Image processing method, device, electronic device and readable storage medium

Abstract

The present application provides an image processing method, device, electronic device and readable storage medium, the method comprising: using a neural network model to obtain a first filter corresponding to a preset first resolution; according to the first filter, rendering a target filter corresponding to a preset target size; using the target filter to process a preset image to be filtered; wherein the preset target size is larger than the preset first size, and the resolution of the image to be filtered is the preset target resolution. In this way, the process of obtaining the target filter is no longer an end-to-end method, but is indirectly obtained based on the low-resolution filter output by the neural network model, so that the coefficients of each intermediate layer can be adjusted. At the same time, since the present application obtains the target filter by rendering upward step by step, the training process is more controllable and overfitting is not prone to occur. In addition, the method of the present application can also save computing resources.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and readable storage medium. Background Art

[0002] To make up for the shortcomings of traditional image processing algorithms, algorithms based on neural networks to improve image quality have been widely used. Compared with traditional algorithms that rely on experience to adjust the filter coefficients to process images, algorithms based on neural networks to improve image quality are data-driven and use training data to learn the filter coefficients.

[0003] The current neural network-based image quality improvement algorithms are all trained using an end-to-end learning method, that is, the training set images are directly input to obtain the output images, and the process of obtaining the filter coefficients in the middle is a black box and cannot be adjusted manually. At the same time, due to the end-to-end processing method, the same computing power is allocated to each pixel in the image for processing, which leads to the current neural network-based image quality improvement algorithm. The computing resource consumption is high. In addition, in the neural network-based image quality improvement algorithm, overfitting of the training data is always inevitable, and as long as there is a slight overfitting, serious artifacts may be generated on the image during testing or application, thereby affecting the image quality. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device and readable storage medium to solve the above-mentioned problems.

[0005] An embodiment of the present application provides an image processing method, including: using a neural network model to obtain a first filter corresponding to a preset first resolution; rendering a target filter corresponding to a preset target resolution based on the first filter; the preset target resolution is greater than the preset first resolution; using the target filter to process a preset image to be filtered; the resolution of the image to be filtered is the preset target resolution.

[0006] In an embodiment of the present application, a first filter corresponding to a preset first size is first obtained through a neural network model, and then a target filter of the target size actually required is obtained by rendering according to the first filter. In this way, a low-resolution filter is rendered to obtain a high-resolution filter by rendering. The process of obtaining the target filter is no longer an end-to-end method, but is indirectly obtained based on a low-resolution filter obtained by training the neural network model, so that the coefficients of each intermediate layer (that is, the coefficients of each filter) can be adjusted, thereby solving the problem that the process of obtaining the filter coefficients in the middle of the traditional algorithm is a black box and cannot be adjusted manually to a certain extent. At the same time, since the present application is a target filter obtained by rendering upward step by step, the training process is more controllable and it is not easy to overfit (even if overfitting occurs, engineers can easily make corrections and adjustments based on filters at all levels, unlike the current end-to-end method, where coefficients cannot be adjusted).

[0007] In addition, by adopting the method of the present application, through the step-by-step upward rendering method, it is necessary to initially obtain only a small-resolution filter, and then render the coefficients of the corresponding high-resolution filter based on the small-resolution filter, so that there is no need to perform the same repeated calculations for each point during the calculation process, thereby saving computing resources.

[0008] Furthermore, a neural network model is used to obtain a first filter corresponding to a preset first resolution, including: inputting a preset input image into the neural network model to obtain a first initial filter; using the first initial filter to filter the input image downsampled to the first resolution to obtain a first filtered image; calculating a loss value between the color of the first filtered image and the color of a preset standard image downsampled to the first resolution; when the loss value has not converged, iteratively updating the parameters of the neural network model until the loss value converges; using a preset network to correct the high-frequency coefficients in the first initial filter when the loss value converges to obtain the first filter.

[0009] In the above implementation process, a neural network model is iterated to obtain a low-resolution first filter as the basis for rendering, so as to ensure that when rendering towards the target filter, the first filter as the basis is a filter that can meet the corresponding image processing requirements, thereby improving the accuracy of the target filter learned by the scheme.

[0010] Furthermore, according to the first filter, a target filter corresponding to a preset target resolution is rendered; including: rendering the first filter to obtain a second initial filter corresponding to a preset second resolution; the preset second resolution is greater than the preset first resolution; using a preset network to correct the high-frequency coefficients in the second initial filter to obtain a corrected second filter; if the preset second resolution is not the preset target resolution, continuing to render the second filter until a target filter corresponding to the preset target resolution is obtained.

[0011] In the above implementation process, the target filter corresponding to the preset target resolution is rendered step by step from the low-resolution filter. The filter coefficient of the target filter depends on the coefficient of each low-resolution filter. Therefore, engineers can debug the algorithm and adjust the target filter by analyzing the coefficients of each low-resolution filter as needed, thereby reducing the risk of serious artifacts on the image during testing or application due to overfitting.

[0012] Furthermore, rendering the first filter to obtain a second initial filter includes: interpolating the first filter according to a multiple of the difference between the preset second resolution and the preset first resolution to obtain the second initial filter.

[0013] In the above implementation process, when rendering upward, rendering upward is performed through a simple interpolation method, which can reduce the amount of calculation and improve learning efficiency.

[0014] Furthermore, the high-frequency coefficients in the second initial filter are corrected using a preset network to obtain a corrected second filter, including: obtaining a first image to be processed when the preset input image is downsampled to the preset second resolution; filtering the first image to be processed using the second initial filter to obtain a second filtered image; extracting high-frequency feature data corresponding to the second filtered image, and regressing the high-frequency feature data using the preset network to obtain high-frequency coefficients after regression; and replacing the high-frequency coefficients in the second initial filter with the high-frequency coefficients after regression to obtain the second filter.

[0015] It should be understood that in an image, the low-frequency part usually represents the internal area of ​​an object, and the color variation of the pixels in the area is not high, while the high-frequency area represents the boundary area between different objects, and the color variation of the pixels in the area is high. However, during rendering, the filter coefficients are upsampled and expanded by methods such as interpolation. Therefore, the filter coefficients obtained in this way will become relatively smooth, performing well in the low-frequency part, but performing poorly in the high-frequency part, so the high-frequency coefficients need to be corrected. In an embodiment of the present application, the high-frequency feature data corresponding to the second filtered image is extracted and regressed, and then the high-frequency coefficients after regression are used to replace the high-frequency coefficients in the first initial filter, so that the first filter can meet the corresponding image processing requirements.

[0016] In addition, in an end-to-end manner, since the input is an unprocessed input image, the output is a processed image. In such a process, all pixels of the input image will undergo the same process (the same convolution process). In the scheme provided in the embodiment of the present application, since the neural network training obtains a coarse-scale (i.e., low-resolution) filter, a filter of the target size is obtained by rendering upward. Therefore, the processing of all pixels of the input image is not treated equally in this process, and through the above scheme, for pixels that do not need to be repeatedly calculated (pixels in low-frequency areas), it can be directly obtained by interpolation, and important pixels (pixels in high-frequency areas) will be recalculated, thereby giving a significantly reduced amount of computation for neural network processing images compared to an end-to-end approach.

[0017] Furthermore, the extracting of high-frequency feature data corresponding to the second filtered image includes: obtaining the position of the high-frequency area in the second filtered image; and extracting the high-frequency feature data corresponding to the position of the high-frequency area in the second filtered image in the feature map output by the neural network model and having a size of the preset second resolution.

[0018] It should be understood that in the process of training the neural network model to obtain the first filter, it is necessary to perform feature extraction step by step to obtain feature maps at each level. Feature maps at each level can be used to learn filters of different resolutions. In an embodiment of the present application, only the high-frequency coefficients need to be corrected for the second initial filter. Therefore, it is only necessary to extract the high-frequency feature data corresponding to the position of the high-frequency area in the second filtered image from the feature map whose size is the preset second resolution output by the neural network model. There is no need to perform neural network processing again, which saves computing resources.

[0019] Furthermore, the preset network is a multi-layer perceptron.

[0020] An embodiment of the present application also provides an image processing device, including: an acquisition module, a rendering module, and a processing module; the acquisition module is used to use a neural network model to acquire a first filter corresponding to a preset first resolution; the rendering module is used to render a target filter corresponding to a preset target resolution based on the first filter; the preset target resolution is greater than the preset first resolution; the processing module is used to use the target filter to process a preset image to be filtered; the resolution of the image to be filtered is the preset target resolution.

[0021] An embodiment of the present application also provides an electronic device, comprising: a processor, a memory and a communication bus; the communication bus is used to realize connection and communication between the processor and the memory; the processor is used to execute one or more programs stored in the memory to realize any of the above-mentioned image processing methods.

[0022] In an embodiment of the present application, a readable storage medium is also provided, wherein the readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement any of the above-mentioned image processing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A basic flow chart of an image processing method provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of a first filter acquisition process provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of a flow chart of a correction process for a second initial filter provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of the structure of a neural network provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of the structure of a vehicle provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0031] Embodiment 1:

[0032] Current neural network-based image processing algorithms are all trained using an end-to-end learning approach, which means that the training set images are directly input to get the output images. The process of obtaining the filter coefficients is a black box and cannot be adjusted manually. Overfitting is very common during neural network training, which results in poor filter coefficients obtained through training and artifacts are easily generated during testing or application.

[0033] In order to solve the problem that end-to-end learning methods are prone to artifacts during testing or application, an indirect learning method based on rendering ideas is proposed in the embodiments of the present application. Figure 1 As shown, Figure 1 An image processing method provided in an embodiment of the present application includes:

[0034] S101: Use a neural network model to obtain a first filter corresponding to a preset first resolution.

[0035] In an embodiment of the present application, a neural network model can be firstly used to iteratively output a first low-resolution filter, so that during rendering, the filter serving as the basis for rendering is a filter that can meet the actual image processing requirements, i.e., the filter coefficients of the filter serving as the basis for rendering are ensured to be credible.

[0036] In the embodiment of the present application, in order to learn and obtain the final target filter, an input image and a standard image corresponding to the input image are pre-set.

[0037] It should be noted that the standard image corresponding to the input image is an image that represents the processing effect of the input image. For example, in an image enhancement algorithm, the standard image is the image after the input image is successfully enhanced.

[0038] When obtaining the first filter, see Figure 2 As shown:

[0039] The preset input image can be input into the neural network model to obtain a first initial filter.

[0040] Assuming that the resolution of the first initial filter is 1 / m of the resolution of the input image, the input image may then be downsampled m times to obtain a target input image with a resolution of 1 / m of the input image.

[0041] The target input image is filtered using a first initial filter to obtain a first filtered image, and a loss value between the color of the first filtered image and the color of a standard image downsampled to a first resolution is calculated.

[0042] If the loss value has not converged, the neural network model parameters are iteratively updated, the first initial filter is updated, and the first initial filter is reused to filter the target input image to obtain a first filtered image, and the loss value between the color of the first filtered image and the color of the standard image downsampled to the first resolution is calculated.

[0043] If the loss value converges, the iteration is terminated, and the high-frequency coefficients in the first initial filter when the loss value converges are corrected using a preset network to obtain a first filter.

[0044] See also Figure 3 The correction process of the first initial filter when the loss value converges includes:

[0045] S301: Obtain the image to be processed.

[0046] It should be understood that in the embodiment of the present application, the preset input image can be downsampled to the first resolution size in advance to obtain the image to be processed.

[0047] S302: Filter the image to be processed using the first initial filter when the loss value converges to obtain a filtered image.

[0048] S303: extract high-frequency feature data corresponding to the filtered image, and use a preset network to regress the high-frequency feature data to obtain high-frequency coefficients after regression.

[0049] It should be understood that the so-called high-frequency feature data refers to feature data that can be used to train the high-frequency coefficients of the filter.

[0050] In an embodiment of the present application, the position of the high-frequency area in the filtered image can be obtained first, and then the high-frequency feature data corresponding to the position of the high-frequency area in the filtered image can be extracted from the feature map with a size of the first resolution output by the neural network model during the aforementioned training.

[0051] In an embodiment of the present application, in order to determine the position of the high-frequency area in the filtered image, a feasible way is to filter the filtered image using a high-frequency filter to obtain the position of the high-frequency area in the filtered image.

[0052] However, usually corresponding to different image processing requirements, the algorithms used are different, the purposes to be achieved are different, and the positioning of the high-frequency area may be different. Therefore, in order to accurately determine the position of the high-frequency area, in an embodiment of the present application, the standard image corresponding to the input image can be downsampled to the first resolution, and then the loss value between each pixel of the filtered image and each pixel of the standard image downsampled to the first resolution is calculated, and then the position of the high-frequency area is determined according to the loss value.

[0053] For example, the position of the pixel whose loss value is greater than the preset threshold can be taken as the position of the high-frequency area. Alternatively, the N pixels with the largest loss value in the filtered image, or the position of the M% pixels with the largest loss value can be taken as the position of the high-frequency area. The values ​​of N and M can be set by engineers according to actual needs.

[0054] It should be understood that in a neural network, a feature map is represented by a matrix. After determining the position of the high-frequency region, the data of the corresponding position can be taken from the feature map matrix of the first resolution according to the position of the high-frequency region to obtain the high-frequency feature data described in the embodiment of the present application.

[0055] In an embodiment of the present application, any regression model can be used to regress the high-frequency feature data. For example, MLP (multi-layer perceptron) can be used to regress the high-frequency feature data to obtain high-frequency coefficients after regression.

[0056] S304: Using the regressed high-frequency coefficients to replace the high-frequency coefficients in the first initial filter, thereby obtaining a first filter.

[0057] The first filter obtained at this time is more reliable in theory because the high-frequency coefficients are regressed and corrected, and the effect is better when rendering upward.

[0058] In order to ensure that the obtained first filter is reliable, in an embodiment of the present application, the first filter can be used to filter the image to be processed of the first resolution, and then the loss value between the color of the filtered first image to be processed and the color of the preset standard image downsampled to the first resolution is calculated.

[0059] If the loss value does not converge, the first filter can be used as the first initial filter and the above Figure 3 The correction process is performed, and the first filter is reused to filter the image to be processed, and then the loss value between the color of the filtered image to be processed and the color of the preset standard image downsampled to the first resolution is calculated.

[0060] If the loss value converges, it is determined that the final first filter is obtained.

[0061] S102: Rendering, according to the first filter, to obtain a target filter corresponding to a preset target resolution.

[0062] It should be noted that, in the embodiment of the present application, rendering refers to the process of calculating a filter with a higher resolution from a low-resolution filter in sequence, which can be achieved through an upsampling operation.

[0063] It should also be noted that in the embodiment of the present application, the resolution of the high-level filter should be higher than the resolution of the low-level filter. For example, the resolution of the target filter should be greater than the resolution of the remaining filters, and the second resolution should be greater than the first resolution.

[0064] It should be understood that in the embodiment of the present application, the target resolution required by the target filter can be determined by engineers according to actual image processing requirements.

[0065] In the embodiment of the present application, the first filter may be rendered to obtain a second initial filter corresponding to a preset second resolution.

[0066] Furthermore, the high frequency coefficients in the second initial filter are corrected using a preset network to obtain a corrected second filter.

[0067] At this time, if the target resolution required by the target filter of this learning is the second resolution, then this learning is completed, and the second filter is the required target filter obtained by learning.

[0068] If the target resolution required by the target filter of this learning is not the second resolution, then the second filter is continued to be rendered to obtain a third initial filter corresponding to the preset third resolution. Then, the high-frequency coefficients in the third initial filter are corrected using the preset network to obtain a corrected third filter.

[0069] If the target resolution required by the target filter learned this time is not the third resolution, the third filter continues to be rendered until a target filter corresponding to the preset target resolution is obtained.

[0070] In the embodiment of the present application, upward rendering can be performed according to the difference in resolution multiples between adjacent levels to obtain an initial filter for the next level.

[0071] For example, in the process of rendering the first filter to obtain the second initial filter, the second initial filter can be obtained by upsampling according to the difference multiple between the preset second resolution and the first resolution.

[0072] For example, the second resolution is 2h*2w, and the first resolution is h*w, so the length and width difference between the second resolution and the first resolution is 2, and the first filter can be upsampled by 2 to obtain the second initial filter.

[0073] In an embodiment of the present application, the resolution corresponding to the filters at each level can be pre-set by an engineer, that is, the engineer can pre-configure how many times rendering is required, the resolution corresponding to the filters at each level, etc.

[0074] In addition, engineers can also configure the rendering multiple and the resolution size of the first filter so that the electronic device can automatically determine whether the filter obtained by the latest rendering and correction is the required target filter. For example, assuming that the size of the configured first filter is 1 / 16 of the input image resolution (1 / 16 in both length and width, the same below), and the target resolution is the resolution of the input image, and the rendering multiple is 2 each time, then at this time, the first, second, and third renderings will obtain filter sizes of 1 / 8, 1 / 4, and 1 / 2 of the input image resolution, respectively. The electronic device will determine that the filter obtained by rendering and correction is not the required target filter and needs to be rendered. The filter size obtained by the fourth rendering and correction is the input image resolution, and the electronic device can determine that the target filter has been obtained, and thus stop rendering.

[0075] It should be noted that in the embodiments of the present application, upsampling can be achieved by interpolation or the like.

[0076] It should be noted that, for the upsampling process, the embodiment of the present application adopts a back-to-forward expansion and reverse processing method instead of a front-to-backward iterative processing process.

[0077] It should be understood that the initial filter obtained by upsampling methods such as interpolation usually performs well in the low-frequency part, but performs poorly in the high-frequency part (this is because the low-frequency part usually represents the area with low color change in the image, such as the internal area of ​​the object, etc., while the high-frequency area usually represents the area with high color change in the image, such as the boundary area between different objects. Therefore, the filter coefficient corresponding to the low-frequency part will be relatively smooth, while the filter coefficient corresponding to the high-frequency part will change dramatically. The filter coefficient obtained by upsampling methods such as interpolation will become relatively smooth, resulting in inaccurate high-frequency coefficients corresponding to the high-frequency area). For this reason, it is necessary to correct the high-frequency coefficients of the initial filter.

[0078] In the embodiment of the present application, the correction of the high frequency coefficients can be achieved by regressing the high frequency coefficients. It should be understood that the method of correcting the high frequency coefficients in each initial filter in the present application is consistent, so the correction process can refer to the high frequency coefficient correction process for the first initial filter mentioned above.

[0079] Exemplarily, the correction process for the second initial filter is as follows:

[0080] First, obtain the first picture to be processed.

[0081] It should be understood that in the embodiment of the present application, the preset input image can be downsampled to the second resolution size in advance to obtain the first image to be processed.

[0082] It should also be understood that for the initial filters of the remaining levels, the input image is downsampled to the resolution size of the corresponding level.

[0083] Then, the first to-be-processed picture is filtered using the second initial filter to obtain a second filtered picture.

[0084] Next, high-frequency feature data corresponding to the second filtered image is extracted, and a preset network is used to regress the high-frequency feature data to obtain high-frequency coefficients after regression.

[0085] It should be understood that the so-called high-frequency feature data refers to feature data that can be used to train the high-frequency coefficients of the filter.

[0086] In an embodiment of the present application, the position of the high-frequency area in the second filtered image can be first obtained, and then the high-frequency feature data corresponding to the position of the high-frequency area in the second filtered image can be extracted from the feature map with a size of the second resolution output by the neural network model.

[0087] It should be understood that in the process of training the neural network model to obtain the first filter, it is necessary to extract features step by step to obtain feature maps at each level. Figure 4 The neural network model shown in the figure consists of 4 nonlinear convolution units (conv+bn+relu, conv represents convolution, bn represents batch normalization, and relu represents activation function) and a 1*n*n*3 convolution layer (the first filter is an n*n*3 filter), and the step size of conv in the 4 nonlinear convolution units is 2. In this way, the length and width of the feature map output by the first nonlinear convolution unit are both 1 / 2 of the input image; the length and width of the feature map output by the second nonlinear convolution unit are both 1 / 2 of the feature map output by the first nonlinear convolution unit, that is, 1 / 4 of the input image; the length and width of the feature map output by the third nonlinear convolution unit are both 1 / 8 of the input image; the length and width of the feature map output by the fourth nonlinear convolution unit are both 1 / 16 of the input image. Figure 4 You will get 4 feature maps of different sizes.

[0088] Therefore, based on the feature map of the second resolution obtained when training the first filter, the high-frequency feature data corresponding to the position of the high-frequency area in the second filtered image can be directly extracted, without the need to reuse the neural network model to extract features from the input image to obtain the feature map of the second resolution.

[0089] Of course, the solution of reusing the neural network model to extract features from the input image to obtain a feature map of the second resolution can also be adopted by the embodiments of the present application.

[0090] Finally, the high-frequency coefficients after regression are used to replace the high-frequency coefficients in the second initial filter, thereby obtaining the second filter.

[0091] Similarly, in order to ensure that the obtained second filter is reliable, in an embodiment of the present application, a second filter can be used to filter the first image to be processed, and then the loss value between the color of the filtered first image to be processed and the color of the preset standard image downsampled to the second resolution is calculated.

[0092] If the loss value has not converged, the second filter can be used as the second initial filter, the above correction process can be re-executed, and the second filter can be used again to filter the first image to be processed, and then the loss value between the color of the filtered first image to be processed and the color of the preset standard image downsampled to the second resolution is calculated.

[0093] If the loss value converges, it is determined that the final second filter is obtained.

[0094] In the embodiment of the present application, a target filter that meets the image processing requirements can be learned through the above method, and then the target filter can be used to perform the corresponding image processing task.

[0095] It should be noted that the loss value convergence described in the embodiments of the present application may be that the loss value is less than a preset loss value threshold, or that the change in the loss value is lower than a preset change value threshold.

[0096] It should be noted that the target resolution corresponding to the target filter obtained by upward rendering in the embodiment of the present application may be different from the resolution of the input image, and of course it may be equal to the resolution of the input image. The specific size of the target resolution is limited by the algorithm actually applied to the target filter. For example, for the demosaic algorithm, the target resolution may be equal to the resolution of the input image.

[0097] S103: Using the target filter to process the preset image to be filtered.

[0098] It should be understood that the image to be filtered described in the embodiments of the present application is an image that needs to be processed after the target filter is obtained and adapted to the processing capability of the target filter.

[0099] It should be noted that the so-called adaptation to the processing capability of the target filter means that the resolution of the image to be filtered is the target resolution corresponding to the target filter, and the processing function to be implemented is also the function of the target filter.

[0100] It should be understood that the function of the target filter is adapted to the aforementioned acquisition process. For example, in the process of obtaining the target filter, the standard picture used is the picture after the input picture is successfully enhanced, then the standard picture is the picture after the input picture is successfully enhanced as the image enhancement function, and at this time, the target filter is used to process the picture to be filtered, which will enhance the display effect of the picture to be filtered.

[0101] The image processing method provided in the embodiment of the present application obtains a first filter corresponding to a preset first size by training with a preset neural network model, and then obtains a target filter of the actually required target size by rendering according to the first filter. In this way, a low-resolution filter is rendered to obtain a high-resolution filter by multi-layer filtering rendering. The process of obtaining the target filter is no longer an end-to-end method, but is indirectly obtained based on the low-resolution filter output by the neural network model. Engineers can implement correction and adjustment of the filter coefficients of the target filter based on filters at all levels, and engineers have the ability to correct and adjust the filter coefficients of the target filter.

[0102] At the same time, due to the existence of filters at various levels, it is also convenient to analyze and debug the entire image processing algorithm.

[0103] In addition, since it is an indirect learning method based on rendering, the obtained target filter is not prone to overfitting. Even if overfitting occurs, engineers can easily make corrections and adjustments based on filters at each level, unlike the current end-to-end method, which cannot adjust coefficients.

[0104] In addition, in an end-to-end manner, since the input is an unprocessed input image, the output is a processed image. In such a process, all pixels of the input image will undergo the same process (the same convolution process). In the scheme provided in the embodiment of the present application, since the neural network training obtains a coarse-scale (i.e., low-resolution) filter, a filter of the target size is obtained by rendering upward. Therefore, the processing of all pixels of the input image is not treated equally in this process. In the scheme of the embodiment of the present application, pixels that do not need to be repeatedly calculated (such as pixels in low-frequency areas) can be directly obtained by interpolation, and important pixels (such as pixels in high-frequency areas) will be recalculated, thereby giving a significantly reduced amount of calculation of the neural network processing image compared to the end-to-end approach.

[0105] In addition, the image processing method provided in the embodiment of the present application can be widely applied to various image processing algorithms to realize the training of filters in the algorithm, thereby improving the reliability of each image processing algorithm.

[0106] Embodiment 2:

[0107] This embodiment is based on the first embodiment and takes the application in the demosaic algorithm as an example to illustrate the solution of the present application.

[0108] Let the input image be raw, let the length and width of raw be h*w; let the standard image RAW corresponding to raw be h*w.

[0109] In the demosaic algorithm, the target resolution is the resolution of the input image, that is, h*w.

[0110] Neural network structure see Figure 4 As shown in the figure, the number of channels in the last convolutional layer is set to 1*n*n*3.

[0111] Input raw into the neural network, which outputs the first initial filter of (h / 16)*(w / 16)*n*n*3.

[0112] The first initial filter is used to filter the image raw1 that is downsampled to (h / 16)*(w / 16) to obtain a filtered image r1.

[0113] The loss is calculated for the filtered image r1 and the image RAW1 which is RAW downsampled to (h / 16)*(w / 16).

[0114] If the loss value does not converge, update the neural network parameters, re-output the first initial filter of (h / 16)*(w / 16)*n*n*3, use the new first initial filter to filter raw1, and obtain a new filtered image r1. Calculate the loss of the new filtered image r1 and RAW1. If the loss value does not converge, repeat the above process.

[0115] If the loss value converges, raw1 is filtered using the first initial filter when the loss value converges to obtain the filtered image r11.

[0116] Calculate the loss between image r11 and RAW1.

[0117] According to the position of the pixel point set A1 whose loss value is greater than the preset threshold in the image r11, the position of the pixel point set A1 in the feature map output by the fourth nonlinear convolution unit of the neural network is determined, and the value corresponding to the pixel point set A1 in the feature map matrix is ​​extracted.

[0118] Use MLP to regress the numerical set, and replace the data at the corresponding position in the filter matrix corresponding to the first initial filter with the regressed numerical values ​​according to the position of each numerical value in the feature map matrix, so as to correct the high-frequency coefficients of the first initial filter and obtain the first filter.

[0119] The first filter is upsampled twice to obtain a second initial filter of (h / 8)*(w / 8)*n*n*3. The filter coefficients of the second initial filter that are more than those of the first filter are obtained by interpolating the filter coefficients of the first filter.

[0120] The second initial filter is used to filter the image raw2 that is downsampled to (h / 8)*(w / 8) to obtain a filtered image r2.

[0121] Calculate the loss value between the image r2 and the image RAW2 which is RAW downsampled to (h / 8)*(w / 8).

[0122] According to the position of the pixel point set A2 whose loss value is greater than the preset threshold in the image r2, the position of the pixel point set A2 in the feature map output by the third nonlinear convolution unit of the neural network is determined, and the value corresponding to the pixel point set A2 in the feature map matrix is ​​extracted.

[0123] Use MLP to regress the numerical set, and replace the data at the corresponding position in the filter matrix corresponding to the second initial filter with the regressed numerical values ​​according to the position of each numerical value in the feature map matrix, so as to correct the high-frequency coefficients of the second initial filter and obtain the second filter.

[0124] The second filter is upsampled twice to obtain a third initial filter of (h / 4)*(w / 4)*n*n*3. The filter coefficients of the third initial filter that are more than those of the second filter are obtained by interpolating the filter coefficients of the second filter.

[0125] The third initial filter is used to filter the image raw3 that is downsampled to (h / 4)*(w / 4) to obtain a filtered image r3.

[0126] Calculate the loss value between the image r3 and the image RAW3 which is obtained by downsampling RAW to (h / 4)*(w / 4).

[0127] According to the position of the pixel point set A3 whose loss value is greater than the preset threshold in the image r3, the position of the pixel point set A3 in the feature map output by the second nonlinear convolution unit of the neural network is determined, and the value corresponding to the pixel point set A3 in the feature map matrix is ​​extracted.

[0128] Use MLP to regress the numerical set, and replace the data at the corresponding position in the filter matrix corresponding to the third initial filter with the regressed numerical values ​​according to the position of each numerical value in the feature map matrix, so as to correct the high-frequency coefficients of the third initial filter and obtain the third filter.

[0129] The third filter is upsampled twice to obtain a fourth initial filter of (h / 2)*(w / 2)*n*n*3. The filter coefficients of the fourth initial filter that are more than those of the third filter are obtained by interpolating the filter coefficients of the third filter.

[0130] The fourth initial filter is used to filter the image raw4 that is downsampled to (h / 2)*(w / 2) to obtain a filtered image r4.

[0131] Calculate the loss value between the image r4 and the image RAW4 which is obtained by downsampling RAW to (h / 2)*(w / 2).

[0132] According to the position of the pixel point set A4 whose loss value is greater than the preset threshold in the image r4, the position of the pixel point set A4 in the feature map output by the first nonlinear convolution unit of the neural network is determined, and the value corresponding to the pixel point set A4 in the feature map matrix is ​​extracted.

[0133] Use MLP to regress the numerical set, and replace the data at the corresponding position in the filter matrix corresponding to the fourth initial filter with the regressed numerical values ​​according to the position of each numerical value in the feature map matrix, so as to correct the high-frequency coefficients of the fourth initial filter and obtain the fourth filter.

[0134] The fourth filter is upsampled twice to obtain a target initial filter of h*w*n*n*3. The part of the filter coefficients of the target initial filter that are more than those of the fourth filter is obtained by interpolating the filter coefficients of the fourth filter.

[0135] Use the target initial filter to filter raw and obtain the filtered image r5.

[0136] Calculate the loss value between image r5 and RAW.

[0137] According to the position of the pixel point set A5 whose loss value is greater than the preset threshold in the image r5, the position of the pixel point set A5 in the feature map output by the first nonlinear convolution unit of the neural network is determined, and the value corresponding to the pixel point set A5 in the feature map matrix is ​​extracted.

[0138] Use MLP to regress the numerical set, and replace the data at the corresponding position in the filter matrix corresponding to the target initial filter with the regressed numerical values ​​according to the position of each numerical value in the feature map matrix, so as to correct the high-frequency coefficients of the target initial filter and obtain the target filter.

[0139] Finally, apply the target filter to the original image raw to get the final output result.

[0140] The solution of the embodiment of the present application abandons the traditional end-to-end design concept and renders the learned filters layer by layer into high-resolution filters. In this way, since it is an indirect learning method based on rendering, the obtained target filter is not prone to overfitting, so it is less likely to have serious artifacts. In addition, since the target filter is rendered to high resolution, computing power will not be wasted on a large number of pixels that do not require excessive processing (such as low-frequency areas). In addition, the coefficients of each filter can be consulted by engineers, which is also convenient for analyzing and debugging algorithms.

[0141] Embodiment three:

[0142] Based on the same inventive concept, an image processing device is also provided in the embodiment of the present application. Figure 5 As shown, Figure 5 The image processing device 100 corresponding to the method shown in the first embodiment is shown. It should be understood that the specific functions of the image processing device 100 can be referred to the description above, and the detailed description is appropriately omitted here to avoid repetition. The image processing device 100 includes at least one software function module that can be stored in a memory in the form of software or firmware or fixed in the operating system of the image processing device 100. Specifically:

[0143] See also Figure 5As shown, the image processing device 100 includes: an acquisition module 101, a rendering module 102, and a processing module 103. Among them:

[0144] The acquisition module 101 is used to train a preset neural network model to obtain a first filter corresponding to a preset first resolution.

[0145] The rendering module 102 is used to render a target filter corresponding to a preset target resolution according to the first filter; the preset target resolution is greater than the preset first resolution.

[0146] The processing module 103 is used to process a preset image to be filtered using the target filter; the resolution of the image to be filtered is the preset target resolution.

[0147] In an embodiment of the present application, the acquisition module 101 is specifically used to input a preset input image into the neural network model to obtain a first initial filter; use the first initial filter to filter the input image downsampled to the first resolution to obtain a first filtered image; calculate the loss value between the color of the first filtered image and the color of the preset standard image downsampled to the first resolution; when the loss value has not converged, iteratively update the neural network model parameters until the loss value converges; use the preset network to correct the high-frequency coefficients in the first initial filter when the loss value converges to obtain the first filter.

[0148] In an embodiment of the present application, the rendering module 102 is specifically used to render the first filter to obtain a second initial filter corresponding to a preset second resolution; the preset second resolution is greater than the preset first resolution; the high-frequency coefficients in the second initial filter are corrected using a preset network to obtain a corrected second filter; if the preset second resolution is not the preset target resolution, continue to render the second filter until a target filter corresponding to the preset target resolution is obtained.

[0149] In a feasible implementation manner of the embodiment of the present application, the rendering module 102 is specifically configured to interpolate the first filter according to a multiple of the difference between the preset second resolution and the preset first resolution to obtain the second initial filter.

[0150] In another feasible implementation of the embodiment of the present application, the rendering module 102 is specifically used to obtain a first image to be processed when a preset input image is downsampled to the preset second resolution; use the second initial filter to filter the first image to be processed to obtain a second filtered image; extract high-frequency feature data corresponding to the second filtered image, and use the preset network to regress the high-frequency feature data to obtain high-frequency coefficients after regression; use the regressed high-frequency coefficients to replace the high-frequency coefficients in the second initial filter to obtain the second filter.

[0151] In the above feasible implementation manner, the position of the high-frequency area in the second filtered image is obtained; in the feature map output by the neural network model with a size of the preset second resolution, the high-frequency feature data corresponding to the position of the high-frequency area in the second filtered image is extracted.

[0152] In the above feasible implementation manner, the preset network is a multi-layer perceptron.

[0153] It should be understood that, for the sake of brevity, some of the contents described in the first embodiment will not be repeated in this embodiment.

[0154] Embodiment 4:

[0155] This embodiment provides an electronic device, which can be seen in Figure 6 As shown, it includes a processor 601, a memory 602 and a communication bus 603. Among them:

[0156] The communication bus 603 is used to realize the connection and communication between the processor 601 and the memory 602 .

[0157] The processor 601 is used to execute one or more programs stored in the memory 602 to implement the image processing method in the above-mentioned embodiment 1 / 2.

[0158] Understandably, Figure 6 The structure shown is for illustration only. The electronic device may also include Figure 6 More or fewer components as shown, or with Figure 6 The different configurations shown may also include components such as a keyboard, a communication module, and a display screen.

[0159] This embodiment also provides a readable storage medium, such as a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash disk, an SD (Secure Digital Memory Card) card, an MMC (Multimedia Card) card, etc., in which one or more programs for implementing the above steps are stored, and the one or more programs can be executed by one or more processors to implement the image processing method in the above embodiment 1 / 2. No further details will be given here.

[0160] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0161] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0163] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0164] As used herein, a plurality refers to two or more than two.

[0165] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: include: Using a neural network model to obtain a first filter corresponding to a preset first resolution; According to the first filter, rendering is performed to obtain a target filter corresponding to a preset target resolution; The preset target resolution is greater than the preset first resolution; Using the target filter to process the preset image to be filtered; The resolution of the image to be filtered is the preset target resolution; Using a neural network model to obtain a first filter corresponding to a preset first resolution includes: Inputting a preset input image into the neural network model to obtain a first initial filter; Using the first initial filter to filter the input image downsampled to the first resolution, to obtain a first filtered image; Calculating a loss value between a color of the first filtered image and a color of a preset standard image downsampled to the first resolution; When the loss value has not converged, iteratively updating the parameters of the neural network model until the loss value converges; The high-frequency coefficients in the first initial filter when the loss value converges are corrected using a preset network to obtain the first filter.

2. The image processing method according to claim 1, characterized in that: According to the first filter, rendering is performed to obtain a target filter corresponding to a preset target resolution; comprising: Rendering the first filter to obtain a second initial filter corresponding to a preset second resolution; the preset second resolution is greater than the preset first resolution; Correcting the high frequency coefficients in the second initial filter using a preset network to obtain a corrected second filter; If the preset second resolution is not the preset target resolution, the second filter continues to be rendered until a target filter corresponding to the preset target resolution is obtained.

3. The image processing method according to claim 2, characterized in that: Rendering the first filter to obtain a second initial filter includes: The first filter is interpolated according to the difference multiple between the preset second resolution and the preset first resolution to obtain the second initial filter.

4. The image processing method according to claim 2, wherein: Correcting the high frequency coefficients in the second initial filter using a preset network to obtain a corrected second filter includes: Obtaining a first image to be processed when a preset input image is downsampled to the preset second resolution; Using the second initial filter to filter the first to-be-processed picture to obtain a second filtered picture; Extracting high-frequency feature data corresponding to the second filtered image, and regressing the high-frequency feature data using the preset network to obtain high-frequency coefficients after regression; The high-frequency coefficients after regression are used to replace the high-frequency coefficients in the second initial filter to obtain the second filter.

5. The image processing method according to claim 4, characterized in that: The extracting high-frequency feature data corresponding to the second filtered image includes: Obtaining a position of a high-frequency area in the second filtered image; In the feature map output by the neural network model with a size of the preset second resolution, the high-frequency feature data corresponding to the position of the high-frequency area in the second filtered image is extracted.

6. The image processing method according to claim 4, characterized in that: The preset network is a multi-layer perceptron.

7. An image processing device, characterized in that: include: Acquisition module, rendering module, and processing module; The acquisition module is used to acquire a first filter corresponding to a preset first resolution using a neural network model; The rendering module is used to render a target filter corresponding to a preset target resolution according to the first filter; The preset target resolution is greater than the preset first resolution; The processing module is used to process a preset image to be filtered using the target filter; The resolution of the image to be filtered is the preset target resolution; The acquisition module is specifically used to: input a preset input image into the neural network model to obtain a first initial filter; use the first initial filter to filter the input image downsampled to the first resolution to obtain a first filtered image; calculate the loss value between the color of the first filtered image and the color of the preset standard image downsampled to the first resolution; when the loss value has not converged, iteratively update the neural network model parameters until the loss value converges; use the preset network to correct the high-frequency coefficients in the first initial filter when the loss value converges to obtain the first filter.

8. An electronic device, characterized in that: include: Processor, memory and communication bus; The communication bus is used to realize the connection and communication between the processor and the memory; The processor is used to execute one or more programs stored in the memory to implement the image processing method according to any one of claims 1 to 6.

9. A readable storage medium, characterized in that: The readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the image processing method according to any one of claims 1 to 6.

Images