An image denoising model generation method, an image denoising method, an image denoising device, a storage medium, and an image denoising apparatus

Abstract

The application provides an image denoising model generation method and an image denoising method. N first Raw images of different Bayer arrays in the same scene are obtained, the N first Raw images of different Bayer arrays are fused to obtain N corresponding second Raw images, the first Raw image is subtracted from the corresponding second Raw image pixel by pixel to obtain a corresponding residual noise image, a sample image database is established based on the first Raw image and the corresponding residual noise image, model training is performed, and an image denoising model is obtained. The image denoising model is input into the above image denoising model after format conversion processing of a to-be-processed Raw image, a residual noise image is obtained, then the residual noise image and the corresponding input image are fused to obtain a first denoising image, and corresponding format conversion processing is performed on the first denoising image to obtain a final denoising image. The image denoising model generated based on the application realizes automatic image denoising, can obviously reduce picture noise, and the image denoising method has high efficiency, good universality and strong robustness.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to an image denoising model generation method, an image denoising method, a device, a storage medium and equipment. BACKGROUND

[0002] Whether it is a CCD or a CMOS chip, high temperature will inevitably be generated during work. If the temperature of the chip is increased, the noise signal is too strong, and color speckles with different brightness will be formed on the picture, which is more obvious in the dark area. These points are the noise points. The generation of thermal noise is because the thermal current is superimposed on the normal signal current, so that the signal current of part of the pixel points is greater than the normal sensing current, finally leading to the signal intensity on this pixel point being stronger, which is shown as higher brightness on the Raw image. After the color processing of the rear-end ISP, such pixel points will lead to noise problems such as deeper or brighter colors.

[0003] At present, for the denoising of these noise points, the existing Raw image denoising method can generally only be used for denoising processing of Raw images with the same Bayer array under the same Sensor parameter condition, but Raw images generally have multiple Bayer patterns, and the parameters of the Sensor of different cameras are also different, leading to poor generality of the Raw denoising realized on the existing denoising method. In addition, the Raw image generally needs to be processed by the ISP into a visible Raw image, and the existing denoising method generally uses SSIM and other objective indexes as a reference on the Raw image itself, which leads to the existing denoising method tending to smooth the Raw image, resulting in loss of clarity of the Raw image.

[0004] In summary, the existing Raw image denoising method has the problems of poor generality and loss of clarity, and it is necessary to improve on this basis. SUMMARY

[0005] The embodiments of the present application provide an image denoising model generation method, an image denoising method, a device, a storage medium and equipment, which are used to solve the problems of poor generality and loss of clarity of the existing Raw image denoising method.

[0006] In a first aspect, the embodiments of the present application provide an image denoising model generation method, which comprises the following steps:

[0007] Obtain N first Raw images of different Bayer arrays under the same scene, N being an integer greater than or equal to 2;

[0008] Respectively fuse the N first Raw images of different Bayer arrays to obtain N second Raw images corresponding thereto;

[0009] subtracting the second Raw image corresponding to the first Raw image pixel by pixel to obtain a corresponding residual noise image;

[0010] establishing a sample image database based on the first Raw image and the residual noise image;

[0011] performing model training based on the sample image database to obtain an image denoising model.

[0012] In a possible implementation manner, the first Raw images of the N different Bayer arrays are fused respectively to obtain N second Raw images, and the specific process includes the following steps.

[0013] selecting any one of the N first Raw images as a reference image in sequence;

[0014] detecting difference points of the reference image and other N-1 first Raw images respectively;

[0015] performing corresponding processing on the reference image according to the difference points to obtain a to-be-fused image;

[0016] performing pixel-by-pixel weighted fusion processing on the to-be-fused image to obtain a corresponding second Raw image.

[0017] In a possible implementation manner, the difference points of the reference image and other N-1 first Raw images are detected respectively, and the specific process includes the following steps.

[0018] dividing the N first Raw images into G grids, wherein G is an integer greater than or equal to 2;

[0019] calculating difference points of each grid content of the reference image and corresponding grid content of other N-1 first Raw images by using an alignment algorithm.

[0020] In a possible implementation manner, the corresponding processing on the reference image according to the difference points to obtain a to-be-fused image includes the following steps.

[0021] if it is detected that the difference points are caused by movement in a chaotic direction, deleting corresponding grids in the N first Raw images to obtain N to-be-fused images;

[0022] if it is detected that the difference points are caused by movement in a single direction, performing translation on the corresponding grids of the N-1 first Raw images relative to the reference image in the opposite direction of the single direction, and taking the N-1 translated images and the reference image as N to-be-fused images.

[0023] With reference to the first aspect, in a possible implementation manner, based on the first Raw image and the residual noise image, a sample image database is established, specifically including:

[0024] If the Bayer arrays of the first Raw image and the residual noise image are inconsistent, the first Raw image and the residual noise image are converted to obtain the first Raw image and the residual noise image with consistent Bayer arrays.

[0025] With reference to the first aspect, in a possible implementation manner, based on the sample image database, model training is performed to obtain an image denoising model, specifically including:

[0026] The first Raw image with consistent Bayer arrays is taken as input image data, and the residual noise image with consistent Bayer arrays corresponding to the first Raw image is taken as corresponding output image data, and model training is performed to obtain an image denoising model.

[0027] The second aspect, the embodiment of the application provides an image denoising method, the method comprises the following steps:

[0028] Obtaining a to-be-processed Raw image;

[0029] According to the Bayer array of the to-be-processed Raw image and a preset image denoising model, a corresponding input image is obtained;

[0030] The input image is input into a corresponding image denoising model for denoising processing to obtain a residual noise image;

[0031] The residual noise image and the corresponding input image are pixel-by-pixel fused to obtain a first denoised image;

[0032] The preset image denoising model is obtained by the image denoising model generation method of any one of the first aspect.

[0033] With reference to the second aspect, in a possible implementation manner, according to the Bayer array of the to-be-processed Raw image and a preset image denoising model, a corresponding input image is obtained, specifically including:

[0034] It is judged whether the Bayer array of the to-be-processed Raw image is consistent with the Bayer array of the preset image denoising model;

[0035] If the Bayer array of the to-be-processed Raw image is inconsistent with the Bayer array of the preset image denoising model, the to-be-processed Raw image is converted to obtain a corresponding input image;

[0036] if the to-be-processed Raw image is consistent with the Bayer array of the preset image denoising model, the to-be-processed Raw image is taken as the corresponding input image;

[0037] wherein the input image is consistent with the Bayer array of the corresponding image denoising model.

[0038] With reference to the second aspect, in a possible implementation manner, the residual noise image and the corresponding input image are subjected to pixel-by-pixel fusion processing to obtain a first denoising image, and the first denoising image is subjected to corresponding processing according to a judgment result to obtain a final denoising image corresponding to the to-be-processed Raw image.

[0039] The residual noise image and the corresponding input image are subjected to pixel-by-pixel weighted fusion processing to generate a first denoising image.

[0040] With reference to the second aspect, in a possible implementation manner, after the residual noise image and the corresponding input image are subjected to pixel-by-pixel fusion processing to obtain a first denoising image, the method further includes:

[0041] determining whether the first denoising image is consistent with the Bayer array of the corresponding to-be-processed Raw image;

[0042] According to the judgment result, the first denoising image is subjected to corresponding processing to obtain a final denoising image corresponding to the to-be-processed Raw image.

[0043] With reference to the second aspect, in a possible implementation manner, according to the judgment result, the first denoising image is subjected to corresponding processing to obtain a final denoising image corresponding to the to-be-processed Raw image, and the corresponding processing includes:

[0044] if the first denoising image is not consistent with the Bayer array of the corresponding to-be-processed Raw image, the first denoising image is subjected to conversion processing to obtain a final denoising image consistent with the Bayer array of the corresponding to-be-processed Raw image;

[0045] if the first denoising image is consistent with the Bayer array of the corresponding to-be-processed Raw image, the first denoising image is taken as a final denoising image of the corresponding to-be-processed Raw image.

[0046] A third aspect, an image denoising model generation device is provided, and the device includes:

[0047] an acquisition module configured to acquire N first Raw images of different Bayer arrays in a same scene, N being an integer greater than or equal to 2;

[0048] a first processing module configured to subject the N first Raw images of different Bayer arrays to fusion processing respectively to obtain N second Raw images corresponding respectively.

[0049] a second processing module configured to subtract the second Raw image corresponding to the first Raw image pixel by pixel to obtain a residual noise image corresponding thereto;

[0050] a third processing module configured to establish a sample image database based on the first Raw image and the residual noise image;

[0051] a training module configured to perform model training based on the sample image database to obtain an image denoising model.

[0052] In a fourth aspect, an embodiment of the present application provides an image denoising device, and the device comprises:

[0053] an acquisition module configured to acquire a Raw image to be processed;

[0054] a first processing module configured to obtain an input image corresponding to the Raw image to be processed and a Bayer array of a preset image denoising model;

[0055] a second processing module configured to input the input image into a corresponding image denoising model to perform denoising processing to obtain a residual noise image;

[0056] a third processing module configured to perform pixel-by-pixel fusion processing on the residual noise image and the input image corresponding thereto to obtain a first denoised image;

[0057] The preset image denoising model is obtained by the image denoising model generation method of any one of the first aspect.

[0058] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is characterized in that when the computer program is executed by a processor, the steps of the method of any one of the first aspect or the second aspect are implemented.

[0059] In a sixth aspect, an embodiment of the present application provides an electronic device, and the electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor is characterized in that when the processor executes the computer program, the steps of the method of any one of the first aspect or the second aspect are implemented.

[0060] The application provides an image denoising model generation method and an image denoising method. N first Raw images of different Bayer arrays in the same scene are obtained, the N first Raw images of different Bayer arrays are respectively fused to obtain N second Raw images, the first Raw images are subtracted from the corresponding second Raw images pixel by pixel to obtain residual noise images, a sample image database is established based on the first Raw images and the corresponding residual noise images, model training is performed based on the sample image database to obtain an image denoising model, the image denoising model is input into the above image denoising model after format conversion processing of a to-be-processed Raw image, a residual noise image is obtained, then the residual noise image and the corresponding input image are fused to obtain a first denoising image, and corresponding format conversion processing is performed on the first denoising image to obtain a final denoising image. The image denoising model generated based on the application realizes automatic image denoising, can obviously reduce picture noise, improve the sharpening degree of the image, make the image observation more comfortable, make a small-size image sensor obtain a picture quality similar to a larger-size sensor, and the image denoising method has high efficiency, good universality and strong robustness. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is a flowchart of an image denoising model generation method provided by an embodiment of the application;

[0062] Figure 2 is a flowchart of an image denoising method provided by an embodiment of the application;

[0063] Figure 3 is a structural schematic diagram of an image denoising model generation device provided by an embodiment of the application;

[0064] Figure 4 is a structural schematic diagram of an image denoising device provided by an embodiment of the application;

[0065] Figure 5 is a structural schematic diagram of an electronic device provided by an embodiment of the application. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical scheme and beneficial effects of the application clearer, the application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0067] In the prior art, the Raw image is original data converted by a CMOS or CCD image sensor from a light source signal captured into a digital signal, is lossless, and contains original color information of an object. A data format of the Raw image generally adopts a Bayer array arrangement mode. A color filter array (CFA) is generated through a filter sheet. Since the human eye is more sensitive to green waveband color, the Bayer array data format contains 50% green information and 25% red and blue information. The Bayer array is a 4x4 array, which is composed of 8 green, 4 blue and 4 red pixels. When converting a gray scale pattern into a color picture, 9 operations are performed in a 2x2 matrix, and finally a color pattern is generated. The Bayer array generally has four formats: RGGB (a), BGGR (b), GBRG (c), and GRBG (d).

[0068] The specific implementation of the present application is described in detail below in combination with specific embodiments.

[0069] Example One:

[0070] Figure 1 An implementation process of an image denoising model generation method provided by an embodiment of the present application is shown. The image denoising model generation method provided by the embodiment of the present application can be applied to a computing device, which can be but is not limited to various personal computers, notebook computers, smart phones, tablet computers and panoramic cameras.

[0071] For ease of illustration, Figure 1 Only parts related to the embodiments of the present application are shown, and are described in detail as follows.

[0072] As Figure 1 shown, the image denoising model generation method includes the following steps:

[0073] S101, acquiring N first Raw images of different Bayer arrays under the same scene;

[0074] The first Raw image can be acquired by any device with a shooting function, such as a digital camera, a panoramic camera, a mobile phone, a tablet computer, a sports camera, etc. In addition, the first Raw image can also be a Raw image obtained through any image processing mode, such as image transformation, image stitching, image segmentation, image synthesis, image compression, image enhancement, image restoration, etc. In addition, the first Raw image can be a panoramic image or a normal planar image.

[0075] In order to obtain the N first Raw images of different Bayer arrays in the same scene, a plurality of manners can be used, for example: in a first manner, a plurality of Raw images in the same scene can be continuously captured by one shooting device, or a plurality of Raw images in the same scene can be captured by a plurality of shooting devices, and then N first Raw images with consistent Bayer arrays can be selected from the plurality of Raw images; in a second manner, N Raw images in the same scene can be continuously captured by one shooting device, or a plurality of Raw images in the same scene can be captured by a plurality of shooting devices, and then N first Raw images with consistent Bayer arrays can be obtained through a Bayer array conversion method; in a third manner, N first Raw images of different Bayer arrays in the same scene can be continuously captured by one shooting device or a plurality of shooting devices. It should be understood that the N first Raw images of different Bayer arrays in the same scene can also be obtained by using different combinations of the above-mentioned manners or by using any other manner, and thus enumeration is not performed here.

[0076] fusing the N first Raw images of different Bayer arrays respectively to obtain N second Raw images corresponding to the N first Raw images respectively;

[0077] In one embodiment, S102 specifically includes the following steps:

[0078] S1021: any one of the N first Raw images is selected as a reference image in turn;

[0079] S1022: differences between the reference image and other N-1 first Raw images are detected respectively;

[0080] S1023: the reference image is processed according to the differences to obtain a to-be-fused image;

[0081] S1024: the to-be-fused image is subjected to pixel-by-pixel weighted fusion processing to obtain a corresponding second Raw image.

[0082] In one embodiment, in step S1021, any one of the N first Raw images is selected as a reference image in turn, and one manner that can be used is that the N first Raw images are sequentially marked as a first image, a second image, a third image,..., and an Nth image, and then the first image, the second image, the third image,..., and the Nth image are sequentially selected as reference images respectively.

[0083] In one embodiment, in step S1022, differences between the reference image and other N-1 first Raw images are detected respectively, and specifically includes the following steps:

[0084] The N first Raw images are each divided into G grids, where G is an integer greater than or equal to 2;

[0085] The difference points of each grid content of the reference image and the corresponding grid content of the other N-1 first Raw images are calculated by using an alignment algorithm.

[0086] In an embodiment, the N first Raw images are each divided into G grids, where G is an integer greater than or equal to 2. The image can be divided into multiple grids in multiple ways. Common grids can be divided into two categories: structured grid and unstructured grid. The connection between the unit nodes of the structured grid has only a limited number of ways. All internal nodes in the grid area generated by the structured grid have the same number of adjacent units and the same number of adjacent nodes, corresponding to a two-dimensional plane or a three-dimensional curved surface. The grid unit generated by the structured grid is generally a quadrilateral, and the three-dimensional entity space is mainly a hexahedron. The nodes between the grid units of the unstructured grid can be connected in any form. The unstructured grid does not need to have the same adjacent units. The number of connections between different nodes in the grid area can be different. The grid unit generated by the unstructured grid is generally a triangle, and the three-dimensional entity space is mainly a tetrahedron. Common structured grid generation methods generally include mapping method and geometric decomposition method. The unstructured grid generation method generally includes octree method, quadtree method, and Delaunay triangulation method.

[0087] It should be understood that the above is only a common grid generation method. The embodiments of the present application are not limited thereto. The N first Raw images can be divided into G grids by using any one or more of the above methods, or other grid generation methods in the prior art. Details are not described herein.

[0088] Further, it should be understood that the G grids divided can be of any shape, such as an AXA grid or an AXB grid, where A and B are not equal and are both integers greater than or equal to 2.

[0089] In an embodiment, the difference points of each grid content of the reference image and the corresponding grid content of the other N-1 first Raw images are calculated by using an alignment algorithm. The alignment algorithm can be any image alignment algorithm, for example, a method based on pixel value cross-correlation, a sequential similarity detection algorithm, a phase correlation algorithm based on Fourier transform, a mutual information-based alignment algorithm, and an image alignment method based on optimization, such as Lucas-Kanade algorithm and improved algorithm thereof.

[0090] In one embodiment, in step S1023, according to the difference points, the reference image is processed to obtain the to-be-fused images, specifically including the following steps:

[0091] If it is detected that the difference points are caused by movement in a random direction, the corresponding grid in the N first Raw images is deleted to obtain N to-be-fused images.

[0092] If it is detected that the difference points are caused by movement in a single direction, the grid corresponding to the N-1 first Raw images is translated in the opposite direction of the single direction relative to the reference image, and the N-1 translated images and the reference image are taken as N to-be-fused images.

[0093] In one embodiment, the difference caused by movement in a random direction is relative to the difference caused by movement in a single direction, that is, the difference caused by movement in a random direction contains movement difference in at least two directions. Further, the difference caused by movement refers to the offset difference in two grids in a set direction, for example, grid 1 and grid 2, grid 1 is a grid on the corresponding first Raw image, and grid 2 is a corresponding grid on the reference image, both of which are set to detect difference points in X and Y directions. If there is an offset difference between grid 1 and grid 2 in X and Y directions, it means that the difference point between grid 1 and grid 2 is caused by movement in a random direction, and then grid 1 and grid 2 are deleted. Further, as described above, if there is an offset difference between grid 1 and grid 2 in X or Y direction, it means that the difference point between grid 1 and grid 2 is caused by movement in a single direction, and then grid 2 is translated in the opposite direction of the single offset direction relative to grid 1 to ensure that there is no difference point in any direction between the translated grid 1 and grid 2.

[0094] In one embodiment, in step 1024, the to-be-fused images are processed by pixel-by-pixel weighted fusion to obtain corresponding second Raw images. Generally, the pixel-by-pixel weighted fusion processing is directly weighted calculation on the pixel points of the image, and the algorithm for selecting the weight value directly affects the effect of the fused image. Based on how to effectively select the fusion weight coefficient, it can also be divided into average weighted fusion algorithm (directly taking the average value of the corresponding pixels of two images), multi-scale weighted gradient fusion algorithm, principal component analysis (PCA), etc.

[0095] It should be understood that the pixel-by-pixel weighted fusion processing in step S1024 is not the only way of the fusion processing in step S102, but only one of the optional ways, and the pixel-by-pixel weighted fusion processing can also be replaced by other pixel-level image fusion methods, such as a simple point-by-point pixel addition fusion processing and a fusion method based on multi-scale transformation, wherein the point-by-point pixel addition fusion processing generally refers to an operation of adding pixels at corresponding positions of two or more images of the same size to generate a new image containing information of the two or more images. It should be understood that the addition of pixel values at corresponding coordinate positions of two or more [i3] 255 gray scale images will inevitably exceed the maximum gray scale representation range 255. Obviously, the result of the image addition operation needs to be processed, and there are three basic methods: one is to take the average value of the added gray scale values of two or more pixels as the addition result; one is to make a proportional reduction according to the minimum value and the maximum value of the addition results of all pixel gray scale values of the two or more images, so that the result gray scale value meets the gray scale value range of 0 to 255; and the other is to take 255 when the added value of the two or more pixel gray scale values exceeds 255.

[0096] Further, in the embodiment of the present application, the plurality of to-be-fused images can be subjected to pixel-by-pixel average weighted fusion processing, and then the average value of the added gray scale values of the plurality of pixels is taken as the addition result, that is, as the pixel gray scale value of the corresponding second Raw image. In this process, because the average value of the added gray scale values of the plurality of pixels is taken as the pixel gray scale value of the corresponding second Raw image, the random noise between the to-be-fused images can be effectively reduced.

[0097] : subtracting the corresponding second Raw image from the first Raw image pixel by pixel to obtain a corresponding residual noise image;

[0098] In one embodiment, the corresponding second Raw image is subtracted from the first Raw image pixel by pixel to obtain a corresponding residual noise image, wherein the pixel-by-pixel subtraction generally refers to a subtraction operation of pixels at corresponding positions of two images of the same size to generate a new image containing information of the two images. When the subtraction result of the pixel values at corresponding coordinate positions of two 255 gray scale images is greater than or equal to zero, the subtraction result is taken as the gray scale value of the pixel at the corresponding position in the result image. When the subtraction result is less than zero, the absolute value of the subtraction result is taken as the result value for certain special application purposes. Further, in the embodiment of the present application, the corresponding second Raw image is subtracted from the first Raw image pixel by pixel to obtain a corresponding residual noise image, wherein the residual noise image contains difference information of the first Raw image and the second Raw image.

[0099] : based on the first Raw image and the residual noise image, a sample image database is established;

[0100] In one embodiment, the establishing of the sample image database based on the first Raw image and the residual noise image specifically comprises:

[0101] If the Bayer arrays of the first Raw image and the residual noise image are inconsistent, the first Raw image and the residual noise image are converted to obtain the first Raw image and the residual noise image with consistent Bayer arrays.

[0102] In one embodiment, the conversion processing is to convert the Bayer array format of the first Raw image and the corresponding residual noise image, so that the Bayer array format of the first Raw image and the corresponding residual noise image in the sample image database is consistent. Further, to clearly illustrate how to perform the conversion processing, the following is taken as an example: if the Bayer array format of the first Raw image is “RGGB”, when the Bayer array format of the corresponding residual noise image is not “RGGB”, the following conversion processing can be taken: when the Bayer array format of the corresponding residual noise image is “GBRG”, one pixel is added to the upper and lower boundaries of the corresponding residual noise image; when the Bayer array format of the corresponding residual noise image is “GRBG”, one pixel is added to the left and right boundaries of the corresponding residual noise image; when the Bayer array format of the corresponding residual noise image is “BGGR”, one pixel is added to the upper, lower, left and right boundaries of the corresponding residual noise image.

[0103] Further, to establish the sample image database, the image data sets in M groups of scenes can be respectively obtained through the above-mentioned method of obtaining N images in the same scene, where M is an integer greater than or equal to 2, and details are not repeated here.

[0104] : based on the sample image database, model training is performed to obtain an image denoising model.

[0105] In one embodiment, the model training based on the sample image database to obtain the image denoising model specifically comprises:

[0106] The first Raw image consistent with the Bayer array is taken as input image data, and the residual noise image corresponding to the first Raw image is taken as corresponding output image data to perform model training to obtain an image denoising model.

[0107] In one embodiment, after the construction of the sample image data set is completed, the first Raw image set is taken as training input, and the corresponding residual noise image set is taken as target output to perform model training according to a preset training algorithm to obtain the image denoising model for image denoising processing.

[0108] The training algorithm is a machine learning algorithm, and the machine learning algorithm can process data through continuous feature learning. The machine learning algorithm can include a decision tree algorithm, a logistic regression algorithm, a Bayesian algorithm, a neural network algorithm (which can include a deep neural network algorithm, a convolutional neural network algorithm, and a recurrent neural network algorithm), a clustering algorithm, and the like.

[0109] It should be noted that the selection of the training algorithm for training the image denoising model can be selected by a person skilled in the art according to actual needs. For example, the convolutional neural network algorithm can be selected to perform model training to obtain the image denoising model.

[0110] It should be understood that the sample image data can include a set of image data under M groups of scenes, and each group of scenes includes N first Raw images inconsistent with the Bayer array. The sample image data includes the first Raw image and the corresponding residual noise image under the M groups of scenes. Further, when the values of M and N become larger and larger, the training data in the sample image database contains a large amount of training data, and the obtained image denoising model is more and more accurate.

[0111] Embodiment two:

[0112] Figure 2 An implementation process of an image denoising method provided by the embodiment two of the present application is shown. The image denoising method provided by the embodiment of the present application can be applied to a computing device, which can be but is not limited to various personal computers, notebook computers, smart phones, tablet computers and panoramic cameras.

[0113] For ease of illustration, Figure 2 Only parts related to the embodiments of the present application are shown, and the details are as follows:

[0114] As Figure 2 shown, the embodiment of the present application provides an image denoising method, which includes the following steps:

[0115] S201: Obtain a Raw image to be processed;

[0116] In one embodiment, the to-be-processed Raw image can be obtained by photographing by any device with photographing function, such as a digital camera, a panoramic camera, a mobile phone, a tablet computer, an action camera, etc. In addition, the to-be-processed Raw image can also be a Raw image obtained by any image processing method, such as image transformation, image stitching, image segmentation, image synthesis, image compression, image enhancement, image restoration, etc. In addition, the to-be-processed Raw image can be a panoramic image, or a normal planar image.

[0117] According to the Bayer array of the to-be-processed Raw image and a preset image denoising model, an input image corresponding to the to-be-processed Raw image is obtained; wherein the preset image denoising model is obtained by the image denoising model generation method in any one of Embodiment One.

[0118] In one embodiment, according to the Bayer array of the to-be-processed Raw image and a preset image denoising model, an input image corresponding to the to-be-processed Raw image is obtained, specifically including:

[0119] S2021: determining whether the Bayer array of the to-be-processed Raw image is consistent with the Bayer array of a preset image denoising model;

[0120] S2022: if the Bayer array of the to-be-processed Raw image is not consistent with the Bayer array of a preset image denoising model, then performing conversion processing on the to-be-processed Raw image to obtain an input image corresponding to the to-be-processed Raw image;

[0121] S2023: if the Bayer array of the to-be-processed Raw image is consistent with the Bayer array of a preset image denoising model, then taking the to-be-processed Raw image as an input image corresponding to the to-be-processed Raw image;

[0122] wherein the input image is consistent with the Bayer array of a corresponding image denoising model.

[0123] In one embodiment, in step S2022, if the to-be-processed Raw image is inconsistent with the Bayer array of the preset image denoising model, the to-be-processed Raw image is subjected to conversion processing, that is, the to-be-processed Raw image is converted into the Bayer array format, so as to make the to-be-processed Raw image consistent with the Bayer array format of the preset image denoising model. Further, in order to clearly illustrate how to perform the conversion processing, the following is taken as an example: if the Bayer array format of the preset image denoising model is "RGGB", when the Bayer array format of the to-be-processed Raw image is not "RGGB", the following conversion processing can be performed: when the Bayer array format of the to-be-processed Raw image is "GBRG", one pixel is added to the upper and lower boundaries of the to-be-processed Raw image; when the Bayer array format of the to-be-processed Raw image is "GRBG", one pixel is added to the left and right boundaries of the to-be-processed Raw image; and when the Bayer array format of the to-be-processed Raw image is "BGGR", one pixel is added to the upper, lower, left and right boundaries of the to-be-processed Raw image.

[0124] : inputting the input image into the preset image denoising model to perform denoising processing, to obtain a residual noise image;

[0125] In one embodiment, the input image is input into the preset image denoising model, and the preset image denoising model is obtained by any one of the image denoising model generation methods. Therefore, after the input image is input into the preset image denoising model, the output image, that is, the residual noise image, is the residual noise image corresponding to the to-be-processed Raw image. Since the to-be-processed Raw image may have been subjected to conversion processing in the Bayer array format before being input into the preset image denoising model, the residual noise image obtained may be inconsistent with the Bayer array format of the corresponding to-be-processed Raw image.

[0126] : performing pixel-by-pixel fusion processing on the residual noise image and the corresponding input image, to obtain a first denoising image;

[0127] In one embodiment, the residual noise image and the corresponding input image are subjected to pixel-by-pixel fusion processing, to obtain a first denoising image, which specifically includes:

[0128] The residual noise image and the corresponding input image are subjected to pixel-by-pixel weighted fusion processing, to generate a first denoising image.

[0129] In an embodiment, the residual noise image and the corresponding input image are subjected to pixel-wise weighted fusion processing to generate a first denoised image, wherein the pixel-wise weighted fusion processing is generally a weighted calculation directly on the pixel points of the image, and the algorithm for selecting the weight value will directly affect the effect of the fused image. Based on how to effectively select the fusion weight coefficient, the pixel-wise weighted fusion processing can be generally divided into an average weighted fusion algorithm (directly taking the average value of the corresponding pixels of the two images), a multi-scale weighted gradient fusion algorithm, principal component analysis (PCA), and the like.

[0130] It should be understood that the pixel-wise weighted fusion processing in the above steps is not the only way of the pixel-wise fusion processing in step S204, but is only one optional way. The pixel-wise weighted fusion processing can also be replaced by other pixel-level image fusion methods, such as a simple pixel-wise addition fusion processing and a multi-scale transformation-based fusion method. The pixel-wise addition fusion processing generally refers to an operation of adding the pixels at the corresponding positions of two or more images of the same size to generate a new image containing the information of the two or more images. It should be understood that the addition of the pixel values at the corresponding coordinate positions of two or more 255 grayscale level images will inevitably exceed the maximum grayscale representation range 255. Apparently, the result of the image addition operation needs to be processed. There are three basic methods: one is to take the average value of the added pixel grayscale values as the addition result; one is to make a proportional reduction according to the minimum value and the maximum value of the addition result of all pixel grayscale values of the two or more images, so that the result grayscale value meets the grayscale value range of 0 to 255; and the other is to take 255 when the added value of the two or more pixel grayscale values exceeds 255.

[0131] Further, in the embodiment of the present application, the residual noise image and the corresponding input image can be subjected to pixel-wise average weighted fusion processing, and then the average value of the added pixel grayscale values is taken as the addition result, that is, as the pixel grayscale value of the corresponding first denoised image. In this process, because the average value of the added pixel grayscale values is taken as the pixel grayscale value of the corresponding first denoised image, the random noise between the images to be fused can be effectively reduced.

[0132] In an embodiment, the image denoising method further includes the following steps after step S204:

[0133] determining whether the first denoised image is consistent with the Bayer array of the corresponding Raw image to be processed;

[0134] According to the judgment result, the first denoising image is processed correspondingly to obtain the final denoising image corresponding to the Raw image to be processed.

[0135] Wherein, as described above, before the Raw image to be processed is input into the preset image denoising model, it is needed to judge whether the Bayer array format of the Raw image to be processed is consistent with the preset image denoising model, if not, conversion processing is needed, so the Bayer array format of the input image, the corresponding residual noise image and the corresponding first denoising image is consistent, but it is not necessarily consistent with the Bayer array of the corresponding Raw image to be processed, therefore it is needed to judge whether the first denoising image is consistent with the Bayer array of the corresponding Raw image to be processed, and it is needed to process the first denoising image correspondingly according to the judgment result, so as to obtain the final denoising image corresponding to the Raw image to be processed.

[0136] In one embodiment, according to the judgment result, the first denoising image is processed correspondingly to obtain the final denoising image corresponding to the Raw image to be processed, specifically including:

[0137] If the first denoising image is not consistent with the Bayer array of the corresponding Raw image to be processed, the first denoising image is converted to obtain the final denoising image consistent with the Bayer array of the corresponding Raw image to be processed;

[0138] If the first denoising image is consistent with the Bayer array of the corresponding Raw image to be processed, the first denoising image is taken as the final denoising image of the corresponding Raw image to be processed.

[0139] Further, if the first denoising image is inconsistent with the Bayer array of the corresponding to-be-processed Raw image, the first denoising image is subjected to conversion processing, that is, the first denoising image is subjected to Bayer array format conversion, so as to make the first denoising image consistent with the Bayer array format of the corresponding to-be-processed Raw image. Further, in order to clearly illustrate how to perform the conversion processing, the following is taken as an example: if the Bayer array format of the to-be-processed Raw image is "RGGB", when the Bayer array format of the corresponding first denoising image is not "RGGB", the following conversion processing can be taken: when the Bayer array format of the corresponding first denoising image is "GBRG", one pixel is added to the upper and lower boundaries of the corresponding first denoising image; when the Bayer array format of the corresponding first denoising image is "GRBG", one pixel is added to the left and right boundaries of the corresponding first denoising image; and when the Bayer array format of the corresponding first denoising image is "BGGR", one pixel is added to the upper, lower, left and right boundaries of the corresponding first denoising image.

[0140] The Bayer array format of the to-be-processed Raw image is consistent with the Bayer array format of the corresponding final denoising image.

[0141] Example Three:

[0142] Figure 3 The structure of the image denoising model generation device 3 provided by the third embodiment of the present application is shown, and only the parts related to the embodiments of the present application are shown for the purpose of illustration.

[0143] In the embodiments of the present application, the image denoising model generation device 3 comprises an acquisition module 31, a first processing module 32, a second processing module 33, a third processing module 34 and a training module 35, wherein:

[0144] The acquisition module 31 is configured to acquire N first Raw images of different Bayer arrays under the same scene, N being an integer greater than or equal to 2.

[0145] The first processing module 32 is configured to perform fusion processing on the N first Raw images of different Bayer arrays respectively, to obtain N second Raw images corresponding thereto.

[0146] The second processing module 33 is configured to subtract the corresponding second Raw image from the first Raw image pixel by pixel, to obtain a residual noise image corresponding thereto.

[0147] The third processing module 34 is configured to establish a sample image database based on the first Raw image and the residual noise image.

[0148] The training module 35 is configured to train a model based on the sample image database to obtain an image denoising model.

[0149] In the embodiments of the present application, each module of the image denoising generation apparatus can be implemented by a corresponding hardware or software unit, and each module can be an independent software or hardware unit, or can be integrated into a software or hardware unit, which does not limit the present application.

[0150] Example Four:

[0151] Figure 4 The structure of the image denoising apparatus 4 provided in the fourth embodiment of the present application is shown, and only parts related to the embodiments of the present application are shown for ease of illustration.

[0152] In the embodiments of the present application, the image denoising apparatus 4 includes an acquisition module 41, a first processing module 42, a second processing module 43, and a third processing module 44, wherein:

[0153] The acquisition module 41 is configured to acquire a to-be-processed Raw image.

[0154] The first processing module 42 is configured to obtain a corresponding input image according to the to-be-processed Raw image and a Bayer array of a preset image denoising model.

[0155] The second processing module 43 is configured to input the input image into a corresponding image denoising model for denoising processing to obtain a residual noise image.

[0156] The third processing module 44 is configured to perform pixel-by-pixel fusion processing on the residual noise image and the corresponding input image to obtain a first denoising image.

[0157] The preset image denoising model is obtained by the image denoising model generation method in any one of the embodiments.

[0158] In the embodiments of the present application, each module of the image denoising apparatus can be implemented by a corresponding hardware or software unit, and each module can be an independent software or hardware unit, or can be integrated into a software or hardware unit, which does not limit the present application.

[0159] Example Five:

[0160] In one embodiment, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps in the above image denoising model generation method embodiments are implemented, for example, Figure 1 The steps S101 to S105 shown.

[0161] Or, the computer program is executed by the processor to realize the functions of each unit in each of the above device embodiments, for example, Figure 3 the functions of units 31 to 35 shown.

[0162] In an embodiment, a computer readable storage medium is provided, which stores a computer program, the computer program being executed by a processor to realize the steps in the above image denoising method embodiments, for example, Figure 2 the steps S201 to S204 shown.

[0163] Or, the computer program is executed by the processor to realize the functions of each unit in each of the above device embodiments, for example, Figure 4 the functions of units 41 to 44 shown.

[0164] The computer readable storage medium of the embodiments of the present application can include any entity or device capable of carrying computer program code, recording medium, such as ROM / RAM, magnetic disk, optical disk, flash memory, etc.

[0165] Example Six:

[0166] Figure 5 The structure of the electronic device provided by the sixth embodiment of the present application is shown, only the parts related to the embodiments of the present application are shown for the convenience of description.

[0167] In an embodiment, the electronic device 5 of the present application includes a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. The processor 50 executes the computer program 52 to realize the steps in the above image denoising model generation method embodiments, for example Figure 1 the steps S101 to S105 shown. Or, the processor 50 executes the computer program 52 to realize the functions of each module in the above device embodiments, for example Figure 3 the functions of modules 31 to 35 shown.

[0168] In an embodiment, the electronic device 5 of the present application includes a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. The processor 50 executes the computer program 52 to realize the steps in the above image denoising method embodiments, for example Figure 2 the steps S201 to S204 shown. Or, the processor 50 executes the computer program 52 to realize the functions of each module in the above device embodiments, for example Figure 4 the functions of modules 41 to 44 shown.

[0169] The electronic device of the embodiment of the present application can be a smart phone, a personal computer, a panoramic camera itself, or the like. The steps implemented when the processor 50 of the electronic device 5 executes the computer program 52 to implement the image denoising model generation method or the image denoising method can refer to the description of the foregoing method embodiments, and will not be described herein again.

[0170] The above merely describes preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, and improvement within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating an image denoising model, characterized in that: The method comprises: Obtain the first raw images of N different Bayer arrays under the same scene, where N is an integer greater than or equal to 2; fusing the N first Raw images of different Bayer arrays to obtain corresponding N second Raw images; Subtracting the corresponding second Raw image from the first Raw image pixel by pixel to obtain a corresponding residual noise image; Establishing a sample image database based on the first Raw image and the residual noise image; Model training is performed based on the sample image database to obtain an image denoising model.

2. The method according to claim 1, wherein The N first Raw images of different Bayer arrays are fused separately to obtain corresponding N second Raw images, specifically including: Sequentially select any one of the N first Raw images as a reference image; Detecting differences between the reference image and the other N-1 first Raw images respectively; According to the difference points, the reference image is processed accordingly to obtain an image to be fused; The images to be fused are subjected to pixel-by-pixel weighted fusion processing to obtain a corresponding second Raw image.

3. The method according to claim 2, wherein Detecting differences between the reference image and the other N-1 first Raw images respectively, specifically including: Divide the N first Raw images into G grids, where G is an integer greater than or equal to 2; An alignment algorithm is used to calculate the difference between the content of each grid of the reference image and the content of the corresponding grids of the other N-1 first Raw images.

4. The method according to claim 2, wherein According to the difference points, the reference image is processed accordingly to obtain the image to be fused, specifically including: If it is detected that the difference point is caused by movement in a disorderly direction, the corresponding grids in the N first Raw images are deleted to obtain N images to be fused; If it is detected that the difference point is caused by movement in a single direction, the grids corresponding to the N-1 first Raw images are translated relative to the reference image in the opposite direction of the single direction, and the N-1 translated images and the reference image are used as the N images to be fused.

5. The method according to claim 1, wherein Establishing a sample image database based on the first Raw image and the residual noise image specifically includes: If the Bayer arrays of the first Raw image and the residual noise image are inconsistent, conversion processing is performed on the first Raw image and the residual noise image to obtain a first Raw image and a residual noise image with consistent Bayer arrays.

6. The method according to claim 5, wherein Model training is performed based on the sample image database to obtain an image denoising model, specifically including: The first Raw image with consistent Bayer array is used as input image data, and the residual noise image with consistent Bayer array corresponding to the first Raw image is used as corresponding output image data to perform model training to obtain an image denoising model.

7. An image denoising method, characterized in that: The method comprises: Get the raw image to be processed; Obtaining a corresponding input image according to the raw image to be processed and the Bayer array of a preset image denoising model; Input the input image into the corresponding image denoising model to perform denoising processing to obtain a residual noise image; Performing pixel-by-pixel fusion processing on the residual noise image and the corresponding input image to obtain a first denoised image; The preset image denoising model is obtained by the image denoising model generation method according to any one of claims 1 to 6.

8. The method according to claim 7, wherein Obtaining a corresponding input image according to the raw image to be processed and the Bayer array of the preset image denoising model, specifically including: Determine whether the Bayer array of the raw image to be processed is consistent with the Bayer array of a preset image denoising model; If the raw image to be processed is inconsistent with the Bayer array of the preset image denoising model, converting the raw image to obtain a corresponding input image; If the raw image to be processed is consistent with the Bayer array of the preset image denoising model, the raw image to be processed is used as the corresponding input image; The input image is consistent with the Bayer array of the corresponding image denoising model.

9. The method according to claim 7, wherein The residual noise image is subjected to pixel-by-pixel fusion processing with the corresponding input image to obtain a first denoised image, specifically comprising: The residual noise image is subjected to pixel-by-pixel weighted fusion processing with the corresponding input image to generate a first denoised image.

10. The method according to claim 7, wherein: After performing pixel-by-pixel fusion processing on the residual noise image and the corresponding input image to obtain a first denoised image, the method further includes: Determining whether the Bayer array of the first denoised image is consistent with that of the corresponding raw image to be processed; According to the judgment result, the first denoised image is processed accordingly to obtain a final denoised image corresponding to the raw image to be processed.

11. The method according to claim 10, wherein: According to the judgment result, the first denoised image is processed accordingly to obtain a final denoised image corresponding to the raw image to be processed, specifically including: If the first denoised image is inconsistent with the Bayer array of the corresponding raw image to be processed, converting the first denoised image to obtain a final denoised image that is consistent with the Bayer array of the corresponding raw image to be processed; If the first denoised image is consistent with the Bayer array of the corresponding raw image to be processed, the first denoised image is used as the final denoised image of the corresponding raw image to be processed.

12. An image denoising model generation device, characterized in that: The device comprises: Acquisition module: used to acquire N first Raw images of different Bayer arrays under the same scene, where N is an integer greater than or equal to 2; A first processing module is used to fuse the N first Raw images of different Bayer arrays respectively to obtain corresponding N second Raw images; A second processing module is configured to subtract the corresponding second Raw image from the first Raw image pixel by pixel to obtain a corresponding residual noise image; A third processing module: configured to establish a sample image database based on the first Raw image and the residual noise image; Training module: used to perform model training based on the sample image database to obtain an image denoising model.

13. An image noise reduction device, characterized in that: The device comprises: Acquisition module: used to obtain the Raw image to be processed; A first processing module is configured to obtain a corresponding input image according to the raw image to be processed and the Bayer array of a preset image denoising model; The second processing module is used to input the input image into the corresponding image denoising model to perform denoising processing to obtain a residual noise image; A third processing module is used to perform pixel-by-pixel fusion processing on the residual noise image and the corresponding input image to obtain a first denoised image; The preset image denoising model is obtained by the image denoising model generation method according to any one of claims 1 to 6.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

Images