An image denoising model generation method, an image denoising method, an image denoising device, a storage medium, and an image denoising apparatus

Abstract

The application provides an image denoising model generation method and an image denoising method. N first Raw images of different Bayer arrays in the same scene are obtained, the N first Raw images of different Bayer arrays are fused to obtain N second Raw images, a sample image database is established based on the first Raw images and the second Raw images, model training is performed, and an image denoising model is obtained. The image denoising model is input into the image denoising model after format conversion processing of a to-be-processed Raw image, a first denoising image is obtained, corresponding format conversion processing is performed on the first denoising image, and a final denoising image is obtained. The image denoising model generated based on the application can realize automatic image denoising, can obviously reduce picture noise, and the image denoising method has high efficiency, good universality and strong robustness.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to an image denoising model generation method, an image denoising method, a device, a storage medium and equipment. BACKGROUND

[0002] Whether it is a CCD or a CMOS chip, high temperature will inevitably be generated during work. If the temperature of the chip is increased, the noise signal is too strong, and color speckles with different brightness will be formed on the picture, which is more obvious in the dark area. These points are the noise points. The generation of thermal noise is because the thermal current is superimposed on the normal signal current, so that the signal current of part of the pixel points is greater than the normal sensing current, finally leading to the signal intensity on this pixel point being stronger, which is shown as higher brightness on the Raw image. After the color processing of the rear-end ISP, such pixel points will lead to noise problems such as deeper or brighter colors.

[0003] At present, for the denoising of these noise points, the existing Raw image denoising method can generally only be used for denoising processing of Raw images with the same Bayer array under the same Sensor parameter condition, but Raw images generally have multiple Bayer patterns, and the parameters of the Sensor of different cameras are also different, leading to poor generality of the Raw denoising realized on the existing denoising method. In addition, the Raw image generally needs to be processed by the ISP into a visible Raw image, and the existing denoising method generally uses SSIM and other objective indexes as a reference on the Raw image itself, which leads to the existing denoising method tending to smooth the Raw image, resulting in loss of clarity of the Raw image.

[0004] In summary, the existing Raw image denoising method has the problems of poor generality and loss of clarity, and it is necessary to improve on this basis. SUMMARY

[0005] The embodiments of the present application provide an image denoising model generation method, an image denoising method, a device, a storage medium and equipment, which are used to solve the problems of poor generality and loss of clarity of the existing Raw image denoising method.

[0006] In a first aspect, the embodiments of the present application provide an image denoising model generation method, which comprises the following steps:

[0007] Obtain N first Raw images of different Bayer arrays under the same scene, N being an integer greater than or equal to 2;

[0008] Respectively fuse the N first Raw images of different Bayer arrays to obtain N second Raw images corresponding thereto;

[0009] establishing a sample image database based on the first Raw image and the second Raw image;

[0010] performing model training based on the sample image database to obtain an image denoising model.

[0011] With reference to the first aspect, in a possible implementation manner, the first Raw image of the N different Bayer arrays is fused respectively to obtain the corresponding N second Raw images, and the specific implementation manner comprises the following steps.

[0012] selecting any one of the N first Raw images as a reference image in sequence;

[0013] detecting difference points of the reference image and other N-1 first Raw images respectively;

[0014] performing corresponding processing on the reference image according to the difference points to obtain a to-be-fused image;

[0015] performing pixel-by-pixel weighted fusion processing on the to-be-fused image to obtain a corresponding second Raw image.

[0016] With reference to the first aspect, in a possible implementation manner, the difference points of the reference image and other N-1 first Raw images are detected respectively, and the specific implementation manner comprises the following steps.

[0017] dividing the N first Raw images into G grids, wherein G is an integer greater than or equal to 2;

[0018] calculating, by using an alignment algorithm, difference points of each grid content of the reference image and corresponding grid content of other N-1 first Raw images.

[0019] With reference to the first aspect, in a possible implementation manner, the corresponding processing is performed on the reference image according to the difference points to obtain a to-be-fused image, and the specific implementation manner comprises the following steps.

[0020] if it is detected that the difference points are caused by movement in a chaotic direction, deleting corresponding grids in the N first Raw images to obtain N to-be-fused images;

[0021] if it is detected that the difference points are caused by movement in a single direction, performing translation on corresponding grids of the N-1 first Raw images relative to the reference image in the opposite direction of the single direction, and taking the N-1 translated images and the reference image as N to-be-fused images.

[0022] With reference to the first aspect, in a possible implementation manner, the sample image database is established based on the first Raw image and the second image, and the specific implementation manner comprises the following steps.

[0023] If the Bayer arrays of the first Raw image and the second Raw image are inconsistent, the first Raw image and the second Raw image are converted to obtain the first Raw image and the second Raw image with consistent Bayer arrays.

[0024] With reference to the first aspect, in a possible implementation manner, the model training is performed based on the sample image database to obtain an image denoising model, and the image denoising model is obtained by specifically including:

[0025] The first Raw image with consistent Bayer array is taken as input image data, and the second Raw image with consistent Bayer array corresponding to the first Raw image is taken as corresponding output image data, and the model training is performed to obtain an image denoising model.

[0026] The second aspect, the embodiment of the present application provides an image denoising method, the method comprises the following steps:

[0027] Obtaining a to-be-processed Raw image;

[0028] According to the Bayer array of the to-be-processed Raw image and the preset image denoising model, a corresponding input image is obtained;

[0029] The input image is input into the corresponding image denoising model for denoising processing to obtain a first denoised image;

[0030] The preset image denoising model is obtained by the image denoising model generation method of any one of the first aspect.

[0031] With reference to the second aspect, in a possible implementation manner, according to the Bayer array of the to-be-processed Raw image and the preset image denoising model, a corresponding input image is obtained, and the method specifically includes:

[0032] It is judged whether the Bayer array of the to-be-processed Raw image is consistent with the Bayer array of the preset image denoising model;

[0033] If the Bayer array of the to-be-processed Raw image is inconsistent with the Bayer array of the preset image denoising model, the to-be-processed Raw image is converted to obtain a corresponding input image;

[0034] If the Bayer array of the to-be-processed Raw image is consistent with the Bayer array of the preset image denoising model, the to-be-processed Raw image is taken as a corresponding input image;

[0035] The input image is consistent with the Bayer array of the corresponding image denoising model.

[0036] With reference to the second aspect, in a possible implementation manner, after the input image is input into the corresponding image denoising model to perform denoising processing to obtain a first denoised image, the method further includes:

[0037] determining whether the first denoised image is consistent with a Bayer array of the corresponding to-be-processed Raw image;

[0038] performing corresponding processing on the first denoised image according to the determination result to obtain a final denoised image corresponding to the to-be-processed Raw image.

[0039] With reference to the second aspect, in a possible implementation manner, performing corresponding processing on the first denoised image according to the determination result to obtain a final denoised image corresponding to the to-be-processed Raw image includes:

[0040] if the first denoised image is not consistent with the Bayer array of the corresponding to-be-processed Raw image, performing conversion processing on the first denoised image to obtain a final denoised image consistent with the Bayer array of the corresponding to-be-processed Raw image;

[0041] if the first denoised image is consistent with the Bayer array of the corresponding to-be-processed Raw image, taking the first denoised image as the final denoised image of the corresponding to-be-processed Raw image.

[0042] Third aspect, an image denoising model generation device is provided in the embodiment of the present application, and the device includes:

[0043] an acquisition module configured to acquire N first Raw images of different Bayer arrays in a same scene, N being an integer greater than or equal to 2;

[0044] a first processing module configured to perform fusion processing on the N first Raw images of different Bayer arrays respectively to obtain N second Raw images corresponding thereto;

[0045] a second processing module configured to establish a sample image database based on the first Raw images and the second Raw images;

[0046] a training module configured to perform model training based on the sample image database to obtain an image denoising model.

[0047] Fourth aspect, an image denoising device is provided in the embodiment of the present application, and the device includes:

[0048] a first processing module configured to obtain a corresponding input image according to a Bayer array of a preset image denoising model and a to-be-processed Raw image;

[0049] The second processing module is configured to input the input image into a corresponding image denoising model to perform denoising processing, and obtain a first denoised image.

[0050] The preset image denoising model is obtained by the image denoising model generation method of any one of the first aspect.

[0051] In the fifth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is characterized in that when the computer program is executed by a processor, the steps of the method of any one of the first aspect or the second aspect are implemented.

[0052] In the sixth aspect, the present application provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the electronic device is characterized in that when the processor executes the computer program, the steps of the method of any one of the first aspect or the second aspect are implemented.

[0053] The present application provides an image denoising model generation method and an image denoising method. N first Raw images of different Bayer arrays in the same scene are obtained, the N first Raw images of different Bayer arrays are fused respectively to obtain N second Raw images corresponding to the N first Raw images, a sample image database is established based on the first Raw images and the corresponding second Raw images, model training is performed based on the sample image database to obtain an image denoising model, a Raw image to be processed is input into the image denoising model after format conversion processing to obtain a first denoised image, and corresponding format conversion processing is performed on the first denoised image to obtain a final denoised image. The image denoising model generated based on the present application realizes automatic image denoising, can obviously reduce picture noise, improve the sharpness of the image, make the image more comfortable to watch, make a small-size image sensor obtain a picture quality similar to a larger-size sensor, and the image denoising method has high efficiency, good versatility, and strong robustness. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a flowchart of the image denoising model generation method provided by the first embodiment of the present application;

[0055] Figure 2 is a flowchart of the image denoising method provided by the second embodiment of the present application;

[0056] Figure 3 is a structural schematic diagram of the image denoising model generation device provided by the third embodiment of the present application;

[0057] Figure 4 is a structural schematic diagram of the image denoising device provided by the fourth embodiment of the present application;

[0058] Figure 5 Figure 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the objectives, technical solutions and beneficial effects of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0060] In the prior art, a Raw image is original data of a digital signal converted by a CMOS or CCD image sensor from a light source signal captured by the CMOS or CCD image sensor, is lossless, and contains original color information of an object. A data format of the Raw image generally adopts a Bayer array arrangement manner. A color filter array (CFA) is generated by a filter sheet. Since the human eye is more sensitive to green color bands, the Bayer array data format contains 50% green information and 25% red and blue information. The Bayer array is a 4x4 array, which is composed of 8 green, 4 blue and 4 red pixels. When converting a gray scale pattern into a color picture, 9 operations are performed in a 2x2 matrix, and finally a color pattern is generated. The Bayer array generally has four formats: RGGB (a), BGGR (b), GBRG (c) and GRBG (d).

[0061] The specific implementation of the present application will be described in detail below in combination with specific embodiments.

[0062] Embodiment One

[0063] Figure 1 Figure 1 shows an implementation process of an image denoising model generation method provided by an embodiment of the present application. The image denoising model generation method provided by the embodiment of the present application can be applied to a computing device, which can be but is not limited to various personal computers, notebook computers, smart phones, tablet computers and panoramic cameras.

[0064] For ease of illustration, Figure 1 Only parts related to the embodiments of the present application are shown, and are described in detail as follows: Figure 1 As shown in Figure 1, the image denoising model generation method includes the following steps:

[0065] S101: Obtain N first Raw images of different Bayer arrays under the same scene;

[0066] The first Raw image can be obtained by photographing by any device with photographing function, such as a digital camera, a panoramic camera, a mobile phone, a tablet computer, an action camera, etc. In addition, the first Raw image can also be a Raw image obtained by any image processing method, such as image transformation, image stitching, image segmentation, image synthesis, image compression, image enhancement, image restoration, etc. In addition, the first Raw image can be a panoramic image, or a normal planar image.

[0067] In order to obtain N first Raw images with different Bayer arrays in the same scene, a plurality of methods can be used, for example: the first method is to continuously photograph a plurality of Raw images in the same scene by one photographing device, or to photograph a plurality of Raw images in the same scene by a plurality of photographing devices, and then to select N first Raw images with inconsistent Bayer arrays from the plurality of Raw images; the second method is to continuously photograph N Raw images in the same scene by one photographing device, or to photograph a plurality of Raw images in the same scene by a plurality of photographing devices, and then to obtain N first Raw images with inconsistent Bayer arrays by Bayer array conversion method; the third method is to continuously photograph N first Raw images with different Bayer arrays in the same scene by one photographing device or a plurality of photographing devices. It should be understood that different combinations of the above listed methods or any other method can also be used to obtain N first Raw images with different Bayer arrays in the same scene, which will not be listed one by one.

[0068] S102: The N first Raw images with different Bayer arrays are fused respectively to obtain N corresponding second Raw images;

[0069] In one embodiment, S102 specifically includes the following steps:

[0070] S1021: Any one of the N first Raw images is selected as a reference image in turn;

[0071] S1022: Differences between the reference image and other N-1 first Raw images are detected respectively;

[0072] S1023: The reference image is processed according to the differences to obtain a to-be-fused image;

[0073] S1024: The to-be-fused image is pixel-by-pixel weighted and fused to obtain a corresponding second Raw image.

[0074] In one embodiment, in step S1021, any one of the N first Raw images is selected as the reference image in turn, and one way that can be used is that the N first Raw images are marked as the 1st image, the 2nd image, the 3rd image, …, and the Nth image in turn, and then the 1st image, the 2nd image, the 3rd image, …, and the Nth image are selected as the reference image in turn.

[0075] In one embodiment, in step S1022, the difference points of the reference image and the other N-1 first Raw images are detected respectively, and the specific steps include the following steps:

[0076] The N first Raw images are all divided into G grids, where G is an integer greater than or equal to 2.

[0077] An alignment algorithm is used to calculate the difference points of each grid content of the reference image and the corresponding grid content of the other N-1 first Raw images.

[0078] In one embodiment, the N first Raw images are all divided into G grids, where G is an integer greater than or equal to 2. The image can be divided into multiple grids in multiple ways, and common grids can be divided into two categories: structured grid and unstructured grid. The connection between the unit nodes of the structured grid has only a limited number of ways, and all internal nodes in the generated grid body area have the same number of adjacent units and the same number of adjacent nodes. For a two-dimensional plane or a three-dimensional curved surface, the grid unit generated by the structured grid is generally a quadrilateral, and the three-dimensional entity space is mainly a hexahedron. In the unstructured grid, the nodes between the grid units can be connected in any form, and do not need to have the same adjacent units. The number of connections between different nodes in the grid area can be different. For a two-dimensional plane or a three-dimensional curved surface, the grid unit generated by the unstructured grid is generally a triangle, and the three-dimensional entity space is mainly a tetrahedron. Common structured grid generation methods generally include mapping method, geometric decomposition method, etc. The unstructured grid generation method generally includes octree method, quadtree method, Delaunay triangulation method, etc.

[0079] It should be understood that the above is only to list some common grid generation methods, and the embodiments of the present application are not limited thereto. The embodiments of the present application can divide the N first Raw images into G grids by using any one or more of the above methods, or can use other grid generation methods in the prior art, which will not be described here.

[0080] Further, it should be understood that the G grids divided can be of any shape, such as AXA grid or AXB grid, where A and B are not equal and are both integers greater than or equal to 2.

[0081] In one embodiment, a difference point between each grid content of the reference image and the corresponding grid content of the other N-1 first Raw images is calculated by using an alignment algorithm, wherein the alignment algorithm can be any image alignment algorithm, for example, a method based on pixel value cross-correlation, a sequential similarity detection algorithm, a phase correlation algorithm based on Fourier transform, a mutual information based alignment algorithm, an image alignment method based on optimization, for example, Lucas-Kanade algorithm and its improved algorithm.

[0082] In one embodiment, in step S1023, the reference image is processed according to the difference point to obtain a to-be-fused image, which specifically includes the following steps:

[0083] If it is detected that the difference point is caused by movement in a random direction, the corresponding grid in the N first Raw images is deleted to obtain N to-be-fused images.

[0084] If it is detected that the difference point is caused by movement in a single direction, the corresponding grid of the N-1 first Raw images is translated in the opposite direction of the single direction relative to the reference image, and the N-1 translated images and the reference image are taken as N to-be-fused images.

[0085] In one embodiment, the difference caused by movement in a random direction is relative to the difference caused by movement in a single direction, that is, the difference caused by movement in a random direction contains movement difference in at least two directions. Further, the difference caused by movement refers to the offset difference of two grids in a set direction, for example, grid 1 and grid 2, grid 1 is a grid on the corresponding first Raw image, and grid 2 is the corresponding grid on the reference image, both of which are set to detect the difference point in the X and Y directions. If grid 1 and grid 2 have offset difference in the X and Y directions, it means that the difference point between grid 1 and grid 2 is caused by movement in a random direction, and grid 1 and grid 2 are deleted. Further, as described above, if grid 1 and grid 2 only have offset difference in the X or Y direction, it means that the difference point between grid 1 and grid 2 is caused by movement in a single direction, and grid 2 is translated in the opposite direction of the single offset direction relative to grid 1 to ensure that there is no difference point between the translated grid 1 and grid 2 in any direction.

[0086] In one embodiment, in step 1024, the to-be-fused images are subjected to pixel-by-pixel weighted fusion processing to obtain a corresponding second Raw image, wherein the pixel-by-pixel weighted fusion processing is generally a weighted calculation directly on the pixel points of the images, and the algorithm for selecting the weight directly affects the effect of the fused image. Based on how to effectively select the fusion weight coefficient, the pixel-by-pixel weighted fusion processing can be generally divided into an average weighted fusion algorithm (directly taking the average of the corresponding pixels of two images for processing), a multi-scale weighted gradient fusion algorithm, principal component analysis (PCA), and the like.

[0087] It should be understood that the pixel-by-pixel weighted fusion processing in step S1024 is not the only way of the fusion processing in step S102, but only one optional way. The pixel-by-pixel weighted fusion processing can also be replaced by other pixel-level image fusion methods, such as a simple pixel-by-pixel addition fusion processing and a multi-scale transformation-based fusion method. The pixel-by-pixel addition fusion processing generally refers to an operation of adding corresponding position pixels of two or more images of the same size to generate a new image containing information of the two or more images. It should be understood that the addition of the pixel values of two or more 255 gray scale images at corresponding coordinate positions will inevitably exceed the maximum gray scale representation range 255. Obviously, the result of the image addition operation needs to be processed. There are three basic methods: one is to take the average of the added gray scale values of two or more pixels as the addition result; one is to make a proportional reduction according to the minimum and maximum values of the addition results of all pixel gray scale values of the two or more images, so that the result gray scale value meets the gray scale value range of 0 to 255; and the other is to take 255 when the added value of the two or more pixel gray scale values exceeds 255.

[0088] Further, in the embodiment of the present application, the multiple to-be-fused images can be subjected to pixel-by-pixel average weighted fusion processing, and then the average of the added values of the multiple pixel gray scale values is taken as the addition result, that is, as the pixel gray scale value of the corresponding second Raw image. In this process, because the average of the added values of the multiple pixel gray scale values is taken as the pixel gray scale value of the corresponding second Raw image, the random noise between the to-be-fused images can be effectively reduced.

[0089] S103: establishing a sample image database based on the first Raw image and the second Raw image;

[0090] In one embodiment, the establishing of the sample image database based on the first Raw image and the second Raw image specifically includes:

[0091] If the Bayer arrays of the first Raw image and the second Raw image are inconsistent, the first Raw image and the second Raw image are converted to obtain the first Raw image and the second Raw image with consistent Bayer arrays.

[0092] In one embodiment, the conversion process, i.e., the conversion of the Bayer array format of the first Raw image and the corresponding second Raw image, is to make the Bayer array format of the first Raw image and the corresponding second Raw image in the sample image database consistent. Further, to clearly illustrate how to perform the conversion process, the following example is given: if the Bayer array format of the first Raw image is “RGGB”, when the Bayer array format of the corresponding second Raw image is not “RGGB”, the conversion process can be performed in the following manner: when the Bayer array format of the corresponding second Raw image is “GBRG”, one pixel is added to the upper and lower boundaries of the corresponding second Raw image; when the Bayer array format of the corresponding second Raw image is “GRBG”, one pixel is added to the left and right boundaries of the corresponding second Raw image; and when the Bayer array format of the corresponding second Raw image is “BGGR”, one pixel is added to the upper, lower, left and right boundaries of the corresponding second Raw image.

[0093] Further, to establish the sample image database, the image data sets under M groups of scenes can be obtained by the above-mentioned method of obtaining N images under the same scene, where M is an integer greater than or equal to 2, and details are not repeated here.

[0094] S104: model training based on the sample image database to obtain an image denoising model.

[0095] In one embodiment, the model training based on the sample image database to obtain an image denoising model specifically includes:

[0096] The first Raw image with consistent Bayer array is taken as input image data, and the second Raw image with consistent Bayer array corresponding to the first Raw image is taken as corresponding output image data, and model training is performed to obtain an image denoising model.

[0097] In an embodiment, after the construction of the sample image data set is completed, the first Raw image set can be taken as the training input, the corresponding second Raw image set can be taken as the target output, and the model training is performed according to the preset training algorithm, so as to train the image denoising model for image denoising processing.

[0098] The training algorithm is a machine learning algorithm, and the machine learning algorithm can process data through continuous feature learning. The machine learning algorithm can include a decision tree algorithm, a logistic regression algorithm, a Bayesian algorithm, a neural network algorithm (which can include a deep neural network algorithm, a convolutional neural network algorithm, and a recurrent neural network algorithm), a clustering algorithm, and the like.

[0099] It should be noted that the selection of the training algorithm for training the image denoising model can be selected by a person skilled in the art according to actual needs. For example, the convolutional neural network algorithm can be selected for model training to obtain the image denoising model.

[0100] It should be understood that the sample image data can include a set of image data under M groups of scenes, and each group of scenes includes N Bayer array inconsistent first Raw images. The sample image data includes the first Raw images and the corresponding second Raw images under the M groups of scenes. Further, when the values of M and N become larger and larger, the sample image database contains a large amount of training data, and the obtained image denoising model is more and more accurate.

[0101] Embodiment two:

[0102] Figure 2 An implementation process of an image denoising method provided by an embodiment of the present application is shown. The image denoising method provided by the embodiment of the present application can be applied to a computing device, which can be but is not limited to various personal computers, notebook computers, smart phones, tablet computers, and panoramic cameras.

[0103] For ease of illustration, Figure 2 Only parts related to the embodiments of the present application are shown, and the details are as follows: Figure 2 As shown in the figure, the embodiment of the present application provides an image denoising method, which includes the following steps:

[0104] S201: Obtain a Raw image to be processed;

[0105] In an embodiment, the to-be-processed Raw image can be obtained by photographing by any device with photographing function, such as a digital camera, a panoramic camera, a mobile phone, a tablet computer, an action camera, etc. In addition, the to-be-processed Raw image can also be a Raw image obtained by any image processing method, such as image transformation, image stitching, image segmentation, image synthesis, image compression, image enhancement, image restoration, etc. In addition, the to-be-processed Raw image can be a panoramic image, or can be a normal planar image.

[0106] S202: obtaining a corresponding input image according to the Bayer array of the to-be-processed Raw image and a preset image denoising model; wherein the preset image denoising model is obtained by the image denoising model generation method in any one of Embodiment One.

[0107] In an embodiment, obtaining a corresponding input image according to the Bayer array of the to-be-processed Raw image and a preset image denoising model specifically includes:

[0108] S2021: determining whether the Bayer array of the to-be-processed Raw image is consistent with the Bayer array of the preset image denoising model;

[0109] S2022: if the Bayer array of the to-be-processed Raw image is not consistent with the Bayer array of the preset image denoising model, performing conversion processing on the to-be-processed Raw image to obtain a corresponding input image;

[0110] S2023: if the Bayer array of the to-be-processed Raw image is consistent with the Bayer array of the preset image denoising model, taking the to-be-processed Raw image as a corresponding input image;

[0111] In an embodiment, the input image is consistent with the Bayer array of the corresponding image denoising model.

[0112] In one embodiment, in step S2022, if the to-be-processed Raw image is inconsistent with the Bayer array of the preset image denoising model, the to-be-processed Raw image is converted, that is, the to-be-processed Raw image is converted into the Bayer array format, so as to make the to-be-processed Raw image consistent with the Bayer array format of the preset image denoising model. Further, in order to clearly illustrate how to perform the conversion, the following is taken as an example: if the Bayer array format of the preset image denoising model is "RGGB", when the Bayer array format of the to-be-processed Raw image is not "RGGB", the following conversion can be performed: when the Bayer array format of the to-be-processed Raw image is "GBRG", one pixel is added to the upper and lower boundaries of the to-be-processed Raw image; when the Bayer array format of the to-be-processed Raw image is "GRBG", one pixel is added to the left and right boundaries of the to-be-processed Raw image; and when the Bayer array format of the to-be-processed Raw image is "BGGR", one pixel is added to the upper, lower, left and right boundaries of the to-be-processed Raw image.

[0113] S203: inputting the input image into the preset image denoising model to perform denoising processing, to obtain a first denoised image;

[0114] In one embodiment, the input image is input into the preset image denoising model, and the preset image denoising model is obtained by any one of the image denoising model generation methods. Therefore, after the input image is input into the preset image denoising model, the output image, that is, the first denoised image, is a second Raw image corresponding to the to-be-processed Raw image. Since the to-be-processed Raw image may have been converted in the Bayer array format before being input into the preset image denoising model, the second Raw image obtained may be inconsistent with the Bayer array format of the corresponding to-be-processed Raw image.

[0115] In one embodiment, the image denoising method further includes the following steps after step S203:

[0116] determining whether the first denoised image is consistent with the Bayer array of the corresponding to-be-processed Raw image;

[0117] According to the determination result, the first denoised image is processed to obtain a final denoised image corresponding to the to-be-processed Raw image.

[0118] If the Bayer array format of the first denoising image is inconsistent with that of the corresponding Raw image, the first denoising image needs to be converted to the Bayer array format consistent with that of the corresponding Raw image, so that the Bayer array format of the first denoising image is consistent with that of the corresponding Raw image.

[0119] In one embodiment, according to the judgment result, the first denoising image is processed to obtain the final denoising image corresponding to the Raw image to be processed, specifically including:

[0120] If the Bayer array format of the first denoising image is inconsistent with that of the corresponding Raw image, the first denoising image needs to be converted to the Bayer array format consistent with that of the corresponding Raw image, so that the Bayer array format of the first denoising image is consistent with that of the corresponding Raw image.

[0121] If the Bayer array format of the first denoising image is consistent with that of the corresponding Raw image, the first denoising image is taken as the final denoising image of the corresponding Raw image to be processed.

[0122] Further, if the Bayer array format of the first denoising image is inconsistent with that of the corresponding Raw image to be processed, the first denoising image needs to be converted to the Bayer array format consistent with that of the corresponding Raw image to be processed, that is, the Bayer array format of the first denoising image is converted, so that the Bayer array format of the first denoising image is consistent with that of the corresponding Raw image to be processed. Further, in order to clearly illustrate how to perform the conversion processing, the following is taken as an example: if the Bayer array format of the Raw image to be processed is "RGGB", when the Bayer array format of the corresponding first denoising image is not "RGGB", the following conversion processing can be taken: when the Bayer array format of the corresponding first denoising image is "GBRG", one pixel is added to the upper and lower boundaries of the corresponding first denoising image; when the Bayer array format of the corresponding first denoising image is "GRBG", one pixel is added to the left and right boundaries of the corresponding first denoising image; when the Bayer array format of the corresponding first denoising image is "BGGR", one pixel is added to the upper, lower, left and right boundaries of the corresponding first denoising image.

[0123] The final denoised image corresponding to the to-be-processed Raw image should be consistent with the Bayer array format of the to-be-processed Raw image.

[0124] Embodiment three:

[0125] Figure 3 The structure of the image denoising model generation device 3 provided in Embodiment three of the present application is shown, and only parts related to the embodiments of the present application are shown for ease of illustration.

[0126] In the embodiments of the present application, the image denoising model generation device 3 comprises an acquisition module 31, a first processing module 32, a second processing module 33, and a training module 34, wherein:

[0127] The acquisition module 31 is configured to acquire N first Raw images of different Bayer arrays under the same scene, N being an integer greater than or equal to 2.

[0128] The first processing module 32 is configured to perform fusion processing on the N first Raw images of different Bayer arrays respectively to obtain corresponding N second Raw images.

[0129] The second processing module 33 is configured to establish a sample image database based on the first Raw images and the second Raw images.

[0130] The training module 34 is configured to perform model training based on the sample image database to obtain an image denoising model.

[0131] In the embodiments of the present application, each module of the image denoising generation device can be realized by a corresponding hardware or software unit, and each module can be an independent software or hardware unit, or can be integrated into a software or hardware unit, which does not limit the present application.

[0132] Embodiment four:

[0133] Figure 4 The structure of the image denoising device 4 provided in Embodiment four of the present application is shown, and only parts related to the embodiments of the present application are shown for ease of illustration.

[0134] In the embodiments of the present application, the image denoising device 4 comprises an acquisition module 41, a first processing module 42, and a second processing module 43, wherein:

[0135] The acquisition module 41 is configured to acquire a to-be-processed Raw image.

[0136] The first processing module 42 is configured to obtain a corresponding input image according to the Bayer array of a preset image denoising model and the to-be-processed Raw image.

[0137] The second processing module 43 is configured to input the input image into a corresponding image denoising model to perform denoising processing to obtain a first denoised image.

[0138] The preset image denoising model is obtained by any one of the image denoising model generation methods in Embodiment One.

[0139] In the embodiments of the present application, each module of the image denoising device can be realized by a corresponding hardware or software unit, and each module can be an independent software or hardware unit, or can be integrated into a software or hardware unit, which does not limit the present application.

[0140] Embodiment Five

[0141] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the above image denoising model generation method embodiments, for example, Figure 1 the steps S101 to S104 shown.

[0142] Alternatively, the computer program is executed by the processor to implement the functions of each unit in the above device embodiments, for example, Figure 3 the functions of the units 31 to 34 shown.

[0143] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the above image denoising method embodiments, for example, Figure 2 the steps S201 to S203 shown.

[0144] Alternatively, the computer program is executed by the processor to implement the functions of each unit in the above device embodiments, for example, Figure 4 the functions of the units 41 to 43 shown.

[0145] The computer-readable storage medium of the embodiments of the present application can include any entity or device capable of carrying computer program code, recording medium, such as ROM / RAM, magnetic disk, optical disk, flash memory, etc.

[0146] Embodiment Six

[0147] Figure 5 The structure of the electronic device provided by Embodiment Six of the present application is shown, and only the parts related to the embodiments of the present application are shown for ease of illustration.

[0148] In one embodiment, the electronic device 5 of the present application comprises a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. The processor 50 implements the steps in the above-mentioned image denoising model generation method embodiments when executing the computer program 52, for example Figure 1 the steps S101 to S104 shown in the above-mentioned image denoising model generation method embodiments. Alternatively, the processor 50 implements the functions of the modules in the above-mentioned device embodiments when executing the computer program 52, for example Figure 3 the functions of the modules 31 to 34 shown in the above-mentioned image denoising model generation method embodiments.

[0149] In one embodiment, the electronic device 5 of the present application comprises a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. The processor 50 implements the steps in the above-mentioned image denoising method embodiments when executing the computer program 52, for example Figure 2 the steps S201 to S203 shown in the above-mentioned image denoising method embodiments. Alternatively, the processor 50 implements the functions of the modules in the above-mentioned device embodiments when executing the computer program 52, for example Figure 4 the functions of the modules 41 to 43 shown in the above-mentioned image denoising method embodiments.

[0150] The electronic device of the embodiments of the present application can be a smartphone, a personal computer, or a panoramic camera itself, etc. The steps implemented by the processor 50 in the electronic device 5 when executing the computer program 52 to implement the image denoising model generation method or the image denoising method can refer to the descriptions of the above-mentioned method embodiments, which will not be repeated here.

[0151] The above-mentioned is only the preferred embodiment of the present application, and does not limit the present application. Any modification, equivalent replacement, and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for generating an image denoising model, characterized in that, The method includes: Obtain the first raw images of N different Bayer arrays in the same scene, where N is an integer greater than or equal to 2; The N first raw images of different Bayer arrays are fused to obtain the corresponding N second raw images; A sample image database is established based on the first Raw image and the second Raw image; The image denoising model is obtained by training the model based on the sample image database.

2. The method as described in claim 1, characterized in that, The N first raw images with different Bayer arrays are fused to obtain the corresponding N second raw images, specifically including: Select any one of the N first Raw images as the reference image in sequence; The differences between the reference image and the other N-1 first Raw images are detected respectively; Based on the differences, the reference image is processed accordingly to obtain the image to be fused. The images to be fused are subjected to pixel-by-pixel weighted fusion processing to obtain the corresponding second Raw image.

3. The method as described in claim 2, characterized in that, Detecting the differences between the reference image and the other N-1 first raw images, specifically including: The N first Raw images are each divided into G grids, where G is an integer greater than or equal to 2; An alignment algorithm is used to calculate the differences between each grid content of the reference image and the corresponding grid content of the other N-1 first Raw images.

4. The method as described in claim 2, characterized in that, Based on the differences, the reference image is processed accordingly to obtain the image to be fused, specifically including: If the difference is detected as a difference caused by movement in a random direction, then the corresponding grid in the N first Raw images is deleted to obtain N images to be fused. If the difference is detected as a difference caused by movement in a single direction, then the grids corresponding to the N-1 first Raw images are translated in the opposite direction to the reference image in that single direction, and the N-1 translated images and the reference image are used as N images to be fused.

5. The method as described in claim 1, characterized in that, Based on the first Raw image and the second Raw image, a sample image database is established, specifically including: If the Bayer arrays of the first Raw image and the second Raw image are inconsistent, then the first Raw image and the second Raw image are converted to obtain a first Raw image and a second Raw image with consistent Bayer arrays.

6. The method as described in claim 5, characterized in that, Based on the aforementioned sample image database, a model is trained to obtain an image denoising model, specifically including: The first Raw image with the same Bayer array is used as the input image data, and the second Raw image with the same Bayer array as the first Raw image is used as the corresponding output image data. The model is trained to obtain the image denoising model.

7. An image denoising method, characterized in that, The method includes: Obtain the raw image to be processed; The corresponding input image is obtained based on the Raw image to be processed and the Bayer array of the preset image denoising model; The input image is input into the corresponding image denoising model for denoising processing to obtain the first denoised image; The preset image denoising model is obtained by the image denoising model generation method according to any one of claims 1-6.

8. The method as described in claim 7, characterized in that, Based on the raw image to be processed and the Bayer array of the preset image denoising model, the corresponding input image is obtained, specifically including: Determine whether the Bayer array of the raw image to be processed is consistent with the Bayer array of the preset image denoising model; If the raw image to be processed is inconsistent with the Bayer array of the preset image denoising model, the raw image to be processed is converted to obtain the corresponding input image. If the raw image to be processed is consistent with the Bayer array of the preset image denoising model, then the raw image to be processed is used as the corresponding input image; The input image is consistent with the Bayer array of the corresponding image denoising model.

9. The method as described in claim 7, characterized in that, After inputting the input image into the corresponding image denoising model for denoising processing to obtain the first denoised image, the process further includes: Determine whether the Bayer array of the first denoised image is consistent with the corresponding Raw image to be processed; Based on the judgment result, the first denoised image is processed accordingly to obtain the final denoised image corresponding to the raw image to be processed.

10. The method as described in claim 9, characterized in that, Based on the judgment result, the first denoised image is processed accordingly to obtain the final denoised image corresponding to the raw image to be processed, specifically including: If the Bayer array of the first denoised image is inconsistent with that of the corresponding Raw image to be processed, the first denoised image is converted to obtain the final denoised image that is consistent with the Bayer array of the corresponding Raw image to be processed. If the first denoised image has the same Bayer array as the corresponding Raw image to be processed, then the first denoised image is used as the final denoised image of the corresponding Raw image to be processed.

11. An image denoising model generation device, characterized in that, The device includes: Acquisition module: used to acquire the first raw image of N different Bayer arrays in the same scene, where N is an integer greater than or equal to 2; First processing module: used to fuse the N first raw images of different Bayer arrays respectively to obtain the corresponding N second raw images; The second processing module is used to establish a sample image database based on the first Raw image and the second Raw image. Training module: Used to train the model based on the sample image database to obtain the image denoising model.

12. An image noise reduction device, characterized in that, The device includes: Acquisition module: Used to acquire the raw image to be processed; First processing module: used to obtain the corresponding input image based on the raw image to be processed and the Bayer array of the preset image denoising model; The second processing module is used to input the input image into the corresponding image denoising model for denoising processing to obtain the first denoised image. The preset image denoising model is obtained by the image denoising model generation method according to any one of claims 1-6.

13. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 10.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 10.

Images