Noise reduction model training method, image noise reduction method, equipment and product
By fine-tuning the real data of the first noise reduction model (large model) and indirectly distilling the second noise reduction model (small model), the problem of the existing noise reduction model poor performance in complex noise scenarios is solved, and a more efficient and robust image noise reduction effect is achieved.
Patent Information
- Application Number
- CN202510180247.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-19
- Publication Date
- 2025-06-06
AI Technical Summary
Existing noise reduction models perform poorly when dealing with complex noise scenarios, especially in low-light scenarios. Traditional methods are difficult to improve model performance and effectiveness, and at the same time increase the difficulty of model learning.
By obtaining several real data pairs, the trained first noise reduction model (big model) is fine-tuned, and the fine-tuned large model is used to indirectly distillate the second noise reduction model (small model) to obtain the target noise reduction model.
It improves the generalization ability and robustness of the target noise reduction model, reduces the need for real data and the difficulty of learning real data pairs directly in small models, and improves the image noise reduction effect.
Smart Images
Figure CN120107102A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of computer vision and image processing, and in particular to a denoising model training method, an image denoising method, a device and a product. Background Art
[0002] In image denoising tasks, the industry usually uses Poisson-Gaussian distribution (PG) to synthesize shot noise and readout noise data for training denoising models. However, in some scenarios, such as low-light scenarios, the performance of readout noise is more complicated and does not always conform to the Gaussian distribution. It also contains other noise components, such as fixed pattern noise, truncation noise, and quantization noise. These factors lead to unsatisfactory performance of traditional denoising models.
[0003] The related technologies for improving the performance of traditional denoising models usually include the following two methods:
[0004] The first method is to use the large model and the collected real noise data to construct a pseudo-real data pair to fine-tune the small model. However, the core problem of this method is that the performance of the small model is highly dependent on the effect of the large model, and the training of the large model is still based on synthetic data, so the performance of the resulting denoising small model has an upper limit.
[0005] The second method is to use long and short exposure and multi-frame methods. In this method, real data pairs are collected through long and short exposure or multi-frame shooting to fine-tune the small model. This method also has some limitations: it is difficult for the small model to accurately learn the mapping of the real label, and the denoising results may have false textures, false colors, etc.; and the collection of real data pairs is difficult, usually requiring a relatively static scene, and the effect of the small model is highly related to the diversity of the samples, resulting in smearing and other problems in the denoising results.
[0006] It can be seen that how to improve the performance and effect of the denoising model while reducing the difficulty of model learning is a technical problem that people in this field urgently need to solve. Summary of the invention
[0007] The purpose of the present invention is to provide a denoising model training method, an image denoising method, a device and a product to solve the technical problem of how to improve the performance and effect of the denoising model while reducing the difficulty of model learning.
[0008] In order to solve the above technical problems, the present invention provides a method for training a denoising model, comprising:
[0009] Get some real data pairs;
[0010] Using some real data to fine-tune the trained first denoising model;
[0011] The trained second denoising model is indirectly distilled through the fine-tuned first denoising model to obtain a target denoising model, wherein the scale of the first denoising model is larger than the scale of the second denoising model.
[0012] Preferably, the method of indirectly distilling the trained second denoising model through the fine-tuned first denoising model to obtain a target denoising model includes:
[0013] Obtaining a distilled data pair through the fine-tuned first denoising model;
[0014] The trained second denoising model is fine-tuned using the distilled data to obtain a target denoising model.
[0015] Preferably, obtaining the distilled data pair through the fine-tuned first denoising model comprises:
[0016] Get real noise samples;
[0017] The fine-tuned first denoising model is run according to the real noise sample to obtain the distilled data pair.
[0018] Preferably, running the fine-tuned first denoising model according to the real noise sample to obtain the distilled data pair includes:
[0019] The real noise sample is input into the fine-tuned first denoising model to obtain a clean sample, and the clean sample and the real noise sample form a distilled data pair.
[0020] Preferably, the fine-tuning training of the trained second denoising model using the distilled data to obtain a target denoising model includes:
[0021] Inputting the real noise sample into the trained second denoising model to obtain corresponding denoised sample data;
[0022] The trained second denoising model is optimized by using the denoised sample data and the clean samples to obtain the target denoising model.
[0023] Preferably, at least one of the trained first denoising model and the trained second denoising model is obtained by training with a synthetic sample training set, and the number of samples in the synthetic sample training set is greater than the number of the real data pairs.
[0024] In order to solve the above technical problem, the present invention also provides an image denoising method, the method comprising:
[0025] Acquire an image to be denoised;
[0026] The image to be denoised is input into a target denoising model to obtain a denoised image; wherein the target denoising model is the target denoising model in the above-mentioned denoising model training method.
[0027] In order to solve the above technical problems, the present invention also provides a training device for a noise reduction model, comprising:
[0028] An acquisition module, used to acquire several real data pairs;
[0029] A fine-tuning module, used for fine-tuning the trained first denoising model using a plurality of real data;
[0030] A distillation module is used to indirectly distill the trained second denoising model through the fine-tuned first denoising model to obtain a target denoising model, wherein the scale of the first denoising model is larger than the scale of the second denoising model.
[0031] In order to solve the above technical problem, the present invention further provides an electronic device, comprising:
[0032] Memory for storing computer programs;
[0033] A processor is used to implement the steps of the above-mentioned denoising model training method or the above-mentioned image denoising method when executing the computer program.
[0034] In order to solve the above technical problems, the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned denoising model training method or the above-mentioned image denoising method.
[0035] In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned denoising model training method or the above-mentioned image denoising method are implemented.
[0036] The present invention first obtains a number of real data pairs; then fine-tunes the trained first denoising model using the real data pairs; finally, indirectly distills the trained second denoising model through the fine-tuned first denoising model to obtain a target denoising model. Since the scale of the first denoising model is larger than that of the second denoising model, the first denoising model can be called a large model, and the second denoising model can be called a small model. First, compared with the previous large model refinement method that only relies on synthetic data to train the large model, in the method provided by the present invention, since the large model is fine-tuned using several real data pairs, when the fine-tuned first denoising model is used to indirectly distill the trained second denoising model, the generalization ability and robustness of the obtained target denoising model can be improved; secondly, different from the method of driving the small model with a large amount of real data, the method provided by the present invention uses the strong prior of "learning features are more robust and less dependent on sample diversity" of the large model to select real data pairs to fine-tune the large model, and then indirectly distill the small model, thereby reducing the demand for real data; thirdly, compared with the previous method of using long and short exposure and multi-frame methods to collect real data pairs to fine-tune the small model, in the method provided by the present invention, the small model is indirectly distilled through the fine-tuned large model, thereby reducing the difficulty of the small model directly learning the real data pairs, and can also ensure a sufficiently complete real input noise domain distribution. It can be seen that the target denoising model finally provided by the present invention improves the performance and effect of the model compared with the related art, and reduces the difficulty of model learning. Therefore, the target denoising model improves the image denoising effect when performing image denoising. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0038] Figure 1 A flowchart of a method for training a denoising model provided by an embodiment of the present invention;
[0039] Figure 2 An overall schematic diagram of a method for training a denoising model provided by an embodiment of the present invention;
[0040] Figure 3 A comparison effect diagram of a noise reduction model involved in an embodiment of the present invention;
[0041] Figure 4 A structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0043] As mentioned in the background technology, traditional image denoising models still have significant limitations when dealing with complex noise in certain scenes. Therefore, there is an urgent need for a more effective solution to improve the performance of image denoising models while reducing the difficulty of learning the models.
[0044] Based on this, the present invention proposes a denoising model distillation method based on real data driving. The core of the present invention is to provide a denoising model training method, an image denoising method, a device and a product. The scheme combines the advantages of large model refinement method, long and short exposure and real data of multi-frame schemes, which not only improves the performance of the image denoising model, but also overcomes the limitations of traditional methods, thereby solving the technical problems of poor performance and effect of the denoising model and high difficulty of model learning. Overall, it provides an efficient and robust solution for image denoising tasks.
[0045] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0046] Figure 1 A flowchart of a method for training a noise reduction model provided by an embodiment of the present invention, such as Figure 1 As shown, the method includes:
[0047] S10: Obtain several real data pairs;
[0048] S11: fine-tune the trained first denoising model using some real data;
[0049] S12: Perform indirect distillation on the trained second denoising model through the fine-tuned first denoising model to obtain a target denoising model.
[0050] In this embodiment, the denoising model may be an image denoising model based on the RAW domain or an image denoising model based on other domains. The data pair refers to a clean image (noisy image) and a noisy image. The scale of the first denoising model is greater than the scale of the second denoising model. Since the scale of the first denoising model is greater than the scale of the second denoising model, the first denoising model may be referred to as a large model and the second denoising model may be referred to as a small model.
[0051] In the denoising model training method, firstly, a trained first denoising model and a trained second denoising model are obtained.
[0052] In order to reduce the difficulty of collecting a large number of real samples, in implementation, at least one of the trained first denoising model and the trained second denoising model can be obtained by training with a synthetic sample training set, and the number of samples in the synthetic sample training set is greater than the number of real data pairs, that is, the number of real data pairs is relatively small, and the number of samples in the synthetic sample training set is large. For example, a few-shot real data pairs can be used.
[0053] It should be noted that pre-training the large denoising model and the small denoising model respectively in combination with the synthetic sample training set, and then directly obtaining the trained first denoising model and the trained second denoising model, can reduce the difficulty of directly learning real data and learn sufficiently robust and stable features.
[0054] When obtaining the synthetic sample training set, clean sample data and noise parameters may be obtained, and noise sample data corresponding to the clean sample data may be synthesized according to the clean sample data and the noise parameters, thereby forming a synthetic sample training set.
[0055] For example, the noise parameters may be formed using a Poisson-Gaussian noise model of the corresponding sensor.
[0056] For another example, obtaining noise parameters includes: obtaining an image that satisfies a first preset exposure condition and an image that satisfies a second preset exposure condition; wherein the exposure conditions include at least light intensity and different image gains; the light intensity value under the first preset exposure condition is greater than the light intensity value under the second preset exposure condition; optionally, the light intensity value under the first preset exposure condition is less than the preset light intensity.
[0057] Further, the first noise parameter can be determined according to the relationship between the variance and signal strength of the image satisfying the first preset exposure condition; the second noise parameter can be determined according to the variance of the image satisfying the second preset exposure condition. The first noise parameter is the shot-noise parameter K, and the second noise parameter is the read-noise parameter B.
[0058] In implementation, high-definition texture sample sets from the Internet, including but not limited to DIV2K, DF2K, etc., can be collected as clean sample data; the shot-noise parameter K and read-noise parameter B of the image sensor are obtained based on the Poisson-Gaussian model calibration, and based on K and B, noise samples corresponding to the high-definition texture samples in the synthetic sample training set are synthesized to form pre-trained noisy-clean (noise sample, high-definition texture sample) data pairs, and the synthetic sample training set includes multiple groups of data pairs.
[0059] After obtaining the synthetic sample training set, the large model is trained to obtain the trained large model (i.e., the trained first denoising model). The basic models used in training include but are not limited to Restormer (an efficient Transformer model), Uformer (U-shaped Transformer for Image Restoration), NAFNet (NonlinearActivation Free Network), etc. The l1 loss function can be used when optimizing model training, and the training cycle (epoch) can be set to 80 to 120 rounds, for example, 80 epochs, 100 epochs or 120 epochs. The cosine annealing learning rate scheduler can be set in the training framework.
[0060] Similarly, after obtaining the synthetic sample training set, the small model is trained to obtain a trained small model (i.e., a trained second denoising model). The basic models used in training include but are not limited to Deep Neural Convolutional Network (DnCNN), PMRID, Fast and Flexible Denoising Network (FFDNet), Content-Boosted Denoising Network (CBDNet), Residual Information Denoising Network (RIDNet), etc. When performing model training optimization, the l1 loss function can be used, and the training cycle (epoch) can be set to 80 to 120 rounds, for example, 80 epochs, 100 epochs or 120 epochs. A cosine annealing learning rate scheduler can be set in the training framework.
[0061] Please also see Figure 1 and Figure 2 The above-mentioned trained first denoising model and trained second denoising model can be called a pre-training stage, which is specifically driven by pre-training synthetic data (i.e., synthetic sample training set) to realize the pre-training of the denoising large model and the denoising small model.
[0062] After obtaining the trained first denoising model and the trained second denoising model, the trained first denoising model is first fine-tuned, which is called the large model refinement stage. In the large model refinement stage, several real data pairs are obtained, and then the trained first denoising model is fine-tuned using the several real data pairs.
[0063] The aforementioned real data are collected real data with rich sample diversity, and a number of real data pairs can be constructed based on the real data.
[0064] Still taking the RAW domain as an example, when constructing or acquiring several real data pairs, you can collect long and short exposure images to construct The data pair only needs to satisfy the exposure equivalence formula as a whole.
[0065] Specifically, obtaining several real data pairs includes:
[0066] In at least one group of static scenes, images with different exposure times are collected based on multiple groups of preset image gains; wherein, the images with different exposure times include images with an exposure time less than the preset time and images with an exposure time greater than or equal to the preset time, the images with an exposure time less than the preset time are clean images, and the images with an exposure time greater than or equal to the preset time are noisy images; there is no limitation on the preset time, which is determined according to actual conditions.
[0067] A clean image and a noisy image with the same exposure level are obtained, and the clean image and the noisy image with the same exposure level are used as a set of true data pairs.
[0068] Among them, obtaining a clean image and a noisy image with the same exposure level includes:
[0069] Acquire a first preset exposure time and a first preset sensitivity of a clean image to be acquired; acquire the clean image to be acquired according to the first preset exposure time and the first preset sensitivity; acquire a second preset sensitivity and a preset image gain of a noise image to be acquired; determine the second preset exposure time of the noise image to be acquired according to the first preset exposure time, the first preset sensitivity, the second preset sensitivity and the preset image gain; acquire the noise image to be acquired according to the second preset sensitivity, the preset image gain and the second preset exposure time; use the acquired clean image to be acquired and the noise image to be acquired as the clean image and the noise image with the same exposure level.
[0070] Taking the RAW domain as an example, in the implementation, 1 to 3 groups of different static scenes are selected, and the data pairs constructed by collecting long and short exposure images based on the required low-light image gains dgain5, 15, 30, 50 and 90 are respectively , that is, we get several real data pairs.
[0071] The exposure equivalent formula for overall satisfaction is as follows:
[0072] ;
[0073] in, Indicates the specific time of long exposure. The specific time of short exposure needs to be calculated. and is a known quantity, such as Set to 100, Set to 1550.
[0074] After obtaining a number of real data pairs, the first denoising model trained can be fine-tuned using the real data pairs to obtain a fine-tuned first denoising model. When fine-tuning the first denoising model, an l1 loss function can be used, and the training cycle (epoch) of the overall model can be between 80 and 120 epochs, for example, the training cycle is 80 epochs, 100 epochs or 120 epochs, and the learning rate can be a fixed learning rate, for example, a fixed learning rate of 4e-5, 3e-5 or 5e-5.
[0075] It should be noted that the first denoising model that is fine-tuned and pre-trained with real data can utilize the prior knowledge that the large model "learns features more robustly and is not highly dependent on sample diversity". After fine-tuning the large model with a relatively small amount of real data, it can better capture specific scenes, such as complex noise features in low-light scenes, thereby optimizing the upper limit of the effect of the large model and the subsequent indirect distillation of the small model.
[0076] After obtaining the fine-tuned first denoising model, the trained second denoising model is indirectly distilled through the fine-tuned first denoising model to obtain the target denoising model.
[0077] In this embodiment, different from the method of using a large amount of real data to drive a small model, this patent uses the strong prior that the large model "learns features more robustly and is not highly dependent on sample diversity" to select real data to fine-tune the large model and then indirectly distill the small model, thereby reducing the demand for real data.
[0078] In summary, the present invention first obtains a number of real data pairs; then uses the real data pairs to fine-tune the trained first denoising model; finally, the fine-tuned first denoising model is used to indirectly distill the trained second denoising model to obtain the target denoising model. Since the scale of the first denoising model is larger than that of the second denoising model, the first denoising model can be called a large model, and the second denoising model can be called a small model.
[0079] Compared with the previous large model refinement method that only relies on synthetic data to train the large model, in the method provided by the present invention, since the large model is fine-tuned using a number of real data pairs, when the fine-tuned first denoising model is used to indirectly distill the trained second denoising model, the generalization ability and robustness of the obtained target denoising model can be improved;
[0080] Secondly, different from the method of driving a small model with a large amount of real data, the method provided by the present invention uses the strong prior of "learning features of the large model are more robust and less dependent on sample diversity" to select a number of real data pairs to fine-tune the large model, and then indirectly distill the small model, thereby reducing the demand for real data;
[0081] Thirdly, compared with the previous method of using long-short exposure and multi-frame methods to collect real data pairs and fine-tune the small model, the method provided by the present invention indirectly distills the small model through the fine-tuned large model, which reduces the difficulty of the small model directly learning the real data pairs and can also ensure a sufficiently complete real input noise domain distribution. It can be seen that the target denoising model finally provided by the present invention improves the performance and effect of the model compared with the related technology, while reducing the difficulty of model learning.
[0082] In summary, the model distillation method based on real data drive proposed in this patent not only improves the performance of the denoising model, but also overcomes the limitations of traditional methods, providing an efficient and robust solution for image denoising tasks. Therefore, the target denoising model can also improve the image denoising effect when performing image denoising.
[0083] In order to better reflect the technical effects of the embodiments of the present invention, the following is a comparison and explanation of the original image, the effect image output after the second denoising model performs denoising on the original image, and the effect image output after the target denoising model performs denoising on the original image.
[0084] The original picture is Figure 3 The noisy image on the left, Figure 3 The middle image is the output of the second denoising model before fine-tuning. It can be seen that it removes a lot of noise in the original image and restores the basic image elements. However, in some image areas, such as grass and resin branches, the processing is relatively poor. Figure 3 The performance of the output of the target denoising model on the right is worse. Figure 3 The details of the original noisy image are completely restored, and the overall image denoising effect is the best. That is, the target denoising model obtained based on indirect distillation combines the advantages of related technologies, reduces image pseudo-textures and pseudo-colors, and improves the image denoising effect.
[0085] In the implementation, the trained second denoising model is indirectly distilled through the fine-tuned first denoising model to obtain the target denoising model, including:
[0086] Obtaining a distilled data pair through the fine-tuned first denoising model;
[0087] The trained second denoising model is fine-tuned using the distilled data to obtain a target denoising model.
[0088] During implementation, this solution can input noisy image data as distillation input data into the fine-tuned large model, thereby generating clean samples, and then using the clean samples as high-quality distillation data combined with noisy image data to form distillation data pairs, providing diverse training samples, providing rich and diverse samples for the training of small models, and enriching the diversity of samples used for fine-tuning small models.
[0089] On the other hand, the small model is fine-tuned by running the distilled data pairs formed by the large model. This process avoids the sample diversity problem caused by directly collecting "data pairs" and reduces the learning difficulty of the small model. The target denoising model obtained can learn the noise feature mapping more efficiently and reduce the occurrence of false textures, false colors and other phenomena.
[0090] Further, the distilled data pair obtained by the fine-tuned first denoising model includes:
[0091] Get real noise samples;
[0092] Run the fine-tuned first denoising model on the real noise samples to obtain distilled data pairs.
[0093] Specifically, the real noise sample can be input as the distillation input data into the fine-tuned first denoising model to obtain a clean sample, and the clean sample and the distillation input data form a distillation data pair.
[0094] During implementation, the real noise samples obtained through actual collection can be used as the distillation input data noisy and input into the fine-tuned first denoising model to predict the pseudo-clean samples of the corresponding input data, also called clean samples clean, thereby constructing the clean-noisy distillation data pairs needed in the distillation stage.
[0095] It should be noted that this solution pre-trains the first denoising model and the second denoising model with synthetic data, and refines the denoising model with real data. While narrowing the learning domain gap, it reduces the denoising model's demand for real data pairs and the difficulty of directly learning real data pairs.
[0096] Subsequently, the obtained distilled data pairs can be used to fine-tune the pre-trained denoising model. The denoising model obtained after fine-tuning is the final target denoising model with improved effect.
[0097] In order to enable those skilled in the art to better understand the process of the small model distillation stage, a specific embodiment is provided below for explanation.
[0098] When obtaining real noise samples, for example, we use image sensors to collect 14 groups of indoor and outdoor noisy scenes at ISO 1600, 3200, 6400, 12800, 25600, 51200 and 102400, and try to ensure more than 10 frames in each group to construct the input data in the distillation data pair to ensure the diversity of noise and samples in the distillation stage. We use the fine-tuned first denoising model to predict the pseudo-clean samples of the corresponding input data for the obtained distillation input data, and construct the clean-noisy data pair in the distillation stage.
[0099] After obtaining the clean-noisy data pairs in the distillation stage, the target denoising model can be obtained by fine-tuning the second denoising model obtained by pre-training.
[0100] During implementation, the method of fine-tuning the trained second denoising model using the distilled data to obtain a target denoising model may include:
[0101] Input the distilled input data into the trained second denoising model to obtain the corresponding denoised sample data;
[0102] The trained second denoising model is optimized through denoising sample data and clean samples to obtain the target denoising model.
[0103] The l1 loss function can be used for fine-tuning, and the training cycle (epoch) is 80 to 120. For example, the trainer is 80, 100 or 120 epochs. The cosine annealing learning rate scheduler can be used in the model fine-tuning process, and its learning rate ranges from 4×10⁻ 5 Linear decay to 1×10⁻ 6 , or you can also start from 5×10⁻ 5 Linear decay to 2×10⁻ 6 .
[0104] The above-mentioned stage of indirectly distilling the trained second denoising model through the fine-tuned first denoising model to obtain the target denoising model can be called the small model distillation stage.
[0105] In this scheme, the input data with rich sample diversity collected from the real world is used to input into the fine-tuned large model to generate clean samples, which are then used as high-quality distillation data to form distillation data pairs, providing diverse training samples. This provides rich and diverse samples for the training of small models, avoiding the limitations of real data due to its difficulty in collecting data, and enriching the diversity of samples used for fine-tuning small models.
[0106] On the other hand, the small model is fine-tuned by running the distilled data pairs formed by the large model. This process avoids the sample diversity problem caused by directly collecting "data pairs" and reduces the learning difficulty of the small model. The target denoising model obtained can learn noise feature mapping more efficiently, reduce the occurrence of false textures, false colors and other phenomena, and ensure a sufficiently complete real input noise domain distribution. Overall, the small model performs better in denoising tasks.
[0107] In order to make those skilled in the art better understand the overall technical solution of the present invention, the following Figure 2 The training method of the denoising model of the present invention is described. Figure 2 An overall schematic diagram of a denoising model training method provided by an embodiment of the present invention.
[0108] like Figure 2 As shown in the figure, the training process includes the data preparation stage, the pre-training stage, the large model refinement stage and the small model distillation stage.
[0109] In the data preparation stage, the main tasks are to obtain clean sample data, use clean sample data to synthesize noisy sample data, obtain real data pairs, and obtain real input data; in the pre-training stage: use clean sample data and noisy sample data to pre-train the denoising large model (i.e., the first denoising model) and the denoising small model (i.e., the second denoising model); in the large model refinement stage: use real data pairs to fine-tune the large model to obtain a refined denoising large model.
[0110] In the small model distillation stage: input the real input data into the refined denoising large model (i.e. the first denoising model after fine-tuning), and output the noisy-clean distilled data (i.e. the distilled data pair). Use the noisy-clean distilled data to distill the small model to obtain the refined denoising small model, i.e. the target denoising model.
[0111] In the training method of the denoising model provided by the present invention, firstly, a number of real data pairs are obtained; then, the trained first denoising model is fine-tuned using the number of real data pairs; finally, the trained second denoising model is indirectly distilled by the fine-tuned first denoising model to obtain the target denoising model. Since the scale of the first denoising model is larger than that of the second denoising model, the first denoising model can be called a large model, and the second denoising model can be called a small model. Firstly, compared with the previous large model refinement method that only relies on synthetic data to train the large model, in the method provided by the present invention, since the large model is fine-tuned using a number of real data, when the trained second denoising model is indirectly distilled using the fine-tuned first denoising model, the generalization ability and robustness of the obtained target denoising model are improved; secondly, different from the method of driving the small model with a large amount of real data, the method provided by the present invention uses the strong prior of the large model that "the learning features are more robust and the degree of dependence on sample diversity is not high", selects a number of real data pairs to fine-tune the large model, and then indirectly distills the small model, thereby reducing the demand for real data; thirdly, compared with the previous method of using long The short exposure and multi-frame methods are used to collect real data pairs to fine-tune the small model. In the method provided by the present invention, the small model is indirectly distilled through the fine-tuned large model, which reduces the difficulty of the small model directly learning the real data pairs, and can also ensure a sufficiently complete real input noise domain distribution; and the small model is fine-tuned by collecting distilled input data and constructing the input data pairs of the distillation stage through the large model, avoiding the difficulty of directly collecting real data pairs and the problem of sample diversity; in addition, through the method of synthetic data pre-training + real data pair refinement, while narrowing the learning domain gap, the denoising model's demand for real data pairs and the difficulty of directly learning real data pairs are reduced. It can be seen that the denoising model training method provided by the present invention improves the performance and effect of the model and reduces the difficulty of model learning.
[0112] A method for training a denoising model is described above. The present invention also provides an image denoising method, which comprises:
[0113] Acquire an image to be denoised;
[0114] The image to be denoised is input into the target denoising model to obtain a denoised image; wherein the target denoising model is the target denoising model in the above-mentioned denoising model training method.
[0115] The above has described in detail the embodiment of obtaining the target denoising model, which will not be repeated here. When denoising an image, first obtain the image to be denoised. There is no limitation on the selected image to be denoised, and the selection is made according to the actual situation. The image to be denoised is input into the target denoising model, and the denoised image corresponding to the image to be denoised can be output through the target denoising model. The denoised image obtained by the model has been greatly improved in many aspects such as the denoising effect.
[0116] In the above-mentioned embodiments, the training method of the denoising model is described in detail. The present invention also provides an embodiment corresponding to the training device of the denoising model.
[0117] It should be noted that the present invention describes the embodiments of the device part from two perspectives, one is based on the perspective of functional modules, and the other is based on the perspective of hardware.
[0118] The denoising model training device provided by the embodiment of the present invention includes:
[0119] An acquisition module, used to acquire several real data pairs;
[0120] A fine-tuning module, used for fine-tuning the trained first denoising model using a plurality of real data;
[0121] The distillation module is used to indirectly distill the trained second denoising model through the fine-tuned first denoising model to obtain a target denoising model, wherein the scale of the first denoising model is larger than that of the second denoising model.
[0122] Since the embodiments of the device part correspond to the embodiments of the method part, the embodiments of the device part refer to the description of the embodiments of the method part, which will not be described here. And it has the same beneficial effects as the training method of the denoising model mentioned above.
[0123] Figure 4 This is a structural diagram of an electronic device provided by an embodiment of the present invention. This embodiment is based on the hardware perspective, such as Figure 4 As shown, the electronic equipment includes:
[0124] A memory 20, for storing computer programs;
[0125] The processor 21 is used to implement the method of training a denoising model or the steps of the image denoising method mentioned in the above embodiments when executing a computer program.
[0126] The processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 may be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA).
[0127] The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen.
[0128] In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.
[0129] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices, flash memory storage devices.
[0130] In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein after the computer program is loaded and executed by the processor 21, it can implement the method for training the denoising model or the relevant steps of the image denoising method disclosed in any of the aforementioned embodiments.
[0131] In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, and the storage method may be temporary storage or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include but is not limited to the data involved in the above-mentioned method for training the denoising model or the image denoising method.
[0132] In some embodiments, the electronic device may further include a display screen 22 , an input / output interface 23 , a communication interface 24 , a power source 25 , and a communication bus 26 .
[0133] Those skilled in the art will understand that Figure 4 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure.
[0134] The electronic device provided by an embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the following method: a method for training a denoising model or an image denoising method, and the effect is the same as above.
[0135] The present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned denoising model training method or the above-mentioned image denoising method.
[0136] Finally, the present invention also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps recorded in the above method embodiment are implemented.
[0137] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store program codes.
[0138] The computer-readable storage medium provided by the present invention includes the above-mentioned method for training a denoising model or an image denoising method, and the effect is the same as above.
[0139] The above is a detailed introduction to the training method of the denoising model, the image denoising method, the device and the product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can refer to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.
[0140] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
Claims
1. A method for training a denoising model, characterized in that: include: Get some real data pairs; Using some real data to fine-tune the trained first denoising model; The trained second denoising model is indirectly distilled through the fine-tuned first denoising model to obtain a target denoising model, wherein the scale of the first denoising model is larger than the scale of the second denoising model.
2. The method for training a denoising model according to claim 1, characterized in that: The method indirectly distills the trained second denoising model through the fine-tuned first denoising model to obtain a target denoising model, including: Obtaining a distilled data pair through the fine-tuned first denoising model; The trained second denoising model is fine-tuned using the distilled data to obtain a target denoising model.
3. The method for training a denoising model according to claim 2, characterized in that: The step of obtaining a distilled data pair through the fine-tuned first denoising model comprises: Get real noise samples; The fine-tuned first denoising model is run according to the real noise sample to obtain the distilled data pair.
4. The method for training a denoising model according to claim 3, characterized in that: The step of running the fine-tuned first denoising model according to the real noise sample to obtain the distilled data pair includes: The real noise sample is input into the fine-tuned first denoising model to obtain a clean sample, and the clean sample and the real noise sample form a distilled data pair.
5. The method for training a denoising model according to claim 4, characterized in that: The step of fine-tuning the trained second denoising model using the distilled data to obtain a target denoising model includes: Inputting the real noise sample into the trained second denoising model to obtain corresponding denoised sample data; The trained second denoising model is optimized by using the denoised sample data and the clean samples to obtain the target denoising model.
6. The method for training a denoising model according to any one of claims 1 to 5, characterized in that: At least one of the trained first denoising model and the trained second denoising model is obtained by training with a synthetic sample training set, and the number of samples in the synthetic sample training set is greater than the number of the real data pairs.
7. An image denoising method, characterized in that: The method comprises: Acquire an image to be denoised; The image to be denoised is input into a target denoising model to obtain a denoised image; wherein the target denoising model is the target denoising model in the denoising model training method according to any one of claims 1 to 6.
8. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the denoising model training method according to any one of claims 1 to 6 or the steps of the image denoising method according to claim 7 when executing the computer program.
9. A computer program product, characterized in that It comprises a computer program / instruction, which, when executed by a processor, implements the steps of the training method of the denoising model as described in any one of claims 1 to 6 or the steps of the image denoising method as described in claim 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the training method of the denoising model according to any one of claims 1 to 6 or the steps of the image denoising method according to claim 7 are implemented.