A model training method, denoising method, model, device and storage medium

Abstract

The embodiment of the present application provides a model training method, a denoising method, a model, a device and a storage medium. In the embodiment of the present application, a first adversarial network including a background generator and a noise discriminator can be constructed in the denoising model. Based on this, the noise features in the noisy image can be determined according to the background image extracted from the noisy image by the background generator, and the noise features can be discriminated by the noise discriminator. In this way, through the mutual confrontation between the background generator and the noise discriminator, the first constraint parameter corresponding to the background generator can be calculated, so that the background generator can be unsupervisedly trained based on the first constraint parameter. In the embodiment of the present application, the training constraints for the background generator can be determined based on the noisy image itself, so that the unsupervised training of the background generator in the denoising model can be realized. Since the noisy image in the natural scene can be used as a training sample, the generalization ability and accuracy of the background generator can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a model training method, a denoising method, a model, a device and a storage medium. Background Art

[0002] Image De-raining technology, as an image enhancement technology in the direction of deep learning and computer vision, has demonstrated great practical value in scenarios such as surveillance videos.

[0003] At present, it is difficult to collect paired clean and rainy training samples in real scenes. Therefore, it is necessary to use PS synthesis and other techniques to add raindrops to the original clean pictures, and then rely on rainy-clean paired images as training sets for full supervision training of the model. However, the characteristics of synthetic raindrops are quite different from those of raindrops in real scenes, resulting in insufficient generalization ability of the model generated by this training method, and the deraining accuracy in real natural scenes is too low. Summary of the invention

[0004] Multiple aspects of the present application provide a model training method, a denoising method, a model, a device and a storage medium to improve the generalization ability and accuracy of the denoising model.

[0005] The present application embodiment provides a model training method applicable to a denoising model, wherein the denoising model includes a first adversarial network, and the method includes:

[0006] Extracting a background image from a noisy image using the background generator in the first adversarial network;

[0007] Determining noise features contained in the noisy image according to the background image;

[0008] Using the noise discriminator in the first adversarial network to discriminate the noise feature, and calculating the first constraint parameter corresponding to the background generator according to the discrimination result;

[0009] Based on the first constraint parameter, unsupervised training is performed on the background generator in the denoising model.

[0010] The present application also provides a model training method, including:

[0011] In response to a request to call a target service, a processing resource corresponding to the target service is determined, and the following steps are performed using the processing resource corresponding to the target service:

[0012] The background generator in the first adversarial network in the denoising model is used to extract the background image from the noisy image;

[0013] Determining noise features contained in the noisy image according to the background image;

[0014] Using the noise discriminator in the first adversarial network to discriminate the noise feature, and calculating the first constraint parameter corresponding to the background generator according to the discrimination result;

[0015] Based on the first constraint parameter, unsupervised training is performed on the background generator in the denoising model.

[0016] The embodiment of the present application also provides a model training method, which is applicable to a denoising model, wherein the denoising model includes a background generator and a background enhancer, and the method includes:

[0017] Extracting a background image from a noisy image using the background generator;

[0018] The background enhancer is used to calculate a difference index between the noisy image and the background image at a specified blur level as a constraint parameter corresponding to the background generator;

[0019] The constraint parameters are used to perform unsupervised training on the background generator in the denoising model.

[0020] The present application also provides a model training method, including:

[0021] In response to a request to call a target service, a processing resource corresponding to the target service is determined, and the following steps are performed using the processing resource corresponding to the target service:

[0022] The background generator in the denoising model is used to extract the background image from the noisy image;

[0023] The background enhancer in the denoising model is used to calculate the difference index between the noisy image and the background image at a specified blur level as a constraint parameter corresponding to the background generator;

[0024] The constraint parameters are used to perform unsupervised training on the background generator in the denoising model.

[0025] The embodiment of the present application also provides a computing device, including a memory and a processor;

[0026] The memory is used to store one or more computer instructions;

[0027] The processor is coupled to the memory and configured to execute the one or more computer instructions for:

[0028] A background generator included in the first adversarial network in the denoising model is used to extract a background image from the noisy image;

[0029] Determining noise features contained in the noisy image according to the background image;

[0030] Using the noise discriminator in the first adversarial network to discriminate the noise feature, and calculating the first constraint parameter corresponding to the background generator according to the discrimination result;

[0031] Based on the first constraint parameter, unsupervised training is performed on the background generator in the denoising model.

[0032] The embodiment of the present application also provides a computing device, including a memory and a processor;

[0033] The memory is used to store one or more computer instructions;

[0034] The processor is coupled to the memory and configured to execute the one or more computer instructions for:

[0035] The background generator in the denoising model is used to extract the background image from the noisy image;

[0036] The background enhancer in the denoising model is used to calculate the difference index between the noisy image and the background image at a specified blur level as a constraint parameter corresponding to the background generator;

[0037] The constraint parameters are used to perform unsupervised training on the background generator in the denoising model.

[0038] The embodiment of the present application also provides a denoising model, comprising a first adversarial network, wherein the first adversarial network includes a background generator and a noise discriminator;

[0039] The background generator is used to extract a background image from a noisy image;

[0040] The rain discriminator is used to obtain noise features contained in the noisy image and discriminate the noise features to generate first constraint parameters corresponding to the background generator; wherein the noise features are determined based on the background image;

[0041] The background generator is further used to update its own model parameters based on the first constraint parameters.

[0042] The present application also provides a denoising method, including:

[0043] In response to a denoising request for a target image, acquiring the target image;

[0044] Extracting a background image from the target image using a background generator included in the first adversarial network in the denoising model to denoise the target image;

[0045] The training process of the background generator includes:

[0046] Extracting a sample background image from a noisy sample image using the background generator;

[0047] Determining, according to the sample background image, a sample noise feature contained in the noisy sample image;

[0048] Using the noise discriminator in the first adversarial network to discriminate the sample noise feature, and calculating the first constraint parameter corresponding to the background generator according to the discrimination result;

[0049] Based on the first constraint parameter, unsupervised training is performed on the background generator in the denoising model.

[0050] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the aforementioned model training method.

[0051] In an embodiment of the present application, a first adversarial network including a background generator and a noise discriminator can be constructed in a denoising model, wherein the background generator is a model unit in the denoising model that can perform denoising on the input image. On this basis, the background generator can be used to extract a background image from a noisy image; thereafter, the noise features in the noisy image can be determined based on the background image, and the noise discriminator can be used to discriminate the noise features. In this way, through the mutual confrontation between the background generator and the noise discriminator, the first constraint parameter corresponding to the background generator can be calculated, so that the background generator can be unsupervisedly trained based on the first constraint parameter. Accordingly, in an embodiment of the present application, the training constraints for the background generator can be determined based on the noisy image itself, so that the unsupervised training of the background generator in the denoising model can be realized, and then a large number of unlabeled noisy images in natural scenes can be involved in the training of the background generator; since the noisy images in natural scenes can be used as training samples, the generalization ability and accuracy of the background generator can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0053] Figure 1 A schematic diagram of a denoising model provided by an exemplary embodiment of the present application;

[0054] Figure 2aA flowchart of a model training method provided for an exemplary embodiment of the present application;

[0055] Figure 2b A flowchart of another model training method provided for an exemplary embodiment of the present application;

[0056] Figure 3 A processing logic diagram of a background enhancer provided for an exemplary embodiment of the present application;

[0057] Figure 4a A flowchart of another model training method provided for an exemplary embodiment of the present application;

[0058] Figure 4b A flowchart of another model training method provided for an exemplary embodiment of the present application;

[0059] Figure 5 A schematic diagram of a computing device provided for another exemplary embodiment of the present application;

[0060] Figure 6 A schematic diagram of the structure of another computing device provided for another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0062] In view of the technical problems such as low generalization ability and insufficient precision of the existing denoising model, in some embodiments of the present application, a first adversarial network including a background generator and a noise discriminator can be constructed in the denoising model, wherein the background generator is a model unit in the denoising model that can denoise the input image. On this basis, the background generator can be used to extract the background image from the noisy image; then, the noise features in the noisy image can be determined according to the background image, and the noise discriminator can be used to discriminate the noise features. In this way, through the mutual confrontation between the background generator and the noise discriminator, the first constraint parameter corresponding to the background generator can be calculated, so that the background generator can be unsupervisedly trained based on the first constraint parameter. Accordingly, in the embodiment of the present application, the training constraints for the background generator can be determined based on the noisy image itself, so that the unsupervised training of the background generator in the denoising model can be realized, and then a large number of unlabeled noisy images in natural scenes can be involved in the training of the background generator; since the noisy images in natural scenes can be used as training samples, the generalization ability and precision of the background generator can be effectively improved.

[0063] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0064] Figure 1 A schematic diagram of the structure of a denoising model provided for an exemplary embodiment of the present application. Figure 2a A flowchart of a model training method provided for an exemplary embodiment of the present application. The present embodiment of the application proposes an unsupervised training scheme for a denoising model, which can involve a large number of unlabeled noisy images in natural scenes in the training process of the denoising model, that is, the noisy images in natural scenes can be directly used as training samples of the denoising model. Compared with the traditional supervised training scheme, the unsupervised training scheme provided in this embodiment can effectively improve the generalization ability and accuracy of the denoising model in natural scenes.

[0065] The denoising model training method provided in this embodiment can be applied to various application scenarios that require denoising processing to improve the performance of the denoising models used in these application scenarios. These application scenarios include but are not limited to video surveillance scenarios, audio and video editing scenarios, etc. In different application scenarios, the sample objects or the objects of denoising processing may not be exactly the same. For example, the sample objects or the objects of denoising processing may be surveillance videos, game videos, etc.

[0066] In this embodiment, the denoising model can be used to denoise an image, and is particularly suitable for denoising an image containing Gaussian noise. The characteristics of Gaussian noise are high frequency and statistically conform to Gaussian distribution. Gaussian noise may include rain noise, snow noise, etc. Accordingly, the noisy image in this embodiment may be an image with rain and snow, and the noise features contained in the noisy image may be rain and snow features. Therefore, the denoising model provided in this embodiment can be used to denoise an image.

[0067] refer to Figure 1 The denoising model in this embodiment includes a first adversarial network, and the first adversarial network includes a background generator 10 and a noise discriminator 20. In this embodiment, in the training stage of the denoising model, the background generator 10 is mainly optimized; and in the use stage of the denoising model, the background image output by the background generator 10 can be used as the model output result of the denoising model, thereby realizing the denoising function of the denoising model.

[0068] based on Figure 1 The denoising model shown in Figure 2a The model training method provided in this embodiment may include:

[0069] Step 100, extracting a background image from a noisy image using the background generator 10 in the first adversarial network;

[0070] Step 101: determining noise features contained in a noisy image according to a background image;

[0071] Step 102: using the noise discriminator 20 in the first adversarial network to discriminate the noise feature, and calculating the first constraint parameter corresponding to the background generator 10 according to the discrimination result;

[0072] Step 103: Based on the first constraint parameter, perform unsupervised training on the background generator 10 in the denoising model.

[0073] Since the model training process is an iterative process, the logic of each iterative process is similar. Therefore, for ease of description, the following will take a single iterative process as an example to illustrate the model training scheme. It should be understood that in this embodiment, the following iterative process can be repeated multiple times to achieve the training goal.

[0074] refer to Figure 1 and Figure 2a, for the background generator 10 in the first adversarial network, it can be used to extract a background image from a noisy image. Based on this, in step 100, the background generator 10 can be used to extract a background image from a noisy image. Among them, the denoised image can be any noisy image in a natural scene, which does not need to be labeled or have a corresponding clean image. It is worth noting that in the model training stage, the model parameters of the background generator 10 are constantly being optimized. Therefore, in step 100, the background generator 10 can perform denoising on the noisy image used this time based on the model parameters determined in the previous iterative process to generate a background image.

[0075] Among them, the background generator 10 in this embodiment can adopt a residual network, a pyramid structure network or other neural network structures, which is not limited in this embodiment.

[0076] In step 101, the noise features contained in the noisy image can be determined based on the background image. Figure 1 The denoising model may further include a noise feature processing module 30, which may be used to determine the noise features contained in the noisy image. In an optional implementation, the noise feature may be obtained by subtracting the noisy image from the background image. Of course, other implementations may also be used to determine the noise features contained in the noisy image, and the present embodiment is not limited thereto.

[0077] refer to Figure 2a In step 102, the noise discriminator 20 in the first adversarial network can be used to discriminate the noise features. For the rain discriminator, it can be used to obtain the noise features contained in the noisy image and discriminate the noise features.

[0078] To this end, in this embodiment, true samples and false samples may be prepared for the noise discriminator 20, and the true samples and false samples may be provided to the noise discriminator 20 so that the noise discriminator 20 performs a discrimination calculation.

[0079] In an optional implementation scheme, a noise synthesis image can be constructed based on noise features as a false sample of the noise discriminator 20; the noisy image is used as a true sample of the noise discriminator 20; and the noise discriminator 20 is used to discriminate the noise synthesis image containing noise features.

[0080] refer to Figure 1 In practical applications, the noise feature processing module 30 may fuse the noise feature with a noise-free image to obtain a noise composite image. The noise-free image may be any image that does not contain noise features, and the image content in the noise-free image may be independent of the image content in the noise-containing image.

[0081] Thus, from the perspective of the first adversarial network, the noise discriminator 20 can discriminate the noise synthesis image, and the noise synthesis image contains the noise features in the noisy image, and the noise features are determined based on the background image generated by the background generator 10. Therefore, the noise discriminator 20 can indirectly discriminate the background image generated by the background generator 10. The background generator 10 and the noise discriminator 20 can confront each other and improve performance at the same time.

[0082] Of course, other optional implementation schemes can also be used in this embodiment to distinguish the noise characteristics. For example, the noise characteristics can be directly used as false samples of the noise discriminator 20, and the real noise map can be used as a true sample to distinguish the noise characteristics. However, considering that the real noise map is difficult to obtain, in this embodiment, the aforementioned noise synthesis map method is still preferred for distinguishing the noise characteristics.

[0083] In step 102, the first constraint parameter corresponding to the background generator 10 can also be calculated according to the discrimination result of the noise discriminator 20. As mentioned above, the background generator 10 and the noise discriminator 20 constitute a first adversarial network. From the perspective of the first adversarial network, the loss function and update gradient of the background generator 10 can be calculated based on the discrimination result of the noise discriminator 20 as the first constraint parameter for training the background generator 10. Accordingly, in this embodiment, the first constraint parameter can be determined based on the noise image itself through the first adversarial network including the background generator 10 and the noise discriminator 20 as the training constraint for the background generator 10.

[0084] In this embodiment, the first constraint parameter can be any constraint parameter used for regression, such as L2 loss function, L1 loss function, etc. This embodiment does not limit the function type of the first constraint function. Similarly, the constraint functions of other aspects mentioned below are also the same.

[0085] On this basis, in step 103, the background generator 10 in the denoising model may be unsupervisedly trained based on the first constraint parameter. In practical applications, the background generator 10 may be back-propagated based on the first constraint parameter to update the model parameters of the background generator 10.

[0086] From this iterative process, the noisy image in the real scene can be used as the input of the denoising model; by performing special conversion processing on the intermediate processing result (background image) in the first adversarial network included in the denoising model, the adversarial training of the first adversarial network can be realized, thereby realizing the unsupervised training of the background generator 10 in the first adversarial network. Accordingly, the denoising model based on the first adversarial network can be obtained by looping the above adversarial training process for the first adversarial network until the first adversarial network finally converges. For example, a video is used as a training sample for the denoising model, so that multiple video frames in the video can be used as a training sample set to loop the above adversarial training process for the first adversarial network until the first adversarial network finally converges.

[0087] In addition, it is worth mentioning that the training scheme for the denoising model provided in this embodiment is a scheme for self-supervised training based on denoised images in natural scenes. This scheme can be used alone to train the denoising model, and can also be combined with the traditional supervised training scheme. A large number of denoised images in natural scenes are added to the labeled sample set, thereby improving the performance of the denoising model under the traditional training scheme.

[0088] In this embodiment, after the training of the background generator 10 in the denoising model is completed, the background generator 10 can be used to extract and output the background image from the scene image to denoise the scene image. The scene image can be any image in a natural scene that needs to be denoised.

[0089] In this embodiment, a first adversarial network including a background generator and a noise discriminator can be constructed in the denoising model, wherein the background generator is a model unit in the denoising model that can perform denoising on the input image. On this basis, the background generator can be used to extract the background image from the noisy image; thereafter, the noise features in the noisy image can be determined based on the background image, and the noise discriminator can be used to discriminate the noise features. In this way, through the mutual confrontation between the background generator and the noise discriminator, the first constraint parameter corresponding to the background generator can be calculated, so that the background generator can be unsupervisedly trained based on the first constraint parameter. Accordingly, in the embodiment of the present application, the training constraints for the background generator can be determined based on the noisy image itself, so that the unsupervised training of the background generator in the denoising model can be realized, and then a large number of unlabeled noisy images in natural scenes can be involved in the training of the background generator; since the noisy images in natural scenes can be used as training samples, the generalization ability and accuracy of the background generator can be effectively improved.

[0090] In the above or following embodiments, reference Figure 1The denoising model may further include a background enhancer 40. In this embodiment, the background enhancer 40 and the background generator 10 may cooperate with each other to determine the second constraint parameter corresponding to the background generator 10.

[0091] Figure 3 A processing logic diagram of a background enhancer 40 provided by an exemplary embodiment of the present application. Figure 3 In this embodiment, the background enhancer 40 can be used to calculate the difference index between the noisy image and the background image at a specified blur level as the second constraint parameter corresponding to the background generator 10. The specified blur level can be one or more. For example, Figure 3 3 blur levels are shown in FIG. 3, 5, and 9. Of course, in this embodiment, the blur level is not limited to Figure 3 The quantities and values ​​shown in the figure can be flexibly set as required in actual applications.

[0092] In this embodiment, the background enhancer 40 can be used to perform Gaussian blur on the noisy image and the background image to obtain blurred images of the noisy image and the background image at a specified blur degree; the gradient difference between the blurred images of the noisy image and the background image at the specified blur degree is calculated as a difference index.

[0093] Since noise features are all high-frequency features, the high-frequency features in the noisy image and the background image can be blurred by Gaussian blurring, while the low-frequency features are retained. From the dimension of low-frequency features, the closer the low-frequency features in the noisy image and the background image are, the more accurate the background image generated by the background generator 10 is. Therefore, the gradient difference between the blurred images of the noisy image and the background image at a specified blur level can be used as a loss function to constrain the training of the background generator 10.

[0094] Based on this, in this embodiment, the second constraint parameter determined in this embodiment and the first constraint parameter determined based on the first adversarial network in the above embodiment can be combined to provide two aspects of training constraints for the background generator 10, so as to perform unsupervised training on the background generator 10. This can further improve the generalization ability and accuracy of the background generator 10.

[0095] In the above or following embodiments, reference Figure 1 The denoising model may also include a second adversarial network, wherein the second adversarial network includes a background generator 10 and a background discriminator 50.

[0096] In this embodiment, the background generator 10 is combined with the background discriminator 50 to form a second adversarial network, wherein the background discriminator 50 can be used to discriminate the background image. Based on this, in this embodiment, the background discriminator 50 can be used to discriminate the background image; according to the discrimination result of the background discriminator 50 on the background image, the third constraint parameter corresponding to the background generator 10 is calculated; and the third constraint function is used to perform unsupervised training on the background generator 10 in the denoising model.

[0097] Under the second adversarial network, the background image generated by the background generator 10 can be used as a false sample of the background discriminator 50, and any noise-free image in a natural scene can be used as a true sample of the background discriminator 50 to train the background discriminator 50; thereby, based on the discrimination result of the background discriminator 50 on the background image, the loss function and updated gradient of the background generator 10 can be calculated as the third constraint parameter in this embodiment.

[0098] On this basis, in this embodiment, the third constraint parameter determined in this embodiment and the first constraint parameter determined based on the first adversarial network in the above embodiment can be combined to provide two aspects of training constraints for the background generator 10. Of course, the second constraint function determined based on the background enhancer 40 in the above embodiment can also be combined to provide three aspects of training constraints for the background generator 10 to perform unsupervised training on the background generator 10. This can further improve the generalization ability and accuracy of the background generator 10.

[0099] In the above or following embodiments, reference Figure 1 , the denoising model may also include a data protection network. In this embodiment, the background generator 10 may cooperate with the noise generator 60 to form a data protection network. Among them, the noise generator 60 can be used to perform noise reconstruction on the background image to obtain a noisy reconstruction image. The noise generator 60 can generate noise features, and fuse the generated noise features with the background image generated by the background generator 10 as a noisy reconstruction image. In practical applications, the noise generator 60 can be trained in advance so that the noise generator 60 has the ability to generate noise features. For example, the noise generator 60 can cooperate with the noise discriminator 20 in the aforementioned embodiment to form an adversarial network to achieve training of the noise generator 60.

[0100] The noise generator 60 in this embodiment may adopt a residual network, a pyramid structure network or other neural network structures, which is not limited in this embodiment.

[0101] Based on this, in this embodiment, the noise generator 60 can be used to perform noise reconstruction on the background image to obtain a noisy reconstructed image; based on the difference between the noisy image and the noisy reconstructed image, the fourth constraint parameter corresponding to the background generated image is calculated; and the fourth constraint parameter is used to perform unsupervised training on the background generator 10 in the denoising model.

[0102] The applicant discovered during the research that the background generator 10 may remove the background content incorrectly during the denoising process of the noisy image, resulting in incomplete information in the generated background image, which affects the accuracy of the background image. To this end, in this embodiment, the difference between the noisy image and the noisy reconstruction image can be calculated as a loss function of the background generator 10, that is, the fourth constraint parameter in this embodiment, so as to determine the training constraint on the data integrity dimension for the background generator 10.

[0103] On this basis, in this embodiment, the fourth constraint parameter determined in this embodiment and the first constraint parameter determined based on the first adversarial network in the above embodiment can be combined to provide two aspects of training constraints for the background generator 10. Of course, the second constraint function determined based on the background enhancer 40 and the third constraint parameter determined based on the second adversarial network in the above embodiment can be combined to provide more aspects of training constraints for the background generator 10 to perform unsupervised training on the background generator 10. This can further improve the generalization ability and accuracy of the background generator 10.

[0104] In one possible design, the model training solution provided in the above embodiment can be implemented on a cloud server or cloud platform. Figure 2b A flow chart of a model training method provided by an exemplary embodiment of the present application, refer to Figure 2b , the method comprising:

[0105] Step 200: In response to a request to call a target service, determine a processing resource corresponding to the target service, and use the processing resource corresponding to the target service to perform the following steps:

[0106] Step 201, extracting a background image from a noisy image using a background generator in a first adversarial network in a denoising model;

[0107] Step 202: determining noise features contained in the noisy image according to the background image;

[0108] Step 203: using the noise discriminator in the first adversarial network to discriminate the noise feature, and calculating the first constraint parameter corresponding to the background generator according to the discrimination result;

[0109] Step 204: Based on the first constraint parameter, perform unsupervised training on the background generator in the denoising model.

[0110] In an optional embodiment, after completing the training of the background generator in the denoising model, the background generator can be called to extract and output the background image from the scene image using the processing resources corresponding to the target service to denoise the scene image.

[0111] In this embodiment, the target service can be deployed on a cloud server or cloud platform, and the cloud server or cloud platform can receive a request to call the target service to provide an image denoising function to the outside. The target service can include a denoising model, and the cloud server or cloud platform can also train the denoising model according to the above-mentioned step logic, so that the target service can support the image denoising function.

[0112] Regarding the image denoising solution and the training solution for the denoising model that can be provided by the target service, please refer to the relevant description in the aforementioned embodiments, which will not be repeated here, but this should not cause any loss to the scope of protection of this application.

[0113] Figure 4a A flowchart of another model training method provided for an exemplary embodiment of the present application. Figure 1 The denoising model provided is referenced Figure 4a , the model training method provided in this embodiment may include:

[0114] Step 400, extracting a background image from a noisy image using a background generator;

[0115] Step 401: Calculate the difference index between the noisy image and the background image at a specified blur level using a background enhancer, and use it as a constraint parameter corresponding to a background generator;

[0116] Step 402: Use the constraint parameters to perform unsupervised training on the background generator in the denoising model.

[0117] Among them, in step 301, the noisy image and the background image can be Gaussian blurred by a background enhancer to obtain blurred images of the noisy image and the background image at a specified blur degree; the gradient difference between the blurred images of the noisy image and the background image at the specified blur degree is calculated as a difference index.

[0118] The model training method provided in this embodiment can perform training constraints on the background generator from the background enhancement dimension alone. Of course, as mentioned in the previous embodiments, the background enhancement dimension provided in this embodiment can be combined with one or more of the second adversarial network dimension for background images, the first adversarial network dimension for noise features, or the data protection dimension provided in the previous embodiments to perform training constraints on the background generator.

[0119] Among them, the relevant technical details of the model training method provided in this embodiment can be referred to the description in the previous embodiment. In order to save space, they will not be repeated here, but this should not cause loss of the protection scope of this application.

[0120] In one possible design, the model training solution provided in the above embodiment can be implemented on a cloud server or cloud platform. Figure 4b A flow chart of a model training method provided by an exemplary embodiment of the present application, refer to Figure 4b , the method comprising:

[0121] Step 500: In response to a request to call a target service, determine a processing resource corresponding to the target service, and use the processing resource corresponding to the target service to perform the following steps:

[0122] Step 501, extracting a background image from a noisy image using a background generator in a denoising model;

[0123] Step 502: Calculate the difference index between the noisy image and the background image at a specified blur level using the background enhancer in the denoising model as a constraint parameter corresponding to the background generator;

[0124] Step 503: Use the constraint parameters to perform unsupervised training on the background generator in the denoising model.

[0125] In an optional embodiment, after completing the training of the background generator in the denoising model, the background generator can be called to extract and output the background image from the scene image using the processing resources corresponding to the target service to denoise the scene image.

[0126] In this embodiment, the target service can be deployed on a cloud server or cloud platform, and the cloud server or cloud platform can receive a request to call the target service to provide an image denoising function to the outside. The target service can include a denoising model, and the cloud server or cloud platform can also train the denoising model according to the above-mentioned step logic, so that the target service can support the image denoising function.

[0127] Regarding the image denoising solution and the training solution for the denoising model that can be provided by the target service, please refer to the relevant description in the aforementioned embodiments, which will not be repeated here, but this should not cause any loss to the scope of protection of this application.

[0128] Based on the above denoising model and model training method, in some other embodiments, a denoising method may be provided, which may be implemented based on the trained denoising model, and the denoising model may be integrated into a computing device. The denoising method may include:

[0129] In response to a denoising request for a target image, acquiring the target image;

[0130] A background generator included in the first adversarial network in the denoising model is used to extract a background image from the target image to denoise the target image;

[0131] Among them, the training process of the background generator includes:

[0132] A background generator is used to extract a sample background image from a noisy sample image;

[0133] Determine, according to the sample background image, the sample noise feature contained in the noisy sample image;

[0134] Using the noise discriminator in the first adversarial network to discriminate the noise characteristics of the sample, and calculating the first constraint parameter corresponding to the background generator according to the discrimination result;

[0135] Based on the first constraint parameter, the background generator in the denoising model is trained unsupervised.

[0136] The denoising method provided in this embodiment can be applied to various scenarios where image denoising is required, for example, the rain and snow removal scenario mentioned above. The target image can be an image containing noise. Of course, in practical applications, there is no need to pre-screen the image to determine whether it contains noise. The video or image in the application scenario can be directly input into the denoising model. In the denoising model, the image without noise can be output basically in its original appearance, while the image containing noise can be input as the background image, that is, the image after the noise is removed.

[0137] In this embodiment, the target image is denoised mainly by the background generator. According to the description of the training scheme of the denoising model in the previous embodiment, the training of the background generator needs to rely on the auxiliary cooperation of other model units such as the background discriminator. However, in the process of denoising the target image using the denoising model, only the output result of the background generator, that is, the background image, can be used as the denoising result. Of course, in practical applications, the denoising process of the target image can also mobilize other model units in its denoising model to jointly perform a model iteration. In this way, the model can be continuously optimized during the application of the denoising model.

[0138] Among them, regarding the training process of the background generator in the denoising model, reference can be made to the relevant description in the aforementioned embodiment. In order to save space, it will not be described in detail here, but this should not cause loss of the protection scope of this application.

[0139] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 103 can be device A; for another example, the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0140] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different networks, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0141] Above Figure 2a and Figure 4a The model training method provided in the associated embodiments may be executed by a model training device, which may be implemented as software or as a combination of software and hardware, and which may be integrated in a computing device. Figure 5 A schematic diagram of a computing device provided by another exemplary embodiment of the present application is shown in FIG. Figure 5 As shown, the computing device includes: a memory 50 and a processor 51.

[0142] The processor 51 is coupled to the memory 50 and is used to execute the computer program in the memory 50 to:

[0143] A background generator included in the first adversarial network in the denoising model is used to extract a background image from the noisy image;

[0144] Determine the noise features contained in the noisy image according to the background image;

[0145] Using the noise discriminator in the first adversarial network to discriminate the noise feature, and calculating the first constraint parameter corresponding to the background generator according to the discrimination result;

[0146] Based on the first constraint parameter, the background generator in the denoising model is trained unsupervised.

[0147] In an optional embodiment, when the processor 51 uses the noise discriminator in the first adversarial network to discriminate the noise feature, it is used to:

[0148] Based on the noise features, a noise synthesis image is constructed as a false sample for the noise discriminator;

[0149] Use noisy images as true samples for noise discriminators;

[0150] The noise discriminator is used to discriminate the noise composite image containing noise features.

[0151] In an optional embodiment, when constructing a noise synthesis graph based on the noise characteristics, the processor 51 is configured to:

[0152] The noise features are fused with a noise-free image to obtain a noise composite image.

[0153] In an optional embodiment, when determining the noise features contained in the noisy image based on the background image, the processor 51 is configured to:

[0154] The noise feature is obtained by subtracting the noisy image from the background image.

[0155] In an optional embodiment, the denoising model further includes a background enhancer, and the processor 51 is further configured to:

[0156] The background enhancer is used to calculate the difference index between the noisy image and the background image at a specified blur level as the second constraint parameter corresponding to the background generator;

[0157] The background generator in the denoising model is trained unsupervised using the second constraint function.

[0158] In an optional embodiment, when the processor 51 calculates the difference index between the noisy image and the background image at a specified blur level using the background enhancer, it is configured to:

[0159] Using a background enhancer to perform Gaussian blurring on the noisy image and the background image, so as to obtain blurred images of the noisy image and the background image at a specified blurring degree;

[0160] The gradient difference between the noisy image and the blurred image of the background image at a specified blur level is calculated as the difference index.

[0161] In an optional embodiment, the denoising model further includes a second adversarial network, the second adversarial network includes a background generator and a background discriminator, and the processor 51 is further used to:

[0162] Using a background discriminator to discriminate the background image;

[0163] Calculate the third constraint parameter corresponding to the background generator according to the discrimination result of the background discriminator on the background image;

[0164] The background generator in the denoising model is trained unsupervised using the third constraint function.

[0165] In an optional embodiment, the denoising model further includes a data protection network, the data protection network includes a background generator and a noise generator, and the processor 51 is further used for:

[0166] Using a noise generator to perform noise reconstruction on the background image to obtain a noisy reconstruction image;

[0167] Calculate the fourth constraint parameter corresponding to the background generation image according to the difference between the noisy image and the noisy reconstructed image;

[0168] The fourth constraint parameter is used to perform unsupervised training on the background generator in the denoising model.

[0169] In an optional embodiment, the background generator adopts one or more of a residual network or a pyramid structure network.

[0170] In an optional embodiment, the processor 51 is further configured to:

[0171] After completing the training of the background generator in the denoising model, the background generator is used to extract and output the background image from the scene image to denoise the scene image.

[0172] Further, if Figure 5 As shown, the computing device also includes: a communication component 52, a power supply component 53 and other components. Figure 5 Only some components are shown schematically, and it does not mean that the computing device only includes Figure 5 Components shown.

[0173] It is worth noting that the technical details in the above-mentioned embodiments of the computing device can refer to the relevant description in the aforementioned method embodiment. In order to save space, they will not be repeated here, but this should not cause a loss in the scope of protection of this application.

[0174] Figure 6 A schematic diagram of the structure of another computing device provided for another exemplary embodiment of the present application, the computing device may include a memory 60 and a processor 61.

[0175] The processor 61 is coupled to the memory 50 and is used to execute the computer program in the memory 50 to:

[0176] The background generator in the denoising model is used to extract the background image from the noisy image;

[0177] The background enhancer in the denoising model is used to calculate the difference index between the noisy image and the background image at a specified blur level, which is used as the constraint parameter corresponding to the background generator.

[0178] Unsupervised training of the background generator in the denoising model using constrained parameters.

[0179] In an optional embodiment, the processor 61, when calculating the difference between the noisy image and the background image at a specified blur level using the background enhancer, is configured to:

[0180] Using a background enhancer to perform Gaussian blurring on the noisy image and the background image, so as to obtain blurred images of the noisy image and the background image at a specified blurring degree;

[0181] The gradient difference between the noisy image and the blurred image of the background image at a specified blur level is calculated as the difference index.

[0182] and Figure 5 Different from the computing device shown in the figure, the computing device provided in this embodiment can perform training constraints on the background generator from the background enhancement dimension. Of course, as mentioned in the aforementioned embodiments, the background enhancement dimension provided in this embodiment can be combined with one or more of the second adversarial network dimension for the background image, the first adversarial network dimension for the noise feature, or the data protection dimension provided in the aforementioned embodiments to perform training constraints on the background generator.

[0183] It is worth noting that the technical details in the above-mentioned embodiments of the computing device can refer to the relevant description in the aforementioned method embodiment. In order to save space, they will not be repeated here, but this should not cause a loss in the scope of protection of this application.

[0184] Further, if Figure 6 As shown, the computing device also includes: a communication component 62, a power supply component 63 and other components. Figure 6 Only some components are shown schematically, and it does not mean that the computing device only includes Figure 6 Components shown.

[0185] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by a computing device in the above method embodiment.

[0186] Above Figure 5 and Figure 6 The memory in the computer is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0187] Above Figure 5 and 6The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0188] Above Figure 5 and Figure 6 The power supply component in the device provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0189] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0190] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0191] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0193] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0194] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0195] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0196] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0197] The above is only the embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A model training method, applicable to a denoising model, characterized in that: The denoising model includes a background generator, a noise discriminator and a noise generator, and the method includes: Extracting a background image from a noisy image using the background generator; Determining noise features contained in the noisy image according to the background image; The noise feature is fused with a noise-free image to obtain a noise synthesis image, and the noise synthesis image is used as a false sample of the noise discriminator; the noise-containing image is used as a true sample of the noise discriminator; the noise discriminator is used to discriminate the noise synthesis image containing the noise feature, and according to the discrimination result, the first constraint parameter corresponding to the background generator is calculated; Performing noise reconstruction on the background image using the noise generator to obtain a noisy reconstructed image; calculating a fourth constraint parameter corresponding to the background generator according to a difference between the noisy image and the noisy reconstructed image; Based on the first constraint parameter and the fourth constraint parameter, unsupervised training is performed on the background generator in the denoising model.

2. The method according to claim 1, characterized in that The step of determining the noise features contained in the noisy image based on the background image includes: The noise feature is obtained by subtracting the noisy image from the background image.

3. The method according to claim 1, characterized in that The denoising model further comprises a background enhancer, and the method further comprises: Utilizing the background enhancer to calculate a difference index between the noisy image and the background image at a specified blur level as a second constraint parameter corresponding to the background generator; The second constraint function is used to perform unsupervised training on the background generator in the denoising model.

4. The method according to claim 3, characterized in that The step of calculating the difference index between the noisy image and the background image at a specified blur level by using the background enhancer includes: Using the background enhancer to perform Gaussian blurring on the noisy image and the background image to obtain blurred images of the noisy image and the background image at the specified blur level; The gradient difference between the noisy image and the blurred image of the background image at the specified blur level is calculated as the difference index.

5. The method according to claim 1 or 3, characterized in that: The denoising model further includes a second adversarial network, the second adversarial network includes the background generator and the background discriminator, and the method further includes: Using the background discriminator to discriminate the background image; Calculating a third constraint parameter corresponding to the background generator according to a discrimination result of the background discriminator on the background image; The third constraint function is used to perform unsupervised training on the background generator in the denoising model.

6. The method according to claim 1, characterized in that The background generator adopts one or more of a residual network or a pyramid structure network.

7. The method according to claim 1, characterized in that Also includes: After the training of the background generator in the denoising model is completed, the background generator is used to extract and output a background image from a scene image to denoise the scene image.

8. The method according to claim 7, characterized in that The scene image and the noisy image are images with rain and snow, and the noise features are rain and snow features.

9. A model training method, characterized in that: include: In response to a request to call a target service, a processing resource corresponding to the target service is determined, and the following steps are performed using the processing resource corresponding to the target service: Extracting a background image from a noisy image using a background generator in a denoising model; the denoising model includes a background generator, a noise discriminator and a noise generator; Determining noise features contained in the noisy image according to the background image; The noise feature is fused with a noise-free image to obtain a noise synthesis image, and the noise synthesis image is used as a false sample of the noise discriminator; Using the noisy image as a true sample of the noise discriminator; Using a noise discriminator to discriminate the noise synthesis image containing the noise feature, and calculating a first constraint parameter corresponding to the background generator according to the discrimination result; The noise generator is used to perform noise reconstruction on the background image to obtain a noisy reconstructed image; according to the difference between the noisy image and the noisy reconstructed image, a fourth constraint parameter corresponding to the background generator is calculated; based on the first constraint parameter and the fourth constraint parameter, unsupervised training is performed on the background generator in the denoising model.

10. A model training method, applicable to a denoising model, characterized in that: The denoising model includes a background generator and a background enhancer, and the method includes: Extracting a background image from a noisy image using the background generator; wherein the background generator has been trained according to the model training method according to claims 1 to 8; Utilizing the background enhancer to calculate a difference index between the noisy image and the background image at a specified blur level as a constraint parameter corresponding to the background generator; The constraint parameters are used to perform unsupervised training on the background generator in the denoising model.

11. The method according to claim 10, characterized in that The step of calculating the difference between the noisy image and the background image at a specified blur level by using the background enhancer includes: Using the background enhancer to perform Gaussian blurring on the noisy image and the background image to obtain blurred images of the noisy image and the background image at the specified blur level; The gradient difference between the noisy image and the blurred image of the background image at the specified blur level is calculated as the difference index.

12. A model training method, characterized in that: include: In response to a request to call a target service, a processing resource corresponding to the target service is determined, and the following steps are performed using the processing resource corresponding to the target service: Extracting a background image from a noisy image using a background generator in a denoising model; wherein the background generator has been trained according to the model training method according to claims 1 to 8; The background enhancer in the denoising model is used to calculate the difference index between the noisy image and the background image at a specified blur level as a constraint parameter corresponding to the background generator; The constraint parameters are used to perform unsupervised training on the background generator in the denoising model.

13. A computing device, characterized in that: including memory and processor; The memory is used to store one or more computer instructions; The processor is coupled to the memory and configured to execute the one or more computer instructions for: Extracting a background image from a noisy image using a background generator included in a first adversarial network in a denoising model; the denoising model includes a background generator, a noise discriminator and a noise generator; Determining noise features contained in the noisy image according to the background image; The noise feature is fused with a noise-free image to obtain a noise synthesis image, and the noise synthesis image is used as a false sample of the noise discriminator; Using the noisy image as a true sample of the noise discriminator; Using the noise discriminator in the first adversarial network to discriminate the noise synthesis image containing the noise feature, and calculating the first constraint parameter corresponding to the background generator according to the discrimination result; Performing noise reconstruction on the background image using the noise generator to obtain a noisy reconstructed image; calculating a fourth constraint parameter corresponding to the background generator according to a difference between the noisy image and the noisy reconstructed image; Based on the first constraint parameter and the fourth constraint parameter, unsupervised training is performed on the background generator in the denoising model.

14. A computing device, characterized in that including memory and processor; The memory is used to store one or more computer instructions; The processor is coupled to the memory and configured to execute the one or more computer instructions for: Extracting a background image from a noisy image using a background generator in a denoising model; wherein the background generator has been trained according to the model training method according to claims 1 to 8; The background enhancer in the denoising model is used to calculate the difference index between the noisy image and the background image at a specified blur level as a constraint parameter corresponding to the background generator; The constraint parameters are used to perform unsupervised training on the background generator in the denoising model.

15. A denoising model, characterized in that: It comprises a first adversarial network, the first adversarial network comprises a background generator and a noise discriminator, and the denoising model further comprises: a noise generator; The background generator is used to extract a background image from a noisy image; The noise discriminator is used to obtain the noise features contained in the noisy image, fuse the noise features with a noise-free image to obtain a noise synthesis image, and use the noise-containing image as a false sample of the noise discriminator; use the noisy image as a true sample of the noise discriminator; discriminate the noise synthesis image containing the noise features, and discriminate the noise features to generate a first constraint parameter corresponding to the background generator; wherein the noise features are determined according to the background image; The noise generator is used to: perform noise reconstruction on the background image to obtain a noisy reconstruction image; and calculate a fourth constraint parameter corresponding to the background generator according to the difference between the noisy image and the noisy reconstruction image; The background generator is further used to update its own model parameters based on the first constraint parameter and the fourth constraint parameter.

16. A denoising method, characterized in that: include: In response to a denoising request for a target image, acquiring the target image; Extracting a background image from the target image using a background generator included in the first adversarial network in the denoising model to denoise the target image; The denoising model includes a background generator, a noise discriminator and a noise generator; The training process of the background generator includes: Extracting a sample background image from a noisy sample image using the background generator; Determining, according to the sample background image, a sample noise feature contained in the noisy sample image; The sample noise feature is fused with a noise-free image to obtain a noise synthesis image, and the image is used as a false sample of the noise discriminator; the noise sample image is used as a true sample of the noise discriminator; the noise discriminator in the first adversarial network is used to discriminate the noise synthesis image containing the noise feature, and according to the discrimination result, the first constraint parameter corresponding to the background generator is calculated; Performing noise reconstruction on the background image using the noise generator to obtain a noisy reconstructed image; calculating a fourth constraint parameter corresponding to the background generator according to a difference between the noisy image and the noisy reconstructed image; Based on the first constraint parameter and the fourth constraint parameter, unsupervised training is performed on the background generator in the denoising model.

17. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the model training method described in any one of claims 1-10 or the denoising method described in claim 16.

Images