Model training method, image noise reduction method and device

Abstract

The invention provides a model training method and device and an image noise reduction method and device, and the method comprises the steps: training a first cyclic generative adversarial network model through employing a low-noise sample simulation image and a high-resolution sample real image, and obtaining a target cyclic generative adversarial network model; the target reconstruction image corresponding to the target real image is generated by using the target cyclic generative adversarial network model, and due to the style migration characteristic of the cyclic generative adversarial network model, the target reconstruction image can have the dual advantages of high resolution and low noise, the noise removal effect of the image can be improved to a certain extent, and the image quality is improved. When the target real image is the low-dose STEM image, the obtained target reconstructed image can meet the quality requirement of the low-dose STEM image in actual scientific research.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a model training method, an image denoising method and a device. Background Art

[0002] In the field of materials science, Scanning Transmission Electron Microscopy (STEM) has become an important tool for studying the structure and properties of materials due to its high spatial resolution and sensitivity. In particular, when studying electron-sensitive materials (such as zeolite molecular sieves, metal-organic frameworks (MOFs) and perovskites, STEM can provide atomic-level resolution. In order to protect the integrity of the material and maintain the physical and chemical properties of the material itself, low-electron dose imaging methods can usually be used. However, while protecting the material, low-electron dose imaging also introduces more noise into the image. The noise problem directly limits the application of low-dose STEM images in microstructure analysis and molecular behavior observation.

[0003] In the related art, principal component analysis PCA, independent component analysis ICA and conventional filtering algorithms can be used to remove image noise from low-dose STEM images and improve image quality.

[0004] However, the image noise removal effect of the above method cannot meet the quality requirements of low-dose STEM images in actual scientific research, and the image noise removal effect is still not high. Summary of the invention

[0005] In view of the above problems, the embodiments of the present application provide a model training method, an image denoising method, an apparatus, an electronic device and a readable storage medium to overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect of an embodiment of the present application, a model training method is provided, the method comprising: Generate a sample simulated image corresponding to the sample real image; wherein the resolution of the sample real image is smaller than the resolution of the sample simulated image; and the sample simulated image does not contain image noise; Input the sample real image into the first loop generative adversarial network model to obtain a first reconstructed image output by the first loop generative adversarial network model, and input the sample simulated image into the first loop generative adversarial network to obtain a second reconstructed image output by the first loop generative adversarial network model; Determine a model loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image; Based on the model loss value, the first model parameter of the first cyclic generative adversarial network model is adjusted to obtain a target cyclic generative adversarial network model; the target cyclic generative adversarial network is used to generate a target reconstructed image based on a target real image.

[0007] Optionally, determining a model loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image includes: Determining a cycle consistency loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image; Determining a focal loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image; Determining a perceptual total loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image; Based on the cycle consistency loss value, the focal loss value and the perceptual total loss value, a model loss value of the first cycle generative adversarial network model is determined.

[0008] Optionally, determining the focal loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image includes: Determining a first confidence level of the first reconstructed image relative to the sample real image; determining a second confidence level of the second reconstructed image relative to the sample simulated image; Based on the first confidence and the second confidence, a focal loss value of the first cyclic generative adversarial network model is determined.

[0009] Optionally, determining the perceptual total loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image includes: Inputting the sample simulated image and the first reconstructed image into a pre-trained convolutional neural network model respectively, and obtaining a first image feature corresponding to the sample simulated image and a second image feature corresponding to the first reconstructed image output by each network layer from the convolutional neural network model; Inputting the sample real image and the second reconstructed image into the convolutional neural network model respectively, and obtaining the third image feature corresponding to the sample real image and the fourth image feature corresponding to the second reconstructed image output by each network layer from the convolutional neural network model; Determine a first perceptual loss value of the first recurrent generative adversarial network model based on the first image feature and the second image feature, and determine a second perceptual loss value of the first recurrent generative adversarial network model based on the third image feature and the fourth image feature; The first perceptual loss value and the second perceptual loss value are added to obtain a perceptual total loss value of the first cyclic generative adversarial network model.

[0010] Optionally, inputting the sample real image into a first recurrent generative adversarial network model to obtain a first reconstructed image output by the first recurrent generative adversarial network model, and inputting the sample simulated image into the first recurrent generative adversarial network to obtain a second reconstructed image output by the first recurrent generative adversarial network model include: Input the sample real image into the first generator in the first loop generative adversarial network model to obtain a third reconstructed image output by the first generator, and input the sample simulated image into the second generator in the first loop generative adversarial network model to obtain a fourth reconstructed image output by the second generator; The third reconstructed image is input into the second generator in the first loop generative adversarial network model to obtain the first reconstructed image output by the second generator, and the fourth reconstructed image is input into the first generator in the first loop generative adversarial network model to obtain the second reconstructed image output by the first generator.

[0011] Optionally, adjusting a first model parameter of the first cyclic generative adversarial network model based on the model loss value to obtain a target cyclic generative adversarial network model includes: Based on the focal loss value in the model loss value, the model weight parameter in the first model parameter of the first loop generative adversarial network model is adjusted; based on the perceived total loss value and the cycle consistency loss value in the model loss value, the other model parameters in the first model parameter except the model weight parameter are adjusted to obtain the target loop generative adversarial network model.

[0012] In a second aspect, an embodiment of the present application provides an image noise reduction method, the method comprising: Get the real target image; Generate a target simulated image corresponding to the target real image; wherein the target simulated image does not contain image noise; The target real image and the target simulated image are input into a target recurrent generative adversarial network model to obtain a target reconstructed image output by the target recurrent generative adversarial network model; wherein the target recurrent generative adversarial network model is obtained based on any of the model training methods described above.

[0013] Optionally, inputting the target real image and the target simulated image into a target recurrent generative adversarial network model to obtain a target reconstructed image output by the target recurrent generative adversarial network model comprises: Inputting the target real image and the target simulated image into a target recurrent generative adversarial network model to obtain a first reconstructed image and a second reconstructed image output by the target recurrent generative adversarial network model; The first reconstructed image and the second reconstructed image are input into a preset filter to obtain a target reconstructed image output by the preset filter.

[0014] In a third aspect, an embodiment of the present application provides a model training device, the device comprising: A generating module, used for generating a sample simulated image corresponding to a sample real image; wherein the resolution of the sample real image is smaller than the resolution of the sample simulated image; and the sample simulated image does not contain image noise; An input-output module, configured to input the sample real image into the first recurrent generative adversarial network model to obtain a first reconstructed image output by the first recurrent generative adversarial network model, and input the sample simulated image into the first recurrent generative adversarial network to obtain a second reconstructed image output by the first recurrent generative adversarial network model; A determination module, configured to determine a model loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image; An adjustment module is used to adjust the first model parameter of the first loop generative adversarial network model based on the model loss value to obtain a target loop generative adversarial network model; the target loop generative adversarial network is used to generate a target reconstructed image based on a target real image.

[0015] Optionally, the determining module includes: A first determination submodule, configured to determine a cycle consistency loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image; A second determination submodule, configured to determine a focal loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image; A third determination submodule is used to determine a perceptual total loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image and the second reconstructed image; The fourth determination submodule is used to determine the model loss value of the first cycle generative adversarial network model based on the cycle consistency loss value, the focus loss value and the perceived total loss value.

[0016] Optionally, the second determining submodule includes: A first determining unit, configured to determine a first confidence level of the first reconstructed image relative to the sample real image; a second determining unit, configured to determine a second confidence level of the second reconstructed image relative to the sample simulation image; The third determining unit is used to determine the focal loss value of the first loop generative adversarial network model based on the first confidence level and the second confidence level.

[0017] Optionally, the third determining submodule includes: A first input acquisition unit, used to input the sample simulated image and the first reconstructed image into a pre-trained convolutional neural network model, and obtain, from the convolutional neural network model, first image features corresponding to the sample simulated image and second image features corresponding to the first reconstructed image output by each network layer; A second input acquisition unit, used to input the sample real image and the second reconstructed image into the convolutional neural network model respectively, and obtain a third image feature corresponding to the sample real image and a fourth image feature corresponding to the second reconstructed image output by each network layer from the convolutional neural network model; a fourth determining unit, configured to determine a first perceptual loss value of the first cyclic generative adversarial network model based on the first image feature and the second image feature, and to determine a second perceptual loss value of the first cyclic generative adversarial network model based on the third image feature and the fourth image feature; A computing unit is configured to add the first perceptual loss value and the second perceptual loss value to obtain a total perceptual loss value of the first cyclic generative adversarial network model.

[0018] Optionally, the input-output module includes: A first input-output submodule is used to input the sample real image into the first generator in the first recurrent generative adversarial network model to obtain a third reconstructed image output by the first generator, and input the sample simulated image into the second generator in the first recurrent generative adversarial network model to obtain a fourth reconstructed image output by the second generator; The second input-output submodule is used to input the third reconstructed image into the second generator in the first recurrent generative adversarial network model to obtain the first reconstructed image output by the second generator, and to input the fourth reconstructed image into the first generator in the first recurrent generative adversarial network model to obtain the second reconstructed image output by the first generator.

[0019] Optionally, the adjustment module includes: An adjustment submodule is used to adjust the model weight parameters in the first model parameters of the first loop generative adversarial network model based on the focal loss value in the model loss value, and adjust other model parameters in the first model parameters except the model weight parameters based on the perceived total loss value and the cycle consistency loss value in the model loss value to obtain a target loop generative adversarial network model.

[0020] In a fourth aspect, an embodiment of the present application provides an image noise reduction device, the device comprising: An acquisition module is used to acquire a real target image; A generating module, used for generating a target simulated image corresponding to the target real image; wherein the target simulated image does not contain image noise; An input-output module is used to input the target real image and the target simulated image into a target recurrent generative adversarial network model to obtain a target reconstructed image output by the target recurrent generative adversarial network model; wherein the target recurrent generative adversarial network model is obtained based on any of the model training methods described above.

[0021] Optionally, the input-output module includes: A first input-output submodule is used to input the target real image and the target simulated image into a target recurrent generative adversarial network model to obtain a first reconstructed image and a second reconstructed image output by the target recurrent generative adversarial network model; The second input-output submodule is used to input the first reconstructed image and the second reconstructed image into a preset filter to obtain a target reconstructed image output by the preset filter.

[0022] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory, characterized in that the processor executes the computer program to implement the model training method as described in any one of the above, or the image denoising method.

[0023] In a sixth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the model segmentation method or image denoising method as described in any one of the above is implemented.

[0024] The specific beneficial effects are: The embodiment of the present application generates a sample simulated image corresponding to a sample real image; wherein the resolution of the sample real image is smaller than that of the sample simulated image; the sample simulated image does not contain image noise, inputs the sample real image into a first loop generative adversarial network model, obtains a first reconstructed image output by the first loop generative adversarial network model, inputs the sample simulated image into the first loop generative adversarial network, obtains a second reconstructed image output by the first loop generative adversarial network model, determines a model loss value of the first loop generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image, and adjusts a first model parameter of the first loop generative adversarial network model based on the model loss value to obtain a target loop generative adversarial network. Model; The target cycle generative adversarial network is used to generate a target reconstructed image based on a target real image. The first cycle generative adversarial network model can be trained using low-noise sample simulation images and high-resolution sample real images to obtain a target cycle generative adversarial network model, and then the target cycle generative adversarial network model can be used to generate a target reconstructed image corresponding to the target real image. Due to the style transfer characteristics of the cycle generative adversarial network model, the target reconstructed image can have the dual advantages of high resolution and low noise, which can improve the noise removal effect of the image to a certain extent. When the target real image is a low-dose STEM image, the obtained target reconstructed image can meet the quality requirements of low-dose STEM images in actual scientific research. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0026] Figure 1 It is a flowchart of a model training method provided in an embodiment of the present application; Figure 2 It is a flowchart of another model training method provided in an embodiment of the present application; Figure 3 It is a flowchart of an image noise reduction method provided in an embodiment of the present application; Figure 4 is a schematic diagram of a low-dose STEM image provided in an embodiment of the present application; Figure 5 It is a logic block diagram of a model training device provided in an embodiment of the present application; Figure 6It is a logic block diagram of an image noise reduction device provided in an embodiment of the present application; Figure 7 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The exemplary embodiments of the present application will be described in more detail below in conjunction with the accompanying drawings in the embodiments of the present application. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to enable the scope of the present application to be fully communicated to those skilled in the art.

[0028] Reference Figure 1 , Figure 1 A flow chart of a model training method provided in an embodiment of the present application, the method comprising: Step 101, generating a sample simulated image corresponding to a sample real image; wherein the resolution of the sample real image is smaller than the resolution of the sample simulated image; and the sample simulated image does not contain image noise.

[0029] In an embodiment of the present application, the sample real image may be an image of any objective scene that exists in reality. In the present application, a microscopic image obtained by a scanning transmission electron microscope under experimental conditions is taken as an example, but the sample real image is not limited thereto. The above-mentioned microscopic image may be a low electron dose image corresponding to an electron sensitive material (such as a zeolite molecular sieve, a metal organic framework (MOFs) and a perovskite, etc.). After obtaining the sample real image, the chemical structure file (such as a CIF file) of the material in the sample real image may be used to generate a sample simulation image that matches the experimental conditions of the sample real image using advanced electron microscope simulation software (such as PrismaticSTEM Simulator). Generally, since the scanning projection electron microscope has a higher resolution, the resolution of the sample real image may be less than the resolution of the sample simulation image. At the same time, image noise will inevitably be introduced during the shooting process, and in the simulation software, the obtained sample simulation image is an image under an ideal state, so the sample simulation image may not contain image noise.

[0030] Step 102: input the sample real image into the first loop generative adversarial network model to obtain a first reconstructed image output by the first loop generative adversarial network model, and input the sample simulated image into the first loop generative adversarial network to obtain a second reconstructed image output by the first loop generative adversarial network model.

[0031] In an embodiment of the present application, a cycle generative adversarial network model (CycleGAN) is a deep learning model for unsupervised image conversion. It can convert between two different fields without paired training data. In the cycle generative adversarial network model, two generators and two discriminators may be included, wherein the generator is used to process the input image, and the discriminator is used to judge the difference between the output image and the input image. The sample real image can be input into the first cycle generative adversarial network model to obtain the first reconstructed image output by the first cycle generative adversarial network model, and the sample simulated image can also be input into the first cycle generative adversarial network model to obtain the second reconstructed image output by the first cycle generative adversarial network model. Since the cycle generative adversarial network model has two generators, the cycle generative adversarial network model can have two input ports, and the sample real image and the sample simulated image can be input simultaneously from different input ports when inputting the first cycle generative adversarial network model. In this way, the image style of the sample real image and the image style of the sample simulated image can be included in the first reconstructed image and the second reconstructed image.

[0032] Step 103: determine a model loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image and the second reconstructed image.

[0033] In an embodiment of the present application, the model loss value of the first cyclic generative adversarial network model can be calculated based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image. The model loss value may include the generation loss of the generator, the discrimination loss of the discriminator, and the cycle consistency loss. At the same time, other types of loss values ​​can also be used to assist model training, such as similarity loss value.

[0034] Step 104: Based on the model loss value, adjust the first model parameter of the first loop generative adversarial network model to obtain a target loop generative adversarial network model; the target loop generative adversarial network is used to generate a target reconstructed image based on the target real image.

[0035] In an embodiment of the present application, the first model parameter of the first loop generative adversarial network model can be adjusted according to the model loss value, so that the target loop generative adversarial network model can be obtained. When adjusting the first model parameter, multiple adjustments can be made. After each parameter adjustment is completed, the process of steps 101 to 104 can be repeated to adjust the first model parameter again. The adjustment direction of the first model parameter can be the direction in which the model loss value (or absolute value) is reduced so that the model converges. When the model loss value meets the convergence condition, or the number of training times of the first loop generative adversarial network model reaches a preset number of times, the training can be stopped, and the first loop generative adversarial network model obtained after the last model parameter adjustment is determined as the target loop generative adversarial network model. The target loop generative adversarial network model can be used to generate a target reconstructed image based on a target real image.

[0036] In an embodiment of the present application, a sample simulated image corresponding to a sample real image is generated; wherein the resolution of the sample real image is smaller than that of the sample simulated image; and the sample simulated image does not contain image noise, the sample real image is input into a first loop generative adversarial network model to obtain a first reconstructed image output by the first loop generative adversarial network model, the sample simulated image is input into the first loop generative adversarial network to obtain a second reconstructed image output by the first loop generative adversarial network model, based on the sample real image, the sample simulated image, the first reconstructed image and the second reconstructed image, the model loss value of the first loop generative adversarial network model is determined, based on the model loss value, the first model parameter of the first loop generative adversarial network model is adjusted, and the target loop generative adversarial network model is obtained. Network model; the target cyclic generative adversarial network is used to generate a target reconstructed image based on a target real image. The first cyclic generative adversarial network model can be trained using low-noise sample simulation images and high-resolution sample real images to obtain a target cyclic generative adversarial network model, and then the target cyclic generative adversarial network model can be used to generate a target reconstructed image corresponding to the target real image. Due to the style transfer characteristics of the cyclic generative adversarial network model, the target reconstructed image can have the dual advantages of high resolution and low noise, which can improve the noise removal effect of the image to a certain extent. When the target real image is a low-dose STEM image, the obtained target reconstructed image can meet the quality requirements of low-dose STEM images in actual scientific research.

[0037] Reference Figure 2 , Figure 2 A flowchart of another model training method provided in an embodiment of the present application, the method may include: Step 201: Generate a sample simulated image corresponding to the sample real image.

[0038] In the embodiments of the present application, the implementation content of this step can refer to the embodiment content of step 101, which will not be repeated here.

[0039] Step 202: input the sample real image into the first generator in the first recurrent generative adversarial network model to obtain the third reconstructed image output by the first generator, and input the sample simulated image into the second generator in the first recurrent generative adversarial network model to obtain the fourth reconstructed image output by the second generator.

[0040] In an embodiment of the present application, the sample real image and the sample simulated image can be input into the first loop generative adversarial network model from different input ports, and different input ports correspond to different generators. The sample real image can be input into the first generator in the first loop generative adversarial network model, so as to obtain the third reconstructed image output by the first generator, and the sample simulated image can also be input into the second generator of the first loop generative adversarial network model, so as to obtain the fourth reconstructed image output by the second generator. The third reconstructed image and the fourth reconstructed image are images obtained by intermediate stage processing, and they only have one of the image styles of their corresponding input images, and do not contain the image styles of two input images at the same time.

[0041] Step 203: input the third reconstructed image into the second generator in the first recurrent generative adversarial network model to obtain the first reconstructed image output by the second generator, and input the fourth reconstructed image into the first generator in the first recurrent generative adversarial network model to obtain the second reconstructed image output by the first generator.

[0042] In an embodiment of the present application, after the second generator in the first cyclic generative adversarial network model outputs the fourth reconstructed image, the third reconstructed image can be input into the second generator, thereby obtaining the first reconstructed image output by the second generator, and after the first generator in the first cyclic generative adversarial network model outputs the third reconstructed image, the fourth reconstructed image can be input into the first generator, thereby obtaining the second reconstructed image output by the first generator. In this way, the first reconstructed image and the second image can both contain the image styles of the sample real image and the sample simulated image.

[0043] For example, if the sample real image has a higher resolution and contains image noise, and the sample simulated image has a lower resolution and does not contain image noise, then between the first reconstructed image and the second reconstructed image, there is a reconstructed image with a lower resolution and contains image noise, and there is also a reconstructed image with a higher resolution and does not contain image noise.

[0044] In an embodiment of the present application, by inputting the sample real image into the first generator in the first loop generative adversarial network model to obtain the third reconstructed image output by the first generator, and inputting the sample simulated image into the second generator in the first loop generative adversarial network model to obtain the fourth reconstructed image output by the second generator, the third reconstructed image is input into the second generator in the first loop generative adversarial network model to obtain the first reconstructed image output by the second generator, and the fourth reconstructed image is input into the first generator in the first loop generative adversarial network model to obtain the second reconstructed image output by the first generator, the first reconstructed image and the second reconstructed image corresponding to the sample real image and the sample virtual image can be obtained by jointly obtaining the first generator and the second generator in the first loop generative adversarial network model, thereby improving the accuracy of the first reconstructed image and the second reconstructed image to a certain extent.

[0045] Step 204: determine a cycle consistency loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image.

[0046] In an embodiment of the present application, the cycle consistency loss value of the first cyclic generative adversarial network model can be calculated according to the sample real image, the sample simulated image, the first reconstructed image and the second reconstructed image, and the calculation method thereof is shown in the following formula 1: (Formula 1) In the above formula 1, represents the cycle consistency loss value, Represents the image dataset where the sample real image x is located, represents the sample real image, represents the third reconstructed image output by the first generator A, represents the first reconstructed image output by the second generator B, Represents a sample simulated image The image dataset where represents the fourth reconstructed image output by the second generator B, Represents the second reconstructed image output by the first generator A, symbol " ” represents the L1 norm, which is used to measure the difference between the reconstructed image and the original image (the sample real image or the sample simulated image). Indicates sum.

[0047] Step 205: determine the focal loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image and the second reconstructed image.

[0048] In an embodiment of the present application, in a microscopic image, there will be sparse key areas (such as single molecule sites), and the image features of this part of the area are not obvious. Therefore, focal loss can be introduced in model training. Focal Loss is an improved cross entropy loss function, which is mainly used to solve the class imbalance problem in target detection. It improves the performance of the model by reducing the loss contribution of simple samples and enhancing the focus on difficult-to-classify samples. The focal loss value of the first cycle generative adversarial network model can be calculated based on the sample real image, the sample simulated image, the first reconstructed image and the second reconstructed image, and the calculation method is shown in Formula 2 below: (Formula 2) In the above formula 2, represents the focal loss value, i represents the number of the sample real image and the sample simulated image, is a balancing factor used to adjust the weights of samples of different categories. is an adjustment factor used to reduce the loss weight of easy-to-classify samples. is the predicted probability of the model for the sample real image, which can be expressed by the joint similarity of the first reconstructed image, the second reconstructed image, the sample real image and the sample simulated image, that is, ,in, represents the modulus of the first reconstructed image, represents the modulus of the second reconstructed image, represents the model of the real image of the sample, Represents the modulus of the sample simulated image.

[0049] Optionally, step 205 may include the following sub-steps: Sub-step 2051: determining a first confidence level of the first reconstructed image relative to the sample real image.

[0050] In an embodiment of the present application, in the first recurrent generative adversarial network model, there may be two discriminators for discriminating the authenticity of the first reconstructed image and the second reconstructed image, and generating confidences of the first reconstructed image and the second reconstructed image. The first reconstructed image and the sample real image may be input into the first discriminator, so that the first confidence of the first reconstructed image relative to the sample real image output by the first discriminator may be obtained.

[0051] Sub-step 2052, determining a second confidence level of the second reconstructed image relative to the sample simulation image.

[0052] In an embodiment of the present application, the second reconstructed image and the sample virtual image may be input into a second discriminator to obtain a second confidence level of the second reconstructed image relative to the sample simulated image output by the second discriminator.

[0053] Sub-step 2053, based on the first confidence and the second confidence, determining the focal loss value of the first loop generative adversarial network model.

[0054] In an embodiment of the present application, the focal loss value of the first cyclic generative adversarial network model can be calculated according to the first confidence level and the second confidence level according to the method shown in Formula 2 above. In Formula 2, It can be the first confidence level or the second confidence level.

[0055] In an embodiment of the present application, by determining a first confidence of the first reconstructed image relative to the sample real image, determining a second confidence of the second reconstructed image relative to the sample simulated image, and based on the first confidence and the second confidence, determining the focus loss value of the first cyclic generative adversarial network model, the focus loss value can be calculated based on the first confidence and the second confidence, which can improve the accuracy and usability of the focus loss value to a certain extent.

[0056] Step 206: Determine a total perceptual loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image.

[0057] In an embodiment of the present application, ‌Perceptual Loss‌ is a loss function based on deep learning, which is mainly used in image generation and style transfer tasks, aiming to improve the visual realism and quality of the generated images. Perceptual loss measures the similarity between the image to be generated and the target image by comparing the difference between the feature representations in the pre-trained convolutional neural network (CNN). The pre-trained convolutional neural network model can be used to combine the sample real image, the sample simulated image, the first reconstructed image and the second reconstructed image to calculate the total perceptual loss value of the first cycle generative adversarial network model.

[0058] Optionally, step 206 may include the following sub-steps: Sub-step 2061, input the sample simulated image and the first reconstructed image into a pre-trained convolutional neural network model respectively, and obtain from the convolutional neural network model the first image features corresponding to the sample simulated image and the second image features corresponding to the first reconstructed image output by each network layer.

[0059] In an embodiment of the present application, the pre-trained convolutional neural network may have multiple network layers, and the sample simulated image and the first reconstructed image may be respectively input into the pre-trained convolutional neural network model. After completing the above input, the first image features corresponding to the sample simulated image output by each network layer and the second image features corresponding to the first reconstructed image may be obtained from the convolutional neural network model.

[0060] Sub-step 2062, input the sample real image and the second reconstructed image into the convolutional neural network model respectively, and obtain the third image features corresponding to the sample real image and the fourth image features corresponding to the second reconstructed image output by each network layer from the convolutional neural network model.

[0061] In an embodiment of the present application, the sample real image and the second reconstructed image can be respectively input into a pre-trained convolutional neural network model. After completing the above input, the third image feature corresponding to the sample real image output by each network layer and the fourth image feature corresponding to the second reconstructed image can be obtained from the convolutional neural network model.

[0062] Sub-step 2063: Determine a first perceptual loss value of the first loop generative adversarial network model based on the first image feature and the second image feature, and determine a second perceptual loss value of the first loop generative adversarial network model based on the third image feature and the fourth image feature.

[0063] In an embodiment of the present application, a first perceptual loss value of a first cyclic generative adversarial network model may be calculated based on the first image feature and the second image feature, and the calculation method thereof is shown in Formula 3 below: (Formula 3) In the above formula 3, represents the first perceptual loss value, i represents the network layer number of the convolutional neural network model, represents the second image feature extracted by network layer i, Represents the first image feature extracted by network layer i, symbol " ” represents the L2 norm, which is used to measure the difference in feature space.

[0064] In an embodiment of the present application, the second perceptual loss value of the first cyclic generative adversarial network model can be calculated according to the third image feature and the fourth image feature, and the calculation method thereof is shown in the following formula 4: (Formula 4) In the above formula 4, represents the second perceptual loss value, i represents the network layer number of the convolutional neural network model, represents the second image feature extracted by network layer i, Represents the first image feature extracted by network layer i, symbol " ” represents the L2 norm, which is used to measure the difference in feature space.

[0065] Sub-step 2064, adding the first perceptual loss value and the second perceptual loss value to obtain the total perceptual loss value of the first cyclic generative adversarial network model.

[0066] In an embodiment of the present application, the first perceptual loss value and the second perceptual loss value may be added to obtain a perceptual total loss value of the first cyclic generative adversarial network model, as shown in Formula 5 below: (Formula 5) In the above formula 5, represents the perceived total loss value, represents the first perceptual loss value, Represents the second perceptual loss value.

[0067] In an embodiment of the present application, the sample simulated image and the first reconstructed image are respectively input into a pre-trained convolutional neural network model, and the first image feature corresponding to the sample simulated image output by each network layer and the second image feature corresponding to the first reconstructed image are obtained from the convolutional neural network model, the sample real image and the second reconstructed image are respectively input into the convolutional neural network model, and the third image feature corresponding to the sample real image output by each network layer and the fourth image feature corresponding to the second reconstructed image are obtained from the convolutional neural network model, based on the first image feature and the second image feature, the first perceptual loss value of the first loop generative adversarial network model is determined, based on the third image feature and the fourth image feature, the second perceptual loss value of the first loop generative adversarial network model is determined, the first perceptual loss value and the second perceptual loss value are added to obtain the perceptual total loss value of the first loop generative adversarial network model, the image features of the sample real image, the sample simulated image, the first reconstructed image and the second reconstructed image can be extracted through the convolutional neural network, and the perceptual total loss value of the model can be calculated according to the image features, which improves the accuracy of the perceptual total loss value to a certain extent, and can improve the image generation quality of the target loop generative adversarial network model finally obtained.

[0068] Step 207: Determine a model loss value of the first cycle generative adversarial network model based on the cycle consistency loss value, the focus loss value, and the perceived total loss value.

[0069] In an embodiment of the present application, the model loss value of the first cycle generative adversarial network model can be calculated based on the cycle consistency loss value, the focus loss value and the total perceptual loss value. One calculation method is shown in Formula 6 below: (Formula 6) In the above formula 6, is the model loss value, is the perceptual loss value, is the focus loss value, is the cycle consistency loss value, , Both are balance coefficients.

[0070] In an embodiment of the present application, the cycle consistency loss value of the first cyclic generative adversarial network model is determined based on the sample real image, the sample simulated image, the first reconstructed image and the second reconstructed image; the focus loss value of the first cyclic generative adversarial network model is determined based on the sample real image, the sample simulated image, the first reconstructed image and the second reconstructed image; the perceptual total loss value of the first cyclic generative adversarial network model is determined based on the sample real image, the sample simulated image, the first reconstructed image and the second reconstructed image; the model loss value of the first cyclic generative adversarial network model is determined based on the cycle consistency loss value, the focus loss value and the perceptual total loss value; the cycle consistency loss value, the focus loss value and the perceptual total loss value can be calculated, and the model loss value is constituted by the cycle consistency loss value, the focus loss value and the perceptual total loss value, which improves the accuracy and reliability of the model loss value to a certain extent, and can improve the image generation quality of the target cyclic generative adversarial network model finally obtained.

[0071] Step 208: Based on the focal loss value in the model loss value, adjust the model weight parameter in the first model parameter of the first loop generative adversarial network model; based on the perceived total loss value and the loop consistency loss value in the model loss value, adjust other model parameters in the first model parameter except the model weight parameter to obtain the target loop generative adversarial network model.

[0072] In the embodiment of the present application, since the focus loss value can be used to solve the class imbalance problem in target detection, and this type of problem is mainly solved by adjusting the model weight parameters in the model, therefore, when adjusting the first model parameter of the first loop generative adversarial network model by the model loss value, the model weight parameters in the first model parameter can be adjusted by the focus loss value in the model loss value, so that the image output by the model can have more image features of sparse key areas. The perceived total loss value and the cycle consistency loss value in the model loss value do not contribute to the model weight parameters. Therefore, the other model parameters in the first model parameter except the model weight parameters can be adjusted by the perceived total loss value and the cycle consistency loss value in the model loss value. After completing the above adjustment, the implementation process of step 201 to step 208 can be repeated to adjust the first model parameters multiple times. When the model loss value meets the preset convergence condition, or the number of training times of the first loop generative adversarial network model reaches the preset number, the adjustment of the first model parameter can be stopped, and the first loop generative adversarial network model obtained after the last adjustment is determined as the target loop generative adversarial network model.

[0073] In an embodiment of the present application, by adjusting the model weight parameters in the first model parameters of the first loop generative adversarial network model based on the focal loss value in the model loss value, and adjusting other model parameters in the first model parameters except the model weight parameters based on the perceived total loss value and the loop consistency loss value in the model loss value, a target loop generative adversarial network model is obtained. Targeted parameter adjustments can be made to the first loop generative adversarial network model according to the characteristics of each loss value contained in the model loss value, which improves the efficiency of model training to a certain extent and can improve the image generation quality of the target loop generative adversarial network model.

[0074] Reference Figure 3 , Figure 3 A schematic diagram of a process of an image noise reduction method provided in an embodiment of the present application, the method may include: Step 301, obtaining a target real image.

[0075] In the embodiments of the present application, the target real image can be an image of any objective scene that exists in reality. In the present application, a microscopic image taken by a scanning transmission electron microscope under experimental conditions is taken as an example, but the sample real image is not limited to this, and other types of images can still use the method described in the present application. The above-mentioned microscopic image can be a low electron dose image corresponding to an electron sensitive material (such as zeolite molecular sieves, metal organic frameworks (MOFs) and perovskites, etc.). The material to be studied can be photographed using a scanning transmission electron microscope, so that the target real image can be obtained.

[0076] Step 302: Generate a target simulated image corresponding to the target real image; wherein the target simulated image does not contain image noise.

[0077] In the embodiment of the present application, after obtaining the target real image, the chemical structure file (such as CIF file) of the material in the target real image can be used to generate a target simulated image that matches the experimental conditions of the sample real image using advanced electron microscope simulation software (such as Prismatic STEM Simulator). Since the target virtual image is an image under an ideal state, the target simulated image may not contain image noise.

[0078] Step 303, input the target real image and the target simulated image into a target recurrent generative adversarial network model to obtain a target reconstructed image output by the target recurrent generative adversarial network model; wherein the target recurrent generative adversarial network model is obtained based on any of the model training methods described above.

[0079] In an embodiment of the present application, a target real image and a target simulated image can be input into a target recurrent generative adversarial network model, so as to obtain a target reconstructed image output by the target recurrent generative adversarial network model. The target recurrent generative adversarial network model can be obtained based on any of the above model training methods.

[0080] Optionally, step 303 may include the following sub-steps: Sub-step 3031, inputting the target real image and the target simulated image into a target recurrent generative adversarial network model to obtain a first reconstructed image and a second reconstructed image output by the target recurrent generative adversarial network model.

[0081] In an embodiment of the present application, the target real image and the target virtual image can be input into the target recurrent generative adversarial network model, so as to obtain the first reconstructed image and the second reconstructed image output by the target recurrent generative adversarial network model. Among the first reconstructed image and the second reconstructed image, there is a reconstructed image with a lower resolution and containing image noise, and there is also a reconstructed image with a higher resolution and no image noise.

[0082] For example, Figure 4 As shown, Figure 4 A schematic diagram of a low-dose STEM image provided in the present application. In the figure, 10 is a real image of the target obtained by actual shooting, 20 is a simulated image of the target generated by the simulation software, and 30 and 40 are the first reconstructed image and the second reconstructed image output by the target cyclic generation adversarial network model. The image corresponding to 40 has a higher image resolution and lower image noise.

[0083] Sub-step 3032: input the first reconstructed image and the second reconstructed image into a preset filter to obtain a target reconstructed image output by the preset filter.

[0084] In an embodiment of the present application, a preset filter can be designed according to the image features of the first reconstructed image and the second reconstructed image to screen the desired target reconstructed image from the first reconstructed image and the second reconstructed image. If the target real image is a low-dose STEM image, there is a reconstructed image containing high noise and low resolution in the first reconstructed image and the second reconstructed image, and the grayscale value of the image is close to the middle range, so the preset filter can be a band-stop filter. After the first reconstructed image and the second reconstructed image are input into the preset filter, the preset filter can output a reconstructed image with high resolution and low noise, so that the output reconstructed image can be used as the target reconstructed image.

[0085] Continuing with the above example, after the reconstructed image corresponding to 30 and the reconstructed image corresponding to 40 are input into the preset filter, the preset filter can output the reconstructed image corresponding to 40, that is, the reconstructed image corresponding to 40 is the target reconstructed image.

[0086] In an embodiment of the present application, a target real image and a target simulated image are input into a target recurrent generative adversarial network model to obtain a first reconstructed image and a second reconstructed image output by the target recurrent generative adversarial network model, and the first reconstructed image and the second reconstructed image are input into a preset filter to obtain a target reconstructed image output by the preset filter. The two reconstructed images can be filtered through the preset filter to obtain the target reconstructed image, which can avoid manual screening, save time cost, and improve the generation efficiency of the target reconstructed image to a certain extent.

[0087] In an embodiment of the present application, a target real image is acquired to generate a target simulated image corresponding to the target real image; wherein the target simulated image does not contain image noise, and the target real image and the target simulated image are input into a target recurrent generative adversarial network model to obtain a target reconstructed image output by the target recurrent generative adversarial network model; wherein the target recurrent generative adversarial network model is obtained based on any of the model training methods described above, and the trained target recurrent generative adversarial network model can be used in combination with the target real image to generate a target reconstructed image, and after style transfer using the target recurrent generative adversarial network model, the target reconstructed image can have the resolution characteristics of the target real image and does not contain image noise, therefore, the noise removal effect of the image is improved to a certain extent, and when the target real image is a low-dose STEM image, the obtained target reconstructed image can meet the quality requirements of low-dose STEM images in actual scientific research.

[0088] Reference Figure 5 , Figure 5 A logical block diagram of a model training device provided in an embodiment of the present application, the device 500 may include: A generating module 501 is used to generate a sample simulated image corresponding to a sample real image; wherein the resolution of the sample real image is smaller than the resolution of the sample simulated image; and the sample simulated image does not contain image noise; An input-output module 502 is used to input the sample real image into the first recurrent generative adversarial network model to obtain a first reconstructed image output by the first recurrent generative adversarial network model, and input the sample simulated image into the first recurrent generative adversarial network to obtain a second reconstructed image output by the first recurrent generative adversarial network model; A determination module 503 is used to determine a model loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image and the second reconstructed image; The adjustment module 504 is used to adjust the first model parameter of the first loop generative adversarial network model based on the model loss value to obtain a target loop generative adversarial network model; the target loop generative adversarial network is used to generate a target reconstructed image based on a target real image.

[0089] Optionally, the determining module 503 includes: A first determination submodule, configured to determine a cycle consistency loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image; A second determination submodule, configured to determine a focal loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image; A third determination submodule is used to determine a perceptual total loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image and the second reconstructed image; The fourth determination submodule is used to determine the model loss value of the first cycle generative adversarial network model based on the cycle consistency loss value, the focus loss value and the perceived total loss value.

[0090] Optionally, the second determining submodule includes: A first determining unit, configured to determine a first confidence level of the first reconstructed image relative to the sample real image; a second determining unit, configured to determine a second confidence level of the second reconstructed image relative to the sample simulation image; The third determining unit is used to determine the focal loss value of the first loop generative adversarial network model based on the first confidence level and the second confidence level.

[0091] Optionally, the third determining submodule includes: A first input acquisition unit, used to input the sample simulated image and the first reconstructed image into a pre-trained convolutional neural network model, and obtain, from the convolutional neural network model, first image features corresponding to the sample simulated image and second image features corresponding to the first reconstructed image output by each network layer; A second input acquisition unit, used to input the sample real image and the second reconstructed image into the convolutional neural network model respectively, and obtain a third image feature corresponding to the sample real image and a fourth image feature corresponding to the second reconstructed image output by each network layer from the convolutional neural network model; a fourth determining unit, configured to determine a first perceptual loss value of the first cyclic generative adversarial network model based on the first image feature and the second image feature, and to determine a second perceptual loss value of the first cyclic generative adversarial network model based on the third image feature and the fourth image feature; A computing unit is configured to add the first perceptual loss value and the second perceptual loss value to obtain a total perceptual loss value of the first cyclic generative adversarial network model.

[0092] Optionally, the input-output module 502 includes: A first input-output submodule is used to input the sample real image into the first generator in the first recurrent generative adversarial network model to obtain a third reconstructed image output by the first generator, and input the sample simulated image into the second generator in the first recurrent generative adversarial network model to obtain a fourth reconstructed image output by the second generator; The second input-output submodule is used to input the third reconstructed image into the second generator in the first recurrent generative adversarial network model to obtain the first reconstructed image output by the second generator, and to input the fourth reconstructed image into the first generator in the first recurrent generative adversarial network model to obtain the second reconstructed image output by the first generator.

[0093] Optionally, the adjustment module 504 includes: An adjustment submodule is used to adjust the model weight parameters in the first model parameters of the first loop generative adversarial network model based on the focal loss value in the model loss value, and adjust other model parameters in the first model parameters except the model weight parameters based on the perceived total loss value and the cycle consistency loss value in the model loss value to obtain a target loop generative adversarial network model.

[0094] The model training device in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. Exemplarily, the electronic device can be a GPUBOX, a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0095] The model training device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, a Linux operating system, a Windows operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0096] The model training device provided in the embodiment of the present application can achieve Figure 1 to Figure 2 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0097] Reference Figure 6 , Figure 6 A logic block diagram of an image noise reduction device provided in an embodiment of the present application, the device 600 may include: An acquisition module 601 is used to acquire a real target image; A generating module 602 is used to generate a target simulated image corresponding to the target real image; wherein the target simulated image does not contain image noise; The input-output module 603 is used to input the target real image and the target simulated image into the target recurrent generative adversarial network model to obtain the target reconstructed image output by the target recurrent generative adversarial network model; wherein the target recurrent generative adversarial network model is obtained based on any of the model training methods described above.

[0098] Optionally, the input-output module 603 includes: A first input-output submodule is used to input the target real image and the target simulated image into a target recurrent generative adversarial network model to obtain a first reconstructed image and a second reconstructed image output by the target recurrent generative adversarial network model; The second input-output submodule is used to input the first reconstructed image and the second reconstructed image into a preset filter to obtain a target reconstructed image output by the preset filter.

[0099] The image noise reduction device in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. Exemplarily, the electronic device can be a GPUBOX, a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0100] The image noise reduction device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, a Linux operating system, a Windows operating system, etc., or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0101] The image noise reduction device provided in the embodiment of the present application can achieve Figure 3 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0102] The present application embodiment provides an electronic device, see Figure 7 The electronic device 70 includes: a processor 701, a memory 702, and a computer program 7021 stored in the memory 702 and executable on the processor 701. When the processor 701 executes the program, the model training method of the aforementioned embodiment or the image denoising method is implemented.

[0103] An embodiment of the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the model training method disclosed in the embodiment of the present application, or the steps in the image denoising method, are implemented.

[0104] The embodiment of the present application also provides a computer program product. When the computer program product is run on an electronic device, the processor implements the model training method disclosed in the embodiment of the present application, or the steps in the image denoising method.

[0105] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0106] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0107] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0108] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0109] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0110] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0111] The above is a detailed introduction to a resource decoupling system, execution method and device for deep learning applications provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A model training method, characterized in that: The method comprises: Generate a sample simulated image corresponding to the sample real image; wherein the resolution of the sample real image is smaller than the resolution of the sample simulated image; and the sample simulated image does not contain image noise; Input the sample real image into the first loop generative adversarial network model to obtain a first reconstructed image output by the first loop generative adversarial network model, and input the sample simulated image into the first loop generative adversarial network to obtain a second reconstructed image output by the first loop generative adversarial network model; Determine a model loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image; Based on the model loss value, the first model parameter of the first cyclic generative adversarial network model is adjusted to obtain a target cyclic generative adversarial network model; the target cyclic generative adversarial network is used to generate a target reconstructed image based on a target real image.

2. The method according to claim 1, characterized in that The determining, based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image, a model loss value of the first cyclic generative adversarial network model comprises: Determining a cycle consistency loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image; Determining a focal loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image; Determining a total perceptual loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image; Based on the cycle consistency loss value, the focal loss value and the perceptual total loss value, a model loss value of the first cycle generative adversarial network model is determined.

3. The method according to claim 2, characterized in that The determining, based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image, of a focal loss value of the first cyclic generative adversarial network model comprises: Determining a first confidence level of the first reconstructed image relative to the sample real image; determining a second confidence level of the second reconstructed image relative to the sample simulated image; Based on the first confidence and the second confidence, a focal loss value of the first cyclic generative adversarial network model is determined.

4. The method according to claim 2, characterized in that: The determining, based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image, of the perceived total loss value of the first cyclic generative adversarial network model comprises: Inputting the sample simulated image and the first reconstructed image into a pre-trained convolutional neural network model respectively, and obtaining a first image feature corresponding to the sample simulated image and a second image feature corresponding to the first reconstructed image output by each network layer from the convolutional neural network model; Inputting the sample real image and the second reconstructed image into the convolutional neural network model respectively, and obtaining the third image feature corresponding to the sample real image and the fourth image feature corresponding to the second reconstructed image output by each network layer from the convolutional neural network model; Determine a first perceptual loss value of the first recurrent generative adversarial network model based on the first image feature and the second image feature, and determine a second perceptual loss value of the first recurrent generative adversarial network model based on the third image feature and the fourth image feature; The first perceptual loss value and the second perceptual loss value are added to obtain a perceptual total loss value of the first cyclic generative adversarial network model.

5. The method according to claim 1, characterized in that The step of inputting the sample real image into a first cyclic generative adversarial network model to obtain a first reconstructed image output by the first cyclic generative adversarial network model, and inputting the sample simulated image into the first cyclic generative adversarial network to obtain a second reconstructed image output by the first cyclic generative adversarial network model comprises: Input the sample real image into the first generator in the first loop generative adversarial network model to obtain a third reconstructed image output by the first generator, and input the sample simulated image into the second generator in the first loop generative adversarial network model to obtain a fourth reconstructed image output by the second generator; The third reconstructed image is input into the second generator in the first loop generative adversarial network model to obtain the first reconstructed image output by the second generator, and the fourth reconstructed image is input into the first generator in the first loop generative adversarial network model to obtain the second reconstructed image output by the first generator.

6. The method according to claim 1 or 2, characterized in that: The adjusting, based on the model loss value, a first model parameter of the first cyclic generative adversarial network model to obtain a target cyclic generative adversarial network model includes: Based on the focal loss value in the model loss value, the model weight parameter in the first model parameter of the first loop generative adversarial network model is adjusted; based on the perceived total loss value and the cycle consistency loss value in the model loss value, the other model parameters in the first model parameter except the model weight parameter are adjusted to obtain the target loop generative adversarial network model.

7. An image denoising method, characterized in that: The method comprises: Get the real target image; Generate a target simulated image corresponding to the target real image; wherein the target simulated image does not contain image noise; The target real image and the target simulated image are input into a target recurrent generative adversarial network model to obtain a target reconstructed image output by the target recurrent generative adversarial network model; wherein the target recurrent generative adversarial network model is obtained based on the model training method described in any one of claims 1 to 6.

8. The method according to claim 7, characterized in that The step of inputting the target real image and the target simulated image into a target recurrent generative adversarial network model to obtain a target reconstructed image output by the target recurrent generative adversarial network model comprises: Inputting the target real image and the target simulated image into a target recurrent generative adversarial network model to obtain a first reconstructed image and a second reconstructed image output by the target recurrent generative adversarial network model; The first reconstructed image and the second reconstructed image are input into a preset filter to obtain a target reconstructed image output by the preset filter.

9. A model training device, characterized in that: The device comprises: A generating module, used for generating a sample simulated image corresponding to a sample real image; wherein the resolution of the sample real image is smaller than the resolution of the sample simulated image; and the sample simulated image does not contain image noise; An input-output module, configured to input the sample real image into the first recurrent generative adversarial network model to obtain a first reconstructed image output by the first recurrent generative adversarial network model, and input the sample simulated image into the first recurrent generative adversarial network to obtain a second reconstructed image output by the first recurrent generative adversarial network model; A determination module, configured to determine a model loss value of the first cyclic generative adversarial network model based on the sample real image, the sample simulated image, the first reconstructed image, and the second reconstructed image; An adjustment module is used to adjust the first model parameter of the first loop generative adversarial network model based on the model loss value to obtain a target loop generative adversarial network model; the target loop generative adversarial network is used to generate a target reconstructed image based on a target real image.

10. An image noise reduction device, characterized in that: The device comprises: An acquisition module is used to acquire a real target image; A generating module, used for generating a target simulated image corresponding to the target real image; wherein the target simulated image does not contain image noise; An input-output module is used to input the target real image and the target simulated image into a target recurrent generative adversarial network model to obtain a target reconstructed image output by the target recurrent generative adversarial network model; wherein the target recurrent generative adversarial network model is obtained based on the model training method described in any one of claims 1 to 6.

Images