Image processing method, device, equipment and storage medium

Abstract

The present application discloses an image processing method, device, equipment and storage medium, which relates to the field of image processing technology. The image processing method includes: inputting the image to be denoised into the decoupler in the target generation network to obtain the target feature information of the image to be denoised; performing noise elimination according to the target feature information, the controller in the target generation network and the reconstructor in the target generation network to obtain the target processed image, and the joint structure corresponding to the decoupler, the controller and the reconstructor is composed of multiple downsampling layers, multiple upsampling layers, a head layer, a tail layer, a multi-layer perceptron and a codec layer. By extracting the feature information of the image to be denoised, and then performing noise elimination according to the feature information of the image to be denoised, the denoised image is obtained, which achieves the purpose of removing the compressed noise while retaining the details and edge information in the video.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to image processing methods, devices, equipment and storage media. Background Art

[0002] Deep learning, especially convolutional neural network (CNN), has been widely used in the field of video denoising. This type of method is in a leading position in the current image noise suppression ability and has played an important role in broadening the scope of denoising problems. However, a key problem faced by current algorithms is poor generalization ability. The current model is mainly trained for specific types of noise. When encountering new types of noise that have not been seen during training, the effect may be significantly reduced; in practical applications, factors such as hardware conditions and network bandwidth may limit the actual effect of video denoising technology; how to retain the details and edge information in the video while removing compressed noise is also a problem that needs to be solved. Summary of the invention

[0003] The main purpose of the present application is to provide an image processing method, apparatus, device and storage medium, aiming to solve the technical problem of not being able to retain details and edge information in the video while removing compression noise.

[0004] To achieve the above object, the present application proposes an image processing method, which includes: Inputting the image to be denoised into the decoupler in the target generation network to obtain target feature information of the image to be denoised; Noise elimination is performed according to the target feature information, the controller in the target generation network and the reconstructor in the target generation network to obtain a target processed image. The joint structure corresponding to the decoupler, the controller and the reconstructor consists of multiple downsampling layers, multiple upsampling layers, a head layer, a tail layer, a multi-layer perceptron and a codec layer.

[0005] In one embodiment, the step of performing noise elimination according to the target feature information, the controller in the target generation network, and the reconstructor in the target generation network to obtain a target processed image includes: Inputting the target feature information into a controller in the target generation network to obtain a noise control signal; Noise elimination is performed according to the target feature information, the noise control signal and the reconstructor in the target generation network to obtain a target processed image.

[0006] In one embodiment, before the step of inputting the image to be denoised into the decoupler in the target generation network to obtain the target feature information of the image to be denoised, the step further includes: Input the initial noise image to the initial generation network for image generation to obtain a generated noise image; Inputting the generated noise image and the sample noise image into the target discrimination network for image discrimination to obtain a target discrimination result; The initial generation network is trained according to the target discrimination result and the generation loss function to obtain a target generation network.

[0007] In one embodiment, before the step of training the initial generation network according to the target discrimination result and the generation loss function to obtain the target generation network, the step further includes: Calculate the loss function according to the target discrimination result to obtain a relative average adversarial loss function; Calculate the loss function between pixels according to the generated noise image and the sample noise image to obtain a pixel difference loss function; Calculate the loss function between structures according to the generated noise image and the sample noise image to obtain a structural similarity loss function; A generation loss function is obtained according to the relative average adversarial loss function, the pixel difference loss function and the structural similarity loss function.

[0008] In one embodiment, the step of inputting the generated noise image and the sample noise image into a target discrimination network for image discrimination to obtain a target discrimination result includes: Inputting the generated noise image into the target discrimination network for image discrimination to obtain the probability of generated image; Input the sample noise image to the target discrimination network for image discrimination and obtain the sample image probability; The target discrimination result is obtained according to the generated image probability and the sample image probability.

[0009] In one embodiment, the step of training the initial generation network according to the target discrimination result and the generation loss function to obtain the target generation network includes: Adjusting parameters of the initial generation network according to the target discrimination result to obtain an adjusted generation network; The adjustment generation network is trained according to the training sample set and the generation loss function to obtain the target generation network.

[0010] In one embodiment, after the step of performing noise elimination according to the target feature information, the controller in the target generation network and the reconstructor in the target generation network to obtain a target processed image, the step further includes: Performing quality assessment on the target processed image according to a preset image assessment method to obtain a quality assessment result; Determine corresponding optimization hyperparameters according to the quality assessment result; The hyperparameters of the target generation network are adjusted according to the optimized hyperparameters to obtain an optimized target generation network.

[0011] In addition, to achieve the above-mentioned purpose, the present application also proposes an image processing device, the image processing device comprising: An input module, used for inputting the image to be denoised into the decoupler in the target generation network to obtain target feature information of the image to be denoised; A processing module is used to perform noise elimination according to the target feature information, the controller in the target generation network and the reconstructor in the target generation network to obtain a target processed image. The joint structure corresponding to the decoupler, the controller and the reconstructor consists of multiple downsampling layers, multiple upsampling layers, a head layer, a tail layer, a multi-layer perceptron and a codec layer.

[0012] In addition, to achieve the above objectives, the present application also proposes an image processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image processing method described above.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image processing method described above are implemented.

[0014] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the image processing method described above are implemented.

[0015] The present application obtains the target feature information of the image to be denoised by inputting the image to be denoised into the decoupler in the target generation network; performs noise elimination according to the target feature information, the controller in the target generation network and the reconstructor in the target generation network to obtain the target processed image, and the joint structure corresponding to the decoupler, the controller and the reconstructor is composed of multiple downsampling layers, multiple upsampling layers, a head layer, a tail layer, a multi-layer perceptron and a codec layer. By extracting the feature information of the image to be denoised and then performing noise elimination according to the feature information of the image to be denoised, the denoised image is obtained, which achieves the purpose of removing compression noise while retaining the details and edge information in the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the image processing method of the present application; Figure 2 A schematic diagram of a generator network structure provided in Example 1 of the image processing method of the present application; Figure 3 A schematic diagram of the joint network structure of a decoupler, a reconstructor and a controller provided in the first embodiment of the image processing method of the present application; Figure 4 A schematic diagram of the structure of the downsampling layer provided in the first embodiment of the image processing method of the present application; Figure 5 A schematic diagram of the structure of an upsampling layer provided in Embodiment 1 of the image processing method of the present application; Figure 6 A schematic diagram of a flow chart provided for the second embodiment of the image processing method of the present application; Figure 7 A schematic diagram of the discriminator network structure provided for the second embodiment of the image processing method of the present application; Figure 8 This is a schematic diagram of the module structure of the image processing device according to an embodiment of the present application; Fig. 9 Schematic diagram of the device structure of the hardware operating environment involved in the image processing method in the embodiment of the present application.

[0019] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0020] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0021] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0022] The main solution of the embodiment of the present application is: input the image to be denoised into the decoupler in the target generation network to obtain the target feature information of the image to be denoised; perform noise elimination according to the target feature information, the controller in the target generation network and the reconstructor in the target generation network to obtain the target processed image, and the joint structure corresponding to the decoupler, the controller and the reconstructor consists of multiple downsampling layers, multiple upsampling layers, a head layer, a tail layer, a multi-layer perceptron and a codec layer.

[0023] Deep learning, especially convolutional neural network (CNN), has been widely used in the field of video denoising. This type of method is in a leading position in the current image noise suppression ability and has played an important role in broadening the scope of denoising problems. However, a key problem faced by current algorithms is poor generalization ability. The current model is mainly trained for specific types of noise. When encountering new types of noise that have not been seen during training, the effect may be significantly reduced; in practical applications, factors such as hardware conditions and network bandwidth may limit the actual effect of video denoising technology; how to retain the details and edge information in the video while removing compressed noise is also a problem that needs to be solved.

[0024] The present application obtains the target feature information of the image to be denoised by inputting the image to be denoised into the decoupler in the target generation network; performs noise elimination according to the target feature information, the controller in the target generation network and the reconstructor in the target generation network to obtain the target processed image, and the joint structure corresponding to the decoupler, the controller and the reconstructor is composed of multiple downsampling layers, multiple upsampling layers, a head layer, a tail layer, a multi-layer perceptron and a codec layer. By extracting the feature information of the image to be denoised and then performing noise elimination according to the feature information of the image to be denoised, the denoised image is obtained, which achieves the purpose of removing compression noise while retaining the details and edge information in the video.

[0025] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an image processing device capable of realizing the above functions, etc. The following takes the image processing device as the execution subject as an example to illustrate this embodiment and the following embodiments.

[0026] Based on this, the present application embodiment provides an image processing method, referring to Figure 1 , Figure 1 This is a flowchart diagram of the first embodiment of the image processing method of the present application.

[0027] In this embodiment, the image processing method includes steps S10 to S20: Step S10, inputting the image to be denoised into the decoupler in the target generation network to obtain target feature information of the image to be denoised; It should be noted that the present embodiment adopts Generative Adversarial Networks (GAN), that is, the generator and discriminator networks are used simultaneously to compare the generated results during training. This method has the following two advantages. First, the trained network can retain more effective details and edge feature information; second, when predicting the results, only the generator reasoning is needed, which greatly increases the efficiency of the algorithm operation.

[0028] It can be understood that the image to be denoised refers to the image to be denoised, including images and lightweight videos. The target generation network refers to the generator network after adversarial training. The generator network consists of three parts: decoupler, controller and reconstructor. The decoupler is used to extract the high- and low-dimensional feature information of the input image. The controller controls the reconstructor to eliminate noise by fitting the image noise distribution by understanding the high-dimensional and deep information. The target feature information refers to the high- and low-dimensional feature information of the input image.

[0029] In a specific implementation, an image to be denoised is obtained, and then the image to be denoised is input into a decoupler in a generator network after adversarial training, and then the high- and low-dimensional feature information, i.e., target feature information, in the image to be denoised is extracted through the decoupler.

[0030] Step S20, noise elimination is performed according to the target feature information, the controller in the target generation network and the reconstructor in the target generation network to obtain a target processed image, and the joint structure corresponding to the decoupler, the controller and the reconstructor is composed of multiple downsampling layers, multiple upsampling layers, a head layer, a tail layer, a multi-layer perceptron and a codec layer.

[0031] It can be understood that the target processed image refers to the image after noise processing.

[0032] In a specific implementation, the high- and low-dimensional feature information of the input image is input into the controller in the generator network to obtain a fitted noise distribution, and the reconstructor in the generator network is informed to perform noise elimination based on the fitted noise distribution to obtain a noise-processed image, i.e., a target processed image.

[0033] In a feasible implementation, step S20 may include steps A21-A22: Step A21, inputting the target feature information into a controller in the target generation network to obtain a noise control signal; It should be noted that the noise control signal refers to a signal used by the controller to control the reconstructor to eliminate noise.

[0034] In a specific implementation, the high- and low-dimensional feature information of the input image is input into the controller in the generator network to obtain a fitted noise distribution, and then a signal for controlling the reconstructor to eliminate noise, namely a noise control signal, is generated according to the fitted noise distribution.

[0035] Step A22, performing noise elimination according to the target feature information, the noise control signal and the reconstructor in the target generation network to obtain a target processed image.

[0036] In a specific implementation, the controller controls the reconstructor to remove noise by combining the signal with the high- and low-dimensional feature information of the input image, and controls the reconstructor in the generator network to remove noise from the image to be processed, and finally obtains the image after noise removal, that is, the target processed image.

[0037] In a feasible implementation manner, step S20 may include steps B21 to B23: Step B21, performing quality assessment on the target processed image according to a preset image assessment method to obtain a quality assessment result; It is understandable that the preset image evaluation methods include visual inspection, PSNR (peak signal-to-noise ratio) and SSIM (structural similarity index) and other image evaluation methods, and the quality evaluation result refers to the quality evaluation result of the image after noise elimination.

[0038] In the specific implementation, the quality of the image after noise removal is evaluated according to image evaluation methods such as visual inspection, PSNR and SSIM. The visual inspection, PSNR value and SSIM value are combined to comprehensively evaluate the quality of the generated image. Different weights are assigned based on specific needs for summary calculation to obtain the quality evaluation result of the image after noise removal.

[0039] Step B22, determining corresponding optimization hyperparameters according to the quality assessment result; It can be understood that the optimized hyperparameters refer to the optimized hyperparameters, including the learning rate, batch size and other hyperparameters. The learning rate, batch size and other hyperparameters of the generator network are adjusted according to the quality evaluation results of the image after noise elimination, and finally the optimized hyperparameters are obtained.

[0040] Step B23, adjusting the hyperparameters of the target generation network according to the optimized hyperparameters to obtain an optimized target generation network.

[0041] In the specific implementation, the hyperparameters of the target generator network are optimized according to the obtained optimized hyperparameters to obtain the optimized generator network, and the training effect is optimized. Finally, the generator network is optimized repeatedly until the image quality generated by the generator and the discrimination ability of the discriminator reach the expected level.

[0042] It should be noted that the generator network consists of three parts: decoupler, controller and reconstructor. The decoupler is used to extract high- and low-dimensional feature information of the input image, and the controller controls the reconstructor to eliminate noise by fitting the image noise distribution by understanding the high-dimensional and deep information. The complete algorithm structure is shown in Figure 2 As shown, the joint network structure of the decoupler, reconstructor and controller is as follows Figure 3 As shown in the figure, the structure consists of three downsampling layers, three upsampling layers, a Head layer, a Tail layer, an MLP (multi-layer perceptron) and a codec layer. The Head and Tail layers are composed of only one convolution filter, which respectively obtains low-dimensional and shallow information of the image and fused and reconstructed feature information. The codec layer is used to extract deeper image information, and the output of the encoding layer flows into the MLP layer to fit the image noise distribution, thereby controlling the direction of reconstructing image features in each upsampling process, thereby better eliminating the input image noise.

[0043] It can be understood that the structure of the downsampling layer is shown in Figure 4, which consists of two basic layers (BasicBlock), a maximum pooling layer (Maxpool) and a convolutional layer (Conv). BasicBlock extracts high-dimensional features of the input, Maxpool reduces the image size, and Conv increases the input dimension to retain more feature information. The structure of the upsampling layer is shown in Figure 4. Figure 5 As shown in the figure, it consists of a transposed convolution layer (ConvT), a convolution layer (Conv) and an attention layer (QFAttention). ConvT increases the image size, Conv reduces the input dimension to lightweight the network, and QFAttention combines with a multi-layer perceptron (MLP) to recover and supplement the feature information forgotten during the upsampling process.

[0044] In this embodiment, the target feature information of the image to be denoised is obtained by inputting the image to be denoised into the decoupler in the target generation network; noise is eliminated according to the target feature information, the controller in the target generation network and the reconstructor in the target generation network to obtain a target processed image, and the joint structure corresponding to the decoupler, the controller and the reconstructor is composed of multiple downsampling layers, multiple upsampling layers, a head layer, a tail layer, a multi-layer perceptron and a codec layer. By extracting the feature information of the image to be denoised and then eliminating the noise according to the feature information of the image to be denoised, the denoised image is obtained, which achieves the goal of removing the compression noise while retaining the details and edge information in the video.

[0045] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 6 Before step S10, the image processing method further includes steps S01 to S03: Step S01, inputting an initial noise image into an initial generation network to generate an image, thereby obtaining a generated noise image; It can be understood that the initial noise image refers to the noisy sample image used to train the generator network, the initial generation network refers to the generator network that has not undergone adversarial training, and the generated noise image refers to the noise image generated by the initial generation network, wherein the noise image generated by the initial generation network is a false noise image.

[0046] In a specific implementation, the noisy sample image used to train the generator network is input into the initial generator network for image generation, and then the output of the initial generator network is obtained to determine the noisy image generated by the generator network.

[0047] Step S02, inputting the generated noise image and the sample noise image into the target discrimination network for image discrimination to obtain a target discrimination result; It can be understood that the target discriminant network refers to the discriminator network in the adversarial generation network, which is used for adversarial training with the generator network, the sample noise image refers to the noise sample image used for the discriminator network, and the sample noise image is a real noise image, and the target discrimination result refers to the result of discriminating whether the input image is true or false.

[0048] In a specific implementation, the noise image generated by the generator network and the noise sample image used in the discriminator network are both input into the discriminator network. The discriminator network determines whether the noise image generated by the generator network and the noise sample image used in the discriminator network are true or false, and finally determines the result of judging whether the input image is true or false, that is, the target judgment result.

[0049] In a feasible implementation, step S02 may include steps A021 to A023: Step A021, inputting the generated noise image into the target discrimination network for image discrimination to obtain the generated image probability; It can be understood that the probability of generating an image refers to the probability that the noise image generated by the generator network is judged as real.

[0050] In a specific implementation, the noise image generated by the generator network is input into the discriminator network for image discrimination, and then the probability that the noise image (false noise image) generated by the generator network is judged by the discriminator network as true is obtained, that is, the probability of generating the image.

[0051] Step A022, input the sample noise image to the target discrimination network for image discrimination, and obtain the sample image probability; It can be understood that the sample image probability refers to the probability of judging the noise sample image used in the discriminator network as false.

[0052] In a specific implementation, the noise sample image used for the discriminator network is input into the discriminator network for image discrimination, and then the probability that the discriminator network discriminates that the noise sample image used for the discriminator network (real noise image) is false is obtained, that is, the probability of generating an image.

[0053] Step A023, obtaining a target discrimination result according to the generated image probability and the sample image probability.

[0054] In a specific implementation, the probability of a noise image generated by the generator network being judged as true and the probability of a noise sample image used for the discriminator network being judged as false are summarized and calculated, and then the target discrimination result is determined based on the calculation results.

[0055] Step S03, training the initial generation network according to the target discrimination result and the generation loss function to obtain a target generation network.

[0056] It can be understood that the generation loss function refers to the total loss function of the generator network.

[0057] In the specific implementation, the initial generator network is adversarially trained based on the result of the discriminator network distinguishing whether the input image is true or false combined with the total loss function of the generator network, and the final generator network, i.e., the target generator network, is obtained based on the training results.

[0058] In a feasible implementation manner, step S03 may include steps A031 to A034: Step A031, calculating a loss function according to the target discrimination result to obtain a relative average adversarial loss function; It can be understood that the relative average adversarial loss function refers to the generator loss function based on the adversarial loss function.

[0059] In specific implementation, the relative average adversarial loss function aims to solve the problem of gradient vanishing in traditional GAN ​​and improve training stability. It modifies the output of the discriminator to more accurately reflect the difference between the generated samples and the real samples, and then calculates the relative average adversarial loss function of the generator network through the target discriminant results.

[0060] Step A032, calculating the loss function between pixels according to the generated noise image and the sample noise image to obtain a pixel difference loss function; It can be understood that the pixel difference loss function refers to a regularized loss function that is conducive to network convergence.

[0061] In the specific implementation, in addition to the adversarial loss for the generator loss function, this embodiment also supplements the regularization loss that is beneficial to network convergence, and then determines the regularization loss function that is beneficial to network convergence by comparing the pixel difference between the noise image generated by the generator network and the noise sample image used for discriminator network training, that is, the pixel difference loss function.

[0062] Step A033, calculating a loss function between structures according to the generated noise image and the sample noise image to obtain a structural similarity loss function; It can be understood that the inter-structure loss function refers to the Multi-Scale Structural Similarity Index (MSSIM) loss function of reconstructing the color saturation of the image.

[0063] In the specific implementation, in addition to the adversarial loss function for the generator, this embodiment also supplements the MSSIM loss function for the color saturation of the reconstructed image. The multi-scale structural similarity index loss function of the color saturation of the reconstructed image is determined by comparing the structural differences between the noise image generated by the generator network and the noise sample image used for training the discriminator network, that is, the loss function between structures.

[0064] Step A034, obtaining a generation loss function according to the relative average adversarial loss function, the pixel difference loss function and the structural similarity loss function.

[0065] In the specific implementation, weighted calculation is performed based on the calculated relative average adversarial loss function, pixel difference loss function and structural similarity loss function, and the total loss function of the generator network is obtained according to the settlement result. Through the above settings, the generator must not only try to deceive the discriminator, but also ensure that the generated image is highly consistent with the real image in terms of pixel level and structure.

[0066] In a feasible implementation, step S03 may include steps B031-B032: Step B031, adjusting parameters of the initial generation network according to the target discrimination result to obtain an adjusted generation network; It can be understood that adjusting the generative network refers to the generator network after parameter adjustment, and based on the result of the discriminator network discriminating whether the image is true or false, the parameters of the generator network are adjusted to optimize the generator network, thereby obtaining the generator network after parameter adjustment.

[0067] Step B032: training the adjusted generation network according to the training sample set and the generation loss function to obtain the target generation network.

[0068] In a specific implementation, the training sample set refers to a set of noisy sample images used to train the generator network, and then the generator network with adjusted parameters is adversarially trained by combining the set of noisy sample images used to train the generator network with the total loss function of the generator network to finally obtain the target generator network.

[0069] It should be noted that, for the discriminator network, this embodiment adopts a relativistic discriminator, and the specific structure is as follows: Figure 7 As shown in the figure, the main body of the network adopts the commonly used U-net structure. Each layer consists of a convolution filter, a normalization layer, and a LeakReLU activation function. The normalization layer uses spectral normalization to increase the stability of GAN training.

[0070] It should be noted that the main feature of the GAN network lies in its unique training method. The spatial distribution of the input data is approximated by the confrontation between the generator and the discriminator. Therefore, the network needs to train the generator network and the discriminator network at the same time in one cycle. The loss function is as follows: ; ; The adversarial loss function of the discriminator is: ; The adversarial loss function of the generator is: ; in, Express expectations, represents the discriminator, Represent the predicted label and the real label respectively. In addition, in addition to the adversarial loss, the loss function of the generator also supplements the regularization loss L1 that is beneficial to network convergence and the MSSIM loss of the reconstructed image color saturation. The formula is as follows: ; in, Network prediction value, is the true value.

[0071] ; in, and Represent the mean and variance of the predicted and real images respectively, and represents a constant, N is a multi-scale component, represents the covariance between the predicted and true images. and represents the weight coefficient of different multi-scale components. Therefore, the total loss of the generator is: .

[0072] In this embodiment, an initial noise image is input to an initial generation network for image generation to obtain a generated noise image; the generated noise image and the sample noise image are input to a target discrimination network for image discrimination to obtain a target discrimination result; the initial generation network is trained according to the target discrimination result and the generation loss function to obtain a target generation network. By adopting a generative adversarial network (GAN), that is, using both the generator and the discriminator network to compare the generation results during training, the trained network can retain more effective details and edge feature information, and only the generator needs to be used for reasoning, which greatly increases the efficiency of the algorithm.

[0073] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the image processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0074] This application also provides an image processing device, please refer to Figure 8 , the image processing device comprises: An input module 10 is used to input the image to be denoised into a decoupler in a target generation network to obtain target feature information of the image to be denoised; The processing module 20 is used to perform noise elimination according to the target feature information, the controller in the target generation network and the reconstructor in the target generation network to obtain a target processed image. The joint structure corresponding to the decoupler, the controller and the reconstructor consists of multiple downsampling layers, multiple upsampling layers, a head layer, a tail layer, a multi-layer perceptron and a codec layer.

[0075] Optionally, the processing module 20 is further used for: Inputting the target feature information into a controller in the target generation network to obtain a noise control signal; Noise elimination is performed according to the target feature information, the noise control signal and the reconstructor in the target generation network to obtain a target processed image.

[0076] Optionally, the input module 10 is further used for: Input the initial noise image to the initial generation network for image generation to obtain a generated noise image; Inputting the generated noise image and the sample noise image into the target discrimination network for image discrimination to obtain a target discrimination result; The initial generation network is trained according to the target discrimination result and the generation loss function to obtain a target generation network.

[0077] Optionally, the input module 10 is further used for: Calculate the loss function according to the target discrimination result to obtain a relative average adversarial loss function; Calculate the loss function between pixels according to the generated noise image and the sample noise image to obtain a pixel difference loss function; Calculate the loss function between structures according to the generated noise image and the sample noise image to obtain a structural similarity loss function; A generation loss function is obtained according to the relative average adversarial loss function, the pixel difference loss function and the structural similarity loss function.

[0078] Optionally, the input module 10 is further used for: Inputting the generated noise image into the target discrimination network for image discrimination to obtain the probability of generated image; Input the sample noise image to the target discrimination network for image discrimination and obtain the sample image probability; The target discrimination result is obtained according to the generated image probability and the sample image probability.

[0079] Optionally, the input module 10 is further used for: Adjusting parameters of the initial generation network according to the target discrimination result to obtain an adjusted generation network; The adjustment generation network is trained according to the training sample set and the generation loss function to obtain the target generation network.

[0080] Optionally, the processing module 20 is further used for: Performing quality assessment on the target processed image according to a preset image assessment method to obtain a quality assessment result; Determine corresponding optimization hyperparameters according to the quality assessment result; The hyperparameters of the target generation network are adjusted according to the optimized hyperparameters to obtain an optimized target generation network.

[0081] The image processing device provided by the present application adopts the image processing method in the above embodiment, which can solve the technical problem that details and edge information in the video cannot be retained while removing compression noise. Compared with the prior art, the beneficial effects of the image processing device provided by the present application are the same as the beneficial effects of the image processing method provided by the above embodiment, and other technical features in the image processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0082] The present application provides an image processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image processing method in the above-mentioned embodiment one.

[0083] Reference below Fig. 9 , which shows a schematic diagram of the structure of an image processing device suitable for implementing an embodiment of the present application. The image processing device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The image processing device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0084] like Fig. 9 As shown, the image processing device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the image processing device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the image processing device to communicate with other devices wirelessly or by wire to exchange data. Although the image processing device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0085] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0086] The image processing device provided by the present application adopts the image processing method in the above embodiment, which can solve the technical problem that details and edge information in the video cannot be retained while removing compression noise. Compared with the prior art, the beneficial effects of the image processing device provided by the present application are the same as the beneficial effects of the image processing method provided by the above embodiment, and other technical features in the image processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0087] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0088] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0089] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the image processing method in the above-mentioned embodiment.

[0090] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0091] The computer-readable storage medium may be included in the image processing device; or may exist independently without being assembled into the image processing device.

[0092] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an image processing device, the image processing device: inputs the image to be denoised into the decoupler in the target generation network to obtain target feature information of the image to be denoised; performs noise elimination according to the target feature information, the controller in the target generation network and the reconstructor in the target generation network to obtain a target processed image, and the corresponding joint structure of the decoupler, the controller and the reconstructor consists of multiple downsampling layers, multiple upsampling layers, a head layer, a tail layer, a multi-layer perceptron and a codec layer.

[0093] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0094] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0095] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0096] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned image processing method, and can solve the technical problem of not being able to retain details and edge information in the video while removing compression noise. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the image processing method provided in the above-mentioned embodiment, and will not be described in detail here.

[0097] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned image processing method when executed by a processor.

[0098] The computer program product provided by the present application can solve the technical problem of not being able to retain details and edge information in the video while removing compression noise. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the image processing method provided by the above embodiment, and will not be described in detail here.

[0099] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An image processing method, characterized in that: The image processing method comprises: Inputting the image to be denoised into the decoupler in the target generation network to obtain target feature information of the image to be denoised; Noise elimination is performed according to the target feature information, the controller in the target generation network and the reconstructor in the target generation network to obtain a target processed image. The joint structure corresponding to the decoupler, the controller and the reconstructor consists of multiple downsampling layers, multiple upsampling layers, a head layer, a tail layer, a multi-layer perceptron and a codec layer.

2. The method according to claim 1, characterized in that The step of performing noise elimination according to the target feature information, the controller in the target generation network and the reconstructor in the target generation network to obtain a target processed image comprises: Inputting the target feature information into a controller in the target generation network to obtain a noise control signal; Noise elimination is performed according to the target feature information, the noise control signal and the reconstructor in the target generation network to obtain a target processed image.

3. The method according to claim 1, characterized in that Before the step of inputting the image to be denoised into the decoupler in the target generation network to obtain the target feature information of the image to be denoised, the method further includes: Input the initial noise image to the initial generation network for image generation to obtain a generated noise image; Inputting the generated noise image and the sample noise image into the target discrimination network for image discrimination to obtain a target discrimination result; The initial generation network is trained according to the target discrimination result and the generation loss function to obtain a target generation network.

4. The method according to claim 3, characterized in that Before the step of training the initial generation network according to the target discrimination result and the generation loss function to obtain the target generation network, the step further includes: Calculate the loss function according to the target discrimination result to obtain a relative average adversarial loss function; Calculate the loss function between pixels according to the generated noise image and the sample noise image to obtain a pixel difference loss function; Calculate the loss function between structures according to the generated noise image and the sample noise image to obtain a structural similarity loss function; A generation loss function is obtained according to the relative average adversarial loss function, the pixel difference loss function and the structural similarity loss function.

5. The method according to claim 3, characterized in that The step of inputting the generated noise image and the sample noise image into the target discrimination network for image discrimination to obtain the target discrimination result includes: Inputting the generated noise image into the target discrimination network for image discrimination to obtain the probability of generated image; Input the sample noise image to the target discrimination network for image discrimination to obtain the sample image probability; The target discrimination result is obtained according to the generated image probability and the sample image probability.

6. The method according to claim 3, characterized in that The step of training the initial generation network according to the target discrimination result and the generation loss function to obtain the target generation network comprises: Adjusting parameters of the initial generation network according to the target discrimination result to obtain an adjusted generation network; The adjustment generation network is trained according to the training sample set and the generation loss function to obtain the target generation network.

7. The method according to any one of claims 1 to 6, characterized in that After the step of performing noise elimination according to the target feature information, the controller in the target generation network and the reconstructor in the target generation network to obtain a target processed image, the method further includes: Performing quality assessment on the target processed image according to a preset image assessment method to obtain a quality assessment result; Determine corresponding optimization hyperparameters according to the quality assessment result; The hyperparameters of the target generation network are adjusted according to the optimized hyperparameters to obtain an optimized target generation network.

8. An image processing device, characterized in that: The device comprises: An input module, used for inputting the image to be denoised into the decoupler in the target generation network to obtain target feature information of the image to be denoised; A processing module is used to perform noise elimination according to the target feature information, the controller in the target generation network and the reconstructor in the target generation network to obtain a target processed image. The joint structure corresponding to the decoupler, the controller and the reconstructor consists of multiple downsampling layers, multiple upsampling layers, a head layer, a tail layer, a multi-layer perceptron and a codec layer.

9. An image processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image processing method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

Images