Training method, image processing method, device and storage medium for generating adversarial network
By initializing and optimizing generators and judges in the generative adversarial network, the problem of noise and rough edges in super-resolution images is solved, and the automatic supplementation of image details and improvement of effect is achieved.
Patent Information
- Application Number
- CN201910104171.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-01-18
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2039-02-14
AI Technical Summary
Existing super-resolution image generation methods, such as SRGAN, have problems with noise and edge roughness, affecting the image effect.
By initializing the generator, global judge and pixel judge in the generative adversarial network, fixing the parameters of any two models and updating the remaining models until the convergence conditions are met, the parameter optimization of the generative adversarial network is achieved, thereby processing at super resolution Automatically supplement image details and avoid noise and rough edges.
It realizes the optimization of each model parameter in the generative adversarial network, automatically supplements image details, avoids noise and edge roughness, and improves image processing effect.
Smart Images

Figure CN109801221B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to computer technology, and in particular to a training method, an image processing method, a device and a storage medium for generating an adversarial network. Background Art
[0002] Super-resolution technology (SR) refers to reconstructing a corresponding super-resolution image from an observed low-resolution image, which can improve the image resolution. It is referred to as super-resolution and has important application value in fields such as satellite images and medical imaging. SR can be divided into two categories: multi-frame super-resolution and single-frame super-resolution. Single-frame super-resolution processes an image to obtain a super-resolution image corresponding to the image. Multi-frame super-resolution processes multiple images to obtain super-resolution images corresponding to multiple images. SR based on deep learning is mainly single image super-resolution reconstruction (SISR) based on single-frame super-resolution to reconstruct a single low-resolution image.
[0003] SISR includes network models such as Super-Resolution Convolutional Neural Network (SRCNN) and Super Resolution Generative Adversarial Network (SRGAN).
[0004] SRCNN is a super-resolution model that can perform super-resolution on images based on a fixed magnification, but SRCNN cannot restore random details of the image.
[0005] SRGAN is a deep learning-based super-resolution network model that can automatically supplement image details. However, in some scenarios, the generated super-resolution images such as Figure 1 As shown, there is obvious distorted noise at position 101, and the edges are rough, which affects the image effect. Summary of the invention
[0006] The embodiments of the present invention are intended to provide a training method, an image processing method, an apparatus, and a storage medium for a generative adversarial network, which can improve the image processing effect of the generative adversarial network.
[0007] The technical solution of the embodiment of the present invention is achieved as follows:
[0008] An embodiment of the present invention provides a training method for a generative adversarial network, the method comprising:
[0009] Initialize the generator in the generative adversarial network;
[0010] Obtaining a global decision loss and a pixel decision loss based on the generator;
[0011] Initialize a global decision device in the generative adversarial network based on the global decision loss;
[0012] Initialize a pixel decision device in the generative adversarial network based on the pixel decision loss;
[0013] The parameters of any two models among the generator, the global judger and the pixel judger are fixed, and the parameters of the remaining models are updated until the convergence condition of the generative adversarial network is met.
[0014] In the above scheme, the initializing the global decision maker in the generative adversarial network based on the global decision loss includes: iteratively updating the parameters of the global decision maker based on the global decision loss until the global decision loss converges;
[0015] Initializing a pixel decider in the generative adversarial network based on the pixel decision loss includes iteratively updating parameters of the pixel decider based on the pixel decision loss until the pixel decision loss converges.
[0016] In the above scheme, in the generator: the training image is convolved to obtain a first feature map, and the first feature map is convolved to obtain a second feature map; the first feature map and the second feature map are fused to obtain a first fused feature map; pixels of the first fused feature map are rearranged to obtain a rearranged feature map; and the rearranged feature map is convolved to obtain a reconstructed image.
[0017] In the above scheme, in the global decider: the reconstructed image is convolved to obtain at least two feature maps; the third feature map and the fourth feature map in the at least two feature maps are fused to obtain a second fused feature map; and a decision is made based on the second fused feature map to obtain a first probability of judging the reconstructed image as an unnatural image.
[0018] In the above scheme, in the pixel determiner: the reconstructed image is convolved to obtain a fifth feature map of the same size as the reconstructed image; each pixel in the fifth feature map is judged to obtain a first probability matrix for judging each pixel in the fifth feature map as a pixel of a non-natural image.
[0019] In the above scheme, the convergence condition of the generative adversarial network is met, including:
[0020] The parameters of the generator, the global decision maker and the pixel decision maker are iteratively updated until the perceptual loss of the generator, the global decision loss of the global decision maker and the pixel decision loss of the pixel decision maker all converge.
[0021] An embodiment of the present invention provides an image processing method, the method comprising:
[0022] Get the image to be processed;
[0023] In the generator, convolution processing is performed on the image to be processed to obtain a target image corresponding to the image to be processed;
[0024] The generator is obtained by training the global judge and the pixel judge in the generative adversarial network where the generator is located through the following processing:
[0025] Initializing the generator, the global determiner and the pixel determiner;
[0026] The parameters of the generator, the global judger and the pixel judger are updated until the convergence condition of the generative adversarial network is met.
[0027] In the above solution, the step of performing convolution processing on the image to be processed to obtain a target image corresponding to the image to be processed includes:
[0028] Convolving the image to be processed to obtain a sixth feature map, and convolving the sixth feature map to obtain a seventh feature map;
[0029] Fusing the sixth feature map and the seventh feature map to obtain a third fused feature map;
[0030] Rearranging pixels of the third fused feature map to obtain a rearranged feature map;
[0031] Convolution is performed on the rearranged feature map to obtain the target image.
[0032] In the above solution, the updating of the parameters of the generator, the global determiner and the pixel determiner includes:
[0033] The parameters of any two models among the generator, the global decision maker and the pixel decision maker are fixed, and the parameters of the remaining models are updated.
[0034] An embodiment of the present invention provides a training device for generating an adversarial network, the device comprising:
[0035] The initial generation unit is used to initialize the generator in the generative adversarial network;
[0036] A loss determination unit, configured to obtain a global decision loss and a pixel decision loss based on the generator;
[0037] An initial global unit, configured to initialize a global decision maker in the generative adversarial network based on the global decision loss;
[0038] An initial pixel unit, used to initialize a pixel decision device in the generative adversarial network based on the pixel decision loss;
[0039] An updating unit is used to fix the parameters of any two models among the generator, the global judge and the pixel judge, and to update the parameters of the remaining models until the convergence condition of the generative adversarial network is met.
[0040] An embodiment of the present invention provides an image processing device, the device comprising:
[0041] An acquisition unit, used for acquiring an image to be processed;
[0042] A super-resolution unit, used for performing convolution processing on the image to be processed in a generator of a generative adversarial network to obtain a target image corresponding to the image to be processed;
[0043] The generator is obtained by the following processing based on the global judger and the pixel judger in the generative adversarial network:
[0044] Initializing the generator, the global determiner and the pixel determiner;
[0045] The parameters of the generator, the global judger and the pixel judger are updated until the convergence condition of the generative adversarial network is met.
[0046] An embodiment of the present invention provides a training device for generating an adversarial network, the device comprising: a memory, a communication bus, and a processor, wherein:
[0047] The memory is used to store computer programs;
[0048] The communication bus is used to realize the connection and communication between the processor and the memory;
[0049] The processor is used to implement the steps of the above-mentioned training method for generating an adversarial network when executing the computer program stored in the memory.
[0050] An embodiment of the present invention provides an image processing device, the device comprising: a memory, a communication bus and a processor, wherein:
[0051] The memory is used to store computer programs;
[0052] The communication bus is used to realize the connection and communication between the processor and the memory;
[0053] The processor is used to implement the steps of the above-mentioned image processing method when executing the computer program stored in the memory.
[0054] An embodiment of the present invention provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, the training method of the above-mentioned generative adversarial network or the above-mentioned image processing method is implemented.
[0055] The embodiments of the present invention have the following beneficial effects:
[0056] In a generative adversarial network including a generator, a global judge and a pixel judge, when training the generative adversarial network, the parameters of the remaining models are updated while the parameters of two models are fixed, thereby realizing the adversarial relationship between the global judge, the pixel judge and the generator, so that the parameters of each model in the generative adversarial network are optimized, and the generator has the performance of automatically supplementing image details while avoiding image noise, rough edges and other problems when performing super-resolution processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 It is a schematic diagram of the effect of super-resolution image in the related art;
[0058] Figure 2 This is a schematic diagram of the structure of the GAN in the embodiment of the present invention. Figure 1 ;
[0059] Figure 3 is a schematic diagram of a network structure of an embodiment of the present invention;
[0060] Figure 4 Schematic diagram of the training method of the generative adversarial network according to the embodiment of the present invention Figure 1 ;
[0061] Figure 5 It is a schematic diagram of the structure of SRCNN according to an embodiment of the present invention;
[0062] Figure 6 This is a schematic diagram of the structure of the GAN in the embodiment of the present invention. Figure 2 ;
[0063] Figure 7 is a flowchart of an image processing method according to an embodiment of the present invention;
[0064] Figure 8 This is a schematic diagram of the structure of SRGAN;
[0065] Fig. 9 is a schematic diagram of the structure of a generator in an embodiment of the present invention;
[0066] Fig.10 is a schematic diagram of the structure of a residual block in an embodiment of the present invention;
[0067] Fig.11 is a schematic diagram of the structure of a global decision device in an embodiment of the present invention;
[0068] Fig.12 is a schematic diagram of the structure of a local decision device in an embodiment of the present invention;
[0069] Fig.13 is a schematic diagram of the structure of a supervisor in an embodiment of the present invention;
[0070] Fig.14 FIG. 1 is a flow chart of a training method for generating an adversarial network according to an embodiment of the present invention. Figure 2 ;
[0071] Fig.15 Schematic diagram of the comparison of super-resolution image effects in the embodiments of the present invention Figure 1 ;
[0072] Fig.16 Schematic diagram of the comparison of super-resolution image effects in the embodiments of the present invention Figure 2 ;
[0073] Fig.17 It is a schematic diagram of the composition structure of a training device for generating an adversarial network according to an embodiment of the present invention;
[0074] Fig.18 is a schematic diagram of the composition structure of an image processing device according to an embodiment of the present invention;
[0075] Fig.19 It is a schematic diagram of the hardware composition structure of a training device or an image processing device for generating an adversarial network according to an embodiment of the present invention. DETAILED DESCRIPTION
[0076] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.
[0077] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0078] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present invention have the same meanings as those commonly understood by those skilled in the art of the technical field of the embodiments of the present invention. The terms used in the embodiments of the present invention are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.
[0079] Before further describing the embodiments of the present invention in detail, the nouns and terms involved in the embodiments of the present invention are described. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations.
[0080] 1) Generative Adversarial Networks (GAN), also known as Generative Adversarial Networks, is a deep learning model. The model produces fairly good output through the mutual game learning of at least two modules in the framework: the generative model G (Generative Model) and the discriminative model D (Discriminative Model). For example: G is a model for making super-resolution images (also called reconstructed images in this article), and D is a model for detecting whether it is an original natural image. The goal of G is to make D unable to judge whether the super-resolution image generated by G is an unnatural image. D should try its best to distinguish whether the input image is an original natural image or an unnatural image generated by G, and continuously iterate and update the parameters of G and D until the generative adversarial network meets the convergence conditions.
[0081] 2) Generator, i.e., generative model, is used to generate high-resolution images from low-resolution images. The generator can be a convolutional neural network based on deep learning.
[0082] 3) The discriminator is a discriminant model that determines whether the input image x is an unnatural image generated by the generator or a natural image. The discriminator outputs a probability value D1(x) in the range of 0-1. When D1(x) is 0, it means that the input image x is a natural image. When D1(x) is 1, it means that the input image x is an unnatural image.
[0083] 4) Natural images, images that have not been processed by the generative adversarial network to increase resolution, can be images directly captured by image acquisition devices such as cameras.
[0084] 5) Unnatural images, images obtained through resolution enhancement processing using a generative adversarial network.
[0085] 6) Training images: Low-resolution images for training the generative adversarial network can be obtained by downsampling high-resolution original natural images.
[0086] 7) Original natural images, high-resolution images for training generative adversarial networks, are natural images. The corresponding low-resolution training images can be obtained through downsampling.
[0087] The embodiments of the present invention can provide a training method, an image processing method, a device and a storage medium for a generative adversarial network. In practical applications, the training method for a generative adversarial network can be implemented by a training device for a generative adversarial network, and each functional module in the training device for a generative adversarial network can be collaboratively implemented by hardware resources of a device (such as a terminal device, a server or a server cluster), such as computing resources such as a processor, and communication resources (such as for supporting various communication methods such as optical cables and cellular).
[0088] The training method of the generative adversarial network of the embodiment of the present invention can be applied to Figure 2 The GAN shown in Figure 2 As shown, it includes three convolutional neural networks: a generator 201, a global decider 202 and a pixel decider 203. Among them, the generator 201 is used to improve the resolution of the low-resolution image and output the image with enlarged resolution, that is, the reconstructed image. The global decider 202 is used to determine whether the received image is an unnatural image generated by the generator or an original natural image, and output the probability of the judgment. The pixel decider 203 is used to determine whether the pixels of the received image are pixels of the unnatural image generated by the generator or the pixels of the original natural image, and output the probability matrix of the judgment.
[0089] The generator 201, the global judge 202 and the pixel judge 203 are trained, the generator 201, the global judge 202 and the pixel judge 203 are initialized, and the parameters of any two models in the generator 201, the global judge 202 and the pixel judge 203 are fixed, and the parameters of the remaining models are updated until the convergence condition of the generative adversarial network is met, and the generative adversarial network completes the training, thereby obtaining the optimal parameters of the GAN based on mutual competition. The perceptual loss is determined based on the probability output by the global judge 202 and the probability matrix output by the pixel judge 203, and the parameters of the generator 201 are updated based on the perceptual loss.
[0090] exist Figure 2 Based on the GAN shown in FIG. 1 , a system architecture of applying GAN in an embodiment of the present invention can be as follows: Figure 3 As shown, it includes: a client 31 and a server 32, wherein the client 31 or the server 32 is deployed with Figure 2 The GAN shown in Figure 2 After the GAN training shown is completed, the generator 201 trained in the GAN can be deployed in the client 31, or the generator 201 trained in the GAN can be deployed on the server 32.
[0091] A mobile application (App) for image processing may be installed on the client 31 , and a super-resolution request for improving the resolution of the image to be processed may be triggered through the image processing App.
[0092] When the trained generator 201 is deployed in the client 31, the generator 201 deployed in the client 31 receives the image to be processed, performs convolution on the image to be processed, and obtains a target image with improved resolution.
[0093] In a specific usage scenario, such as Figure 3 As shown, the trained generator 201 is deployed in the server 32, and the client 31 sends the image to be processed 33 to the server 32 through a super-resolution request. After receiving the super-resolution request, the server 32 performs convolution on the image to be processed 33 carried by the super-resolution request through the deployed generator 201 to obtain a target image 34 with an enlarged resolution, and sends the target image 34 to the client 31.
[0094] Next, combine Figure 2 and Figure 3 The network architecture diagram shown illustrates various embodiments of the training method, image processing method, device and storage medium for generating adversarial networks provided in the embodiments of the present invention.
[0095] The present invention provides a training method for generating an adversarial network. Figure 4 As shown, it will be combined Figure 4 The steps shown are explained.
[0096] S401, initializing the generator in the generative adversarial network.
[0097] right Figure 2 When the generator in the GAN shown is initialized, a training image and an original natural image corresponding to the training image are obtained, and the training image is input into the generator. The training image and the original natural image are the same image with different resolutions, the training image is a low-resolution image, and the original natural image is a high-resolution image.
[0098] The training image can be obtained by downsampling the original natural image. For example, for an original natural image of size M*N, the image in the s*s window of the original natural image is converted into a pixel, and the value of this pixel is the average of all pixels in the s*s window, thereby downsampling by s times to obtain a training image with a size of (M / s)*(N / s) and a resolution reduced by s times relative to the original natural image.
[0099] After receiving the training image, the generator performs resolution enhancement processing on the training image to obtain a reconstructed image. Here, when the size of the training image is H×W, the size of the reconstructed image is rH×rW, where r is the magnification of the resolution. Here, r and s can be the same size.
[0100] The original natural image corresponding to the reconstructed image and the training image is calculated by formula (1) to obtain the mean square error loss Loss_mse of the generator.
[0101] Loss_mse =∑ ( G(x i ) - y i ) 2 Formula (1);
[0102] Among them, x i is the i-th training image, i is an integer between 0 and the total number of training images in the training image set; G(x i ) is the reconstructed image generated by the generator G based on the i-th training image; i For x i The corresponding original natural image.
[0103] The parameters of the generator are iteratively updated through the mean square error loss, so that the mean square error loss of the generator is reduced until the mean square error loss converges.
[0104] When the parameters of the generator are iteratively updated through the mean square error loss, the parameters of the generator are continuously updated through the optimization method to reduce the mean square error loss of the generator. When the decrease value of the mean square error loss of the generator after multiple updates is less than the set first threshold, it is considered that the mean square error loss of the generator has converged, and the updating of the parameters of the generator is stopped.
[0105] For example: the generator obtains reconstructed image 1 based on the training image, and the mean square error loss based on the reconstructed image 1 is Loss_mse1. The parameters of the generator are updated for the first time based on the optimization processing, and the training image is input into the updated generator to obtain reconstructed image 2. Based on the reconstructed image 2, the mean square error loss of the generator is Loss_mse2. The parameters of the generator are updated for the second time based on the optimization processing, and the training image is input into the updated generator to obtain reconstructed image 3. Based on the reconstructed image 3, the mean square error loss is Loss_mse3. Until the mean square error loss converges, stop updating the parameters of the generator and complete the initialization of the generator.
[0106] The optimization method used in the optimization process may include: a method for optimizing the loss function such as a gradient descent method, a Newton method, and a quasi-Newton method. It should be noted that in the embodiment of the present invention, there is no restriction on the optimization method used in the optimization process.
[0107] In the gradient descent method, the negative gradient direction of the current position is used as the search direction because this direction is the fastest descent direction of the current position. The closer the steepest descent method is to the target value, the smaller the step size and the slower the progress. When the loss function is a convex function, the solution of the gradient descent method is a global solution.
[0108] Newton's method is a method for approximately solving equations over real and complex numbers. It uses the first few terms of the Taylor series of the function f(x) to find the roots of the equation f(x) = 0.
[0109] The quasi-Newton method improves the defect of the Newton method that the inverse matrix of the complex Hessian matrix needs to be solved each time. It uses a positive definite matrix to approximate the inverse of the Hessian matrix, thereby simplifying the complexity of the operation.
[0110] S402: Obtain a global decision loss and a pixel decision loss based on the generator.
[0111] Here, the global decision loss is determined based on the global difference between the original natural image and the reconstructed image output by the generator; and the pixel decision loss is determined based on the pixel difference between the original natural image and the reconstructed image output by the generator.
[0112] After completing the initialization of the generator in S401, the training image is input into the generator, and the generator outputs a reconstructed image. A global decision loss is obtained based on the global difference between the original natural image and the reconstructed image, and a pixel decision loss is obtained based on the pixel difference between the original natural image and the reconstructed image.
[0113] In some embodiments, determining the global decision loss based on a global difference between the original natural image and the reconstructed image output by the generator comprises:
[0114] The generator performs resolution enhancement processing on the training image to obtain the reconstructed image; the global decider performs judgment on the reconstructed image and the original natural image to obtain a first initial probability of judging the reconstructed image as a non-natural image and a second initial probability of judging the original natural image as a non-natural image; wherein the training image is obtained by downsampling the original natural image; and the global judgment loss is obtained based on the first initial probability and the second initial probability.
[0115] The global judge judges the input image to determine whether the input image is an unnatural image, and outputs the probability of judging the input image as an unnatural image. The probability output by the global judge is between 0 and 1. When the output probability is greater than the probability threshold, the judgment result of the global judge indicates that the input image is judged as an unnatural image. When the output probability is less than the probability threshold, the judgment result of the global judge indicates that the input image is judged as a natural image. The probability threshold can be 0.5.
[0116] Here, the parameters of the generator are fixed, and a reconstructed image is obtained in the generator based on the training image. The reconstructed image and the original natural image are judged in the global judge to obtain the first initial probability of judging the reconstructed image as an unnatural image and the second initial probability of judging the original natural image as an unnatural image. The first initial probability and the second initial probability are calculated by formula (2) to obtain the global judgment loss Loss_global_d of the global judge.
[0117] Loss_global_d = ∑ - log(D1(G(x i ))) + ∑ - log( 1-D1( y i ) ) formula (2);
[0118] Among them, D1(G(x i )) is the global decision maker D1 that reconstructs the image G(x i ) is judged as the first probability of an unnatural image; D1(y i ) is the global decision maker D1 that converts the original natural image y i The second probability of judging as an unnatural image. At this time, the first probability is the first initial probability, and the second probability is the second initial probability.
[0119] In some embodiments, determining the pixel decision loss based on the pixel difference between the original natural image and the reconstructed image output by the generator comprises:
[0120] The generator performs resolution enhancement processing on the training image to obtain the reconstructed image; the pixel determiner performs judgment on the reconstructed image and the original natural image to obtain a first initial probability matrix for judging pixels of the reconstructed image as pixels of a non-natural image, and a second initial probability matrix for judging pixels of the original natural image as pixels of a non-natural image; wherein the training image is obtained by downsampling the original natural image; and the pixel judgment loss is obtained based on the first initial probability matrix and the second initial probability matrix.
[0121] The pixel judger judges the input image to determine whether each pixel in the input image is a pixel of an unnatural image, that is, whether the input image is a pixel generated by the generator, and outputs a probability matrix for judging the pixels of the input image as pixels of an unnatural image, wherein the probability matrix is a matrix composed of the probabilities of each pixel being judged as a pixel of an unnatural image. The size of the output probability matrix is the same as the size of the input image, and each element corresponds to a different pixel.
[0122] Each element of the probability matrix output by the pixel decision device is between 0 and 1. When the output element is greater than the element threshold, it indicates that the pixel corresponding to the element in the input image is judged as a pixel of an unnatural image. When the output element is less than the element threshold, it indicates that the pixel corresponding to the element in the input image is judged as a pixel of a natural image. The element threshold can be 0.5.
[0123] Here, the parameters of the generator are fixed, and a reconstructed image is obtained in the generator based on the training image. The pixel judger judges the reconstructed image and the original natural image to obtain a first initial probability matrix for judging the pixels of the reconstructed image as pixels of the non-natural image and a second initial probability matrix for judging the pixels of the original natural image as pixels of the non-natural image. The first initial probability matrix and the second initial probability matrix are calculated by formula (3) to obtain the pixel judgment loss Loss_pixel_d of the pixel judger.
[0124] Loss_pixel_d = ∑ - log(sum(D2(G(x i )))) + ∑ - log( sum(1-D2( y i )) )
[0125] Formula (3);
[0126] Among them, D2(G(x i )) is the pixel decision device D2 that reconstructs the image G(x i ) is the first probability matrix of pixels judged as pixels of unnatural images; D2(y i ) is the pixel decision device D2 that converts the original natural image y i The second probability matrix of the pixel judged as a pixel of an unnatural image, sum() is a matrix summation of the elements of the matrix. At this time, the first probability matrix is the first initial probability matrix, and the second probability matrix is the second initial probability matrix.
[0127] S403: Initialize a global decision maker in the generative adversarial network based on the global decision loss.
[0128] Wherein, the parameters of the global decision device are iteratively updated based on the global decision loss until the global decision loss converges.
[0129] The global decision loss is obtained through S402, and the parameters of the global decision device are updated based on the global decision loss, so that the global decision loss of the global decision device is reduced, and the parameters of the global decision device are continuously updated through multiple iterations until the global decision loss converges.
[0130] The parameters of the global decision maker are continuously updated through the optimization method to reduce the global decision loss of the global decision maker. When the decrease value of the global decision loss of the global decision maker after multiple iterations is less than the set second threshold, it is considered that the global decision loss of the global decision maker has converged, and the updating of the parameters of the global decision maker is stopped.
[0131] For example: the global judge obtains the global judgment loss Loss_global_d1 based on the reconstructed image and the original natural image, and the parameters of the global judge are updated for the first time based on the optimization processing. After the first update, the global judge obtains the global judgment loss Loss_global_d2 based on the reconstructed image and the original natural image. The parameters of the global judge are updated for the second time based on the optimization processing. After the second update, the global judge obtains the global judgment loss Loss_global_d3 based on the reconstructed image and the original natural image. The parameters of the global judge continue to be updated until the global judgment loss converges, stop updating the parameters of the global judge, and complete the initialization of the global judge.
[0132] The optimization method used in the optimization process may include: methods for optimizing the loss function such as gradient descent method, Newton method and quasi-Newton method. It should be noted that in the embodiment of the present invention, there is no restriction on the optimization method used in the optimization process.
[0133] S404: Initialize a pixel decision device in the generative adversarial network based on the pixel decision loss.
[0134] Wherein, the parameters of the pixel decision device are iteratively updated based on the pixel decision loss until the pixel decision loss converges.
[0135] The pixel decision loss is obtained through S402, and the parameters of the pixel decider are iteratively updated based on the pixel decision loss, so that the pixel decision loss of the pixel decider is reduced, and the parameters of the pixel decider are continuously updated through multiple iterations until the pixel decision loss converges.
[0136] The parameters of the pixel decision maker are continuously updated by an optimization method to reduce the pixel decision loss of the pixel decision maker. When the decrease value of the pixel decision loss of the pixel decision maker after multiple iterations is less than the set third threshold, it is considered that the pixel decision loss of the pixel decision maker has converged, and the updating of the parameters of the pixel decision maker is stopped.
[0137] For example: the pixel decider obtains the pixel decision loss Loss_pixel_d1 based on the reconstructed image and the original natural image, and the parameters of the pixel decider are updated for the first time based on the optimization processing. After the first update, the pixel decider obtains the pixel decision loss Loss_pixel_d 2 based on the reconstructed image and the original natural image. The parameters of the pixel decider are updated for the second time based on the optimization processing. After the second update, the pixel decider obtains the pixel decision loss Loss_pixel_d3 based on the reconstructed image and the original natural image. The parameters of the pixel decider continue to be updated until the pixel decision loss converges, stop updating the parameters of the pixel decider, and complete the initialization of the pixel decider.
[0138] The optimization method used in the optimization process may include: methods for optimizing the loss function such as gradient descent method, Newton method and quasi-Newton method. It should be noted that in the embodiment of the present invention, there is no restriction on the optimization method used in the optimization process.
[0139] In the embodiment of the present invention, the structure of the pixel decision device can be a pixel classification model such as a series of fully convolutional neural networks, where the input and output images have the same size. The embodiment of the present invention does not impose any limitation on the structure of the pixel classification model used by the pixel decision device.
[0140] S405, fixing the parameters of any two models among the generator, the global judgement device and the pixel judgement device, and updating the parameters of the remaining models until the convergence condition of the generative adversarial network is met.
[0141] After the generator, global decision maker and pixel decision maker in the network are initialized based on S401, S403 and S404, the parameters of the generator, global decision maker and pixel decision maker are updated. Wherein, the parameters of the generator, the global decision maker and the pixel decision maker are iteratively updated in the following manner until the convergence condition of the generative adversarial network is met, including: fixing the parameters of any two models of the generator, the global decision maker and the pixel decision maker, and updating the parameters of the remaining models until the parameters of the generator, the global decision maker and the pixel decision maker are all updated once.
[0142] Here, the parameters of the two models in the generator, global decision maker, and pixel decision maker are fixed, and the parameters of the remaining model are updated until the convergence condition of the generative adversarial network is met. The convergence condition of the generative adversarial network can be that the number of iterations reaches the set iteration threshold, or that the network converges.
[0143] The parameters of the two models in the generator, global discriminator and pixel discriminator are fixed, and the parameters of the remaining models are updated, including the following three update cases:
[0144] Case 1: Fix the parameters of the global discriminator and the pixel discriminator, and update the parameters of the generator.
[0145] Case 2: Fix the parameters of the generator and the pixel judgement, and update the parameters of the global generator.
[0146] Case 3: Fix the parameters of the generator and the global judgement, and update the parameters of the pixel generator.
[0147] In some embodiments, until the convergence condition of the generative adversarial network is met, the step includes iteratively updating the parameters of the generator, the global decider, and the pixel decider until the perceptual loss of the generator, the global decision loss of the global decider, and the pixel decision loss of the pixel decider all converge.
[0148] Here, the parameters of the generator, global discriminator, and pixel discriminator are iteratively updated until the convergence conditions of the generative adversarial network are met.
[0149] In practical applications, the execution order of the three update situations is not limited in any way. When the parameters of the generator, the global judge and the pixel judge are updated once, it is determined that an iteration is completed.
[0150] The parameters of the generator are updated based on the optimization method, so that the perceptual loss of the generator is reduced. The parameters of the global decision maker are updated based on the optimization method, so that the global decision loss of the global decision maker is reduced. The parameters of the pixel decision maker are updated based on the optimization method, so that the pixel decision loss of the pixel decision maker is reduced. The optimization methods used by different models can be the same or different.
[0151] Here, the parameters of the global discriminator and the pixel discriminator are fixed, the perceptual loss 1 is obtained based on the reconstructed image 1 of the training image, and the parameters of the generator are updated once based on the perceptual loss 1. The training image is received in the updated generator to obtain the reconstructed image 2. The parameters of the generator and the pixel discriminator are fixed, the global decision loss 2 of the global discriminator is obtained based on the reconstructed image 2 and the original natural image, and the parameters of the global discriminator are updated once based on the global decision loss 2. The parameters of the generator and the global discriminator are fixed, the pixel decision loss 2 of the pixel discriminator is obtained based on the reconstructed image 2 and the original natural image, and the parameters of the pixel discriminator are updated once based on the pixel decision loss 2.
[0152] At this point, the update of the parameters of the generator, global decision maker, and pixel decision maker is completed. If any of the perceptual loss of the generator, the global decision loss of the global decision maker, and the pixel decision loss of the pixel decision maker has not converged, continue to perform the above update:
[0153] Maintain the parameters of the global judge and the pixel judge, obtain the perceptual loss 2 based on the reconstructed image 2 of the training image, and update the parameters of the generator once based on the perceptual loss 2. Receive the training image in the updated generator to obtain the reconstructed image 3. Fix the parameters of the generator and the pixel judge, obtain the global decision loss 3 of the global judge based on the reconstructed image 3 and the original natural image, and update the parameters of the global judge once based on the global decision loss 3. Fix the parameters of the generator and the global judge, obtain the pixel decision loss 3 of the pixel judge based on the reconstructed image 3 and the original natural image, and update the parameters of the pixel judge once based on the pixel decision loss 3.
[0154] At this time, the parameters of the generator, global decider and pixel decider are updated again. If any of the perceptual loss of the generator, the global decision loss of the global decider and the pixel decision loss of the pixel decider has not converged, the above update is continued until the perceptual loss of the generator, the global decision loss of the global decider and the pixel decision loss of the pixel decider all converge.
[0155] In an embodiment of the present invention, in a generative adversarial network including a generator, a global judge and a pixel judge, when the generative adversarial network is trained, the parameters of the remaining models are updated while fixing the parameters of two models, thereby realizing the adversarial relationship between the global judge, the pixel judge and the generator, so that the parameters of each model in the generative adversarial network are optimized, so that the generator has the performance of automatically supplementing image details while avoiding problems such as image noise and rough edges when performing super-resolution processing.
[0156] The following describes the three updating situations involved in S405 respectively.
[0157] Case 1
[0158] The fixing of parameters of any two models among the generator, the global decision maker and the pixel decision maker, and updating parameters of the remaining models, comprises:
[0159] The parameters of the global judge and the pixel judge are fixed; the training image is subjected to resolution enhancement processing in the generator to obtain the reconstructed image; the reconstructed image is judged in the global judge and the pixel judge respectively to obtain a first probability of judging the reconstructed image as an unnatural image and a first probability matrix of judging the pixels of the reconstructed image as pixels of the unnatural image; a global adversarial loss is determined based on the first probability, a pixel adversarial loss is determined based on the first probability matrix, and a perceptual loss is determined based on the global adversarial loss and the pixel adversarial loss; the parameters of the generator are updated based on the perceptual loss.
[0160] Here, the parameters of the global decision maker and the pixel decision maker are fixed, and the generator performs convolution on the training image to obtain a reconstructed image with a resolution magnified by r times. The global decision maker makes a decision on the reconstructed image to obtain a first probability of judging the reconstructed image as an unnatural image. The pixel decision maker makes a decision on the reconstructed image to obtain a first probability matrix of judging each pixel in the reconstructed image as a pixel of an unnatural image.
[0161] The first probability is calculated by formula (4) to obtain the global adversarial loss Loss_global_gan,
[0162] Loss_global_gan = ∑ - log(D1(G(x i ))) Formula (4);
[0163] Among them, D1(G(x i )) is the global decision maker D1 based on the reconstructed image G(x i ) to obtain the first probability.
[0164] The first probability matrix is calculated by formula (5) to obtain the pixel adversarial loss Loss_pixel_gan,
[0165] Loss_pixel_gan = ∑ - log(sum(D2(G(x i )))) Formula (5);
[0166] Among them, D2(G(x i )) is the pixel decision device D2 based on the reconstructed image G(x i ) to obtain the first probability matrix, sum(D2(G(x i ))) is the probability matrix D2(G(xi )) performs matrix summation, that is, sums the values of all elements in the pixel matrix.
[0167] The perceptual loss is calculated based on the global adversarial loss and the pixel adversarial loss, and the parameters of the generator are updated using an optimization method based on the perceptual loss so that the perceptual loss is reduced.
[0168] The perceptual loss is a cost function based on adversarial learning, and can be obtained by weighted summing the global adversarial loss and the pixel adversarial loss based on the weights corresponding to the global adversarial loss and the pixel adversarial loss. In an embodiment of the present invention, the weights corresponding to the global adversarial loss and the pixel adversarial loss may be the same or different, for example, the corresponding weights are 1e-3 and 1e-3 respectively.
[0169] Case 2
[0170] The fixing of parameters of any two models among the generator, the global decision maker and the pixel decision maker, and updating parameters of the remaining models, comprises:
[0171] The parameters of the generator and the pixel judge are fixed; the training image is subjected to resolution enhancement processing in the generator to obtain the reconstructed image; the reconstructed image and the original natural image are judged in the global judge respectively to obtain a first probability of judging the reconstructed image as a non-natural image and a second probability of judging the original natural image as a non-natural image; wherein the training image is obtained by downsampling the original natural image; a global decision loss is obtained based on the first probability and the second probability, and the parameters of the global judge are updated based on the global decision loss.
[0172] Here, the parameters of the generator and the pixel judge are fixed, and the generator performs convolution on the training image to obtain a reconstructed image with a resolution magnified by r times. The global judge judges the reconstructed image and the original natural image respectively, and obtains a first probability of judging the reconstructed image as an unnatural image and a second probability of judging the original natural image as an unnatural image.
[0173] The first probability and the second probability are calculated by formula (2) to obtain the global decision loss Loss_global_d of the global decision maker.
[0174] Loss_global_d = ∑ - log(D1(G(x i ))) + ∑ - log( 1-D1(y i ) ) formula (2);
[0175] Among them, D1(G(x i )) is the global decision maker D1 that reconstructs the image G(xi ) is judged as the first probability of an unnatural image; D1(y i ) is the global decision maker D1 that transforms the original natural image y i The second probability of judging the image as unnatural.
[0176] According to the global decision loss, the parameters of the global decision device are updated using the optimization processing corresponding to the optimization method, so that the global decision loss is reduced.
[0177] Case 3
[0178] The fixing of parameters of any two models among the generator, the global decision maker and the pixel decision maker, and updating parameters of the remaining models, comprises:
[0179] The parameters of the generator and the global judge are fixed; the training image is subjected to resolution enhancement processing in the generator to obtain the reconstructed image; in the pixel judge, the reconstructed image and the original natural image are judged respectively to obtain a first probability matrix for judging the pixels of the reconstructed image as pixels of a non-natural image, and a second probability matrix for judging the pixels of the original natural image as pixels of a non-natural image; wherein the training image is obtained by downsampling the original natural image; a pixel judgment loss is obtained based on the first probability matrix and the second probability matrix, and the parameters of the pixel judge are updated based on the pixel judgment loss.
[0180] Here, the parameters of the generator and the pixel judger are fixed, and the generator performs convolution on the training image to obtain a reconstructed image with a resolution magnified by r times. The pixel judger judges the reconstructed image and the original natural image respectively to obtain a first probability matrix for judging the pixels of the reconstructed image as pixels of the non-natural image, and a second probability matrix for judging the pixels of the original natural image as pixels of the non-natural image.
[0181] The first probability matrix and the second probability matrix are calculated by formula (3) to obtain the pixel decision loss Loss_pixel_d of the pixel decision device.
[0182] Loss_pixel_d = ∑ - log(sum(D2(G(x i )))) + ∑ - log( sum(1-D2( y i )) )
[0183] Formula (3).
[0184] According to the pixel decision loss, the parameters of the pixel decision device are updated using the optimization processing corresponding to the optimization method, so that the pixel decision loss is reduced.
[0185] In the network training method provided by the embodiment of the present invention, the parameters of the global judge and the pixel judge are fixed, and the parameters of the generator are updated; the parameters of the generator and the pixel judge are fixed, and the parameters of the global judge are updated; the parameters of the generator and the global judge are fixed, and the parameters of the pixel judge are updated, thereby completing an update of the generative adversarial network, and performing multiple iterative updates until the convergence conditions of the generative adversarial network are met, and the updating of the parameters of the generator, the global judge and the pixel judge is stopped.
[0186] In an embodiment of the present invention, a training image is input into a generator to obtain a reconstructed image output by the generator, and the reconstructed image is input into a global judge and a pixel judge respectively to obtain a first probability output by the judge and a first probability matrix output by the pixel judge, and the perceptual loss is determined by the first probability output by the global judge and the first probability matrix output by the pixel judge, and the parameters of the generator are updated based on the perceptual loss, and the difference between the reconstructed image output by the generator and the original natural image is minimized by continuously updating the parameters of the generator, the global judge and the pixel judge.
[0187] In the embodiment of the present invention, Figure 2 The structure of the generator 201 shown can adopt the structure of super-resolution models such as SRCNN, fast super-resolution convolutional neural network (FSRCNN), sub-pixel convolutional neural network (ESPCN), residual super-resolution network (ResidualSuper-Resolution Network, ResSRNet). The embodiment of the present invention does not impose any limitation on the super-resolution model adopted by the generator.
[0188] The structure of SRCNN can be as follows Figure 5 As shown in the figure, bicubic interpolation is used to enlarge the training image to the size of the original natural image, and then three convolutional layers are used to fit the nonlinear mapping, and the high-resolution reconstructed image result is obtained by reconstructing the mapping result. The functions of the three convolutions are: image block extraction and feature representation, feature nonlinear mapping and final reconstruction. The sizes of the convolution kernels used in the three convolutional layers can be 9x9, 1x1 and 5x5 respectively, and the number of channels of the first two convolutional layers are 64 and 32 respectively.
[0189] FSRCNN can be divided into five parts. The first part: extract features from the training image through a convolution layer with a convolution kernel size of 5×5; the second part: reduce the dimension of the output of the first part through a convolution layer with a convolution kernel size of 1×1; the third part: perform non-phenomenal mapping on the output of the second part through multiple convolution layers with a convolution kernel size of 3×3 in series. The fourth part: expand the dimension of the output of the third part through a convolution layer with a convolution kernel size of 1×1, which is equivalent to the inverse process of contraction. The fifth part: upsample the output of the fourth part through a deconvolution layer, and enlarge the size of the output of the fourth part by r times according to the step size r of the convolution kernel of the deconvolution layer.
[0190] The network structure of ESPCN includes sub-pixel convolution layers. ESPCN convolves the training image in sequence through three convolution layers to obtain a channel number of r 2 The feature map of the same size as the training image, where r is the method multiple. Then r of each pixel in the feature map 2 The channels are rearranged into an r×r region, corresponding to an r×r sub-block in the high-resolution image, so that the feature map of size H×W is rearranged into a high-resolution image of rH×rW.
[0191] ResSRNet includes multiple residual blocks and two sub-pixel convolutional layers. Each residual block contains two convolutional layers with a kernel size of 3×3, followed by a batch normalization layer (BN) and PReLU as an activation function. Two sub-pixel convolutional layers are used to increase the feature size.
[0192] In some embodiments, the resolution enhancement processing of the training image to obtain the reconstructed image includes: convolving the training image to obtain a first feature map, convolving the first feature map to obtain a second feature map; fusing the first feature map and the second feature map to obtain a first fused feature map; rearranging pixels of the first fused feature map to obtain a rearranged feature map; and convolving the rearranged feature map to obtain the reconstructed image.
[0193] At this time, the structure of the generator is ResSRNet, which includes multiple cascaded convolutional layers. An addition layer is connected between the first convolutional layer and the second convolutional layer, and the first convolutional layer is connected to multiple residual blocks. The output of the residual block is input to the addition layer, and a sub-pixel layer, i.e., a pixel rearrangement layer, is connected before the last convolutional layer.
[0194] The first convolutional layer extracts features from the input training image to obtain a low-resolution image carrying low-frequency information, i.e., the first feature map. The first feature map is input into the residual block to obtain a residual image carrying the residual information between the low-frequency information and the high-frequency information of the training image, i.e., the second feature map. The first feature map and the second feature map are added to obtain the third feature map. The first feature map and the second feature map are fused through the addition layer to obtain the first fused feature map. The first fused feature map is input into the sub-pixel layer, the size of the first fused feature map is increased through the sub-pixel layer, and the rearranged feature map is convolved to obtain a reconstructed image with the same number of channels as the training image.
[0195] It should be noted that before the first fused feature map is input into the sub-pixel layer, feature extraction can be performed on the first fused feature map through multiple cascaded convolutional layers to obtain higher frequency information of the training image.
[0196] In an embodiment of the present invention, Figure 2 The structure of the global decision maker 202 shown may adopt the structure of classification models such as ResNet, VGG16, VGG19, etc. The embodiment of the present invention does not impose any limitation on the structure of the classification model adopted by the global decision maker.
[0197] In some embodiments, judging the reconstructed image to obtain a first probability that the reconstructed image is a non-natural image includes: convolving the reconstructed image to obtain at least two feature maps; fusing a third feature map and a fourth feature map of the at least two feature maps to obtain a second fused feature map; and judging based on the second fused feature map to obtain a first probability that the reconstructed image is a non-natural image.
[0198] Here, the reconstructed image is convolved to obtain at least two feature maps, and the features of the third feature map and the fourth feature map of the at least two features are fused to obtain a second fused feature map, wherein the third feature map and the fourth feature map may be feature maps with different frequencies of features, for example: the third feature map is a feature image carrying low-frequency information extracted by a low-level convolutional layer; the fourth feature map is a feature image carrying high-frequency information extracted by a high-level convolutional layer. The second fused feature map is input into a fully connected layer, and the extracted features are integrated through the fully connected layer to obtain local information with category distinction in the convolution layer, and the integrated features are discriminated through the activation function connected after the fully connected layer to obtain a first probability of discriminating the reconstructed image as an unnatural image.
[0199] Here, the reconstructed image can be convolved to obtain a third feature map, and the third feature map can be convolved to obtain a fourth feature map. At this time, the structure of the global decision device can be ResNet, including multiple cascaded convolutional layers and fully connected layers, and the input of any high-level convolutional layer includes the output of the previous convolutional layer and the output of any low-level convolutional layer, wherein at least one convolutional layer is included between the low-level convolutional layer and the high-level convolutional layer. For example: the input of the high-level convolutional layer 8 includes the output of the previous convolutional layer 7 and the output of the low-level convolutional layer 3. Here, an addition layer can be connected between the high-level convolutional layer and the previous convolutional layer, so that before the output of the low-level convolutional layer and the output of the previous convolutional layer of the high-level convolutional layer are input into the high-level convolutional layer, the output of the low-level convolutional layer and the output of the previous convolutional layer of the high-level convolutional layer are added through the addition layer, and the addition result is input into the high-level convolutional layer.
[0200] In an embodiment of the present invention, Figure 2 The structure of the pixel decision device 203 shown can be a pixel classification model with the same size of input and output images, such as a series of fully convolutional neural networks. The embodiment of the present invention does not impose any limitation on the structure of the pixel classification model used by the pixel decision device.
[0201] In some embodiments, judging the reconstructed image to obtain a first probability matrix for judging pixels of the reconstructed image as pixels of a non-natural image includes: convolving the reconstructed image to obtain a fifth feature map of the same size as the reconstructed image; judging each pixel in the fifth feature map to obtain a first probability matrix for judging each pixel in the fifth feature map as a pixel of a non-natural image.
[0202] At this time, the structure of the pixel decision device can be a series-connected full convolutional network, including multiple cascaded convolutional layers, where the number of channels of the last convolutional layer is the same as the number of channels of the convolution of the input image, for example, the number of channels is 3 at the same time.
[0203] The reconstructed image is feature extracted through multiple cascaded convolutional layers to obtain a feature map that is the same size as the reconstructed image and carries the pixel-level features of the reconstructed image, i.e., the fifth feature map. Each pixel in the fifth feature map is judged through an activation function connected after the last convolutional layer to obtain the probability of judging the feature corresponding to each pixel in the fifth feature map as a pixel of a non-natural image, and the first probability matrix is formed by the probabilities of each pixel.
[0204] It should be noted that, in the embodiment of the present invention, the pixel determiner classifies the extracted low-level pixel-level features, and the low-level pixel-level features can be identified without a complex network. Therefore, the pixel determiner does not need a very deep and complex network structure.
[0205] When the parameters of the global discriminator and the pixel discriminator are fixed and the parameters of the generator are updated, determining the perceptual loss based on the global adversarial loss and the pixel adversarial loss may include:
[0206] Extract features from the training image and the original natural image respectively; determine the content loss corresponding to the features of the training image and the features of the original natural image based on the mean square error loss function; and weight the content loss, the global adversarial loss and the pixel adversarial loss to obtain the perceptual loss.
[0207] Here, when extracting features from training images and original natural images, high-level features can be extracted from the training images and original natural images through a VGG network, wherein the high-level features are features output by a convolutional layer that meets a specific depth.
[0208] The calculation formula of content loss Loss_content is shown in formula (6):
[0209] Loss_content = ∑ ( VGG(G(x i )) - VGG(y i ) ) 2 Formula (6);
[0210] Among them, VGG(G(x i )) is the reconstructed image G(x) using the VGG network i ) for high-level features of feature extraction; VGG(y i ) is to use the VGG network to process the original natural image y i High-level features for feature extraction.
[0211] Here, the VGG network can be a high-level feature extraction network such as VGG16, VGG19, etc., in which the input image y is extracted i When the high-level features are used, the network structure of the VGG network may not include the last layer classifier in VGG16 and VGG19.
[0212] At this time, the global adversarial loss, the pixel adversarial loss and the content loss are weighted and summed based on the weights corresponding to the global adversarial loss, the pixel adversarial loss and the content loss. In an embodiment of the present invention, the weights corresponding to the global adversarial loss, the pixel adversarial loss and the content loss may be the same or different, for example, the corresponding weights are 2e-6, 1e-3 and 1e-3 respectively.
[0213] exist Figure 2 Based on the GAN shown in FIG. 1 , the training method and method of the generative adversarial network of the embodiment of the present invention can be applied to Figure 6 The GAN shown in Figure 6 As shown, it includes four convolutional neural networks: generator 201, global decision maker 202, pixel decision maker 203 and supervisor 204. Supervisor 204 is used to judge whether there is a difference in image information such as color volume and brightness between the unnatural image generated by the generator and the original natural image in the local area. Generator 201, global decision maker 202 and pixel decision maker 203 are trained alternately to compete with each other and obtain the best parameters of GAN.
[0214] based on Figure 6 The structure shown in the figure weights the content loss, the global adversarial loss and the pixel adversarial loss to obtain the perceptual loss, including: in the supervisor of the generative adversarial network, respectively average pooling the reconstructed image and the original natural image to obtain a first downsampled image corresponding to the reconstructed image and a second downsampled image corresponding to the original natural image; determining the downsampled loss based on the first downsampled image and the second downsampled image; weighting the content loss, the global adversarial loss, the pixel adversarial loss and the downsampled loss to obtain the perceptual loss.
[0215] The supervisor includes a mean pooling layer, and the size and step size of the convolution kernel of the mean pooling layer are the same as the resolution magnification of the generator. For example: when the resolution magnification of the generator is 4, the convolution kernel used by the mean pooling layer is (4,4) with a step size of 4. For another example: when the resolution magnification of the generator is 3, the convolution kernel used by the mean pooling layer is (3,3) with a step size of 3.
[0216] After determining the first downsampled image and the second downsampled image, the first downsampled image and the second downsampled image are calculated by formula (7) to obtain the downsampling loss Loss_downscale of the supervisor,
[0217] Loss_downscale = ∑ ( R(G(x i )) - R(y i ) ) 2 Formula (7);
[0218] Among them, R(G(x i ) is the supervisory image reconstructed by G(x i ) is downsampled to obtain the first downsampled image, R(y i ) is the supervisory model for the original natural image y i The second down-sampled image is obtained by down-sampling.
[0219] It should be noted that the downsampling loss can supervise the reconstructed image and the original training image generated by the generator, and there is no obvious brightness and color change in the corresponding local area. There is no limitation on the loss function used to determine the downsampling loss corresponding to the first downsampled image and the second downsampled image.
[0220] Based on the weights corresponding to the content loss, the global adversarial loss, the pixel adversarial loss and the downsampling loss, the content loss, the global adversarial loss, the pixel adversarial loss and the downsampling loss are weighted and summed to obtain the perceptual loss. In an embodiment of the present invention, the weights corresponding to the content loss, the global adversarial loss, the pixel adversarial loss and the downsampling loss may be the same or different, for example, the corresponding weights are 2e-6, 1e-3, 1e-3 and 1 respectively.
[0221] The training method for a generative adversarial network provided in an embodiment of the present invention uses a supervisor to monitor whether there are obvious brightness and color changes in corresponding local areas between the reconstructed image generated by the generator and the training image, thereby ensuring that the GAN only improves the resolution of the low-resolution image without changing the brightness or color of the image.
[0222] based on Figure 2 or Figure 6 The GAN shown in the embodiment of the present invention provides an image processing method such as Figure 7 As shown, it will be combined Figure 7 The steps shown are explained.
[0223] S701: Obtain an image to be processed.
[0224] When a low-resolution image to be processed needs to be super-resolution-enhanced, a super-resolution request can be initiated. Here, the super-resolution request can be initiated based on the client or the server. The super-resolution request for improving the resolution of the image to be processed can be triggered by the image processing app.
[0225] When a super-resolution request is triggered on the client, the client receives the super-resolution request and directly processes the request according to the location where the GAN is deployed, or sends the super-resolution request to the server so that the server processes the super-resolution request.
[0226] S702: In a generator of a generative adversarial network, convolution processing is performed on the image to be processed to obtain a target image corresponding to the image to be processed.
[0227] The generator is obtained by the following processing based on the global judger and the pixel judger in the generative adversarial network:
[0228] Initializing the generator, the global judger and the pixel judger; updating the parameters of the generator, the global judger and the pixel judger until the convergence condition of the generative adversarial network is met.
[0229] Here, the structure of GAN can be as follows Figure 2 or Figure 6 As shown in the figure, when processing a super-resolution request, the image to be processed corresponding to the super-resolution request is input into the generator. In the generator, the image to be processed is received, and convolution processing is performed on the image to be processed to improve the resolution of the image to be processed, thereby obtaining a target image. Here, when the size of the image to be processed is H×W, the size of the target image output by the generator is rH×rW, where r is the resolution magnification of the generator.
[0230] In the embodiment of the present invention, the structure of the generator may adopt the structure of super-resolution models such as SRCNN, FSRCNN, ESPCN, ResSRNet, etc. The embodiment of the present invention does not impose any limitation on the super-resolution model adopted by the generator.
[0231] In some embodiments, performing convolution processing on the image to be processed to obtain a target image corresponding to the image to be processed includes:
[0232] Convolving the image to be processed to obtain a sixth feature map, convolving the sixth feature map to obtain a seventh feature map; fusing the sixth feature map and the seventh feature map to obtain a third fused feature map; rearranging pixels of the third fused feature map to obtain a rearranged feature map; convolving the rearranged feature map to obtain the target image.
[0233] At this time, the structure of the generator can be ResSRNet, including multiple cascaded convolutional layers, an addition layer is connected between the first convolutional layer and the second convolutional layer, and multiple residual blocks are connected to the first convolutional layer. The output of the residual block is connected to the addition layer, and a sub-pixel layer, i.e., a pixel rearrangement layer, is connected before the last convolutional layer.
[0234] The training of the generator can be specifically referred to the training method of the above-mentioned generative adversarial network, which will not be described in detail here.
[0235] In the image processing method provided by the embodiment of the present invention, after obtaining GAN through the above-mentioned generative adversarial network training method, the resolution of the image to be processed is improved based on the generator in the GAN to obtain a target image with enlarged resolution, while automatically supplementing the image details, avoiding the existence of noise, rough edges and other problems in the target image.
[0236] In practical applications, the image processing method provided by the embodiment of the present invention can be applied to super-resolution scenarios of video cover images, video screenshots, user screenshots, shared screenshots, avatars, etc. Taking the video cover image as an example, for a cover image with too low resolution, the image processing method provided by the embodiment of the present invention can improve the resolution of the cover image and improve the user's visual experience.
[0237] It should be noted that when it is necessary to display the stored image to be processed, the image processing method provided by the embodiment of the present invention performs super-resolution processing on the image to be processed, that is, resolution enhancement processing, and after obtaining the target image, the target image is displayed. Thereby, while providing the user with a visual experience, the space required for image storage is reduced. Here, taking 4 times super-resolution, that is, the resolution method multiple of the generator is 4 times as an example, when storing the image, the low-resolution image is stored, that is, compressed storage. When decompression is required, the image is super-resolutioned 4 times to obtain the decompressed super-resolution image. At this time, the area of the low-resolution image is 1 / 16 of the area of the super-resolution image, which can effectively reduce the resources required for storing or transmitting the image. Taking 100 PNG images (2040x1356) with a decompressed image of 420MB as an example, before super-resolution, the image volume is only 30MB (PNG picture, 508x339), saving 92.86% of the volume, which can greatly save the traffic and bandwidth used for data transmission.
[0238] In the related art, the network model for super-resolution processing of images can use SRCNN with fixed magnification and Figure 8 SRGAN shown.
[0239] In the process of training SRCNN with fixed magnification, the existing high-resolution image is artificially downsampled to obtain a low-resolution image with a fixed magnification, and then the low-resolution image is interpolated to obtain a super-resolution image after cubic interpolation. The super-resolution model is obtained by training the mapping relationship between the super-resolution image after cubic interpolation and the high-resolution image. For example, when SRCNN is required to perform 2x super-resolution, an SRCNN model that processes 2x cubic interpolation is trained, the image to be super-resolved is interpolated 2x cubically, and the interpolated result is input into the SRCNN model that processes 2x cubic interpolation to obtain the super-resolution result.
[0240] However, the loss function of SRCNN is an averaged super-resolution mapping. Therefore, the image processing method using SRCNN for super-resolution cannot restore the random details of the image.
[0241] The network structure of SRGAN is as follows Figure 8As shown, it includes a generator 801 and a judger 802. The generator 801 and the judger 802 are deep neural network models.
[0242] Generator 801, which can be referred to as G, is a super-resolution model, whose input is a low-resolution image x i , the output is a super-resolution image G(x i ).
[0243] The judgement device 802, which can be referred to as D, is used to distinguish whether the input image is generated by the generator. It inputs a high-resolution image z and outputs a number D(z) in the range of 0-1. D(z) is used to determine whether the input image is generated by the generator, where 0 means no and 1 means yes.
[0244] Here, the low-resolution image is x i , x i The corresponding high-resolution image is y i , the generator is based on x i The generated super-resolution image is G(x i ).
[0245] The super-resolution G(x i ) or high-resolution image y i Input D, and the image obtained by D is G(x i ), or the high-resolution image y in the database i When the losses of the generator 801 and the discriminator 802 no longer decrease significantly, a usable super-resolution model can be obtained.
[0246] The loss function for training the generator G includes the following three loss functions:
[0247] a. Content loss Loss_content:
[0248] Content loss is a loss function based on the mean square error. The VGG model is used to extract features except for the last layer of the network structure. The feature extracted is recorded as VGG(x), where x is the input image.
[0249] Content loss is the network structure of the VGG model except the last layer for high-resolution image y i With super-resolution G(x i ) is the square error of the feature after feature extraction, and the calculation formula is formula (8),
[0250] Loss_content = ∑ ( VGG(G(x i )) - VGG(y i ) ) 2 Formula (8).
[0251] b. Mean square error loss (minimum square loss) Loss_mse
[0252] The minimum square loss is the generated super-resolution image G(x i ) and high-resolution image y i The square error is calculated as follows:
[0253] Loss_mse =∑ ( G(x i ) - y i ) 2 Formula (9).
[0254] c. Adversarial loss loss_gan
[0255] Based on the probability definition of the training image on the discriminator, we try to make the data generated by the generator deceive the discriminator. The calculation formula is formula (10):
[0256] Loss_gan = ∑ - logD(G(x i )) Formula (10).
[0257] The calculation formula of the loss function loss_d used in training the discriminator D is formula (11),
[0258] loss_d = - ∑ log( D(G(x i )) ) - ∑ log( D(y i ) ) formula (11).
[0259] train Figure 8 The SRGAN method shown includes:
[0260] S1. Use Loss_mse to train the generator G and initialize the parameters of the generator until Loss_mse converges.
[0261] S2, based on G(x generated by generator G i ), and the real high-resolution image y i , use loss_d to train the discriminator D and initialize the parameters of the discriminator D until loss_d converges.
[0262] S3. Fix the parameters of the discriminator D, and use the weighted sum of loss_content, loss_mse, and loss_gan, i.e., the perceptual loss, to update the parameters of the generator G. The weights of the three loss functions can be 1e-6, 1, and 1e-3, respectively.
[0263] S4. Fix the parameters of the generator G, and update the parameters of the discriminator D once using loss_d, so that the discriminator D can further distinguish the super-resolution image G(x i ) generated by the generator G from the high-resolution image y i .
[0264] S5. Repeat S3 and S4 until the weighted sum of Loss_content, Loss_mse, and Loss_gan converges.
[0265] However, in some scenarios, the super-resolution images generated by super-resolving images through SRGAN are as Figure 1 shown, there will be obvious distorted noise at the 101 position, and the edges are rough, affecting the image effect.
[0266] To overcome the problem that the above-mentioned SRCNN cannot restore the random details of the image and the rough edges of the super-resolution images generated by SRGAN, in the embodiments of the present invention, a generative adversarial network as Figure 6 shown is proposed, including: a generator 201, a global discriminator 202, a pixel discriminator 203, and a supervisor 204. Below, each part will be described separately.
[0267] Generator 201
[0268] The generator G is a convolutional neural network, and its network structure can be as Fig. 9 shown, adopting a fully convolutional residual neural network, which sequentially includes 4 convolutional layers: convolutional layer 1 to convolutional layer 4. Among them, between convolutional layer 1 and convolutional layer 2, there are 17 cascaded residual blocks and an addition layer. The output of the 17th residual block and the output of convolutional layer 1 are used as the inputs of the adder. A pixel rearrangement layer is connected between convolutional layer 3 and convolutional layer 4.
[0269] The working process of the generator G is as follows:
[0270] Use a convolutional layer 1 to perform preliminary feature extraction on the input image to obtain the preliminarily extracted features. Adopt cascaded residual blocks 1 to 17 to continuously perform feature extraction on the preliminarily extracted features and supplement features and textures to the output of the previous residual block. The output of the 17th residual module is supplemented to the preliminarily extracted features through the addition layer. Use cascaded convolutional layer 2 and convolutional layer 3 - pixel rearrangement layer - convolutional layer 4 to magnify the resolution of the image output by the addition layer to the specified multiple of super-resolution to obtain the super-resolution image.
[0271] The network structure of the jth residual block can be as Fig.10 shown, including cascaded convolutional layer j1, convolutional layer j2, and addition layer j1. Where 0 < j ≤ 17. The addition layer j1 adds the corresponding elements of the input of the jth residual block and the output of convolutional layer j2.
[0272] The pixel rearrangement layer arranges the pixels differently for different magnifications. Taking 4x super-resolution as an example, the pixel rearrangement layer rearranges every 16 channels into a large image to increase the size of the image. When rearranging pixels, the pixel rearrangement layer arranges the 16 channel pixels with the same corresponding rows and columns into a 4x4 pixel block. After all pixels are arranged into 4x4 pixel blocks, all pixel blocks are pieced together into a large image, thereby turning 1 pixel into 4x4, or 16 pixels. Taking 3x super-resolution as an example, the pixel rearrangement layer arranges every 9 channels into a large image.
[0273] In the generator G, the high-level features passing through the residual block and the low-level features output by the convolutional layer 1 or the previous residual block are added to improve the super-resolution effect of small objects to a certain extent.
[0274] When the method magnification of the generator is 4 times, the network parameters of the generator can be shown in Table 1. Among them, the low-resolution image input to the convolution layer 1 is a 3-channel image including RGB channels, and the super-resolution image output by the convolution layer 4 is a 3-channel image including RGB channels. The number of channels of the pixel rearrangement layer is the square of the magnification.
[0275] Table 1 Network parameter examples of the quadruple super-resolution generator G
[0276]
[0277] Global decision maker 202
[0278] The global decision maker D1 is used to promote the super-resolution image generated by the generator G to be as similar as possible to the real high-resolution image at the global level. The global decision maker D1 is a convolutional neural network, and the network structure can be shown as follows: Fig.11 As shown: it includes 11 convolutional layers, an addition layer and a fully connected layer; the addition layer is located between the convolutional layer 11 and the fully connected layer, and is used to add the output of the convolutional layer 8 to the output of the convolutional layer 11, and input it into the subsequent fully connected layer, and predict the probability that the input image is a real high-resolution image through the fully connected layer and the activation function connected to the fully connected layer, and judge whether the input image is a generator structure according to the output probability. The network parameters of the global decision device D1 can be shown in Table 2.
[0279] Table 2 Network parameter examples of global decision maker D1
[0280]
[0281] Pixel decision device 203
[0282] The pixel determiner D2 may also be referred to as the local determiner D2, which is used to promote the super-resolution image generated by the generator G at the local pixel level, so that each generated pixel is as close as possible to the pixel in the real high-resolution image. For example, the pixels on the edge should be natural, smooth and not rough like the real high-resolution image, and there should be no noise.
[0283] The local decision maker D2 is a series of fully convolutional neural networks, and the network structure is as follows: Fig.12 As shown, it is a 5-layer series-connected full convolutional neural network, including convolutional layer 1 to convolutional layer 5. Among them, the input and output of the local decision device D2 are the same in size. The network parameters of the local decision device D2 can be shown in Table 3.
[0284] Table 3 Network parameter examples of local decision maker D2
[0285]
[0286] Supervisor 204
[0287] The supervisor R is used to supervise the super-resolution image generated by the generator G and the original low-resolution image. In the corresponding local areas, there is no obvious brightness or color change, ensuring that the super-resolution model only improves the resolution of the low-resolution image without changing the brightness or color of the image.
[0288] The network structure of the supervisor R can be as follows Fig.13 As shown in Figure 1, it is a mean pooling layer, the convolution kernel size and step size of the mean pooling layer are both super-resolution magnifications, so as to obtain an image with the same resolution size as the original image, and the filling method is the same. The super-resolution image is downsampled through the supervisor R to obtain a low-resolution image based on the downsampling of the super-resolution image.
[0289] based on Figure 6 The generative adversarial network shown in the figure, an embodiment of the present invention provides a training method for a generative adversarial network, wherein the loss function involved may include the following seven loss functions:
[0290] a. Content loss:
[0291] Loss_content = ∑ ( VGG(G(x i )) - VGG(y i ) ) 2
[0292] b. Downsampling loss:
[0293] Loss_downscale = ∑ ( R(G(x i )) - R(y i ) ) 2
[0294] c. Global adversarial loss:
[0295] Loss_global_gan = ∑ - log(D1(G(x i )))
[0296] d. Pixel adversarial loss:
[0297] Loss_pixel_gan = ∑ - log(sum(D2(G(x i ))))
[0298] e. Least square loss:
[0299] Loss_mse =∑ ( G(x i ) - y i ) 2
[0300] f. Global decision loss:
[0301] Loss_global_d = ∑ - log(D1(G(x i ))) + ∑ - log( 1-D1( y i ) )
[0302] g. Pixel decision loss
[0303] Loss_pixel_d = ∑ - log(sum(D2(G(x i )))) + ∑ - log( sum(1-D2( y i )) )
[0304] The training method of the generative adversarial network based on the above seven loss functions is as follows Fig.14 As shown, including:
[0305] S1401. Initialize the generator G through the least square loss.
[0306] Use the minimum square loss Loss_mse to train the generator G and initialize the parameters of the generator G. Use the optimization method to continuously update the parameters of the generator G to reduce the value of Loss_mse.
[0307] The parameters of the generator G are iteratively updated, and the difference between two consecutive Loss_mse is calculated. When the difference between two consecutive Loss_mse is less than the set threshold α, it is determined that the Loss_mse decrease value is lower than the set threshold α, and the updating of the parameters of the generator G is stopped to obtain the initialized generator G.
[0308] S1402 , initializing the global decision device D1 through the global decision loss.
[0309] Initialize the global decision maker D1, and continuously update the parameters of the global decision maker D1 through the global decision loss Loss_global_d to reduce the value of Loss_global_d, so that D1 can distinguish the super-resolution image constructed by G from the real high-resolution image as much as possible.
[0310] The parameters of the global decision device D1 are iteratively updated, and the difference between two consecutive Loss_global_d is calculated. When the difference between two consecutive Loss_global_d is less than the set threshold β, it is determined that the Loss_global_d decrease value is lower than the set threshold β, and the updating of the parameters of the global decision device D1 is stopped to obtain the initialized global decision device D1.
[0311] S1403, initializing the local decision device D2 through pixel decision loss.
[0312] Initialize the local discriminator D2, and continuously update the parameters of the local discriminator D2 through the pixel decision loss Loss_pixel_d to reduce the value of Loss_pixel_d, so that the local discriminator D2 can distinguish the super-resolution pixels constructed by the generator G from the real high-resolution pixels as much as possible, that is, when a super-resolution image is input, the local discriminator D2 outputs a matrix of all 1s (pixel probability matrix) with the same size as the super-resolution image; when a real high-resolution image is input, the local discriminator D2 outputs a matrix of all zeros with the same size as the high-resolution image.
[0313] The parameters of the local decision device D2 are iteratively updated, and the difference between two consecutive Loss_pixel_d is calculated. When the difference between two consecutive Loss_pixel_d is less than the set threshold γ, it is determined that the Loss_pixel_d drop value is lower than the set threshold γ, and the updating of the parameters of the local decision device D2 is stopped to obtain the initialized local decision device D2.
[0314] S1404, fix the parameters of the global decision maker D1 and the local decision maker D2, obtain the perceptual loss according to the content loss, downsampling loss, global adversarial loss and pixel adversarial loss, and optimize the generator G through the perceptual loss.
[0315] The parameters of the global decision maker D1 and the local decision maker D2 are fixed, and the parameters of the generator G are updated once using the optimization method, so that the weighted sum of the content loss Loss_content, the downsampling loss Loss_downscale, the global adversarial loss Loss_global_gan, and the pixel adversarial loss Loss_pixel_gan, i.e., the perceptual loss, is reduced. Among them, when weighting Loss_content, Loss_downscale, Loss_global_gan, and Loss_pixel_gan, the corresponding weights can be: 2e-6, 1, 1e-3, and 1e-3, respectively.
[0316] In practical applications, when weighting loss_content, loss_downscale, loss_global_gan, and loss_pixel_gan to determine the perceptual loss, the corresponding weights are not limited in any way and can be set according to actual needs.
[0317] S1405, fixing the parameters of the generator G and the local decision maker D2, and optimizing the global decision maker D1 through the global decision loss.
[0318] The parameters of the generator G and the local decision maker D2 are fixed, and the parameters of the global decision maker D1 are updated once using the optimization method to reduce Loss_global_d.
[0319] S1406, fixing the parameters of the generator G and the global decision maker D1, and optimizing the local decision maker D2 through the pixel decision loss.
[0320] The parameters of the generator G and the global decision maker D1 are fixed, and the parameters of the local decision maker D2 are updated once using the optimization method to reduce Loss_pixel_d.
[0321] S1407. Determine whether the decrease value of the perceptual loss is less than the threshold ε, whether the decrease value of the global decision loss is less than the threshold β, and whether the decrease value of the pixel decision loss is less than the threshold γ?
[0322] If the decrease value of the perceptual loss is less than the threshold ε, the decrease value of the global decision loss is less than the threshold β, and the decrease value of the pixel decision loss is less than the threshold, that is, the perceptual loss, global decision loss and pixel loss all converge, then the process ends and the generator G is obtained.
[0323] If the condition that the decrease value of the perceptual loss is less than the threshold ε, the decrease value of the global decision loss is less than the threshold β, and the decrease value of the pixel decision loss is less than the threshold γ is not satisfied, then S24 is executed until the decrease value of the perceptual loss is less than the threshold ε, the decrease value of the global decision loss is less than the threshold β, and the decrease value of the pixel decision loss is less than the threshold γ, that is, the perceptual loss, the global decision loss and the pixel loss all converge, then the process ends and the generator G is obtained.
[0324] In practical applications, the values of the threshold α, the threshold β, the threshold γ and the threshold ε can be set according to actual needs.
[0325] It should be noted that the network structure of the generator in the embodiment of the present invention is not limited to Fig. 9 The structure shown in the figure, the structure of the global decision device D1 is not limited to Fig.11 The structure shown in the figure, the structure of the local decision device D2 is not limited to Fig.12 The structure shown can be set according to actual needs.
[0326] pass Fig.14 The training method of the generative adversarial network shown is Figure 6 The GAN shown in the figure is trained, and the super-resolution image generated by the trained GAN has the advantages of being able to supplement high-frequency details, weak noise, and smooth edges compared to the super-resolution images generated by the interpolation method, SRCNN, and SRGAN, as shown in Table 4. Among them, in Table 4, method 1 is an interpolation method, method 2 is an image processing method based on SRCNN, method 3 is an image processing method based on SRGAN, and method 4 is an image processing method based on Figure 6 The image processing method of GAN shown.
[0327] Table 4 Comparison of indicators of super-resolution images generated by various methods based on low-resolution images
[0328]
[0329] right Fig.15 When processing the A image in Fig.15 The B image in is a super-resolution image obtained by the generator generated by the network training method of the present invention, Fig.15 The C image in the figure is obtained by Figure 8 The super-resolution image obtained by SRGAN is shown. Fig.15 The comparison of the image quality at position 1501 and position 1502 shows that Fig.15 The B-image in the image is cleaner and has no jagged noise.
[0330] right Fig.16 When processing the A image in Fig.16The B image in is a super-resolution image obtained by the generator generated by the network training method of the present invention, Fig.16 The C image in the figure is obtained by Figure 8 The super-resolution image obtained by SRGAN is shown. Fig.16 The comparison of the image quality at position 1601 and position 1602 shows that Fig.16 The B image in the image has no jagged noise and the edges are smoother.
[0331] It can be seen that the super-resolution image obtained by the training method of the generative adversarial network provided by the embodiment of the present invention suppresses Figure 8 The noise existing in SRGAN shown avoids the problem of rough edges, reduces jagged noise, makes edges smoother, and ensures that there will be no obvious changes in hue and brightness in the corresponding local area after super-resolution, thereby improving the image quality of the super-resolution algorithm and improving the user's image quality experience.
[0332] The embodiment of the present invention also provides a training device for generating an adversarial network, such as Fig.17 As shown, the device comprises:
[0333] An initial generation unit 1701, used to initialize the generator in the generative adversarial network;
[0334] A loss determining unit 1702, configured to obtain a global decision loss and a pixel decision loss based on the generator;
[0335] An initial global unit 1703, configured to initialize a global decision device in the generative adversarial network based on the global decision loss;
[0336] An initial pixel unit 1704, configured to initialize a pixel decision device in the generative adversarial network based on the pixel decision loss;
[0337] The updating unit 1705 is used to fix the parameters of any two models among the generator, the global judge and the pixel judge, and update the parameters of the remaining models until the convergence condition of the generative adversarial network is met.
[0338] The embodiment of the present invention also provides an image processing device, such as Fig.18 As shown, the device comprises:
[0339] An acquisition unit 1801 is used to acquire an image to be processed;
[0340] The super-resolution unit 1802 is used to perform convolution processing on the image to be processed in the generator of the generative adversarial network to obtain a target image corresponding to the image to be processed;
[0341] The generator is obtained by the following processing based on the global judge and the pixel judge in the generative adversarial network:
[0342] Initializing the generator, the global determiner and the pixel determiner;
[0343] The parameters of the generator, the global judger and the pixel judger are updated until the convergence condition of the generative adversarial network is met.
[0344] An embodiment of the present invention further provides a training device for generating an adversarial network, the device comprising: a memory, a communication bus, and a processor, wherein:
[0345] The memory is used to store computer programs;
[0346] The communication bus is used to realize the connection and communication between the processor and the memory;
[0347] The processor is used to implement the steps of the above-mentioned training method for generating an adversarial network when executing the computer program stored in the memory.
[0348] The embodiment of the present invention further provides an image processing device, the device comprising: a memory, a communication bus and a processor, wherein:
[0349] The memory is used to store computer programs;
[0350] The communication bus is used to realize the connection and communication between the processor and the memory;
[0351] The processor is used to implement the steps of the above-mentioned image processing method when executing the computer program stored in the memory.
[0352] Fig.19 A schematic diagram of the hardware structure of a training device or image processing device for generating an adversarial network according to an embodiment of the present invention is shown in FIG. Fig.19 As shown, the device 1900 may include: a processor 1901, at least one communication bus 1902, at least one external communication interface 1903 and a memory 1904. The communication bus 1902 is configured to achieve connection and communication between these components. The external communication interface 1903 may include a standard wired interface and a wireless interface.
[0353] In practical applications, the training device for generating an adversarial network and the image processing device may be the same physical entity or different physical entities.
[0354] An embodiment of the present invention further provides a storage medium, namely, a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the above-mentioned training method of generating an adversarial network is implemented.
[0355] An embodiment of the present invention further provides a storage medium, namely, a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the above-mentioned image processing method is implemented.
[0356] The description of the above device embodiment and computer-readable storage medium embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment and computer-readable storage medium embodiment of the present invention, please refer to the description of the method embodiment of the present invention for understanding.
[0357] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A training method for generating an adversarial network, characterized in that: The method comprises: Initialize the generator in the generative adversarial network; Obtaining a global decision loss based on the generator; Determining pixel differences between an original natural image and a reconstructed image output by the generator, the pixel differences comprising a first initial probability matrix of pixels of the reconstructed image being determined as pixels of a non-natural image and a second initial probability matrix of pixels of the original natural image being determined as pixels of a non-natural image; Obtaining a pixel decision loss based on the first initial probability matrix and the second initial probability matrix; Initialize a global decision device in the generative adversarial network based on the global decision loss; Initialize a pixel decision device in the generative adversarial network based on the pixel decision loss; The parameters of any two models among the generator, the global judger and the pixel judger are fixed, and the parameters of the remaining models are updated until the convergence condition of the generative adversarial network is met.
2. The method according to claim 1, characterized in that The obtaining of a global decision loss based on the generator comprises: The global decision loss is determined based on a global difference between an original natural image and a reconstructed image output by the generator.
3. The method according to claim 2, characterized in that The determining the global decision loss based on the global difference between the original natural image and the reconstructed image output by the generator comprises: Performing resolution enhancement processing on the training image in the generator to obtain the reconstructed image; The reconstructed image and the original natural image are judged in the global judger to obtain a first initial probability of judging the reconstructed image as a non-natural image and a second initial probability of judging the original natural image as a non-natural image; wherein the training image is obtained by downsampling the original natural image; The global decision loss is obtained based on the first initial probability and the second initial probability.
4. The method according to claim 1, characterized in that: The determining of the pixel difference between the original natural image and the reconstructed image output by the generator, and obtaining the pixel decision loss based on the first initial probability matrix and the second initial probability matrix, comprises: Performing resolution enhancement processing on the training image in the generator to obtain the reconstructed image; The reconstructed image and the original natural image are judged in the pixel judger to obtain the first initial probability matrix for judging the pixels of the reconstructed image as pixels of a non-natural image, and the second initial probability matrix for judging the pixels of the original natural image as pixels of a non-natural image; wherein the training image is obtained by downsampling the original natural image.
5. The method according to any one of claims 1 to 4, characterized in that The fixing of parameters of any two models among the generator, the global decision maker and the pixel decision maker, and updating parameters of the remaining models, comprises: Fixing parameters of the global determiner and the pixel determiner; Performing resolution enhancement processing on the training image in the generator to obtain the reconstructed image; In the global determiner and the pixel determiner, the reconstructed image is determined respectively to obtain a first probability of determining the reconstructed image as an unnatural image and a first probability matrix of determining pixels of the reconstructed image as pixels of the unnatural image; Determine a global adversarial loss based on the first probability, determine a pixel adversarial loss based on the first probability matrix, and determine a perceptual loss based on the global adversarial loss and the pixel adversarial loss; Parameters of the generator are updated based on the perceptual loss.
6. The method according to any one of claims 1 to 4, characterized in that The fixing of parameters of any two models among the generator, the global decision maker and the pixel decision maker, and updating parameters of the remaining models, comprises: Fixing parameters of the generator and the pixel determiner; Performing resolution enhancement processing on the training image in the generator to obtain the reconstructed image; The reconstructed image and the original natural image are judged in the global judger respectively to obtain a first probability of judging the reconstructed image as a non-natural image and a second probability of judging the original natural image as a non-natural image; wherein the training image is obtained by downsampling the original natural image; A global decision loss is obtained based on the first probability and the second probability, and parameters of the global decision device are updated based on the global decision loss.
7. The method according to any one of claims 1 to 4, characterized in that The fixing of parameters of any two models among the generator, the global decision maker and the pixel decision maker, and updating parameters of the remaining models, comprises: Fixing parameters of the generator and the global decider; Performing resolution enhancement processing on the training image in the generator to obtain the reconstructed image; In the pixel decision device, the reconstructed image and the original natural image are respectively decided to obtain a first probability matrix for deciding pixels of the reconstructed image as pixels of a non-natural image, and a second probability matrix for deciding pixels of the original natural image as pixels of a non-natural image; wherein the training image is obtained by downsampling the original natural image; A pixel decision loss is obtained based on the first probability matrix and the second probability matrix, and parameters of the pixel decider are updated based on the pixel decision loss.
8. The method according to claim 5, characterized in that The determining of the perceptual loss based on the global adversarial loss and the pixel adversarial loss comprises: Extracting features from the training image and the original natural image respectively; Determining content losses corresponding to features of the training image and features of the original natural image based on a mean square error loss function; The content loss, the global adversarial loss and the pixel adversarial loss are weighted to obtain the perceptual loss.
9. The method according to claim 8, characterized in that The step of weighting the content loss, the global adversarial loss, and the pixel adversarial loss to obtain the perceptual loss includes: In the supervisor of the generative adversarial network, average pooling is performed on the reconstructed image and the original natural image respectively to obtain a first downsampled image corresponding to the reconstructed image and a second downsampled image corresponding to the original natural image; determining a downsampling loss based on the first downsampled image and the second downsampled image; The content loss, the global adversarial loss, the pixel adversarial loss and the downsampling loss are weighted to obtain the perceptual loss.
10. An image processing method, characterized in that: The method comprises: Get the image to be processed; In a generator of a generative adversarial network, convolution processing is performed on the image to be processed to obtain a target image corresponding to the image to be processed; The generator is obtained by the following processing based on the global judger and the pixel judger in the generative adversarial network: Initializing the generator, the global determiner and the pixel determiner; The pixel decision device is initialized by a pixel decision loss, the pixel decision loss is determined based on a pixel difference between an original natural image and a reconstructed image output by the generator, the pixel difference comprising a first initial probability matrix of pixels of the reconstructed image being determined as pixels of a non-natural image, and a second initial probability matrix of pixels of the original natural image being determined as pixels of a non-natural image; The parameters of the generator, the global judger and the pixel judger are updated until the convergence condition of the generative adversarial network is met.
11. A training device for generating an adversarial network, characterized in that: The device comprises: The initial generation unit is used to initialize the generator in the generative adversarial network; A loss determination unit is used to obtain a global decision loss based on the generator; determine a pixel difference between an original natural image and a reconstructed image output by the generator, wherein the pixel difference includes a first initial probability matrix of pixels of the reconstructed image being determined as pixels of a non-natural image, and a second initial probability matrix of pixels of the original natural image being determined as pixels of a non-natural image; and obtain a pixel decision loss based on the first initial probability matrix and the second initial probability matrix; An initial global unit, configured to initialize a global decision maker in the generative adversarial network based on the global decision loss; An initial pixel unit, used to initialize a pixel decision device in the generative adversarial network based on the pixel decision loss; An updating unit is used to fix the parameters of any two models among the generator, the global judge and the pixel judge, and to update the parameters of the remaining models until the convergence condition of the generative adversarial network is met.
12. An image processing device, characterized in that: The device comprises: An acquisition unit, used for acquiring an image to be processed; A super-resolution unit, used for performing convolution processing on the image to be processed in a generator of a generative adversarial network to obtain a target image corresponding to the image to be processed; The generator is obtained by the following processing based on the global judger and the pixel judger in the generative adversarial network: Initializing the generator, the global determiner and the pixel determiner; The pixel decision device is initialized by a pixel decision loss, the pixel decision loss is determined based on a pixel difference between an original natural image and a reconstructed image output by the generator, the pixel difference comprising a first initial probability matrix of pixels of the reconstructed image being determined as pixels of a non-natural image, and a second initial probability matrix of pixels of the original natural image being determined as pixels of a non-natural image; The parameters of the generator, the global judger and the pixel judger are updated until the convergence condition of the generative adversarial network is met.
13. A training device for generating an adversarial network, characterized in that: The device comprises: a memory, a communication bus and a processor, wherein: The memory is used to store computer programs; The communication bus is used to realize the connection and communication between the processor and the memory; The processor is used to implement the steps of the training method for generating an adversarial network as described in any one of claims 1 to 9 when executing the computer program stored in the memory.
14. An image processing device, characterized in that: The device comprises: a memory, a communication bus and a processor, wherein: The memory is used to store computer programs; The communication bus is used to realize the connection and communication between the processor and the memory; The processor is used to implement the steps of the image processing method according to claim 10 when executing the computer program stored in the memory.
15. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, it implements the training method of the generative adversarial network described in any one of claims 1 to 9, or the image processing method described in claim 10.
