Image processing method and device based on generative adversarial network and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides an image processing method based on a generative adversarial network, and the method comprises the steps: injecting a Gaussian noise vector matched with the dimension of a current feature map at the output end of the last residual dense block of cascaded residual dense blocks of a pre-trained generator, generating an initial high-resolution image according to the primary mapping relation; inputting the initial high-resolution image into a discriminator, extracting features through a convolutional layer of the discriminator, and generating a quality matrix containing local detail scores; calculating a composite loss value based on the quality matrix, and synchronously updating an expanded convolutional layer parameter of the pre-trained generator and a convolution kernel weight of the discriminator according to the composite loss value; and when the fluctuation amplitude of the composite loss value after continuous preset times of iteration is less than a preset threshold value, stopping adversarial training on the generative adversarial network, and outputting an optimized super-resolution image by the current pre-trained generator. According to the technical scheme, the visual authenticity of the amplified image can be remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to an image processing method and device based on a generative adversarial network and a storage medium. BACKGROUND

[0002] The existing image super-resolution technology is mainly based on a convolutional neural network, and a mapping relationship from a low-resolution image to a high-resolution image is learned through an end-to-end learning. However, the above existing method has the following defects: 1) insufficient detail reconstruction capability, which is manifested in that a traditional convolutional neural network structure is difficult to effectively capture high-frequency detail information, and especially when a large-multiple super-resolution task is processed, a reconstructed image is prone to have a texture blur and an edge jaggy phenomenon; 2) lack of image authenticity, which is manifested in that a method based on a common optimization method, such as a mean square error optimization method, is prone to cause a generated image to be too smooth and lack natural textures in a real scene. SUMMARY

[0003] The present application provides an image processing method and device based on a generative adversarial network and a storage medium, which can significantly improve the visual authenticity of an enlarged image.

[0004] In one aspect, the present application provides an image processing method based on a generative adversarial network, which comprises: inputting a to-be-processed low-resolution image into a pre-trained generator, injecting a Gaussian noise vector matching a current feature map dimension at an output end of a last residual dense block of a cascaded residual dense block of the pre-trained generator, generating an initial high-resolution image according to a primary mapping relationship, the pre-trained generator being obtained after training of the generative adversarial network, and the primary mapping relationship being a mapping relationship from a low-resolution image to a high-resolution image established in a training process of the generator; inputting the initial high-resolution image into the discriminator, extracting features through a convolutional layer of the discriminator and generating a quality matrix containing a local detail score; calculating a composite loss value based on the quality matrix, and synchronously updating an expansion convolutional layer parameter of the pre-trained generator and a convolutional kernel weight of the discriminator according to the composite loss value; stopping the adversarial training of the generative adversarial network when a composite loss value after continuous preset times of iterations has a fluctuation amplitude less than a preset threshold, inputting the to-be-processed low-resolution image into a current pre-trained generator, and outputting an optimized super-resolution image from the current pre-trained generator.

[0005] In another aspect, the present application provides an image processing device based on a generative adversarial network, which comprises: The first generation module is configured to input a to-be-processed low-resolution image into a pre-trained generator, inject a Gaussian noise vector matching a current feature map dimension at an output end of a last residual dense block of a cascade residual dense block of the pre-trained generator, and generate an initial high-resolution image according to a primary mapping relationship, wherein the pre-trained generator is obtained after training of a generative adversarial network, and the primary mapping relationship is a mapping relationship from a low-resolution image to a high-resolution image established during training of the generator; The second generation module is configured to input the initial high-resolution image into the discriminator, extract features through a convolutional layer of the discriminator, and generate a quality matrix containing a local detail score. The update module is configured to calculate a composite loss value based on the quality matrix, and synchronously update parameters of a dilated convolutional layer of the pre-trained generator and weights of a convolutional kernel of the discriminator according to the composite loss value. The output module is configured to stop adversarial training of the generative adversarial network when a fluctuation amplitude of the composite loss value after continuous preset times of iteration is less than a preset threshold, input the to-be-processed low-resolution image into a current pre-trained generator, and output an optimized super-resolution image from the current pre-trained generator.

[0006] In a third aspect, the present application provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the technical solution of the image processing method based on the generative adversarial network when executing the computer program.

[0007] In a fourth aspect, the present application provides a storage medium, which stores a computer program, and the computer program implements the steps of the technical solution of the image processing method based on the generative adversarial network when executed by a processor.

[0008] From the technical solutions provided in the application, on the one hand, the low-resolution-to-high-resolution primary mapping relationship established in the pre-training process of the generator of the generative adversarial network can capture the low-frequency, medium-frequency and high-frequency features of the image synchronously, and solve the problem of insufficient capture of high-frequency information by the traditional single-scale convolutional network; on the other hand, the generator is pre-trained to establish the low-resolution-to-high-resolution primary mapping relationship, so that the generator has the basic image reconstruction capability before adversarial optimization, thereby avoiding the problem of serious distortion of the initial reconstructed image caused by the traditional generative adversarial network starting training from randomly initialized parameters, and providing a high-quality initial input for subsequent adversarial learning; and on the third aspect, the composite loss value is calculated based on the quality matrix, and the dilated convolution layer parameters of the generator and the convolution kernel weights of the discriminator module are updated synchronously according to the composite loss value, so that the three types of optimization objectives of pixel-level fidelity, adversarial detail authenticity and perceptual similarity can be synergistically acted, thereby significantly improving the visual authenticity of the generated image. In summary, the technical solutions of the application can significantly improve the visual authenticity of the enlarged image. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0010] Figure 1 is a flowchart of the image processing method based on the generative adversarial network provided by the embodiments of the present application; Figure 2 is a structural schematic diagram of the image processing device based on the generative adversarial network provided by the embodiments of the present application; Figure 3 is a structural schematic diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0011] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0012] In this specification, adjectives such as "first" and "second" can merely be used to differentiate one element from another without necessarily implying an actual relationship or order. Where the context permits, reference to an element or component or step (etc.) can be construed as referring to one or more elements, components, or steps (etc.) individually or collectively.

[0013] In this specification, the sizes of the respective parts shown in the drawings can not be drawn to scale in order to emphasize certain features.

[0014] The existing image super-resolution technology is mainly based on a convolutional neural network, and the mapping relationship from a low-resolution image to a high-resolution image is learned through end-to-end learning. However, the above existing method has the following defects: 1) insufficient detail reconstruction capability, which is manifested in that the traditional convolutional neural network structure is difficult to effectively capture high-frequency detail information, and especially when processing a large multiple super-resolution task, the reconstructed image is prone to texture blurring and edge jaggedness; 2) lack of image authenticity, which is manifested in that the method based on the common optimization method, such as mean square error optimization, is easy to cause the generated image to be too smooth, lacking natural texture in the real scene.

[0015] In view of the above problems of the prior art, the present application proposes an image processing method based on a generative adversarial network, the flow chart of which is as shown in the accompanying Figure 1 The method mainly includes steps S101 to S104, which are described in detail as follows: Step S101: inputting a low-resolution image to be processed into a pre-trained generator, injecting a Gaussian noise vector matching the dimension of the current feature map at the output end of the last residual dense block of the cascaded residual dense block of the pre-trained generator, and generating an initial high-resolution image according to a primary mapping relationship, wherein the pre-trained generator is obtained after training of a generative adversarial network, and the primary mapping relationship is a mapping relationship from a low-resolution image to a high-resolution image established in the training process of the generator.

[0016] The pre-trained generator is obtained after training of the generative adversarial network, which means that before the example method starts, there will be a process of training the generator of the generative adversarial network, including: constructing a generative adversarial network containing a generator and a discriminator, wherein the generator contains a cascaded residual dense block; pre-training the generator of the generative adversarial network by using a training set, wherein multi-scale features of the training set are extracted through the dilated convolution layer of the cascaded residual dense block to establish a primary mapping relationship from a low-resolution image to a high-resolution image. The following will be described in detail. Figure 1

[0017] ​In the embodiments of the present application, the adversarial generation network includes a generator and a discriminator. The role or goal of the generator is to become a "counterfeit master", that is, the generator receives an input (for example, a low-resolution image), and attempts to generate a high-resolution image that is difficult to distinguish from a real image (for example, a high-resolution image), and the task is to make the generated image as close as possible to a real high-quality image in terms of details, textures and overall authenticity, so as to "deceive" the discriminator. The role or goal of the discriminator is just "opposed" to the generator, that is, the discriminator is to become a "gem expert", which receives an image and judges whether the image is "real" (from a real high-resolution image dataset) or "fake" (from the generator). Through continuous learning, the ability to distinguish true and false is improved, so as to force the generator to continuously improve its generation quality. Simply, the generator and the discriminator interact in a game process: the generator strives to improve the counterfeiting technology to pass the identification, and the discriminator strives to improve the identification ability to prevent being deceived. This adversarial training eventually enables the generator to produce extremely realistic results.

[0018] The generator of the above-mentioned embodiments includes a cascaded residual dense block, and each residual dense block of the cascaded residual dense block, specifically, is composed of a plurality of dilated convolution layers and standard convolution layers connected alternately: first, a standard convolution layer is used to focus on extracting local detailed features, and then a dilated convolution layer is connected to expand the receptive field and capture the low-frequency global context information of the image. The above-mentioned operations are alternately performed to form a multi-level feature extraction path. At the end of each residual dense block, a multi-scale feature fusion structure (for example, a channel attention mechanism) is configured to adaptively weight and fuse the feature maps output by the foregoing different convolution layers, so as to be able to simultaneously capture the low-frequency global features, the medium-frequency texture features and the high-frequency edge features of the image, and effectively solve the problem of insufficient capture of high-frequency information by the traditional single-scale convolution network in combination with the local detail enhancement capability of the standard convolution layer.

[0019] In the embodiments of the present application, the multi-scale features of the training set are extracted through the dilated convolution layer of the cascaded residual dense block of the generator, the generator is pre-trained, that is, the pre-training stage of the generator, after the pre-training stage is completed, the primary mapping relationship from low resolution to high resolution is established, which can make the generator have the basic image reconstruction capability before the adversarial optimization, thereby avoiding the problem of serious distortion of the initial reconstruction image caused by the traditional generative adversarial network directly starting from the random initialization parameters, and providing a high-quality initial input for the subsequent adversarial learning. It should be noted that the training set of the above embodiments refers to a group of data used to train the model, which usually contains paired image samples, and the specific content is as follows: training set={(low resolution image 1, corresponding high resolution real image 1), (low resolution image 2, corresponding high resolution real image 2), …}, wherein the high resolution real image is a clear and high-quality original image, which is the target of model learning, and the low resolution image is usually obtained by artificially degrading the high resolution real image, for example, using bicubic downsampling and then enlarging to the target size to simulate the effect of reducing the image resolution. In this way, the model can learn the mapping relationship from low-quality input to high-quality output. As an embodiment of the present application, the multi-scale features of the training set are extracted through the dilated convolution layer of the cascaded residual dense block of the generator, which can be realized through the following steps S1 to S4, which are described in detail as follows: Step S1: extracting low-frequency global features of the training set through the convolution layer with a dilation rate of r in the cascaded residual dense block.

[0020] The dilation rate (Dilation Rate) is the core parameter of the dilated convolution, which is used to control the interval between the sampling points of the convolution kernel. In the standard convolution, when the interval between adjacent convolution kernel elements is 1, the interval of the dilated convolution is r-1, where r is the dilation rate. If the size of the convolution kernel is 3x3, then for the dilated convolution with a dilation rate of r, its actual coverage range is equivalent to , for example: r=2, its actual coverage range is equivalent to a 5x5 region, and the sampling point interval is 1 pixel; r=4, its actual coverage range is equivalent to a 9x9 region, and the sampling point interval is 3 pixels. Convolution layers with different dilation rates r are used in parallel, which can capture local details and global context information at the same time. The essence of the dilation rate is to control the sampling interval of the convolution kernel to expand the receptive field under the premise of constant parameters. Since the size of the feature map is usually large near the input layer, the size of the feature map of the shallow network is large, and a small dilation rate can effectively capture low-frequency global features. Therefore, in the embodiments of the present application, when the low-frequency global features of the training set are extracted through the convolution layer with a dilation rate of r in the cascaded residual dense block, the dilation rate can be taken as a small value, for example, =2, at this time, if the size of the convolution kernel is 3x3, the dilated convolution can cover a 5x5 local region, which is sufficient to capture low-frequency global features.

[0021] Step S2: Extracting the medium-frequency texture features of the training set through the convolution layer with the dilation rate of 2 in the cascaded residual dense block.

[0022] Since the feature map size extracted by the medium-layer network (i.e., the network between the shallow-layer network and the deep-layer network) is reduced, the local and global information needs to be balanced. Therefore, in the embodiment of the present application, when the medium-frequency texture features of the training set are extracted through the convolution layer with the dilation rate of 2 in the cascaded residual dense block, the dilation rate can be taken as a value slightly larger than 2, for example, = 4. At this time, if the convolution kernel size is 3x3, the dilated convolution can cover a 9x9 area.

[0023] Step S3: Extracting the high-frequency edge features of the training set through the convolution layer with the dilation rate of 6 in the cascaded residual dense block.

[0024] The high-frequency features are generally rapidly changing edges or textures (such as object contours, hair), which need dense local sampling. Since the feature map size extracted by the deep-layer network is further reduced relative to the feature map size extracted by the medium-layer network, the sparse but significant high-frequency edges need to be captured. Therefore, in the embodiment of the present application, when the high-frequency edge features of the training set are extracted through the convolution layer with the dilation rate of 6 in the cascaded residual dense block, the dilation rate can be taken as a value larger than 6, for example, = 6. At this time, if the convolution kernel size is 3x3, the dilated convolution can cover a 13x13 area.

[0025] Step S4: Fusing the feature maps output after the convolution layers with the dilation rates of 2, 4 and 6 are used to extract the low-frequency global features, the medium-frequency texture features and the high-frequency edge features, respectively, through channel attention weighting to obtain the multi-scale features of the training set.

[0026] Specifically, the feature maps output after the convolution layers with the dilation rates of 2, 4 and 6 are used to extract the low-frequency global features, the medium-frequency texture features and the high-frequency edge features, respectively, are fused through channel attention weighting to obtain the multi-scale features of the training set. ​​​​​​​​​​​The feature maps outputted by the convolutional layers with expansion rates of 、 and in the cascaded residual dense block are received as input. For each of the received feature maps, a global average pooling operation is performed to compress the feature map in the spatial dimension (height and width) and to count the average value of all feature values in each channel. A one-dimensional vector equal to the number of channels of each feature map is outputted as the channel description vector corresponding to the feature map. For each channel description vector, a two-layer fully connected network is used to generate a channel attention weight matrix. The multiple channel description vectors corresponding to different expansion rate convolutional layers generated by the foregoing global average pooling are received as input. Each channel description vector is sequentially inputted into a two-layer fully connected neural network for processing. The first fully connected layer is used to reduce the dimension and introduce a nonlinear transformation, and the second fully connected layer is used to restore the original channel dimension and output the weight score. The weight score outputted by the two-layer fully connected network is activated by a Sigmoid function and normalized to the interval (0, 1) to generate the attention weight corresponding to each channel. A channel attention weight matrix with the same dimension as the input channel description vector is outputted. Each feature map outputted by each expansion rate convolutional layer is multiplied with the corresponding channel attention weight matrix in the channel. The feature maps outputted by the convolutional layers with expansion rates of 、 and in the cascaded residual dense block are received as input. For each of the received feature maps, a global average pooling operation is performed to compress the feature map in the spatial dimension (height and width) and to count the average value of all feature values in each channel. A one-dimensional vector equal to the number of channels of each feature map is outputted as the channel description vector corresponding to the feature map. For each channel description vector, a two-layer fully connected network is used to generate a channel attention weight matrix. The multiple channel description vectors corresponding to different expansion rate convolutional layers generated by the foregoing global average pooling are received as input. Each channel description vector is sequentially inputted into a two-layer fully connected neural network for processing. The first fully connected layer is used to reduce the dimension and introduce a nonlinear transformation, and the second fully connected layer is used to restore the original channel dimension and output the weight score. The weight score outputted by the two-layer fully connected network is activated by a Sigmoid function and normalized to the interval (0, 1) to generate the attention weight corresponding to each channel. A channel attention weight matrix with the same dimension as the input channel description vector is outputted. Each feature map outputted by each expansion rate convolutional layer is multiplied with the corresponding channel attention weight matrix in the channel.the original feature maps output by the convolutional layers and the channel attention weight matrix corresponding to each original feature map generated by the foregoing steps; for each pair of original feature maps and the channel attention weight matrix corresponding thereto, a channel-by-channel multiplication operation is performed, that is, all elements in each channel of the feature map are multiplied by the attention weight of the corresponding channel; and a set of channel attention weighted feature maps is output. The feature maps obtained by multiplication are spliced to obtain the multi-scale features of the training set, which can be achieved in the following manner: receiving a plurality of weighted feature maps corresponding to the expansion rates 、 and output by the foregoing channel attention weighting step as input; along the channel dimension, the plurality of weighted feature maps are spliced in the order of the feature maps corresponding to the expansion rates 、 and (or in the order of feature extraction) to form a fused feature map with a channel number being the sum of the channel numbers of the input feature maps; and outputting the fused feature map as the multi-scale features of the training set.

[0027] In a traditional generative adversarial network, the image output by the generator does not inject noise, which means that the image output by the generator lacks a detail disturbance mechanism. Comparative experiments have also proved that not injecting noise will result in a significant decrease in the detail diversity of the image generated by the generator and an increase in the flat area artifact rate, which will cause the discriminator to be unable to effectively guide the generation of details in the adversarial training. Therefore, in the embodiments of the present application, a low-resolution image to be processed is input into a pre-trained generator, and a Gaussian noise vector matching the dimension of the current feature map is injected at the output end of the last residual dense block of the cascaded residual dense blocks of the trained generator. It should be noted that each residual dense block of the cascaded residual dense blocks processes the input low-resolution image to be processed or feature map in turn, that is, the low-resolution image to be processed is input into the first residual dense block, the feature map output by the first residual dense block is input into the second residual dense block, the feature map output by the second residual dense block is input into the third residual dense block, and so on, until the feature map output by the last residual dense block is obtained. Therefore, the "current feature map" here refers to the feature map output by the last residual dense block of the cascaded residual dense blocks. In addition, the dimension of the Gaussian noise vector needs to match the dimension of the current feature map, that is, the Gaussian noise vector is spatially aligned with the current feature map to avoid introducing distortion.

[0028] Specifically, injecting a Gaussian noise vector matching the dimension of the current feature map at the output end of the last residual dense block of the cascaded residual dense blocks of the trained generator can be achieved through steps S1011 to S1013, which are described in detail as follows: Step S1011: Generate a random noise matrix with the same dimensions as the current feature map.

[0029] The noise values ​​of the random noise matrix can follow a normal distribution with a mean of 0 and a variance of 0.1. Specifically, generating a random noise matrix with the same dimension as the current feature map can be done by: obtaining the dimension information of the current feature map output by the last residual dense block in the cascaded residual dense blocks of the generator, where the dimension information includes height H, width W, and number of channels C; and generating a random noise matrix with dimension [H, W, C] based on the height H, width W, and number of channels C, where each element value in the random noise matrix is ​​sampled from a predefined random distribution.

[0030] Step S1012: Adjust the noise intensity of the random noise matrix using learnable parameters.

[0031] Among them, learnable parameters This can be adjusted by calculating the average gradient magnitude of the current feature map. The basic scaling factor is generated by mapping using the sigmoid function. ,in, k These are trainable parameters; obtained through the following formula The superimposed momentum adjustment term adjusts the learnable parameter. ,in, Momentum factor This is the average change in the composite loss value over the most recent preset number of iterations (e.g., 3). From the above learnable parameters... The adjustment method reveals that: by mapping the base scaling factor using the sigmoid function, the noise intensity in regions with large gradients (which exhibit rich details) is significantly reduced, thus preventing the destruction of details. Furthermore, the addition of momentum adjustment terms in the later stages of training... When reduced, it can automatically enhance noise diversity.

[0032] Specifically, through learnable parameters Adjusting the noise intensity of a random noise matrix can be done by: receiving a random noise matrix as input; and then adjusting each element of the random noise matrix with a learnable parameter. Multiplying these results in a random noise matrix adjusted for noise intensity. As for the average gradient magnitude of the current feature map... The specific calculation process can be as follows: calculate the gradient magnitude of all channels at each spatial location in the current feature map; calculate the average of all calculated gradient magnitudes to obtain the average gradient magnitude representing the overall detail richness of the current feature map. .

[0033] Step S1013: Add the random noise matrix after noise intensity adjustment to the current feature map element by element.

[0034] Here, the element-wise addition of the noise intensity-adjusted random noise matrix to the current feature map can be achieved as follows: the current feature map output by the last residual dense block in the cascaded residual dense block of the generator; the current feature map and the intensity-adjusted random noise matrix (which has the same dimension as the current feature map) are then added at corresponding positions; the noise-injected feature map is output for the generator's subsequent upsampling and image reconstruction processes.

[0035] Step S102: Input the initial high-resolution image into the discriminator, extract features through the convolutional layer of the discriminator, and generate a quality matrix containing local detail scores.

[0036] Specifically, the output feature map of the last convolutional layer of the discriminator can be segmented into l×l local regions; for each local region, a detail score can be calculated according to the following formula. :

[0037] Where N is the number of channels in the final convolutional layer output feature map of the discriminator. The feature value on the nth channel, This represents the super-resolution image output by the generator. This represents a real high-resolution image paired with a low-resolution training image. and Let i and j represent the pixels with row and column coordinates (i, j) respectively. Therefore, the detail score... The value is actually obtained by calculating the difference between the corresponding local features of the generated image and the real image in the discriminator feature space.

[0038] In the above embodiments, after calculating the detail score Then, when the last layer feature map of the discriminator is segmented into m x m local regions, each region has a unique spatial location index on the feature map, such as region (1, 1), region (1, 2), ..., region (l, m); the quality matrix is ​​an m x m matrix, that is, the calculated detail score of each local region. The subscripts i and j directly correspond to the row and column indices of the region in the quality matrix. For example, assuming m=4, the feature map is divided into 4x4=16 regions. The score for each region is calculated, then the score of the first region located in the top left corner is... The rating of the second area to its right is... The score is located in the last area in the bottom right corner. Finally, these 16 scores are arranged into a 4x4 matrix according to their inherent spatial location; this matrix is ​​the quality matrix. It is equivalent to a detail score heatmap, which visually displays the local quality of different regions of the generated image.

[0039] Step S103: Calculate the composite loss value based on the quality matrix, and update the parameters of the dilated convolutional layer of the generator and the convolutional kernel weights of the discriminator synchronously according to the composite loss value.

[0040] Specifically, calculating the composite loss value based on the mass matrix can be done by: based on the mass matrix, and according to the formula... Calculate the composite loss value ,in, The loss value is at the pixel level. To combat the loss value, To perceive the similarity loss value, , and They are respectively , and The weights of perceptual similarity loss. The image is obtained as follows: the generator output image and the real high-resolution image are input into a pre-trained VGG19 network; the first image is extracted from the second image. Convolutional layer, first Convolutional layers and the first Feature maps of the convolutional layer; calculate the first convolutional layer. Convolutional layer, first Convolutional layers and the first Cosine similarity loss of feature maps in convolutional layers As the perceptual similarity loss value, where, Image output by the generator The three-dimensional feature tensor extracted during forward propagation of the pre-trained network to the l-th layer. For true high-resolution images The feature vector extracted during the pre-training of the network's l-th layer is used as a benchmark reference for perceptual similarity calculation. As an embodiment of this application, synchronously updating the parameters of the dilated convolutional layers of the generator and the convolutional kernel weights of the discriminator based on the composite loss value can be: based on the composite loss value... The algorithm uses backpropagation to simultaneously calculate the gradients of the generator's dilated convolutional layer parameters and the discriminator's convolutional kernel weights. Based on the calculated gradients, the algorithm uses gradient descent to simultaneously update the generator's dilated convolutional layer parameters and the discriminator's convolutional kernel weights.

[0041] Step S104: When the fluctuation range of the composite loss value after a preset number of consecutive iterations is less than a preset threshold, stop the adversarial training of the generative adversarial network and input the low-resolution image to be processed into the current pre-trained generator, and the current pre-trained generator outputs the optimized super-resolution image.

[0042] It should be noted that the current pre-trained generator refers to the generator obtained when the adversarial training of the generative adversarial network is stopped when the fluctuation of the composite loss value after a preset number of consecutive iterations is less than a preset threshold.

[0043] From the above appendix Figure 1 As illustrated in the example image processing method based on generative adversarial networks (GANs), on the one hand, by establishing a primary mapping relationship from low resolution to high resolution during the pre-training process of the generator in the GAN, low-frequency, mid-frequency, and high-frequency features of the image can be captured simultaneously, solving the problem of insufficient high-frequency information capture by traditional single-scale convolutional networks. On the other hand, pre-training the generator and establishing a primary mapping relationship from low resolution to high resolution enables the generator to have basic image reconstruction capabilities before adversarial optimization, thus avoiding the severe distortion of the reconstructed image in the early stages caused by traditional GANs starting training directly from randomly initialized parameters, and providing high-quality initial input for subsequent adversarial learning. Thirdly, by calculating a composite loss value based on the quality matrix and synchronously updating the parameters of the generator's dilated convolutional layers and the kernel weights of the discriminator module according to the composite loss value, the three optimization objectives of pixel-level fidelity, adversarial detail realism, and perceptual similarity can work synergistically, thereby significantly improving the visual realism of the generated image. In summary, the technical solution of this application can significantly improve the visual realism of the image after magnification.

[0044] Please see the appendix Figure 2 This application provides an image processing apparatus based on a generative adversarial network. The apparatus may include a first generation module 201, a second generation module 202, an update module 203, and an output module 204, as detailed below: The first generation module 201 is used to input the low-resolution image to be processed into the pre-trained generator, inject a Gaussian noise vector matching the dimension of the current feature map into the output of the last residual dense block of the cascaded residual dense block of the pre-trained generator, and generate an initial high-resolution image according to the primary mapping relationship. The pre-trained generator is obtained after training the generative adversarial network, and the primary mapping relationship is the mapping relationship from low-resolution image to high-resolution image established during the generator training process. The second generation module 202 is used to input the initial high-resolution image into the discriminator, extract features through the convolutional layer of the discriminator, and generate a quality matrix containing local detail scores. Update module 203 is used to calculate the composite loss value based on the quality matrix, and to synchronously update the parameters of the dilated convolutional layer of the generator and the convolutional kernel weights of the discriminator after pre-training according to the composite loss value; The output module 204 is used to stop the adversarial training of the generative adversarial network when the fluctuation amplitude of the composite loss value after a preset number of consecutive iterations is less than a preset threshold, and input the low-resolution image to be processed into the current pre-trained generator, and output the optimized super-resolution image by the current pre-trained generator.

[0045] From the above appendix Figure 2 As illustrated in the example of the generative adversarial network (GAN)-based image processing device, on the one hand, by establishing a primary mapping relationship from low resolution to high resolution during the pre-training process of the GAN generator, low-frequency, mid-frequency, and high-frequency features of the image can be captured simultaneously, solving the problem of insufficient high-frequency information capture by traditional single-scale convolutional networks. On the other hand, pre-training the generator and establishing a primary mapping relationship from low resolution to high resolution enables the generator to possess basic image reconstruction capabilities before adversarial optimization, thereby avoiding the severe distortion of the reconstructed image in the early stages caused by traditional GANs directly training from randomly initialized parameters, and providing high-quality initial input for subsequent adversarial learning. Thirdly, by calculating a composite loss value based on the quality matrix and synchronously updating the parameters of the generator's dilated convolutional layers and the convolutional kernel weights of the discriminator module according to the composite loss value, the three optimization objectives of pixel-level fidelity, adversarial detail realism, and perceptual similarity can work synergistically, thereby significantly improving the visual realism of the generated image. In summary, the technical solution of this application can significantly improve the visual realism of the image after magnification.

[0046] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 3 As shown, the electronic device 3 in this embodiment mainly includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a program for an image processing method based on a generative adversarial network. When the processor 30 executes the computer program 32, it implements the steps described in the embodiment of the image processing method based on a generative adversarial network, for example... Figure 1 The steps S101 to S104 are shown. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 2 The functions of the first generation module 201, the second generation module 202, the update module 203, and the output module 204 are shown.

[0047] For example, the computer program 32 of the image processing method based on generative adversarial networks mainly includes: inputting the low-resolution image to be processed into a pre-trained generator; injecting a Gaussian noise vector matching the dimension of the current feature map into the output of the last residual dense block of the cascaded residual dense block of the pre-trained generator; generating an initial high-resolution image according to the primary mapping relationship, wherein the pre-trained generator is obtained after training the generative adversarial network, and the primary mapping relationship is the mapping relationship from low-resolution image to high-resolution image established during the generator training process; inputting the initial high-resolution image into a discriminator; extracting features through the convolutional layer of the discriminator and generating a quality matrix containing local detail scores; calculating a composite loss value based on the quality matrix; synchronously updating the dilated convolutional layer parameters of the pre-trained generator and the convolutional kernel weights of the discriminator according to the composite loss value; stopping the adversarial training of the generative adversarial network when the fluctuation amplitude of the composite loss value after a preset number of consecutive iterations is less than a preset threshold, and inputting the low-resolution image to be processed into the current pre-trained generator, and outputting the optimized super-resolution image by the current pre-trained generator. The computer program 32 can be divided into one or more modules / units. One or more modules / units are stored in the memory 31 and executed by the processor 30 to complete this application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 32 in the electronic device 3. For example, the computer program 32 can be divided into the functions of a first generation module 201, a second generation module 202, an update module 203, and an output module 204 (a module in a virtual device). The specific functions of each module are as follows: The first generation module 201 is used to input the low-resolution image to be processed into a pre-trained generator, inject a Gaussian noise vector matching the dimension of the current feature map into the output of the last residual dense block of the cascaded residual dense block of the pre-trained generator, and generate an initial high-resolution image according to the primary mapping relationship. Here, the pre-trained generator is obtained after training a generative adversarial network, and the primary mapping relationship is the mapping from low-resolution image to high-resolution image established during the generator training process. The mapping relationship of resolution images; the second generation module 202 is used to input the initial high-resolution image into the discriminator, extract features through the convolutional layer of the discriminator and generate a quality matrix containing local detail scores; the update module 203 is used to calculate the composite loss value based on the quality matrix, and synchronously update the dilated convolutional layer parameters of the pre-trained generator and the convolutional kernel weights of the discriminator according to the composite loss value; the output module 204 is used to stop the adversarial training of the generative adversarial network when the fluctuation amplitude of the composite loss value after a preset number of consecutive iterations is less than a preset threshold, and input the low-resolution image to be processed into the current pre-trained generator, and output the optimized super-resolution image by the current pre-trained generator.

[0048] Electronic device 3 may include, but is not limited to, processor 30 and memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0049] The processor 30 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0050] The memory 31 can be an internal storage unit of the electronic device 3, such as a hard disk or RAM. The memory 31 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 31 can include both internal and external storage units of the electronic device 3. The memory 31 is used to store computer programs and other programs and data required by the electronic device. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0051] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed. That is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above-described device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0052] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0053] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0054] In the embodiments provided in this application, it should be understood that the disclosed apparatus / device and method can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0055] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0056] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0057] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program for the image processing method based on generative adversarial networks can be stored in a storage medium. When executed by a processor, this computer program can implement the steps of the various method embodiments described above, namely, inputting the low-resolution image to be processed into a pre-trained generator, injecting a Gaussian noise vector matching the current feature map dimension into the output of the last residual dense block of the cascaded residual dense blocks of the pre-trained generator, and generating an initial high-resolution image according to the primary mapping relationship. Here, the pre-trained generator is obtained by training a generative adversarial network. The initial mapping relationship is the mapping relationship from low-resolution images to high-resolution images established during the generator training process. The initial high-resolution image is input into the discriminator, and features are extracted through the convolutional layer of the discriminator to generate a quality matrix containing local detail scores. A composite loss value is calculated based on the quality matrix, and the parameters of the dilated convolutional layer of the pre-trained generator and the convolutional kernel weights of the discriminator are updated synchronously according to the composite loss value. When the fluctuation amplitude of the composite loss value after a preset number of consecutive iterations is less than a preset threshold, the adversarial training of the generative adversarial network is stopped, and the low-resolution image to be processed is input into the current pre-trained generator, which outputs the optimized super-resolution image. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. The storage medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of the storage medium may be appropriately added to or subtracted from the contents according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the storage medium may not include electrical carrier signals and telecommunication signals.

[0058] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application. The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the protection scope of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this invention.

Claims

1. An image processing method based on generative adversarial networks, characterized in that, The method includes: The low-resolution image to be processed is input into the pre-trained generator. A Gaussian noise vector matching the dimension of the current feature map is injected into the output of the last residual dense block of the cascaded residual dense block of the pre-trained generator. An initial high-resolution image is generated according to the primary mapping relationship. The pre-trained generator is obtained after training the generative adversarial network. The primary mapping relationship is the mapping relationship from low-resolution image to high-resolution image established during the training of the generator. The initial high-resolution image is input into the discriminator, and features are extracted through the convolutional layer of the discriminator to generate a quality matrix containing local detail scores. Calculate the composite loss value based on the quality matrix, and update the parameters of the dilated convolutional layer of the pre-trained generator and the weights of the convolutional kernel of the discriminator synchronously according to the composite loss value. When the fluctuation range of the composite loss value after a preset number of consecutive iterations is less than a preset threshold, the adversarial training of the generative adversarial network is stopped, and the low-resolution image to be processed is input into the current pre-trained generator, which then outputs the optimized super-resolution image.

2. The image processing method based on generative adversarial networks as described in claim 1, characterized in that, Before inputting the low-resolution image to be processed into the pre-trained generator, the method further includes: constructing a generative adversarial network containing the generator and a discriminator, wherein the generator contains cascaded residual dense blocks; and pre-training the generator of the generative adversarial network using a training set, wherein multi-scale features of the training set are extracted through dilated convolutional layers of the cascaded residual dense blocks to establish a primary mapping relationship from low-resolution images to high-resolution images. The extraction of multi-scale features from the training set through the dilated convolutional layers of the cascaded residual dense blocks includes: The expansion rate in the cascaded residual dense block is The convolutional layers extract low-frequency global features from the training set; The expansion rate in the cascaded residual dense block is The convolutional layers extract the mid-frequency texture features of the training set; The expansion rate in the cascaded residual dense block is The convolutional layers extract high-frequency edge features from the training set; The expansion rate , and The convolutional layers extract the low-frequency global features, mid-frequency texture features, and high-frequency edge features. The output feature maps are then fused using channel attention weighting to obtain the multi-scale features of the training set.

3. The image processing method based on generative adversarial networks as described in claim 2, characterized in that, The expansion rate is , and The feature maps output by the convolutional layers are subjected to channel attention weighted fusion to obtain the multi-scale features of the training set, including: Global average pooling is performed on the feature maps output by each dilation rate convolutional layer to generate channel description vectors; For each channel description vector, a two-layer fully connected network is used to process it to generate a channel attention weight matrix; Multiply the feature map output by each dilation rate convolutional layer channel by channel with the corresponding channel attention weight matrix; The feature maps obtained after multiplication are concatenated to obtain the multi-scale features of the training set.

4. The image processing method based on generative adversarial networks as described in claim 1, characterized in that, The step of injecting a Gaussian noise vector matching the dimension of the current feature map into the output of the last residual dense block of the cascaded residual dense blocks of the generator after training includes: Generate a random noise matrix with the same dimensions as the current feature map; Learnable parameters Adjust the noise intensity of the random noise matrix; The random noise matrix after noise intensity adjustment is added element by element to the current feature map.

5. The image processing method based on generative adversarial networks as described in claim 4, characterized in that, Before each injection of a Gaussian noise vector matching the dimension of the current feature map, the learnable parameters are adjusted as follows. : Calculate the average gradient magnitude of the current feature map. ; The basic scaling factor is generated by mapping using the sigmoid function. The k For trainable parameters, the The current feature map; Through the following formula Overlay a momentum adjustment term to adjust the learnable parameter. The The momentum factor, the This is the average change of the composite loss value over the most recent preset number of iterations.

6. The image processing method based on generative adversarial networks according to claim 1, characterized in that, The calculation of the composite loss value based on the quality matrix includes: Based on the aforementioned quality matrix, and according to the formula Calculate the composite loss value The The loss value is at the pixel level. To combat the loss value, the To perceive the similarity loss value, the , and The respective , and The weight.

7. The image processing method based on generative adversarial networks according to claim 6, characterized in that, The perceptual similarity loss It is obtained in the following way: The generated output image and the corresponding real high-resolution image during the generator training process are input into the pre-trained VGG19 network; Extract the first one respectively Convolutional layer, first Convolutional layers and the first Feature maps of convolutional layers; Calculate the first Convolutional layer, first Convolutional layers and the first Cosine similarity loss of feature maps in convolutional layers As the perceived similarity loss value, the The image output by the generator Forward propagation through the pre-trained network to the [number]th ... The three-dimensional feature tensor extracted during layer-by-layer processing, the For true high-resolution images Through the pre-trained network Feature tensors extracted at each layer.

8. An image processing apparatus based on generative adversarial networks, characterized in that, The device includes: The first generation module is used to input the low-resolution image to be processed into the pre-trained generator, inject a Gaussian noise vector matching the dimension of the current feature map into the output of the last residual dense block of the cascaded residual dense block of the pre-trained generator, and generate an initial high-resolution image according to the primary mapping relationship. The pre-trained generator is obtained after training the generative adversarial network, and the primary mapping relationship is the mapping relationship from low-resolution image to high-resolution image established during the training of the generator. The second generation module is used to input the initial high-resolution image into the discriminator, extract features through the convolutional layer of the discriminator, and generate a quality matrix containing local detail scores. The update module is used to calculate the composite loss value based on the quality matrix, and synchronously update the parameters of the dilated convolutional layer of the pre-trained generator and the convolutional kernel weights of the discriminator according to the composite loss value; The output module is used to stop the adversarial training of the generative adversarial network when the fluctuation amplitude of the composite loss value after a preset number of consecutive iterations is less than a preset threshold, and input the low-resolution image to be processed into the current pre-trained generator, and the current pre-trained generator outputs the optimized super-resolution image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images