Image generation method and device, computer equipment and computer readable storage medium

Abstract

The invention relates to the technical field of image processing, in particular to an image generation method and device, computer equipment and a computer readable storage medium. The method comprises the following steps: acquiring a first resolution image; performing a preset operation on the first resolution image to generate a target image; inputting the target image into an image recognition model comprising a generator and a discriminator for super-resolution processing, and determining a probability corresponding to each output image; taking the output image corresponding to the maximum probability as a second resolution image corresponding to the first resolution image; wherein the resolution of the second resolution image is greater than that of the first resolution image. By adopting the method, the quality of the reconstructed image can be improved, and meanwhile, the application scene of the SISR method based on deep learning is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image generation method, apparatus, computer device and computer-readable storage medium. Background Art

[0002] As a basic task in the field of computer vision, Single Image Super-resolution Reconstruction (SISR) technology has been widely used in monitoring, remote sensing, medical treatment and other fields.

[0003] The existing single image super-resolution enhancement methods mainly include traditional SISR methods and deep learning-based SISR methods. Among them, the traditional SISR method uses a fixed mathematical method to predict and calculate the pixel values ​​of the interpolation point positions in the super-resolution image based on the known pixels in the low-resolution image; with the rise of deep learning, the SISR method based on deep learning has greatly improved the fitting ability of the model. Compared with the traditional image SISR method, the reconstructed image has a better effect. It not only completes the task of high-scale super-resolution reconstruction, but also obtains high-quality results.

[0004] However, traditional SISR methods suffer from the problem of poor quality of reconstructed images, while deep learning-based SISR methods suffer from heavy computational burden and overfitting risk of models due to a large number of parameters, which greatly limits their application in real-world scenarios.

[0005] Therefore, how to improve the application scenarios of deep learning-based SISR methods while improving the quality of reconstructed images has become an urgent problem to be solved. Summary of the invention

[0006] Based on this, it is necessary to provide an image generation method, device, computer equipment and computer-readable storage medium to improve the application scenarios of the deep learning-based SISR method while improving the quality of the reconstructed image in response to the above-mentioned technical problems.

[0007] The embodiment of the present application provides an image generation method, comprising: acquiring a first resolution image; performing a preset operation on the first resolution image to generate a target image; wherein the preset operation includes one or more of image cropping and down sampling; inputting the target image into an image recognition model including a generator and a discriminator for super-resolution processing, and determining the probability corresponding to each output image; wherein the generator includes an encoder and a decoder, the input of the generator is the target image, the output of the generator is the input of the discriminator, the output of the discriminator is the probability corresponding to each output image, the encoder includes one or more 3×3 standard convolution kernels, one or more multi-scale feature fusion The invention relates to a method for realizing a novel neural network based on a neural network, comprising a BMFB module using a combined technology, one or more depth-separable convolution BSConv modules, and one or more channel attention mechanism ECA modules, the decoder comprising at least one upsampling module, and at least one 3×3 standard convolution kernel, the discriminator comprising multiple 3×3 standard convolution kernels, multiple 4×4 convolution kernels, multiple batch normalization BN layers, a flat layer and a classifier, the activation functions of the 3×3 standard convolution kernel and the BN layer are both Gaussian error linear unit GELU functions; the output image corresponding to the maximum probability is used as the second resolution image corresponding to the first resolution image; wherein the resolution of the second resolution image is greater than that of the first resolution image.

[0008] In one embodiment, before acquiring the first resolution image, the image generation method provided by the present disclosure further includes: acquiring a training sample image and a labeling result of the training sample image; wherein the training sample image includes a training image having a resolution less than a preset resolution, and the labeling result includes a labeling image corresponding to the training image, and the resolution of the labeling image is greater than the resolution of the training image;

[0009] Performing preset operations on the training sample image to obtain a processed image;

[0010] Inputting the processed image into the generator for learning, and obtaining a first learning image corresponding to the processed image;

[0011] Based on the mean absolute error between the first learning image and the labeled image, a first loss value for the current time is obtained;

[0012] When the current first loss value is less than the previous first loss value, the network parameters and the first actual weight value of the generator are adjusted based on the current first loss value, and the preset operation is performed on the next training sample image until the first loss value corresponding to the next training sample image is less than or equal to the first target loss value, and the generator is determined to be converged, and the first actual weight value corresponding to the generator when the generator converges is used as the first weight value corresponding to the generator;

[0013] Input the training sample image into the converged generator to obtain a second learning image corresponding to the training sample image;

[0014] Inputting the second learning image into the discriminator for learning, and obtaining a plurality of third learning images after the discriminator has learned the second learning image;

[0015] Calculate the second loss value of the labeled images corresponding to the plurality of third learning images and the training sample images at the current time based on the relative average discriminator;

[0016] Based on the second loss value of the current time, re-determine the third loss value of the current time corresponding to the generator;

[0017] When the third loss value of the current time is less than the third loss value of the previous time, and the second loss value of the current time is less than the second loss value of the previous time, adjust the first weight value, the network parameter of the discriminator and the second actual weight value, and perform a prediction operation on the next training sample image until the second loss value corresponding to the next training sample image is less than or equal to the second target loss value, and the third loss value is less than or equal to the third target loss value, determine that the discriminator converges, and obtain the adjusted first weight value and the adjusted second actual weight value;

[0018] The weight value of the generator is set to the adjusted first weight value, and the weight value of the discriminator is set to the adjusted second actual weight value, to obtain an image recognition model.

[0019] In one embodiment, the sum of the adjusted first weight value and the adjusted second actual weight value is equal to 1.

[0020] In one embodiment, calculating the second loss value of the labeled image corresponding to the plurality of third learning images and the training sample image at the current time based on the relative average discriminator includes:

[0021] Input the labeled images corresponding to the plurality of third learning images and the training sample images into the relative average discriminator for calculation to obtain the second loss value of the current time; wherein the relative average discriminator includes:

[0022] D Ra (x r , x f )=σ(C(x r )-E xf [C(xf)]);

[0023] Among them, D Ra (x r , x f ) represents the second loss value of the current time, x r represents the labeled image corresponding to the training sample image, x frepresents the second learning image output by the generator, σ is the Sigmoid function, C(xf) represents the third learning image output by the discriminator, E xf [C(xf)] represents an operation of taking the average value of all third learning images output by the discriminator.

[0024] In one embodiment, the current second loss value is input into the loss formula for calculation, and the current third loss value corresponding to the generator is re-determined; wherein the loss formula includes:

[0025]

[0026] in, represents the third loss value, D Ra (x r , x f ) represents the second loss value, E xr [log(D Ra (x r , x f ))] represents the operation of averaging the second loss value of the discriminator.

[0027] The embodiment of the present application provides an image generation device, including: an acquisition module, used to acquire a first resolution image; a processing module, used to perform a preset operation on the first resolution image acquired by the acquisition module to generate a target image; wherein the preset operation includes one or more of image cropping and down sampling; the processing module is also used to input the target image into an image recognition model including a generator and a discriminator for super-resolution processing, and determine the probability corresponding to each output image; wherein the generator includes an encoder and a decoder, the input of the generator is the target image, the output of the generator is the input of the discriminator, the output of the discriminator is the probability corresponding to each output image, and the encoder includes one or more 3×3 standard convolution kernels, one or more The invention discloses a method for processing a BMFB module that adopts multi-scale feature fusion technology, one or more depth-separable convolution BSConv modules, and one or more channel attention mechanism ECA modules. The decoder includes at least one upsampling module and at least one 3×3 standard convolution kernel. The discriminator includes multiple 3×3 standard convolution kernels, multiple 4×4 convolution kernels, multiple batch normalization BN layers, a flat layer and a classifier. The activation functions of the 3×3 standard convolution kernel and the BN layer are both Gaussian error linear unit GELU functions. The processing module is also used to use the output image corresponding to the maximum probability as the second resolution image corresponding to the first resolution image. The resolution of the second resolution image is greater than that of the first resolution image.

[0028] In one embodiment, the acquisition module is further used to acquire a training sample image and a marking result of the training sample image; wherein the training sample image includes a training image having a resolution less than a preset resolution, and the marking result includes a marked image corresponding to the training image, and the resolution of the marked image is greater than the resolution of the training image;

[0029] The processing module is also used to perform a preset operation on the training sample image acquired by the acquisition module to obtain a processed image;

[0030] The processing module is further used to input the processed image into the generator for learning, so as to obtain a first learning image corresponding to the processed image;

[0031] The processing module is further used to obtain a first loss value for a current time based on a mean absolute error between the first learning image and the labeled image;

[0032] The processing module is further used to adjust the network parameters and the first actual weight value of the generator based on the current first loss value when the current first loss value is less than the previous first loss value, and perform a preset operation on the next training sample image until the first loss value corresponding to the next training sample image is less than or equal to the first target loss value, determine that the generator converges, and use the first actual weight value corresponding to the generator when the generator converges as the first weight value corresponding to the generator;

[0033] The processing module is further used to input the training sample image into the converged generator to obtain a second learning image corresponding to the training sample image;

[0034] The processing module is further used to input the second learning image into the discriminator for learning, and obtain a plurality of third learning images after the discriminator learns the second learning image;

[0035] The processing module is further used to calculate the second loss value of the labeled images corresponding to the plurality of third learning images and the training sample images at the current time based on the relative average discriminator;

[0036] The processing module is further used to re-determine the third loss value of the current time corresponding to the generator based on the second loss value of the current time;

[0037] The processing module is further used to adjust the first weight value, the network parameters of the discriminator and the second actual weight value when the third loss value of the current time is less than the third loss value of the previous time and the second loss value of the current time is less than the second loss value of the previous time, and perform a prediction operation on the next training sample image until the second loss value corresponding to the next training sample image is less than or equal to the second target loss value and the third loss value is less than or equal to the third target loss value, determine that the discriminator converges, and obtain the adjusted first weight value and the adjusted second actual weight value;

[0038] The processing module is also used to set the weight value of the generator to the adjusted first weight value, and set the weight value of the discriminator to the adjusted second actual weight value to obtain an image recognition model.

[0039] In one embodiment, the sum of the adjusted first weight value and the adjusted second actual weight value is equal to 1.

[0040] In one embodiment, the processing module is specifically used to input the labeled images corresponding to the plurality of third learning images and the training sample images into the relative average discriminator for calculation to obtain the second loss value of the current time; wherein the relative average discriminator includes:

[0041] D Ra (x r , x f )=σ(C(x r )-E xf [C(xf)]);

[0042] Among them, D Ra (x r , x f ) represents the second loss value of the current time, x r represents the labeled image corresponding to the training sample image, x f represents the second learning image output by the generator, σ is the Sigmoid function, C(xf) represents the third learning image output by the discriminator, E xf [C(xf)] represents an operation of taking the average value of all third learning images output by the discriminator.

[0043] In one embodiment, the processing module is specifically used to input the current second loss value into the loss formula for calculation, and re-determine the current third loss value corresponding to the generator; wherein the loss formula includes:

[0044]

[0045] in, represents the third loss value, D Ra (x r , x f ) represents the second loss value, EE xr [log(D Ra (x r , x f ))] represents the operation of averaging the second loss value of the discriminator.

[0046] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the image generation method provided in any embodiment of the present application are implemented.

[0047] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image generation method provided in any embodiment of the present application are implemented.

[0048] The image generation method, apparatus, computer device, and computer-readable storage medium provided in the embodiments of the present application are derived using the unique technical features of the image generation method to achieve the beneficial effect of solving the technical problems raised in the background technology.

[0049] The image generation method provided by the present application generates a target image by performing a preset operation on the first resolution image when the first resolution image is processed by SISR technology; then, the target image is input into an image recognition model including a generator and a discriminator for super-resolution processing to determine the probability corresponding to each output image; since the image recognition model uses one or more 3×3 standard convolution kernels, one or more BMFB modules using multi-scale feature fusion technology, one or more depth-separable convolution BSConv modules, and one or more channel attention mechanism ECA modules instead of the traditional BN layer, such a design not only improves the reuse rate of features, but also avoids the interference of the BN layer on the feature space, saving storage space. At the same time, the activation functions of both the 3×3 standard convolution kernel and the BN layer are Gaussian error linear unit GELU functions, which enhances the nonlinear factors of the mapping process and makes it more suitable for the BSConv module. At the same time, by introducing the ECA module, the image recognition model can repeatedly capture the feature relationship of each channel without adding a large number of parameters. At the same time, the BSConv module decomposes the standard convolution into a 1×1 convolution in the point direction and a convolution in the depth direction, thereby reducing the dimension of the data output by the generator. Since the image recognition model actually requires less storage space and less computation, it can be applied to more scenarios. Afterwards, the output image corresponding to the maximum probability is used as the second resolution image corresponding to the first resolution image. At the same time, since the image recognition model can extract more image features, the quality of the second resolution image can be improved. In this way, the SISR technology processing of the first resolution image can be realized. The problem of how to improve the application scenarios of the SISR method based on deep learning while improving the quality of the reconstructed image is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 An application scenario diagram of an image generation method in an embodiment;

[0051] Figure 2 is one of the flowchart diagrams of the image generating method in one embodiment;

[0052] Figure 3 A schematic diagram of a network structure of a generator of an image generation method in one embodiment;

[0053] Figure 4 A schematic diagram of a network structure of a BRRMB of a generator of an image generation method in one embodiment;

[0054] Figure 5 A schematic diagram of a network structure of a BRMB of a generator of an image generation method in one embodiment;

[0055] Figure 6 FIG2 is a second flow chart of an image generating method in an embodiment;

[0056] Figure 7 The third flowchart of the image generating method in one embodiment;

[0057] Figure 8 FIG4 is a fourth flow chart of an image generating method in an embodiment;

[0058] Fig. 9 is a schematic structural diagram of an image generating device in one embodiment;

[0059] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] BSConv in the disclosed embodiment: the full name (Bilateral Slice Convolution) is a convolution operation used for point cloud data processing, mainly used to improve the effect of point cloud feature extraction.

[0062] ESRGAN in the embodiments of the present disclosure: the full name (Enhanced Super-Resolution Generative Adversarial Network) is an advanced super-resolution generative adversarial network used to convert low-resolution (LR) images into high-resolution (HR) images.

[0063] ECA in the disclosed embodiments: full name (Efficient Channel Attention) is a channel attention mechanism for improving the performance of convolutional neural networks (CNNs). It reduces computational complexity while maintaining good performance by improving the channel attention mechanism.

[0064] The GELU activation layer in the embodiment of the present disclosure: the full name (Gaussian Error Linear Unit) is a nonlinear activation function widely used in deep learning.

[0065] The relative average discriminator in the disclosed embodiment is an improved discriminator design in generative adversarial networks (GANs) that aims to improve the stability of training and the quality of generated images. Its main idea is to transform the output of the discriminator from absolute values ​​to relative values. This design helps to improve the adversarial relationship between the generator and the discriminator during training.

[0066] Mini-batch in the embodiments of the present disclosure is a concept commonly used in machine learning, especially deep learning. When training a neural network, we usually do not process the entire data set at once, but divide the data into smaller batches for processing. Each such batch is called a mini-batch.

[0067] Deep-wise Convolution in the embodiments of the present disclosure: This is an important concept in deep learning, usually referred to as Depthwise Convolution. It is an effective convolution operation method used to reduce the amount of calculation and improve network efficiency.

[0068] The BMFB module (Bilateral and Multi-scale Feature Block) in the disclosed embodiment is a module in a convolutional neural network that aims to improve the effect of feature extraction and fusion, especially when processing complex image tasks. The BMFB module combines the advantages of bilateral feature extraction and multi-scale feature fusion, which can effectively improve the performance of the network.

[0069] The batch normalization BN layer in the embodiment of the present disclosure, whose full name is Batch Normalization, is a technology used in deep learning, which aims to solve the problem of internal covariate shift.

[0070] The image generation method provided in this application can be applied to Figure 1In the application environment shown in FIG. 1 . The image generation method is applied to an image generation system. The image generation system includes a terminal 102 and a server 104. The terminal 102 and the server 104 communicate through a network. When a user uses the terminal 102, the user can perform a super-resolution processing operation on the first resolution image selected in the terminal 102. At this time, the terminal 102 responds to the user performing a super-resolution processing operation on the selected low-resolution image, and the terminal 102 sends the first resolution image to the server 104. After receiving the first resolution image transmitted by the terminal 102, the server 104 performs a preset operation on the first resolution image to generate a target image; the server 104 inputs the target image into an image recognition model including a generator and a discriminator for super-resolution processing, and determines the probability corresponding to each output image; the server 104 uses the output image corresponding to the maximum probability as the second resolution image corresponding to the first resolution image. After that, the server 104 sends the second resolution image to the terminal 102. After receiving the second resolution image sent by the server 104, the terminal 102 displays it. Since the first resolution image and the second resolution image display the same content, and the resolution of the second resolution image is greater than that of the first resolution image, the user can see the first resolution image with a higher resolution, thereby improving the user experience. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0071] In one embodiment, Figure 2 As shown, a method for generating an image is provided. This embodiment mainly applies the method to Figure 1 Take server 104 in FIG. 1 as an example.

[0072] S11. Acquire a first resolution image.

[0073] The first resolution image is an image with a resolution smaller than a preset resolution.

[0074] S12. Perform a preset operation on the first resolution image to generate a target image; wherein the preset operation includes one or more of image cropping and down sampling.

[0075] S13. Input the target image into an image recognition model including a generator and a discriminator for super-resolution processing, and determine the probability corresponding to each output image; wherein the generator includes an encoder and a decoder, the input of the generator is the target image, the output of the generator is the input of the discriminator, and the output of the discriminator is the probability corresponding to each output image. The encoder includes one or more 3×3 standard convolution kernels, one or more BMFB modules using multi-scale feature fusion technology, one or more depth-separable convolution BSConv modules, and one or more channel attention mechanism ECA modules. The decoder includes at least one upsampling module and at least one 3×3 standard convolution kernel. The discriminator includes multiple 3×3 standard convolution kernels, multiple 4×4 convolution kernels, multiple batch normalization BN layers, a flat layer and a classifier. The activation functions of the 3×3 standard convolution kernel and the BN layer are both Gaussian Error Linear Units (GELU) functions.

[0076] Specifically, Figure 3 As shown in the figure, the image recognition model includes a generator and a discriminator. The generator adopts the residual structure of the residual, and its purpose is to produce realistic fake images HR to confuse them with real images. The discriminator is a classification network modified from the VGG (Visual Geometry Group Network) network, and its role is to identify the GT (ground truth) composed of the generator and the fake image.

[0077] Stage 1 is represented by a dotted line, and only the generator participates in training in stage 1.

[0078] Stage 2 is represented by a dotted line, which indicates adversarial training, where the generator and discriminator are trained together.

[0079] like Figure 3 As shown in the figure, the image recognition model is trained in two stages. In stage 1, the generator is trained by the L1 loss function. In this process, the discriminator is fixed, and the generator regards the image HR generated based on the training image LR as the real image GT, just like a traditional convolutional neural network. In the second stage, both the generator and the discriminator are trained by adversarial loss. By introducing the discriminator, the training results of the previous stage are used as pre-trained weights to complete the adversarial training.

[0080] For example, the encoder in the generator includes two 3×3 standard convolution kernels (Conv1 and Conv2), N BRRMB modules, and the decoder includes three upsampling modules (upsampling module 1, upsampling module 2, and upsampling module 3) and two 3×3 standard convolution kernels (Conv3 and Conv4). The network structure of the generator is as follows: Figure 4 shown.

[0081] Specifically, after receiving the first resolution image, the encoder convolves the first resolution image with Conv1 to obtain the first convolution image. Afterwards, Conv1 inputs the first convolution image into the BRRMB module 1 of the encoder for convolution to obtain the second convolution image. Afterwards, BRRMB module 1 inputs the second convolution image into the BRRMB module 2 of the encoder for convolution to obtain the third convolution image. Afterwards, BRRMB module N-1 inputs the nth convolution image into the BRRMB module N of the encoder for convolution to obtain the N+1th convolution image. Afterwards, BRRMB module N inputs the N+1th convolution image into Conv1 for convolution to obtain the N+2th convolution image. Afterwards, the encoder performs image fusion on the first convolution image and the N+2th convolution image to generate the N+3th convolution image, and uses the N+3th convolution image as the input of the decoder. Wherein, n∈[1,N], and N is an integer greater than or equal to 1, and N represents the total number of BRRMB modules contained in the encoder.

[0082] Exemplarily, the encoder may include 23 BRRMB modules.

[0083] Specifically, a BRRMB module is composed of one or more BRMB modules and a first residual scaling parameter β1. For example, taking a BRRMB module comprising three BRMB modules (BRMB module 1, BRMB module 2 and BRMB module 3) and a residual scaling parameter β1 as an example, the network structure of BRRMB module 1 is as follows: Figure 4 shown.

[0084] BRMB module 1 in BRRMB module 1 convolves the first convolution image input by Conv1 to obtain the n-1th target image; BRMB module 1 inputs the n-1th target image to BRMB module 2 in BRRMB module 1 for convolution to obtain the n-2th target image. BRMB module 2 inputs the n-2th target image to BRMB module 3 in BRRMB module 1 for convolution to obtain the n-3th target image. The n-3th target image is multiplied by the residual scaling parameter β1 to obtain the n-4th target image, and the n-4th target image is used as the N+1th convolution image.

[0085] It should be noted that the network structure of BRRMB module 1 is the same as the network structure of other BRRMB modules, which will not be repeated here.

[0086] Among them, for the BRRMB module n whose input image is not the first convolution image, the following operations are performed:

[0087] BRMB module 1 in BRRMB module n convolves the nth convolution image output by BRRMB module n-1 to obtain the n-1th target image; BRMB module 1 inputs the n-1th target image to BRMB module 2 in BRRMB module n for convolution to obtain the n-2th target image; BRMB module 2 inputs the n-2th target image to BRMB module 3 in BRRMB module n for convolution to obtain the n-3th target image; BRMB module 1 in BRRMB module n multiplies the n-3th target image by the residual scaling parameter β1 to obtain the n-4th target image, and uses the n-4th target image as the n+1th convolution image.

[0088] Exemplarily, β1 may be 0.2.

[0089] Specifically, a BRMB module consists of one or more BMFB modules that use multi-scale feature fusion technology, a deep separable convolution BSConv module, and a channel attention mechanism ECA module. For example, taking a BRMB module consisting of BMFB modules (BMFB module 1, BMFB module 2, BMFB module 3 and BMFB module 4), a BSConv module 1, an ECA module and a residual scaling parameter β2 as an example, the network structure of BRMB module 1 in BRRMB module 1 is as follows: Figure 4 shown.

[0090] Exemplarily, β2 may be 0.2.

[0091] BRMB module 1 in BRRMB module 1 uses the first convolution image as input to BMFB module 1, BMFB module 2, BMFB module 3, BMFB module 4, and BSConv module 1 in BRMB module 1 in BRRMB module 1, respectively;

[0092] The output of BMFB module 1 in BRMB module 1 in BRRMB module 1 is used as the input of BMFB module 2, BMFB module 3, BBMFB module 4, and BSConv module 1 in BRMB module 1 in BRRMB module 1;

[0093] The output of BMFB module 2 in BRMB module 1 in BRRMB module 1 is used as the input of BMFB module 3, BMFB module 4, and BSConv module 1 in BRMB module 1 in BRRMB module 1;

[0094] The output of BMFB module 3 in BRMB module 1 in BRRMB module 1 serves as the input of BMFB module 4 in BRMB module 1 in BRRMB module 1 and BSConv module 1.

[0095] BMFB module 1 in BRMB module 1 in BRRMB module 1 convolves the first convolution image input by Conv1 to obtain the n-1-1th processed image;

[0096] The BMFB module 2 in the BRMB module 1 in the BRRMB module 1 convolves the n-1-1th processed image and the first convolution image to obtain the n-1-2th processed image;

[0097] The BMFB module 3 in the BRMB module 1 in the BRRMB module 1 convolves the n-1-1th processed image, the n-1-2th processed image and the first convolution image to obtain the n-1-3th processed image;

[0098] The BMFB module 4 in the BRMB module 1 in the BRRMB module 1 convolves the n-1-1th processed image, the n-1-2th processed image, the n-1-3th processed image and the first convolution image to obtain the n-1-4th processed image;

[0099] The BSConv module 1 in the BRMB module 1 in the BRRMB module 1 convolves the n-1-1th processed image, the n-1-2th processed image, the n-1-3th processed image, the n-1-4th processed image and the first convolution image to obtain the n-1-5th processed image;

[0100] The BSConv module 1 in the BRMB module 1 in the BRRMB module 1 inputs the n-1-5th processed image into the ECA module in the BRMB module 1 in the BRRMB module 1 for feature collection to obtain the n-1-6th processed image.

[0101] In the BRRMB module 1, the BRMB module 1 multiplies the n-1-6th processed image by the residual scaling parameter β2 to obtain the n-1-7th target image.

[0102] The BRRMB module 1 in the BRMB module 1 performs image fusion on the n-1-7th target image and the first convolution image to obtain the n-1-8th target image, and uses the n-1-8th processed image as the n-1th target image.

[0103] The BMFB module 1 in the BRMB module 2 in the BRRMB module 1 performs convolution on the n-1th target image input by the BRMB module 1 in the BRRMB module 1 to obtain the n-2-1th processed image;

[0104] The BMFB module 2 in the BRMB module 2 in the BRRMB module 1 convolves the n-2-1th processed image and the n-1th target image to obtain the n-2-2th processed image;

[0105] The BMFB module 3 in the BRMB module 2 in the BRRMB module 1 convolves the n-2-1th processed image, the n-2-2th processed image, and the n-1th target image to obtain the n-2-3th processed image;

[0106] The BMFB module 4 in the BRMB module 2 in the BRRMB module 1 convolves the n-2-1th processed image, the n-2-2th processed image, the n-2-3th processed image, and the n-1th target image to obtain the n-2-4th processed image;

[0107] The BSConv module 1 in the BRMB module 2 in the BRRMB module 1 performs convolution on the n-2-1th processed image, the n-2-2th processed image, the n-2-3th processed image, the n-2-4th processed image and the n-1th target image to obtain the n-2-5th processed image;

[0108] The BSConv module 1 in the BRMB module 2 in the BRRMB module 1 inputs the n-2-5th processed image to the ECA module in the BRMB module 2 in the BRRMB module 1 for feature collection to obtain the n-2-6th processed image.

[0109] In the BRRMB module 1, the BRMB module 2 multiplies the n-2-6th processed image by the residual scaling parameter β2 to obtain the n-2-7th target image.

[0110] The BRMB module 2 in the BRRMB module 1 performs image fusion on the n-2-7th target image and the n-1th target image to obtain the n-2-8th target image, and uses the n-2-8th processed image as the n-2th target image.

[0111] The BMFB module 1 in the BRMB module 3 in the BRRMB module 1 performs convolution on the n-2th target image input by the BRMB module 2 in the BRRMB module n to obtain the n-3-1th processed image;

[0112] The BMFB module 2 in the BRMB module 3 in the BRRMB module 1 convolves the n-3-1th processed image and the n-2th target image to obtain the n-3-2th processed image;

[0113] The BMFB module 3 in the BRMB module 3 in the BRRMB module 1 convolves the n-3-1th processed image, the n-3-2th processed image, and the n-2th target image to obtain the n-3-3th processed image;

[0114] The BMFB module 4 in the BRMB module 3 in the BRRMB module 1 convolves the n-3-1th processed image, the n-3-2th processed image, the n-3-3th processed image, and the n-2th target image to obtain the n-3-4th processed image;

[0115] The BSConv module 1 in the BRMB module 3 in the BRRMB module 1 performs convolution on the n-3-1th processed image, the n-3-2th processed image, the n-3-3th processed image, the n-3-4th processed image and the n-2th target image to obtain the n-3-5th processed image;

[0116] The BSConv module 1 in the BRMB module 3 in the BRRMB module 1 inputs the n-3-5th processed image into the ECA module for feature collection to obtain the n-3-6th processed image.

[0117] BRMB module 3 in BRRMB module 1 multiplies the n-3-6th processed image by the residual scaling parameter β2 to obtain the n-3-7th target image.

[0118] The BRMB module 3 in the BRRMB module 1 performs image fusion on the n-3-7th target image and the n-2th target image to obtain the n-3-8th target image, and uses the n-3-8th processed image as the n-3th target image.

[0119] It should be noted that the network structure of BRMB module 1 is the same as the network structure of other BRMB modules, which will not be repeated here.

[0120] Among them, for the BRRMB module n whose input is not the first convolution image, the following operations are performed:

[0121] BMFB module 1 in BRMB module 1 in BRRMB module n convolves the nth convolution image output by BRRMB module n-1 to obtain the n-1-1th processed image;

[0122] The BMFB module 2 in the BRMB module 1 in the BRRMB module n convolves the n-1-1th processed image and the nth convolution image to obtain the n-1-2th processed image;

[0123] The BMFB module 3 in the BRMB module 1 in the BRRMB module n convolves the n-1-1th processed image, the n-1-2th processed image and the nth convolution image to obtain the n-1-3th designated image;

[0124] The BMFB module 4 in the BRMB module 1 in the BRRMB module n convolves the n-1-1th processed image, the n-1-2th processed image, the n-1-3th processed image and the nth convolution image to obtain the n-1-4th processed image;

[0125] BSConv module 1 in BRMB module 1 in BRRMB module n convolves the n-1-1th processed image, the n-1-2th processed image, the n-1-3th processed image, the n-1-4th processed image and the nth convolution image to obtain the n-1-5th processed image;

[0126] The BSConv module 1 in the BRMB module 1 in the BRRMB module n inputs the n-1-5th processed image to the ECA module in the BRMB module 1 in the BRRMB module n for feature collection to obtain the n-1-6th processed image.

[0127] BRMB module 1 in BRRMB module n multiplies the n-1-6th processed image by the residual scaling parameter β2 to obtain the n-1-7th processed image.

[0128] BRMB module 1 in BRRMB module n performs image fusion on the n-1-7th processed image and the n-th convolution image to obtain the n-1-8th processed image, and uses the n-1-8th processed image as the n-1th target image.

[0129] BMFB module 1 in BRMB module 2 in BRRMB module n performs convolution on the n-1th target image input by BRMB module 1 in BRRMB module n to obtain the n-2-1th processed image;

[0130] The BMFB module 2 in the BRMB module 2 in the BRRMB module n performs convolution on the n-2-1th processed image and the n-1th target image to obtain the n-2-2th processed image;

[0131] The BMFB module 3 in the BRMB module 2 in the BRRMB module n performs convolution on the n-2-1th processed image, the n-2-2th processed image and the n-1th target image to obtain the n-2-3th processed image;

[0132] The BMFB module 4 in the BRMB module 2 in the BRRMB module n performs convolution on the n-2-1th processed image, the n-2-2th processed image, the n-2-3th processed image and the n-1th target image to obtain the n-2-4th processed image;

[0133] The BSConv module 1 in the BRMB module 2 in the BRRMB module n performs convolution on the n-2-1th processed image, the n-2-2th processed image, the n-2-3th processed image, the n-2-4th processed image and the n-1th target image to obtain the n-2-5th processed image;

[0134] The BSConv module 1 in the BRMB module 2 in the BRRMB module n inputs the n-2-5th processed image to the ECA module in the BRMB module 2 in the BRRMB module n for feature collection to obtain the n-2-6th processed image.

[0135] The BRMB module 2 in the BRRMB module n multiplies the n-2-6th processed image by the residual scaling parameter β2 to obtain the n-2-7th target image.

[0136] The BRMB module 2 in the BRRMB module n performs image fusion on the n-2-7th target image and the n-1th target image to obtain the n-2-8th target image, and uses the n-2-8th processed image as the n-2th target image.

[0137] BMFB module 1 in BRMB module 3 in BRRMB module n performs convolution on the n-2th target image input by BRMB module 2 in BRRMB module n to obtain the n-3-1th processed image;

[0138] The BMFB module 2 in the BRMB module 3 in the BRRMB module n performs convolution on the n-3-1th processed image and the n-2th target image to obtain the n-3-2th processed image;

[0139] The BMFB module 3 in the BRMB module 3 in the BRRMB module n performs convolution on the n-3-1th processed image, the n-3-2th processed image and the n-2th target image to obtain the n-3-3th processed image;

[0140] The BMFB module 4 in the BRMB module 3 in the BRRMB module n performs convolution on the n-3-1th processed image, the n-3-2th processed image, the n-3-3th processed image, and the n-2th target image to obtain the n-3-4th processed image;

[0141] The BSConv module 1 in the BRMB module 3 in the BRRMB module n performs convolution on the n-3-1th processed image, the n-3-2th processed image, the n-3-3th processed image, the n-3-4th processed image and the n-2th target image to obtain the n-3-5th processed image;

[0142] The BSConv module in the BRMB module 3 in the BRRMB module n inputs the n-3-5th processed image into the ECA module for feature collection to obtain the n-3-6th processed image.

[0143] BRMB module 3 in BRRMB module n multiplies the n-3-6th processed image by the residual scaling parameter β2 to obtain the n-3-7th target image.

[0144] BRMB module 3 in BRRMB module n performs image fusion on the n-3-7th target image and the n-2th target image to obtain the n-3-8th target image, and uses the n-3-8th processed image as the n-3th target image.

[0145] Specifically, a BMFB module is composed of multiple depth-separable convolutional BSConv modules. For example, a BMFB module includes three BSConv modules, namely BSConv module 2, BSConv module 3 and BSConv module 4, and the activation functions of BSConv module 3 and BSConv module 4 are both GELU functions. The network structure of BMFB module 1 is as follows: Figure 4 shown.

[0146] BSConv module 2 in BMFB module 1 in BRMB module 1 in BRRMB module 1 convolves the first convolution image input by Conv1 to obtain the 1-1-1-2 preset image;

[0147] The BSConv module 3 in the BMFB module 1 in the BRMB module 1 in the BRRMB module 1 convolves the 1-1-1-2th input of the BSConv module 2 to obtain the 1-1-1-3th preset image, and multiplies the 1-1-1-3th preset image with the GELU function to obtain the 1-1-1-5th preset image.

[0148] The BSConv module 4 in the BMFB module 1 in the BRMB module 1 in the BRRMB module 1 convolves the first convolution image input by Conv1 to obtain the 1-1-1-4th preset image, and multiplies the 1-1-1-4th preset image with the GELU function to obtain the 1-1-1-6th preset image.

[0149] The BMFB module 1 in the BRMB module 1 in the BRRMB module 1 superimposes the 1-1-1-5th preset image and the 1-1-1-6th preset image to obtain the n-1-1th processed image.

[0150] It should be noted that the network structure of BMFB module 1 is the same as the network structure of other BMFB modules, which will not be repeated here.

[0151] Among them, for the BRRMB module n whose input is not the first convolution image, the following operations are performed:

[0152] Specifically, after receiving the N+3th convolution image output by the encoder, the decoder upsamples the N+3th convolution image through the upsampling module 1 of the encoder to obtain a first upsampled image. At the same time, the N+3th convolution image is used as the input of the upsampling module 2 and the upsampling module 3 of the encoder respectively. The upsampling module 1 inputs the first upsampled image to the upsampling module 2. The upsampling module 2 upsamples both the first upsampled image and the N+3th convolution image to obtain a second upsampled image. The upsampling module 2 inputs the second upsampled image to the upsampling module 3. The upsampling module 2 upsamples both the second upsampled image and the N+3th convolution image to obtain a third upsampled image. The upsampling module 3 inputs the third upsampled image to Conv3 of the decoder for convolution to obtain the N+4th convolution image. Conv3 inputs the N+4th convolution image to Conv4 for convolution to obtain the N+5th convolution image. The decoder uses the N+5th convolution image as the input of the discriminator.

[0153] Exemplarily, the discriminator includes 1 first structure, 5 second structures (second structure 1, second structure 2, second structure 3, second structure 4 and second structure 5 respectively), 1 third structure, 1 flat layer and a classifier, the first structure includes 1 3×3 standard convolution kernel (such as Conv5), and the activation function of Conv5 is the GELU function, each second structure includes 1 4×4 convolution kernel (such as Conv6), 1 3×3 standard convolution kernel (such as Conv7) and 2 BN layers (BN layer 1 and BN layer 2 respectively), and the activation function of the BN layer is the GELU function, and the third structure includes 1 4×4 convolution kernel (such as Conv8) and 1 BN layer, and the activation function of the BN layer is the GELU function. For example, the network structure of the discriminator is as follows: Figure 5 shown.

[0154] Conv5 of the first structure in the discriminator convolves the N+5th convolution image input by the generator to obtain the N+6th convolution image, and multiplies the N+6th convolution image with the GELU function to obtain the N+7th convolution image. The first structure in the discriminator inputs the N+7th convolution image to the second structure 1.

[0155] Conv6 in the second structure 1 in the discriminator convolves the N+7th convolution image input by the first structure to obtain the N+8th convolution image, and inputs the N+8th convolution image into the BN layer 1 in the second structure 1 in the discriminator for normalization to obtain the N+9th convolution image, and multiplies the N+9th convolution image with the GELU function to obtain the N+10th convolution image; then, the BN layer 1 in the second structure 1 in the discriminator inputs the N+10th convolution image into Conv7 in the second structure 1 in the discriminator. Conv7 in the second structure 1 in the discriminator convolves the N+10th convolution image to obtain the N+11th convolution image, and inputs the N+11th convolution image into the BN layer 2 in the second structure 1 in the discriminator for normalization to obtain the N+12th convolution image, and multiplies the N+12th convolution image with the GELU function to obtain the N+13th convolution image.

[0156] The second structure 1 in the discriminator inputs the N+13th convolution image to the second structure 2 in the discriminator.

[0157] Conv6 in the second structure 2 in the discriminator convolves the N+13th convolution image input by the first structure to obtain the N+14th convolution image, and inputs the N+14th convolution image to the BN layer 1 in the second structure 2 in the discriminator for normalization to obtain the N+15th convolution image, and multiplies the N+15th convolution image with the GELU function to obtain the N+16th convolution image; then, the BN layer 1 in the second structure 2 in the discriminator inputs the N+16th convolution image to Conv7 in the second structure 2 in the discriminator. Conv7 in the second structure 2 in the discriminator convolves the N+16th convolution image to obtain the N+17th convolution image, and inputs the N+17th convolution image to the BN layer 2 in the second structure 2 in the discriminator for normalization to obtain the N+18th convolution image, and multiplies the N+18th convolution image with the GELU function to obtain the N+19th convolution image.

[0158] The second structure 2 in the discriminator inputs the N+19th convolution image to the second structure 3 in the discriminator.

[0159] Conv6 in the second structure 3 in the discriminator convolves the N+19th convolution image input by the first structure to obtain the N+20th convolution image, and inputs the N+20th convolution image to the BN layer 1 in the second structure 3 in the discriminator for normalization to obtain the N+21th convolution image, and multiplies the N+21th convolution image with the GELU function to obtain the N+22th convolution image; then, the BN layer 1 in the second structure 3 in the discriminator inputs the N+22th convolution image to Conv7 in the second structure 3 in the discriminator. Conv7 in the second structure 3 in the discriminator convolves the N+22th convolution image to obtain the N+23th convolution image, and inputs the N+23th convolution image to the BN layer 2 in the second structure 3 in the discriminator for normalization to obtain the N+24th convolution image, and multiplies the N+24th convolution image with the GELU function to obtain the N+25th convolution image.

[0160] The second structure 3 in the discriminator inputs the N+25th convolution image to the second structure 4 in the discriminator.

[0161] Conv6 in the second structure 4 in the discriminator convolves the N+25th convolution image input by the first structure to obtain the N+26th convolution image, and inputs the N+26th convolution image to the BN layer 1 in the second structure 4 in the discriminator for normalization to obtain the N+27th convolution image, and multiplies the N+27th convolution image with the GELU function to obtain the N+28th convolution image; then, the BN layer 1 in the second structure 4 in the discriminator inputs the N+29th convolution image to Conv7 in the second structure 4 in the discriminator. Conv7 in the second structure 4 in the discriminator convolves the N+27th convolution image to obtain the N+28th convolution image, and inputs the N+28th convolution image to the BN layer 2 in the second structure 4 in the discriminator for normalization to obtain the N+29th convolution image, and multiplies the N+29th convolution image with the GELU function to obtain the N+30th convolution image.

[0162] The second structure 4 in the discriminator inputs the N+30th convolution image to the second structure 5 in the discriminator.

[0163] Conv6 in the second structure 5 in the discriminator convolves the N+30th convolution image input by the first structure to obtain the N+31th convolution image, and inputs the N+31th convolution image to the BN layer 1 in the second structure 5 in the discriminator for normalization to obtain the N+32th convolution image, and multiplies the N+32th convolution image with the GELU function to obtain the N+33th convolution image; then, the BN layer 1 in the second structure 5 in the discriminator inputs the N+33th convolution image to Conv7 in the second structure 5 in the discriminator. Conv7 in the second structure 5 in the discriminator convolves the N+33th convolution image to obtain the N+34th convolution image, and inputs the N+34th convolution image to the BN layer 2 in the second structure 5 in the discriminator for normalization to obtain the N+35th convolution image, and multiplies the N+35th convolution image with the GELU function to obtain the N+36th convolution image.

[0164] The second structure 5 in the discriminator inputs the N+36th convolution image to the third structure in the discriminator.

[0165] Conv8 in the third structure in the discriminator convolves the N+36th convolution image input by the second structure 5 in the discriminator to obtain the N+37th convolution image, and inputs the N+37th convolution image into the BN layer in the third structure in the discriminator for normalization to obtain the N+38th convolution image, and multiplies the N+38th convolution image with the GELU function to obtain the N+39th convolution image.

[0166] The third structure in the discriminator inputs the N+39th convolution image into the flat layer in the discriminator for aggregation, and obtains the actual convolution image output by each channel. After that, the flat layer in the discriminator inputs the actual convolution image output by each channel into the classifier for classification, and obtains the probability corresponding to each actual convolution image.

[0167] S14 uses the output image corresponding to the maximum probability as the second resolution image corresponding to the first resolution image; wherein the first resolution image and the second resolution image display the same content, and the resolution of the second resolution image is greater than that of the first resolution image.

[0168] In the above-mentioned image generation method, the unique technical features of the image generation method are used for deduction, thereby achieving the beneficial effect of being able to solve the technical problems raised in the background technology.

[0169] As can be seen from the above, the image generation method provided by the embodiment of the present application introduces one or more 3×3 standard convolution kernels, one or more BMFB modules using multi-scale feature fusion technology, one or more depth-separable convolution BSConv modules, and one or more channel attention mechanism ECA modules in the image recognition model, instead of the traditional BN layer. Such a design not only improves the reuse rate of features, but also avoids the interference of the BN layer on the feature space, saving storage space. At the same time, the activation functions of both the 3×3 standard convolution kernel and the BN layer are Gaussian error linear unit GELU functions, which enhances the nonlinear factors of the mapping process and makes it more suitable for the BSConv module. At the same time, by introducing the ECA module, the image recognition model can repeatedly capture the feature relationship of each channel without adding a large number of parameters. At the same time, the BSConv module decomposes the standard convolution into a 1×1 convolution in the point direction and a convolution in the depth direction, thereby reducing the dimension of the data output by the generator. Since the actual storage space required for the image recognition model is reduced and the amount of calculation is reduced, more scenarios can be applied.

[0170] In one embodiment, in combination Figure 2 ,like Figure 6 As shown, before executing S11, the image generating method provided by the embodiment of the present disclosure further includes S15-S25.

[0171] S15, obtaining a training sample image and a marking result of the training sample image; wherein the training sample image includes a training image having a resolution less than a preset resolution, and the marking result includes a marked image corresponding to the training image, and the resolution of the marked image is greater than the resolution of the training image;

[0172] S16, performing a preset operation on the training sample image to obtain a processed image;

[0173] S17, inputting the processed image into the generator for learning, to obtain a first learning image corresponding to the processed image;

[0174] S18, obtaining a first loss value for the current time based on the mean absolute error between the first learning image and the labeled image;

[0175] S19, when the current first loss value is less than the previous first loss value, adjust the network parameters and the first actual weight value of the generator based on the current first loss value, and perform a preset operation on the next training sample image until the first loss value corresponding to the next training sample image is less than or equal to the first target loss value, determine that the generator converges, and use the first actual weight value corresponding to the generator when the generator converges as the first weight value corresponding to the generator;

[0176] In some examples, when training an image recognition model, the generator of the image recognition model is trained first, and the discriminator does not participate in the training. When training the generator, it is first necessary to perform a preset operation on the training sample image (GT), such as the preset operation including one or more of image cropping and down sampling. The processed image (GT) after the preset operation cut ) will be used as the low-resolution image LR cut Then, the low-resolution image LR cut Assign to the training group epoch number k, and use epoch number k low-resolution image LR cut The generator in the image recognition model is trained and the loss value of the generator is calculated using the mean absolute error (L1 loss). The back-propagation process updates the weights of the generator in the image recognition model until the mean absolute error converges. Finally, the first weight value θ1 of the generator in the image recognition model is output.

[0177] Exemplarily, the process of training the generator is as follows:

[0178] Input: Ground truth (GT).

[0179] Output: model weight value θ1 at stage 1;

[0180] Requirements: Image cropping size: C, number of complete data set processing times: K, scale factor: S;

[0181] GT cut =Cut(GT,C);

[0182] LR cut = Downsampling (GT cut , S);

[0183] for(k=1;k≤K;k++)do;

[0184] Take gradient descent step on;

[0185] ▽θ|GT cut -Nete(LR cut , S)|;

[0186] end for

[0187] θ1←θ

[0188] Returns: θ1.

[0189] In this way, the first weight value θ1 of the generator in the image recognition model is obtained.

[0190] S20, inputting the training sample image into the converged generator to obtain a second learning image corresponding to the training sample image;

[0191] S21, inputting the second learning image into the discriminator for learning, and obtaining a plurality of third learning images after the discriminator learns the second learning image;

[0192] S22, calculating the second loss value of the labeled images corresponding to the plurality of third learning images and the training sample images at the current time based on the relative average discriminator;

[0193] S23, based on the current second loss value, re-determine the current third loss value corresponding to the generator;

[0194] S24, when the third loss value of the current time is less than the third loss value of the previous time, and the second loss value of the current time is less than the second loss value of the previous time, adjust the first weight value, the network parameters of the discriminator and the second actual weight value, and perform a prediction operation on the next training sample image, until the second loss value corresponding to the next training sample image is less than or equal to the second target loss value, and the third loss value is less than or equal to the third target loss value, determine that the discriminator converges, and obtain the adjusted first weight value and the adjusted second actual weight value;

[0195] In some examples, when training an image recognition model, after the generator converges, the generator and the discriminator are trained simultaneously. First, a preset operation needs to be performed on the training sample image (GT), such as one or more of image cropping and down sampling. The processed image (GT) after the preset operation cut ) will be used as the low-resolution image LR cut Then, the low-resolution image LR cut Assign it to the training group epoch number k, and use the epoch number k low-resolution image LR cut Input to the generator in the image recognition model. At the same time, the output of the generator is used as the input of the discriminator for training, and the loss value of the generator is calculated using the relative average discriminator (RaD). The back propagation process updates the weight value of the generator and the weight value of the discriminator in the image recognition model until the relative average discriminator converges. Finally, the first weight value θ of the generator in the image recognition model is output G and the second actual weight value δ after the discriminator is adjusted D .

[0196] For example, after the generator converges, the process of training the generator and the discriminator simultaneously is as follows:

[0197] Input: Ground truth (GT).

[0198] Output: The generator model weight is θ G , the discriminator model weight is δ D ;

[0199] Requirements: The weight of the stage1 model is θ1, and the weight of the generator pre-trained model is Net θpre ), image cropping size: C, number of complete data set processing times: K, scale factor: S.

[0200] GT cut =Cut(GT,C);

[0201] LR cut = Downsampling(GTcut, S);

[0202] Initialization:Netopre←θ1;

[0203] Nete←θ pre ;

[0204] for(n=1;k≤N;n++)do;

[0205] SR=Net θ (LR cut , S);

[0206] Take gradient descent step on

[0207] / / Compute the total loss by Eq.1:

[0208] Eq.1: ▽ δ (-E GTcut [log(D Raδ (GT cut ,SR))]-E SR [1-log(D Raδ (SR, GT cut ))]);

[0209] Take gradient descent step on

[0210] / / Compute the total loss by Eq.2:

[0211] Eq.2: ▽δ (-E GTcut [log(1-D Raδ (GT cut , Net θ (LR cut , S)))]-E SR [log(D Raδ (Net θ (LR cut , S), GT cut )]);

[0212] end for

[0213] θ G ←θ

[0214] δ D ←δ

[0215] Returns: θ G , δ D ;.

[0216] In this way, the first weight value θ after the generator in the image recognition model is adjusted is obtained. G and the second actual weight value δ after the discriminator is adjusted D .

[0217] S25. Set the weight value of the generator to the adjusted first weight value, and set the weight value of the discriminator to the adjusted second actual weight value, to obtain an image recognition model.

[0218] In some examples, after the generator and the discriminator in the image recognition model have converged, the image recognition model is obtained by setting the weight value of the generator to the adjusted first weight value and the weight value of the discriminator to the adjusted second actual weight value. After that, the image recognition model can perform super-resolution processing on the input first resolution image to determine the probability corresponding to each output image. The output image corresponding to the maximum probability is used as the second resolution image corresponding to the first resolution image.

[0219] Exemplarily, the image recognition model can perform super-resolution processing on the input first-resolution image in the following process:

[0220] Input: first resolution image LR;

[0221] Output: the probability corresponding to each output image;

[0222] Requirement: The first weight value θ after the generator is adjusted G , scale factor: S;

[0223] Initialization:Net θ ←θ G ;

[0224] HR = Netθ(LR, S).

[0225] Returns: The probability corresponding to each output image.

[0226] In this way, based on the probability corresponding to each output image output by the image recognition model, the output image corresponding to the maximum probability can be used as the second resolution image corresponding to the first resolution image.

[0227] From the above, it can be seen that the image generation method provided in the embodiment of the present application makes the model lightweight by introducing deep separable convolution (BSConv) in the generator in the image recognition model to replace the traditional convolution; secondly, we propose a new compensation mechanism to compensate for the impact of the lightweight process on the function of the image recognition model. The compensation mechanism includes a channel attention mechanism (ECA) module, a BMFB module of multi-scale feature fusion technology, and a 3×3 standard convolution kernel and BN layer with an activation function of GELU. This compensation mechanism enables the image recognition model to better integrate information from different channels and granularities. In addition, reducing the number of parameters also reduces the risk of overfitting of the image recognition model, which helps to obtain better perceptual quality.

[0228] In one embodiment, the sum of the adjusted first weight value and the adjusted second actual weight value is equal to 1.

[0229] In one embodiment, in combination Figure 6 ,like Figure 7 As shown, the above S22 can be specifically implemented through the following S221.

[0230] S221, inputting the labeled images corresponding to the plurality of third learning images and the training sample images into a relative average discriminator for calculation to obtain the second loss value of the current time; wherein the relative average discriminator includes:

[0231] D Ra (x r , x f )=σ(C(x r )-E xf [C(xf)]);

[0232] Among them, D Ra (x r , x f ) represents the second loss value of the current time, x r represents the labeled image corresponding to the training sample image, x frepresents the second learning image output by the generator, σ is the Sigmoid function, C(xf) represents the third learning image output by the discriminator, E xf [C(xf)] represents an operation of taking the average value of all third learning images output by the discriminator.

[0233] In some examples, the operation of averaging all the third study images includes averaging pixel values ​​of all the third study images.

[0234] In one embodiment, the relative average discriminator may be RaD (Relativistic average Discriminator).

[0235] From the above, it can be seen that in the image generation method provided by the embodiment of the present disclosure, the relative average delay discriminator (RAD) is used as the loss function of the discriminator in the image recognition model, so that the discriminator can predict the relative authenticity of the image, which helps to reduce the generation of artifacts and thus improve the visual quality of the generated image.

[0236] In one embodiment, in combination Figure 6 ,like Figure 8 As shown, the above S23 can be specifically implemented through the following S231.

[0237] S231, input the current second loss value into the loss formula for calculation, and re-determine the current third loss value corresponding to the generator; wherein the loss formula includes:

[0238]

[0239] in, Denotes the third loss value, D Ra (x r , x f ) represents the second loss value, E xr [log(D Ra (x r , x f ))] represents the operation of averaging the second loss value of the discriminator.

[0240] From the above, it can be seen that the image generation method provided in the embodiment of the present disclosure, when training the discriminator, by simultaneously training the discriminator and the generator, can enable the image recognition model to better output the second resolution image and improve the user experience.

[0241] It should be understood that although Figure 2-8The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-8 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0242] In one embodiment, Fig. 9 As shown, an image generation device is provided, comprising: an acquisition module and a processing module, wherein:

[0243] The acquisition module is further used to acquire the training sample image and the marking result of the training sample image; wherein the training sample image includes a training image with a resolution less than a preset resolution, and the marking result includes a marked image corresponding to the training image, and the resolution of the marked image is greater than the resolution of the training image;

[0244] The processing module is also used to perform a preset operation on the training sample image acquired by the acquisition module to obtain a processed image;

[0245] The processing module is further used to input the processed image into the generator for learning, so as to obtain a first learning image corresponding to the processed image;

[0246] The processing module is further used to obtain a first loss value for a current time based on a mean absolute error between the first learning image and the labeled image;

[0247] The processing module is further used to adjust the network parameters and the first actual weight value of the generator based on the current first loss value when the current first loss value is less than the previous first loss value, and perform a preset operation on the next training sample image until the first loss value corresponding to the next training sample image is less than or equal to the first target loss value, determine that the generator converges, and use the first actual weight value corresponding to the generator when the generator converges as the first weight value corresponding to the generator;

[0248] The processing module is further used to input the training sample image into the converged generator to obtain a second learning image corresponding to the training sample image;

[0249] The processing module is further used to input the second learning image into the discriminator for learning, and obtain a plurality of third learning images after the discriminator learns the second learning image;

[0250] The processing module is further used to calculate the second loss value of the labeled images corresponding to the plurality of third learning images and the training sample images at the current time based on the relative average discriminator;

[0251] The processing module is further used to re-determine the third loss value of the current time corresponding to the generator based on the second loss value of the current time;

[0252] The processing module is further used to adjust the first weight value, the network parameters of the discriminator and the second actual weight value when the third loss value of the current time is less than the third loss value of the previous time and the second loss value of the current time is less than the second loss value of the previous time, and perform a prediction operation on the next training sample image until the second loss value corresponding to the next training sample image is less than or equal to the second target loss value and the third loss value is less than or equal to the third target loss value, determine that the discriminator converges, and obtain the adjusted first weight value and the adjusted second actual weight value;

[0253] The processing module is also used to set the weight value of the generator to the adjusted first weight value, and set the weight value of the discriminator to the adjusted second actual weight value to obtain an image recognition model.

[0254] In one embodiment, the sum of the adjusted first weight value and the adjusted second actual weight value is equal to 1.

[0255] In one embodiment, the processing module is specifically used to input the labeled images corresponding to the plurality of third learning images and the training sample images into the relative average discriminator for calculation to obtain the second loss value of the current time; wherein the relative average discriminator includes:

[0256] D Ra (x r , x f )=σ(C(x r )-E xf [C(xf)]);

[0257] Among them, D Ra (x r , x f ) represents the second loss value of the current time, x r represents the labeled image corresponding to the training sample image, x f represents the second learning image output by the generator, σ is the Sigmoid function, C(xf) represents the third learning image output by the discriminator, E xf [C(xf)] represents an operation of taking the average value of all third learning images output by the discriminator.

[0258] In one embodiment, the processing module is specifically used to input the current second loss value into the loss formula for calculation, and re-determine the current third loss value corresponding to the generator; wherein the loss formula includes:

[0259]

[0260] in, Denotes the third loss value, D Ra (x r , x f ) represents the second loss value, EE xr [log(D Ra (x r , x f ))] represents the operation of averaging the second loss value of the discriminator.

[0261] As can be seen from the above, the image generation method provided by the embodiment of the present application introduces one or more 3×3 standard convolution kernels, one or more BMFB modules using multi-scale feature fusion technology, one or more depth-separable convolution BSConv modules, and one or more channel attention mechanism ECA modules in the image recognition model, instead of the traditional BN layer. Such a design not only improves the reuse rate of features, but also avoids the interference of the BN layer on the feature space, saving storage space. At the same time, the activation functions of both the 3×3 standard convolution kernel and the BN layer are Gaussian error linear unit GELU functions, which enhances the nonlinear factors of the mapping process and makes it more suitable for the BSConv module. At the same time, by introducing the ECA module, the image recognition model can repeatedly capture the feature relationship of each channel without adding a large number of parameters. At the same time, the BSConv module decomposes the standard convolution into a 1×1 convolution in the point direction and a convolution in the depth direction, thereby reducing the dimension of the data output by the generator. Since the actual storage space required for the image recognition model is reduced and the amount of calculation is reduced, more scenarios can be applied.

[0262] For the specific definition of the image generation device, please refer to the definition of the image generation method above, which will not be repeated here. Each module in the above-mentioned image generation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0263] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data of the image recognition model. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image generation method is implemented.

[0264] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0265] In one embodiment, the image generation device provided by the present application can be implemented in the form of a computer program. The computer program can be Fig.10 The computer device shown in the figure is run. The memory of the computer device can store various program modules constituting the image generating device, for example, Fig. 9 The acquisition module and the processing module shown in the figure. The computer program composed of various program modules enables the processor to execute the steps of the image generation method of each embodiment of the present application described in this specification.

[0266] For example, Fig.10 The computer device shown can be Fig. 9 The acquisition module in the image generation device shown executes S11. The computer device can execute S12, S13, and S14 through the processing module.

[0267] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0268] Acquire a first resolution image;

[0269] Performing a preset operation on the first resolution image to generate a target image; wherein the preset operation includes one or more of image cropping and down sampling;

[0270] Input the target image into an image recognition model including a generator and a discriminator for super-resolution processing, and determine the probability corresponding to each output image; wherein the generator includes an encoder and a decoder, the input of the generator is the target image, the output of the generator is the input of the discriminator, and the output of the discriminator is the probability corresponding to each output image. The encoder includes one or more 3×3 standard convolution kernels, one or more BMFB modules using multi-scale feature fusion technology, one or more depth-separable convolution BSConv modules, and one or more channel attention mechanism ECA modules. The decoder includes at least one upsampling module and at least one 3×3 standard convolution kernel. The discriminator includes multiple 3×3 standard convolution kernels, multiple 4×4 convolution kernels, multiple batch normalization BN layers, a flat layer and a classifier, and the activation functions of the 3×3 standard convolution kernel and the BN layer are both GELU functions.

[0271] The output image corresponding to the maximum probability is used as the second resolution image corresponding to the first resolution image; wherein the first resolution image and the second resolution image display the same content, and the resolution of the second resolution image is greater than that of the first resolution image.

[0272] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0273] Acquire a training sample image and a labeling result of the training sample image; wherein the training sample image includes a training image having a resolution less than a preset resolution, and the labeling result includes a labeling image corresponding to the training image, and the resolution of the labeling image is greater than the resolution of the training image;

[0274] Performing preset operations on the training sample image to obtain a processed image;

[0275] Inputting the processed image into the generator for learning, and obtaining a first learning image corresponding to the processed image;

[0276] Based on the mean absolute error between the first learning image and the labeled image, a first loss value for the current time is obtained;

[0277] When the current first loss value is less than the previous first loss value, the network parameters and the first actual weight value of the generator are adjusted based on the current first loss value, and the preset operation is performed on the next training sample image until the first loss value corresponding to the next training sample image is less than or equal to the first target loss value, and the generator is determined to be converged, and the first actual weight value corresponding to the generator when the generator converges is used as the first weight value corresponding to the generator;

[0278] Input the training sample image into the converged generator to obtain a second learning image corresponding to the training sample image;

[0279] Inputting the second learning image into the discriminator for learning, and obtaining a plurality of third learning images after the discriminator has learned the second learning image;

[0280] Calculate the second loss value of the labeled images corresponding to the plurality of third learning images and the training sample images at the current time based on the relative average discriminator;

[0281] Based on the second loss value of the current time, re-determine the third loss value of the current time corresponding to the generator;

[0282] When the third loss value of the current time is less than the third loss value of the previous time, and the second loss value of the current time is less than the second loss value of the previous time, adjust the first weight value, the network parameter of the discriminator and the second actual weight value, and perform a prediction operation on the next training sample image until the second loss value corresponding to the next training sample image is less than or equal to the second target loss value, and the third loss value is less than or equal to the third target loss value, determine that the discriminator converges, and obtain the adjusted first weight value and the adjusted second actual weight value;

[0283] The weight value of the generator is set to the adjusted first weight value, and the weight value of the discriminator is set to the adjusted second actual weight value, to obtain an image recognition model.

[0284] In one embodiment, when the processor executes the computer program, the following steps are further implemented: the sum of the adjusted first weight value and the adjusted second actual weight value is equal to 1.

[0285] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0286] Input the labeled images corresponding to the plurality of third learning images and the training sample images into the relative average discriminator for calculation to obtain the second loss value of the current time; wherein the relative average discriminator includes:

[0287] D Ra (x r , x f )=σ(C(x r )-E xf [C(xf)]);

[0288] Among them, D Ra (x r , x f ) represents the second loss value of the current time, x r represents the labeled image corresponding to the training sample image, x f represents the second learning image output by the generator, σ is the Sigmoid function, C(xf) represents the third learning image output by the discriminator, E xf[C(xf)] represents an operation of taking the average value of all third learning images output by the discriminator.

[0289] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0290] The second loss value of the current time is input into the loss formula for calculation, and the third loss value of the current time corresponding to the generator is re-determined; wherein the loss formula includes:

[0291]

[0292] in, represents the third loss value, D Ra (x r , x f ) represents the second loss value, EE xr [log(D Ra (x r , x f ))] represents the operation of averaging the second loss value of the discriminator.

[0293] As can be seen from the above, the image generation method provided by the embodiment of the present application introduces one or more 3×3 standard convolution kernels, one or more BMFB modules using multi-scale feature fusion technology, one or more depth-separable convolution BSConv modules, and one or more channel attention mechanism ECA modules in the image recognition model, instead of the traditional BN layer. Such a design not only improves the reuse rate of features, but also avoids the interference of the BN layer on the feature space, saving storage space. At the same time, the activation functions of both the 3×3 standard convolution kernel and the BN layer are Gaussian error linear unit GELU functions, which enhances the nonlinear factors of the mapping process and makes it more suitable for the BSConv module. At the same time, by introducing the ECA module, the image recognition model can repeatedly capture the feature relationship of each channel without adding a large number of parameters. At the same time, the BSConv module decomposes the standard convolution into a 1×1 convolution in the point direction and a convolution in the depth direction, thereby reducing the dimension of the data output by the generator. Since the actual storage space required for the image recognition model is reduced and the amount of calculation is reduced, more scenarios can be applied.

[0294] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0295] Acquire a first resolution image;

[0296] Performing a preset operation on the first resolution image to generate a target image; wherein the preset operation includes one or more of image cropping and down sampling;

[0297] Input the target image into an image recognition model including a generator and a discriminator for super-resolution processing, and determine the probability corresponding to each output image; wherein the generator includes an encoder and a decoder, the input of the generator is the target image, the output of the generator is the input of the discriminator, and the output of the discriminator is the probability corresponding to each output image. The encoder includes one or more 3×3 standard convolution kernels, one or more BMFB modules using multi-scale feature fusion technology, one or more depth-separable convolution BSConv modules, and one or more channel attention mechanism ECA modules. The decoder includes at least one upsampling module and at least one 3×3 standard convolution kernel. The discriminator includes multiple 3×3 standard convolution kernels, multiple 4×4 convolution kernels, multiple batch normalization BN layers, a flat layer and a classifier, and the activation functions of the 3×3 standard convolution kernel and the BN layer are both GELU functions.

[0298] The output image corresponding to the maximum probability is used as the second resolution image corresponding to the first resolution image; wherein the first resolution image and the second resolution image display the same content, and the resolution of the second resolution image is greater than that of the first resolution image.

[0299] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0300] Acquire a training sample image and a labeling result of the training sample image; wherein the training sample image includes a training image having a resolution less than a preset resolution, and the labeling result includes a labeling image corresponding to the training image, and the resolution of the labeling image is greater than the resolution of the training image;

[0301] Performing preset operations on the training sample image to obtain a processed image;

[0302] Inputting the processed image into the generator for learning, and obtaining a first learning image corresponding to the processed image;

[0303] Based on the mean absolute error between the first learning image and the labeled image, a first loss value for the current time is obtained;

[0304] When the current first loss value is less than the previous first loss value, the network parameters and the first actual weight value of the generator are adjusted based on the current first loss value, and the preset operation is performed on the next training sample image until the first loss value corresponding to the next training sample image is less than or equal to the first target loss value, and the generator is determined to be converged, and the first actual weight value corresponding to the generator when the generator converges is used as the first weight value corresponding to the generator;

[0305] Input the training sample image into the converged generator to obtain a second learning image corresponding to the training sample image;

[0306] Inputting the second learning image into the discriminator for learning, and obtaining a plurality of third learning images after the discriminator has learned the second learning image;

[0307] Calculate the second loss value of the labeled images corresponding to the plurality of third learning images and the training sample images at the current time based on the relative average discriminator;

[0308] Based on the second loss value of the current time, re-determine the third loss value of the current time corresponding to the generator;

[0309] When the third loss value of the current time is less than the third loss value of the previous time, and the second loss value of the current time is less than the second loss value of the previous time, adjust the first weight value, the network parameter of the discriminator and the second actual weight value, and perform a prediction operation on the next training sample image until the second loss value corresponding to the next training sample image is less than or equal to the second target loss value, and the third loss value is less than or equal to the third target loss value, determine that the discriminator converges, and obtain the adjusted first weight value and the adjusted second actual weight value;

[0310] The weight value of the generator is set to the adjusted first weight value, and the weight value of the discriminator is set to the adjusted second actual weight value, to obtain an image recognition model.

[0311] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: the sum of the adjusted first weight value and the adjusted second actual weight value is equal to 1.

[0312] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0313] Input the labeled images corresponding to the plurality of third learning images and the training sample images into the relative average discriminator for calculation to obtain the second loss value of the current time; wherein the relative average discriminator includes:

[0314] D Ra (x r , x f )=σ(C(x r )-E xf [C(xf)]);

[0315] Among them, d Ra (x r , x f ) represents the second loss value of the current time, x r represents the labeled image corresponding to the training sample image, x frepresents the second learning image output by the generator, σ is the Sigmoid function, C(xf) represents the third learning image output by the discriminator, E xf [C(xf)] represents an operation of taking the average value of all third learning images output by the discriminator.

[0316] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0317] The second loss value of the current time is input into the loss formula for calculation, and the third loss value of the current time corresponding to the generator is re-determined; wherein the loss formula includes:

[0318]

[0319] in, Denotes the third loss value, D Ra (x r , x f ) represents the second loss value, E xr [log(D Ra (x r , x f ))] represents the operation of averaging the second loss value of the discriminator.

[0320] As can be seen from the above, the image generation method provided by the embodiment of the present application introduces one or more 3×3 standard convolution kernels, one or more BMFB modules using multi-scale feature fusion technology, one or more depth-separable convolution BSConv modules, and one or more channel attention mechanism ECA modules in the image recognition model, instead of the traditional BN layer. Such a design not only improves the reuse rate of features, but also avoids the interference of the BN layer on the feature space, saving storage space. At the same time, the activation functions of both the 3×3 standard convolution kernel and the BN layer are Gaussian error linear unit GELU functions, which enhances the nonlinear factors of the mapping process and makes it more suitable for the BSConv module. At the same time, by introducing the ECA module, the image recognition model can repeatedly capture the feature relationship of each channel without adding a large number of parameters. At the same time, the BSConv module decomposes the standard convolution into a 1×1 convolution in the point direction and a convolution in the depth direction, thereby reducing the dimension of the data output by the generator. Since the actual storage space required for the image recognition model is reduced and the amount of calculation is reduced, more scenarios can be applied.

[0321] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static random access memory (SRAM) and dynamic random access memory (DRAM).

[0322] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0323] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An image generation method, characterized in that: Methods include: Acquire a first resolution image; Performing a preset operation on the first resolution image to generate a target image; wherein the preset operation includes one or more of image cropping and down sampling; The target image is input into an image recognition model including a generator and a discriminator for super-resolution processing, and the probability corresponding to each output image is determined; wherein the generator includes an encoder and a decoder, the input of the generator is the target image, the output of the generator is the input of the discriminator, and the output of the discriminator is the probability corresponding to each output image, the encoder includes one or more 3×3 standard convolution kernels, one or more BMFB modules using multi-scale feature fusion technology, one or more depth-separable convolution BSConv modules, and one or more channel attention mechanism ECA modules, the decoder includes at least one upsampling module, and at least one 3×3 standard convolution kernel, the discriminator includes multiple 3×3 standard convolution kernels, multiple 4×4 convolution kernels, multiple batch normalization BN layers, a flat layer and a classifier, and the activation functions of the 3×3 standard convolution kernel and the BN layer are both Gaussian error linear unit GELU functions; The output image corresponding to the maximum probability is used as the second resolution image corresponding to the first resolution image; wherein the first resolution image and the second resolution image display the same content, and the resolution of the second resolution image is greater than that of the first resolution image.

2. The image generation method according to claim 1, characterized in that: Before acquiring the first resolution image, the method further includes: Acquire a training sample image and a marking result of the training sample image; wherein the training sample image includes a training image having a resolution less than a preset resolution, and the marking result includes a marked image corresponding to the training image, and the resolution of the marked image is greater than the resolution of the training image; Performing a preset operation on the training sample image to obtain a processed image; Inputting the processed image into the generator for learning to obtain a first learning image corresponding to the processed image; Obtaining a first loss value for a current time based on a mean absolute error between the first learning image and the labeled image; In the case where the current first loss value is less than the previous first loss value, adjusting the network parameters and the first actual weight value of the generator based on the current first loss value, and performing a preset operation on the next training sample image until the first loss value corresponding to the next training sample image is less than or equal to the first target loss value, determining that the generator has converged, and using the first actual weight value corresponding to the generator when it converges as the first weight value corresponding to the generator; Inputting the training sample image into the converged generator to obtain a second learning image corresponding to the training sample image; Inputting the second learning image into the discriminator for learning, and obtaining a plurality of third learning images after the discriminator has learned the second learning image; Calculate the second loss value of the labeled images corresponding to the plurality of third learning images and the training sample images at the current time based on the relative average discriminator; Based on the second loss value of the current time, re-determine the third loss value of the current time corresponding to the generator; When the third loss value of the current time is less than the third loss value of the previous time, and the second loss value of the current time is less than the second loss value of the previous time, adjust the first weight value, the network parameter of the discriminator and the second actual weight value, and perform a prediction operation on the next training sample image until the second loss value corresponding to the next training sample image is less than or equal to the second target loss value, and the third loss value is less than or equal to the third target loss value, determine that the discriminator converges, and obtain the adjusted first weight value and the adjusted second actual weight value; The weight value of the generator is set to the adjusted first weight value, and the weight value of the discriminator is set to the adjusted second actual weight value to obtain the image recognition model.

3. The image generation method according to claim 2, characterized in that: The sum of the adjusted first weight value and the adjusted second actual weight value is equal to 1.

4. The image generation method according to claim 2, characterized in that: The calculating the second loss value of the labeled image corresponding to the plurality of third learning images and the training sample image at the current time based on the relative average discriminator includes: Inputting the labeled images corresponding to the plurality of third learning images and the training sample images into a relative average discriminator for calculation to obtain the second loss value of the current time; wherein the relative average discriminator includes: D Ra (x r ,x f )=σ(C(x r )-E xf [C(xf)); Among them, D Ra (x r , x f ) represents the second loss value of the current time, x r represents the labeled image corresponding to the training sample image, x f represents the second learning image output by the generator, σ is the Sigmoid function, C(xf) represents the third learning image output by the discriminator, E xf [C(xf)] represents an operation of taking the average value of all third learning images output by the discriminator.

5. The image generation method according to claim 2, characterized in that: The re-determining the third loss value of the current time corresponding to the generator based on the second loss value of the current time includes: The second loss value of the current time is input into the loss formula for calculation, and the third loss value of the current time corresponding to the generator is re-determined; wherein the loss formula includes: in, represents the third loss value, D Ra (x r , x f ) represents the second loss value, E xr [log(D Ra (x r ,× f ))] represents an operation of averaging the second loss values ​​of the discriminator.

6. An image generating device, characterized in that: The device comprises: An acquisition module, used for acquiring a first resolution image; A processing module, configured to perform a preset operation on the first resolution image acquired by the acquisition module to generate a target image; wherein the preset operation includes one or more of image cropping and down sampling; The processing module is further used to input the target image into an image recognition model including a generator and a discriminator for super-resolution processing, and determine the probability corresponding to each output image; wherein the generator includes an encoder and a decoder, the input of the generator is the target image, the output of the generator is the input of the discriminator, and the output of the discriminator is the probability corresponding to each output image, the encoder includes one or more 3×3 standard convolution kernels, one or more BMFB modules using multi-scale feature fusion technology, one or more depth-separable convolution BSConv modules, and one or more channel attention mechanism ECA modules, the decoder includes at least one upsampling module, and at least one 3×3 standard convolution kernel, the discriminator includes multiple 3×3 standard convolution kernels, multiple 4×4 convolution kernels, multiple batch normalization BN layers, a flat layer and a classifier, and the activation functions of the 3×3 standard convolution kernel and the BN layer are both Gaussian error linear unit GELU functions; The processing module is further used to use the output image corresponding to the maximum probability as a second resolution image corresponding to the first resolution image; wherein the resolution of the second resolution image is greater than that of the first resolution image.

7. The image generating device according to claim 6, characterized in that: The acquisition module is further used to acquire a training sample image and a marking result of the training sample image; wherein the training sample image includes a training image having a resolution less than a preset resolution, and the marking result includes a marked image corresponding to the training image, and the resolution of the marked image is greater than the resolution of the training image; The processing module is further used to perform a preset operation on the training sample image acquired by the acquisition module to obtain a processed image; The processing module is further used to input the processed image into the generator for learning, so as to obtain a first learning image corresponding to the processed image; The processing module is further used to obtain a first loss value for a current time based on a mean absolute error between the first learning image and the labeled image; The processing module is further used to adjust the network parameters and the first actual weight value of the generator based on the first loss value of the current time when the first loss value of the current time is less than the first loss value of the previous time, and perform a preset operation on the next training sample image until the first loss value corresponding to the next training sample image is less than or equal to the first target loss value, determine that the generator converges, and use the first actual weight value corresponding to the convergence of the generator as the first weight value corresponding to the generator; The processing module is further used to input the training sample image into the converged generator to obtain a second learning image corresponding to the training sample image; The processing module is further used to input the second learning image into the discriminator for learning, so as to obtain a plurality of third learning images after the discriminator learns the second learning image; The processing module is further used to calculate the second loss value of the labeled images corresponding to the plurality of third learning images and the training sample images at the current time based on the relative average discriminator; The processing module is further used to re-determine the current third loss value corresponding to the generator based on the current second loss value; The processing module is further used to adjust the first weight value, the network parameters of the discriminator and the second actual weight value when the third loss value of the current time is less than the third loss value of the previous time, and the second loss value of the current time is less than the second loss value of the previous time, and perform a prediction operation on the next training sample image until the second loss value corresponding to the next training sample image is less than or equal to the second target loss value, and the third loss value is less than or equal to the third target loss value, determine that the discriminator converges, and obtain the adjusted first weight value and the adjusted second actual weight value; The processing module is further used to set the weight value of the generator to the adjusted first weight value, and set the weight value of the discriminator to the adjusted second actual weight value, so as to obtain the image recognition model.

8. The image generation method according to claim 7, characterized in that: The processing module is specifically used to input the labeled images corresponding to the plurality of third learning images and the training sample images into a relative average discriminator for calculation to obtain the second loss value of the current time; wherein the relative average discriminator includes: D Ra (x r ,x f )=σ(C(x r )-E xf [C(xf)]); Among them, D Ra (x r , x f ) represents the second loss value of the current time, x r represents the labeled image corresponding to the training sample image, x f represents the second learning image output by the generator, σ is the Sigmoid function, C(xf) represents the third learning image output by the discriminator, E xf [C(xf)] represents an operation of taking the average value of all third learning images output by the discriminator.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images