GAN network-based super-resolution image processing method, device, equipment and medium

Abstract

Embodiments of the present disclosure relate to a super-resolution image processing method and device based on a GAN network, equipment and a medium, wherein the method comprises: obtaining a first feature of a positive sample image corresponding to an input sample image, and a third feature corresponding to a reference sample image; determining a binary cross entropy (BCE) loss function according to the first feature and the third feature, extracting a fourth feature corresponding to the positive sample image, a fifth feature corresponding to a negative sample image, and a sixth feature corresponding to the reference sample image, and determining a second contrastive learning loss function according to the fourth feature, the fifth feature and the sixth feature; training parameters of a generation model according to the BCE loss function and the second contrastive learning loss function, obtaining a target super-resolution network, and performing super-resolution processing on a test image according to the target super-resolution network to obtain a target super-resolution image. Thus, the richness of the image details of the target super-resolution image output by the target super-resolution network is ensured, and the purity of the target super-resolution image is improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a super-resolution image processing method, apparatus, device and medium based on GAN network. Background Technology

[0002] Image super-resolution processing involves enlarging the resolution of an image, transforming a low-resolution image into a high-resolution super-resolution image. It is often used to enhance image quality in scenarios such as short video frames.

[0003] In related technologies, super-resolution networks are used to process low-resolution input images to output high-resolution super-resolution images. The training of super-resolution networks is mainly carried out using a training framework based on Generative Adversarial Networks (GANs). That is, an additional discriminant module is used to judge the super-resolution images generated by the network and the real high-resolution images, thereby promoting the progress of super-resolution networks.

[0004] However, when learning from training sample images, especially those with a wide input domain, GAN networks learn to distinguish between super-resolution images and real high-resolution images from multiple feature levels. This leads to the introduction of some complex and rare noise and artifacts, resulting in the generated super-resolution images containing more artifacts and noise. Summary of the Invention

[0005] To address, or at least partially address, the aforementioned technical problems, this disclosure provides a super-resolution image processing method, apparatus, device, and medium based on a GAN network. This solves the problem in the prior art where the GAN network takes the super-resolution image and the real high-resolution image as input for judgment. However, if the output super-resolution image contains complex noise or rare artifacts, the feature extraction layer of the discriminator in the GAN network may selectively ignore these "outliers," causing the noise and artifacts to be accepted by the discriminator and introduced into the super-resolution image. This results in the generated super-resolution image containing a large number of artifacts and noise, leading to low image quality.

[0006] This disclosure provides a super-resolution image processing method based on a GAN network. The method includes: acquiring a positive sample image, a negative sample image, and a reference sample image, wherein the positive sample image is the ground truth super-resolution image corresponding to the input sample image, the negative sample image is an image obtained by fusing and adding noise to the input sample image and the positive sample image, and the reference sample image is an image output by the input sample image after the image quality is reduced by the generative model of the generative adversarial GAN ​​network to be trained.

[0007] The GAN network discrimination model extracts a first feature corresponding to the positive sample image and a third feature corresponding to the reference sample image, and performs discrimination processing on the first feature and the third feature respectively to obtain a first score corresponding to the positive sample image and a second score corresponding to the reference sample image. The binary cross-entropy (BCE) loss function is determined based on the first score and the second score.

[0008] A fourth feature corresponding to the positive sample image, a fifth feature corresponding to the negative sample image, and a sixth feature corresponding to the reference sample image are extracted through a preset network. A second contrastive learning loss function is determined based on the fourth feature, the fifth feature, and the sixth feature. The second contrastive learning loss function is used to make the features of the reference sample image close to the features of the positive sample image and far away from the features of the negative sample image.

[0009] The parameters of the generative model are trained by backpropagation based on the BCE loss function and the second contrastive learning loss function to obtain the target super-resolution network, and the test image is super-resolution processed by the target super-resolution network to obtain the target super-resolution image.

[0010] This disclosure also provides a super-resolution image processing apparatus based on a GAN network, the apparatus comprising:

[0011] The first acquisition module is used to acquire positive sample images, negative sample images, and reference sample images, wherein the positive sample image is the ground truth super-resolution image corresponding to the input sample image, the negative sample image is an image that has been fused and noise-added by the input sample image and the positive sample image, and the reference sample image is an image output by the input sample image after the image quality has been reduced by the generative model of the generative adversarial network to be trained.

[0012] The second acquisition module is used to extract a first feature corresponding to the positive sample image and a third feature corresponding to the reference sample image through the GAN network discrimination model, and to perform discrimination processing on the first feature and the third feature respectively to obtain a first score corresponding to the positive sample image and a second score corresponding to the reference sample image, and to determine the binary cross-entropy (BCE) loss function based on the first score and the second score.

[0013] The determination module is used to extract a fourth feature corresponding to the positive sample image, a fifth feature corresponding to the negative sample image, and a sixth feature corresponding to the reference sample image through a preset network, and to determine a second contrastive learning loss function based on the fourth feature, the fifth feature, and the sixth feature, wherein the second contrastive learning loss function is used to make the features of the reference sample image close to the features of the positive sample image and far away from the features of the negative sample image;

[0014] The third acquisition module is used to train the parameters of the generative model by backpropagation based on the BCE loss function and the second contrastive learning loss function, and to acquire the target super-resolution network, so as to perform super-resolution processing on the test image based on the target super-resolution network to acquire the target super-resolution image.

[0015] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the super-resolution image processing method based on a GAN network as provided in this disclosure.

[0016] This disclosure also provides a computer-readable storage medium storing a computer program for executing the GAN-based super-resolution image processing method provided in this disclosure.

[0017] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0018] The super-resolution image processing scheme provided in this disclosure acquires a positive sample image, a negative sample image, and a reference sample image. The positive sample image is the ground truth super-resolution image corresponding to the input sample image. The negative sample image is an image obtained by fusing and adding noise to the input sample image and the positive sample image. The reference sample image is the output image of the input sample image after its quality has been reduced by a generative adversarial network (GAN) model to be trained. A first feature corresponding to the positive sample image and a third feature corresponding to the reference sample image are extracted using a GAN network discriminative model. The first and third features are then processed separately to obtain a first score corresponding to the positive sample image and a second score corresponding to the reference sample image. The GAN network is trained using a pre-defined network. A first score and a second score are used to determine the binary cross-entropy (BCE) loss function. A fourth feature corresponding to the positive sample image, a fifth feature corresponding to the negative sample image, and a sixth feature corresponding to the reference sample image are extracted. A second contrastive learning loss function is then determined based on these features. This second contrastive learning loss function aims to make the features of the reference sample image closer to those of the positive sample image and further away from those of the negative sample image. Backpropagation is then used to train the parameters of the generative model based on the BCE and second contrastive learning loss functions, obtaining the target super-resolution network. This network is then used to perform super-resolution processing on the test image to obtain the target super-resolution image. Thus, by super-supervising the feature extraction process of the GAN network based on the loss function, the sensitivity of the discriminative model to noise and artifacts is improved, reducing the discrimination and training difficulty of the model. This achieves an improvement in the purity of the target super-resolution image while maintaining the richness of image details in the output image of the target super-resolution network. Attached Figure Description

[0019] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 A flowchart illustrating a super-resolution image processing method based on a GAN network provided in this embodiment of the present disclosure;

[0021] Figure 2 A schematic diagram illustrating a scenario for acquiring negative sample images, provided in an embodiment of this disclosure;

[0022] Figure 3 A schematic diagram illustrating another scenario for acquiring negative sample images provided in this embodiment of the disclosure;

[0023] Figure 4 A flowchart illustrating another super-resolution image processing method based on a GAN network provided in this embodiment of the present disclosure;

[0024] Figure 5 This is a schematic diagram of a super-resolution image processing scenario provided by an embodiment of the present disclosure;

[0025] Figure 6 A flowchart illustrating another super-resolution image processing method based on a GAN network provided in this embodiment of the present disclosure;

[0026] Figure 7 A flowchart illustrating another super-resolution image processing method based on a GAN network provided in this embodiment of the present disclosure;

[0027] Figure 8 This is a schematic diagram of another super-resolution image processing scenario provided by an embodiment of this disclosure;

[0028] Figure 9 This is a schematic diagram of another super-resolution image processing scenario provided by an embodiment of this disclosure;

[0029] Figure 10 This is a schematic diagram of another super-resolution image processing scenario provided by an embodiment of this disclosure;

[0030] Figure 11 This is a schematic diagram of the structure of a super-resolution image processing device provided in an embodiment of the present disclosure;

[0031] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0032] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0033] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0034] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0035] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0036] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0037] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0038] To address the aforementioned issues, this disclosure provides a super-resolution image processing method based on a GAN network. In this method, a contrastive learning loss (CR loss) function is introduced into the training process of the GAN network's discriminative model. By supervising its feature extraction process, the GAN network's discriminative model can more easily distinguish between the super-resolution image output by the network and the real high-resolution image. This makes the GAN network's discriminative model more sensitive to noise and artifacts, while also reducing the difficulty of discrimination and training. This method can be applied to various image enhancement tasks and their GAN network training frameworks.

[0039] The method will be described below with reference to specific embodiments.

[0040] Figure 1 This is a flowchart illustrating a super-resolution image processing method based on a GAN network, provided in an embodiment of this disclosure. This method can be executed by a GAN-based super-resolution image processing device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the method includes:

[0041] Step 101: Obtain positive sample images, negative sample images, and reference sample images. The positive sample image is the ground truth super-resolution image corresponding to the input sample image. The negative sample image is the image that is fused and noise-added by the input sample image and the positive sample image. The reference sample image is the image output by the generative model of the generative adversarial network to be trained, which reduces the image quality of the input sample image.

[0042] In this embodiment, in order to better simulate the real image degradation process, the ground truth super-resolution image corresponding to the input sample image is obtained as the positive sample image. The positive sample image is a real high-definition image. The image output by the generative model of the GAN network to be trained after the input sample image has been degraded is obtained as the reference sample image. At the same time, the negative sample image corresponding to the input sample image is also obtained to ensure that in the subsequent training process, in addition to considering the distance from the positive sample image, it is also considered to stay as far away from the negative sample image as possible, so as to further improve the training effect.

[0043] It should be noted that the methods for obtaining negative sample images differ in different application scenarios, as shown in the following examples:

[0044] In one embodiment of this disclosure, since the input sample image and the output reference sample image and positive sample image have different sizes, the input sample image is upsampled to obtain a candidate sample image with the same size as the positive sample image. Then, a negative sample image is generated based on the candidate sample image and the positive sample image. Thus, the positive sample image is fused to generate the negative sample image, making the negative sample image slightly closer to the positive sample image, thereby increasing the training difficulty and avoiding too fast convergence.

[0045] In this embodiment, refer to Figure 2 It can determine a first weight corresponding to the candidate sample image, for example, the first weight can be 0.5, and determine a second weight corresponding to the positive sample image, for example, 0.5, where the sum of the first weight and the second weight is 1.

[0046] The first product of the candidate sample image and the first weight, and the second product of the positive sample image and the second weight are summed to obtain a fused image. Further, after obtaining the fused image, random Gaussian noise is added to the fused image to generate negative sample images, thereby improving the realism of the negative sample images and ensuring training effectiveness. For example, random Gaussian noise can be introduced to weight and sum the fused images to obtain negative sample images.

[0047] In another embodiment of this disclosure, reference is made to Figure 3 The positive sample image is downsampled based on a preset downsampling resolution to obtain a downsampled sample image. The downsampled image and the input sample image have the same size. Then, the downsampled sample image and the input sample image are fused to obtain a fused image. The fused image is upsampled to obtain an image of the corresponding size as a negative sample image, so that the negative sample image is slightly close to the positive sample image, thereby increasing the training difficulty and avoiding too fast convergence. Of course, in this embodiment, after upsampling the fused image to obtain an image of the corresponding size, random Gaussian noise can be added to generate a negative sample image to improve the realism of the negative sample image and ensure the training effect.

[0048] Step 102: Extract the first feature corresponding to the positive sample image and the third feature corresponding to the reference sample image through the GAN network discriminant model, and perform discriminant processing on the first feature and the third feature respectively to obtain the first score corresponding to the positive sample image and the second score corresponding to the reference sample image. Determine the binary cross-entropy (BCE) loss function based on the first score and the second score.

[0049] In this embodiment, in order to further improve the performance of the GAN network discrimination model, the first score and the second score are trained adversarially based on the Binary Cross Entropy Loss (BCE) function used for binary classification, so as to ensure that the super-resolution result is closer to the positive sample image.

[0050] In this embodiment, the first feature and the third feature are processed according to the discrimination model to obtain the first score corresponding to the positive sample image and the second score corresponding to the reference sample image.

[0051] In this embodiment, during adversarial training based on the GAN network, the first feature and the second feature are processed according to the discrimination model to obtain the first score corresponding to the positive sample image and the second score corresponding to the reference sample image.

[0052] Then, the BCE loss function is determined based on the first score and the second score.

[0053] In this embodiment, the first score and the second score are trained adversarially using the cross-entropy loss function (BCE) used for binary classification, thereby ensuring that the super-resolution result is closer to the high-frequency result of the positive sample image.

[0054] Step 103: Extract the fourth feature corresponding to the positive sample image, the fifth feature corresponding to the negative sample image, and the sixth feature corresponding to the reference sample image through a preset network, and determine the second contrastive learning loss function based on the fourth, fifth, and sixth features. The second contrastive learning loss function is used to make the features of the reference sample image close to the features of the positive sample image and far away from the features of the negative sample image.

[0055] In this embodiment, a positive sample image, a negative sample image, and a reference sample image are input into a pre-trained VGG network to obtain a fourth feature corresponding to the positive sample image, a fifth feature corresponding to the negative sample image, and a sixth feature corresponding to the reference sample image.

[0056] In this embodiment, the positive sample image, negative sample image, and reference sample image are input into the deep convolutional neural network VGG network for feature extraction, obtaining the fourth feature corresponding to the positive sample image, the fifth feature corresponding to the negative sample image, and the sixth feature corresponding to the reference sample image, so as to facilitate the training of the super-resolution network based on the feature dimensions.

[0057] The second contrastive learning loss function is determined based on the fourth, fifth, and sixth features. The second contrastive learning loss function is used to make the features of the reference sample image close to the features of the positive sample image and far away from the features of the negative sample image.

[0058] In this embodiment, in order to train the super-resolution network, a second contrastive learning loss function is determined based on the fourth feature, the fifth feature, and the sixth feature. The second contrastive learning loss function is used to make the features of the reference sample image close to the features of the positive sample image and far away from the features of the negative sample image. This ensures that the reference sample image and the positive sample image are close at the feature level while being far away from the negative sample image, thereby avoiding the introduction of some artifacts and noise.

[0059] Therefore, there is no need to introduce a large number of fake sample images for generative adversarial learning. The super-resolution network is trained solely based on the calculation of the loss values ​​of positive and negative samples in the feature dimension. Compared with traditional generative adversarial networks (GANs), GANs are prone to introducing artifacts and noise because the adversarial loss function they use only emphasizes that the network output is close to the ground truth (positive sample images) in the training set, but does not consider its distance from negative sample images, thus introducing artifacts and noise. In this embodiment, not only is the network output close to the ground truth (positive sample images), but it is also distanced from some flawed negative samples, reducing the introduced artifacts and noise.

[0060] It should be noted that the method for determining the second contrastive learning loss function based on the fourth, fifth, and sixth features differs in different application scenarios, as shown in the following example:

[0061] In one embodiment of this disclosure, such as Figure 4 As shown, the second contrastive learning loss function is determined based on the fourth, fifth, and sixth features, including:

[0062] Step 401: Determine the fourth loss function based on the fourth and sixth features.

[0063] In this embodiment, a fourth loss function is determined based on the sixth feature corresponding to the positive sample image and the third feature corresponding to the reference sample image, wherein the fourth loss function represents the distance between the reference sample image and the positive sample image.

[0064] The fourth loss function can be calculated based on any algorithm for calculating loss values. For example, it can be calculated based on the L1 loss function, which is the mean absolute error (MAE), used to calculate the average distance between the fourth feature and the sixth feature.

[0065] For example, it can be calculated based on the L2 loss function, which is the mean square error (MSE), used to calculate the average of the squared differences between the fourth and sixth features.

[0066] Step 402: Determine the fifth loss function based on the fifth and sixth features.

[0067] In this embodiment, a fifth loss function is determined based on the fifth feature corresponding to the negative sample image and the sixth feature corresponding to the reference sample image, wherein the fifth loss function represents the distance between the reference sample image and the negative sample image.

[0068] The fifth loss function can be calculated based on any algorithm for calculating loss values. For example, it can be calculated based on the L1 loss function, which is the mean absolute error (MAE), used to calculate the average distance between the fifth feature and the sixth feature to obtain the fifth loss function.

[0069] For example, a second loss function can be calculated based on the L2 loss function, which is the mean square error (MSE). The average of the squared differences between the fifth and sixth features is used as the second loss function to obtain the fifth loss function.

[0070] Step 403: Determine the second contrastive learning loss function based on the fourth and fifth loss functions.

[0071] In this embodiment, a second contrastive learning loss function is determined based on a fourth loss function and a fifth loss function. The second contrastive learning loss function is used to make the features of the reference sample image close to the features of the positive sample image and far away from the features of the negative sample image.

[0072] It should be noted that the method for determining the second contrastive learning loss function based on the fourth and fifth loss functions differs in different application scenarios, as shown in the following example:

[0073] In one embodiment of this disclosure, the ratio between the fourth loss function and the fifth loss function is calculated to obtain the second contrastive learning loss function, wherein the fourth loss function is an L1 loss function representing the average absolute error between the fourth feature and the sixth feature; and the fifth loss function is an L1 loss function representing the average absolute error between the fifth feature and the sixth feature.

[0074] In this embodiment, when the fourth feature is φ + The fifth characteristic is φ - When the sixth feature is φ, the corresponding fourth loss function is L1(φ, φ). + The fifth loss function is L1(φ, φ). - If ), then the corresponding second contrastive learning loss function is given by the following formula (1), where CR is the second contrastive learning loss function:

[0075]

[0076] In another embodiment of this disclosure, the sum of the fourth loss function and the fifth loss function is calculated, and the ratio of the fourth loss function and the sum of the loss functions is calculated as the second contrastive learning function. Thus, the distance between the reference sample image and the positive sample image, as well as the loss contrast relationship between the reference sample image and the negative sample image, are determined based on the ratio.

[0077] Step 104: Train the generator model by backpropagation based on the BCE loss function and the second contrastive learning loss function to obtain the target super-resolution network, and then perform super-resolution processing on the test image based on the target super-resolution network to obtain the target super-resolution image.

[0078] In this embodiment, the parameters of the generative model are trained by backpropagation using the BCE loss function and the second contrastive learning loss function to obtain the target super-resolution network. The test image is then super-resolution processed using the target super-resolution network to obtain the target super-resolution image.

[0079] Therefore, in this embodiment, while ensuring that the reference sample image and the positive sample are close at the high-frequency information level when training the target super-resolution network, the similarity between the reference sample image and the positive sample at the feature level is further enhanced based on adversarial training.

[0080] For example, such as Figure 5 As shown, when the sample image is a landscape image, the first feature is The third characteristic is F D The first score is D + The second score is D, and the BCE loss function is BCE(D). + D), the fourth feature is φ + The fifth characteristic is φ -The sixth feature is φ, and the fourth loss function is L1(φ, φ). + The fifth loss function is L1(φ, φ). - Based on the fourth and fifth loss functions, the corresponding second contrastive learning loss function is determined to be CR(φ). - ,φ,φ + Given an input sample image of LR, a positive sample image of GT, a negative sample image of Neg, and a reference sample image of SR, then the reference... Figure 5 The first and third features are processed separately according to the discrimination model to obtain a first score corresponding to the positive sample image and a second score corresponding to the reference sample image. The BCE loss function is determined based on the first and second scores. The generative model of the GAN network is trained using the BCE loss function and the second contrastive learning loss function to obtain the target super-resolution network. The generative model of the GAN network is trained based on two loss functions to further ensure consistency between the super-resolution result (target super-resolution image) and the positive sample image. The generative model of the GAN network is trained based on multiple loss functions to further ensure consistency between the super-resolution result (target super-resolution image) and the positive sample image in high-frequency information, while avoiding the introduction of artifacts and noise, thus improving the detail purity of the super-resolution image.

[0081] In summary, the super-resolution image processing method based on GAN network of this disclosure performs discrimination processing on the first feature and the third feature according to the discrimination model to obtain a first score corresponding to the positive sample image and a second score corresponding to the reference sample image. A BCE loss function is determined based on the first score and the second score. The positive sample image, the negative sample image, and the reference sample image are input into a pre-trained VGG network to obtain a fourth feature corresponding to the positive sample image, a fifth feature corresponding to the negative sample image, and a sixth feature corresponding to the reference sample image. Furthermore, a second contrastive learning loss function is determined based on the fourth feature, the fifth feature, and the sixth feature. The second contrastive learning loss function is used to make the features of the reference sample image close to the features of the positive sample image and far away from the features of the negative sample image. The generative model of the GAN network is trained according to the BCE loss function and the second contrastive learning loss function to obtain a target super-resolution network. The target super-resolution network is then used to perform super-resolution processing on the test image to obtain the target super-resolution image. Therefore, by combining the distances between the input sample image and the positive and negative sample images respectively, the target super-resolution network is trained using the loss value at the feature level. While ensuring the richness of image details in the target super-resolution image output by the target super-resolution network, the purity of the target super-resolution image is further improved.

[0082] In practical applications, in order to further make the reference samples and positive samples close to each other at the feature level while keeping them far away from the negative samples, thereby avoiding the introduction of some artifacts and noise, the model can be trained at the feature level by combining the GAN network discriminative model.

[0083] like Figure 6 As shown, the method also includes:

[0084] Step 601: Extract the second feature corresponding to the negative sample image through the GAN network discriminative model, and determine the first contrastive learning loss function based on the first feature, the second feature and the third feature. The first contrastive learning loss function is used to make the features of the reference sample image close to the features of the negative sample image and far away from the features of the positive sample image.

[0085] In this embodiment, positive sample images, negative sample images, and reference sample images are input into the GAN network discriminative model for feature extraction to obtain a first feature corresponding to the positive sample image, a second feature corresponding to the negative sample image, and a third feature corresponding to the reference sample image.

[0086] In this embodiment, positive sample images, negative sample images, and reference sample images are input into the GAN network discriminative model for feature extraction, obtaining a first feature corresponding to the positive sample image, a second feature corresponding to the negative sample image, and a third feature corresponding to the reference sample image, so as to facilitate the training of the super-resolution network based on the feature dimensions.

[0087] Then, a first contrastive learning loss function is determined based on the first feature, the second feature, and the third feature, wherein the first contrastive learning loss function is used to make the features of the reference sample image close to the features of the negative sample image and far away from the features of the positive sample image.

[0088] In this embodiment, in order to train the super-resolution network, a first contrastive learning loss function is determined based on the first feature, the second feature, and the third feature. The first contrastive learning loss function is used to make the features of the reference sample image close to the features of the negative sample image and far away from the features of the positive sample image. That is, it emphasizes the attention to noise and artifacts, makes the reference sample image far away from the positive sample features, and reduces the probability that the discrimination model will "selectively" ignore complex noise and rare artifacts.

[0089] It should be noted that the method for determining the first contrastive learning loss function based on the first, second, and third features differs in different application scenarios, as shown in the following example:

[0090] In one embodiment of this disclosure, such as Figure 7 As shown, the first contrastive learning loss function is determined based on the first feature, the second feature, and the third feature, including:

[0091] Step 701: Determine the first loss function based on the second and third features.

[0092] In this embodiment, a first loss function is determined based on the first feature corresponding to the negative sample image and the third feature corresponding to the reference sample image, wherein the first loss function represents the distance between the reference sample image and the negative sample image.

[0093] The first loss function can be calculated based on any algorithm for calculating loss values. For example, it can be calculated based on the L1 loss function, which is the mean absolute error (MAE), used to calculate the average distance between the second and third features.

[0094] For example, it can be calculated based on the L2 loss function, which is the mean square error (MSE), used to calculate the average of the squared differences between the second and third features.

[0095] Step 702: Determine the second loss function based on the first feature and the third feature.

[0096] In this embodiment, a second loss function is determined based on a first feature corresponding to a positive sample image and a third feature corresponding to a reference sample image, wherein the first loss function represents the distance between the reference sample image and the positive sample image.

[0097] The second loss function can be calculated based on any algorithm for calculating loss values. For example, it can be calculated based on the L1 loss function, which is the mean absolute error (MAE), used to calculate the average distance between the first feature and the third feature.

[0098] For example, a second loss function can be calculated based on the L2 loss function. The L2 loss function is the mean square error (MSE), which is used to calculate the average of the squared differences between the first feature and the third feature as the second loss function.

[0099] Step 703: Determine the first contrastive learning loss function based on the first loss function and the second loss function.

[0100] In this embodiment, a contrastive learning loss function is determined based on a first loss function and a second loss function. The contrastive learning loss function is used to make the features of the reference sample image move away from the features of the positive sample image and approach the features of the negative sample image.

[0101] It should be noted that the method for determining the contrastive learning loss function based on the first loss function and the second loss function differs in different application scenarios, as shown in the following example:

[0102] In one embodiment of this disclosure, the ratio between a first loss function and a second loss function is calculated to obtain a first contrastive learning loss function, wherein the first loss function is an L1 loss function representing the average absolute error between the second feature and the third feature; and the second loss function is an L1 loss function representing the average absolute error between the first feature and the third feature.

[0103] In this embodiment, when the first feature is The second characteristic is The third characteristic is F D When the first loss function is L1(F), then the corresponding first loss function is L1(F). D , The second loss function is L1(F). D , If ), then the corresponding first contrastive learning loss function is given by the following formula (2), where CR is the first contrastive learning loss function:

[0104]

[0105] In another embodiment of this disclosure, the sum of the first loss function and the second loss function is calculated, and the ratio of the first loss function and the sum of the loss functions is calculated as a contrastive learning function. Based on the ratio, the distance between the reference sample image and the positive sample image, as well as the loss contrast relationship between the reference sample image and the negative sample image, are determined.

[0106] Step 602: Perform backpropagation training to generate the model parameters based on the BCE loss function, the first contrastive learning loss function, and the second contrastive learning loss function to obtain the target super-resolution network.

[0107] In this embodiment, the generative model of the GAN network is trained according to the BCE loss function, the first contrastive learning loss function, and the second contrastive learning loss function to obtain the target super-resolution network.

[0108] In this embodiment, the generative model of the GAN network is trained according to the BCE loss function, the first contrastive learning loss function, and the second contrastive learning loss function. That is, the network parameters of the generative model of the GAN network are adjusted according to the loss values ​​of the BCE loss function, the first contrastive learning loss function, and the second contrastive learning loss function until the loss value of the BCE loss function is less than a preset loss threshold, the loss value of the first contrastive learning loss function is also less than the corresponding loss threshold, and the loss value of the second contrastive learning loss function is also less than the corresponding loss threshold, so as to obtain the target super-resolution network after training.

[0109] Therefore, in this embodiment, while ensuring that the reference sample image and the positive sample are close at the high-frequency information level when training the target super-resolution network, the similarity between the reference sample image and the positive sample at the feature level is further enhanced based on adversarial training.

[0110] For example, such as Figure 8 As shown, when the sample image is a landscape image, the first feature is The third characteristic is F D The first score is D + The second score is D, and the BCE loss function is BCE(D). + D), the first contrastive learning loss function is The second contrastive learning loss function is CR(φ) - ,φ,φ + Given an input sample image of LR, a positive sample image of GT, and a reference sample image of SR, then the reference... Figure 8 The first and third features are processed separately according to the discrimination model to obtain a first score corresponding to the positive sample image and a second score corresponding to the reference sample image. The BCE loss function is determined based on the first and second scores. The generative model of the GAN network is trained by combining the BCE loss function, the first contrastive learning loss function, and the second contrastive learning loss function to obtain the target super-resolution network. The generative model of the GAN network is trained based on the two loss functions to further ensure consistency between the super-resolution result (target super-resolution image) and the positive sample image. Furthermore, the feature extraction process of the GAN network is supervised and trained based on the first contrastive learning loss function to improve the sensitivity of the discrimination model to noise and artifacts.

[0111] In the embodiments of this disclosure, when training the generative model of the GAN network, a third loss function can be determined based on the reference sample image and the positive sample image. For example, the L1 loss function representing the mean absolute error can be determined based on the reference sample image and the positive sample image, or the L2 loss function representing the average of the squared differences can be determined based on the reference sample image and the positive sample image. Then, the generative model of the GAN network is trained based on the BCE loss function, the third loss function, the first contrastive learning loss function, and the second contrastive learning loss function. That is, the network parameters of the generative model of the GAN network are adjusted based on the BCE loss function, the third loss function, the first contrastive learning loss function, and the second contrastive learning loss function until the loss value of the third loss function is less than a preset loss threshold, the loss value of the BCE loss function is less than a preset loss threshold, the loss value of the first contrastive learning loss function is less than a corresponding loss threshold, and the loss value of the second contrastive learning loss function is less than a corresponding loss threshold, so as to obtain the target super-resolution network after training.

[0112] For example, such as Figure 9 As shown, with Figure 8 Taking the scenario shown as an example, a third loss function L1(GT,SR) is determined based on the reference sample image and the positive sample image. The generative model of the GAN network is trained together based on the third loss function, the first contrastive learning function, the BCE loss function and the second contrastive learning function. The generative model of the GAN network is trained based on multiple loss functions to ensure that the super-resolution result (target super-resolution image) and the positive sample image maintain consistency in high-frequency information, while avoiding the introduction of artifacts and noise, and improving the detail purity of the super-resolution image.

[0113] Therefore, in this embodiment, while ensuring that the reference sample image and positive sample are close in feature level while being far away from the negative sample image during the training of the target super-resolution network, some artifacts and noise are avoided. Furthermore, training is conducted based on a third loss function of the reference sample image and positive sample image, further enhancing the closeness between them at the feature level. Moreover, the feature extraction process of the discriminative model is supervised based on the first contrastive learning loss function, making the discriminative model more sensitive to noise and artifacts, thus improving the purity of the target super-resolution image generated based on the target super-resolution network.

[0114] Of course, in one embodiment of this disclosure, the target super-resolution network can be trained separately based on the first contrastive learning function.

[0115] In this embodiment, the generative model of the GAN network is trained according to a first contrastive learning loss function. For example, a preset threshold corresponding to the first contrastive learning loss function is pre-set. When the loss value of the first contrastive learning loss function is greater than the preset threshold, the network parameters of the generative model of the GAN network are adjusted until the loss value of the first contrastive learning loss function is no greater than the preset threshold. At this point, the corresponding target super-resolution network is obtained. Thus, during the training process, the target super-resolution network adds a CR loss for the feature extraction part of the discriminative model. Therefore, the trained target super-resolution model can significantly improve the super-resolution effect on lower-quality images, with noticeable improvements in noise suppression and detail generation. Consequently, by super-resolution processing the test image based on the target super-resolution network to obtain the target super-resolution image, the image achieves high purity while improving the detail richness.

[0116] For example, such as Figure 10 As shown, when the sample image is a landscape image, the first feature is The second characteristic is The third characteristic is F D The first contrastive learning loss function is Given an input sample image of LR, a positive sample image of GT, a negative sample image of Neg, and a reference sample image of SR, then the reference... Figure 10 A first contrastive learning loss function is determined based on the first feature, the second feature, and the third feature, wherein the first contrastive learning loss function is used to make the features of the reference sample image approach the features of the negative sample image and move away from the features of the positive sample image.

[0117] Therefore, positive sample images, negative sample images, and reference sample images are fed into the feature extraction part of the GAN network. Simultaneously, a loss function (CR) is calculated on the features of these three images. This causes the GAN to tend to make the features of the reference image (SR) closer to those of the negative sample image during feature extraction, thus emphasizing the GAN's focus on noise and artifacts, while keeping the features of the reference image away from those of the positive sample image. This reduces the probability of the GAN network selectively ignoring complex noise and rare artifacts. Due to the existence of the CR loss for the GAN feature extraction part, the subsequent GAN discrimination module can more easily distinguish between super-resolution image features and real high-resolution image features, thereby reducing the training difficulty of the GAN network on complex datasets.

[0118] In summary, the super-resolution image processing method based on GAN network in this embodiment supervises the feature extraction process of the discriminative model based on the first contrastive learning loss function, making the discriminative model more sensitive to noise and artifacts. While ensuring that the reference sample image and positive sample are close to each other at the feature level while being far away from the negative sample image when training the target super-resolution network, it avoids the introduction of some artifacts and noise. Furthermore, it trains based on the third loss function of the reference sample image and positive sample image, which strengthens the closeness between the reference sample image and positive sample at the feature level.

[0119] To implement the above embodiments, this disclosure also proposes a super-resolution image processing device based on a GAN network. Figure 11 This is a schematic diagram of a super-resolution image processing device based on a GAN network, provided as an embodiment of this disclosure. This device can be implemented by software and / or hardware and is generally integrated into an electronic device. Figure 11 As shown, the device includes: a first acquisition module 1110, a second acquisition module 1120, a determination module 1130, and a third acquisition module 1140, wherein,

[0120] The first acquisition module 1110 is used to acquire a positive sample image, a negative sample image, and a reference sample image, wherein the positive sample image is the ground truth super-resolution image corresponding to the input sample image, the negative sample image is an image that has been fused and noise-added by the input sample image and the positive sample image, and the reference sample image is an image output by the input sample image after the image quality has been reduced by the generative model of the generative adversarial network to be trained.

[0121] The second acquisition module 1120 is used to extract a first feature corresponding to the positive sample image and a third feature corresponding to the reference sample image through the GAN network discrimination model, and to perform discrimination processing on the first feature and the third feature respectively to obtain a first score corresponding to the positive sample image and a second score corresponding to the reference sample image, and to determine the binary cross-entropy (BCE) loss function based on the first score and the second score.

[0122] The determining module 1130 is used to extract a fourth feature corresponding to the positive sample image, a fifth feature corresponding to the negative sample image, and a sixth feature corresponding to the reference sample image through a preset network, and to determine a second contrastive learning loss function based on the fourth feature, the fifth feature, and the sixth feature, wherein the second contrastive learning loss function is used to make the features of the reference sample image close to the features of the positive sample image and far away from the features of the negative sample image;

[0123] The third acquisition module 1140 is used to train the parameters of the generative model by backpropagation based on the BCE loss function and the second contrastive learning loss function, and to acquire the target super-resolution network, so as to perform super-resolution processing on the test image based on the target super-resolution network to acquire the target super-resolution image.

[0124] The super-resolution image processing apparatus based on GAN network provided in this disclosure can execute the super-resolution image processing method based on GAN network provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0125] To implement the above embodiments, this disclosure also proposes a computer program product, including a computer program / instructions, which, when executed by a processor, implements the GAN-based super-resolution image processing method described in the above embodiments.

[0126] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure.

[0127] The following is a detailed reference. Figure 12 The diagram illustrates a structural schematic suitable for implementing the electronic device 1300 in the embodiments of this disclosure. The electronic device 1300 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 12The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0128] like Figure 12 As shown, the electronic device 1300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage device 1308 into a random access memory (RAM) 1303. The RAM 1303 also stores various programs and data required for the operation of the electronic device 1300. The processing unit 1301, ROM 1302, and RAM 1303 are interconnected via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0129] Typically, the following devices can be connected to I / O interface 1305: input devices 1306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1309. Communication device 1309 allows electronic device 1300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 12 An electronic device 1300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0130] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1309, or installed from storage device 1308, or installed from ROM 1302. When the computer program is executed by processing device 1301, it performs the functions defined in the GAN-based super-resolution image processing method of embodiments of this disclosure.

[0131] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0132] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0133] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0134] The aforementioned computer-readable medium carries one or more programs. When the electronic device executes the aforementioned one or more programs, the electronic device causes the following: to acquire a positive sample image, a negative sample image, and a reference sample image, wherein the positive sample image is a ground truth super-resolution image corresponding to the input sample image, the negative sample image is an image obtained by fusing and adding noise to the input sample image and the positive sample image, and the reference sample image is an image output by the input sample image after image quality reduction processing by a generative adversarial network (GAN) to be trained; to extract a first feature corresponding to the positive sample image and a third feature corresponding to the reference sample image through a GAN network discriminative model; and to perform discriminative processing on the first feature and the third feature respectively to acquire a first feature corresponding to the positive sample image. The system generates a first super-resolution image and a second score corresponding to the reference sample image. Based on the first and second scores, a binary cross-entropy (BCE) loss function is determined. A fourth feature corresponding to the positive sample image, a fifth feature corresponding to the negative sample image, and a sixth feature corresponding to the reference sample image are extracted using a pre-defined network. A second contrastive learning loss function is then determined based on the fourth, fifth, and sixth features. This second contrastive learning loss function aims to make the features of the reference sample image approach the features of the positive sample image and distance them from the features of the negative sample image. Backpropagation is then performed using the BCE loss function and the second contrastive learning loss function to train the parameters of the generative model, obtaining the target super-resolution network. The target super-resolution network is then used to perform super-resolution processing on the test image to obtain the target super-resolution image.

[0135] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0137] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0138] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0139] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0140] According to one or more embodiments of this disclosure, this disclosure provides a super-resolution image processing method based on a GAN network, comprising:

[0141] Obtain positive sample images, negative sample images, and reference sample images, wherein the positive sample image is the ground truth super-resolution image corresponding to the input sample image, the negative sample image is an image that is fused and noise-added by the input sample image and the positive sample image, and the reference sample image is an image output by the input sample image after the image quality is reduced by the generative model of the generative adversarial network to be trained;

[0142] The GAN network discrimination model extracts a first feature corresponding to the positive sample image and a third feature corresponding to the reference sample image, and performs discrimination processing on the first feature and the third feature respectively to obtain a first score corresponding to the positive sample image and a second score corresponding to the reference sample image. The binary cross-entropy (BCE) loss function is determined based on the first score and the second score.

[0143] A fourth feature corresponding to the positive sample image, a fifth feature corresponding to the negative sample image, and a sixth feature corresponding to the reference sample image are extracted through a preset network. A second contrastive learning loss function is determined based on the fourth feature, the fifth feature, and the sixth feature. The second contrastive learning loss function is used to make the features of the reference sample image close to the features of the positive sample image and far away from the features of the negative sample image.

[0144] The parameters of the generative model are trained by backpropagation based on the BCE loss function and the second contrastive learning loss function to obtain the target super-resolution network, and the test image is super-resolution processed by the target super-resolution network to obtain the target super-resolution image.

[0145] According to one or more embodiments of this disclosure, in the super-resolution image processing method based on GAN networks provided in this disclosure, the process of generating negative sample images includes:

[0146] The input sample image is upsampled to obtain a candidate sample image with the same size as the positive sample image;

[0147] Determine a first weight corresponding to the candidate sample image, and determine a second weight corresponding to the positive sample image;

[0148] The fused image is obtained by summing the first product result of the candidate sample image and the first weight, and the second product result of the positive sample image and the second weight;

[0149] Random Gaussian noise is added to the fused image to generate the negative sample image.

[0150] According to one or more embodiments of this disclosure, in the super-resolution image processing method based on GAN networks provided in this disclosure,

[0151] Determining the second contrastive learning loss function based on the fourth feature, the fifth feature, and the sixth feature includes:

[0152] The fourth loss function is determined based on the fourth feature and the sixth feature;

[0153] The fifth loss function is determined based on the fifth feature and the sixth feature;

[0154] The second contrastive learning loss function is determined based on the fourth loss function and the fifth loss function.

[0155] According to one or more embodiments of this disclosure, in the super-resolution image processing method based on GAN networks provided in this disclosure,

[0156] Determining the second contrastive learning loss function based on the fourth loss function and the fifth loss function includes:

[0157] Calculate the ratio between the fourth loss function and the fifth loss function to obtain the second contrastive learning loss function, wherein the fourth loss function is an L1 loss function representing the average absolute error between the fourth feature and the sixth feature; and the fifth loss function is an L1 loss function representing the average absolute error between the fifth feature and the sixth feature.

[0158] According to one or more embodiments of this disclosure, the super-resolution image processing method based on GAN networks provided in this disclosure further includes:

[0159] Also includes:

[0160] The GAN network discrimination model extracts a second feature corresponding to the negative sample image, and a first contrastive learning loss function is determined based on the first feature, the second feature, and the third feature. The first contrastive learning loss function is used to make the features of the reference sample image close to the features of the negative sample image and far away from the features of the positive sample image.

[0161] The step of training the parameters of the generative model through backpropagation based on the BCE loss function and the second contrastive learning loss function to obtain the target super-resolution network includes:

[0162] The parameters of the generative model are trained by backpropagation based on the BCE loss function, the first contrastive learning loss function, and the second contrastive learning loss function to obtain the target super-resolution network.

[0163] According to one or more embodiments of this disclosure, in the super-resolution image processing method based on GAN networks provided in this disclosure,

[0164] Determining the first contrastive learning loss function based on the first feature, the second feature, and the third feature includes:

[0165] A first loss function is determined based on the second feature and the third feature;

[0166] A second loss function is determined based on the first feature and the third feature;

[0167] The first contrastive learning loss function is determined based on the first loss function and the second loss function.

[0168] According to one or more embodiments of this disclosure, the super-resolution image processing method based on GAN networks provided in this disclosure further includes:

[0169] Determining the first contrastive learning loss function based on the first loss function and the second loss function includes:

[0170] Calculate the ratio between the first loss function and the second loss function to obtain the first contrastive learning loss function, wherein the first loss function is an L1 loss function representing the average absolute error between the second feature and the third feature; and the second loss function is an L1 loss function representing the average absolute error between the first feature and the third feature.

[0171] According to one or more embodiments of this disclosure, in the super-resolution image processing method based on GAN networks provided in this disclosure,

[0172] Also includes:

[0173] A third loss function is determined based on the reference sample image and the positive sample image;

[0174] The step of training the parameters of the generative model through backpropagation based on the BCE loss function, the first contrastive learning loss function, and the second contrastive learning loss function to obtain the target super-resolution network includes:

[0175] The parameters of the generative model are trained by backpropagation based on the BCE loss function, the third loss function, the second contrastive learning loss function, and the first contrastive learning loss function to obtain the target super-resolution network. According to one or more embodiments of this disclosure, this disclosure provides a super-resolution image processing apparatus based on a GAN network, comprising:

[0176] The first acquisition module is used to acquire positive sample images, negative sample images, and reference sample images, wherein the positive sample image is the ground truth super-resolution image corresponding to the input sample image, the negative sample image is an image that has been fused and noise-added by the input sample image and the positive sample image, and the reference sample image is an image output by the input sample image after the image quality has been reduced by the generative model of the generative adversarial network to be trained.

[0177] The second acquisition module is used to extract a first feature corresponding to the positive sample image and a third feature corresponding to the reference sample image through the GAN network discrimination model, and to perform discrimination processing on the first feature and the third feature respectively to obtain a first score corresponding to the positive sample image and a second score corresponding to the reference sample image, and to determine the binary cross-entropy (BCE) loss function based on the first score and the second score.

[0178] The determination module is used to extract a fourth feature corresponding to the positive sample image, a fifth feature corresponding to the negative sample image, and a sixth feature corresponding to the reference sample image through a preset network, and to determine a second contrastive learning loss function based on the fourth feature, the fifth feature, and the sixth feature, wherein the second contrastive learning loss function is used to make the features of the reference sample image close to the features of the positive sample image and far away from the features of the negative sample image;

[0179] The third acquisition module is used to train the parameters of the generative model by backpropagation based on the BCE loss function and the second contrastive learning loss function, and to acquire the target super-resolution network, so as to perform super-resolution processing on the test image based on the target super-resolution network to acquire the target super-resolution image.

[0180] According to one or more embodiments of this disclosure, in the super-resolution image processing apparatus based on GAN networks provided in this disclosure, the first acquisition module is specifically used for:

[0181] The input sample image is upsampled to obtain a candidate sample image with the same size as the positive sample image;

[0182] Determine a first weight corresponding to the candidate sample image, and determine a second weight corresponding to the positive sample image;

[0183] The fused image is obtained by summing the first product result of the candidate sample image and the first weight, and the second product result of the positive sample image and the second weight;

[0184] Random Gaussian noise is added to the fused image to generate the negative sample image.

[0185] According to one or more embodiments of this disclosure, in the GAN-based super-resolution image processing apparatus provided by this disclosure, the determining module is specifically used for:

[0186] A first loss function is determined based on the second feature and the third feature;

[0187] A second loss function is determined based on the first feature and the third feature;

[0188] The first contrastive learning loss function is determined based on the first loss function and the second loss function.

[0189] According to one or more embodiments of this disclosure, the super-resolution image processing apparatus based on GAN networks provided in this disclosure further includes:

[0190] The first loss function determination module is used to determine the fourth loss function based on the fourth feature and the sixth feature;

[0191] The second loss function determination module is used to determine the fifth loss function based on the fifth feature and the sixth feature;

[0192] The third loss function determination module is used to determine the second contrastive learning loss function based on the fourth loss function and the fifth loss function.

[0193] According to one or more embodiments of this disclosure, in the GAN-based super-resolution image processing apparatus provided by this disclosure, the third loss function determination module is specifically used for:

[0194] Calculate the ratio between the fourth loss function and the fifth loss function to obtain the second contrastive learning loss function, wherein the fourth loss function is an L1 loss function representing the average absolute error between the fourth feature and the sixth feature; and the fifth loss function is an L1 loss function representing the average absolute error between the fifth feature and the sixth feature.

[0195] According to one or more embodiments of this disclosure, the super-resolution image processing apparatus based on GAN networks provided in this disclosure further includes:

[0196] The extraction module is used to extract a second feature corresponding to the negative sample image through the GAN network discrimination model, and determine a first contrastive learning loss function based on the first feature, the second feature and the third feature, wherein the first contrastive learning loss function is used to make the features of the reference sample image close to the features of the negative sample image and far away from the features of the positive sample image;

[0197] The third acquisition module is specifically used for:

[0198] The parameters of the generative model are trained by backpropagation based on the BCE loss function, the first contrastive learning loss function, and the second contrastive learning loss function to obtain the target super-resolution network.

[0199] According to one or more embodiments of this disclosure, in the GAN-based super-resolution image processing apparatus provided by this disclosure, the extraction module is specifically used for:

[0200] A first loss function is determined based on the second feature and the third feature;

[0201] A second loss function is determined based on the first feature and the third feature;

[0202] The first contrastive learning loss function is determined based on the first loss function and the second loss function.

[0203] According to one or more embodiments of this disclosure, in the GAN-based super-resolution image processing apparatus provided by this disclosure, the extraction module is specifically used for:

[0204] Calculate the ratio between the first loss function and the second loss function to obtain the first contrastive learning loss function, wherein the first loss function is an L1 loss function representing the average absolute error between the second feature and the third feature; and the second loss function is an L1 loss function representing the average absolute error between the first feature and the third feature.

[0205] According to one or more embodiments of this disclosure, the super-resolution image processing apparatus based on GAN networks provided in this disclosure further includes:

[0206] The fourth loss function determination module is used to determine the third loss function based on the reference sample image and the positive sample image;

[0207] The third acquisition module is specifically used to perform backpropagation training of the parameters of the generative model based on the BCE loss function, the third loss function, the second contrastive learning loss function and the first contrastive learning loss function, to acquire the target super-resolution network.

[0208] According to one or more embodiments of this disclosure, this disclosure provides an electronic device, including:

[0209] processor;

[0210] Memory used to store the processor's executable instructions;

[0211] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the super-resolution image processing method based on GAN network as described in any of the present disclosure.

[0212] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium storing a computer program for performing a super-resolution image processing method based on a GAN network as described in any of the present disclosure.

[0213] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0214] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0215] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A super-resolution image processing method based on GAN networks, characterized in that, include: Obtain positive sample images, negative sample images, and reference sample images, wherein the positive sample image is the ground truth super-resolution image corresponding to the input sample image, the negative sample image is an image that is fused and noise-added by the input sample image and the positive sample image, and the reference sample image is an image output by the input sample image after the image quality is reduced by the generative model of the generative adversarial network to be trained; The GAN network discrimination model extracts a first feature corresponding to the positive sample image and a third feature corresponding to the reference sample image, and performs discrimination processing on the first feature and the third feature respectively to obtain a first score corresponding to the positive sample image and a second score corresponding to the reference sample image. The binary cross-entropy (BCE) loss function is determined based on the first score and the second score. A fourth feature corresponding to the positive sample image, a fifth feature corresponding to the negative sample image, and a sixth feature corresponding to the reference sample image are extracted through a preset network. A second contrastive learning loss function is determined based on the fourth, fifth, and sixth features. The second contrastive learning loss function is used to make the features of the reference sample image close to the features of the positive sample image and far away from the features of the negative sample image. The preset network is a pre-trained feature extraction network. The parameters of the generative model are trained by backpropagation based on the BCE loss function and the second contrastive learning loss function to obtain the target super-resolution network, and the test image is super-resolution processed by the target super-resolution network to obtain the target super-resolution image.

2. The method according to claim 1, characterized in that, The process of generating the negative sample image includes: The input sample image is upsampled to obtain a candidate sample image with the same size as the positive sample image; Determine a first weight corresponding to the candidate sample image, and determine a second weight corresponding to the positive sample image; The fused image is obtained by summing the first product result of the candidate sample image and the first weight, and the second product result of the positive sample image and the second weight; Random Gaussian noise is added to the fused image to generate the negative sample image.

3. The method according to claim 1, characterized in that, Determining the second contrastive learning loss function based on the fourth feature, the fifth feature, and the sixth feature includes: The fourth loss function is determined based on the fourth feature and the sixth feature; The fifth loss function is determined based on the fifth feature and the sixth feature; The second contrastive learning loss function is determined based on the fourth loss function and the fifth loss function.

4. The method according to claim 3, characterized in that, Determining the second contrastive learning loss function based on the fourth loss function and the fifth loss function includes: Calculate the ratio between the fourth loss function and the fifth loss function to obtain the second contrastive learning loss function, wherein the fourth loss function is an L1 loss function representing the average absolute error between the fourth feature and the sixth feature; and the fifth loss function is an L1 loss function representing the average absolute error between the fifth feature and the sixth feature.

5. The method according to claim 1, characterized in that, Also includes: The GAN network discrimination model extracts a second feature corresponding to the negative sample image, and a first contrastive learning loss function is determined based on the first feature, the second feature, and the third feature. The first contrastive learning loss function is used to make the features of the reference sample image close to the features of the negative sample image and far away from the features of the positive sample image. The step of training the parameters of the generative model through backpropagation based on the BCE loss function and the second contrastive learning loss function to obtain the target super-resolution network includes: The parameters of the generative model are trained by backpropagation based on the BCE loss function, the first contrastive learning loss function, and the second contrastive learning loss function to obtain the target super-resolution network.

6. The method according to claim 5, characterized in that, Determining the first contrastive learning loss function based on the first feature, the second feature, and the third feature includes: A first loss function is determined based on the second feature and the third feature; A second loss function is determined based on the first feature and the third feature; The first contrastive learning loss function is determined based on the first loss function and the second loss function.

7. The method according to claim 6, characterized in that, Determining the first contrastive learning loss function based on the first loss function and the second loss function includes: Calculate the ratio between the first loss function and the second loss function to obtain the first contrastive learning loss function, wherein the first loss function is an L1 loss function representing the average absolute error between the second feature and the third feature; and the second loss function is an L1 loss function representing the average absolute error between the first feature and the third feature.

8. The method according to claim 5, characterized in that, Also includes: A third loss function is determined based on the reference sample image and the positive sample image; The step of training the parameters of the generative model through backpropagation based on the BCE loss function, the first contrastive learning loss function, and the second contrastive learning loss function to obtain the target super-resolution network includes: The parameters of the generative model are trained by backpropagation based on the BCE loss function, the third loss function, the second contrastive learning loss function, and the first contrastive learning loss function to obtain the target super-resolution network.

9. A super-resolution image processing device based on a GAN network, characterized in that, include: The first acquisition module is used to acquire positive sample images, negative sample images, and reference sample images, wherein the positive sample image is the ground truth super-resolution image corresponding to the input sample image, the negative sample image is an image that has been fused and noise-added by the input sample image and the positive sample image, and the reference sample image is an image output by the input sample image after the image quality has been reduced by the generative model of the generative adversarial network to be trained. The second acquisition module is used to extract a first feature corresponding to the positive sample image and a third feature corresponding to the reference sample image through the GAN network discrimination model, and to perform discrimination processing on the first feature and the third feature respectively to obtain a first score corresponding to the positive sample image and a second score corresponding to the reference sample image, and to determine the binary cross-entropy (BCE) loss function based on the first score and the second score. The determining module is used to extract a fourth feature corresponding to the positive sample image, a fifth feature corresponding to the negative sample image, and a sixth feature corresponding to the reference sample image through a preset network, and to determine a second contrastive learning loss function based on the fourth feature, the fifth feature, and the sixth feature. The second contrastive learning loss function is used to make the features of the reference sample image close to the features of the positive sample image and far away from the features of the negative sample image. The preset network is a pre-trained feature extraction network. The third acquisition module is used to train the parameters of the generative model by backpropagation based on the BCE loss function and the second contrastive learning loss function, and to acquire the target super-resolution network, so as to perform super-resolution processing on the test image based on the target super-resolution network to acquire the target super-resolution image.

10. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the super-resolution image processing method based on GAN network as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the super-resolution image processing method based on GAN network as described in any one of claims 1-8.

Images