Training method and device of image super-resolution model, equipment, medium and product

Abstract

The invention discloses a training method and device of an image super-resolution model, equipment, a medium and a product. Inputting the first image and the second image into a first image super-resolution model, wherein the first image super-resolution model comprises a first initial image degradation sub-model, an initial image super-resolution sub-model and a second initial image degradation sub-model; performing degradation processing on the first image by using the first initial image degradation sub-model, and then performing super-resolution processing to obtain a fourth image; performing super-resolution processing on the second image by using the initial image super-resolution sub-model, and then performing degradation processing to obtain a sixth image; under the condition that the loss value does not meet the condition, adjusting parameters of the model, and returning to input the first image and the second image into the first image super-resolution model; and obtaining a second image super-resolution model under the condition that the loss value meets the condition. Even in the face of unknown degradation, accurate high-resolution images can be constructed, and actual application requirements are met.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a training method, apparatus, device, storage medium, and computer program product for an image super-resolution model. Background Technology

[0002] Super-resolution technology, as an important branch of computer vision, aims to reconstruct high-resolution images with rich details and realistic visuals from low-resolution images using models. This technology, with its ability to improve the quality of low-resolution images, has broad application prospects in many fields such as security monitoring, old photo restoration, remote video communication, and facial recognition. For example, in old photo restoration, it can restore facial details in faded or low-resolution old photos, thus restoring the value of historical images.

[0003] Existing image super-resolution models typically rely on a large number of pairs of low-resolution and high-resolution images (i.e., each low-resolution image corresponds to a high-resolution labeled image of the same scene and the same target) for model training. By learning the mapping relationship between pairs of training data, super-resolution reconstruction can be achieved, enabling the reconstruction of high-resolution images from low-resolution images under specific degradation modes.

[0004] However, in real-world applications, the degradation process of low-resolution images is often affected by a variety of complex and unknown factors, such as random noise interference, non-uniform blurring, compression distortion, and changes in illumination. Existing image super-resolution models can only reconstruct high-resolution images from low-resolution images under specific degradation modes, and their generalization performance is limited. Therefore, when faced with unknown degradation in real-world applications, it is difficult to construct realistic high-resolution images and cannot meet the needs of practical applications. Summary of the Invention

[0005] This application provides a training method, apparatus, device, storage medium, and computer program product for an image super-resolution model, which can construct accurate high-resolution images.

[0006] In a first aspect, embodiments of this application provide a method for training an image super-resolution model, comprising: Acquire the first image and the second image; The first image and the second image are input into the first image super-resolution model, which includes a first initial image degradation sub-model, an initial image super-resolution sub-model, and a second initial image degradation sub-model. The first image is degraded using the first initial image degradation sub-model to obtain the third image, and the third image is super-resolution using the initial image super-resolution sub-model to obtain the fourth image. The second image is super-resolution processed using the initial image super-resolution sub-model to obtain the fifth image. The fifth image is then degraded using the second initial image degradation sub-model to obtain the sixth image. If the first loss value between the second and third images, the second loss value between the third and first images, the third loss value between the fourth and first images, the fourth loss value between the first and fifth images, the fifth loss value between the fifth and second images, and the sixth loss value between the sixth and second images do not meet the first preset training stopping condition, adjust the parameters of the first image super-resolution model and return to inputting the first and second images into the first image super-resolution model. When the first loss value, second loss value, third loss value, fourth loss value, fifth loss value and sixth loss value satisfy the first preset training stopping condition, a second image super-resolution model is obtained. The second image super-resolution model includes a first image degradation sub-model, a target image super-resolution sub-model and a second image degradation sub-model.

[0007] In one possible implementation, if the first loss value between the second and third images, the second loss value between the third and first images, the third loss value between the fourth and first images, the fourth loss value between the first and fifth images, the fifth loss value between the fifth and second images, and the sixth loss value between the sixth and second images do not meet the first preset training stopping condition, the parameters of the first image super-resolution model are adjusted, including: Obtain the first loss value between the second image and the third image, the second loss value between the third image and the first image, the third loss value between the fourth image and the first image, the fourth loss value between the first image and the fifth image, the fifth loss value between the fifth image and the second image, and the sixth loss value between the sixth image and the second image; Obtain the first weights corresponding to the first, second, third, fourth, fifth, and sixth loss values, respectively; Based on the first weight, the first loss value, the second loss value, the third loss value, the fourth loss value, the fifth loss value, and the sixth loss value are weighted and summed to obtain the first target loss value of the first image super-resolution model; If the first target loss value does not meet the first preset training stopping condition, adjust the parameters of the first image super-resolution model.

[0008] In one possible implementation, a second image super-resolution model is obtained when the first loss value, second loss value, third loss value, fourth loss value, fifth loss value, and sixth loss value satisfy a first preset training stopping condition, including: When the first target loss value meets the first preset training stopping condition, the second image super-resolution model is obtained.

[0009] In one possible implementation, obtaining a first loss value between the second image and the third image includes: The first loss value between the second and third images is calculated using the following adversarial loss calculation formula:

[0010] in, This represents the first loss value. This represents the second image. This represents the third image. and Let represent the probability distributions followed by the second and third images, respectively. This represents the score given by the discriminator corresponding to the first initial image degradation sub-model.

[0011] In one possible implementation, obtaining a second loss value between the third image and the first image includes: The second loss value between the third image and the first image is calculated using the following formula:

[0012] in, This represents the second loss value, and N represents the number of images input to the first image super-resolution model each time. Represents the first image. This represents the third image. This indicates bicubic interpolation upsampling.

[0013] In one possible implementation, the third loss value includes the first sub-loss value and the second sub-loss value; obtaining the third loss value between the fourth image and the first image includes: The third loss value between the fourth image and the first image is calculated using the following formula:

[0014] in, This represents the first sub-loss value. This represents the second sub-loss value. Represents the first image. This represents the fourth image. and Let represent the probability distributions followed by the first and fourth images, respectively. This represents the score given by the discriminator corresponding to the super-resolution sub-model of the initial image.

[0015] In one possible implementation, obtaining a fourth loss value between the first image and the fifth image includes: The fourth loss value between the first and fifth images is calculated using the following adversarial loss calculation formula:

[0016] in, This represents the fourth loss value. Represents the first image. This represents the fifth image. and Let represent the probability distributions followed by the first and fifth images, respectively. This represents the score given by the discriminator corresponding to the super-resolution sub-model of the initial image.

[0017] In one possible implementation, obtaining a fifth loss value between the fifth image and the second image includes: The fifth loss value between the fifth image and the second image is calculated using the following formula:

[0018] in, This represents the fifth loss value, and N represents the number of images input to the first image super-resolution model each time. This represents the second image. This represents the fifth image. This indicates bicubic interpolation downsampling.

[0019] In one possible implementation, the sixth loss value includes a third sub-loss value and a fourth sub-loss value; obtaining the sixth loss value between the sixth image and the second image includes: The sixth loss value between the sixth image and the second image is calculated using the following formula:

[0020] in, This represents the third sub-loss value. This represents the fourth sub-loss value. This represents the second image. This represents the sixth image. and Let represent the probability distributions followed by the second and sixth images, respectively. This represents the score given by the discriminator corresponding to the second initial image degradation sub-model.

[0021] In one possible implementation embodiment, before adjusting the parameters of the first image super-resolution model if the first target loss value does not meet the first preset training stopping condition, the method further includes: The first image and the fourth image are input into the prior extraction model. Using the prior extraction model, the depth map, reflection map, global illumination direction and projection view of the first image, as well as the depth map, reflection map, global illumination direction and projection view of the fourth image, are extracted. Based on the depth map, reflection map, global illumination direction and projection view of the first image, the corresponding three-dimensional image of the first image is constructed. Based on the depth map, reflection map, global illumination direction and projection view of the fourth image, the corresponding three-dimensional image of the fourth image is constructed. The three-dimensional images corresponding to the first image and the four images are projected onto a two-dimensional plane to obtain the first reconstructed image corresponding to the first image and the second reconstructed image corresponding to the fourth image. Obtain the seventh loss value between the first and second reconstructed images; Obtain the second weight of the seventh loss value; Based on the first weight and the second weight, the first target loss value and the seventh loss value are weighted and summed to obtain the second target loss value; If the first target loss value does not meet the first preset training stopping condition, adjust the parameters of the first image super-resolution model, including: If the second target loss value does not meet the second preset training stopping condition, adjust the parameters of the first image super-resolution model; The prior extraction model is a model trained using image samples and their corresponding label images.

[0022] In one possible implementation, obtaining a seventh loss value between the first reconstructed image and the second reconstructed image includes: The seventh loss value between the first and second reconstructed images is calculated using the following formula:

[0023] in, This represents the seventh loss value, and N represents the number of images input to the first image super-resolution model each time. This represents the first reconstructed image. This represents the second reconstructed image.

[0024] Secondly, embodiments of this application provide a training apparatus for an image super-resolution model, comprising: The acquisition module is used to acquire the first image and the second image; The input module is used to input the first image and the second image into the first image super-resolution model. The first image super-resolution model includes a first initial image degradation sub-model, an initial image super-resolution sub-model, and a second initial image degradation sub-model. The processing module is used to perform degradation processing on the first image using the first initial image degradation sub-model to obtain the third image, and to perform super-resolution processing on the third image using the initial image super-resolution sub-model to obtain the fourth image. The processing module is also used to perform super-resolution processing on the second image using the initial image super-resolution sub-model to obtain the fifth image, and to perform degradation processing on the fifth image using the second initial image degradation sub-model to obtain the sixth image; The adjustment module is used to adjust the parameters of the first image super-resolution model when the first loss value between the second image and the third image, the second loss value between the third image and the first image, the third loss value between the fourth image and the first image, the fourth loss value between the first image and the fifth image, the fifth loss value between the fifth image and the second image, and the sixth loss value between the sixth image and the second image do not meet the first preset training stopping condition, and then return to input the first image and the second image into the first image super-resolution model. The determination module is used to obtain a second image super-resolution model when the first loss value, second loss value, third loss value, fourth loss value, fifth loss value and sixth loss value satisfy the first preset training stopping condition. The second image super-resolution model includes a first image degradation sub-model, a target image super-resolution sub-model and a second image degradation sub-model.

[0025] Thirdly, embodiments of this application provide an electronic device, the device comprising: A processor and a memory storing computer program instructions; the training method for the above-mentioned image super-resolution model is implemented when the processor executes the computer program instructions.

[0026] Fourthly, embodiments of this application provide a computer storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above-described training method for the image super-resolution model is implemented.

[0027] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by the processor of an electronic device, enable the electronic device to execute the above-described image super-resolution model training method.

[0028] The image super-resolution model training method, apparatus, device, storage medium, and computer program product of this application embodiment acquire a first image and a second image; input the first image and the second image into a first image super-resolution model, the first image super-resolution model including a first initial image degradation sub-model, an initial image super-resolution sub-model, and a second initial image degradation sub-model; perform degradation processing on the first image using the first initial image degradation sub-model to obtain a third image; perform super-resolution processing on the third image using the initial image super-resolution sub-model to obtain a fourth image; perform super-resolution processing on the second image using the initial image super-resolution sub-model to obtain a fifth image; perform degradation processing on the fifth image using the second initial image degradation sub-model to obtain a sixth image; and perform training on the image between the second image and the third image. If the first loss value, the second loss value between the third image and the first image, the third loss value between the fourth image and the first image, the fourth loss value between the first image and the fifth image, the fifth loss value between the fifth image and the second image, and the sixth loss value between the sixth image and the second image do not meet the first preset training stopping condition, the parameters of the first image super-resolution model are adjusted, and the first image and the second image are returned to be input into the first image super-resolution model; if the first loss value, the second loss value, the third loss value, the fourth loss value, the fifth loss value, and the sixth loss value meet the first preset training stopping condition, a second image super-resolution model is obtained, which includes a first image degradation sub-model, a target image super-resolution sub-model, and a second image degradation sub-model.

[0029] This application trains an image super-resolution model using unpaired training data. By degrading the first image before super-resolution and then super-resolutioning the second image before degradation, the loss value of each step of the model is determined, and the model parameters are adjusted to optimize the model. When all loss values ​​meet the training stopping condition, a well-trained image super-resolution model is obtained. This model does not rely on the mapping relationship between paired training data and can construct accurate high-resolution images even in the face of unknown degradation, meeting the needs of practical applications. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart illustrating a training method for an image super-resolution model provided in one embodiment of this application; Figure 2 This is a network framework diagram of a first image super-resolution model provided in another embodiment of this application; Figure 3 This is a network framework diagram of the initial image super-resolution sub-model provided in another embodiment of this application; Figure 4 This is a flowchart illustrating a training method for an image super-resolution model provided in another embodiment of this application; Figure 5 This is a flowchart illustrating a training method for an image super-resolution model provided in another embodiment of this application; Figure 6 This is one of the schematic diagrams of simulation results provided in another embodiment of this application; Figure 7 This is a second schematic diagram of the simulation results provided in another embodiment of this application; Figure 8 This is the third schematic diagram of the simulation results provided in another embodiment of this application; Figure 9 This is a schematic diagram of the structure of a training device for an image super-resolution model provided in another embodiment of this application; Figure 10 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0032] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0033] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0034] It should be noted that the acquisition, storage, use, and processing of data in this application embodiment all comply with the relevant provisions of national laws and regulations.

[0035] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0036] Super-resolution technology, as an important branch of computer vision, aims to reconstruct high-resolution images with rich details and realistic visuals from low-resolution images using models. This technology, with its ability to improve the quality of low-resolution images, has broad application prospects in many fields such as security monitoring, old photo restoration, remote video communication, and facial recognition. For example, in old photo restoration, it can restore facial details in faded or low-resolution old photos, thus restoring the value of historical images.

[0037] Existing image super-resolution models typically rely on a large number of pairs of low-resolution and high-resolution images (i.e., each low-resolution image corresponds to a high-resolution labeled image of the same scene and the same target) for model training. By learning the mapping relationship between pairs of training data, super-resolution reconstruction can be achieved, enabling the reconstruction of high-resolution images from low-resolution images under specific degradation modes.

[0038] However, in real-world applications, the degradation process of low-resolution images is often affected by a variety of complex and unknown factors, such as random noise interference, non-uniform blurring, compression distortion, and changes in illumination. Existing image super-resolution models can only reconstruct high-resolution images from low-resolution images under specific degradation modes, and their generalization performance is limited. Therefore, when faced with unknown degradation in real-world applications, it is difficult to construct realistic high-resolution images and cannot meet the needs of practical applications.

[0039] To address the problems of existing technologies, this application provides a training method, apparatus, device, storage medium, and computer program product for an image super-resolution model. This application utilizes unpaired training data to train the image super-resolution model. By degrading a first image before super-resolution, and then super-resolution a second image before degradation, the loss values ​​for each step of the model are determined, and the model parameters are adjusted to optimize the model. When all loss values ​​meet the training stopping condition, a well-trained image super-resolution model is obtained. This model does not rely on the mapping relationship between paired training data and can construct accurate high-resolution images even in the face of unknown degradation, meeting the needs of practical applications.

[0040] To more clearly illustrate the methods provided in the embodiments of this application, the terms involved in the embodiments of this application will be explained below.

[0041] Cycle-GAN (Constant Adversarial Networks): A deep learning model for unsupervised image-to-image translation.

[0042] The LS3D-W balanced dataset is a subset of the Large Scale 3D Faces in-the-Wild dataset (LS3D-W).

[0043] Lightweight Image Inpainting Generative Adversarial Network (LRGAN) is an improved model based on generative adversarial networks.

[0044] Fraser Inception Distance (FID): A core metric for evaluating the quality of images generated by a model. The core function of FID is to measure the difference in distribution between the generated image set and the real image set in a high-dimensional feature space. A lower FID value indicates that the feature distributions of the two sets of images are more similar, meaning that the generated images are more realistic and of better quality.

[0045] Kernel Inception Distance (KID): KID evaluates the diversity and quality of generated images by calculating the kernel distance between the real and generated images in the feature space. A lower KID value means that the generated image is closer to the distribution of the real image in the feature space, that is, the generated image can better simulate the features of the real image, and the higher the quality and diversity of the generated image.

[0046] Skew Inception Distance (SID): By considering the third moment (skewness) of the feature distribution, SID can more comprehensively describe the differences in feature distribution. The lower the SID value, the closer the feature distribution of the generated image is to the feature distribution of the real image, which means that the quality of the generated image is higher and it can more realistically simulate the features of the real image.

[0047] Mean Absolute Difference (MAD): MAD measures the difference between a generated image and a real image. It is calculated by summing the absolute differences in the grayscale values ​​of corresponding pixels in the generated and real images, and then averaging the sum. A lower MAD value indicates a smaller pixel-level difference between the generated and real images. This means the generated image can more accurately reproduce the details and features of the real image, and is visually closer to the real image, thus resulting in higher quality.

[0048] Learned Perceptual Image Patch Similarity (LPIPS): LPIPS aims to simulate the perception of the human visual system to more accurately measure the similarity or quality differences between images. The lower the LPIPS value, the more similar the generated image is to the real image in perceptual features, which means the higher the quality of the generated image.

[0049] The Natural Image Quality Evaluator (NIQE) is a no-reference image quality assessment metric. Based on a statistical model of a natural scene, it evaluates image quality by calculating the statistical deviation between the tested image and an ideal natural image. A lower NIQE value indicates that the image quality is closer to natural and sharp visual characteristics.

[0050] Multi-Scale Image Quality (MUSIQ): MUSIQ is a metric used to evaluate image quality. The higher the MUSIQ value, the better the image quality, meaning the generated image is closer to the feature distribution of a high-quality natural image, and the less distortion and artificial processing artifacts there are.

[0051] CLIPIQA: CLIPIQA stands for "CLIP Image Quality Assessment." It is an image quality assessment method based on the CLIP (Contrastive Language-Image Pretraining) model. A higher CLIP value indicates that the generated image is more similar to a high-quality image in terms of visual content and semantics.

[0052] Generative Adversarial Networks (GANs) Prior Embedded Networks (GPENs) are a technique for image inpainting, such as in face photo restoration, which can make blurry photos clearer.

[0053] Semi-cycled generative adversarial networks (SCGANs) mitigate the adverse effects of the domain gap between real-world low-resolution face images and synthetic low-resolution images by establishing two independent degradation branches during forward and backward cyclic consistent reconstruction processes while sharing the same recovery branch.

[0054] The training method of the image super-resolution model provided in the embodiments of this application will be described in detail below.

[0055] like Figure 1 As shown, the training method for the image super-resolution model provided in this application embodiment includes the following steps S110 to S160.

[0056] S110, Obtain the first image and the second image.

[0057] The resolution of the first image is higher than that of the second image. For example, the resolution of the first image is 64×64, and the resolution of the second image is 16×16.

[0058] In some embodiments, an unpaired high-resolution first image and a low-resolution second image are acquired. It is understood that the first image is a high-resolution image, which is a high-quality image; and the second image is a low-resolution image, which is a low-quality image.

[0059] In some embodiments, a large number of first images are obtained from a high-resolution face dataset (such as the Flickr Faces-High-Quality (FFHQ) dataset, containing 20,000 images). A large number of second images are obtained from a pre-defined dataset (such as the Widerface dataset, containing 4,000 images). The first and second images are used as training data to train a super-resolution model.

[0060] S120. Input the first image and the second image into the first image super-resolution model. The first image super-resolution model includes a first initial image degradation sub-model, an initial image super-resolution sub-model, and a second initial image degradation sub-model.

[0061] In some embodiments, the first image super-resolution model is constructed based on the cycle consistency principle in CycleGAN, using the forward and backward mappings as the network framework. A specific network framework diagram is shown below. Figure 2 As shown. The first image and the second image are input into the first image super-resolution model, and the forward process is performed by the first initial image degradation sub-model. and the initial image super-resolution sub-model The process is completed collaboratively. The first initial image degradation sub-model is used to degrade the first image, resulting in the third image. Then, the initial image super-resolution sub-model is used to super-resolution the third image, resulting in the fourth image. The backward process is performed using the second initial image degradation sub-model. and the initial image super-resolution sub-model This was done collaboratively. The second image was super-resolution processed using the initial image super-resolution sub-model to obtain the fifth image. The fifth image was then degraded using the second initial image degradation sub-model to obtain the sixth image. A discriminator was then used. Determine the loss from the first image to the third image using a discriminator. Determine the loss from the third image to the fourth image using a discriminator. Determine the loss from the second image to the fifth image using a discriminator. The loss from the fifth image to the sixth image is determined, and the model parameters are then optimized based on these losses.

[0062] In some embodiments, such as Figure 3 As shown, the initial image super-resolution sub-model 210 is implemented using stacked residual blocks (ResBlock) 211 to ensure the transmission of information flow. Each residual block contains two bilinear interpolation upsampling modules to achieve image scale changes. The residual block 211 includes a convolution (conv) layer 2111 and a rectified linear unit (ReLU) layer 2112 for image transformation. The backbone of both the first and second initial image degradation sub-models is still stacked residual blocks. That is, the first initial image degradation sub-model, the initial image super-resolution sub-model, and the second initial image degradation sub-model use the same encoder and decoder structure. The discriminator is implemented using the same number of stacked residual blocks as the initial image super-resolution sub-model.

[0063] In one example, the encoder contains four average pooling layers, and the decoder contains two pixel shuffle upsampling modules to achieve the corresponding image feature scale changes.

[0064] S130. The first image is degraded using the first initial image degradation sub-model to obtain the third image. The third image is then super-resolved using the initial image super-resolution sub-model to obtain the fourth image.

[0065] In some embodiments, after inputting the first image and the second image into the first image super-resolution model, during the forward pass, the first image is input into the first initial image degradation sub-model, and the first initial image degradation sub-model is used to perform degradation processing on the first image to obtain the third image. Then, the third image is input into the initial image super-resolution sub-model, and the initial image super-resolution sub-model is used to perform super-resolution processing on the third image to obtain the fourth image.

[0066] In one example, the first image The resolution is 64×64. The first image... Input to the first initial image degradation sub-model The first image is processed using the first initial image degradation sub-model. The third image is generated after degradation processing. Third image The resolution is 16×16. Then the third image... Input to the initial image super-resolution sub-model Using the initial image super-resolution sub-model to analyze the third image Super-resolution processing was performed to generate the fourth image. Fourth image The resolution is 64×64.

[0067] Understandable and For paired low-resolution and high-resolution images.

[0068] S140. The second image is super-resolution processed using the initial image super-resolution sub-model to obtain the fifth image. The fifth image is then degraded using the second initial image degradation sub-model to obtain the sixth image.

[0069] In some embodiments, after inputting the first image and the second image into the first image super-resolution model, during the backward process, the second image is input into the initial image super-resolution sub-model, and the initial image super-resolution sub-model is used to perform super-resolution processing on the second image to obtain the fifth image. Then, the fifth image is input into the second initial image degradation sub-model, and the second initial image degradation sub-model is used to perform degradation processing on the fifth image to obtain the sixth image.

[0070] In one example, the second image The resolution is 16×16. The second image... Input to the initial image super-resolution sub-model The second image is obtained by using the super-resolution sub-model of the initial image. Super-resolution processing was performed to generate the fifth image. Fifth image The resolution is 64×64. Then the fifth image... Input to the second initial image degradation sub-model Using the second initial image degradation sub-model For the fifth image The sixth image was generated after degradation processing. The sixth image The resolution is 16×16.

[0071] S150, if the first loss value between the second image and the third image, the second loss value between the third image and the first image, the third loss value between the fourth image and the first image, the fourth loss value between the first image and the fifth image, the fifth loss value between the fifth image and the second image, and the sixth loss value between the sixth image and the second image do not meet the first preset training stopping condition, adjust the parameters of the first image super-resolution model, and return to inputting the first image and the second image into the first image super-resolution model.

[0072] In some embodiments, the loss value of the first image super-resolution model in the forward process for transforming the first image into the third image, and the loss value for transforming the third image into the fourth image, are calculated. The loss value of the first image super-resolution model in the backward process for transforming the second image into the fifth image, and the loss value for transforming the fifth image into the sixth image, are calculated. The first image super-resolution model is trained using the loss values ​​from image transformations in the forward and backward processes.

[0073] In some embodiments, the loss value for converting the first image into the third image can be calculated in two ways: one is to calculate a first loss value between the second and third images, and the other is to calculate a second loss value between the third image and the first image. The first loss value reflects the probability that the third image conforms to the distribution domain of a true low-quality image, and is measured using a true second image. The second loss value reflects the degree of distortion that occurs during the conversion of the first image into the third image. Training the first image super-resolution model by calculating the first and second loss values ​​enables the low-quality and high-quality images generated by the first image super-resolution model to be more accurate, and the high-quality images to be more realistic.

[0074] In some embodiments, training a first initial image degradation sub-model in a first image super-resolution model by calculating a first loss value and a second loss value can make the low-quality images generated by the first initial image degradation sub-model more realistic.

[0075] In some embodiments, a third loss value is calculated between the fourth image and the first image, and this third loss value is used as the loss value for transforming the third image into the fourth image. The third loss value reflects the loss between the high-quality image generated by the super-resolution model of the first image and the real high-quality image.

[0076] In some embodiments, the loss value for transforming the second image into the fifth image can be calculated in two ways: one is to calculate a fourth loss value between the first and fifth images, and the other is to calculate a fifth loss value between the fifth and second images. The fourth loss value reflects the probability that the fifth image conforms to the distribution domain of a true high-quality image, and is measured using the true first image. The fifth loss value reflects the degree of distortion that occurs during the transformation of the second image into the fifth image. Training the first image super-resolution model by calculating the fourth and fifth loss values ​​enables the low-quality and high-quality images generated by the first image super-resolution model to be more accurate, and the high-quality images to be more realistic.

[0077] In some embodiments, training the initial image super-resolution sub-model in the first image super-resolution model by calculating the fourth and fifth loss values ​​can make the high-quality images generated by the initial image super-resolution sub-model more realistic.

[0078] In some embodiments, the first preset training stopping condition includes sub-preset training stopping conditions corresponding to the first loss value between the second and third images, the second loss value between the third and first images, the third loss value between the fourth and first images, the fourth loss value between the first and fifth images, the fifth loss value between the fifth and second images, and the sixth loss value between the sixth and second images, respectively. If the first loss value between the second and third images, the second loss value between the third and first images, the third loss value between the fourth and first images, the fourth loss value between the first and fifth images, the fifth loss value between the fifth and second images, and the sixth loss value between the sixth and second images do not meet the first preset training stopping condition, the parameters of the first image super-resolution model are adjusted, including: If the first loss value between the second and third images does not meet the first sub-preset training stopping condition, the second loss value between the third and first images does not meet the second sub-preset training stopping condition, the third loss value between the fourth and first images does not meet the third sub-preset training stopping condition, the fourth loss value between the first and fifth images does not meet the fourth sub-preset training stopping condition, the fifth loss value between the fifth and second images does not meet the fifth sub-preset training stopping condition, or the sixth loss value between the sixth and second images does not meet the sixth sub-preset training stopping condition, the parameters of the first image super-resolution model are adjusted.

[0079] In this embodiment, if the first loss value between the second and third images does not meet the first sub-preset training stopping condition, or if the second loss value between the third and first images does not meet the first sub-preset training stopping condition, the parameters of the first initial image degradation sub-model are adjusted. If the third loss value between the fourth and first images does not meet the third sub-preset training stopping condition, the parameters of the initial image super-resolution sub-model are adjusted. If the fourth loss value between the first and fifth images does not meet the fourth sub-preset training stopping condition, or if the fifth loss value between the fifth and second images does not meet the fifth sub-preset training stopping condition, the parameters of the initial image super-resolution sub-model are adjusted. If the sixth loss value between the sixth and second images does not meet the sixth sub-preset training stopping condition, the parameters of the second initial image degradation sub-model are adjusted.

[0080] In some embodiments, if the first loss value between the second image and the third image, the second loss value between the third image and the first image, the third loss value between the fourth image and the first image, the fourth loss value between the first image and the fifth image, the fifth loss value between the fifth image and the second image, and the sixth loss value between the sixth image and the second image do not meet the first preset training stopping condition, the parameters of the first image super-resolution model are adjusted, including: Obtain the first weights corresponding to the first, second, third, fourth, fifth, and sixth loss values. Based on these first weights, perform a weighted sum of the first, second, third, fourth, fifth, and sixth loss values ​​to obtain the first target loss value of the first image super-resolution model. If the first target loss value does not meet the first preset training stopping condition, adjust the parameters of the first image super-resolution model. The first weights corresponding to the first, second, third, fourth, fifth, and sixth loss values ​​are pre-set according to actual needs.

[0081] In some embodiments, if the first loss value between the second image and the third image, the second loss value between the third image and the first image, the third loss value between the fourth image and the first image, the fourth loss value between the first image and the fifth image, the fifth loss value between the fifth image and the second image, and the sixth loss value between the sixth image and the second image do not meet the first preset training stopping condition, the parameters of the first image super-resolution model are adjusted, and the input of the first image and the second image into the first image super-resolution model is returned.

[0082] In some embodiments, the parameters of the first image super-resolution model are adjusted according to the first loss value, the second loss value, the third loss value, the fourth loss value, the fifth loss value, and the sixth loss value until the first loss value, the second loss value, the third loss value, the fourth loss value, the fifth loss value, and the sixth loss value satisfy the first preset training stopping condition, thereby obtaining the second image super-resolution model.

[0083] In this embodiment, after inputting the first and second images into the first image super-resolution model, since the parameters of the first image super-resolution model have been adjusted, the recalculated first, second, third, fourth, fifth, and sixth loss values ​​may be different. If the recalculated first, second, third, fourth, fifth, and sixth loss values ​​still do not meet the first preset training stopping condition, the parameters of the first image super-resolution model are further adjusted, and the first and second images are input into the first image super-resolution model again, until the first, second, third, fourth, fifth, and sixth loss values ​​meet the first preset training stopping condition, thus obtaining the second image super-resolution model. The second image super-resolution model represents the trained image super-resolution model.

[0084] In some embodiments, the first preset training stopping condition is that the loss value converges to a minimum.

[0085] In some embodiments, the first preset training stopping condition is that the loss value is less than a preset threshold.

[0086] S160. If the first loss value, second loss value, third loss value, fourth loss value, fifth loss value and sixth loss value satisfy the first preset training stopping condition, a second image super-resolution model is obtained. The second image super-resolution model includes a first image degradation sub-model, a target image super-resolution sub-model and a second image degradation sub-model.

[0087] In some embodiments, the first preset training stopping condition includes sub-preset training stopping conditions corresponding to the first loss value between the second and third images, the second loss value between the third and first images, the third loss value between the fourth and first images, the fourth loss value between the first and fifth images, the fifth loss value between the fifth and second images, and the sixth loss value between the sixth and second images, respectively. When the first, second, third, fourth, fifth, and sixth loss values ​​satisfy the first preset training stopping condition, a second image super-resolution model is obtained, including: The second image super-resolution model is obtained when the first loss value between the second and third images satisfies the first sub-preset training stopping condition, the second loss value between the third and first images satisfies the second sub-preset training stopping condition, the third loss value between the fourth and first images satisfies the third sub-preset training stopping condition, the fourth loss value between the first and fifth images satisfies the fourth sub-preset training stopping condition, the fifth loss value between the fifth and second images satisfies the fifth sub-preset training stopping condition, and the sixth loss value between the sixth and second images satisfies the sixth sub-preset training stopping condition.

[0088] In some embodiments, when the first loss value, second loss value, third loss value, fourth loss value, fifth loss value, and sixth loss value satisfy a first preset training stopping condition, a second image super-resolution model is obtained, including: The first weights corresponding to the first, second, third, fourth, fifth, and sixth loss values ​​are obtained. Based on these first weights, the first, second, third, fourth, fifth, and sixth loss values ​​are weighted and summed to obtain the first target loss value of the first image super-resolution model. When the first target loss value satisfies a first preset training stopping condition, the second image super-resolution model is obtained. The first weights corresponding to the first, second, third, fourth, fifth, and sixth loss values ​​are pre-set according to actual needs.

[0089] This application utilizes unpaired training data to train an image super-resolution model. By degrading the first image before super-resolution and then super-resolution the second image before degradation, the loss value of each step in the model is determined, and the model parameters are adjusted to optimize the model. When all loss values ​​meet the training stopping condition, a well-trained image super-resolution model is obtained. This model does not rely on the mapping relationship between paired training data and can construct accurate high-resolution images even in the face of unknown degradation, meeting the needs of practical applications.

[0090] In some embodiments, such as Figure 4 As shown, S150 may specifically include S151 to S154.

[0091] S151. Obtain the first loss value between the second image and the third image, the second loss value between the third image and the first image, the third loss value between the fourth image and the first image, the fourth loss value between the first image and the fifth image, the fifth loss value between the fifth image and the second image, and the sixth loss value between the sixth image and the second image.

[0092] In some embodiments, during each iteration of the image super-resolution model training, the following values ​​are calculated during the forward and backward processes: a first loss value between the second and third images, a second loss value between the third and first images, a third loss value between the fourth and first images, a fourth loss value between the first and fifth images, a fifth loss value between the fifth and second images, and a sixth loss value between the sixth and second images.

[0093] In some embodiments, obtaining a first loss value between the second image and the third image in S151 above includes: The first loss value between the second and third images is calculated using the following adversarial loss calculation formula (1): (1) in, This represents the first loss value. This represents the second image. This represents the third image. and Let represent the probability distributions followed by the second and third images, respectively. This represents the score given by the discriminator corresponding to the first initial image degradation sub-model.

[0094] In one example, a discriminator is used. Calculate the second image With the third image The first loss value between .

[0095] It should be noted that the second image This is a true low-quality image, so It can be set to a constant. Therefore, it affects the first loss value. The parameters are The generated The more realistic and consistent with the distribution domain of real low-quality images, the better the discriminator... The higher the score given, The higher the first loss value The smaller.

[0096] By setting the above formula (1) in the embodiments of this application, the third image output by the first initial image degradation sub-model can be made more realistic and more in line with the distribution domain of real low-quality images, thereby making the trained second image super-resolution model better meet business needs.

[0097] In some embodiments, obtaining the second loss value between the third image and the first image in S151 above includes: The second loss value between the third image and the first image is calculated using the following formula (2): (2) in, This represents the second loss value, and N represents the number of images input to the first image super-resolution model each time. Represents the first image. This represents the third image. This indicates bicubic interpolation upsampling.

[0098] In this embodiment, the third image is upsampled using bicubic interpolation and then compared with the first image for loss calculation.

[0099] By setting the above formula (2) in the embodiments of this application, the structural information of the third image output by the first initial image degradation sub-model will not be distorted too much.

[0100] In some embodiments, in S151 above, the third loss value includes the first sub-loss value and the second sub-loss value; obtaining the third loss value between the fourth image and the first image includes: The third loss value between the fourth image and the first image is calculated using the following formula (3): (3) in, This represents the first sub-loss value. This represents the second sub-loss value. Represents the first image. This represents the fourth image. and Let represent the probability distributions followed by the first and fourth images, respectively. This represents the score given by the discriminator corresponding to the super-resolution sub-model of the initial image.

[0101] In this embodiment, Characterizes the cycle consistency loss between the fourth image and the first image. Characterize the adversarial loss between the fourth image and the first image.

[0102] In one example, a discriminator is used. Calculate the fourth image With the first image The second sub-loss value between .

[0103] It should be noted that the first image For true high-quality images, so It can be set to a constant. Therefore, it affects the second sub-loss value. The parameters are The generated The more realistic and consistent with the distribution domain of real high-quality images, the better the discriminator... The higher the score given, The higher the value, the higher the second sub-loss value. The smaller.

[0104] By setting the above formula (3) in the embodiments of this application, the initial image super-resolution sub-model can output an image that conforms to the distribution domain of a real high-quality image, and the output image has correct pixel and structural information.

[0105] In some embodiments, obtaining a fourth loss value between the first image and the fifth image in S151 above includes: The fourth loss value between the first and fifth images is calculated using the following adversarial loss calculation formula (4): (4) in, This represents the fourth loss value. Represents the first image. This represents the fifth image. and Let represent the probability distributions followed by the first and fifth images, respectively. This represents the score given by the discriminator corresponding to the super-resolution sub-model of the initial image.

[0106] In one example, a discriminator is used. Calculate the first image With the fifth image The fourth loss value between .

[0107] It should be noted that the first image For true high-quality images, so It can be set to a constant. Therefore, it affects the fourth loss value. The parameters are The fifth image generated. The more realistic and consistent with the distribution domain of real high-quality images, the better the discriminator... The higher the score given, The higher the value, the higher the fourth loss value. The smaller.

[0108] By setting the above formula (4) in the embodiments of this application, the fifth image output by the initial image super-resolution sub-model can be made more realistic and more in line with the distribution domain of real high-quality images, thereby making the trained second image super-resolution model better meet business needs.

[0109] In some embodiments, obtaining a fifth loss value between the fifth image and the second image in S151 above includes: The fifth loss value between the fifth image and the second image is calculated using the following formula (5): (5) in, This represents the fifth loss value, and N represents the number of images input to the first image super-resolution model each time. This represents the second image. This represents the fifth image. This indicates bicubic interpolation downsampling.

[0110] In this embodiment, the fifth image is downsampled using bicubic interpolation and then compared with the second image for loss calculation.

[0111] By setting the above formula (5) in the embodiments of this application, the structural information of the fifth image output by the initial image super-resolution sub-model will not be distorted too much, thus maintaining the structure and identity information of the generated super-resolution image.

[0112] In some embodiments, in S151 above, the sixth loss value includes a third sub-loss value and a fourth sub-loss value; obtaining the sixth loss value between the sixth image and the second image includes: The sixth loss value between the sixth image and the second image is calculated using the following formula (6): (6) in, This represents the third sub-loss value. This represents the fourth sub-loss value. This represents the second image. This represents the sixth image. and Let represent the probability distributions followed by the second and sixth images, respectively. This represents the score given by the discriminator corresponding to the second initial image degradation sub-model.

[0113] In this embodiment, Characterizes the cycle consistency loss between the sixth image and the second image. Characterize the adversarial loss between the sixth image and the second image.

[0114] In one example, a discriminator is used. Calculate the sixth image With the second image The fourth sub-loss value between .

[0115] It should be noted that the second image This is a true low-quality image, so It can be set to a constant. Therefore, it affects the fourth sub-loss value. The parameters are The generated sixth image The more realistic and consistent with the distribution domain of real low-quality images, the better the discriminator... The higher the score given, The higher the value, the higher the fourth sub-loss value. The smaller.

[0116] By setting the above formula (6) in the embodiments of this application, the second initial image degradation sub-model can output an image that conforms to the distribution domain of real low-quality images, and the output image has correct pixel and structural information.

[0117] It should be noted that, due to the two discriminators... , The input image size is 16×16, therefore it can contain two max-pooling layers. The other two discriminators... , The input image size is 64×64, so it can contain four max pooling layers.

[0118] S152. Obtain the first weights corresponding to the first loss value, second loss value, third loss value, fourth loss value, fifth loss value and sixth loss value respectively.

[0119] The first weights corresponding to the first, second, third, fourth, fifth, and sixth loss values ​​are pre-set according to actual needs.

[0120] S153. Based on the first weight, the first loss value, the second loss value, the third loss value, the fourth loss value, the fifth loss value, and the sixth loss value are weighted and summed to obtain the first target loss value of the first image super-resolution model.

[0121] In some embodiments, all loss values ​​in the forward and backward processes are weighted and summed to obtain the first target loss value of the first image super-resolution model.

[0122] S154. If the first target loss value does not meet the first preset training stopping condition, adjust the parameters of the first image super-resolution model.

[0123] In some embodiments, the forward and backward processes are performed simultaneously during training. Each loss term participates in backpropagation and model optimization simultaneously according to a set first weight. The first initial image degradation sub-model, the initial image super-resolution sub-model, and the second initial image degradation sub-model mutually incentivize each other, ultimately enabling the image super-resolution sub-model to super-resolution real low-quality images. If the first target loss value does not meet the first preset training stopping condition, the parameters of the first image super-resolution model are adjusted. After inputting the first and second images back into the first image super-resolution model, since the parameters of the first image super-resolution model have been adjusted, the recalculated first, second, third, fourth, fifth, and sixth loss values ​​may be different. If the first target loss value obtained through the recalculated first, second, third, fourth, fifth, and sixth loss values ​​still does not meet the first preset training stopping condition, the parameters of the first image super-resolution model are further adjusted, and the first and second images are input back into the first image super-resolution model until the first target loss value meets the first preset training stopping condition, resulting in the second image super-resolution model. The second image super-resolution model represents the trained image super-resolution model.

[0124] In this embodiment, the forward and backward processes are performed simultaneously during training. Each loss is simultaneously involved in backpropagation and model optimization according to the set first weight. The first initial image degradation sub-model, the initial image super-resolution sub-model, and the second initial image degradation sub-model mutually stimulate each other, ultimately enabling the image super-resolution sub-model to have the ability to super-resolution real low-quality images.

[0125] In some embodiments, the above-mentioned S160 may specifically include: When the first target loss value meets the first preset training stopping condition, the second image super-resolution model is obtained.

[0126] In this embodiment, the forward and backward processes are performed simultaneously during training. The first initial image degradation sub-model, the initial image super-resolution sub-model, and the second initial image degradation sub-model mutually stimulate each other. The training of the model is terminated by determining whether to use the first target loss value, so that the final image super-resolution sub-model has the ability to super-resolution real low-quality images.

[0127] In the embodiments provided in this application, in addition to the basic loss values ​​in the training process described above, highly reliable 3D face prior information is also introduced to assist in the training of the image super-resolution model.

[0128] A prior extraction model is constructed. A large number of high-quality image samples are obtained from high-resolution datasets (such as the Flickr Faces High-Quality (FFHQ) dataset, containing 70,000 images; and the CelebA High-Quality (CelebAHQ) dataset, containing 30,000 images). For example, the pixel size of the high-quality image samples is 1024×1024. These high-quality image samples are used as the training dataset for the prior extraction model.

[0129] The prior extraction model's network structure mainly consists of an encoder and a decoder. The activation function of the final layer is the tanh function. By inputting the training dataset into the prior extraction model, three-dimensional feature information about high-quality image samples can be obtained. During the training of the prior extraction model, the size of each image in the training dataset is first downsampled to 64×64, with each batch containing 64 images. After the images are input into the prior extraction model, the encoder-decoder structure outputs the depth map and reflectance map (i.e., the three-dimensional feature information of the input image). The projection viewpoint and global illumination direction information of the input image are generated by an independent encoder in the prior extraction model. The total number of training iterations for this prior extraction model is a preset value (e.g., ...). (times), parameter optimization selects the Adam optimizer, the training loss tends to a stable state, or the number of iterations of the model reaches a preset value, the prior extraction model stops training, and the network structure and corresponding model parameters are saved.

[0130] In some embodiments, the prior extraction model can reconstruct a 3D face from a single image and render it back to a 2D plane. Given an image The prior extraction model uses a neural network consisting of multiple subnetworks. Mapping an image into a set of elements containing three-dimensional feature information Including depth maps Reflectance spectrum Global illumination direction and projection angle A 3D face can then be reconstructed from this set of elements, and reconstructed into a corresponding image through projection and rendering. This process can be represented by the following formula (7): (7) in, This represents the illumination function, which can be based on the depth map. Reflectance pattern a and illumination direction From a standard perspective Generate a 3D face; global projection view. Represents the standard viewpoint and the actual input image. The shift between perspectives; This represents the reprojection function, which reprojects the aforementioned 3D face onto a 2D plane to obtain the corresponding reconstructed face image. .

[0131] It should be noted that the prior extraction model is a supervised model.

[0132] In some embodiments, such as Figure 5 As shown, before S154 above, the method may also include S310 to S350.

[0133] S310. Input the first image and the fourth image into the prior extraction model. Using the prior extraction model, extract the depth map, reflection map, global illumination direction and projection view of the first image, as well as the depth map, reflection map, global illumination direction and projection view of the fourth image. Based on the depth map, reflection map, global illumination direction and projection view of the first image, construct the three-dimensional image corresponding to the first image. Based on the depth map, reflection map, global illumination direction and projection view of the fourth image, construct the three-dimensional image corresponding to the fourth image.

[0134] The prior extraction model is a model trained using image samples and their corresponding label images.

[0135] In some embodiments, during the aforementioned forward process, for the initial image super-resolution sub-model In terms of having pairs Furthermore, under the training framework of the first image super-resolution model, low-quality images are generated. It fully simulates various complex degradation scenarios in the real world. Therefore, we can use a pre-trained prior extraction model to constrain the initial image super-resolution sub-model.

[0136] The depth map, reflectance map, global illumination direction and projection view are extracted from the first image of the real high-quality image to construct the three-dimensional image corresponding to the first image. The process is as follows: (8) (8) in, , , These represent the depth map, reflectance map, projection viewpoint, and global illumination direction of the first image, respectively. ( ) indicates the prior extraction model. The depth map, reflection map, global illumination direction, and projection view are derived from real high-quality images, and therefore can be used as a reference for correcting the 3D structural features of the face output by the super-resolution sub-model of the initial image.

[0137] From the third image The depth map, reflectance map, global illumination direction, and projection view are extracted from the reconstructed fourth image. Based on the depth map, reflectance map, global illumination direction, and projection view of the fourth image, a three-dimensional image corresponding to the fourth image is constructed. This process is as follows: (9) (9) in, , , These represent the depth map, reflectance map, projection viewpoint, and global illumination direction of the fourth image, respectively. () represents the prior extraction model.

[0138] S320. Project the three-dimensional image corresponding to the first image and the three-dimensional image corresponding to the fourth image onto a two-dimensional plane to obtain the first reconstructed image corresponding to the first image and the second reconstructed image corresponding to the fourth image.

[0139] In some embodiments, the three-dimensional image corresponding to the first image is projected onto a two-dimensional plane using the following formula (10) to obtain the first reconstructed image corresponding to the first image: (10) in, , , These represent the depth map, reflectance map, projection viewpoint, and global illumination direction of the first image, respectively. Represents the illumination function. Represents the reprojection function; This represents the first reconstructed image.

[0140] Using the following formula (11), the three-dimensional image corresponding to the fourth image is projected onto a two-dimensional plane to obtain the second reconstructed image corresponding to the fourth image: (11) in, , , These represent the depth map, reflectance map, projection viewpoint, and global illumination direction of the fourth image, respectively. Represents the illumination function. Represents the reprojection function; This represents the second reconstructed image.

[0141] S330, Obtain the seventh loss value between the first reconstructed image and the second reconstructed image.

[0142] In some embodiments, to align the first reconstructed image and the second reconstructed image, the above-mentioned S330 may specifically include: The seventh loss value between the first and second reconstructed images is calculated using the following formula (12): (12) in, This represents the seventh loss value, and N represents the number of images input to the first image super-resolution model each time. This represents the first reconstructed image. This represents the second reconstructed image.

[0143] In this embodiment, formula (12) is used to correct the three-dimensional information of the initial image super-resolution sub-model, thereby guiding the training and optimization process of the initial image super-resolution sub-model and making it pay more attention to the three-dimensional structure of the generated image.

[0144] S340, Obtain the second weight of the seventh loss value.

[0145] The second weight of the seventh loss value is pre-set according to actual needs.

[0146] S350. Based on the first weight and the second weight, the first target loss value and the seventh loss value are weighted and summed to obtain the second target loss value.

[0147] In some embodiments, the first loss value, the second loss value, the third loss value, the fourth loss value, the fifth loss value, the sixth loss value, and the seventh loss value are weighted and summed according to the first weight and the second weight to obtain the second target loss value.

[0148] Specifically, S154 may include: If the second target loss value does not meet the second preset training stopping condition, adjust the parameters of the first image super-resolution model.

[0149] With the help of a three-dimensional feature correction mechanism, this application embodiment relies on three-dimensional prior information (i.e., depth map, reflection map, global illumination direction, and projection viewpoint) from real high-quality images to correct the geometric structure, contour, and other features of the image generated by the initial image super-resolution sub-model in high-dimensional space, making it closer to the distribution domain of the real image.

[0150] It should be noted that the seventh loss value only participates in the training process of the above-mentioned forward process. Through the training process, the initial image super-resolution sub-model fully learns the three-dimensional prior information of high-quality images, but does not participate in the inference stage. Therefore, the image super-resolution model does not need to extract and incorporate three-dimensional prior information during inference, thus not increasing the inference latency of the backbone model, the computational cost of the inference stage, or the computational complexity.

[0151] The backward process does not introduce 3D prior information to guide the initial image super-resolution sub-model. Firstly, the backward process refers to low-quality images... After obtaining the initial image super-resolution sub-model After passing through get The backward process does not introduce 3D prior information to guide the initial image super-resolution sub-model because it lacks high-quality, high-resolution images to serve as the generated images. The truth labels are missing, making it impossible to calculate the loss value of the 3D prior information. Because the data initially given for this unsupervised task is unpaired. , However, the embodiments of this application require extracting three-dimensional prior information from high-quality images (extracted from real high-resolution image boxes), therefore it can only be obtained from... The forward process acquires the image, while the backward process only uses low-resolution real images. This approach cannot effectively incorporate 3D prior information. Furthermore, the ultimate goal of this application's embodiments is to train a high-quality image super-resolution sub-model; therefore, the output of the image super-resolution sub-model is processed using a loss function. Control is implemented, especially in low-resolution images. Adding a loss term with 3D prior information does not significantly help the image super-resolution sub-model.

[0152] The embodiments of this application can be divided into four steps in terms of process. First, a prior extraction model is constructed. Second, an unsupervised first image super-resolution model is constructed. Then, the three-dimensional prior information extracted by the prior extraction model is used to assist in the training of the first image super-resolution model, ultimately obtaining an unsupervised image super-resolution model with strong generalization ability and good performance. The embodiments of this application extract highly reliable three-dimensional prior information from high-quality images in a non-paired training set. Subsequently, a three-dimensional feature correction mechanism is used to fully integrate the extracted three-dimensional prior information into the training of the unsupervised first image super-resolution model to improve the naturalness, fidelity, and three-dimensional structure of the generated image.

[0153] This application's embodiments utilize unpaired datasets for training within an unsupervised model framework, enabling the model to perform higher-quality super-resolution on images from the low-resolution domain. This results in strong generalization performance. Furthermore, the introduction of 3D prior information from high-quality images further enhances the model's ability to learn the texture distribution of high-resolution, high-quality images. Therefore, the trained model not only possesses strong generalization capabilities but also performs high-quality super-resolution on complex image data.

[0154] The following describes the effect of image super-resolution using the embodiments of this application.

[0155] Low-resolution images are selected from the image test sample set and input into the trained image super-resolution model for testing. The image test sample set can be an open-source dataset or a self-constructed dataset. The following section describes the simulation process and results using an open-source dataset.

[0156] The system was run on an Intel(R) Xeon(R) Silver 4114 2.20GHz CPU, an NVIDIA Tesla V100 processor, and an Ubuntu 16.04 operating system. Python was used as the programming language, and PyTorch was used as the deep learning network framework. The training datasets included CelebAHQ, FFHQ, and Widerface, with 30,000 images in CelebAHQ, 70,000 in FFHQ, and 4,000 in Widerface. The test sets included the FFHQ dataset, the LS3D-W balanced dataset, the Widerface dataset, and the WebFace dataset. The LS3D-W balanced and FFHQ datasets contained only high-resolution real-world face images and no low-quality real-world images. Therefore, the low-quality images used in the super-resolution models for the input images in the tests were artificially synthesized using a specific degradation method (e.g., bilinear downsampling), while the Widerface and WebFace datasets only contained low-quality real-world images and no corresponding high-quality images. To ensure a fair and objective evaluation of the performance of the embodiments in this application, all models compared were trained on the same training dataset. To verify the general effectiveness of the 3D feature correction mechanism in this application's embodiments within existing mainstream unsupervised face super-resolution frameworks, experimental evaluations were conducted based on two other widely accepted unsupervised frameworks: CycleGAN and LRGAN.

[0157] The results obtained from the simulation experiments are shown in Table 1 and Table 2. Figure 6 , Figure 7 , Figure 8As shown in Table 1, [1] represents the results of GPEN, [2] represents the results of RestoreFormer, a high-quality blind face restoration model based on non-degenerate key-value pairs, [3] represents the results of Generative and Controllable Face Super Resolution Method Without Facial and GAN Priors (GCFSR), [4] represents the results of the vector quantization face restoration model CodeForme, [6] represents the results of Self-Supervised Photo Upsampling via LatentSpace Exploration of Generative Models (PULSE), [7] represents the results of LRGAN, [8] represents the results of SCGAN, and [9] represents the results of the embodiments of this application. Figure 6 and Figure 7 In the diagram, [1] represents a low-quality image, [2] represents the result of GPEN, [3] represents the result of RestoreFormer, [4] represents the result of GCFSR, [5] represents the result of VQFR, [6] represents the result of CodeFormer, [7] represents the result of PULSE, [8] represents the result of LRGAN, [9] represents the result of SCGAN, and

[10] represents the result of an embodiment of this application. Figure 6

[11] in the table represents the real images (high-quality images) corresponding to low-quality images in the datasets FFHQ and LS3D-W balanced. Table 2 shows the testing of the CycleGAN, LRGAN and SCGAN frameworks on different datasets. Ablation experiments were conducted on these network frameworks by adding prior extraction models to verify the effectiveness of the prior extraction models. Figure 8 This is a visual evaluation of the ablation experiment of the three-dimensional feature extraction model (Table 2).

[0158] Table 1

[0159] Table 2

[0160] From Table 1, Figure 6 , Figure 7 It can be seen that the embodiments of this application are significantly better than other algorithms in both objective evaluation indicators and visual quality assessment, and have a great advantage in face super-resolution reconstruction tasks.

[0161] exist Figure 8In Table 2, the three sets of images from left to right (4 columns per set) represent CycleGAN, LRGAN, and SCGAN, respectively. Each set of images has 4 columns, corresponding to the four datasets. Each column of images has 3 rows, where row [1] represents low-quality images, row [2] represents the super-resolution results of the original unsupervised model, and row [3] represents the super-resolution results of the unsupervised model with the introduction of a 3D feature correction mechanism. Figure 8 As can be seen from the above, with the help of the three-dimensional feature correction mechanism in the embodiments of this application, the performance is greatly improved in both objective evaluation indicators and visual quality assessment compared with the original model. This fully demonstrates the universal effectiveness and advanced nature of the three-dimensional prior introduction mechanism in the unsupervised framework.

[0162] Based on the image super-resolution model training method provided in the above embodiments, this application also provides specific implementation methods of the image super-resolution model training apparatus. Please refer to the following embodiments.

[0163] See Figure 9 The image super-resolution model training device 400 provided in this application embodiment includes: The acquisition module 410 is used to acquire the first image and the second image; The input module 420 is used to input the first image and the second image into the first image super-resolution model. The first image super-resolution model includes a first initial image degradation sub-model, an initial image super-resolution sub-model, and a second initial image degradation sub-model. Processing module 430 is used to perform degradation processing on the first image using the first initial image degradation sub-model to obtain the third image, and to perform super-resolution processing on the third image using the initial image super-resolution sub-model to obtain the fourth image; The processing module 430 is also used to perform super-resolution processing on the second image using the initial image super-resolution sub-model to obtain the fifth image, and to perform degradation processing on the fifth image using the second initial image degradation sub-model to obtain the sixth image; The adjustment module 440 is used to adjust the parameters of the first image super-resolution model when the first loss value between the second image and the third image, the second loss value between the third image and the first image, the third loss value between the fourth image and the first image, the fourth loss value between the first image and the fifth image, the fifth loss value between the fifth image and the second image, and the sixth loss value between the sixth image and the second image do not meet the first preset training stopping condition, and then return to input the first image and the second image into the first image super-resolution model. The determination module 450 is used to obtain a second image super-resolution model when the first loss value, second loss value, third loss value, fourth loss value, fifth loss value and sixth loss value satisfy the first preset training stopping condition. The second image super-resolution model includes a first image degradation sub-model, a target image super-resolution sub-model and a second image degradation sub-model.

[0164] In some embodiments, the adjustment module 440 may specifically be used for: Obtain the first loss value between the second image and the third image, the second loss value between the third image and the first image, the third loss value between the fourth image and the first image, the fourth loss value between the first image and the fifth image, the fifth loss value between the fifth image and the second image, and the sixth loss value between the sixth image and the second image; Obtain the first weights corresponding to the first, second, third, fourth, fifth, and sixth loss values, respectively; Based on the first weight, the first loss value, the second loss value, the third loss value, the fourth loss value, the fifth loss value, and the sixth loss value are weighted and summed to obtain the first target loss value of the first image super-resolution model; If the first target loss value does not meet the first preset training stopping condition, adjust the parameters of the first image super-resolution model.

[0165] In some embodiments, the determining module 450 may specifically be used for: When the first target loss value meets the first preset training stopping condition, the second image super-resolution model is obtained.

[0166] In some embodiments, the adjustment module 440 may specifically be used for: The first loss value between the second and third images is calculated using the following adversarial loss calculation formula:

[0167] in, This represents the first loss value. This represents the second image. This represents the third image. and Let represent the probability distributions followed by the second and third images, respectively. This represents the score given by the discriminator corresponding to the first initial image degradation sub-model.

[0168] In some embodiments, the adjustment module 440 may specifically be used for: The second loss value between the third image and the first image is calculated using the following formula:

[0169] in, This represents the second loss value, and N represents the number of images input to the first image super-resolution model each time. Represents the first image. This represents the third image. This indicates bicubic interpolation upsampling.

[0170] In some embodiments, the third loss value includes the first sub-loss value and the second sub-loss value; the adjustment module 440 can specifically be used for: The third loss value between the fourth image and the first image is calculated using the following formula:

[0171] in, This represents the first sub-loss value. This represents the second sub-loss value. Represents the first image. This represents the fourth image. and Let represent the probability distributions followed by the first and fourth images, respectively. This represents the score given by the discriminator corresponding to the super-resolution sub-model of the initial image.

[0172] In some embodiments, the adjustment module 440 may specifically be used for: The fourth loss value between the first and fifth images is calculated using the following adversarial loss calculation formula:

[0173] in, This represents the fourth loss value. Represents the first image. This represents the fifth image. and Let represent the probability distributions followed by the first and fifth images, respectively. This represents the score given by the discriminator corresponding to the super-resolution sub-model of the initial image.

[0174] In some embodiments, the adjustment module 440 may specifically be used for: The fifth loss value between the fifth image and the second image is calculated using the following formula:

[0175] in, This represents the fifth loss value, and N represents the number of images input to the first image super-resolution model each time. This represents the second image. This represents the fifth image. This indicates bicubic interpolation downsampling.

[0176] In some embodiments, the sixth loss value includes a third sub-loss value and a fourth sub-loss value; the adjustment module 440 can specifically be used for: The sixth loss value between the sixth image and the second image is calculated using the following formula:

[0177] in, This represents the third sub-loss value. This represents the fourth sub-loss value. This represents the second image. This represents the sixth image. and Let represent the probability distributions followed by the second and sixth images, respectively. This represents the score given by the discriminator corresponding to the second initial image degradation sub-model.

[0178] In some embodiments, the device 400 may further include: The construction module is used to input the first image and the fourth image into the prior extraction model before adjusting the parameters of the first image super-resolution model when the first target loss value does not meet the first preset training stopping condition. Using the prior extraction model, the depth map, reflection map, global illumination direction and projection view of the first image, as well as the depth map, reflection map, global illumination direction and projection view of the fourth image are extracted. Based on the depth map, reflection map, global illumination direction and projection view of the first image, the corresponding three-dimensional image of the first image is constructed. Based on the depth map, reflection map, global illumination direction and projection view of the fourth image, the corresponding three-dimensional image of the fourth image is constructed. The projection module is used to project the three-dimensional image corresponding to the first image and the three-dimensional image corresponding to the fourth image onto a two-dimensional plane respectively, to obtain the first reconstructed image corresponding to the first image and the second reconstructed image corresponding to the fourth image. The acquisition module 410 is also used to acquire a seventh loss value between the first reconstructed image and the second reconstructed image; The acquisition module 410 is also used to acquire the second weight of the seventh loss value; The determination module 450 is also used to perform a weighted summation of the first target loss value and the seventh loss value according to the first weight and the second weight to obtain the second target loss value; The adjustment module 440 can be specifically used for: If the second target loss value does not meet the second preset training stopping condition, adjust the parameters of the first image super-resolution model; The prior extraction model is a model trained using image samples and their corresponding label images.

[0179] In some embodiments, the acquisition module 410 may be specifically used for: The seventh loss value between the first and second reconstructed images is calculated using the following formula:

[0180] in, This represents the seventh loss value, and N represents the number of images input to the first image super-resolution model each time. This represents the first reconstructed image. This represents the second reconstructed image.

[0181] Each module of the image super-resolution model training device provided in this application embodiment can realize the functions of each step of the image super-resolution model training method provided above, and can achieve its corresponding technical effects. For the sake of brevity, it will not be described in detail here.

[0182] Based on the same inventive concept, embodiments of this application also provide an electronic device.

[0183] Figure 10 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0184] An electronic device may include a processor 501 and a memory 502 storing computer program instructions.

[0185] Specifically, the processor 501 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0186] Memory 502 may include mass storage for data or instructions. For example, and not limitingly, memory 502 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 502 may include removable or non-removable (or fixed) media. Where appropriate, memory 502 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 502 is non-volatile solid-state memory.

[0187] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to one aspect of this disclosure.

[0188] The processor 501 reads and executes computer program instructions stored in the memory 502 to implement any of the image super-resolution model training methods in the above embodiments.

[0189] In one example, the electronic device may also include a communication interface 503 and a bus 510. Wherein, as... Figure 10 As shown, the processor 501, memory 502, and communication interface 503 are connected through bus 510 and complete communication with each other.

[0190] The communication interface 503 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0191] Bus 510 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Linear Predictive Coding (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (Peripheral Component Interconnect-X, PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VESA Local Bus, VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 510 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application contemplates any suitable bus or interconnection. The electronic device can execute the image super-resolution model training method of the embodiments of the present invention, thereby realizing the image super-resolution model training method described above.

[0192] Furthermore, in conjunction with the image super-resolution model training methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the image super-resolution model training methods in the above embodiments.

[0193] This application also provides a computer program product, wherein the instructions in the computer program product, when executed by a processor of an electronic device, cause the electronic device to perform various processes implementing any of the above-described embodiments of the training method for image super-resolution models.

[0194] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0195] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0196] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0197] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0198] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A training method for an image super-resolution model, characterized in that, include: Acquire the first image and the second image; The first image and the second image are input into a first image super-resolution model, which includes a first initial image degradation sub-model, an initial image super-resolution sub-model, and a second initial image degradation sub-model. The first image is degraded using the first initial image degradation sub-model to obtain a third image, and the third image is super-resolution using the first initial image super-resolution sub-model to obtain a fourth image. The second image is super-resolution processed using the initial image super-resolution sub-model to obtain the fifth image, and the fifth image is degraded using the second initial image degradation sub-model to obtain the sixth image. If the first loss value between the second image and the third image, the second loss value between the third image and the first image, the third loss value between the fourth image and the first image, the fourth loss value between the first image and the fifth image, the fifth loss value between the fifth image and the second image, and the sixth loss value between the sixth image and the second image do not meet the first preset training stopping condition, adjust the parameters of the first image super-resolution model, and return to inputting the first image and the second image into the first image super-resolution model; When the first loss value, the second loss value, the third loss value, the fourth loss value, the fifth loss value, and the sixth loss value satisfy the first preset training stopping condition, a second image super-resolution model is obtained. The second image super-resolution model includes a first image degradation sub-model, a target image super-resolution sub-model, and a second image degradation sub-model.

2. The method according to claim 1, characterized in that, When the first loss value between the second image and the third image, the second loss value between the third image and the first image, the third loss value between the fourth image and the first image, the fourth loss value between the first image and the fifth image, the fifth loss value between the fifth image and the second image, and the sixth loss value between the sixth image and the second image do not meet the first preset training stopping condition, adjusting the parameters of the first image super-resolution model includes: Obtain a first loss value between the second image and the third image, a second loss value between the third image and the first image, a third loss value between the fourth image and the first image, a fourth loss value between the first image and the fifth image, a fifth loss value between the fifth image and the second image, and a sixth loss value between the sixth image and the second image; Obtain the first weights corresponding to the first loss value, the second loss value, the third loss value, the fourth loss value, the fifth loss value, and the sixth loss value, respectively; Based on the first weight, the first loss value, the second loss value, the third loss value, the fourth loss value, the fifth loss value, and the sixth loss value are weighted and summed to obtain the first target loss value of the first image super-resolution model; If the first target loss value does not meet the first preset training stopping condition, the parameters of the first image super-resolution model are adjusted.

3. The method according to claim 2, characterized in that, The step of obtaining a second image super-resolution model when the first loss value, the second loss value, the third loss value, the fourth loss value, the fifth loss value, and the sixth loss value satisfy the first preset training stopping condition includes: When the first target loss value satisfies the first preset training stopping condition, a second image super-resolution model is obtained.

4. The method according to claim 2, characterized in that, Obtaining a first loss value between the second image and the third image includes: The first loss value between the second image and the third image is calculated using the following adversarial loss calculation formula: ， in, This represents the first loss value. This represents the second image. This represents the third image. and Let represent the probability distributions followed by the second and third images, respectively. This represents the score given by the discriminator corresponding to the first initial image degradation sub-model.

5. The method according to claim 2, characterized in that, Obtaining the second loss value between the third image and the first image includes: The second loss value between the third image and the first image is calculated using the following formula: ， in, This represents the second loss value, and N represents the number of images input to the first image super-resolution model each time. Represents the first image. This represents the third image. This indicates bicubic interpolation upsampling.

6. The method according to claim 2, characterized in that, The third loss value includes the first sub-loss value and the second sub-loss value; the third loss value between the fourth image and the first image is obtained. include: The third loss value between the fourth image and the first image is calculated using the following formula: ， in, This represents the first sub-loss value. This represents the second sub-loss value. Represents the first image. This represents the fourth image. and Let represent the probability distributions followed by the first and fourth images, respectively. This represents the score given by the discriminator corresponding to the super-resolution sub-model of the initial image.

7. The method according to claim 2, characterized in that, Obtaining the fourth loss value between the first image and the fifth image includes: The fourth loss value between the first image and the fifth image is calculated using the following adversarial loss calculation formula: ， in, This represents the fourth loss value. Represents the first image. This represents the fifth image. and Let represent the probability distributions followed by the first and fifth images, respectively. This represents the score given by the discriminator corresponding to the super-resolution sub-model of the initial image.

8. The method according to claim 2, characterized in that, Obtaining the fifth loss value between the fifth image and the second image includes: The fifth loss value between the fifth image and the second image is calculated using the following formula: ， in, This represents the fifth loss value, and N represents the number of images input to the first image super-resolution model each time. This represents the second image. This represents the fifth image. This indicates bicubic interpolation downsampling.

9. The method according to claim 2, characterized in that, The sixth loss value includes the third sub-loss value and the fourth sub-loss value; the sixth loss value between the sixth image and the second image is obtained. include: The sixth loss value between the sixth image and the second image is calculated using the following formula: ， in, This represents the third sub-loss value. This represents the fourth sub-loss value. This represents the second image. This represents the sixth image. and Let represent the probability distributions followed by the second and sixth images, respectively. This represents the score given by the discriminator corresponding to the second initial image degradation sub-model.

10. The method according to claim 2, characterized in that, Before adjusting the parameters of the first image super-resolution model when the first target loss value does not meet the first preset training stopping condition, the method further includes: The first image and the fourth image are input into a priori extraction model. Using the priori extraction model, the depth map, reflection map, global illumination direction, and projection view of the first image, as well as the depth map, reflection map, global illumination direction, and projection view of the fourth image, are extracted. Based on the depth map, reflection map, global illumination direction, and projection view of the first image, a three-dimensional image corresponding to the first image is constructed. Based on the depth map, reflection map, global illumination direction, and projection view of the fourth image, a three-dimensional image corresponding to the fourth image is constructed. The three-dimensional image corresponding to the first image and the three-dimensional image corresponding to the fourth image are projected onto a two-dimensional plane to obtain the first reconstructed image corresponding to the first image and the second reconstructed image corresponding to the fourth image. Obtain the seventh loss value between the first reconstructed image and the second reconstructed image; Obtain the second weight of the seventh loss value; Based on the first weight and the second weight, the first target loss value and the seventh loss value are weighted and summed to obtain the second target loss value; The step of adjusting the parameters of the first image super-resolution model when the first target loss value does not meet the first preset training stopping condition includes: If the second target loss value does not meet the second preset training stopping condition, adjust the parameters of the first image super-resolution model; The prior extraction model is a model trained using image samples and the corresponding label images of the image samples.

11. The method according to claim 10, characterized in that, The step of obtaining the seventh loss value between the first reconstructed image and the second reconstructed image includes: The seventh loss value between the first reconstructed image and the second reconstructed image is calculated using the following formula: ， in, This represents the seventh loss value, and N represents the number of images input to the first image super-resolution model each time. This represents the first reconstructed image. This represents the second reconstructed image.

12. A training device for an image super-resolution model, characterized in that, include: The acquisition module is used to acquire the first image and the second image; The input module is used to input the first image and the second image into a first image super-resolution model, wherein the first image super-resolution model includes a first initial image degradation sub-model, an initial image super-resolution sub-model, and a second initial image degradation sub-model. The processing module is used to perform degradation processing on the first image using the first initial image degradation sub-model to obtain a third image, and to perform super-resolution processing on the third image using the initial image super-resolution sub-model to obtain a fourth image; The processing module is further configured to perform super-resolution processing on the second image using the initial image super-resolution sub-model to obtain the fifth image, and to perform degradation processing on the fifth image using the second initial image degradation sub-model to obtain the sixth image; The adjustment module is used to adjust the parameters of the first image super-resolution model when the first loss value between the second image and the third image, the second loss value between the third image and the first image, the third loss value between the fourth image and the first image, the fourth loss value between the first image and the fifth image, the fifth loss value between the fifth image and the second image, and the sixth loss value between the sixth image and the second image do not meet the first preset training stopping condition, and then return to input the first image and the second image into the first image super-resolution model; The determination module is used to obtain a second image super-resolution model when the first loss value, the second loss value, the third loss value, the fourth loss value, the fifth loss value, and the sixth loss value satisfy the first preset training stopping condition. The second image super-resolution model includes a first image degradation sub-model, a target image super-resolution sub-model, and a second image degradation sub-model.

13. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the training method for the image super-resolution model as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the training method for the image super-resolution model as described in any one of claims 1-11.

15. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device is able to perform the training method of the image super-resolution model as described in any one of claims 1-11.

Images