A semantic-aware image super-resolution reconstruction method

Abstract

The present invention discloses a semantically co-aware image super-resolution reconstruction method, belonging to the technical field of image super-resolution reconstruction. The method comprises: constructing an original high-resolution image dataset and a low-resolution image dataset; constructing an image super-resolution model; calculating a loss function of the image super-resolution model; training the image super-resolution model using the original high-resolution image dataset and the low-resolution image dataset; inputting the low-resolution image and its corresponding text information into the trained image super-resolution model, and outputting an image super-resolution reconstruction result. The present invention introduces text information and uses a conditional generative adversarial network to perform super-resolution reconstruction tasks. Specifically, the present invention introduces text information into a discriminator to assist in the image super-resolution reconstruction task, and adds a new input layer to the discriminator to support conditional input. The method achieves a balance between visual quality and objective quality, and unleashes the potential of text information.

Description

Technical Field

[0001] The present invention belongs to the technical field of image super-resolution reconstruction, and in particular relates to a semantic joint perception image super-resolution reconstruction method. Background Art

[0002] Image super-resolution (ISR) is a technical means that aims to reconstruct low-resolution images that have been degraded due to imaging device performance limitations or during transmission, processing, storage, etc. into high-resolution images with rich details and higher clarity, making their quality as close as possible to real high-resolution images.

[0003] Image super-resolution reconstruction is essentially an ill-posed problem. Specifically, given a low-resolution image block, due to the loss of some information during the degradation process, the low-resolution image may correspond to multiple different high-resolution image blocks. From a mathematical perspective, the low-resolution image can be and high-resolution images The relationship is simplified to: ,in is the downsampling operator. The pathological nature of this relationship is that, given a low-resolution image Solving high-resolution images When , this solution is not unique, and the tiny low-resolution image Changes in (such as noise interference) may result in the restored high-resolution image There are significant changes. To address this pathological problem, it is usually necessary to introduce prior knowledge. Deep learning-based image super-resolution methods use a large amount of training data to enable the network to learn the intrinsic prior information of the image. In this way, when given a low-resolution image, the network can use this prior knowledge to predict a reasonable high-resolution image that conforms to the statistical characteristics of natural images as much as possible, rather than randomly selecting from many possible solutions. For example, the Super-Resolution Generative Adversarial Network (SRGAN) uses an adversarial training mechanism between the generator and the discriminator to allow the generator to learn how to generate high-resolution images that are more realistic and more consistent with the distribution of natural images. This adversarial training process not only improves the visual quality of the image, but also effectively alleviates the pathological nature of the super-resolution task, making the reconstructed high-resolution image closer to the real image in terms of details and overall quality.

[0004] The main goals of image super-resolution reconstruction are to improve both visual and objective image quality. Visual quality focuses on the subjective perception of the reconstructed image, namely, whether the image appears natural, sharp, and has realistic details. Objective evaluation relies on quantitative metrics, such as peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and structural similarity (SSIM), to measure the difference in quality between the reconstructed image and the true high-resolution image. However, in practical applications, these two evaluation methods can conflict. For example, some reconstructed images, while performing well in objective metrics, may be visually inferior to images with slightly lower objective metrics. This phenomenon, known as the "higher but not necessarily better" resolution problem, demonstrates that relying solely on objective metrics cannot fully reflect the visual quality of an image. Therefore, image super-resolution reconstruction requires striking a balance between these two aspects. By designing a suitable loss function to guide model training, model training can optimize both objective metrics and visual quality. For example, SRGAN introduces a perceptual loss to optimize image quality by mimicking the perceptual properties of the human visual system. ESRGAN further improves on this, attempting to generate super-resolution results that are more visually close to the true image. However, this improvement also introduces new challenges, such as oversharpening and artifacts, which can compromise the naturalness and visual quality of images. To address these issues, Real-ESRGAN extends ESRGAN, attempting to improve visual quality while reducing negative effects like artifacts and oversharpening. However, existing models are mostly based on traditional single-modality image super-resolution (SR) frameworks, optimized with generative adversarial networks (GANs). These methods still cannot fully resolve these challenges, indicating that further exploration and improvement are needed in the field of image super-resolution reconstruction.

[0005] The text-to-image (T2I) diffusion model has developed rapidly. It generates high-quality images based on text information and involves natural language processing and computer vision. Seesr uses the T2I model for image super-resolution tasks and introduces image information as a condition into the T2I model, which is the Controlled T2I diffusion model. The use of pre-trained text-to-image (T2I) models to perform image super-resolution tasks is currently receiving more and more attention. However, fundamentally, the output results of this model depend to a large extent on the accuracy of the text input. It can be said that the correctness of the text information directly determines the correctness of the output results. Once the input text information is incorrect, it will have a catastrophic effect on the super-resolution reconstruction results of the target low-resolution image. For example, in Figure 1In the example above, the model mistakenly super-reconstructed a “person driving a boat” into a “bird”. This clearly shows that when text-driven image generation models are applied to image super-resolution tasks, the results may be completely different from the original high-resolution image. Summary of the Invention

[0006] The purpose of the present invention is to address the above-mentioned deficiencies in the prior art and provide a semantically co-aware image super-resolution reconstruction method to solve the problems of over-sharpening and artifacts in existing image super-resolution reconstruction, as well as the problem that the results of text-driven super-resolution reconstruction image generation task models may be completely different from the original high-resolution image.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] A semantic joint perception image super-resolution reconstruction method comprises the following steps:

[0009] S1. Build original high-resolution image dataset;

[0010] S2, downsample the original high-resolution image to construct a low-resolution image dataset;

[0011] S3. Build an image super-resolution model based on conditional generative adversarial network and text information;

[0012] S4. Calculate the loss function of the image super-resolution model;

[0013] S5. Use the original high-resolution image dataset and the low-resolution image dataset to train an image super-resolution model;

[0014] S6. Input the low-resolution image and its corresponding text information into the trained image super-resolution model, and output the image super-resolution reconstruction result.

[0015] Furthermore, in S2, the image super-resolution model includes:

[0016] A generator that takes a low-resolution image as input and outputs a reconstructed high-resolution image;

[0017] The discriminator inputs the reconstructed high-resolution image and its corresponding text information, the original high-resolution image and its corresponding text information into the discriminator for semantic alignment judgment, and transmits the semantic alignment judgment result to the generator for adversarial training.

[0018] Furthermore, the low-resolution image is input into the generator, and the reconstructed high-resolution image is output, which specifically includes:

[0019] The input low-resolution image is processed through multi-layer convolution operations and dynamic residual processing to generate the core features of the image;

[0020] At the same time, the input low-resolution image is upsampled to the target resolution through bilinear interpolation to obtain a benchmark high-resolution image;

[0021]

[0022] Merge the core features of the image with the baseline high-resolution image to generate a reconstructed high-resolution image:

[0023]

[0024] Where, is the reconstructed high-resolution image; is the core feature of the image; is the benchmark high-resolution image; represents bilinear interpolation; is the input low-resolution image.

[0025] Furthermore, the input low-resolution image is subjected to multi-layer convolution operations and dynamic residual processing to generate the core features of the image, including:

[0026] Input the low-resolution image into the first convolution layer, and dynamically convolute the low-resolution image into the feature space:

[0027]

[0028] Where, is the first image feature; is the weight; is the activation function; It is a dynamic convolution operation;

[0029] The first image feature is input into the residual block for dynamic residual processing:

[0030]

[0031] Where, is the second image feature; represents dynamic residual processing;

[0032] The second image feature is input into the second convolution layer for pixel rearrangement and dynamic convolution processing:

[0033]

[0034] Where, is the third image feature; Indicates pixel rearrangement;

[0035] The third image feature is input into the third convolution layer, dynamic convolution is performed and the channel dimension is adjusted:

[0036]

[0037] Where, is the fourth image feature;

[0038] Input the fourth image feature into the fourth convolution layer and map the fourth image feature to the output channel:

[0039]

[0040] Where, is the core feature of the image.

[0041] Furthermore, the reconstructed high-resolution image and its corresponding text information are input into the discriminator for semantic alignment, which specifically includes:

[0042] Project the semantic features of the text information corresponding to the reconstructed high-resolution image to the target shape:

[0043]

[0044] Where s is the semantic feature of text information; , , They are the semantic features of text information introduced in different layers of convolution; To convert the semantic features of the input text information from its original representation into a multi-dimensional tensor suitable for network processing;

[0045] The reconstructed high-resolution image Input into the first stage convolutional layer, perform feature conversion and combine with the attention mechanism:

[0046]

[0047]

[0048] Where, is the fifth image feature; is the first splicing feature; is the activation function; is the convolution operation; For attention output;

[0049] The first concatenated features are input into the second stage convolutional layer to extract image features and combine them with the attention mechanism:

[0050]

[0051]

[0052] Where, is the sixth image feature; is the second splicing feature;

[0053] The second concatenated features are input into the third stage convolutional layer to extract image features and combine them with the attention mechanism:

[0054]

[0055]

[0056] Where, is the seventh image feature; is the third splicing feature;

[0057] Project the third concatenated feature to a single-channel output through the final convolutional layer:

[0058]

[0059] Where, The final output of the discriminator represents the reconstructed high-resolution image of the input prediction results.

[0060] Furthermore, in the discriminator, an attention mechanism is introduced in the first, second, and third convolutional layers to embed the semantic features of text information into image features.

[0061] Among them, the attention mechanism in the first stage convolutional layer is expressed as:

[0062] Normalize the semantic features of the input fifth image feature and text information:

[0063]

[0064]

[0065] Where, is the fifth image feature after normalization; is the semantic feature of the text information after normalization in the first stage; Indicates standardized processing;

[0066] Use 1×1 convolution to map the normalized fifth image features and the semantic features of the normalized text information in the first stage to the embedding dimension:

[0067]

[0068] Where, is the query matrix, is the bond matrix, is the value matrix;

[0069] Calculate the attention weight matrix:

[0070]

[0071] Where, is the attention weight matrix; is the dimension of the key; is the transpose of the matrix; is the normalization operation;

[0072] Add the attention weight matrix to the value matrix , generating attention output .

[0073] Furthermore, in S4, the loss function of the image super-resolution model includes:

[0074] Generator loss function for:

[0075]

[0076] Where, is the pixel loss, For perception loss, For style loss, is the GAN loss;

[0077] Loss function of the discriminator for:

[0078]

[0079] Where, is the original high-resolution image loss, is the loss of the reconstructed high-resolution image.

[0080] Furthermore, the perceptual loss for:

[0081]

[0082]

[0083] Where, for Loss criterion The perceptual loss of the layer, is the mean absolute error; For the The number of elements in the layer feature map; i is the summation index, used to traverse from 1 to The value of and The generated image and the real image are Feature representation of the layer; is the original high-resolution image; is the reconstructed high-resolution image; For the The weight corresponding to the layer, is the overall weight of the perceptual loss;

[0084] Style Loss for:

[0085]

[0086]

[0087]

[0088] Where, For the Layer feature map The calculated Gram matrix; For the Feature maps of layers; is the number of channels, and are the height and width of the feature map respectively; Represents the activation value of each channel at each spatial position in the deep feature map; for Loss criterion Style loss of the layer; For the The number of elements in the layer Gram matrix; and Reconstructed high-resolution images and the original high-resolution image In the Gram matrix of layer feature representation; is the overall weight of style loss;

[0089] GAN loss for:

[0090]

[0091] Where, is the final output of the discriminator, which is the discriminator's judgment result on the reconstructed high-resolution image.

[0092] Furthermore, the original high-resolution image loss for:

[0093]

[0094] Reconstructed high-resolution image loss for:

[0095]

[0096] Where, is the final output of the discriminator, is the discriminator’s judgment result on the original high-resolution image, It is the discriminator's judgment result on the reconstructed high-resolution image.

[0097] The semantic joint perception image super-resolution reconstruction method provided by the present invention has the following beneficial effects:

[0098] The present invention introduces text information and uses a conditional generative adversarial network to perform super-resolution reconstruction tasks. Specifically, text information is introduced into the discriminator to assist in the image super-resolution reconstruction task, and a new input layer is added to the discriminator to support conditional input. This achieves a balance between visual quality and objective quality, and unleashes the potential of text information. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Figure 1 is the reconstruction result of the SISR model based on the T2I model used in the background technology, where: Figure 1 (a) is the result image of image super-resolution reconstruction using the SISR model based on the T2I model; Figure 1 (b) in the figure is the original high-resolution image.

[0100] Figure 2 This is a network result diagram of the generator in an embodiment of the present invention.

[0101] Figure 3 This is a diagram of the network results of the discriminator in an embodiment of the present invention.

[0102] Figure 4 This is a flowchart of the combination of convolution and attention mechanisms in an embodiment of the present invention.

[0103] Figure 5 Flowchart of the semantic joint perception image super-resolution reconstruction method in an embodiment of the present invention. DETAILED DESCRIPTION

[0104] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0105] Example 1

[0106] The semantic joint perception image super-resolution reconstruction method of this embodiment effectively utilizes text information and can well balance the image visual quality and objective quality. Figure 5 , which specifically includes the following:

[0107] S1. Build original high-resolution image dataset;

[0108] This example collects a large number of high-resolution (HR) images from public datasets. The training datasets include CelebA-HQ, coco, DIV2K, and Flickr2k; the test datasets include Set5, Set14, BSD100, and Urban100. Real low-resolution images are also collected to test the performance of the model.

[0109] S2, downsample the original high-resolution image to construct a low-resolution image dataset;

[0110] This embodiment specifically performs bilinear interpolation downsampling on the original high-resolution image to generate a corresponding low-resolution image;

[0111] The original high-resolution images and low-resolution images are cropped, large images are cropped into small blocks for easy training, and the images are normalized to speed up the training process.

[0112] S3. Build an image super-resolution model based on conditional generative adversarial network and text information;

[0113] The image super-resolution model in this embodiment includes:

[0114] Generator, which inputs the low-resolution image into the generator, undergoes multi-layer convolution operations and dynamic residual processing, and outputs a reconstructed high-resolution image;

[0115] Specifically, refer to Figure 2 The generator is a convolutional neural network architecture. The input is a low-resolution image. After dynamic convolution, residual processing, upsampling, and interpolation upsampling, the output is a reconstructed high-resolution image. The specific process is as follows:

[0116] Input the low-resolution image into the first convolution layer, and dynamically convolute the low-resolution image into the feature space:

[0117]

[0118] Where, The first image feature contains the initially extracted low-level features of the image. It is the first feature representation of the network and is used for subsequent residual block processing. is the weight; is the activation function; It is a dynamic convolution operation; is the input low-resolution image;

[0119] The first image feature is input into the residual block for dynamic residual processing, and the process is repeated multiple times:

[0120]

[0121] Where, The second image feature contains richer high-level feature information. It is the result of stacking multiple residual blocks and is used in the main part of the network. represents dynamic residual processing;

[0122] Next, upsampling is performed, and the second image features are input into the second layer of convolution for pixel rearrangement and dynamic convolution processing:

[0123]

[0124] Where, The third image feature is a high-resolution image feature after pixel rearrangement and dynamic convolution. It represents a higher-resolution feature map. It has been upgraded from a lower resolution to a higher resolution, but still requires further convolution processing to enhance its feature information. Rearrange pixels;

[0125] The third image feature is input into the third convolution layer, dynamic convolution processing is performed and the channel dimension is adjusted. The LeakyReLU activation function is used to perform nonlinear processing on the output of dynamic convolution to improve the feature representation capability:

[0126]

[0127] Where, The fourth image feature contains a feature map with higher resolution and rich features, which provides input for the subsequent final convolution layer and output image generation;

[0128] Input the fourth image feature into the fourth convolution layer and map the fourth image feature to the output channel:

[0129]

[0130] Where, is the core feature of the image.

[0131] At the same time, the input low-resolution image is upsampled to the target resolution through bilinear interpolation to obtain a benchmark high-resolution image;

[0132]

[0133] Merge the core features of the image with the baseline high-resolution image to generate a reconstructed high-resolution image:

[0134]

[0135] Where, The reconstructed high-resolution image is the final output of the generator network. This image contains both the high-frequency features generated by the network and the structural information of the low-resolution input. is the core feature of the image; is the benchmark high-resolution image; It is a bilinear interpolation operation.

[0136] The discriminator inputs the reconstructed high-resolution image and its corresponding text information, and the original high-resolution image and its corresponding text information into the discriminator for semantic alignment judgment, and transmits the semantic alignment judgment result to the generator for adversarial training;

[0137] refer to Figure 3 The discriminator is a hybrid network architecture that combines a convolutional neural network and a Transformer. The semantic feature extractor effectively extracts the semantic features of text information. After the first convolution layer, the image is activated using the LeakyReLU function. It then passes through three layers of convolution. After each layer, the modified spatial transformer (MST) fuses the text information with the image features. The fusion is further processed through convolutional layers, and the final convolution layer converts the feature map into the original channel output, which serves as the model's final discriminant result.

[0138] In the discriminator structure designed in this embodiment, the text information and image information are fused by introducing the attention mechanism in three consecutive convolutional layers. The core of the design lies in feature extraction and semantic enhancement. By extracting low-level to high-level features layer by layer to increase their complexity, the semantic context is dynamically introduced through the attention mechanism to improve the understanding of image details until the discriminator combines local details with global information to generate the final prediction result. The main input of the hybrid network architecture is image information, which is shaped by the number of input samples, width, height, and number of channels. Text information is an auxiliary input and contains semantic information closely related to the image.

[0139] Since the processes of the reconstructed high-resolution image and its corresponding text information and the original high-resolution image and its corresponding text information as input are the same, this embodiment is described using the reconstructed high-resolution image and its corresponding text information as input:

[0140] Project the semantic features of the text information corresponding to the reconstructed high-resolution image to the target shape:

[0141]

[0142] Where s is the semantic feature of text information; , , They are the semantic features of text information introduced in different layers of convolution; The role of the input text information semantic features is to convert them from their original representation into a multi-dimensional tensor that can adapt to network processing;

[0143] The reconstructed high-resolution image Input into the first stage convolutional layer, perform feature conversion and combine with the attention mechanism:

[0144]

[0145]

[0146] Where, is the fifth image feature; is the first splicing feature; is the activation function; is the convolution operation; For attention output;

[0147] Here, the LeakyReLU activation function can handle negative features and avoid the problem of gradient disappearance; the semantic features of text information and the fifth image feature Combined with the attention mechanism, contextual information is added to the features. The attention mechanism can adaptively select useful semantic features, enhance the model's understanding of details, make the feature vector richer, and post-convolution compress the feature channel to enhance the nonlinear fitting ability.

[0148] The first concatenated features are input into the second stage convolutional layer to extract image features and combine them with the attention mechanism:

[0149]

[0150]

[0151] Where, is the sixth image feature; is the second splicing feature;

[0152] Sixth image feature Introducing the second stage semantic features , which can improve the ability to model complex semantic relationships. After embedding semantic information, the richness of convolution features is ensured, and post-convolution maintains channel consistency while adding nonlinear transformations.

[0153] The second concatenated features are input into the third stage convolutional layer to extract image features and combine them with the attention mechanism:

[0154]

[0155]

[0156] Where, is the seventh image feature; is the third splicing feature;

[0157] It is the seventh image feature, and the third stage semantic feature is introduced into it , establishing the deepest level of contextual relationships, enabling the model to better understand the overall image semantics.

[0158] Project the third concatenated feature to a single-channel output through the final convolutional layer:

[0159]

[0160] Where, The final output of the discriminator represents the reconstructed high-resolution image of the input The prediction result is whether the image input to the discriminator is a high-resolution image reconstructed by the generator or the original high-resolution image.

[0161] As a preferred embodiment of this invention, in the discriminator, an attention mechanism is introduced into the first-stage convolutional layer, the second-stage convolutional layer, and the third-stage convolutional layer, thereby embedding the semantic features of the text information into the image features;

[0162] Taking the first stage convolution as an example, refer to Figure 4 , the attention mechanism in the first stage convolutional layer is expressed as:

[0163] Normalize the semantic features of the input fifth image feature and text information:

[0164]

[0165]

[0166] Where, is the fifth image feature after normalization; is the semantic feature of the text information after normalization in the first stage; Indicates standardized processing;

[0167] Use 1×1 convolution to map the normalized fifth image features and the semantic features of the normalized text information in the first stage to the embedding dimension:

[0168]

[0169] Where, is the query matrix, is the bond matrix, is the value matrix;

[0170] Calculate the attention weight matrix:

[0171]

[0172] Where, is the attention weight matrix; is the dimension of the key; is the transpose of the matrix; for;

[0173] Add the attention weight matrix to the value matrix , generating attention output

[0174] S4. Calculate the loss function of the image super-resolution model;

[0175] The loss function of the image super-resolution model of this embodiment includes:

[0176] Generator loss function for:

[0177]

[0178] Where, is the pixel loss, For perception loss, For style loss, is the GAN loss;

[0179] Among them, pixel loss This is used to measure the pixel-level difference between a generated image and a real image. The human eye can clearly perceive many subtle details in an image. For example, in an image of a person's face, what is their skin condition? Is their skin tone even? Are there freckles or spots? Is their facial expression smiling, crying, or angry? Are their hair strands clearly distinct?

[0180] Perceptual loss This approach attempts to simulate how the human visual system perceives images. Generally speaking, it measures things like distinct colors, sharp edges, and realistic texture details. Perceptual loss measures the perceptual differences between generated and real images. It extracts features at different levels from the generated and real images using a pre-trained VGG network. The differences between these features are then compared to reflect the degree of perceptual similarity between the generated and real images.

[0181] Perceptual loss The calculation is:

[0182] First, calculate Loss criterion Perceptual loss of the layer:

[0183]

[0184] The total perceptual loss is:

[0185]

[0186] Where, for Loss criterion The perceptual loss of the layer, It is L1 loss, also known as mean absolute error; For the The number of elements in the layer feature map; i is the summation index, which is used to traverse from 1 to The value of and The generated image and the real image are Feature representation of the layer; is the original high-resolution image; is the reconstructed high-resolution image; For the The weight corresponding to the layer, is the overall weight of the perceptual loss; Represents a specific layer (a certain layer) in the VGG network.

[0187] Style Loss The calculation is:

[0188] Style loss is used to measure the style difference between the generated image and the real image. The Gram matrix of the generated image and the real image is calculated and compared. The Gram matrix is ​​used to capture the correlation between different channels in the feature map. Its calculation formula is as follows:

[0189]

[0190] Loss criterion The style loss of the layer is:

[0191]

[0192] The total style loss is:

[0193]

[0194] Where, For the Layer feature map The calculated Gram matrix; For the Feature maps of layers; is the number of channels, and are the height and width of the feature map respectively; It represents the activation value of each channel at each spatial position in the deep feature map. It is obtained by abstracting the input image through the convolutional network. It reflects the existence and strength of local patterns (such as edges, textures, etc.) in the image. It is a mapping from raw pixels to high-level semantic information. The Gram matrix can be used to obtain the correlation statistics between the various features, which is the key indicator for capturing the image style. The feature map is flattened in the spatial dimension to obtain , is the batch size; for Loss criterion Style loss of the layer; For the The number of elements in the layer Gram matrix; and Reconstructed high-resolution images and the original high-resolution image In the Gram matrix of layer feature representation; is the overall weight of style loss;

[0195] GAN loss for:

[0196]

[0197] Where, is the final output of the discriminator, and is the reconstructed high-resolution image The result of the discrimination is a probability value. The goal of the generator is to maximize this probability value, even if the discriminator cannot correctly identify whether the input image is a reconstructed high-resolution image or an original high-resolution image.

[0198] Loss function of the discriminator for:

[0199]

[0200] Where, is the original high-resolution image loss, is the loss of the reconstructed high-resolution image.

[0201] Among them, the original high-resolution image loss for:

[0202]

[0203] Here is the discriminator's judgment result on the original high-resolution image, which is a probability value. The discriminator's goal is to maximize this probability value, that is, to correctly identify the input as the original high-resolution image.

[0204] Reconstructed high-resolution image loss for:

[0205]

[0206] Where, is the discriminator's judgment result on the reconstructed high-resolution image, which is also a probability value. The discriminator's goal is to minimize this probability value, that is, to correctly identify the input as a reconstructed high-resolution image.

[0207] S5. Use the original high-resolution image dataset and the low-resolution image dataset to train an image super-resolution model. The training process includes:

[0208] Load the pre-trained model, including the pre-trained weights of the generator and discriminator. Initialize the training settings, including the loss function, optimizer, etc.

[0209] Data input: input low-resolution images, original high-resolution images, and corresponding text information into the model;

[0210] Train the generator, input a low-resolution image to the generator, output a reconstructed high-resolution image, and calculate and optimize the generator loss;

[0211] Train the discriminator, input the reconstructed high-resolution image and the original high-resolution image, as well as the corresponding text confidence to the discriminator, calculate the discriminator loss, and update the discriminator parameters;

[0212] The loop iterates and repeats the above training cycle until the stopping condition is met.

[0213] S6. Input the low-resolution image and its corresponding text information into the trained image super-resolution model, and output the image super-resolution reconstruction result.

[0214] This embodiment conducts an ablation experiment to verify the effect of the image super-resolution model of the present invention;

[0215] Among them, the baseline is set to be a super-resolution model similar to the traditional SISR model, which converts low-resolution images to high-resolution images. The ablation experiment results are shown in Table 1. It can be seen that the image super-resolution model proposed in this invention outperforms the baseline in both PSNR and SSIM. The introduction of text information in the baseline improves the quality of the super-resolution reconstruction results of the target image. At the same time, the impact of two different types of text input on the SISR task is considered to further prove that the introduction of text information improves the quality of the image super-resolution reconstruction results. For an image selected from the CelebA-HQ dataset, it contains two types of text description information.

[0216] The following two text messages describe the same subject matter. It's easy to see that the first description is more detailed and literal, making it easier for humans to understand. Both descriptions share the same theme: a woman, detailing her appearance, such as her hair, eyebrows, nose, and makeup. Keywords like "hair," "eyebrows," "nose," "lipstick," "heavy makeup," and "cosmetic" are mentioned.

[0217] The first type of text information is a natural language description, consisting of multiple short sentences that provide a detailed and specific description of a person's appearance. The information provided is more fragmented, with certain features repeated multiple times. However, it provides more detail and contains more implicit context, allowing the model to focus more on image details and overall features. For example, "busy eyebrows" and "arched eyebrows" refer to thick, arched eyebrows. The second type of text description is a comma-separated list of keywords. This is concise and focused, with high information density. Each keyword appears only once, yet it contains all the key information in the first type. This is closer to a machine-processable format, reducing redundant input, resulting in faster model training and inference speeds, but it lacks detailed descriptions.

[0218] The experimental results of two different text information ablation methods are shown in Table 2. It can be seen that the SSIM results are consistent, but there are differences in PSNR and LPIPS; the use of the first text information is slightly better in PSNR, indicating that its reconstruction accuracy at the pixel level is higher, but the difference is very small and almost negligible. The use of the second text information performs better in LPIPS, indicating that it is better in visual perception and is more suitable for scenes with high requirements for image details and texture. The SSIM results of the two text information are consistent, indicating that they all perform equally well in structural recovery. Finally, whether using the first or the second text information, PSNR, LPIPS, and SSIM are all better than the baseline that does not use text information, which fully proves that the introduction of text information does improve the quality of the super-resolution reconstruction results of the target image.

[0219] The first text message: "She is wearing lipstick. She is young, and smiling and has big lips, mouth slightly open, pointy nose, and high cheekbones."

[0220] The second text message: "beautiful, crown, hair, laugh, mouth, smile, tiara, wear, woman,"

[0221] Table 1 Ablation experiment results

[0222]

[0223] Table 2 Ablation experiment results (for two text descriptions, dataset: CelebA-HQ)

[0224]

[0225] Comparative experiments;

[0226] The proposed method is compared with several state-of-the-art SISR methods, including traditional image-to-image SISR methods, BSRGAN and Real-ESRGAN; and a SISR method that introduces other reference information (e.g., the text information proposed in this invention), seesr.

[0227] Table 3 shows the results of comparative experiments on several large datasets, all of which are commonly used in the super-resolution (SR) field. To further demonstrate the proposed method and enhance its credibility, comparative experiments were conducted on four public standard test datasets: Set5, Set14, BSD100, and Urban100. The results are shown in Table 4. The methods selected for comparative experiments are all state-of-the-art models in the field of super-resolution. Seesr is an image super-resolution method based on a diffusion model. It employs a controllable text-to-image diffusion model, using low-resolution images as additional conditional control information to achieve the goal of controlling the text to generate a specific target image, that is, generating a high-resolution image from a low-resolution image. As shown in Table 4, the quantitative comparison results on the Set5, Set14, BSD100, and Urban100 datasets show that the proposed method performs best in terms of PSNR, LPIPS, and SSIM. This means that the proposed method, which incorporates text information to assist in image super-resolution reconstruction, outperforms other methods in terms of pixel-level reconstruction accuracy, visual similarity, and structural similarity. From the data in Table 3 and Table 4, it can be seen that the method proposed in the present invention is superior to other methods.

[0228] Table 3 Comparative experimental results

[0229]

[0230] Table 4 Comparative experimental results of public test datasets

[0231]

[0232] Although the specific embodiments of the invention are described in detail in conjunction with the accompanying drawings, this should not be construed as limiting the scope of protection of this patent. Within the scope described by the claims, various modifications and variations that can be made by those skilled in the art without creative work still fall within the scope of protection of this patent.

Claims

1. A semantic joint perception image super-resolution reconstruction method, characterized by: The following steps are involved: S1. Build original high-resolution image dataset; S2, downsample the original high-resolution image to construct a low-resolution image dataset; S3. Build an image super-resolution model based on conditional generative adversarial network and text information; S4. Calculate the loss function of the image super-resolution model; S5. Use the original high-resolution image dataset and the low-resolution image dataset to train an image super-resolution model; S6. Input the low-resolution image and its corresponding text information into the trained image super-resolution model, and output the image super-resolution reconstruction result; In S3, the image super-resolution model includes: A generator that takes a low-resolution image as input and outputs a reconstructed high-resolution image; The discriminator inputs the reconstructed high-resolution image and its corresponding text information, and the original high-resolution image and its corresponding text information into the discriminator for semantic alignment judgment, and transmits the semantic alignment judgment result to the generator for adversarial training; The reconstructed high-resolution image and its corresponding text information are input into the discriminator for semantic alignment judgment, which includes: Project the semantic features of the text information corresponding to the reconstructed high-resolution image to the target shape: Where s is the semantic feature of text information; , , They are the semantic features of text information introduced in different layers of convolution; To convert the semantic features of the input text information from its original representation into a multi-dimensional tensor suitable for network processing; The reconstructed high-resolution image Input into the first stage convolutional layer, perform feature conversion and combine with the attention mechanism: Where, is the fifth image feature; is the first splicing feature; is the activation function; is the convolution operation; For attention output; The first concatenated features are input into the second stage convolutional layer to extract image features and combine them with the attention mechanism: Where, is the sixth image feature; is the second splicing feature; The second concatenated features are input into the third stage convolutional layer to extract image features and combine them with the attention mechanism: Where, is the seventh image feature; is the third splicing feature; Project the third concatenated feature to a single-channel output through the final convolutional layer: Where, The final output of the discriminator represents the reconstructed high-resolution image of the input prediction results.

2. The semantic joint perception image super-resolution reconstruction method according to claim 1, characterized in that: The low-resolution image is input into the generator and the reconstructed high-resolution image is output, which includes: The input low-resolution image is processed through multi-layer convolution operations and dynamic residual processing to generate the core features of the image; At the same time, the input low-resolution image is upsampled to the target resolution through bilinear interpolation to obtain a benchmark high-resolution image; Merge the core features of the image with the baseline high-resolution image to generate a reconstructed high-resolution image: Where, is the reconstructed high-resolution image; is the core feature of the image; is the benchmark high-resolution image; represents bilinear interpolation; is the input low-resolution image.

3. The semantic joint perception image super-resolution reconstruction method according to claim 2, characterized in that: The input low-resolution image is processed through multi-layer convolution operations and dynamic residual processing to generate the core features of the image, including: Input the low-resolution image into the first convolution layer, and dynamically convolute the low-resolution image into the feature space: Where, is the first image feature; is the weight; is the activation function; It is a dynamic convolution operation; The first image feature is input into the residual block for dynamic residual processing: Where, is the second image feature; represents dynamic residual processing; The second image feature is input into the second convolution layer for pixel rearrangement and dynamic convolution processing: Where, is the third image feature; Indicates pixel rearrangement; Input the third image feature into the third convolution layer, perform dynamic convolution processing and adjust the channel dimension: Where, is the fourth image feature; Input the fourth image feature into the fourth convolution layer and map the fourth image feature to the output channel: Where, is the core feature of the image.

4. The semantic joint perception image super-resolution reconstruction method according to claim 1, characterized in that: In the discriminator, an attention mechanism is introduced in the first-stage convolutional layer, the second-stage convolutional layer, and the third-stage convolutional layer, thereby embedding the semantic features of the text information into the image features; Among them, the attention mechanism in the first stage convolutional layer is expressed as: Normalize the semantic features of the input fifth image feature and text information: Where, is the fifth image feature after normalization; is the semantic feature of the text information after normalization in the first stage; Indicates standardized processing; Use 1×1 convolution to map the normalized fifth image features and the semantic features of the normalized text information in the first stage to the embedding dimension: Where, is the query matrix, is the bond matrix, is the value matrix; Calculate the attention weight matrix: Where, is the attention weight matrix; is the dimension of the key; is the transpose of the matrix; is the normalization operation; Add the attention weight matrix to the value matrix , generating attention output .

5. The semantic joint perception image super-resolution reconstruction method according to claim 1, characterized in that: In S4, the loss function of the image super-resolution model includes: Generator loss function for: Where, is the pixel loss, For perception loss, For style loss, is the GAN loss; Loss function of the discriminator for: Where, is the original high-resolution image loss, is the loss of the reconstructed high-resolution image.

6. The semantic joint perception image super-resolution reconstruction method according to claim 5, characterized in that: Perceptual loss for: Where, for Loss criterion The perceptual loss of the layer, is the mean absolute error; For the The number of elements in the layer feature map; i is the sum index, used to traverse from 1 to The value of and The generated image and the real image are Feature representation of the layer; is the original high-resolution image; is the reconstructed high-resolution image; For the The weight corresponding to the layer, is the overall weight of the perceptual loss; Style Loss for: Where, For the Layer feature map The calculated Gram matrix; For the Feature maps of layers; is the number of channels, and are the height and width of the feature map respectively; Represents the activation value of each channel at each spatial position in the deep feature map; for Loss criterion Style loss of the layer; For the The number of elements in the layer Gram matrix; and Reconstructed high-resolution images and the original high-resolution image In the Gram matrix of layer feature representation; is the overall weight of style loss; GAN loss for: Where, is the final output of the discriminator, which is the discriminator's judgment result on the reconstructed high-resolution image.

7. The semantic joint perception image super-resolution reconstruction method according to claim 5, characterized in that: Original high-resolution image loss for: Reconstructed high-resolution image loss for: Where, is the final output of the discriminator, is the discriminator’s judgment result on the original high-resolution image, It is the discriminator's judgment result on the reconstructed high-resolution image.

Images