Transfomer-based image super-resolution model construction method and image super-resolution method

Abstract

The invention discloses an image super-resolution model construction method based on Transfomer, and the method comprises the steps: obtaining a low-resolution image and a reference image from a public database, carrying out the sampling of the reference image, and constructing a training data set through the low-resolution image, the reference image and the sampled reference image; constructing an initial learning framework comprising a texture feature extraction layer, a relative position coding layer, a correlation embedding layer and a parallel circulation embedding layer, taking the low-resolution image, the reference image and the sampled reference image as the input of the initial learning framework, and outputting a high-resolution image; and training the initial learning framework according to the training data set, and designing a loss function to correct model parameters to obtain a trained image super-resolution model. According to the method, the extraction capability of detail textures is improved, the local distortion degree is reduced, the relation between low resolution and high resolution is better established, and the recovery effect and generalization capability of the model are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and specifically to a Transformer-based image super-resolution model construction method and an image super-resolution method. Background Art

[0002] Image restoration tasks (such as super-resolution, deblurring, etc.) are important tasks in the field of computer vision, which aim to restore the missing parts from damaged or missing image data through algorithmic means, so as to predict images with higher quality and resolution. Image restoration has a wide range of applications and research significance in many fields, such as image compression, video perception processing, medicine and remote sensing imaging, etc.

[0003] In image super-resolution methods, how to reduce the distortion of image texture details has always been a major problem. Whether using generative adversarial networks (SRGAN), residual networks (ResNet), recursive neural networks (RNN), or even Transformer models, etc., their essence is based on a certain model structure and a large number of low-resolution and high-resolution image samples, and then obtain the image processing model through large-capacity sample training. However, these methods cannot handle the distortion of image texture details well.

[0004] To address this problem, some other methods have emerged, such as the super-resolution method based on reference images. The basic idea is still based on the original training framework, on which a texture image (high resolution) similar to the low-resolution image texture to be super-resolved is added, so that the detail texture does not need to rely entirely on unstable training, but is targeted based on the reference image to reduce the local distortion as much as possible.

[0005] Although this method is extremely innovative and uses an attention mechanism as well as correlation embedding, it still has some drawbacks such as slow training speed and the actual distortion level not being as expected for the naked eye. Further improvements are necessary. Summary of the invention

[0006] The present invention aims to solve one of the technical problems existing in the related art at least to a certain extent.

[0007] One object of the present invention is to provide a method for constructing an image super-resolution model based on Transfomer, which is used to construct a model for super-resolution image restoration, especially for the distortion problem of detail texture, using a correction method of a reference image, and embedding a retention mechanism in the current RetNet structure, so that training can adopt different attention to positions at different distances, pay attention to the relationship between pixels close to each other as much as possible, reduce the weight of pixels far away, and better restore.

[0008] Another object of the present invention is to provide an image super-resolution method using a model constructed using the above-mentioned construction method.

[0009] In order to achieve the above-mentioned object, the present invention provides a method for constructing an image super-resolution model based on Transformer, comprising:

[0010] Obtain low-resolution images and reference images from a public database, sample the reference images, and construct a training dataset using the low-resolution images, the reference images, and the sampled reference images;

[0011] Constructing an initial learning framework including a texture feature extraction layer, a relative position encoding layer, a correlation embedding layer and a parallel loop embedding layer, taking a low-resolution image, a reference image and a sampled reference image as inputs of the initial learning framework, and outputting a high-resolution image;

[0012] The initial learning framework is trained according to the training data set, and the loss function is designed to correct the model parameters to obtain a trained image super-resolution model.

[0013] A further preferred technical solution of the present invention is to obtain low-resolution images and reference images from a public database, sample the reference images, and construct a training data set with the low-resolution images, the reference images, and the sampled reference images; comprising:

[0014] Obtain several groups of images from a public database, each group of images includes at least a reference image RF and a low-resolution image LR, assuming that the number of channels of the reference image RF is C, the pixel height is H, and the pixel width is W;

[0015] The low-resolution image LR is upsampled by interpolation and adjusted to the same resolution as the reference image RF as the information source of Q in the model. The reference image RF is then upsampled by interpolation and restored to its original resolution as the information source of K in the model. The original reference image RF is used as the information source of V in the model.

[0016] Preferably, an initial learning framework including a texture feature extraction layer, a relative position encoding layer, a correlation embedding layer and a parallel loop embedding layer is constructed, and a low-resolution image, a reference image and a sampled reference image are used as inputs of the initial learning framework to output a high-resolution image; including

[0017] Constructing a texture feature extraction layer to extract texture information of the low-resolution image, the reference image and the sampled reference image and transform it into feature information;

[0018] Construct a relative position encoding layer to add the position information of image pixels to the feature information;

[0019] Construct a correlation embedding layer to embed the correlation information between the reference image and the low-resolution image into the feature information;

[0020] Construct a parallel recurrent embedding layer, add a retention mechanism, assign different weights to different positions, and calculate the attention matrix consisting of soft attention and hard attention;

[0021] Combine the input low-resolution image and the attention matrix to output a high-resolution image.

[0022] Preferably, a texture feature extraction layer is constructed to extract texture information of a low-resolution image, a reference image and a sampled reference image and transform it into feature information; including:

[0023] Construct a texture feature extraction layer formed by stacking multiple deep neural networks, input the low-resolution image, reference image and sampled reference image into the texture feature extraction layer, and convert the low-resolution image into X Q , the sampled reference image is converted to X K , the reference image is converted to X V ; At the same time, the two-dimensional output is flattened, and each input is equivalent to n features.

[0024] Preferably, a relative position coding layer is constructed to add the position information of the image pixels to the feature information; including:

[0025] Randomly initialize three weight matrices W Q , W K , W V , and X Q , X K , X V Perform matrix multiplication and calculate Q = X Q W Q , K = X K W K , V = X V W V ;

[0026] Then, the relative position embedding method is used to encode the obtained Q and K in the relative position. The calculation formula is:

[0027] Q=X Q W Q ☉Θ

[0028]

[0029] Among them, ⊙ represents element-by-element multiplication, Θ=e inθ , is the complex conjugate of Θ.

[0030] Preferably, a correlation embedding layer is constructed to embed the correlation information between the reference image and the low-resolution image into the feature information; including:

[0031] First, each feature of Q and K is normalized and expressed as:

[0032]

[0033] Among them, q i is the i-th feature of Q, k j is the j-th feature of K;

[0034] Then matrix multiplication is used to calculate the similarity of each feature, expressed as:

[0035] Relative(x)=QK T

[0036] Get a correlation matrix whose elements are r i,j =q i ·k j .

[0037] As a preferred method, a parallel recurrent embedding layer is constructed, a retention mechanism is added, different weights are assigned to different positions, and an attention matrix consisting of soft attention and hard attention is calculated; including:

[0038] Considering that different positions have different influences on the current position, a weight matrix is ​​set to assign different weights to different positions. The specific formula is:

[0039] S(x)=Relative(x)☉D

[0040] Where D is the causal masking matrix, M is the feature dimension of K, N is the feature dimension of Q, n is the horizontal coordinate, m is the vertical coordinate, D is expressed as:

[0041]

[0042] Calculate the soft attention SA and hard attention HA. The soft attention SA is the embodiment of the correlation. The soft attention SA is expressed as:

[0043] SA=maxr i,j

[0044] where r i,j is an element in the correlation matrix;

[0045] Hard attention HA first obtains the feature block with the highest correlation with the position through the correlation matrix, which is expressed as:

[0046]

[0047] Then perform matrix multiplication with V to obtain hard attention HA, which is expressed as:

[0048] HA i =Vh i

[0049] Combine the input low-resolution image and attention matrix to output a high-resolution image; including:

[0050] Add a preprocessed low-resolution image F, combine HA and SA, and get the result, expressed as:

[0051] OUT=F+conv(concat(F,HA))☉SA

[0052] OUT is the output high-resolution image.

[0053] As a preferred method, the loss function combines three loss evaluation functions and adds trainable parameters. The calculation formula is:

[0054]

[0055] in, is the L1 loss function; is the adversarial training loss; is the perceptual loss, representing the reconstruction loss of the feature.

[0056] Another aspect of the present invention provides an image super-resolution method, comprising:

[0057] Using the above construction method, a trained image super-resolution model is obtained;

[0058] The low-resolution image and the reference image are input into the trained image super-resolution model to output a high-resolution image.

[0059] Another aspect of the present invention provides an image super-resolution system, comprising:

[0060] A model construction module, used to obtain a trained image super-resolution model using the above construction method;

[0061] The image generation module is used to input low-resolution images and reference images into the trained image super-resolution model and output high-resolution images.

[0062] Another aspect of the present invention provides a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions enabling a computer to execute the above-mentioned construction method.

[0063] Another aspect of the present invention provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus, and the processor calls the logic instructions in the memory to execute the above-mentioned construction method.

[0064] Yet another aspect of the present invention provides a computer program product, the computer program product comprising a computer program, the computer program being stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer executes the above-mentioned construction method.

[0065] Beneficial effects: The present invention optimizes the framework used by the image super-resolution model. First, relative position coding is added in the feature extraction stage, so that the position information of the image can be fully utilized to maximize the reliability of feature extraction. In addition, a nonlinear part is added, which is not so sensitive to extreme data. Based on the added relative position coding, the correlation between the high-definition image and the reference image is calculated by first normalizing the features and then performing QK direct matrix multiplication as a substitute for cosine similarity, which facilitates subsequent attention calculations. The present invention also uses a parallel loop embedding layer to assign different weights to different positions to improve the training effect. The present invention uses a loss function and a large number of samples for training, and uses a parallel loop training mode to accelerate training.

[0066] In summary, the present invention improves the ability to extract detailed textures, reduces local distortion, better establishes the connection between low resolution and high resolution, and improves the restoration effect and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 The present invention is a flowchart of the method for constructing a Transfomer-based image super-resolution model. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, and they should not be understood as limitations on the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In the description of the present invention, it should be understood that the terms used are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0069] Combine the following Figure 1The invention describes a Transfomer-based image super-resolution model construction method and an image super-resolution method.

[0070] Embodiment 1: This embodiment provides a method for constructing an image super-resolution model based on Transformer, which generally includes:

[0071] Obtain low-resolution images and reference images from a public database, sample the reference images, and construct a training dataset using the low-resolution images, the reference images, and the sampled reference images;

[0072] Constructing an initial learning framework including a texture feature extraction layer, a relative position encoding layer, a correlation embedding layer and a parallel loop embedding layer, taking a low-resolution image, a reference image and a sampled reference image as inputs of the initial learning framework, and outputting a high-resolution image;

[0073] The initial learning framework is trained according to the training data set, and the loss function is designed to correct the model parameters to obtain a trained image super-resolution model.

[0074] The above method is described in detail with a specific embodiment, and the specific steps are:

[0075] S1. Obtain several groups of images from the public database CUFED. Each group of images includes at least a reference image RF, a low-resolution image LR, and a high-resolution image HR. Assume that the number of channels of the reference image RF is C, the pixel height is H, and the pixel width is W.

[0076] The low-resolution image LR is upsampled by interpolation and adjusted to the same resolution as the reference image RF as the information source of Q in the model; then the reference image RF is upsampled by interpolation and restored to its original resolution as the information source of K in the model; the original reference image RF is used as the information source of V in the model. These three images are used for texture migration.

[0077] S2, constructing a texture feature extraction layer to extract texture information of the low-resolution image, the reference image and the sampled reference image and transform it into feature information;

[0078] Construct a texture feature extraction layer formed by stacking multiple deep neural networks, put the low-resolution image, reference image and sampled reference image into multiple vgg stacked deep neural networks, and convert the above three images of C*H*W into a new N, H*W, C*K structure, where N is the batch. The specific method is:

[0079] Using different layers of the vgg network, we perform three-layer progressive feature extraction. The first layer uses the first two layers of vgg to convert the original 3-channel pixels into 64 channels and activate them with a nonlinear activation function. In the second layer, based on the previous results, we further perform convolution and activation functions without changing the number of channels to improve the data representation capabilities of the features. Then, we perform a maximum pooling layer, and then convert the 64 channels into 128 channels for convolution. The third layer is similar to the second layer. First, we perform convolution without changing the channels, then perform maximum pooling, and then convert the 128 channels into 256 channels.

[0080] The above three-layer structure is divided into three levels to extract features, realizing multi-level and different channel features. Q , the sampled reference image is converted to X K , the reference image is converted to X V ; At the same time, the original 3-channel color dimension is increased to 256 channels.

[0081] S3. Construct a relative position encoding layer to add the position information of image pixels to the feature information. Through relative position encoding, the position information of image pixels is added to the calculation, and nonlinear factors are added to make the position of the image participate in the training process. In this way, the super-resolution result can be more consistent with the position expression of image pixels and reduce distortion.

[0082] Randomly initialize three weight matrices W Q , W K , W V , and X Q , X K , X V Perform matrix multiplication and calculate Q = X Q W Q , K = X K W K , V = X V W V ;

[0083] Then, the relative position embedding method is used to encode the obtained Q and K in the relative position. The calculation formula is:

[0084] Q=X Q W Q ☉Θ

[0085]

[0086] Among them, ⊙ represents element-by-element multiplication, Θ=e inθ , is the complex conjugate of Θ.

[0087] After that, the relative position of the code is added with position information in the form of encoding, which is also a prerequisite for accelerated training.

[0088] S4. Construct a correlation embedding layer to embed the correlation information between the reference image and the low-resolution image into the feature information; the correlation embedding layer draws on the idea of ​​RNN to recursively process sequences, so that different time steps are weighted differently during processing, which also strengthens the role of relative position encoding in the previous step. At the same time, nonlinear factors are introduced in the process, which improves the learning ability of the original Transformer module.

[0089] Construct a correlation embedding layer to embed the correlation information between the reference image and the low-resolution image into the feature information; including:

[0090] First, each feature of Q and K is normalized and expressed as:

[0091]

[0092] Among them, q i is the i-th feature of Q, k j is the j-th feature of K;

[0093] Then matrix multiplication is used to calculate the similarity of each feature, expressed as:

[0094] Relative(x)=QK T

[0095] Get a correlation matrix whose elements are r i,j =q i ·k j .

[0096] After this step is completed, a basic processed feature representation that can represent the correlation between QK can be obtained. In fact, this is to normalize first, and then use matrix multiplication to calculate the similarity of each feature as the subsequent calculation information. The essential purpose of similarity embedding is to understand the similarity between a certain pixel of RF and LR, which is also the basis for calculating attention later.

[0097] S5. Construct a parallel recurrent embedding layer, add a retention mechanism, assign different weights to different positions, calculate the attention matrix composed of soft attention and hard attention, and perform parallel recurrent embedding. This is derived from the idea of ​​sequence processing, that is, to reduce some time steps that are too far away and in the future, and to increase the weights of time steps that are close and in the past as much as possible. In image processing, this is to reduce the features that are far away as much as possible.

[0098] Because in the previous step S3, the nonlinear function was used to embed the relative position, and considering that different positions have different influences on the current state, a weight matrix was made to implement the function of different weights in different places. The specific formula is as follows:

[0099] S(x)=Relative(x)⊙D

[0100] Where D is the causal masking matrix, M is the feature dimension of K, N is the feature dimension of Q, n is the horizontal coordinate, m is the vertical coordinate, D is expressed as:

[0101]

[0102] Calculate the soft attention SA and hard attention HA. The soft attention SA is the embodiment of the correlation. The soft attention SA is expressed as:

[0103] SA=maxr i，j

[0104] where r i,j is an element in the correlation matrix;

[0105] Hard attention HA first obtains the feature block with the highest correlation with the position through the correlation matrix, which is expressed as:

[0106]

[0107] Then perform matrix multiplication with V to obtain hard attention HA, which is expressed as:

[0108] HA i =Vh i

[0109] where r i,j It is the correlation mentioned in step S3, which is actually already in S. When actually implementing it, you only need to look at S.

[0110] SA and HA are trained in parallel through parallel loop embedding layers to improve efficiency. The addition of Rentention mechanism is the key point of the present invention, that is, causal masking. We perform a certain masking on the similarity obtained in the previous step. This masking is determined by the position of the feature block. The closer to the feature position, the higher the weight. Therefore, the position information can be emphasized more in training, and the "attention" of the model is more concerned with the nearby feature blocks, which can better improve the effect of the attention mechanism.

[0111] S6. Add a preprocessed low-resolution image F, combined with HA and SA, and the final super-resolution result is expressed as:

[0112] OUT=F+conv(concat(F,HA))☉SA

[0113] OUT is the output high-resolution image.

[0114] S7. Design a loss function, modify the training according to the loss function, and continuously train with a sufficient number of samples. During the process, the simplified training of the previous parallel loop embedding layer will be applied to speed up the training and reach convergence faster.

[0115] The result OUT is compared with the high-resolution image HR and measured by the loss function. The loss function of this embodiment combines three loss evaluation functions and adds trainable parameters. The calculation formula is:

[0116]

[0117] in, is the L1 loss function; is the adversarial training loss; is the perceptual loss, representing the reconstruction loss of the feature.

[0118] The L1 loss function is used as the reconstruction loss function. In the field of super-resolution, L1 is more effective than L2. Here, it targets the image pixels themselves rather than the features. The formula is as follows:

[0119]

[0120] Where C represents the number of channels, H represents the pixel height, W represents the pixel width, and y represents the channel vector of the image.

[0121] The adversarial training loss is used, which is a loss function derived from the GAN network. Its main purpose is to make the distribution of the super-resolved image and the original image as consistent as possible. It is divided into two parts, one is the discriminator loss One is the generator loss In this embodiment, WGAN-GP is used to improve the trainability and consistency of the pixel data distribution between the restored result and the original image as much as possible. The specific formula is as follows:

[0122]

[0123] in represents the data distribution of the pixels of the generated image, represents the data distribution of real image pixels, represents the data distribution we additionally generate, which is between the generated image and the real image. The specific generation formula is as follows:

[0124]

[0125] That is, a linear combination of the real image and the generated image, α can be considered as a hyperparameter.

[0126] One part uses the feature comparison between the high-resolution original image and the super-resolution result, and the other part uses the feature comparison between the super-resolution result and the HA feature, that is, the feature reconstruction loss function, the formula is as follows:

[0127]

[0128] Among them, i and j represent a layer of the network structure, which can be considered as hyperparameters. The one that has the best effect is selected, generally the output of the last layer.

[0129] According to the above loss function, by training a large number of samples, we can obtain an image super-resolution model in which the detailed texture of the restored image is more consistent with the original image.

[0130] Embodiment 2: This embodiment provides an image super-resolution method, comprising:

[0131] Using the construction method of Example 1, a trained image super-resolution model is obtained;

[0132] The low-resolution image and the reference image are input into the trained image super-resolution model to output a high-resolution image.

[0133] Embodiment 3: This embodiment provides an image super-resolution system, comprising:

[0134] A model construction module, used to obtain a trained image super-resolution model using the construction method of Example 1;

[0135] The image generation module is used to input low-resolution images and reference images into the trained image super-resolution model and output high-resolution images.

[0136] Embodiment 4: This embodiment provides a non-transitory computer-readable storage medium, on which computer instructions are stored, and the computer instructions enable a computer to execute the image super-resolution method of embodiment 3, the method comprising the following steps:

[0137] Using the construction method of Example 1, a trained image super-resolution model is obtained;

[0138] The low-resolution image and the reference image are input into the trained image super-resolution model to output a high-resolution image.

[0139] Embodiment 5: This embodiment provides an electronic device, which may include: a processor, a communications interface, a memory, and a communication bus, wherein the processor, the communications interface, and the memory communicate with each other via the communication bus. The processor may call the logic instructions in the memory to execute the image super-resolution method of embodiment 3, which includes the following steps:

[0140] Using the construction method of Example 1, a trained image super-resolution model is obtained;

[0141] The low-resolution image and the reference image are input into the trained image super-resolution model to output a high-resolution image.

[0142] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0143] Embodiment 6: This embodiment provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image super-resolution method of embodiment 3. The method includes the following steps:

[0144] Using the construction method of Example 1, a trained image super-resolution model is obtained;

[0145] The low-resolution image and the reference image are input into the trained image super-resolution model to output a high-resolution image.

[0146] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0147] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing an image super-resolution model based on Transformer. It is characterized in that include: Obtain low-resolution images and reference images from a public database, sample the reference images, and construct a training dataset using the low-resolution images, the reference images, and the sampled reference images; Constructing an initial learning framework including a texture feature extraction layer, a relative position encoding layer, a correlation embedding layer and a parallel loop embedding layer, taking a low-resolution image, a reference image and a sampled reference image as inputs of the initial learning framework, and outputting a high-resolution image; The initial learning framework is trained according to the training data set, and the loss function is designed to correct the model parameters to obtain a trained image super-resolution model.

2. The method for constructing an image super-resolution model based on Transformer according to claim 1, It is characterized in that Obtain low-resolution images and reference images from a public database, sample the reference images, and construct a training dataset using the low-resolution images, reference images, and sampled reference images; including: Obtain several groups of images from a public database, each group of images includes at least a reference image RF and a low-resolution image LR, assuming that the number of channels of the reference image RF is C, the pixel height is H, and the pixel width is W; The low-resolution image LR is upsampled by interpolation and adjusted to the same resolution as the reference image RF as the information source of Q in the model. The reference image RF is then upsampled by interpolation and restored to its original resolution as the information source of K in the model. The original reference image RF is used as the information source of V in the model.

3. The method for constructing an image super-resolution model based on Transformer according to claim 1, It is characterized in that Constructing an initial learning framework including a texture feature extraction layer, a relative position encoding layer, a correlation embedding layer and a parallel loop embedding layer, taking a low-resolution image, a reference image and a sampled reference image as inputs of the initial learning framework, and outputting a high-resolution image; include Constructing a texture feature extraction layer to extract texture information of the low-resolution image, the reference image and the sampled reference image and transform it into feature information; Construct a relative position encoding layer to add the position information of image pixels to the feature information; Construct a correlation embedding layer to embed the correlation information between the reference image and the low-resolution image into the feature information; Construct a parallel recurrent embedding layer, add a retention mechanism, assign different weights to different positions, and calculate the attention matrix consisting of soft attention and hard attention; Combine the input low-resolution image and the attention matrix to output a high-resolution image.

4. The method for constructing an image super-resolution model based on Transformer according to claim 3, It is characterized in that Construct a texture feature extraction layer to extract texture information of the low-resolution image, the reference image and the sampled reference image and transform it into feature information; including: Construct a texture feature extraction layer formed by stacking multiple deep neural networks, input the low-resolution image, reference image and sampled reference image into the texture feature extraction layer, and convert the low-resolution image into X Q , the sampled reference image is converted to X K , the reference image is converted to X V ; At the same time, the two-dimensional output is flattened, and each input is equivalent to n features.

5. The method for constructing an image super-resolution model based on Transformer according to claim 4, It is characterized in that Construct a relative position encoding layer to add the position information of image pixels to the feature information; including: Randomly initialize three weight matrices W Q , Q K , W V , and X Q , X K , X V Perform matrix multiplication and calculate Q = X Q W Q , K = X K W K , V = X V W V ; Then, the relative position embedding method is used to encode the obtained Q and K in the relative position. The calculation formula is: Q=X Q W Q ⊙Θ Among them, ⊙ represents element-by-element multiplication, Θ=e inθ , is the complex conjugate of Θ.

6. The method for constructing an image super-resolution model based on Transformer according to claim 5, It is characterized in that Construct a correlation embedding layer to embed the correlation information between the reference image and the low-resolution image into the feature information; including: First, each feature of Q and K is normalized and expressed as: Among them, q i is the i-th feature of Q, k j is the jth feature of K; Then matrix multiplication is used to calculate the similarity of each feature, expressed as: Relative(x)=QK A Get a correlation matrix whose elements are r i,j =q i ·k j .

7. The method for constructing an image super-resolution model based on Transformer according to claim 6, It is characterized in that Construct a parallel recurrent embedding layer, add a retention mechanism, assign different weights to different positions, and calculate the attention matrix consisting of soft attention and hard attention; including: Considering that different positions have different influences on the current position, a weight matrix is ​​set to assign different weights to different positions. The specific formula is: S(x)=Relative(x)⊙D Where D is the causal masking matrix, M is the feature dimension of K, N is the feature dimension of Q, n is the horizontal coordinate, m is the vertical coordinate, D is expressed as: Calculate the soft attention SA and hard attention HA. The soft attention SA is the embodiment of the correlation. The soft attention SA is expressed as: SA=maxr i,j where r i,j is an element in the correlation matrix; Hard attention HA first obtains the feature block with the highest correlation with the position through the correlation matrix, which is expressed as: Then perform matrix multiplication with V to obtain hard attention HA, which is expressed as: HAH i =Vh i Combine the input low-resolution image and attention matrix to output a high-resolution image; including: Add a preprocessed low-resolution image F, combine HA and SA, and get the result, expressed as: OUT=F+conv(concat(F,HA))⊙SA OUT is the output high-resolution image.

8. The method for constructing an image super-resolution model based on Transformer according to claim 1, It is characterized in that The loss function combines three loss evaluation functions, and the calculation formula is: in, is the L1 loss function; is the adversarial training loss; is the perceptual loss, representing the reconstruction loss of the feature, and α, b, and c are trainable parameters.

9. An image super-resolution method, It is characterized in that include: Using the construction method described in any one of claims 1 to 8, a trained image super-resolution model is obtained; The low-resolution image and the reference image are input into the trained image super-resolution model to output a high-resolution image.

10. An image super-resolution system, It is characterized in that include: A model construction module, used to obtain a trained image super-resolution model using the construction method described in any one of claims 1 to 8; The image generation module is used to input low-resolution images and reference images into the trained image super-resolution model and output high-resolution images.

Images