Dual-path Transformer image super-resolution method and system

Abstract

The present invention relates to the field of image processing, and proposes a dual-path Transformer image super-resolution method and system, designs a dual-path Transformer network, and enhances texture detail information through a deep feature extraction module, wherein a dual-path feature fusion submodule adopts two Transformer branches to explore the long-range dependency in the spatial domain, and then fuses the output features of the upper and lower branches, which not only realizes the information interaction between the two branches and enhances the acquisition of texture detail information, but also can compress channels and reduce the number of network parameters, and a global feature fusion submodule fuses feature information from different levels to obtain deep features to effectively combine the advantages of the Transformer network and the convolutional neural network, and the present invention improves the reconstruction quality of image texture details by image super-resolution.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a dual-path Transformer image super-resolution method and system. Background Art

[0002] With the rapid development of computer vision technology, the requirements for image quality are gradually increasing. However, due to the hardware performance limitations of the acquisition equipment, the image quality is usually uneven, especially for a large number of low-resolution images. In order to meet the image quality requirements, image super-resolution processing is required.

[0003] In the prior art, image super-resolution processing usually adopts deep learning convolutional neural networks. In convolutional neural networks, methods often used to explore long-range dependencies between spatial pixels include using large-sized convolution kernels, pooling operations, increasing the depth of the network, or using dilated convolutions. Although these methods can increase the receptive field to a certain extent, when it comes to a certain layer of the convolutional neural network, the convolution operation is still performed locally. In addition, the use of larger-sized convolution kernels and deeper networks will increase the number of parameters and the amount of calculation of the network model. Pooling operations will also cause certain information loss. The use of dilated convolutions is not only prone to information loss, but also causes some checkerboard artifacts.

[0004] Therefore, how to design an image super-resolution method to reconstruct the long-range dependencies in the image and improve the reconstruction quality of the image texture details. Summary of the invention

[0005] Based on this, the purpose of the present invention is to provide a dual-path Transformer image super-resolution method and system, and a dual-path Transformer network is designed. Texture detail information is enhanced through a deep feature extraction module, wherein a dual-path feature fusion submodule uses two Transformer branches to explore the long-range dependency in the spatial domain, and then fuses the output features of the upper and lower branches, which not only realizes the information interaction between the two branches and enhances the acquisition of texture detail information, but also compresses the channel and reduces the number of network parameters. The global feature fusion submodule fuses feature information from different levels to obtain deep features to effectively combine the advantages of the Transformer network and the convolutional neural network. The present invention improves the reconstruction quality of image texture details by image super-resolution.

[0006] The present invention proposes a dual-path Transformer image super-resolution method, comprising:

[0007] Obtain the low-resolution image to be processed and preprocess it, and input the preprocessed low-resolution image into the two-way Transformer network;

[0008] The shallow feature extraction module obtains the shallow features of the low-resolution image to input into the deep feature extraction module;

[0009] A deep feature extraction module obtains deep features according to the shallow features, and the deep feature extraction module includes a dual-path feature fusion submodule and a global feature fusion submodule, and inputs the deep features into a reconstruction module;

[0010] The reconstruction module reconstructs and generates a high-resolution image according to the deep features.

[0011] In summary, according to the above-mentioned two-way Transformer image super-resolution method, a two-way Transformer network is designed, and texture detail information is enhanced through a deep feature extraction module. The two-way feature fusion submodule uses two Transformer branches to explore the long-range dependency in the spatial domain, and then fuses the output features of the upper and lower branches. It not only realizes the information interaction between the two branches and enhances the acquisition of texture detail information, but also compresses the channel and reduces the number of network parameters. The global feature fusion submodule fuses feature information from different levels to obtain deep features to effectively combine the advantages of the Transformer network and the convolutional neural network. The present invention improves the reconstruction quality of image texture details by image super-resolution. Specifically, a low-resolution image to be processed is obtained and preprocessed, the preprocessed low-resolution image is input into a two-way Transformer network, a shallow feature extraction module obtains shallow features of the low-resolution image and inputs them into a deep feature extraction module to preliminarily obtain shallow features of the low-resolution image, a deep feature extraction module obtains deep features based on the shallow features, the deep feature extraction module includes a two-way feature fusion submodule and a global feature fusion submodule, the deep features are input into a reconstruction module, the acquisition of texture detail information is enhanced, the reconstruction module reconstructs a high-resolution image based on the deep features, and the present invention improves the reconstruction quality of image texture details by image super-resolution.

[0012] Furthermore, the step of the deep feature extraction module acquiring the deep features according to the shallow features specifically includes:

[0013] After the low-resolution image is preprocessed, it is input into a shallow feature extraction module to extract shallow features of the low-resolution image. The shallow feature extraction module includes a 3×3 convolution layer. The algorithm for extracting the shallow features is as follows:

[0014] ,

[0015] in, Represents shallow features, represents a shallow feature extraction operation, Represents a low-resolution image;

[0016] The shallow features are input into a deep feature extraction module, the deep feature extraction module obtains deep features according to the shallow features, and the deep feature extraction module includes a dual-path feature fusion submodule and a global feature fusion submodule;

[0017] The dual-path feature fusion submodule performs dual-path feature extraction and fusion;

[0018] The global feature fusion submodule obtains deep features according to the fused dual-path features;

[0019] The algorithm for obtaining deep features based on shallow features is as follows:

[0020] ,

[0021] in, Represents deep features, Represents a deep feature extraction operation.

[0022] Furthermore, the dual-path feature fusion submodule specifically includes:

[0023] The dual-path feature fusion submodule includes a plurality of identical dual-path feature fusion units, each of which is a dual-path branch structure, the dual-path branch structure includes two identical branches, each branch includes a Transformer block, the Transformer block of the previous dual-path feature fusion unit is connected with the Transformer block of the next dual-path feature fusion unit through a residual structure, the residual structure includes an original branch and a residual branch, and the residual branch includes a Concat block and a 1×1 convolution block;

[0024] The specific algorithm of the dual-path feature fusion submodule is as follows:

[0025] = ( )+ ,

[0026] = ( )+ ,

[0027] = ( ( , )),

[0028] in, represents the output characteristics of the upper branch, represents the output characteristics of the lower branch, Represents the Transformer block operation of the upper branch, Represents the Transformer block operation of the lower branch, represents the input characteristics of the upper branch, represents the input characteristics of the lower branch, represents the fusion features of the upper and lower branches, represents a 1×1 convolution operation, Represents a merge operation. Indicates the ordinal number of the dual-path feature fusion unit.

[0029] Furthermore, the Transformer block specifically includes:

[0030] The Transformer block includes a multi-head attention mechanism layer and a feed-forward network layer. The specific algorithm of the Transformer block is as follows:

[0031] ,

[0032] in, represents the output features of the Transformer block, represents the feedforward network layer processing, represents the multi-head attention mechanism layer processing, Represents the input features of the Transformer block;

[0033] The multi-head attention mechanism layer processes the features of the previous two-way feature fusion unit input into the current two-way feature fusion unit according to point convolution and matrix deformation, and transforms to obtain the query matrix, key matrix and value matrix. The query matrix, key matrix and value matrix are defined as follows:

[0034]

[0035] in, represents the query matrix, represents the key matrix, represents the value matrix, represents the ReLU activation function, represents the matrix deformation operation, represents a 1×1 convolution operation, Represents the input features of the multi-head attention mechanism layer;

[0036] The query matrix, key matrix and value matrix are split into multiple parts according to the number of attention heads. The algorithm of the multi-head attention mechanism is as follows:

[0037] ,

[0038] in, Indicates The first output of the attention The value of pixels, represents the number of attention heads, represents the ordinal numbers of the key matrix and value matrix on the attention head, Indicates height, Indicates width, Represents the pixel ordinal number, represents the transposed matrix, represents a constant that restricts the denominator to be non-zero;

[0039] According to the multi-head attention mechanism, the multi-head attention enhanced features are obtained. The algorithm for obtaining the multi-head attention enhanced features is as follows:

[0040] ,

[0041] in, Represents the multi-head attention enhanced features of the final output of the multi-head attention mechanism layer, represents a 3×3 convolution operation, It represents the output of multiple attention heads combined and adjusted features, Represents the input features of the multi-head attention mechanism layer.

[0042] Furthermore, the feedforward network layer specifically includes:

[0043] The feedforward network layer is a residual structure, the residual structure includes a residual branch and an original branch, the residual branch includes two 1×1 convolution blocks, two GELU layers and a 3×3 grouped convolution block and is arranged in an inverse bottleneck structure;

[0044] The algorithm of the feedforward network layer is as follows:

[0045] ,

[0046] in, represents the final output feature of the feedforward network layer, represents a 1×1 convolution operation, g represents the GELU function, represents a 3×3 grouped convolution operation, Represents the multi-head attention enhanced features of the final output of the multi-head attention mechanism layer.

[0047] Furthermore, the global feature fusion submodule specifically includes:

[0048] The global feature fusion submodule obtains deep features according to the dual-path features output by the dual-path feature fusion submodule, and the global feature fusion submodule includes a Concat block, a 1×1 convolution block and a 3×3 convolution block;

[0049] The specific algorithm of the global feature fusion submodule is as follows:

[0050]

[0051] ,

[0052] in, It represents the dual-path features finally fused and output by the dual-path feature fusion submodule. Represents deep features, represents a 1×1 convolution operation, represents a 3×3 convolution operation, Represents a merge operation. represents the output feature of the upper branch of the Nth dual-path feature fusion unit, It represents the output feature of the lower branch of the Nth dual-path feature fusion unit, and N represents the number of dual-path feature fusion units in the dual-path feature fusion submodule.

[0053] Furthermore, the step of the reconstruction module generating a high-resolution image based on the deep feature reconstruction specifically includes:

[0054] The reconstruction module obtains the deep features. The reconstruction module includes a 3×3 convolution layer and a sub-pixel operation layer. The reconstruction module reconstructs and generates a high-resolution image based on the deep features. The specific reconstruction generation algorithm is as follows:

[0055] ,

[0056] in, represents the high-resolution image generated by the final reconstruction, Represents a rebuild operation. Represents deep features, represents the bilinear interpolation upsampling operation, Represents the original low-resolution image.

[0057] The present invention proposes a dual-path Transformer image super-resolution system, comprising:

[0058] The acquisition module is used to obtain the low-resolution image to be processed and perform preprocessing, and input the preprocessed low-resolution image into the dual-path Transformer network;

[0059] A shallow feature extraction module, used for the shallow feature extraction module to obtain shallow features of the low-resolution image to input into the deep feature extraction module;

[0060] A deep feature extraction module, used for the deep feature extraction module to obtain deep features according to the shallow features, the deep feature extraction module includes a dual-path feature fusion submodule and a global feature fusion submodule, and inputs the deep features into the reconstruction module;

[0061] A reconstruction module is used to reconstruct and generate a high-resolution image based on the deep features.

[0062] The present invention also provides a storage medium, which stores one or more programs, and when the programs are executed by a processor, the two-way Transformer image super-resolution method as described above is implemented.

[0063] The present invention also provides a computer device, the computer device comprising a memory and a processor, wherein:

[0064] The memory is used to store computer programs;

[0065] When the processor is used to execute the computer program stored in the memory, the dual-path Transformer image super-resolution method as described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A flowchart of a dual-path Transformer image super-resolution method proposed in the first embodiment of the present invention;

[0067] Figure 2 A flowchart of a dual-path Transformer image super-resolution method proposed in the second embodiment of the present invention;

[0068] Figure 3 This is a schematic diagram of the structure of the dual-path Transformer image super-resolution method proposed in the third embodiment of the present invention.

[0069] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0070] In order to facilitate understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0071] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0072] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0073] See also Figure 1 , which is a flowchart of a two-way Transformer image super-resolution method proposed in a first embodiment of the present invention, wherein the two-way Transformer image super-resolution method comprises steps S01 to S04, wherein:

[0074] Step S01: obtaining a low-resolution image to be processed and preprocessing it, and inputting the preprocessed low-resolution image into a dual-path Transformer network;

[0075] It should be noted that the preprocessing in this embodiment includes data enhancement processing, which includes randomly rotating the low-resolution image horizontally, vertically and by 90 degrees. In the pre-training of the dual-path Transformer network, the Adam optimizer is used to optimize the parameters of the entire network, and its initial learning rate is set to 0.0005. During the training process, the learning rate is halved every 200 stages.

[0076] Step S02: The shallow feature extraction module obtains shallow features of the low-resolution image to input into the deep feature extraction module;

[0077] It should be noted that, in this embodiment, after the low-resolution image is preprocessed, it is input into a shallow feature extraction module to extract shallow features of the low-resolution image. The shallow feature extraction module includes a 3×3 convolution layer. The algorithm for extracting the shallow features is as follows:

[0078] ,

[0079] in, Represents shallow features, represents a shallow feature extraction operation, Represents a low-resolution image.

[0080] Step S03: The deep feature extraction module obtains deep features according to the shallow features. The deep feature extraction module includes a dual-path feature fusion submodule and a global feature fusion submodule, and inputs the deep features into the reconstruction module;

[0081] It should be noted that in this embodiment, the shallow features are input into a deep feature extraction module, and the deep feature extraction module obtains deep features according to the shallow features. The deep feature extraction module includes a dual-path feature fusion submodule and a global feature fusion submodule;

[0082] The dual-path feature fusion submodule performs dual-path feature extraction and fusion;

[0083] The global feature fusion submodule obtains deep features according to the fused dual-path features;

[0084] The algorithm for obtaining deep features based on shallow features is as follows:

[0085] ,

[0086] in, Represents deep features, Represents deep feature extraction operations;

[0087] In this embodiment, the dual-path feature fusion submodule includes a plurality of identical dual-path feature fusion units, each of which is a dual-path branch structure, the dual-path branch structure includes two identical branches, each branch includes a Transformer block, the Transformer block of the previous dual-path feature fusion unit is connected with the Transformer block of the next dual-path feature fusion unit through a residual structure, the residual structure includes an original branch and a residual branch, and the residual branch includes a Concat block and a 1×1 convolution block;

[0088] The specific algorithm of the dual-path feature fusion submodule is as follows:

[0089] = ( )+ ,

[0090] = ( )+ ,

[0091] = ( ( , )),

[0092] in, represents the output characteristics of the upper branch, represents the output characteristics of the lower branch, Represents the Transformer block operation of the upper branch, Represents the Transformer block operation of the lower branch, represents the input characteristics of the upper branch, represents the input characteristics of the lower branch, represents the fusion features of the upper and lower branches, represents a 1×1 convolution operation, Represents a merge operation. Indicates the ordinal number of the dual-path feature fusion unit;

[0093] In this embodiment, the Transformer block includes a multi-head attention mechanism layer and a feedforward network layer. The specific algorithm of the Transformer block is as follows:

[0094] ,

[0095] in, represents the output features of the Transformer block, represents the feedforward network layer processing, represents the multi-head attention mechanism layer processing, Represents the input features of the Transformer block;

[0096] The multi-head attention mechanism layer processes the features of the previous two-way feature fusion unit input into the current two-way feature fusion unit according to point convolution and matrix deformation, and transforms to obtain the query matrix, key matrix and value matrix. The query matrix, key matrix and value matrix are defined as follows:

[0097]

[0098] in, represents the query matrix, represents the key matrix, represents the value matrix, represents the ReLU activation function, represents the matrix deformation operation, represents a 1×1 convolution operation, Represents the input features of the multi-head attention mechanism layer;

[0099] The query matrix, key matrix and value matrix are split into multiple parts according to the number of attention heads. The algorithm of the multi-head attention mechanism is as follows:

[0100] ,

[0101] in, Indicates The first output of the attention The value of pixels, represents the number of attention heads, represents the ordinal numbers of the key matrix and value matrix on the attention head, Indicates height, Indicates width, Represents the pixel ordinal number, represents the transposed matrix, represents a constant that restricts the denominator to be non-zero;

[0102] Compared with other Transformer modules that use Softmax for normalization, this embodiment uses the denominator part in the above for normalization, and uses the associative law to reduce the computational complexity;

[0103] According to the multi-head attention mechanism, the multi-head attention enhanced features are obtained. The algorithm for obtaining the multi-head attention enhanced features is as follows:

[0104] ,

[0105] in, Represents the multi-head attention enhanced features of the final output of the multi-head attention mechanism layer, represents a 3×3 convolution operation, It represents the output of multiple attention heads combined and adjusted features, Represents the input features of the multi-head attention mechanism layer;

[0106] In this embodiment, the feedforward network layer is a residual structure, and the residual structure includes a residual branch and an original branch. The residual branch includes two 1×1 convolution blocks, two GELU layers and a 3×3 grouped convolution block and is arranged in an inverse bottleneck structure;

[0107] The algorithm of the feedforward network layer is as follows:

[0108] ,

[0109] in, represents the final output feature of the feedforward network layer, represents a 1×1 convolution operation, g represents the GELU function, represents a 3×3 grouped convolution operation, Represents the multi-head attention enhanced features of the final output of the multi-head attention mechanism layer;

[0110] In this embodiment, the global feature fusion submodule obtains deep features according to the dual-path features fused and output by the dual-path feature fusion submodule, and the global feature fusion submodule includes a Concat block, a 1×1 convolution block and a 3×3 convolution block;

[0111] The specific algorithm of the global feature fusion submodule is as follows:

[0112]

[0113] ,

[0114] in, It represents the dual-path features finally fused and output by the dual-path feature fusion submodule. Represents deep features, represents a 1×1 convolution operation, represents a 3×3 convolution operation, Represents a merge operation. represents the output feature of the upper branch of the Nth dual-path feature fusion unit, It represents the output feature of the lower branch of the Nth dual-path feature fusion unit, and N represents the number of dual-path feature fusion units in the dual-path feature fusion submodule.

[0115] Step S04: The reconstruction module reconstructs and generates a high-resolution image based on the deep features;

[0116] It should be noted that in this embodiment, the reconstruction module obtains deep features. The reconstruction module includes a 3×3 convolution layer and a sub-pixel operation layer. The reconstruction module reconstructs and generates a high-resolution image based on the deep features. The specific reconstruction generation algorithm is as follows:

[0117] ,

[0118] in, represents the high-resolution image generated by the final reconstruction, Represents a rebuild operation. Represents deep features, represents the bilinear interpolation upsampling operation, Represents the original low-resolution image.

[0119] In summary, according to the above-mentioned two-way Transformer image super-resolution method, a two-way Transformer network is designed, and texture detail information is enhanced through a deep feature extraction module. The two-way feature fusion submodule uses two Transformer branches to explore the long-range dependency in the spatial domain, and then fuses the output features of the upper and lower branches. It not only realizes the information interaction between the two branches and enhances the acquisition of texture detail information, but also compresses the channel and reduces the number of network parameters. The global feature fusion submodule fuses feature information from different levels to obtain deep features to effectively combine the advantages of the Transformer network and the convolutional neural network. The present invention improves the reconstruction quality of image texture details by image super-resolution. Specifically, a low-resolution image to be processed is obtained and preprocessed, the preprocessed low-resolution image is input into a two-way Transformer network, a shallow feature extraction module obtains shallow features of the low-resolution image and inputs them into a deep feature extraction module to preliminarily obtain shallow features of the low-resolution image, a deep feature extraction module obtains deep features based on the shallow features, the deep feature extraction module includes a two-way feature fusion submodule and a global feature fusion submodule, the deep features are input into a reconstruction module, the acquisition of texture detail information is enhanced, the reconstruction module reconstructs a high-resolution image based on the deep features, and the present invention improves the reconstruction quality of image texture details by image super-resolution.

[0120] See also Figure 2 , which is a flowchart of a two-way Transformer image super-resolution method proposed in a second embodiment of the present invention, wherein the two-way Transformer image super-resolution method comprises steps S11 to S15, wherein:

[0121] Step S11: After the low-resolution image is preprocessed, it is input into a shallow feature extraction module to extract shallow features of the low-resolution image;

[0122] Step S12: inputting shallow features into a deep feature extraction module, and the deep feature extraction module obtains deep features according to shallow features;

[0123] Step S13: The dual-path feature fusion submodule extracts and fuses dual-path features;

[0124] Step S14: the global feature fusion submodule obtains deep features based on the fused dual-path features;

[0125] Step S15: The reconstruction module obtains the deep features and reconstructs a high-resolution image based on the deep features;

[0126] It should be noted that a comparison test was conducted between the dual-path Transformer image super-resolution method in this embodiment and other image super-resolution method network models in the prior art, wherein the comparison results obtained based on the PSNR and SSIM evaluation indicators are as shown in Table 1 below:

[0127] Table 1

[0128]

[0129] As can be seen from the above table, in the Set5 dataset, whether the magnification factor is ×2, ×3 or ×4, the present invention is significantly superior to other image super-resolution method network models in the prior art in terms of both PSNR / SSIM evaluation indicators;

[0130] When the magnification factor is ×2, the comparison of the parameters and calculation amount of the network model of the present invention and other image super-resolution methods is shown in Table 2 below:

[0131] Table 2

[0132]

[0133] It can be seen that the required parameter amount and overall calculation amount of the present invention are also significantly improved compared with other image super-resolution method network models in the prior art.

[0134] In summary, according to the above-mentioned two-way Transformer image super-resolution method, a two-way Transformer network is designed, and texture detail information is enhanced through a deep feature extraction module. The two-way feature fusion submodule uses two Transformer branches to explore the long-range dependency in the spatial domain, and then fuses the output features of the upper and lower branches. It not only realizes the information interaction between the two branches and enhances the acquisition of texture detail information, but also compresses the channel and reduces the number of network parameters. The global feature fusion submodule fuses feature information from different levels to obtain deep features to effectively combine the advantages of the Transformer network and the convolutional neural network. The present invention improves the reconstruction quality of image texture details by image super-resolution. Specifically, a low-resolution image to be processed is obtained and preprocessed, the preprocessed low-resolution image is input into a two-way Transformer network, a shallow feature extraction module obtains shallow features of the low-resolution image and inputs them into a deep feature extraction module to preliminarily obtain shallow features of the low-resolution image, a deep feature extraction module obtains deep features based on the shallow features, the deep feature extraction module includes a two-way feature fusion submodule and a global feature fusion submodule, the deep features are input into a reconstruction module, the acquisition of texture detail information is enhanced, the reconstruction module reconstructs a high-resolution image based on the deep features, and the present invention improves the reconstruction quality of image texture details by image super-resolution.

[0135] See also Figure 3 , which is a schematic diagram of the structure of a dual-path Transformer image super-resolution system proposed in the third embodiment of the present invention, and the system includes:

[0136] The acquisition module 10 is used to acquire the low-resolution image to be processed and perform preprocessing, and input the preprocessed low-resolution image into the dual-path Transformer network;

[0137] A shallow feature extraction module 20, used for the shallow feature extraction module to obtain shallow features of the low-resolution image to input into the deep feature extraction module;

[0138] A deep feature extraction module 30, which is used for the deep feature extraction module to obtain deep features according to the shallow features, and the deep feature extraction module includes a dual-path feature fusion submodule and a global feature fusion submodule, and inputs the deep features into the reconstruction module;

[0139] The reconstruction module 40 is used for the reconstruction module to reconstruct and generate a high-resolution image according to the deep features.

[0140] The present invention also proposes a computer storage medium on which one or more programs are stored, and when the program is executed by a processor, the above-mentioned two-way Transformer image super-resolution method is implemented.

[0141] The present invention also proposes a computer device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the above-mentioned dual-path Transformer image super-resolution method.

[0142] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain storage, communication, propagation or transmission of a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0143] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0144] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0145] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0146] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A two-way Transformer image super-resolution method, characterized in that: include: Obtain the low-resolution image to be processed and preprocess it, and input the preprocessed low-resolution image into the dual-path Transformer network; The shallow feature extraction module obtains the shallow features of the low-resolution image to input into the deep feature extraction module; A deep feature extraction module obtains deep features according to the shallow features, and the deep feature extraction module includes a dual-path feature fusion submodule and a global feature fusion submodule, and inputs the deep features into a reconstruction module; The step of the deep feature extraction module acquiring the deep features according to the shallow features specifically includes: After the low-resolution image is preprocessed, it is input into a shallow feature extraction module to extract shallow features of the low-resolution image. The shallow feature extraction module includes a 3×3 convolution layer. The algorithm for extracting the shallow features is as follows: ， in, Represents shallow features, represents a shallow feature extraction operation, Represents a low-resolution image; The shallow features are input into a deep feature extraction module, the deep feature extraction module obtains deep features according to the shallow features, and the deep feature extraction module includes a dual-path feature fusion submodule and a global feature fusion submodule; The dual-path feature fusion submodule performs dual-path feature extraction and fusion; The dual-path feature fusion submodule specifically includes: The dual-path feature fusion submodule includes a plurality of identical dual-path feature fusion units, each of which is a dual-path branch structure, the dual-path branch structure includes two identical branches, each branch includes a Transformer block, the Transformer block of the previous dual-path feature fusion unit is connected with the Transformer block of the next dual-path feature fusion unit through a residual structure, the residual structure includes an original branch and a residual branch, and the residual branch includes a Concat block and a 1×1 convolution block; The global feature fusion submodule obtains deep features according to the fused dual-path features; The global feature fusion submodule specifically includes: The global feature fusion submodule obtains deep features according to the dual-path features output by the dual-path feature fusion submodule, and the global feature fusion submodule includes a Concat block, a 1×1 convolution block and a 3×3 convolution block; The algorithm for obtaining deep features based on shallow features is as follows: ， in, Represents deep features, Represents deep feature extraction operations; The reconstruction module reconstructs and generates a high-resolution image according to the deep features.

2. The dual-path Transformer image super-resolution method according to claim 1, characterized in that: The specific algorithm of the dual-path feature fusion submodule is as follows: = ( )+ ， = ( )+ ， = ( ( , ))， in, represents the output characteristics of the upper branch, represents the output characteristics of the lower branch, Represents the Transformer block operation of the upper branch, Represents the Transformer block operation of the lower branch, represents the input characteristics of the upper branch, represents the input characteristics of the lower branch, represents the fusion features of the upper and lower branches, represents a 1×1 convolution operation, Represents a merge operation. Indicates the ordinal number of the dual-path feature fusion unit.

3. The dual-path Transformer image super-resolution method according to claim 1, characterized in that: The Transformer block specifically includes: The Transformer block includes a multi-head attention mechanism layer and a feed-forward network layer. The specific algorithm of the Transformer block is as follows: ， in, represents the output features of the Transformer block, represents the feedforward network layer processing, represents the multi-head attention mechanism layer processing, Represents the input features of the Transformer block; The multi-head attention mechanism layer processes the features of the previous two-way feature fusion unit input into the current two-way feature fusion unit according to point convolution and matrix deformation, and transforms to obtain the query matrix, key matrix and value matrix. The query matrix, key matrix and value matrix are defined as follows: in, represents the query matrix, represents the key matrix, represents the value matrix, represents the ReLU activation function, represents the matrix deformation operation, represents a 1×1 convolution operation, Represents the input features of the multi-head attention mechanism layer; The query matrix, key matrix and value matrix are split into multiple parts according to the number of attention heads. The algorithm of the multi-head attention mechanism is as follows: ， in, Indicates The first output of the attention The value of pixels, represents the number of attention heads, represents the ordinal numbers of the key matrix and value matrix on the attention head, Indicates height, Indicates width, Represents the pixel ordinal number, represents the transposed matrix, represents a constant that restricts the denominator to be non-zero; According to the multi-head attention mechanism, the multi-head attention enhanced features are obtained. The algorithm for obtaining the multi-head attention enhanced features is as follows: ， in, Represents the multi-head attention enhanced features of the final output of the multi-head attention mechanism layer, represents a 3×3 convolution operation, It represents the output of multiple attention heads combined and adjusted features, Represents the input features of the multi-head attention mechanism layer.

4. The dual-path Transformer image super-resolution method according to claim 3, characterized in that: The feedforward network layer specifically includes: The feedforward network layer is a residual structure, the residual structure includes a residual branch and an original branch, the residual branch includes two 1×1 convolution blocks, two GELU layers and a 3×3 grouped convolution block and is arranged in an inverse bottleneck structure; The algorithm of the feedforward network layer is as follows: ， in, represents the final output feature of the feedforward network layer, represents a 1×1 convolution operation, g represents the GELU function, represents a 3×3 grouped convolution operation, Represents the multi-head attention enhanced features of the final output of the multi-head attention mechanism layer.

5. The dual-path Transformer image super-resolution method according to claim 1, characterized in that: The specific algorithm of the global feature fusion submodule is as follows: ， in, It represents the dual-path features finally fused and output by the dual-path feature fusion submodule. Represents deep features, represents a 1×1 convolution operation, represents a 3×3 convolution operation, Represents a merge operation. represents the output feature of the upper branch of the Nth dual-path feature fusion unit, It represents the output feature of the lower branch of the Nth dual-path feature fusion unit, and N represents the number of dual-path feature fusion units in the dual-path feature fusion submodule.

6. The dual-path Transformer image super-resolution method according to claim 1, characterized in that: The step of reconstructing the high-resolution image by the reconstruction module according to the deep feature reconstruction specifically includes: The reconstruction module obtains the deep features. The reconstruction module includes a 3×3 convolution layer and a sub-pixel operation layer. The reconstruction module reconstructs and generates a high-resolution image based on the deep features. The specific reconstruction generation algorithm is as follows: ， in, represents the high-resolution image generated by the final reconstruction, Represents a rebuild operation. Represents deep features, represents the bilinear interpolation upsampling operation, Represents the original low-resolution image.

7. A two-way Transformer image super-resolution system, characterized in that: include: The acquisition module is used to obtain the low-resolution image to be processed and perform preprocessing, and input the preprocessed low-resolution image into the dual-path Transformer network; A shallow feature extraction module, used for the shallow feature extraction module to obtain shallow features of the low-resolution image to input into the deep feature extraction module; A deep feature extraction module, used for the deep feature extraction module to obtain deep features according to the shallow features, the deep feature extraction module includes a dual-path feature fusion submodule and a global feature fusion submodule, and inputs the deep features into the reconstruction module; The step of the deep feature extraction module acquiring the deep features according to the shallow features specifically includes: After the low-resolution image is preprocessed, it is input into a shallow feature extraction module to extract shallow features of the low-resolution image. The shallow feature extraction module includes a 3×3 convolution layer. The algorithm for extracting the shallow features is as follows: ， in, Represents shallow features, represents a shallow feature extraction operation, Represents a low-resolution image; The shallow features are input into a deep feature extraction module, the deep feature extraction module obtains deep features according to the shallow features, and the deep feature extraction module includes a dual-path feature fusion submodule and a global feature fusion submodule; The dual-path feature fusion submodule performs dual-path feature extraction and fusion; The dual-path feature fusion submodule specifically includes: The dual-path feature fusion submodule includes a plurality of identical dual-path feature fusion units, each of which is a dual-path branch structure, the dual-path branch structure includes two identical branches, each branch includes a Transformer block, the Transformer block of the previous dual-path feature fusion unit is connected with the Transformer block of the next dual-path feature fusion unit through a residual structure, the residual structure includes an original branch and a residual branch, and the residual branch includes a Concat block and a 1×1 convolution block; The global feature fusion submodule obtains deep features according to the fused dual-path features; The global feature fusion submodule specifically includes: The global feature fusion submodule obtains deep features according to the dual-path features output by the dual-path feature fusion submodule, and the global feature fusion submodule includes a Concat block, a 1×1 convolution block and a 3×3 convolution block; The algorithm for obtaining deep features based on shallow features is as follows: ， in, Represents deep features, Represents deep feature extraction operations; A reconstruction module is used to reconstruct and generate a high-resolution image based on the deep features.

8. A storage medium, characterized in that: The storage medium stores one or more programs, which, when executed by a processor, implement the dual-path Transformer image super-resolution method as described in any one of claims 1-6.

9. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein: The memory is used to store computer programs; When the processor is used to execute the computer program stored in the memory, the dual-path Transformer image super-resolution method described in any one of claims 1-6 is implemented.

Images