An image super-resolution reconstruction method

Abstract

The application provides an image super-resolution reconstruction method, comprising the following steps: S1: extracting shallow features of a low-resolution picture to be reconstructed; S2: inputting the extracted shallow features into a mixed codec to convert the shallow features into deep features; the mixed codec is a U-shaped network structure, comprising a plurality of MSACM modules, each MSACM module comprising a plurality of mixed attention modules; the mixed attention module simultaneously models global context information and local texture information through a parallel double-branch structure comprising a multi-head self-attention branch and a spatial Fourier frequency module branch, and the spatial Fourier frequency module introduces frequency domain characteristics through fast Fourier transform and inverse Fourier transform; S3: reconstructing a high-resolution picture according to the deep features by using a CARAFE up-sampling operator. The application provides an image super-resolution reconstruction method, and solves the problem that existing SR reconstruction technologies need to occupy more computing resources while improving reconstruction performance.

Description

Technical Field

[0001] This invention relates to the field of image super-resolution reconstruction technology, and more specifically, to an image super-resolution reconstruction method. Background Technology

[0002] Image super-resolution (SR) reconstruction aims to reconstruct a corresponding high-resolution (HR) image from a degraded low-resolution (LR) image. As a fundamental and underlying task in the field of computer vision, this technique has always been a key research focus and hot topic.

[0003] Generally, higher image resolution contains richer texture details, resulting in clearer visual effects and making it more suitable for subsequent image processing tasks. Currently, many devices are capable of meeting the clarity requirements of real-world scenarios, but due to limitations in equipment cost and shooting environment, images obtained in many situations often fail to meet high-resolution requirements. SR reconstruction technology solves these problems at the software level. On the one hand, it effectively reduces hardware costs, making device expansion and maintenance more flexible; on the other hand, its software design is not limited by platform, making it more widely applicable. Therefore, SR reconstruction technology has enormous potential in practical applications, such as public safety, medical diagnosis, remote sensing observation, and video / image quality enhancement.

[0004] In recent years, with the explosive progress in research on deep convolutional neural networks (CNNs), deep learning-based methods have attracted widespread attention due to their excellent learning capabilities and real-time processing abilities. Although CNN-based networks have achieved great success in image reconstruction (SR) tasks, their feature representation capabilities remain limited due to their receptive field. While increasing network depth can improve the performance of image SR reconstruction, it also increases the complexity of the network structure. Excessive parameters require greater computational resources, making it difficult for these methods to be widely adopted in practical applications. Furthermore, increasing the network size can also lead to problems such as overfitting. Summary of the Invention

[0005] This invention addresses the problem that existing super-resolution (SR) reconstruction techniques require greater computational resources to improve reconstruction performance, by providing an image super-resolution reconstruction method.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] An image super-resolution reconstruction method includes the following steps:

[0008] S1: Extract shallow features from the low-resolution image to be reconstructed;

[0009] S2: Input the extracted shallow features into the hybrid encoder and decoder to transform them into deep features;

[0010] The hybrid codec has a U-shaped network structure, including multiple MSACM modules, each of which includes several hybrid attention modules. The hybrid attention modules simultaneously model global context information and local texture information through a parallel dual-branch structure including multi-head self-attention branches and spatial Fourier frequency module branches. The spatial Fourier frequency module introduces frequency domain characteristics through fast Fourier transform and inverse Fourier transform.

[0011] S3: High-resolution images are obtained by reconstructing deep features using the CARAFE upsampling operator.

[0012] In the above scheme, a hybrid attention module (HAB) is used as the backbone in the U-shaped hybrid codec. The parallel multi-head self-attention branch and spatial Fourier frequency module branch in the hybrid attention module (HAB) are used to efficiently extract global and local context information of the hierarchy, so as to achieve efficient reconstruction of low-resolution images. The CARAFE upsampling operator is also used to adaptively guide upsampling according to context information, so as to avoid increasing the computational power consumption in the sampling process.

[0013] Preferably, shallow features F are extracted from the low-resolution image using 3×3 convolution and the ReLU nonlinear activation function. e The expression is:

[0014]

[0015] Where σ(·) represents the ReLU nonlinear activation function; Represents a 3×3 convolution; I l This indicates a low-resolution image.

[0016] Preferably, the hybrid codec includes an encoder end and a decoder end, wherein,

[0017] At the encoder end, adjacent MSACM modules are downsampled using 4×4 convolutions with a stride of 2; at the decoder end, adjacent MSACM modules are upsampled using 3×3 convolutions and pixel shuffle operations; the features output by the MSACM modules at the encoder end are transmitted to the MSACM modules at the decoder end through skip connections.

[0018] Preferably, the multi-head self-attention branch first encodes local information through three branches to generate query tensor features, key tensor features, and value tensor features respectively. Then, the dimension of the tensor features is transformed through a dimension transformation operation. The query tensor features and key tensor features are multiplied by matrix multiplication and the Softmax function to generate a transposed attention feature matrix. The value tensor features are then multiplied by the attention feature matrix and subjected to dimension transformation and convolution to obtain the output features of the multi-head self-attention branch.

[0019] The process of multi-head self-attention branching is shown in the following equation:

[0020]

[0021] in, Indicates dimensional reshaping; LN(·) represents a 3×3 depth-separable convolution operation; LN(·) represents a layer normalization operation. F represents the matrix multiplication operation; mDTA This represents the output feature of the multi-head self-attention branch; Softmax(·) represents the Softmax function; Attention(·) represents the self-attention operation; Represent the features of the query tensor; Represent the key tensor features; This represents the features of the value tensor.

[0022] Preferably, the spatial Fourier frequency module branch includes a spatial domain branch and a frequency domain branch, wherein,

[0023] In the spatial domain branch, the features flowing into the spatial domain branch are subjected to a first convolution, a PReLU activation function, and a second convolution, and then added to the original features flowing into the spatial domain branch at the channel level to obtain the output features of the spatial domain branch.

[0024] In the frequency domain branch, the features flowing into the frequency domain branch undergo a first convolution with PReLU activation function, a two-dimensional fast Fourier transform, a second convolution with PReLU activation function, and an inverse two-dimensional fast Fourier transform. These features are then added at the channel level to the original features flowing into the frequency domain branch, and finally integrated through convolution to obtain the output features of the frequency domain branch.

[0025] Finally, the output features of the spatial domain branch and the frequency domain branch are concatenated by the concat operation and then convolved to obtain the output features of the spatial Fourier frequency module branch.

[0026] Preferably, the output features of the multi-head self-attention branch and the output features of the spatial Fourier frequency module branch are fused through an adaptive fusion module. The adaptive fusion module generates two learnable weight tensors and performs channel-level multiplication with the output features of the multi-head self-attention branch and the output features of the spatial Fourier frequency module branch, respectively. Then, the fused features are integrated through a 1×1 convolution.

[0027] In the feedforward module of the hybrid attention module, a deep gated feedforward network is used to further learn and transmit features to obtain the output features F of the hybrid attention module. HAB ;

[0028] The deep-gated feedforward network includes layer normalization, convolution, ReLU activation function, element-wise multiplication, and channel addition operations. Features flowing into the deep-gated feedforward network are first normalized, then flow into two parallel branches for convolution and depthwise convolution, doubling the number of feature channels. Features from one branch are first activated by ReLU, then multiplied element-wise with features from the other branch to suppress irrelevant information, then convolved with 1×1 and finally added to the original features flowing into the deep-gated feedforward network at the channel level. The entire process is shown in the following equation:

[0029]

[0030] in, Representation layer normalization; Indicates a channel-level addition operation; f SFB This represents the operation of branching the spatial Fourier frequency module; f MDTA This indicates the operation of multi-head self-attention branching; f AFM Indicates the operation of the adaptive fusion module; f GDFN This represents the operation of a depth-gated feedforward network; F represents a 1×1 convolution; s This indicates that the features output by the adaptive fusion module are convolved with F after a 1×1 convolution. e Features obtained by channel-level addition.

[0031] Preferably, the output characteristic F of the MSACM module is obtained through the following process. MSACM :

[0032]

[0033] in, This represents the operation of the first hybrid attention module. This represents the operation of the second hybrid attention module. This represents the operation of the nth hybrid attention module.

[0034] Preferably, the specific process of step S3 is as follows:

[0035] The last MSACM module of the hybrid codec outputs deep features F. d F d The upsampled feature F is obtained by using the CARAFE upsampling operator and 3x3 convolution. t F t The high-resolution image I is obtained by adding the dimensions of the upsampled low-resolution image. h The expression is as follows:

[0036]

[0037] in, Represents a 3×3 convolution operation; CARAFE represents the CARAFE upsampling operator; This indicates a dimension addition operation.

[0038] Preferably, it also includes a total loss function that takes into account human visual perception and image quality evaluation metrics:

[0039] L total =PSNR(I h ,I g )+λL Charbonnier (I h ,I g )

[0040] Loss function L Charbonnier As shown in the following formula:

[0041]

[0042] Where PSNR(·) represents the PSNR loss function; I h ,I g Let represent the super-resolution reconstructed image and the original high-resolution image, respectively; λ and ∈ represent different constants.

[0043] Preferably, λ is 0.05 and ∈ is 0.001.

[0044] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0045] This invention provides an image super-resolution reconstruction method. It uses a hybrid attention module as the backbone in a U-shaped hybrid codec. It leverages the ability of the parallel multi-head self-attention branch and spatial Fourier frequency module branch in the hybrid attention module to efficiently extract global and local contextual information at different levels, thereby achieving efficient reconstruction of low-resolution images. It also employs the CARAFE upsampling operator to adaptively guide upsampling based on contextual information, avoiding increased computational consumption during the sampling process. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating the implementation steps of the technical solution of the present invention;

[0047] Figure 2 This is a schematic diagram of the overall framework of the present invention;

[0048] Figure 3 This is a schematic diagram of the MSACM module framework in this invention;

[0049] Figure 4 This is a schematic diagram of the framework of the hybrid attention module in this invention;

[0050] Figure 5 This is a schematic diagram of the MDTA branch framework in this invention;

[0051] Figure 6 This is a schematic diagram of the SFB branch framework in this invention. Detailed Implementation

[0052] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0053] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions;

[0054] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0055] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0056] Example 1

[0057] like Figure 1-4 As shown, an image super-resolution reconstruction method includes the following steps:

[0058] S1: Extract shallow features from the low-resolution image to be reconstructed;

[0059] S2: Input the extracted shallow features into the hybrid encoder and decoder to transform them into deep features;

[0060] The hybrid codec has a U-shaped network structure, including multiple MSACM modules, each of which includes several hybrid attention modules (HAB). The hybrid attention modules (HAB) model global context information and local texture information simultaneously through a parallel dual-branch structure including a multi-head self-attention (MDTA) branch and a spatial Fourier frequency module (SFB) branch. The spatial Fourier frequency module introduces frequency domain characteristics through fast Fourier transform and inverse Fourier transform.

[0061] S3: High-resolution images are obtained by reconstructing deep features using the CARAFE upsampling operator.

[0062] In the specific implementation process, HAB is used as the backbone in the U-shaped hybrid codec. The ability of parallel MDTA and SFB branches in HAB to efficiently extract global and local context information at the hierarchical level is utilized to achieve efficient reconstruction of low-resolution images. The CARAFE upsampling operator is also used to adaptively guide upsampling based on context information to avoid increasing the computational power consumption during the sampling process.

[0063] Example 2

[0064] An image super-resolution reconstruction method includes the following steps:

[0065] S1: Extract shallow features from the low-resolution image to be reconstructed;

[0066] More specifically, shallow features F are extracted from low-resolution images using 3×3 convolution and the ReLU nonlinear activation function. e The expression is:

[0067]

[0068] Where σ(·) represents the ReLU nonlinear activation function; Represents a 3×3 convolution; I l This indicates a low-resolution image.

[0069] S2: Input the extracted shallow features into the hybrid encoder and decoder to transform them into deep features;

[0070] The hybrid codec has a U-shaped network structure, including multiple MSACM modules, each of which includes several hybrid attention modules (HAB). The hybrid attention modules (HAB) model global context information and local texture information simultaneously through a parallel dual-branch structure including a multi-head self-attention (MDTA) branch and a spatial Fourier frequency module (SFB) branch. The spatial Fourier frequency module introduces frequency domain characteristics through fast Fourier transform and inverse Fourier transform.

[0071] More specifically, the hybrid codec includes an encoder end and a decoder end, wherein,

[0072] At the encoder end, adjacent MSACM modules are downsampled using 4×4 convolutions with a stride of 2; at the decoder end, adjacent MSACM modules are upsampled using 3×3 convolutions and pixel shuffle operations; the features output by the MSACM modules at the encoder end are transmitted to the MSACM modules at the decoder end through skip connections.

[0073] The process of transforming shallow features into deep features is as follows:

[0074] F d =F u (F e )

[0075] F u (·) indicates the processing procedure of the hybrid codec.

[0076] More specifically, such as Figure 5 As shown, the Multi-Head Self-Attention (MDTA) branch first encodes local information by performing 1×1 convolution and 3×3 depthwise separable convolution operations through three branches, generating query tensor features, key tensor features, and value tensor features respectively. Then, the dimensions of the tensor features are transformed through a dimension transformation operation. The query tensor features and key tensor features are multiplied by matrix multiplication and the Softmax function to generate a transposed attention feature matrix. The value tensor features are then multiplied by the attention feature matrix and subjected to dimension transformation and 1×1 convolution to obtain the output features of the Multi-Head Self-Attention (MDTA) branch.

[0077] Given First, layer normalization is performed, and then 1×1 convolutions and 3×3 depthwise convolutions are used in the MDTA module of the multi-head self-attention branch to encode the channel-level spatial context. Next, dot product multiplication is performed on the query and key projection to reconstruct a transposed attention graph.

[0078] The process of multi-head self-attention (MDTA) branching is shown in the following equation:

[0079]

[0080] in, Indicates dimensional reshaping; LN(·) represents a 3×3 depth-separable convolution operation; LN(·) represents a layer normalization operation. F represents the matrix multiplication operation; MDTA This represents the output feature of the Multi-Head Self-Attention (MDTA) branch; Softmax(·) represents the Softmax function; Attention(·) represents the self-attention operation; Represent the features of the query tensor; Represent the key tensor features; Represents the features of the value tensor;

[0081] In practice, the MDTA branch models the global context across channels, and its complexity is linear with that of the input image.

[0082] More specifically, such as Figure 6 As shown, the Spatial Fourier Frequency Module (SFB) branch is based on Fast Fourier Convolution (FFC) and includes a spatial domain branch and a frequency domain branch, wherein...

[0083] In the spatial domain branch, the features flowing into the spatial domain branch are subjected to a channel-level addition operation after the first 3×3 convolution, the PReLU activation function, and the second 3×3 convolution, and then the original features flowing into the spatial domain branch are added to obtain the output features of the spatial domain branch.

[0084] In the frequency domain branch, the features flowing into the frequency domain branch are subjected to a first 3×3 convolution with PReLU activation function, a two-dimensional fast Fourier transform, a second 3×3 convolution with PReLU activation function, and an inverse two-dimensional fast Fourier transform. These features are then added at the channel level to the original features flowing into the frequency domain branch, and finally integrated through a 1×1 convolution to obtain the output features of the frequency domain branch.

[0085] Finally, the output features of the spatial domain branch and the frequency domain branch are concatenated by the concat operation, and then subjected to a 1×1 convolution to obtain the output features of the spatial Fourier frequency module (SFB) branch.

[0086] S3: High-resolution images are obtained by reconstructing deep features using the CARAFE upsampling operator.

[0087] More specifically, the specific process of step S3 is as follows:

[0088] The last MSACM module of the hybrid codec outputs deep features F. d F d The upsampled feature F is obtained by using the CARAFE upsampling operator and 3x3 convolution. t F t The high-resolution image I is obtained by adding the dimensions of the upsampled low-resolution image. h The expression is as follows:

[0089]

[0090] in, Represents a 3×3 convolution operation; CARAFE represents the CARAFE upsampling operator; This indicates a dimension addition operation.

[0091] Example 3

[0092] An image super-resolution reconstruction method includes the following steps:

[0093] S1: Extract shallow features from the low-resolution image to be reconstructed;

[0094] More specifically, shallow features F are extracted from low-resolution images using 3×3 convolution and the ReLU nonlinear activation function. e The expression is:

[0095]

[0096] Where σ(·) represents the ReLU nonlinear activation function; Represents a 3×3 convolution; I l This indicates a low-resolution image.

[0097] S2: Input the extracted shallow features into the hybrid encoder and decoder to transform them into deep features;

[0098] The hybrid codec has a U-shaped network structure, including multiple MSACM modules, each of which includes several hybrid attention modules (HAB). The hybrid attention modules (HAB) model global context information and local texture information simultaneously through a parallel dual-branch structure including a multi-head self-attention (MDTA) branch and a spatial Fourier frequency module (SFB) branch. The spatial Fourier frequency module introduces frequency domain characteristics through fast Fourier transform and inverse Fourier transform.

[0099] More specifically, the hybrid codec includes an encoder end and a decoder end, wherein,

[0100] At the encoder end, adjacent MSACM modules are downsampled using 4×4 convolutions with a stride of 2; at the decoder end, adjacent MSACM modules are upsampled using 3×3 convolutions and pixel shuffle operations; the features output by the MSACM modules at the encoder end are transmitted to the MSACM modules at the decoder end through skip connections.

[0101] The process of transforming shallow features into deep features is as follows:

[0102] F d =F u (F e )

[0103] F u (·) indicates the processing procedure of the hybrid codec.

[0104] More specifically, the Multi-Head Self-Attention (MDTA) branch models the global context across channels, with a complexity linear with the input image. The MDTA branch first encodes local information through 1×1 convolutions and 3×3 depthwise separable convolutions in three branches, generating query tensor features, key tensor features, and value tensor features respectively. Then, the tensor features' dimensions are transformed through a dimension transformation operation. The query and key tensor features are multiplied by matrix multiplication and the Softmax function to generate a transposed attention feature matrix. The value tensor feature is then multiplied by the attention feature matrix and subjected to dimension transformation and a 1×1 convolution to obtain the output feature of the MDTA branch.

[0105] Given First, layer normalization is performed, and then 1×1 convolutions and 3×3 depthwise convolutions are used in the MDTA module of the multi-head self-attention branch to encode the channel-level spatial context. Next, dot product multiplication is performed on the query and key projection to reconstruct a transposed attention graph.

[0106] The process of multi-head self-attention (MDTA) branching is shown in the following equation:

[0107]

[0108] in, Indicates dimensional reshaping; LN(·) represents a 3×3 depth-separable convolution operation; LN(·) represents a layer normalization operation. F represents the matrix multiplication operation; MDTA This represents the output feature of the Multi-Head Self-Attention (MDTA) branch; Softmax(·) represents the Softmax function; Attention(·) represents the self-attention operation; Represent the features of the query tensor; Represent the key tensor features; Represents the features of the value tensor;

[0109] More specifically, the Spatial Fourier Frequency Module (SFB) branch is based on Fast Fourier Convolution (FFC) and includes a spatial domain branch and a frequency domain branch, wherein,

[0110] In the spatial domain branch, the features flowing into the spatial domain branch are subjected to a channel-level addition operation after the first 3×3 convolution, the PReLU activation function, and the second 3×3 convolution, and then the original features flowing into the spatial domain branch are added to obtain the output features of the spatial domain branch.

[0111] In the frequency domain branch, the features flowing into the frequency domain branch are subjected to a first 3×3 convolution with PReLU activation function, a two-dimensional fast Fourier transform, a second 3×3 convolution with PReLU activation function, and an inverse two-dimensional fast Fourier transform. These features are then added at the channel level to the original features flowing into the frequency domain branch, and finally integrated through a 1×1 convolution to obtain the output features of the frequency domain branch.

[0112] Finally, the output features of the spatial domain branch and the frequency domain branch are concatenated by the concat operation, and then subjected to a 1×1 convolution to obtain the output features of the spatial Fourier frequency module (SFB) branch.

[0113] In the specific implementation process, the Spatial Fourier Frequency Module (SFB) branch uses Fast Fourier Convolution to extract global information in the frequency domain branch, and then uses the hourglass-based residual module in the spatial domain branch to enhance local feature representation and help the network extract useful information in the image and learn more discriminative feature representations.

[0114] More specifically, shallow features F e First, layer normalization is performed. Then, a parallel dual-branch structure consisting of MDTA and SFB branches is used. The output features of the Multi-Head Self-Attention (MDTA) branch and the output features of the Spatial Fourier Frequency Module (SFB) branch are fused through an Adaptive Fusion Module (AFM). The AFM generates two learnable weight tensors and performs channel-level multiplication with the output features of the MDTA and SFB branches, respectively. Finally, a 1×1 convolution is used to integrate the fused features.

[0115] In the feedforward module of the Hybrid Attention Module (HAB), a Deep Gated Forward Network (GDFN) is used to further learn and transmit features to obtain the output features F of the HAB. HAB ;

[0116] The deep gated feedforward network (GDFN) includes layer normalization, 1×1 convolution, ReLU activation function, element-wise multiplication, and channel addition. Features flowing into the GDFN are first normalized, then fed into two parallel branches for 1×1 and 3×3 depthwise convolutions, doubling the number of feature channels. Features from one branch are first activated by ReLU, then multiplied element-wise with features from the other branch to suppress irrelevant information, followed by a 1×1 convolution and channel-wise addition with the original features flowing into the GDFN. The entire process is illustrated in the following equation:

[0117]

[0118] in, Representation layer normalization; Indicates a channel-level addition operation; f SFB This indicates the operation of the Spatial Fourier Frequency Module (SFB) branch; f MDTA This represents the operation of the multi-head self-attention (MDTA) branch; f AFM Indicates the operation of the Adaptive Fusion Module (AFM); f GDFN This represents the operation of a depth-gated feedforward network (GDFN); F represents a 1×1 convolution; s This indicates that the features output by the Adaptive Fusion Module (AFM) are convolved with F after a 1×1 convolution. e Features obtained by channel-level addition.

[0119] More specifically, the output feature F of the MSACM module is obtained through the following process. MSACM :

[0120]

[0121] in, This represents the operation of the first Hybrid Attention Module (HAB). This represents the operation of the second Hybrid Attention Module (HAB). This represents the operation of the nth Hybrid Attention Module (HAB).

[0122] S3: High-resolution images are obtained by reconstructing deep features using the CARAFE upsampling operator.

[0123] More specifically, the specific process of step S3 is as follows:

[0124] The last MSACM module of the hybrid codec outputs deep features F. d F d The upsampled feature F is obtained by using the CARAFE upsampling operator and 3x3 convolution. t F t The high-resolution image I is obtained by adding the dimensions of the upsampled low-resolution image. h The expression is as follows:

[0125]

[0126] in, Represents a 3×3 convolution operation; CARAFE represents the CARAFE upsampling operator; This indicates a dimension addition operation.

[0127] In the specific implementation process, the CARAFE upsampling operator is used to replace the upsampling operation of the traditional super-resolution network. This operator can adaptively guide upsampling based on context information, and the computational power consumption during the sampling process is comparable to that of the deconvolution operation.

[0128] More specifically, this also includes introducing a total loss function that considers human visual perception and image quality evaluation metrics:

[0129] L total =PSNR(I h ,I g )+λL Charbonnier (I h ,I g )

[0130] Loss function L Charbonnier As shown in the following formula:

[0131]

[0132] Where PSNR(·) represents the PSNR loss function; I h ,I g Let represent the super-resolution reconstructed image and the original high-resolution image, respectively; λ and ∈ represent different constants.

[0133] More specifically, λ is 0.05 and ∈ is 0.001.

[0134] In practice, the constants λ and ∈ can be set based on experience.

[0135] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. An image super-resolution reconstruction method, characterized in that, Includes the following steps: S1: Extract shallow features from the low-resolution image to be reconstructed; S2: Input the extracted shallow features into the hybrid encoder and decoder to transform them into deep features; The hybrid codec has a U-shaped network structure, including multiple MSACM modules, each of which includes several hybrid attention modules. The hybrid attention modules simultaneously model global context information and local texture information through a parallel dual-branch structure including a multi-head self-attention branch and a spatial Fourier frequency module branch. The spatial Fourier frequency module introduces frequency domain characteristics through fast Fourier transform and inverse Fourier transform. The output features of the multi-head self-attention branch and the spatial Fourier frequency module branch are fused through an adaptive fusion module. The adaptive fusion module generates two learnable weight tensors and performs channel-level multiplication with the output features of the multi-head self-attention branch and the spatial Fourier frequency module branch, respectively. Then, the fused features are integrated through a 1×1 convolution. In the feedforward module of the hybrid attention module, a deep gated feedforward network is used to learn and transmit features to obtain the output features of the hybrid attention module. ; The deep-gated feedforward network includes layer normalization, convolution, ReLU activation function, element-wise multiplication, and channel addition operations. Features flowing into the deep-gated feedforward network are first normalized, then fed into two parallel branches for convolution and depthwise convolution, doubling the number of feature channels. Features from one branch are first activated by ReLU, then multiplied element-wise with features from the other branch to suppress irrelevant information, followed by a 1×1 convolution and channel-wise addition with the original features flowing into the deep-gated feedforward network. S3: High-resolution images are reconstructed based on deep features using the CARAFE upsampling operator; The specific process of step S3 is as follows: The last MSACM module of the hybrid codec outputs deep features. , Upsampled features are obtained using the CARAFE upsampling operator and 3x3 convolution. , The high-resolution image is obtained by adding the dimensions of the upsampled low-resolution image to the high-resolution image. .

2. The image super-resolution reconstruction method according to claim 1, characterized in that, Shallow features are extracted from low-resolution images using 3×3 convolution and the ReLU nonlinear activation function. The expression is: in, (•) represents the ReLU nonlinear activation function; (•) indicates a 3×3 convolution; This indicates a low-resolution image.

3. The image super-resolution reconstruction method according to claim 1, characterized in that, The hybrid codec includes an encoder end and a decoder end, wherein, At the encoder end, adjacent MSACM modules are downsampled using 4×4 convolutions with a stride of 2; at the decoder end, adjacent MSACM modules are upsampled using 3×3 convolutions and pixel shuffle operations; the features output by the MSACM modules at the encoder end are transmitted to the MSACM modules at the decoder end through skip connections.

4. The image super-resolution reconstruction method according to claim 1, characterized in that, The multi-head self-attention branch first encodes local information through three branches to generate query tensor features, key tensor features, and value tensor features respectively. Then, the dimension of the tensor features is transformed through a dimension transformation operation. The query tensor features and key tensor features are multiplied by matrix multiplication and the Softmax function to generate a transposed attention feature matrix. The value tensor features are then multiplied by the attention feature matrix and subjected to dimension transformation and convolution to obtain the output features of the multi-head self-attention branch. The process of multi-head self-attention branching is shown in the following equation: in, (•) indicates dimensional reshaping; (•) indicates a 3×3 depth separable convolution operation; (•) indicates layer normalization operation; Represents matrix multiplication operation; This represents the output features of a multi-head self-attention branch; This represents the Softmax function; This indicates a self-attention operation; Represent the features of the query tensor; Represent the key tensor features; Represents the features of the value tensor. This represents a 1×1 convolution.

5. The image super-resolution reconstruction method according to claim 1, characterized in that, The spatial Fourier frequency module branch includes a spatial domain branch and a frequency domain branch, wherein... In the spatial domain branch, the features flowing into the spatial domain branch are subjected to a first convolution, a PReLU activation function, and a second convolution, and then added to the original features flowing into the spatial domain branch at the channel level to obtain the output features of the spatial domain branch. In the frequency domain branch, the features flowing into the frequency domain branch undergo a first convolution with PReLU activation function, a two-dimensional fast Fourier transform, a second convolution with PReLU activation function, and an inverse two-dimensional fast Fourier transform. These features are then added at the channel level to the original features flowing into the frequency domain branch, and finally integrated through convolution to obtain the output features of the frequency domain branch. Finally, the output features of the spatial domain branch and the frequency domain branch are concatenated by the concat operation and then convolved to obtain the output features of the spatial Fourier frequency module branch.

6. The image super-resolution reconstruction method according to claim 1, characterized in that, The specific process of step S2 is shown in the following formula: in, Representation layer normalization; This indicates a channel-level addition operation; This indicates the operation of branching the spatial Fourier frequency module; This represents the operation of multi-head self-attention branches; This indicates the operation of the adaptive fusion module; This represents the operation of a depth-gated feedforward network; Represents a 1×1 convolution; This indicates that the features output by the adaptive fusion module are convolved with a 1×1 convolution and then... Features obtained by channel-level addition.

7. The image super-resolution reconstruction method according to claim 6, characterized in that, The output characteristics of the MSACM module are obtained through the following process. : in, This represents the operation of the first hybrid attention module. This represents the operation of the second hybrid attention module. This represents the operation of the nth hybrid attention module.

8. The image super-resolution reconstruction method according to claim 1, characterized in that, The specific process expression for step S3 is as follows: in, This represents a 3×3 convolution operation; Represents the CARAFE upsampling operator; This represents the dimension addition operation. This indicates a low-resolution image.

9. The image super-resolution reconstruction method according to claim 1, characterized in that, It also includes introducing a total loss function that considers human visual perception and image quality evaluation metrics: loss function As shown in the following formula: in, Represents the PSNR loss function; These represent the super-resolution reconstructed image and the original high-resolution image, respectively. , These represent different constants.

10. The image super-resolution reconstruction method according to claim 9, characterized in that, It is 0.

05. It is 0.001.

Images