Image super-resolution reconstruction method and related apparatus

Abstract

This invention provides an image super-resolution reconstruction method and related apparatus. The method includes: acquiring a low-resolution image to be reconstructed; inputting the low-resolution image into a trained super-resolution reconstruction model MFHFNet to obtain a super-resolution reconstructed image, wherein the MFHFNet consists of a shallow feature extraction layer, a deep feature extraction layer, and an image reconstruction layer, with the residual of the deep feature extraction layer connected to the output of the shallow feature extraction layer; the shallow feature extraction layer performs shallow feature extraction on the low-resolution image to obtain a shallow feature map; the deep feature extraction layer performs deep feature extraction on the shallow feature map to obtain a deep feature map; and the image reconstruction layer performs image reconstruction on the deep feature map to obtain the reconstructed super-resolution reconstructed image. Based on this, this invention can improve the quality of image super-resolution reconstruction.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an image super-resolution reconstruction method and related apparatus. Background Technology

[0002] Image super-resolution (SR) is a classic problem in computer vision and image processing, aiming to reconstruct a high-resolution image from a given low-resolution input image. With the development of deep learning, image super-resolution has made significant progress in computer vision. Image super-resolution methods based on convolutional neural networks (CNNs) have become dominant in this field due to their ability to efficiently represent local details of images through local receptive fields.

[0003] Images are typically rich in information about regions of various sizes. However, existing Transformer-based image super-resolution methods cannot accurately represent features at different scales, and there are regional information conflicts in the fusion of features at different scales. They cannot effectively distinguish the differences between different regions, resulting in artifacts and loss of detail, which affects the quality of image super-resolution reconstruction. Summary of the Invention

[0004] This invention provides an image super-resolution reconstruction method and related apparatus that can accurately represent features at different scales, avoid information conflicts caused by direct fusion of multi-scale features, effectively distinguish the differences in feature maps after fusion at different scales, retain multi-scale fusion features while understanding feature differences at different scales, avoid artifacts and loss of detail, thereby improving the quality of image super-resolution reconstruction.

[0005] In a first aspect, embodiments of the present invention provide an image super-resolution reconstruction method, comprising:

[0006] Obtain the low-resolution image to be reconstructed; The low-resolution image is input into a trained super-resolution reconstruction model MFHFNet to obtain a super-resolution reconstructed image. MFHFNet consists of a shallow feature extraction layer, a deep feature extraction layer, and an image reconstruction layer. The residual of the deep feature extraction layer is connected to the output of the shallow feature extraction layer. The shallow feature extraction layer performs shallow feature extraction on the low-resolution image to obtain a shallow feature map. The deep feature extraction layer performs deep feature extraction on the shallow feature map to obtain a deep feature map. The image reconstruction layer performs image reconstruction on the deep feature map to obtain the reconstructed super-resolution image. The deep feature extraction layer includes a hybrid feature extraction block and a feature enhancement attention block. The hybrid feature extraction block uses a multi-scale feature fusion block and frequency domain self-attention for deep feature extraction. The feature enhancement attention block performs attention feature calculation through addition and subtraction operations on the shallow feature maps.

[0007] In some embodiments, the shallow feature extraction layer is a 3×3 convolutional layer; the deep feature extraction layer consists of n sequentially connected cascaded hybrid feature extraction groups, a 3×3 convolutional layer, and a global residual connection module; the image reconstruction layer consists of a 3×3 convolutional layer, a pixel rearrangement operation, and a 3×3 convolutional layer.

[0008] In some embodiments, the hybrid feature extraction group consists of a plurality of cascaded hybrid feature extraction blocks, a feature enhancement attention block, a 3×3 convolutional layer, and a global residual connection module. The input-output expressions of the hybrid feature extraction group are as follows:

[0009] in, This is the input to the hybrid feature extraction group. The output of the hybrid feature extraction group, Enhance the attention block for the aforementioned feature. This refers to the hybrid feature extraction block.

[0010] In some embodiments, the hybrid feature extraction block is composed of a layer normalization, a pair of parallel frequency domain self-attention and multi-scale feature progressive fusion blocks, a layer normalization, and a multilayer perceptron connected sequentially. The hybrid feature extraction block performs a matrix element-wise addition operation on the parallel frequency domain self-attention and multi-scale feature progressive fusion blocks, and performs a residual concatenation with the original input of the hybrid feature extraction block to obtain a first intermediate feature. The first intermediate feature is then residually concatenated with the output of the multilayer perceptron to obtain the final output of the hybrid feature extraction block. The input-output expression of the hybrid feature extraction block is as follows:

[0011]

[0012] in, This is the original input for the hybrid feature extraction block. This is the first intermediate feature. This is the final output of the hybrid feature extraction block. For the frequency domain self-attention, This refers to the multi-scale feature progressive fusion block. Normalize the layer. This refers to the multilayer perceptron.

[0013] In some embodiments, the input of the frequency domain self-attention By using a 1×1 convolutional layer, the number of channels C is expanded to 3C, becoming... ,in, For spatial dimensions, The number of channels; then, after channel segmentation, the... Transformed into three independent attention branch query matrices with a total of C channels. Key matrix Value matrix , , and Reconstructed through matrix dimensions, respectively, into , and ; and The spatial domain feature map is transformed into the frequency domain feature map using Fast Fourier Transform (FFT), and then the global self-attention weights are calculated. These global self-attention weights are then transformed from the frequency domain feature map into the spatial domain feature map using Inverse Fast Fourier Transform (IFFT), and finally passed through a Sigmoid activation function layer to obtain the final result. Attention weights in the spatial domain AND-value matrix Perform matrix multiplication to obtain the self-attention feature map. ; Input of frequency domain self-attention With self-attention feature map Perform residual connections, and then perform matrix dimension reconstruction again to obtain the output of the frequency domain self-attention. .

[0014] In some embodiments, the original input of the multi-scale feature progressive fusion block is processed by a coarse feature extraction block to obtain a coarse feature map. The coarse feature extraction block operates in parallel with a left branch and a right branch. The left branch passes through a 1×1 convolutional layer, then performs element-wise multiplication with a global average pooling (GAP) and a sigmoid activation function, followed by a 3×3 convolutional layer. The right branch passes through a 3×3 convolutional layer, a 1×1 convolutional layer, a GELU activation function, and a 3×3 convolutional layer sequentially. The outputs of the left and right branches are concatenated along the channels, and then processed by a 1×1 convolutional layer to restore the number of channels to C, resulting in... The output feature map of the coarse feature extraction block is obtained by multi-scale feature extraction through four parallel independent branches. Each of these branches uses a parallel depthwise separable convolutional layer with dilation rates of 1, 2, 4, and 8 to extract features at different scales. The resulting branch features are then progressively fused using a sliding window multi-head self-attention mechanism. Finally, the progressively fused multi-scale features are residually concatenated with the output feature map of the coarse feature extraction block to output the feature map of the progressively fused multi-scale feature block. The input-output expressions of the progressively fused multi-scale feature block are as follows:

[0015]

[0016]

[0017]

[0018]

[0019]

[0020]

[0021] in, This serves as the input to the multi-scale feature progressive fusion block. The output of the coarse feature extraction block. The output of the multi-scale feature progressive fusion block, This is the output of the left branch of the coarse feature extraction block. This is the output of the right branch of the coarse feature extraction block. For Sigmod activation function, Let Gaussian error be the activation function of the linear unit. For global average pooling, This refers to the multi-head self-attention mechanism for the sliding window. For depthwise separable convolutions with an inflation rate of 1, For depthwise separable convolutions with an inflation rate of 2, For depthwise separable convolutions with an inflation rate of 4, For depthwise separable convolutions with an inflation rate of 8, , , All of these are outputs from a gradual fusion process.

[0022] In some embodiments, the feature enhancement attention block is composed of a layer normalization, a feature enhancement attention, a layer normalization and a multilayer perceptron connected in sequence; The input of the feature-enhanced attention block is residually connected with the output of the feature-enhanced attention block to obtain a second intermediate feature. The second intermediate feature is residually connected with the output of the multilayer perceptron to obtain the output feature map of the feature-enhanced attention block. The input to the feature enhancement attention layer is boosted from the number of channels C to 2C through a 1×1 convolutional layer, and then split into two feature maps with the number of channels C. The two feature maps with the number of channels C are then summed element-wise to obtain the final feature map. , Linear projections are performed on the query matrix Q and key matrix K, respectively. The absolute value of the element-wise subtraction of the two feature maps with C channels is used as the value matrix V. Global self-attention weights are calculated on the query matrix Q and key matrix K, and then multiplied with the value matrix V to obtain the self-attention feature map. The self-attention feature map is then subjected to an element-wise matrix addition operation to obtain the output feature map of the feature-enhanced attention.

[0023] Secondly, embodiments of the present invention also provide an image super-resolution reconstruction apparatus, the apparatus comprising: The acquisition module is used to acquire the low-resolution image to be reconstructed; The reconstruction module is used to input the low-resolution image into a trained super-resolution reconstruction model MFHFNet to obtain a super-resolution reconstructed image. MFHFNet consists of a shallow feature extraction layer, a deep feature extraction layer, and an image reconstruction layer. The residual of the deep feature extraction layer is connected to the output of the shallow feature extraction layer. The shallow feature extraction layer performs shallow feature extraction on the low-resolution image to obtain a shallow feature map. The deep feature extraction layer performs deep feature extraction on the shallow feature map to obtain a deep feature map. The image reconstruction layer performs image reconstruction on the deep feature map to obtain the reconstructed super-resolution image. The deep feature extraction layer has a hybrid feature extraction block and a feature enhancement attention block. The hybrid feature extraction block uses a multi-scale feature fusion block and frequency domain self-attention for deep feature extraction. The feature enhancement attention block performs attention feature calculation through addition and subtraction operations on the shallow feature maps.

[0024] Thirdly, embodiments of the present invention also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the image super-resolution reconstruction method as described in the first aspect.

[0025] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for performing the image super-resolution reconstruction method as described in the first aspect.

[0026] According to embodiments of the present invention, the image super-resolution reconstruction method and related apparatus include: acquiring a low-resolution image to be reconstructed; inputting the low-resolution image into a trained super-resolution reconstruction model MFHFNet to obtain a super-resolution reconstructed image, wherein MFHFNet consists of a shallow feature extraction layer, a deep feature extraction layer, and an image reconstruction layer, with the residual of the deep feature extraction layer connected to the output of the shallow feature extraction layer; the shallow feature extraction layer is used to perform shallow feature extraction on the low-resolution image to obtain a shallow feature map; the deep feature extraction layer is used to perform deep feature extraction on the shallow feature map to obtain a deep feature map; the image reconstruction layer is used to perform image reconstruction on the deep feature map to obtain a reconstructed super-resolution reconstructed image; the deep feature extraction layer includes a hybrid feature extraction block and a feature enhancement attention block, wherein the hybrid feature extraction block is used to perform deep feature extraction using a multi-scale feature progressive fusion block and frequency domain self-attention, and the feature enhancement attention block is used to perform attention feature calculation through addition and subtraction operations of the shallow feature maps. This invention inputs the low-resolution image to be reconstructed into a trained super-resolution reconstruction model MFHFNet to obtain a super-resolution reconstructed image. MFHFNet consists of a shallow feature extraction layer, a deep feature extraction layer, and an image reconstruction layer. The hybrid feature extraction block in MFHFNet uses a spatial domain branch multi-scale feature progressive fusion block and a frequency domain branch frequency domain self-attention for feature extraction, thus accurately representing features at different scales and avoiding information conflicts caused by direct fusion of multi-scale features. The feature enhancement attention block is used to calculate attention features through addition and subtraction operations of shallow feature maps, which can distinguish the difference information in the feature maps after fusion at different scales, effectively understanding the feature differences at different scales while preserving the multi-scale fusion features. Based on this, the embodiments of this invention can accurately represent features at different scales, avoid the information conflicts caused by direct fusion of multi-scale features, effectively distinguish the difference information in the feature maps after fusion at different scales, understand the feature differences at different scales while preserving the multi-scale fusion features, avoid artifacts and loss of detail, thereby improving the quality of image super-resolution reconstruction. Attached Figure Description

[0027] Figure 1 This is a main flowchart of an image super-resolution reconstruction method provided in one embodiment of the present invention; Figure 2 This is a schematic diagram of the overall framework of MFHFNet provided in one embodiment of the present invention; Figure 3 This is a schematic diagram of frequency domain self-attention composition provided in one embodiment of the present invention; Figure 4 This is a schematic diagram of the multi-scale feature progressive fusion block composition provided in one embodiment of the present invention; Figure 5This is a schematic diagram of the feature enhancement attention composition provided in one embodiment of the present invention; Figure 6 This is a visualization comparison result of Urban100 img048 images provided in one embodiment of the present invention; Figure 7 This is a visualization comparison result of Urban100 img095 images provided in one embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of an image super-resolution reconstruction device provided in one embodiment of the present invention; Figure 9 This is a schematic diagram of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0029] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the following drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0030] In this embodiment of the invention, the terms "furthermore," "exemplarily," or "optionally" are used as examples, illustrations, or descriptions and should not be construed as being more preferred or advantageous than other embodiments or designs. The use of the terms "furthermore," "exemplarily," or "optionally" is intended to present the relevant concepts in a specific manner.

[0031] To facilitate a more convenient description of the working principle of the embodiments of the present invention, the following introduction of relevant technical scenarios is given first.

[0032] Image super-resolution (SR) is a classic problem in computer vision and image processing, aiming to reconstruct a high-resolution image from a given low-resolution input image. With the development of deep learning, image super-resolution has made significant progress in computer vision. Image super-resolution methods based on convolutional neural networks (CNNs) have become dominant in this field due to their ability to efficiently represent local details of images through local receptive fields.

[0033] Images are typically rich in information about regions of various sizes. However, existing Transformer-based image super-resolution methods cannot accurately represent features at different scales, and there are regional information conflicts in the fusion of features at different scales. They cannot effectively distinguish the differences between different regions, resulting in artifacts and loss of detail, which affects the quality of image super-resolution reconstruction.

[0034] Based on this, the present invention provides an image super-resolution reconstruction method and related apparatus. The image super-resolution reconstruction method includes: acquiring a low-resolution image to be reconstructed; inputting the low-resolution image into a trained super-resolution reconstruction model MFHFNet to obtain a super-resolution reconstructed image, wherein MFHFNet consists of a shallow feature extraction layer, a deep feature extraction layer, and an image reconstruction layer, with the residual of the deep feature extraction layer connected to the output of the shallow feature extraction layer; the shallow feature extraction layer is used to perform shallow feature extraction on the low-resolution image to obtain a shallow feature map; the deep feature extraction layer is used to perform deep feature extraction on the shallow feature map to obtain a deep feature map; the image reconstruction layer is used to reconstruct the image from the deep feature map to obtain the reconstructed super-resolution reconstructed image; the deep feature extraction layer includes a hybrid feature extraction block and a feature enhancement attention block, wherein the hybrid feature extraction block is used to perform deep feature extraction using a multi-scale feature progressive fusion block and frequency domain self-attention, and the feature enhancement attention block is used to perform attention feature calculation through addition and subtraction operations on the shallow feature maps. This invention inputs the low-resolution image to be reconstructed into a trained super-resolution reconstruction model MFHFNet to obtain a super-resolution reconstructed image. MFHFNet consists of a shallow feature extraction layer, a deep feature extraction layer, and an image reconstruction layer. The hybrid feature extraction block in MFHFNet uses a spatial domain branch multi-scale feature progressive fusion block and a frequency domain branch frequency domain self-attention for feature extraction, thus accurately representing features at different scales and avoiding information conflicts caused by direct fusion of multi-scale features. The feature enhancement attention block is used to calculate attention features through addition and subtraction operations of shallow feature maps, which can distinguish the difference information in the feature maps after fusion at different scales, effectively understanding the feature differences at different scales while preserving the multi-scale fusion features. Based on this, the embodiments of this invention can accurately represent features at different scales, avoid the information conflicts caused by direct fusion of multi-scale features, effectively distinguish the difference information in the feature maps after fusion at different scales, understand the feature differences at different scales while preserving the multi-scale fusion features, avoid artifacts and loss of detail, thereby improving the quality of image super-resolution reconstruction.

[0035] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0036] like Figure 1 As shown, Figure 1This is a flowchart of an image super-resolution reconstruction method provided in an embodiment of the present invention. The image super-resolution reconstruction method may include, but is not limited to, steps S101 to S102.

[0037] Step S101: Obtain the low-resolution image to be reconstructed; Step S102: Input the low-resolution image into the trained super-resolution reconstruction model MFHFNet to obtain the super-resolution reconstructed image. MFHFNet consists of a shallow feature extraction layer, a deep feature extraction layer, and an image reconstruction layer. The residual of the deep feature extraction layer is connected to the output of the shallow feature extraction layer. The shallow feature extraction layer is used to extract shallow features from the low-resolution image to obtain a shallow feature map. The deep feature extraction layer is used to extract deep features from the shallow feature map to obtain a deep feature map. The image reconstruction layer is used to reconstruct the image from the deep feature map to obtain the reconstructed super-resolution image. The deep feature extraction layer contains a hybrid feature extraction block and a feature enhancement attention block. The hybrid feature extraction block is used to extract deep features using a multi-scale feature fusion block and frequency domain self-attention. The feature enhancement attention block is used to calculate attention features through addition and subtraction operations of the shallow feature maps.

[0038] Understandably, the training method for the super-resolution reconstruction model MFHFNet includes: constructing an image training dataset, constructing the super-resolution reconstruction network MFHFNet, training the super-resolution reconstruction network MFHFNet using the image training dataset, and obtaining the trained super-resolution reconstruction model MFHFNet.

[0039] It is understandable that, such as Figure 2 As shown, MFHFNet consists of a shallow feature extraction layer, a deep feature extraction layer, and an image reconstruction layer. The shallow feature extraction layer is a 3×3 convolutional layer. This layer extracts shallow features from the input low-resolution image, resulting in a shallow feature map.

[0040] Understandably, the deep feature extraction layer consists of n cascaded hybrid feature extraction groups, a 3×3 convolutional layer, and a global residual connection module. The deep feature extraction layer's residual connection follows the shallow feature extraction layer. The deep feature extraction layer performs deep feature extraction on the input shallow feature map to obtain the deep feature map. The formula is as follows:

[0041] in, As input to the deep feature extraction layer, This is the output of the deep feature extraction layer. It is a 3×3 convolutional layer.

[0042] Understandably, the image reconstruction layer consists of a 3×3 convolutional layer, a pixel rearrangement operation, and another 3×3 convolutional layer. The image reconstruction layer reconstructs the image from the input deep feature map, obtaining a reconstructed high-resolution image.

[0043] It is understandable that, such as Figure 2 As shown in (a), the hybrid feature extraction group consists of multiple cascaded hybrid feature extraction blocks, a feature enhancement attention block, a 3×3 convolutional layer, and a global residual connection module. The formula is expressed as follows:

[0044] in, As input to the hybrid feature extraction group, The output of the hybrid feature extraction group, Enhance attention blocks for features. This is a mixed feature extraction block.

[0045] It is understandable that, such as Figure 2 As shown in (b), the hybrid feature extraction block consists of a layer normalization, a pair of parallel frequency domain self-attention and multi-scale feature progressive fusion blocks, another layer normalization, and a multilayer perceptron connected sequentially. Layer normalization is used to stabilize the training process and improve the model's generalization ability. The output of the parallel frequency domain self-attention and multi-scale feature progressive fusion blocks is obtained by performing element-wise matrix addition and then residually concatenating it with the original input to obtain intermediate features. The intermediate features are then residually concatenated with the output of the multilayer perceptron to obtain the final output of the hybrid feature extraction block. The formula is expressed as follows:

[0046]

[0047] in, As input to the hybrid feature extraction block, As an intermediate feature, The output of the hybrid feature extraction block, For frequency domain self-attention, For multi-scale feature progressive fusion blocks, For layer normalization, It is a multilayer perceptron.

[0048] It is understandable that, such as Figure 3 As shown, the input of frequency domain self-attention By using a 1×1 convolutional layer, the number of channels C is expanded to 3C, becoming... ,in For spatial dimensions, This represents the number of channels. After channel splitting, the... Transformed into three independent attention branch query matrices with a total of C channels. Key matrix Value matrix . , and Reconstructed through matrix dimensions, respectively, into , and . and The spatial domain feature map is transformed into the frequency domain feature map using Fast Fourier Transform (FFT), and then the global self-attention weights are calculated. The global self-attention weights are then transformed from the frequency domain feature map into the spatial domain feature map using Inverse Fast Fourier Transform (IFFT), and finally passed through a Sigmoid activation function layer to obtain the final value. Attention weights in the spatial domain AND-value matrix Perform matrix multiplication to obtain the self-attention feature map. The input of frequency domain self-attention With feature map Perform residual connections, then perform matrix dimension reconstruction again to obtain the output of frequency domain self-attention. .

[0049] It is understandable that, such as Figure 4 As shown, the original input of the multi-scale feature fusion block is processed by a coarse feature extraction block to obtain a coarse feature map. The coarse feature extraction block operates in parallel with two branches. The left branch passes through a 1×1 convolutional layer, then performs element-wise multiplication with a global average pooling (GAP) and a sigmoid activation function, followed by a 3×3 convolutional layer. The right branch passes through a 3×3 convolutional layer, a 1×1 convolutional layer, a GELU activation function, and a 3×3 convolutional layer sequentially. The outputs of the two branches are concatenated along the channels, and then passed through a 1×1 convolutional layer to restore the number of channels to C, resulting in the output feature map of the coarse feature extraction block. The output feature map of the coarse feature extraction block is then processed by four parallel independent branches for multi-scale feature extraction. These four branches each pass through four parallel depthwise separable convolutional layers with dilation rates of 1, 2, 4, and 8, respectively, to extract features at different scales. These branch features are progressively fused in a hierarchical manner. Specifically, feature maps with adjacent dilation rates are sequentially added and input into a Shifted Window Multi-head Self Attention (SW-MSA) mechanism, achieving progressive feature fusion from local details to long-range dependencies. Finally, the progressively fused multi-scale features are residually concatenated with the output feature map of the coarse feature extraction block, outputting the feature map of the progressively fused multi-scale feature block. The formula is as follows:

[0050]

[0051]

[0052]

[0053]

[0054]

[0055]

[0056] in, As input for the multi-scale feature progressive fusion block, This is the output of the coarse feature extraction block. The output of the multi-scale feature progressive fusion block, This is the output of the left branch of the coarse feature extraction block. This is the output of the right branch of the coarse feature extraction block. For Sigmod activation function, Let Gaussian error be the activation function of the linear unit. For global average pooling, This is a multi-head self-attention mechanism for sliding windows. For depthwise separable convolutions with an inflation rate of 1, , , All of these are outputs from a gradual fusion process.

[0057] The hybrid feature extraction block employs a spatial domain branch multi-scale feature progressive fusion block and a frequency domain branch frequency domain self-attention for feature extraction. The multi-scale feature progressive fusion block first performs coarse feature extraction on the input, initially fusing smooth global and local features to prevent the loss of fine-grained information. Then, it uses four parallel branches, each employing depthwise separable convolutions with different dilation rates to progressively cover dependencies from local to ultra-long distances. SW-MSA is used to progressively fuse features at different scales, avoiding information conflicts caused by direct fusion of multi-scale features. The frequency domain self-attention uses Fast Fourier Transform (FFT) to compute self-attention features in the frequency domain, capturing background consistency and global statistical features of the image, which is crucial for the model to understand complex background information.

[0058] It is understandable that, such as Figure 5As shown, the feature enhancement attention block consists of a layer normalization, a feature enhancement attention layer, another layer normalization layer, and a multilayer perceptron connected sequentially. The input of the feature enhancement attention block is residually concatenated with the output of the feature enhancement attention to obtain intermediate features. The intermediate features are then residually concatenated with the output of the multilayer perceptron to obtain the output feature map of the feature enhancement attention block.

[0059] The input for feature enhancement attention is boosted from C to 2C through a 1×1 convolutional layer. Then, channel segmentation is performed to divide it into two feature maps with C channels each. The two feature maps are then element-wise summed to obtain the final output. . Linear projections are performed on the query matrix Q and key matrix K respectively. The absolute value of the element-wise subtraction between the two feature maps is used as the value matrix V. Global self-attention weights are calculated on the query matrix Q and key matrix K, and then multiplied with the value matrix V to obtain the self-attention feature map. The output feature map of feature enhancement attention is obtained by performing element-wise matrix addition with the self-attention feature map.

[0060] Feature-enhanced attention blocks perform attention feature calculations through feature map addition and subtraction operations, distinguishing the differences in feature maps after fusion at different scales, effectively understanding feature differences at different scales while preserving multi-scale fusion features.

[0061] Based on this, the image super-resolution reconstruction method of the present invention can accurately express features at different scales, avoid the information conflict problem of direct fusion of multi-scale features, effectively distinguish the difference information in the feature maps after fusion at different scales, retain the fusion features at multiple scales while understanding the feature differences at different scales, avoid artifacts and loss of detail, thereby improving the quality of image super-resolution reconstruction.

[0062] Understandably, this invention uses DIV2K as the training dataset and five standard test datasets—Set5, Set14, BSD100, Urban100, and Manga109—to evaluate the model's performance. The performance metrics used to evaluate the model are two widely accepted standards in the field of image super-resolution reconstruction: PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity). PSNR is an objective metric based on pixel differences. It measures image fidelity by calculating the mean square error between the reconstructed image and the original high-resolution image, measured in dB. A higher PSNR value indicates smaller differences between the two images and better reconstruction quality. SSIM is a metric more consistent with human visual perception. It evaluates the similarity between two images from three dimensions: brightness, contrast, and structure. The closer the SSIM value is to 1, the more visually similar the two images are, and the more complete the details and structure of the reconstructed image are preserved. All metrics were calculated by converting images from RGB format to YCbCr format and then performing calculations on the Y channel. All experiments were conducted using the PyTorch framework on an NVIDIA GeForce RTX 3090 GPU.

[0063] For the experimental setup, the number of mixed feature extraction groups was set to 6, with 6 mixed feature extraction blocks in each group. The number of channels was set to 180. The number of attention heads for SW-MSA in the multi-scale feature progressive fusion block was set to 6, and the window size was set to 16. During the training phase, the input low-resolution images were randomly horizontally flipped and rotated for data augmentation, and the training set images were randomly cropped into 64×64 pixel image blocks. Each training batch contained 16 randomly input images, and the Adam optimizer combined with the L1 loss function was used for training optimization. The parameters of the Adam optimizer were configured as follows: =0.9, = 0.99. The initial learning rate was set to 2×10⁻⁴, the minimum learning rate was set to 1×10⁻⁶, and cosine annealing was used for updates. The total number of training iterations was 500,000.

[0064] To fully evaluate the image super-resolution reconstruction performance of the MFHFNet method, it is compared with state-of-the-art methods such as Swin-IR, SwinFIR, HAT, MAT, ADT, and PFT. Table 1 shows a quantitative comparison of the various methods at ×2, ×3, and ×4 magnification factors on the benchmark dataset.

[0065] Table 1. Performance of different methods on different datasets

[0066] As shown in Table 1, the MFHFNet method proposed in this invention achieves superior performance compared to other state-of-the-art methods on all datasets. Specifically, at magnification factors of ×2, ×3, and ×4, the MFHFNet method outperforms the Swin-IR method by an average of 0.554 dB, 0.422 dB, and 0.43 dB, respectively.

[0067] This invention selects the Urban100 dataset with rich detailed textures and compares the visualization results of the MFHFNet method with those of the Swin-IR, SwinFIR, HAT, MAT, ADT and PFT methods at a magnification factor of ×4. Figure 6 The visualization comparison results of the img048 image in the Urban100 dataset are presented. The results show that other methods are poor at recovering information in areas of different sizes, while the MFHFNet method can effectively distinguish the details between power lines and buildings behind them. Figure 7 The visualization comparison of the img095 image from the Urban100 dataset is presented. The results show that the MFHFNet method can reconstruct the square shape of the floor compared to other methods. This is due to the ability of the hybrid feature extraction block to extract multi-scale features and the ability of the feature enhancement attention block to distinguish the differences between different regions.

[0068] This invention also conducted an ablation study on the core modules of the MFHFNet method, namely the hybrid feature extraction block and the feature enhancement attention block, to analyze and evaluate the effectiveness of each proposed module, as shown in Table 2. All ablation experiments were implemented using the DIV2K dataset trained on a ×4 magnification factor for fair comparison. The quantitative ablation results in Table 2 were all measured on the Urban100 dataset.

[0069] Table 2 Ablation experimental results of the MFHFNet method on the Urban100 dataset (×4 magnification factor)

[0070] According to the results in Table 2, the accuracy of the metrics decreased significantly after removing the hybrid feature extraction block or the feature enhancement attention block module, which proves the effectiveness of the feature extraction block and the feature enhancement attention.

[0071] For image super-resolution reconstruction models, this invention provides an image super-resolution reconstruction method based on a multi-scale and frequency domain hybrid feature Transformer. This method constructs MFHFNet, which is trained on a training dataset, and achieves better image reconstruction quality. The MFHFNet method outperforms other methods on all datasets.

[0072] In addition, such as Figure 8 As shown, one embodiment of the present invention also discloses an image super-resolution reconstruction apparatus, the apparatus comprising: The acquisition module 110 is used to acquire the low-resolution image to be reconstructed; The reconstruction module 120 is used to input low-resolution images into the trained super-resolution reconstruction model MFHFNet to obtain super-resolution reconstructed images. MFHFNet consists of a shallow feature extraction layer, a deep feature extraction layer, and an image reconstruction layer. The residual of the deep feature extraction layer is connected to the output of the shallow feature extraction layer. The shallow feature extraction layer is used to extract shallow features from the low-resolution image to obtain a shallow feature map. The deep feature extraction layer is used to extract deep features from the shallow feature map to obtain a deep feature map. The image reconstruction layer is used to reconstruct the image from the deep feature map to obtain the reconstructed super-resolution image. The deep feature extraction layer has a hybrid feature extraction block and a feature enhancement attention block. The hybrid feature extraction block is used to extract deep features using a multi-scale feature progressive fusion block and frequency domain self-attention. The feature enhancement attention block is used to calculate attention features through the addition and subtraction operations of the shallow feature maps.

[0073] The image super-resolution reconstruction apparatus of this invention is used to execute the image super-resolution reconstruction method in the above embodiments. Its specific processing procedure is the same as that of the image super-resolution reconstruction method in the above embodiments, and will not be described in detail here.

[0074] In addition, such as Figure 9 As shown, one embodiment of the present invention also discloses an electronic device, including: at least one processor 210; at least one memory 220 for storing at least one program; when the at least one program is executed by the at least one processor 210, it implements the image super-resolution reconstruction method as described in any of the preceding embodiments.

[0075] In addition, one embodiment of the present invention discloses a computer-readable storage medium storing computer-executable instructions for performing the image super-resolution reconstruction method as described in any of the preceding embodiments.

[0076] The system architecture and application scenarios described in the embodiments of this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of system architecture and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0077] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0078] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0079] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process or execution thread, and components may be located on a single computer or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, or a network, such as the Internet interacting with other systems via signals).

Claims

1. An image super-resolution reconstruction method, comprising: Obtain the low-resolution image to be reconstructed; The low-resolution image is input into a trained super-resolution reconstruction model MFHFNet to obtain a super-resolution reconstructed image. MFHFNet consists of a shallow feature extraction layer, a deep feature extraction layer, and an image reconstruction layer. The residual of the deep feature extraction layer is connected to the output of the shallow feature extraction layer. The shallow feature extraction layer performs shallow feature extraction on the low-resolution image to obtain a shallow feature map. The deep feature extraction layer performs deep feature extraction on the shallow feature map to obtain a deep feature map. The image reconstruction layer performs image reconstruction on the deep feature map to obtain the reconstructed super-resolution image. The deep feature extraction layer includes a hybrid feature extraction block and a feature enhancement attention block. The hybrid feature extraction block uses a multi-scale feature fusion block and frequency domain self-attention for deep feature extraction. The feature enhancement attention block performs attention feature calculation through addition and subtraction operations on the shallow feature maps.

2. The method according to claim 1, characterized in that, The shallow feature extraction layer is a 3×3 convolutional layer; the deep feature extraction layer consists of n sequentially connected cascaded hybrid feature extraction groups, a 3×3 convolutional layer, and a global residual connection module; the image reconstruction layer consists of a 3×3 convolutional layer, a pixel rearrangement operation, and a 3×3 convolutional layer.

3. The method according to claim 2, characterized in that, The hybrid feature extraction group consists of multiple cascaded hybrid feature extraction blocks, a feature enhancement attention block, a 3×3 convolutional layer, and a global residual connection module. The input-output expressions of the hybrid feature extraction group are as follows: in, This is the input to the hybrid feature extraction group. The output of the hybrid feature extraction group, Enhance the attention block for the feature. This refers to the hybrid feature extraction block.

4. The method according to claim 3, characterized in that, The hybrid feature extraction block consists of a layer normalization, a pair of parallel frequency domain self-attention and multi-scale feature progressive fusion blocks, a layer normalization, and a multilayer perceptron connected sequentially. The hybrid feature extraction block performs element-wise matrix addition on the parallel frequency domain self-attention and multi-scale feature progressive fusion blocks, and the resulting output is residually concatenated with the original input of the hybrid feature extraction block to obtain a first intermediate feature. The first intermediate feature is then residually concatenated with the output of the multilayer perceptron to obtain the final output of the hybrid feature extraction block. The input-output expression of the hybrid feature extraction block is as follows: in, This is the original input for the hybrid feature extraction block. This is the first intermediate feature. This is the final output of the hybrid feature extraction block. For the frequency domain self-attention, This refers to the multi-scale feature progressive fusion block. Normalize the layer. This refers to the multilayer perceptron.

5. The method according to claim 4, characterized in that, The input of the frequency domain self-attention By using a 1×1 convolutional layer, the number of channels C is expanded to 3C, becoming... ,in, For spatial dimensions, The number of channels; then, after channel segmentation, the... Transformed into three independent attention branch query matrices with a total of C channels. Key matrix Value matrix , , and Reconstructed through matrix dimensions, respectively, into , and ; and The spatial domain feature map is transformed into the frequency domain feature map using Fast Fourier Transform (FFT), and then the global self-attention weights are calculated. These global self-attention weights are then transformed from the frequency domain feature map into the spatial domain feature map using Inverse Fast Fourier Transform (IFFT), and finally passed through a Sigmoid activation function layer to obtain the final result. Attention weights in the spatial domain AND-value matrix Perform matrix multiplication to obtain the self-attention feature map. ; Input of frequency domain self-attention With self-attention feature map Perform residual connections, and then perform matrix dimension reconstruction again to obtain the output of the frequency domain self-attention. .

6. The method according to claim 4, characterized in that, The original input of the multi-scale feature fusion block is processed by a coarse feature extraction block to obtain a coarse feature map. This coarse feature extraction block operates in parallel with a left branch and a right branch. The left branch passes through a 1×1 convolutional layer, then performs element-wise multiplication with a global average pooling (GAP) and a sigmoid activation function, followed by a 3×3 convolutional layer. The right branch passes through a 3×3 convolutional layer, a 1×1 convolutional layer, a GELU activation function, and a 3×3 convolutional layer. The outputs of the left and right branches are concatenated along the channels, and then processed by a 1×1 convolutional layer to restore the number of channels to C, resulting in the coarse feature map. The output feature map of the coarse feature extraction block is extracted through multi-scale feature extraction via four parallel independent branches. Each of these four branches uses a parallel depthwise separable convolutional layer with dilation rates of 1, 2, 4, and 8 to extract features at different scales. The resulting branch features are then progressively fused using a sliding window multi-head self-attention mechanism. Finally, the progressively fused multi-scale features are residually concatenated with the output feature map of the coarse feature extraction block to output the feature map of the progressively fused multi-scale feature block. The input-output expressions of the progressively fused multi-scale feature block are as follows: in, This serves as the input to the multi-scale feature progressive fusion block. The output of the coarse feature extraction block. The output of the multi-scale feature progressive fusion block, This is the output of the left branch of the coarse feature extraction block. This is the output of the right branch of the coarse feature extraction block. For Sigmod activation function, Let Gaussian error be the activation function of the linear unit. For global average pooling, This refers to the multi-head self-attention mechanism for the sliding window. For depthwise separable convolutions with an inflation rate of 1, For depthwise separable convolutions with an inflation rate of 2, For depthwise separable convolutions with an inflation rate of 4, For depthwise separable convolutions with an inflation rate of 8, , , All of these are outputs from a gradual fusion process.

7. The method according to claim 3, characterized in that, The feature enhancement attention block is composed of a layer normalization, a feature enhancement attention, a layer normalization and a multilayer perceptron connected in sequence; The input of the feature-enhanced attention block is residually connected with the output of the feature-enhanced attention block to obtain a second intermediate feature. The second intermediate feature is residually connected with the output of the multilayer perceptron to obtain the output feature map of the feature-enhanced attention block. The input to the feature enhancement attention layer is boosted from the number of channels C to 2C through a 1×1 convolutional layer, and then split into two feature maps with the number of channels C. The two feature maps with the number of channels C are then summed element-wise to obtain the final feature map. , Linear projections are performed on the query matrix Q and key matrix K, respectively. The absolute value of the element-wise subtraction of the two feature maps with C channels is used as the value matrix V. Global self-attention weights are calculated on the query matrix Q and key matrix K, and then multiplied with the value matrix V to obtain the self-attention feature map. The self-attention feature map is then added element-wise to obtain the output feature map of the feature-enhanced attention.

8. An image super-resolution reconstruction device, characterized in that, The device includes: The acquisition module is used to acquire the low-resolution image to be reconstructed; The reconstruction module is used to input the low-resolution image into a trained super-resolution reconstruction model MFHFNet to obtain a super-resolution reconstructed image. MFHFNet consists of a shallow feature extraction layer, a deep feature extraction layer, and an image reconstruction layer. The residual of the deep feature extraction layer is connected to the output of the shallow feature extraction layer. The shallow feature extraction layer performs shallow feature extraction on the low-resolution image to obtain a shallow feature map. The deep feature extraction layer performs deep feature extraction on the shallow feature map to obtain a deep feature map. The image reconstruction layer performs image reconstruction on the deep feature map to obtain the reconstructed super-resolution image. The deep feature extraction layer has a hybrid feature extraction block and a feature enhancement attention block. The hybrid feature extraction block uses a multi-scale feature fusion block and frequency domain self-attention for deep feature extraction. The feature enhancement attention block performs attention feature calculation through addition and subtraction operations on the shallow feature maps.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the image super-resolution reconstruction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions for performing the image super-resolution reconstruction method as described in any one of claims 1 to 7.

Images