High-quality super-resolution image reconstruction methods, apparatus, devices and storage media

Abstract

This application relates to a high-quality super-resolution image reconstruction method, apparatus, device, and storage medium. The method employs a multi-scale feature extraction mechanism, enabling holistic image analysis from a global perspective. The core of the network architecture is a multi-scale feature extraction module, which consists of a non-traditional convolutional layer, an activation layer, and a multi-feature fusion extraction process. This multi-feature fusion extraction component uses dilated convolution after expanding the receptive field, thereby capturing a wider range of features while maintaining model simplicity. The method utilizes a parallel hybrid attention module to enhance the model's detail reconstruction capabilities. By generating feature maps at different scales and inputting them into the PHA module, the model's ability to reconstruct details is comprehensively enhanced. This strategic integration of multi-scale feature extraction and the parallel hybrid attention mechanism overcomes the limitations of existing methods and promotes the generation of high-fidelity super-resolution images.

Description

Technical Field

[0001] This application relates to the field of image reconstruction technology, and in particular to a high-quality super-resolution image reconstruction method, apparatus, device, and storage medium. Background Technology

[0002] Image super-resolution is a key technique in image processing, aiming to recover the complex details of high-resolution images from low-resolution ones. In the field of image super-resolution, methods can be broadly categorized into two types based on the number of input images: single-image methods and multi-image methods. Single-image techniques primarily employ two strategies: interpolation-based methods and learning-based methods. Interpolation-based methods enhance the resolution of low-resolution images through interpolation, but often result in blurry and distorted output images. Conversely, learning-based methods utilize deep learning models to learn image mapping relationships from large datasets of high-resolution images, then apply these learned relationships to low-resolution inputs to generate sharper and more realistic high-resolution reconstructions. Notable examples of this type include Super-Resolution Convolutional Neural Networks (SRCNN), Efficient Subpixel Convolutional Neural Networks (ESPCN), and Super-Resolution Methods Using Generative Adversarial Networks (SRGAN), which have demonstrated significant performance improvements in advancing image super-resolution technology. In contrast, multi-image methods utilize multiple low-resolution images to generate corresponding high-resolution reconstructed images. Among these methods, motion estimation-based methods recover detailed information in high-resolution reconstruction by utilizing motion data between multiple low-resolution images, while self-similarity-based methods extract detailed information in high-resolution reconstruction by utilizing self-similar patterns in multiple low-resolution images. By integrating information from multiple low-resolution inputs, these methods can generate more accurate and precise high-resolution outputs.

[0003] In recent years, with the popularization of deep learning methods, learning-based single-image techniques have become a powerful means of generating high-fidelity super-resolution images. These methods, through sophisticated deep learning architectures, are able to skillfully capture complex mapping relationships between images. Meanwhile, due to the inherent drawbacks of multi-image methods, especially the high data processing and computational costs, single-image techniques have become the mainstream approach widely adopted in industry and academia.

[0004] In 2014, Dong et al. proposed Super-Resolution Convolutional Neural Networks (SRCNN), a revolutionary contribution that revolutionized the field of Single Image Super-Resolution (SISR) by utilizing a series of convolutional layers, rectified linear units, and upsampling layers to learn the mapping from low-resolution to high-resolution images. SRCNN represented a significant breakthrough in applying convolutional neural network models to SISR tasks. The effectiveness of using convolutional neural network models for image super-resolution reconstruction mainly stems from the introduction of depth and residual connections. Subsequently, numerous scholars proposed representative SISR methods, focusing on leveraging the depth and residual connections of convolutional neural networks to advance the field.

[0005] Despite the significant success of existing single-image super-resolution (SISR) methods, a notable shortcoming remains in their design, particularly in preserving details such as edges and dense textures. This limitation hinders the generation of high-quality super-resolution (SR) images. Attention mechanisms introduced by previous researchers have enabled more efficient prioritization of feature components during training [7]. However, it is worth noting that these mechanisms, including channel attention mechanisms, primarily focus on amplifying or attenuating the influence of individual channels based on relevant information. While these mechanisms have shown potential in improving metrics such as peak signal-to-noise ratio (PSNR), they remain inadequate in handling details of image boundaries and textures. Summary of the Invention

[0006] Therefore, it is necessary to provide a high-quality super-resolution image reconstruction method, apparatus, device, and storage medium to address the aforementioned technical problems.

[0007] A high-quality super-resolution image reconstruction method, the method comprising:

[0008] The input image is fed into the shallow feature extraction module to obtain shallow features. The shallow feature extraction module is used to extract low-frequency information from the image using single-layer convolution and output shallow features.

[0009] Shallow features are input into the deep feature extraction module for fine-grained feature extraction, resulting in deep features. The deep feature extraction module includes multiple residual convolution modules and multiple multi-scale feature extraction modules. The multi-scale feature extraction module performs convolution processing on the input features and interacts with the input features through a local feature fusion mechanism. After processing by the multi-feature fusion extraction module, it performs global feature fusion with the input features to obtain multi-scale fused features. The multi-feature fusion extraction module performs multi-scale image feature extraction through feature extraction branches with different expansion rates, and then performs a fusion weighting operation using the PHA module. The fusion weighting result is then fused with the features input to the multi-feature fusion extraction module to obtain multi-scale fused features.

[0010] Deep features are input into the super-resolution reconstruction module to obtain a high-quality reconstructed image. The super-resolution reconstruction module is used to adjust the deep features with different numbers of convolutional layers and sub-pixel convolutional layers to generate a high-quality reconstructed image.

[0011] In one embodiment, the deep feature extraction module includes four residual convolution modules and five multi-scale feature extraction modules.

[0012] The shallow features are the input to the first multi-scale feature extraction module. The first multi-scale feature extraction module, the first residual convolution module, the second multi-scale feature extraction module, the second residual convolution module, the third multi-scale feature extraction module, the third residual convolution module, the fourth multi-scale feature extraction module, the fourth residual convolution module, and the fifth multi-scale feature extraction module are connected in sequence.

[0013] In one embodiment, the residual convolution module includes eight stacked residual blocks.

[0014] In one embodiment, the multi-scale feature extraction module includes: a convolutional group, a multi-feature fusion extraction module, and a fusion module; in the multi-scale feature extraction module:

[0015] The features input to the multi-scale feature extraction module are processed by convolution groups to obtain convolutional features.

[0016] The convolutional features and the features input to the multi-scale feature extraction module exchange information through a local feature fusion mechanism to obtain locally fused features.

[0017] The local fusion features are input into the multi-feature fusion extraction module to obtain multi-scale fusion features.

[0018] The multi-scale mixed features and the features input to the multi-scale feature extraction module are fused through the fusion module to obtain multi-scale fused features.

[0019] In one embodiment, the fusion module employs a multiplication operation.

[0020] In one embodiment, the multi-feature fusion extraction module includes: four feature extraction branches with different expansion rates and a parallel fusion attention module; each feature extraction branch includes a dilated convolutional layer.

[0021] The local fusion features are input into the multi-feature fusion extraction module to obtain multi-scale fusion features, including:

[0022] The local fusion features are processed using the dilated convolutional layer of the first feature extraction branch to obtain the first-scale image features.

[0023] After fusing the local fusion features and the first-scale image features, the second-scale image features are processed by the dilated convolutional layer of the second feature extraction branch.

[0024] After fusing the local fusion features and the second-scale image features, the third-scale image features are processed by the dilated convolutional layer of the third feature extraction branch.

[0025] After fusing the local fusion features and the third-scale image features, the fourth-scale image features are processed by the dilated convolutional layer of the fourth feature extraction branch.

[0026] The first-scale image features, the second-scale image features, the third-scale image features, and the fourth-scale image features are input into the parallel hybrid attention module to obtain hybrid weighted features.

[0027] By fusing the weighted features with the local fusion features, multi-scale hybrid features are obtained.

[0028] In one embodiment, image features at a first scale, a second scale, a third scale, and a fourth scale are input into a parallel hybrid attention module to obtain hybrid weighted features, including:

[0029] Global feature representations are obtained by applying global average pooling to the image features at the first, second, third, and fourth scales, capturing long-range dependencies in both the horizontal and vertical directions while preserving positional relationships. Then, inter-channel correlations are modeled using a 1×1 convolution operation, and after activation with a sigmoid function, the global features in the horizontal and vertical directions are obtained as follows:

[0030] X ni =sigmiod(Conv(AvgPool) H (F n '1)))

[0031] Y nj =sigmiod(Conv(AvgPool) W (F n '1)))

[0032] Among them, X ni ∈R 1×1×W Y represents the global feature in the horizontal direction corresponding to the image feature at scale i, where i = 1, 2, ..., m. nj ∈R 1×H×1 F represents the global feature in the vertical direction corresponding to the image feature at scale j, where j = 1, 2, ..., m; n ' iLet m be the image feature at scale i, and m be the number of scales for image features at different scales.

[0033] The global features in the horizontal and vertical directions are concatenated in the second channel dimension and then activated by the Softmax function in the channel dimension to obtain the multi-scale channel feature weights in the horizontal and vertical directions.

[0034] The multi-scale channel feature weights in the horizontal and vertical directions are normalized to obtain normalized weights in the horizontal and vertical directions.

[0035] The first-scale image features, second-scale image features, third-scale image features, and fourth-scale image features are each multiplied element-wise by their corresponding normalized weights, and the product results are summed to obtain the mixed weighted features:

[0036]

[0037] Among them, F n 'W' represents a mixed weighted feature. X [:,i,:]∈R 1×1×W W y [:,j,:]∈R 1×H×1 For the normalized weights in the horizontal and vertical directions; F n For input features.

[0038] A high-quality super-resolution image reconstruction apparatus, the apparatus comprising:

[0039] The shallow feature extraction unit is used to input the input image into the shallow feature extraction module to obtain shallow features; the shallow feature extraction module is used to extract low-frequency information from the image using single-layer convolution and output shallow features.

[0040] The deep feature extraction unit is used to input shallow features into the deep feature extraction module for fine-grained feature extraction, resulting in deep features. The deep feature extraction module includes multiple residual convolution modules and multiple multi-scale feature extraction modules. The multi-scale feature extraction module performs convolution processing on the input features and interacts with the input features through a local feature fusion mechanism. Then, after processing by the multi-feature fusion extraction module, it performs global feature fusion with the input features to obtain multi-scale fused features. The multi-feature fusion extraction module performs multi-scale image feature extraction through feature extraction branches with different expansion rates, and then performs a fusion weighting operation using the PHA module. The fusion weighting result is then fused with the features input to the multi-feature fusion extraction module to obtain multi-scale fused features.

[0041] The image reconstruction unit is used to input deep features into the super-resolution reconstruction module to obtain a high-quality reconstructed image; the super-resolution reconstruction module is used to adjust the deep features using different numbers of convolutional layers and sub-pixel convolutional layers to generate a high-quality reconstructed image.

[0042] A computer device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of any of the methods described above.

[0043] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above methods.

[0044] The aforementioned high-quality super-resolution image reconstruction method, apparatus, device, and storage medium feature a multi-scale feature extraction mechanism that enables holistic image analysis from a global perspective. The core of the network architecture employed in this method is a multi-scale feature extraction module, which consists of a non-traditional convolutional layer, an activation layer, and a multi-feature fusion extraction process. This multi-feature fusion extraction component uses dilated convolution after expanding the receptive field, thereby capturing a wider range of features while maintaining model simplicity. This method utilizes a parallel hybrid attention module to enhance the model's detail reconstruction capabilities; by generating feature maps at different scales and inputting them into the PHA module, the model's ability to reconstruct details is comprehensively enhanced. This strategic integration of multi-scale feature extraction and the parallel hybrid attention mechanism overcomes the limitations of existing methods and promotes the generation of high-fidelity super-resolution images. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating a high-quality super-resolution image reconstruction method in one embodiment;

[0046] Figure 2 This is a schematic diagram of the general structure of a multi-scale hybrid attention (MSHA) model in another embodiment;

[0047] Figure 3 This is a schematic diagram of the composition of the multi-scale feature extraction module (MFB) in another embodiment;

[0048] Figure 4 This is a schematic diagram illustrating the application of Multi Feature Hybrid Extraction (MFHE) in MFB in another embodiment;

[0049] Figure 5 This is a schematic diagram of a parallel hybrid attention (PHA) module in another embodiment;

[0050] Figure 6The following is a schematic diagram of the results of MSHA and state-of-the-art techniques under a ×4 scaling ratio in another embodiment, wherein (a) is a schematic diagram of the image reconstruction effect of different methods in the Set5 dataset, (b) is a schematic diagram of the image reconstruction effect of different methods in the Set14 dataset, (c) is a schematic diagram of the image reconstruction effect of different methods in the B100 dataset, (d) is a schematic diagram of the image reconstruction effect of different methods in the Urban100 dataset, and (e) is a schematic diagram of the image reconstruction effect of different methods in the Manga109 dataset;

[0051] Figure 7 This is a structural block diagram of a high-quality super-resolution image reconstruction apparatus in one embodiment;

[0052] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] In one embodiment, such as Figure 1 As shown, a high-quality super-resolution image reconstruction method is provided, which includes the following steps:

[0055] Step 100: Input the input image into the shallow feature extraction module to obtain shallow features; the shallow feature extraction module is used to extract low-frequency information from the image using single-layer convolution and output shallow features.

[0056] Specifically, in the shallow feature extraction stage, a single-layer convolution is used to directly extract low-frequency information from the input image. This extracted structure serves as the basis for subsequent deep feature extraction, ensuring that the inherent basic low-frequency components of the image are preserved.

[0057] The input image is processed in the network as follows: Low-resolution image (LR)I LR ∈R H×W×Cin First, the data is fed into the network and passed to the shallow feature extraction module. Here, it undergoes a convolutional layer in the shallow feature extraction module to obtain shallow features F0∈R. H×W×C .

[0058] F0 = C SF (I LR (1)

[0059] Among them, C SF (·) indicates a convolution operation that performs shallow feature extraction.

[0060] Step 102: Input the shallow features into the deep feature extraction module for fine-grained feature extraction to obtain deep features; the deep feature extraction module includes multiple residual convolution modules and multiple multi-scale feature extraction modules; the multi-scale feature extraction module is used to perform convolution processing on the input features and interact with the input features through the local feature fusion mechanism, and then after processing by the multi-feature fusion extraction module, it is fused with the input features for global feature fusion to obtain multi-scale fused features; the multi-feature fusion extraction module is used to perform multi-scale image feature extraction through feature extraction branches with different expansion rates, and then use the PHA module to perform a mixed weighting operation, and fuse the mixed weighting result with the features input to the multi-feature fusion extraction module to obtain multi-scale fused features.

[0061] Specifically, the deep feature extraction module consists of multiple residual convolutional modules (ResBlocks) and multiple multi-scale feature extraction modules (MFBs). MFBs are crucial for recalibrating feature weights and enhancing attention to detailed image components. This mechanism significantly improves the network's ability to capture the fine-grained information required for high-quality super-resolution reconstruction. Subsequently, the deep-extracted features are propagated through the super-resolution reconstruction module to generate high-resolution (HR) images.

[0062] The shallow feature F0 is input into the deep feature extraction module for processing to obtain the deep feature F. DF .

[0063] F DF =C MR (F0) (2)

[0064] Among them, C MR (·) represents the multi-scale and residual operations in the deep feature extraction module, which are performed by multiple residual convolution modules (ResBlocks) and multiple multi-scale feature extraction modules (MFB).

[0065] Subsequently, the deep features F extracted from the complete image DF Input to the super-resolution reconstruction module.

[0066] The deep feature extraction module utilizes a parallel hybrid attention (PHA) module to enhance the model's detail reconstruction capabilities. By generating feature maps at different scales and feeding them into the PHA module, the model's ability to reconstruct details is comprehensively enhanced. This strategic integration of multi-scale feature extraction and the parallel hybrid attention mechanism overcomes the limitations of existing methods and facilitates the generation of high-fidelity super-resolution images.

[0067] Step 104: Input the deep features into the super-resolution reconstruction module to obtain a high-quality reconstructed image; the super-resolution reconstruction module is used to adjust the deep features with different numbers of convolutional layers and sub-pixel convolutional layers to generate a high-quality reconstructed image.

[0068] Specifically, the super-resolution reconstruction module includes varying numbers of convolutional layers and subpixel convolutional layers, tailored to the specific requirements of the desired reconstruction size. This modular approach ensures adaptability and scalability when generating high-resolution images of different resolution specifications.

[0069] Here, the deep features F extracted from the complete image are... DF The image is transmitted to the super-resolution reconstruction module, where it is magnified through sub-pixel convolution to generate a high-quality SR image. SR ∈R H×W×Cout .

[0070] I SR =C UP (F DF (3)

[0071] Among them, I SR For high-quality SR images, C UP (·) represents the operation of different numbers of convolutional layers and subpixel convolutional layers in the deep super-resolution reconstruction module.

[0072] A multi-scale hybrid attention (MSHA) model for high-quality super-resolution image reconstruction is composed of shallow feature extraction, deep feature extraction, and super-resolution reconstruction modules. The general structure of the multi-scale hybrid attention (MSHA) model is as follows: Figure 2 As shown.

[0073] This Multi-Scale Hybrid Attention (MSHA) model introduces a multi-scale feature extraction mechanism, enabling holistic image analysis from a global perspective. The core of this framework is the Multi-Scale Feature Extraction Module (MFB), which consists of a non-traditional convolutional layer, an activation layer, and a Multi-Feature Hybrid Extraction (MFHE) process. The MFHE component uses dilated convolution after expanding the receptive field, thereby capturing a wider range of features while maintaining model simplicity.

[0074] Subsequently, the MSHA model of this application uses the L1 loss function to optimize its parameters.

[0075] In the aforementioned high-quality super-resolution image reconstruction method, a multi-scale feature extraction mechanism is designed, enabling holistic image analysis from a global perspective. The core of the network architecture employed in this method is the multi-scale feature extraction module, which consists of a non-traditional convolutional layer, an activation layer, and a multi-feature fusion extraction process. The multi-feature fusion extraction component uses dilated convolution after expanding the receptive field, thereby capturing a wider range of features while maintaining model simplicity. This method utilizes a parallel hybrid attention module to enhance the model's detail reconstruction capabilities; by generating feature maps at different scales and inputting them into the PHA module, the model's ability to reconstruct details is comprehensively enhanced. This strategic integration of multi-scale feature extraction and the parallel hybrid attention mechanism overcomes the limitations of existing methods and promotes the generation of high-fidelity super-resolution images.

[0076] In one embodiment, the deep feature extraction module includes four residual convolutional modules and five multi-scale feature extraction modules; the shallow features are the input of the first multi-scale feature extraction module, and the first multi-scale feature extraction module, the first residual convolutional module, the second multi-scale feature extraction module, the second residual convolutional module, the third multi-scale feature extraction module, the third residual convolutional module, the fourth multi-scale feature extraction module, the fourth residual convolutional module, and the fifth multi-scale feature extraction module are connected in sequence.

[0077] In one embodiment, the residual convolution module includes eight stacked residual blocks.

[0078] In one embodiment, such as Figure 3 As shown, the multi-scale feature extraction module includes: a convolutional group, a multi-feature fusion extraction module, and a fusion module. In the multi-scale feature extraction module: the features input to the multi-scale feature extraction module are processed by the convolutional group to obtain convolutional features; the convolutional features and the features input to the multi-scale feature extraction module exchange information through a local feature fusion mechanism to obtain local fused features; the local fused features are input to the multi-feature fusion extraction module to obtain multi-scale fused features; the multi-scale fused features and the features input to the multi-scale feature extraction module are fused through the fusion module to obtain multi-scale fused features.

[0079] Specifically, the structure of the multi-scale feature extraction module is as follows: Figure 3 As shown, the input features of the multi-scale feature extraction module are first processed by a convolutional group, such as... Figure 3As shown within the dashed box. This method uses the Gaussian Error Linear Unit (GELU) activation function, which has demonstrated superior performance in various image processing tasks. Unlike the baseline ResBlock, which omits batch normalization (BN), this method reintroduces BN into the convolutional groups. This choice is made because BN improves the regularization of the feature representation in the subsequent feature weighting operation and helps to adjust the input distribution to remain stable. Therefore, this facilitates more effective learning of feature relationships by the attention mechanism and ensures robustness to distribution fluctuations.

[0080] The process of first processing the input features of the multi-scale feature extraction module through a convolutional group is as follows:

[0081] F0'=C G (F0) (4)

[0082] Where F0'∈R H×W×C This represents the features obtained by first processing the input features of the multi-scale feature extraction module through a convolutional group.

[0083] After processing by convolutional groups, F0'∈R H×W×C F0 exchanges information through the Local Feature Fusion (LFF) mechanism and then enters the multi-feature fusion extraction module.

[0084] F0” = C LF (F0+F0') (5)

[0085] Where F0" represents local fusion features, C LF (·) represents the Local Feature Fusion (LFF) mechanism.

[0086] Through an iterative learning process in the Multi-Feature Hybrid Extraction (MFHE) module, complex features are comprehensively captured, and feature weights for each channel are carefully assigned. Once fine-grained processing is complete, the next step is to acquire low-level global features to supplement these details. To facilitate the exchange of high- and low-level feature information, this method employs Global Feature Fusion (GFF).

[0087] F1 = MUL(MFHE(F0”) + F0) (6)

[0088] Where F1 represents the output feature of the multi-feature fusion extraction module, MUL(·) represents the fusion module, and MFHE(·) represents the multi-feature fusion extraction module.

[0089] In one embodiment, the fusion module employs a multiplication operation.

[0090] Specifically, it is worth noting that the fusion module used in this method does not directly add features, but rather performs a multiplication operation. This meticulous approach can precisely control the information flow, minimize information loss, and enable the model to discern the relative importance of features.

[0091] In one embodiment, the multi-feature fusion extraction module includes: four feature extraction branches with different expansion rates and a parallel fusion attention module; each feature extraction branch includes a dilated convolutional layer; inputting local fusion features into the multi-feature fusion extraction module to obtain multi-scale fusion features includes: processing the local fusion features using the dilated convolutional layer of the first feature extraction branch to obtain first-scale image features; fusing the local fusion features and the first-scale image features and processing them using the dilated convolutional layer of the second feature extraction branch to obtain second-scale image features; fusing the local fusion features and the second-scale image features and processing them using the dilated convolutional layer of the third feature extraction branch to obtain third-scale image features; fusing the local fusion features and the third-scale image features and processing them using the dilated convolutional layer of the fourth feature extraction branch to obtain fourth-scale image features; inputting the first-scale image features, the second-scale image features, the third-scale image features, and the fourth-scale image features into the parallel fusion attention module to obtain fusion weighted features; and fusing the fusion weighted features with the local fusion features to obtain multi-scale fusion features.

[0092] Specifically, the multi-feature fusion extraction module includes extraction and fusion processes. For example... Figure 4 As shown, the input feature F n The network employs four 3×3 dilated convolutional layers with different scaling rates to capture multi-scale image features. After the convolution operation in each branch, the processed features are shared between adjacent branches to facilitate information exchange between different scales. This feature extraction part not only maintains the network's depth in a parallel manner but also enhances its width, thereby improving the network's ability to extract fine local textures and broad global semantic information.

[0093] After feature extraction is completed at each branch, the resulting feature maps are passed to the PHA module for weighted blending. Then, the input feature F is... n The features obtained through the hybrid extraction process are fused with residual connections to generate the output feature F. n '.

[0094]

[0095] Where PHA(·) represents a parallel hybrid attention module, C d＝1 (·) represents a dilated convolution operation with a dilation rate of 1.

[0096] In one embodiment, image features at a first scale, a second scale, a third scale, and a fourth scale are input into a parallel hybrid attention module to obtain hybrid weighted features. This includes: using global average pooling to obtain global feature representations from the first, second, third, and fourth scale image features, capturing long-distance dependencies in the horizontal and vertical directions while preserving positional relationships; then modeling inter-channel correlations through a 1×1 convolution operation; and finally activating the features using a sigmoid function to obtain global features in the horizontal and vertical directions.

[0097] X ni =sigmiod(Conv(AvgPool) H (F n '1))) (8)

[0098] Y nj =sigmiod(Conv(AvgPool) W (F n '1))) (9)

[0099] Among them, X ni ∈R 1×1×W Y represents the global feature in the horizontal direction corresponding to the image feature at scale i, where i = 1, 2, ..., m. nj ∈R 1×H×1 F represents the global feature in the vertical direction corresponding to the image feature at scale j, where j = 1, 2, ..., m; n ' i Let m be the image feature at scale i, and m be the number of scales for image features at different scales.

[0100] Global features in the horizontal and vertical directions are concatenated along the second channel dimension and then activated using the Softmax function along the channel dimension to obtain multi-scale channel feature weights in the horizontal and vertical directions. These multi-scale channel feature weights are then normalized to obtain normalized weights in the horizontal and vertical directions. The first-scale, second-scale, third-scale, and fourth-scale image features are then multiplied element-wise by their respective normalized weights, and the products are summed to obtain the mixed weighted features:

[0101]

[0102] Among them, F n 'W' represents a mixed weighted feature. X [:,i,:]∈R 1×1×W W y [:,j,:]∈R1×H×1 For the normalized weights in the horizontal and vertical directions; F n For input features.

[0103] Specifically, the Parallel Hybrid Attention (PHA) module extends the channel attention principle, extracting comprehensive and detailed key features by combining channel and spatial dimensions. It also supports feature fusion across multiple scales. A detailed diagram of PHA is shown below. Figure 5 As shown.

[0104] Given M feature representations of the same size {F n '1,F n '2,...,F n 'm}∈R C×H×W First, global feature representations are obtained through global average pooling to capture long-distance dependencies in the horizontal and vertical directions while preserving positional relationships. Then, the correlation between channels is modeled through 1×1 convolution operations, and global features (i.e., channel descriptors) in the horizontal and vertical directions are generated through the sigmoid activation function as shown in Equations (8) and (9).

[0105] Next, the two sets of orthogonal global features (each corresponding to M scales) are concatenated along the second dimension (channel dimension) to obtain the feature set X'∈R. M×1×W and Y′∈R M×H×1 Subsequently, the multi-scale channel weights are obtained by applying the Softmax function along the channel dimension. Finally, the features are multiplied element-wise with the corresponding normalized weights, and the processed features are summed to obtain the hybrid weighted feature F as shown in formula (10). n '.

[0106] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0107] In a validation implementation, the DIV2K dataset, containing 800 high-quality images, was used for training. This dataset covers a variety of image types, including natural landscapes, portraits, and animals, making it well-suited for evaluating the generalization ability of image super-resolution algorithms. For testing, standard datasets such as Set5, Set14, B100, Urban100, and Manga109 were used. To evaluate the results, we specifically calculated the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) metrics for the Y channel in the converted YCbCr color space.

[0108] For multi-feature extraction, this embodiment uses an inflation rate of (2, 4, 8), which has been extensively validated experimentally and demonstrates superior performance for MFEB. The complete model consists of 32 residual blocks (ResBlocks) and 5 MFBs. Furthermore, the intermediate feature channels of all ResBlocks are consistent with the settings in EDSR, all being 256. For input data, we use LR images with a batch size of 64×4 and their corresponding HR images. The learning rate is initialized to 1e-4 and halved every 200 epochs, finally reaching the final model after 1000 epochs. We use the ADAM optimizer with parameters set to β1 = 0.9, β2 = 0.999, ∈ = 10. -8 The experiments were conducted using the PyTorch framework and performed efficient computations on an Nvidia 3090 GPU.

[0109] (1) Ablation test

[0110] This embodiment conducts multiple ablation experiments on the proposed Multi-Scale Feature Extraction (MFB) module to evaluate its efficiency. First, we explore the necessity of normalization operations in the MFB. While many super-resolution reconstruction networks omit normalization, our MFB requires it to facilitate subsequent hybrid extraction processes. Furthermore, we hypothesize that local feature fusion and global feature fusion within the convolutional groups play corresponding roles in capturing effective features, as shown in Table 1.

[0111] Table 1: Effectiveness of relevant components in MFB on the Set14 dataset at a scaling ratio of 2.

[0112] Module PSNR / SSIM Pure 33.65 / 0.9172 Pure+BN 33.72 / 0.9183 Pure+LFF 33.70 / 0.9173 Pure+GLF 33.76 / 0.9189 Pure+BN+LFF+GLF(MFB) 34.04 / 0.9228

[0113] Next, we delve into the number of MFBs. While the performance of deep convolutional neural networks is partly attributable to the depth of their stacking layers, deeper networks do not always yield better results in super-resolution tasks; optimal depth is crucial. Therefore, we inserted 5 ResBlocks into the baseline at each MFB interval. The results of this exploration regarding the number of MFBs used are presented in Table 2.

[0114] Table 2: Results of different MFB usages in the network for each benchmark at a ×4 scaling ratio.

[0115]

[0116] Finally, we selected several models and compared them with the proposed MSHA in terms of FLOPs, number of parameters, and runtime, as shown in Table 3. Our findings indicate that, in order to fully capture effective features, our model requires more computational workload and more parameters compared to the baseline model. However, through optimization of runtime and overall performance, we achieved better results.

[0117] Table 3: Comparison of the complexity of different methods, evaluating complexity metrics and PSNR on Manga109 (×2).

[0118]

[0119] The results confirm that blindly increasing network depth does not necessarily improve performance. Furthermore, this emphasizes the point that attention mechanisms should not be overused. Improper use not only increases computational costs and reduces model efficiency, but can also lead to overfitting of the model to irrelevant information in the data, thereby reducing the model's generalization ability and overall performance.

[0120] (2) Comparison with the latest technology

[0121] To evaluate the performance of MSHA, this embodiment compares it with 10 state-of-the-art techniques, including WMRN, MSRN, SeaNet, EDSR, RDN, SRFBN, DBPN, WDRN, MGAN, HRAN, DDistill, TPCNN, and ESRT. The datasets and visualizations used in the comparisons were provided by the authors of the respective papers or independently reproduced.

[0122] Quantitative Evaluation: Table 4 shows a clear comparison between the MSHA model proposed in this application and other state-of-the-art methods. Clearly, due to the multi-scale fusion capability of the MSHA model, its ability to capture global information and collect detailed textures surpasses that of other state-of-the-art techniques. This advantage is confirmed by qualitative observations and numerical metrics. In all benchmark tests, our MSHA consistently outperforms other methods in PSNR and SSIM metrics across almost all scales, confirming its effectiveness in improving the quality of super-resolution images.

[0123] Qualitative Evaluation: To verify the actual perceptual effect of MSHA in image processing, this embodiment demonstrates the reconstruction results after applying the MSHA model, as well as the results of other methods on each benchmark dataset. For example... Figure 6 As shown, (a) illustrates the image reconstruction results of different methods in the Set5 dataset, (b) illustrates the image reconstruction results of different methods in the Set14 dataset, (c) illustrates the image reconstruction results of different methods in the B100 dataset, (d) illustrates the image reconstruction results of different methods in the Urban100 dataset, and (e) illustrates the image reconstruction results of different methods in the Manga109 dataset. Figure 6 It is clear that MSHA significantly outperforms most other models in detail reconstruction quality. This example evaluates the image reconstruction performance of different methods on five benchmark datasets, each scaled by ×4. The results show that most methods struggle to achieve satisfactory results in detailed texture regions, providing only blurry results. In contrast, the MSHA model consistently generates sharper and more natural textures. This remarkable result further highlights the effectiveness of our multi-scale hybrid attention mechanism in image super-resolution tasks.

[0124] Table 4: Quantitative evaluation of the average PSNR and SSIM of state-of-the-art methods at scaling ratios of ×2, ×3, and ×4 on multiple benchmarks.

[0125]

[0126]

[0127]

[0128] To address the limitations of existing single-image super-resolution (SISR) methods in recognizing detailed textures and processing large-field-of-view images, this application proposes a multi-scale hybrid attention network (MSHA) model for high-quality super-resolution image reconstruction. The MSHA model utilizes a large receptive field across multiple scales during feature extraction, enabling better inference based on proximity information. Furthermore, the MSHA model prioritizes detail enhancement, significantly improving the perceptual quality of the reconstructed image.

[0129] In one embodiment, such as Figure 7 As shown, a high-quality super-resolution image reconstruction device is provided, comprising: a shallow feature extraction unit, a deep feature extraction unit, and an image reconstruction unit, wherein:

[0130] The shallow feature extraction unit is used to input the input image into the shallow feature extraction module to obtain shallow features; the shallow feature extraction module is used to extract low-frequency information from the image using single-layer convolution and output shallow features.

[0131] The deep feature extraction unit is used to input shallow features into the deep feature extraction module for fine-grained feature extraction, resulting in deep features. The deep feature extraction module includes multiple residual convolution modules and multiple multi-scale feature extraction modules. The multi-scale feature extraction module performs convolution processing on the input features and interacts with the input features through a local feature fusion mechanism. Then, after processing by the multi-feature fusion extraction module, it performs global feature fusion with the input features to obtain multi-scale fused features. The multi-feature fusion extraction module performs multi-scale image feature extraction through feature extraction branches with different expansion rates, and then performs a fusion weighting operation using the PHA module. The fusion weighting result is then fused with the features input to the multi-feature fusion extraction module to obtain multi-scale fused features.

[0132] The image reconstruction unit is used to input deep features into the super-resolution reconstruction module to obtain a high-quality reconstructed image; the super-resolution reconstruction module is used to adjust the deep features using different numbers of convolutional layers and sub-pixel convolutional layers to generate a high-quality reconstructed image.

[0133] In one embodiment, the deep feature extraction module includes four residual convolutional modules and five multi-scale feature extraction modules; the shallow features are the input of the first multi-scale feature extraction module, and the first multi-scale feature extraction module, the first residual convolutional module, the second multi-scale feature extraction module, the second residual convolutional module, the third multi-scale feature extraction module, the third residual convolutional module, the fourth multi-scale feature extraction module, the fourth residual convolutional module, and the fifth multi-scale feature extraction module are connected in sequence.

[0134] In one embodiment, the residual convolution module in the deep feature extraction module includes eight stacked residual blocks.

[0135] In one embodiment, the deep feature extraction module includes a multi-scale feature extraction module comprising: a convolutional group, a multi-feature fusion extraction module, and a fusion module. In the multi-scale feature extraction module: features input to the multi-scale feature extraction module are convolved through the convolutional group to obtain convolutional features; the convolutional features are exchanged with the features input to the multi-scale feature extraction module through a local feature fusion mechanism to obtain locally fused features; the locally fused features are input to the multi-feature fusion extraction module to obtain multi-scale fused features; and the multi-scale fused features and the features input to the multi-scale feature extraction module are fused through the fusion module to obtain multi-scale fused features.

[0136] In one embodiment, the fusion module employs a multiplication operation.

[0137] In one embodiment, the multi-feature fusion extraction module includes: four feature extraction branches with different expansion rates and a parallel fusion attention module; each feature extraction branch includes a dilated convolutional layer; inputting local fusion features into the multi-feature fusion extraction module to obtain multi-scale fusion features includes: processing the local fusion features using the dilated convolutional layer of the first feature extraction branch to obtain first-scale image features; fusing the local fusion features and the first-scale image features and processing them using the dilated convolutional layer of the second feature extraction branch to obtain second-scale image features; fusing the local fusion features and the second-scale image features and processing them using the dilated convolutional layer of the third feature extraction branch to obtain third-scale image features; fusing the local fusion features and the third-scale image features and processing them using the dilated convolutional layer of the fourth feature extraction branch to obtain fourth-scale image features; inputting the first-scale image features, the second-scale image features, the third-scale image features, and the fourth-scale image features into the parallel fusion attention module to obtain fusion weighted features; and fusing the fusion weighted features with the local fusion features to obtain multi-scale fusion features.

[0138] In one embodiment, the parallel hybrid attention module in the deep feature extraction unit is used to obtain global feature representations of the first-scale image features, the second-scale image features, the third-scale image features and the fourth-scale image features through global average pooling, capture long-distance dependencies in the horizontal and vertical directions, and preserve positional relationships. Then, the correlation between channels is modeled through 1×1 convolution operation, and after activation by the sigmoid function, global features in the horizontal and vertical directions as shown in formulas (8) and (9) are obtained.

[0139] The global features in the horizontal and vertical directions are concatenated in the second channel dimension and then activated by the Softmax function in the channel dimension to obtain the multi-scale channel feature weights in the horizontal and vertical directions. The multi-scale channel feature weights in the horizontal and vertical directions are normalized to obtain the normalized weights in the horizontal and vertical directions. The first-scale image features, the second-scale image features, the third-scale image features and the fourth-scale image features are multiplied element by element with the corresponding normalized weights, and the product results are summed to obtain the mixed weighted features as shown in formula (10).

[0140] Specific limitations regarding the high-quality super-resolution image reconstruction apparatus can be found in the limitations of the high-quality super-resolution image reconstruction method described above, and will not be repeated here. Each module in the aforementioned high-quality super-resolution image reconstruction apparatus can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0141] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a high-quality super-resolution image reconstruction method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0142] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0143] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiment.

[0144] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0145] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0146] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A high-quality super-resolution image reconstruction method, characterized in that, The method includes: The input image is fed into the shallow feature extraction module to obtain shallow features; the shallow feature extraction module is used to extract low-frequency information from the image using a single-layer convolution and output shallow features; The shallow features are input into the deep feature extraction module for fine-grained feature extraction to obtain deep features. The deep feature extraction module includes multiple residual convolution modules and multiple multi-scale feature extraction modules. The multi-scale feature extraction module is used to perform convolution processing on the input features and interact with the input features through a local feature fusion mechanism. Then, after processing by the multi-feature fusion extraction module, it is fused with the input features globally to obtain multi-scale fused features. The multi-feature fusion extraction module is used to extract multi-scale image features through feature extraction branches with different expansion rates and then perform a hybrid weighting operation using a parallel hybrid attention module. The hybrid weighting result is fused with the features input to the multi-feature fusion extraction module to obtain multi-scale fused features. The multi-feature fusion extraction module includes: four feature extraction branches with different expansion rates and a parallel hybrid attention module; each feature extraction branch includes a dilated convolutional layer. The deep features are input into the super-resolution reconstruction module to obtain a high-quality reconstructed image; the super-resolution reconstruction module is used to adjust the deep features using different numbers of convolutional layers and sub-pixel convolutional layers to generate a high-quality reconstructed image. The multi-scale image features extracted by the multi-feature hybrid extraction module include: first-scale image features, second-scale image features, third-scale image features, and fourth-scale image features; In the parallel hybrid attention module: The first-scale image features, second-scale image features, third-scale image features, and fourth-scale image features are used to obtain global feature representations through global average pooling, capturing long-distance dependencies in the horizontal and vertical directions while preserving positional relationships. Then, the correlation between channels is modeled through a 1×1 convolution operation, and after activation by the sigmoid function, the global features in the horizontal and vertical directions are obtained as follows: in, For the first i Global features in the horizontal direction corresponding to scale image features. , For the first j Global features in the vertical direction corresponding to scale image features ; For the first i Scale image features m The number of scales for image features at different scales; The global features in the horizontal and vertical directions are concatenated in the second channel dimension and then activated by the Softmax function in the channel dimension to obtain the multi-scale channel feature weights in the horizontal and vertical directions. The multi-scale channel feature weights in the horizontal and vertical directions are normalized to obtain normalized weights in the horizontal and vertical directions. The first-scale image features, the second-scale image features, the third-scale image features, and the fourth-scale image features are each multiplied element-wise by their corresponding normalized weights, and the product results are summed to obtain the mixed weighted features: in, For mixed weighted features, These are the normalized weights in the horizontal and vertical directions; For input features.

2. The method according to claim 1, characterized in that, The deep feature extraction module includes 4 residual convolution modules and 5 multi-scale feature extraction modules; The shallow features are the input to the first multi-scale feature extraction module, and the first multi-scale feature extraction module, the first residual convolution module, the second multi-scale feature extraction module, the second residual convolution module, the third multi-scale feature extraction module, the third residual convolution module, the fourth multi-scale feature extraction module, the fourth residual convolution module, and the fifth multi-scale feature extraction module are connected in sequence.

3. The method according to claim 1, characterized in that, The residual convolution module includes 8 stacked residual blocks.

4. The method according to claim 1, characterized in that, The multi-scale feature extraction module includes: a convolutional group, a multi-feature hybrid extraction module, and a fusion module; in the multi-scale feature extraction module: The features input to the multi-scale feature extraction module are processed by the convolution group to obtain convolutional features; The convolutional features and the features input to the multi-scale feature extraction module exchange information through a local feature fusion mechanism to obtain local fused features; The local fusion features are input into the multi-feature fusion extraction module to obtain multi-scale fusion features; The multi-scale hybrid features and the features input to the multi-scale feature extraction module are fused through the fusion module to obtain multi-scale fused features.

5. The method according to claim 4, characterized in that, The fusion module employs a multiplication operation.

6. The method according to claim 4, characterized in that, The local fusion features are input into the multi-feature fusion extraction module to obtain multi-scale fusion features, including: The local fusion features are processed using a dilated convolutional layer in the first feature extraction branch to obtain image features at the first scale. After fusing the local fusion features and the first-scale image features, the second-scale image features are processed by the dilated convolutional layer of the second feature extraction branch. After fusing the local fusion features and the second-scale image features, the third-scale image features are processed by the dilated convolutional layer of the third feature extraction branch. After fusing the local fusion features and the third-scale image features, the fourth-scale image features are processed by the dilated convolutional layer of the fourth feature extraction branch. The first-scale image features, the second-scale image features, the third-scale image features, and the fourth-scale image features are input into the parallel hybrid attention module to obtain hybrid weighted features; The weighted features are fused with the local fusion features to obtain multi-scale mixed features.

7. A high-quality super-resolution image reconstruction device, characterized in that, The device includes: The shallow feature extraction unit is used to input the input image into the shallow feature extraction module to obtain shallow features; the shallow feature extraction module is used to extract low-frequency information from the image using a single-layer convolution and output shallow features. A deep feature extraction unit is used to input the shallow features into a deep feature extraction module for fine-grained feature extraction to obtain deep features. The deep feature extraction module includes multiple residual convolution modules and multiple multi-scale feature extraction modules. The multi-scale feature extraction module performs convolution processing on the input features and interacts with the input features through a local feature fusion mechanism. Then, after processing by a multi-feature fusion extraction module, it performs global feature fusion with the input features to obtain multi-scale fused features. The multi-feature fusion extraction module performs multi-scale image feature extraction through feature extraction branches with different expansion rates, and then performs a weighted operation using parallel hybrid attention. The weighted result is fused with the features input to the multi-feature fusion extraction module to obtain multi-scale fused features. The multi-feature fusion extraction module includes four feature extraction branches with different expansion rates and a parallel hybrid attention module. Each feature extraction branch includes a dilated convolutional layer. The image reconstruction unit is used to input the deep features into the super-resolution reconstruction module to obtain a high-quality reconstructed image; the super-resolution reconstruction module is used to adjust the deep features using different numbers of convolutional layers and sub-pixel convolutional layers to generate a high-quality reconstructed image. The multi-scale image features extracted by the multi-feature hybrid extraction module include: first-scale image features, second-scale image features, third-scale image features, and fourth-scale image features; The deep feature extraction unit is also used in the parallel hybrid attention module to: obtain global feature representations from the first-scale image features, the second-scale image features, the third-scale image features, and the fourth-scale image features through global average pooling, capturing long-distance dependencies in the horizontal and vertical directions while preserving positional relationships; then modeling inter-channel correlations through 1×1 convolution operations; and finally, after activation by the sigmoid function, obtaining global features in the horizontal and vertical directions as follows: in, For the first i Global features in the horizontal direction corresponding to scale image features. , For the first j Global features in the vertical direction corresponding to scale image features ; For the first i Scale image features m The number of scales for image features at different scales; The global features in the horizontal and vertical directions are concatenated in the second channel dimension and then activated by the Softmax function in the channel dimension to obtain the multi-scale channel feature weights in the horizontal and vertical directions. The multi-scale channel feature weights in the horizontal and vertical directions are normalized to obtain normalized weights in the horizontal and vertical directions. The first-scale image features, the second-scale image features, the third-scale image features, and the fourth-scale image features are each multiplied element-wise by their corresponding normalized weights, and the product results are summed to obtain the mixed weighted features: in, For mixed weighted features, These are the normalized weights in the horizontal and vertical directions; For input features.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images