High-quality super-resolution image reconstruction method and device, equipment and storage medium

Abstract

The invention relates to a high-quality super-resolution image reconstruction method and device, equipment and a storage medium. According to the method, a multi-scale feature extraction mechanism is designed, overall analysis can be carried out on an image from a global perspective, the core of a network architecture of the method is a multi-scale feature extraction module, and the module is composed of a non-traditional convolution layer, an activation layer and a multi-feature mixed extraction process; the multi-feature mixed extraction component uses cavity convolution after expanding a receptive field, so that wider features can be captured while the simplicity of the model is kept. According to the method, the detail reconstruction capability of the model is improved by using the parallel mixed attention module; by generating feature maps on different scales and inputting the feature maps into the PHA module, it is ensured that the ability of the model in the aspect of detail reconstruction is comprehensively enhanced. By means of strategic integration of multi-scale feature extraction and a parallel mixed attention mechanism, the limitation of an existing method is overcome, and generation of a high-fidelity super-resolution image is promoted.

Description

Technical Field

[0001] The present application relates to the field of image reconstruction technology, and in particular to a high-quality super-resolution image reconstruction method, device, equipment and storage medium. Background Art

[0002] Image super-resolution is a key technique in image processing that aims to recover the intricate details of a high-resolution image from a low-resolution image. In the field of image super-resolution, methods can be divided into two categories according to the number of input images: single-image methods and multi-image methods. Single-image techniques mainly adopt two strategies: interpolation-based methods and learning-based methods. Interpolation-based methods increase the resolution of low-resolution images by interpolation, but usually result in blurry and distorted output images. In contrast, learning-based methods use deep learning models to learn image mapping relationships from a large number of high-resolution image datasets, and then apply these learned relationships to low-resolution inputs to generate clearer and more realistic high-resolution reconstructions. Notable examples of this type of method include super-resolution convolutional neural networks (SRCNN), efficient sub-pixel convolutional neural networks (ESPCN), and super-resolution methods using generative adversarial networks (SRGAN), which have demonstrated significant performance improvements in advancing image super-resolution techniques. In contrast, multi-image methods utilize multiple low-resolution images to generate corresponding high-resolution reconstructed images. Among these methods, motion estimation-based methods recover detailed information in high-resolution reconstruction by exploiting motion data between multiple low-resolution images, while self-similarity-based methods exploit self-similar patterns in multiple low-resolution images to extract detailed information in high-resolution reconstruction. By integrating information from multiple low-resolution inputs, these methods are able to generate more precise and accurate high-resolution outputs.

[0003] In recent years, with the popularity of deep learning methods, learning-based single-image techniques have become a powerful means to generate high-fidelity super-resolution images. These methods are able to skillfully capture the complex mapping relationships between images through complex deep learning architectures. At the same time, due to the inherent shortcomings of multi-image methods, especially the high data processing and computational costs, single-image techniques have become the mainstream method widely adopted by industry and academia.

[0004] In 2014, Dong et al. proposed the Super Resolution Convolutional Neural Network (SRCNN), a revolutionary contribution that revolutionized the field of Single Image Super Resolution (SISR) by leveraging a series of convolutional layers, rectified linear units, and upsampling layers to learn the mapping from low-resolution to high-resolution images. SRCNN represents an important breakthrough in applying convolutional neural network models to SISR tasks. The effectiveness of image super-resolution reconstruction using convolutional neural network models mainly stems from the introduction of their depth and residual connections. Since then, many scholars have proposed representative SISR methods, focusing on leveraging the depth and residual connections of convolutional neural networks to advance the development of this field.

[0005] Despite the remarkable success of existing single image super-resolution (SISR) methods, there is still a significant shortcoming in their design, especially in preserving details such as edges and dense textures. This limitation hinders the generation of high-quality super-resolution (SR) images. The attention mechanism introduced by previous researchers enables more effective prioritization of feature components during training [7]. However, it is worth noting that these mechanisms (including channel attention mechanisms) mainly focus on amplifying or weakening the influence of a single channel based on related information. Although these mechanisms have shown potential in improving indicators such as peak signal-to-noise ratio (PSNR), they still fall short in processing details of image boundaries and textures. Summary of the invention

[0006] Based on this, it is necessary to provide a high-quality super-resolution image reconstruction method, device, equipment and storage medium to address the above technical problems.

[0007] A high-quality super-resolution image reconstruction method, the method comprising:

[0008] The input image is input into the shallow feature extraction module to obtain shallow features; the shallow feature extraction module is used to extract low-frequency information from the image using a single-layer convolution and output shallow features.

[0009] The shallow features are input into the deep feature extraction module for fine-grained feature extraction to obtain deep features; the deep feature extraction module includes multiple residual convolution modules and multiple multi-scale feature extraction modules; the multi-scale feature extraction module is used to perform convolution processing on the input features and then interact with the input features with the local feature fusion mechanism, and then perform global feature fusion with the input features after processing by the multi-feature mixed extraction module to obtain multi-scale fusion features; the multi-feature mixed extraction module is used to perform multi-scale image feature extraction through feature extraction branches with different expansion rates, and then use the PHA module to perform mixed weighting operation, and fuse the mixed weighted results with the features input to the multi-feature mixed extraction module to obtain multi-scale fusion features;

[0010] The deep features are input into the super-resolution reconstruction module to obtain a high-quality reconstructed image; the super-resolution reconstruction module is used to adjust the deep features by using different numbers of convolutional layers and sub-pixel convolutional layers to generate a high-quality reconstructed image.

[0011] In one of the embodiments, the deep feature extraction module includes 4 residual convolution modules and 5 multi-scale feature extraction modules.

[0012] The shallow features are the input of the first multi-scale feature extraction module, the first multi-scale feature extraction module, the first residual convolution module, the second multi-scale feature extraction module, the input of the second residual convolution module, the third multi-scale feature extraction module, the input of the third residual convolution module, the fourth multi-scale feature extraction module, the input of the fourth residual convolution module, and the fifth multi-scale feature extraction module are connected in sequence.

[0013] In one embodiment, the residual convolution module includes 8 stacked residual blocks.

[0014] In one embodiment, the multi-scale feature extraction module includes: a convolution group, a multi-feature mixed extraction module and a fusion module; in the multi-scale feature extraction module:

[0015] The features input to the multi-scale feature extraction module are convolved through a convolution group to obtain convolution features.

[0016] The convolutional features and the features input into the multi-scale feature extraction module are exchanged through the local feature fusion mechanism to obtain local fusion features.

[0017] The local fusion features are input into the multi-feature mixed extraction module to obtain multi-scale mixed features.

[0018] The multi-scale mixed features and the features input into the multi-scale feature extraction module are fused through a fusion module to obtain multi-scale fused features.

[0019] In one embodiment, the fusion module uses a multiplication operation.

[0020] In one of the embodiments, the multi-feature mixed extraction module includes: four feature extraction branches with different expansion rates and a parallel mixed attention module; each feature extraction branch includes a hole convolution layer.

[0021] The local fusion features are input into the multi-feature mixed extraction module to obtain multi-scale mixed features, including:

[0022] The local fusion features are processed using the dilated convolutional layer of the first feature extraction branch to obtain the first-scale image features.

[0023] After the local fusion features and the first-scale image features are fused, they are processed using the dilated convolutional layer of the second feature extraction branch to obtain the second-scale image features.

[0024] After the local fusion features and the second-scale image features are fused, they are processed using the dilated convolutional layer of the third feature extraction branch to obtain the third-scale image features.

[0025] The local fusion features and the third-scale image features are fused and then processed using the dilated convolutional layer of the fourth feature extraction branch to obtain the fourth-scale image features.

[0026] The first-scale image features, the second-scale image features, the third-scale image features, and the fourth-scale image features are input into a parallel hybrid attention module to obtain hybrid weighted features.

[0027] The mixed weighted features are fused with the local fusion features to obtain multi-scale mixed features.

[0028] In one embodiment, the first scale image feature, the second scale image feature, the third scale image feature, and the fourth scale image feature are input into a parallel hybrid attention module to obtain a hybrid weighted feature, including:

[0029] The first-scale image features, the second-scale image features, the third-scale image features, and the fourth-scale image features are pooled globally to obtain global feature representation, capturing long-distance dependencies in the horizontal and vertical directions while retaining the positional relationship. Then, the correlation between channels is modeled through a 1×1 convolution operation, and after activation through a sigmoid function, the global features in the horizontal and vertical directions are obtained as follows:

[0030] X ni =sigmiod(Conv(AvgPool H (F n '1)))

[0031] Y nj =sigmiod(Conv(AvgPool W (F n '1)))

[0032] Among them, X ni ∈R 1×1×W is the global feature in the horizontal direction corresponding to the i-th scale image feature, i = 1, 2, ..., m, Y nj ∈R 1×H×1 is the global feature in the vertical direction corresponding to the j-th scale image feature, j = 1, 2, ..., m; F n ' iis the i-th scale image feature, and m is the number of scales of image features of different scales.

[0033] The global features in the horizontal and vertical directions are concatenated in the second channel dimension and activated by the Softmax function in the channel dimension to obtain multi-scale channel feature weights in the horizontal and vertical directions.

[0034] The multi-scale channel feature weights in the horizontal direction and the vertical direction are normalized to obtain the normalized weights in the horizontal direction and the vertical direction.

[0035] The first scale image features, the second scale image features, the third scale image features and the fourth scale image features are respectively multiplied element by element with the corresponding normalized weights, and the product results are summed to obtain the mixed weighted features:

[0036]

[0037] Among them, F n ' is the mixed weighted feature, W X [:,i,:]∈R 1×1×W , W y [:,i,:]∈R 1×H×1 is the normalized weight in the horizontal and vertical directions; F n is the input feature.

[0038] A high-quality super-resolution image reconstruction device, the device comprising:

[0039] The shallow feature extraction unit is used to input the input image into the shallow feature extraction module to obtain shallow features; the shallow feature extraction module is used to extract low-frequency information from the image using a single-layer convolution and output shallow features.

[0040] The deep feature extraction unit is used to input the shallow features into the deep feature extraction module for fine-grained feature extraction to obtain the deep features; the deep feature extraction module includes multiple residual convolution modules and multiple multi-scale feature extraction modules; the multi-scale feature extraction module is used to perform convolution processing on the input features, and then perform global feature fusion with the input features after processing by the multi-feature mixed extraction module to obtain multi-scale fusion features; the multi-feature mixed extraction module is used to perform multi-scale image feature extraction through feature extraction branches with different expansion rates, and then use the PHA module to perform mixed weighting operation, and fuse the mixed weighted results with the features input to the multi-feature mixed extraction module to obtain multi-scale fusion features;

[0041] The image reconstruction unit is used to input the deep features into the super-resolution reconstruction module to obtain a high-quality reconstructed image; the super-resolution reconstruction module is used to adjust the deep features by using different numbers of convolutional layers and sub-pixel convolutional layers to generate a high-quality reconstructed image.

[0042] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0043] A computer-readable storage medium stores a computer program, which implements the steps of any of the above methods when executed by a processor.

[0044] The above-mentioned high-quality super-resolution image reconstruction method, device, equipment and storage medium, the method designs a multi-scale feature extraction mechanism, which can analyze the image as a whole from a global perspective. The core of the network architecture adopted by this method is a multi-scale feature extraction module, which consists of a non-traditional convolution layer, an activation layer and a multi-feature mixed extraction process; the multi-feature mixed extraction component uses a hole convolution after expanding the receptive field, so that it can capture a wider range of features while maintaining the simplicity of the model. This method uses a parallel hybrid attention module to enhance the model's ability to reconstruct details; by generating feature maps at different scales and inputting them into the PHA module, it ensures that the model's ability to reconstruct details is fully enhanced. This strategic integration of multi-scale feature extraction and parallel hybrid attention mechanism overcomes the limitations of existing methods and promotes the generation of high-fidelity super-resolution images. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of a process of a high-quality super-resolution image reconstruction method in one embodiment;

[0046] Figure 2 is a schematic diagram of a general structure of a multi-scale hybrid attention (MSHA) model in another embodiment;

[0047] Figure 3 Schematic diagram of the composition of a multi-scale feature extraction module (MFB) in another embodiment;

[0048] Figure 4 FIG. 1 is a schematic diagram of applying multi-feature hybrid extraction (MFHE) in MFB in another embodiment;

[0049] Figure 5 A schematic diagram of a parallel hybrid attention (PHA) module in another embodiment;

[0050] Figure 6Schematic diagram of the results of MSHA and the latest technology at a visualization ×4 scaling ratio in another embodiment, wherein (a) is a schematic diagram of image reconstruction effects of different methods in the Set5 dataset, (b) is a schematic diagram of image reconstruction effects of different methods in the Set14 dataset, (c) is a schematic diagram of image reconstruction effects of different methods in the B100 dataset, (d) is a schematic diagram of image reconstruction effects of different methods in the Urban100 dataset, and (e) is a schematic diagram of image reconstruction effects of different methods in the Manga109 dataset;

[0051] Figure 7 is a structural block diagram of a high-quality super-resolution image reconstruction device in one embodiment;

[0052] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] In one embodiment, Figure 1 As shown, a high-quality super-resolution image reconstruction method is provided, the method comprising the following steps:

[0055] Step 100: Input the input image into the shallow feature extraction module to obtain shallow features; the shallow feature extraction module is used to extract low-frequency information from the image using a single-layer convolution and output shallow features.

[0056] Specifically, in the shallow feature extraction stage, a single layer of convolution is used to directly extract low-frequency information from the input image. This extracted structure serves as the basis for subsequent deep feature extraction, ensuring that the basic low-frequency components inherent in the image are preserved.

[0057] The input image is processed in the network as follows: Low-resolution image (LR) I LR ∈R H×W×Cin First, it is input into the network and passed to the shallow feature extraction module. Here, it undergoes a layer of convolution in the shallow feature extraction module to obtain the shallow feature F0∈R H×W×C .

[0058] F0=C SF (I LR ) (1)

[0059] Among them, C SF (·) denotes a convolution operation that performs shallow feature extraction.

[0060] Step 102: Input shallow features into a deep feature extraction module for fine-grained feature extraction to obtain deep features; the deep feature extraction module includes multiple residual convolution modules and multiple multi-scale feature extraction modules; the multi-scale feature extraction module is used to perform convolution processing on the input features and then exchange information with the input features and the local feature fusion mechanism, and then perform global feature fusion with the input features after processing by the multi-feature mixed extraction module to obtain multi-scale fusion features; the multi-feature mixed extraction module is used to perform multi-scale image feature extraction through feature extraction branches with different expansion rates, and then use the PHA module to perform mixed weighted operations, and fuse the mixed weighted results with the features input to the multi-feature mixed extraction module to obtain multi-scale fusion features.

[0061] Specifically, the deep feature extraction module consists of multiple residual convolution modules (ResBlocks) and multiple multi-scale feature extraction modules (MFBs). MFBs are crucial in recalibrating feature weights and enhancing attention to detailed image components. This mechanism significantly improves the network's ability to capture fine-grained information required for high-quality super-resolution reconstruction. Subsequently, the deeply extracted features are propagated through the super-resolution reconstruction module to generate high-resolution (HR) images.

[0062] The shallow feature F0 is input into the deep feature extraction module for processing to obtain the deep feature F DF .

[0063] F DF =C MR (F0) (2)

[0064] Among them, C MR (·) represents the multi-scale and residual operations in the deep feature extraction module, which are performed by multiple residual convolution modules (ResBlocks) and multiple multi-scale feature extraction modules (MFBs).

[0065] Then, the deep features F extracted from the complete image are DF Input to the super-resolution reconstruction module.

[0066] The deep feature extraction module uses the parallel hybrid attention (PHA) module to improve the model's ability to reconstruct details. By generating feature maps at different scales and feeding them into the PHA module, it ensures that the model's ability to reconstruct details is fully enhanced. This strategic integration of multi-scale feature extraction and parallel hybrid attention mechanism overcomes the limitations of existing methods and promotes the generation of high-fidelity super-resolution images.

[0067] Step 104: input the deep features into a super-resolution reconstruction module to obtain a high-quality reconstructed image; the super-resolution reconstruction module is used to adjust the deep features by using different numbers of convolutional layers and sub-pixel convolutional layers to generate a high-quality reconstructed image.

[0068] Specifically, the super-resolution reconstruction module includes different numbers of convolutional layers and sub-pixel convolutional layers, adjusted to the specific requirements of the desired reconstruction size. This modular approach ensures adaptability and scalability when generating high-resolution images of different resolution specifications.

[0069] Here, the deep features F extracted from the complete image are DF It is transmitted to the super-resolution reconstruction module and amplified by sub-pixel convolution to generate a high-quality SR image I SR ∈R H×W×Cout .

[0070] I SR =C UP (F DF ) (3)

[0071] Among them, I SR For high-quality SR images, C UP (·) represents the operations of different numbers of convolutional layers and sub-pixel convolutional layers in the deep super-resolution reconstruction module.

[0072] The multi-scale hybrid attention (MSHA) model for high-quality super-resolution image reconstruction consists of shallow feature extraction, deep feature extraction and super-resolution reconstruction modules. The general structure of the multi-scale hybrid attention (MSHA) model is as follows: Figure 2 shown.

[0073] The Multiscale Hybrid Attention (MSHA) model introduces a multiscale feature extraction mechanism that can analyze the image holistically from a global perspective. The core of the framework is the Multiscale Feature Extraction Module (MFB), which consists of an unconventional convolutional layer, an activation layer, and a Multi-Feature Hybrid Extraction (MFHE) process. The MFHE component uses dilated convolutions after expanding the receptive field, so that it can capture a wider range of features while keeping the model simple.

[0074] Subsequently, the MSHA model of this application uses the L1 loss function to optimize its parameters.

[0075] In the above-mentioned high-quality super-resolution image reconstruction method, the method designs a multi-scale feature extraction mechanism, which can analyze the image as a whole from a global perspective. The core of the network architecture adopted by this method is the multi-scale feature extraction module, which consists of a non-traditional convolution layer, an activation layer, and a multi-feature mixed extraction process; the multi-feature mixed extraction component uses a dilated convolution after expanding the receptive field, so that it can capture a wider range of features while maintaining the simplicity of the model. This method uses a parallel hybrid attention module to enhance the model's ability to reconstruct details; by generating feature maps at different scales and inputting them into the PHA module, it ensures that the model's ability to reconstruct details is fully enhanced. This strategic integration of multi-scale feature extraction and parallel hybrid attention mechanisms overcomes the limitations of existing methods and promotes the generation of high-fidelity super-resolution images.

[0076] In one embodiment, the deep feature extraction module includes 4 residual convolution modules and 5 multi-scale feature extraction modules; the shallow feature is the input of the first multi-scale feature extraction module, the first multi-scale feature extraction module, the first residual convolution module, the second multi-scale feature extraction module, the input of the second residual convolution module, the third multi-scale feature extraction module, the input of the third residual convolution module, the fourth multi-scale feature extraction module, the input of the fourth residual convolution module, and the fifth multi-scale feature extraction module are connected in sequence.

[0077] In one embodiment, the residual convolution module includes 8 stacked residual blocks.

[0078] In one embodiment, if Figure 3 As shown, the multi-scale feature extraction module includes: a convolution group, a multi-feature mixed extraction module and a fusion module; in the multi-scale feature extraction module: the features input to the multi-scale feature extraction module are convolved through the convolution group to obtain convolution features; the convolution features and the features input to the multi-scale feature extraction module are exchanged with information through a local feature fusion mechanism to obtain local fusion features; the local fusion features are input to the multi-feature mixed extraction module to obtain multi-scale mixed features; the multi-scale mixed features and the features input to the multi-scale feature extraction module are fused through a fusion module to obtain multi-scale fusion features.

[0079] Specifically, the structure of the multi-scale feature extraction module is as follows: Figure 3 The input features of the multi-scale feature extraction module are first processed by the convolution group, as shown in Figure 3As shown in the dashed box in the figure. This method uses the Gaussian Error Linear Unit (GELU) activation function, which has shown excellent performance in various image processing tasks. Unlike the baseline ResBlock that omits batch normalization (BN), this method reintroduces BN in the convolution group. This choice is because BN improves the regularization of feature representations in subsequent feature weighting operations and helps adjust the input distribution to remain stable. Therefore, this promotes more effective learning of feature relationships by the attention mechanism and ensures robustness to distribution fluctuations.

[0080] The process of first processing the input features of the multi-scale feature extraction module through the convolution group is:

[0081] F0'=C G (F0) (4)

[0082] Among them, F0'∈R H×W×C It represents the features obtained by first processing the input features of the multi-scale feature extraction module through the convolution group.

[0083] After the convolution group processing, F0'∈R H×W×C It exchanges information with F0 through the local feature fusion (LFF) mechanism and then enters the multi-feature hybrid extraction module.

[0084] F0"=C LF (F0+F0') (5)

[0085] Among them, F0" represents the local fusion feature, C LF (·) represents the local feature fusion (LFF) mechanism

[0086] Through the iterative learning process in the multi-feature hybrid extraction (MFHE) module, complex features are fully captured and the feature weights of each channel are prudently assigned. Once the fine-grained processing is completed, the next step is to obtain low-level global features to supplement these details. In order to facilitate the exchange of high-level and low-level feature information, this method adopts global feature fusion (GFF).

[0087] F1=MUL(MFHE(F0”)+F0) (6)

[0088] Among them, F1 represents the output feature of the multi-feature hybrid extraction module, MUL(·) represents the fusion module, and MFHE(·) represents the multi-feature hybrid extraction module.

[0089] In one embodiment, the fusion module uses a multiplication operation.

[0090] Specifically, it is worth noting that the fusion module used in this method does not directly add features, but performs multiplication operations. This meticulous approach can accurately control the flow of information, minimize information loss, and enable the model to discern the relative importance of features.

[0091] In one embodiment, a multi-feature mixed extraction module includes: four feature extraction branches with different expansion rates and a parallel mixed attention module; each feature extraction branch includes a hole convolution layer; the local fusion feature is input into the multi-feature mixed extraction module to obtain a multi-scale mixed feature, including: the local fusion feature is processed by the hole convolution layer of the first feature extraction branch to obtain a first-scale image feature; the local fusion feature and the first-scale image feature are fused and then processed by the hole convolution layer of the second feature extraction branch to obtain a second-scale image feature; the local fusion feature and the second-scale image feature are fused and then processed by the hole convolution layer of the third feature extraction branch to obtain a third-scale image feature; the local fusion feature and the third-scale image feature are fused and then processed by the hole convolution layer of the fourth feature extraction branch to obtain a fourth-scale image feature; the first-scale image feature, the second-scale image feature, the third-scale image feature and the fourth-scale image feature are input into the parallel mixed attention module to obtain a mixed weighted feature; the mixed weighted feature is fused with the local fusion feature to obtain a multi-scale mixed feature.

[0092] Specifically, the multi-feature hybrid extraction module includes extraction and fusion processes. Figure 4 As shown, input feature F n After four 3×3 hole convolution layers with different expansion rates, multi-scale image features are captured. After the convolution operation of each branch, the processed features are shared between adjacent branches to promote information exchange between different scales. This feature extraction part not only maintains the depth of the network in a parallel manner, but also enhances the width of the network, thereby improving the network's ability to extract fine local textures and extensive global semantic information.

[0093] After each branch completes the feature extraction operation, the obtained feature map is passed to the PHA module for mixed weighting operation. n The features obtained through the hybrid extraction process are fused through residual connections to generate the output feature F n '.

[0094]

[0095] Among them, PHA(·) represents the parallel hybrid attention module, C d＝1 (·) represents a dilated convolution operation with a dilation rate of 1.

[0096] In one embodiment, the first scale image features, the second scale image features, the third scale image features and the fourth scale image features are input into a parallel hybrid attention module to obtain a hybrid weighted feature, including: obtaining a global feature representation by global average pooling of the first scale image features, the second scale image features, the third scale image features and the fourth scale image features, capturing long-distance dependencies in the horizontal and vertical directions while retaining the positional relationship, and then modeling the correlation between channels through a 1×1 convolution operation, and after activation by a sigmoid function, obtaining the global features in the horizontal and vertical directions as follows:

[0097] X ni =sigmiod(Conv(AvgPool H (F n '1))) (8)

[0098] Y nj =sigmiod(Conv(AvgPool W (F n '1))) (9)

[0099] Among them, X ni ∈R 1×1×W is the global feature in the horizontal direction corresponding to the i-th scale image feature, i = 1, 2, ..., m, Y nj ∈R 1×H×1 is the global feature in the vertical direction corresponding to the j-th scale image feature, j = 1, 2, ..., m; F n ' i is the i-th scale image feature, and m is the number of scales of image features of different scales.

[0100] The global features in the horizontal and vertical directions are concatenated in the second channel dimension and activated by the Softmax function in the channel dimension to obtain the multi-scale channel feature weights in the horizontal and vertical directions; the multi-scale channel feature weights in the horizontal and vertical directions are normalized to obtain the normalized weights in the horizontal and vertical directions; the first-scale image features, the second-scale image features, the third-scale image features, and the fourth-scale image features are respectively multiplied element by element with the corresponding normalized weights, and the product results are summed to obtain the mixed weighted features:

[0101]

[0102] Among them, F n ' is the mixed weighted feature, W X [:,i,:]∈R 1×1×W , W y [:,i,:]∈R1×H×1 is the normalized weight in the horizontal and vertical directions; F n is the input feature.

[0103] Specifically, the parallel hybrid attention (PHA) module is extended based on the channel attention principle, which extracts comprehensive and detailed key features by combining channel and spatial dimensions. At the same time, it also supports feature fusion across multiple scales. The detailed schematic diagram of PHA is shown in Figure 5 shown.

[0104] Given M feature representations of the same size {F n '1,F n '2,...,F n 'm}∈R C×H×W First, global feature representation is obtained through global average pooling to capture long-distance dependencies in the horizontal and vertical directions while preserving positional relationships. Subsequently, the correlation between channels is modeled through a 1×1 convolution operation, and the global features (i.e., channel descriptors) in the horizontal and vertical directions are generated through the sigmoid activation function as shown in formulas (8) and (9).

[0105] Next, two sets of orthogonal global features (each set corresponds to M scales) are concatenated in the second dimension (channel dimension) to obtain the feature set X'∈R M×1×W and Y′∈R M×H×1 . Then, by applying the Softmax function on the channel dimension, the multi-scale channel weights are obtained. Finally, the features are element-wise multiplied with the corresponding normalized weights, and the processed features are summed to obtain the hybrid weighted feature F as shown in formula (10): n '.

[0106] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0107] In a validation example, the DIV2K dataset, which contains 800 high-quality images, was used for training. This dataset covers a variety of image types, including natural scenery, portraits, animals, etc., making it very suitable for evaluating the generalization ability of image super-resolution algorithms. For testing, we used standard datasets such as Set5, Set14, B100, Urban100, and Manga109. To evaluate the results, we calculated the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) metrics specifically for the Y channel in the converted YCbCr color space.

[0108] In terms of multi-feature extraction, this embodiment uses expansion rates of (2, 4, 8), which have been extensively experimentally verified and show excellent performance for MFEB. The complete model consists of 32 residual blocks (ResBlocks) and 5 MFBs. In addition, the intermediate feature channels of all ResBlocks are consistent with the settings in EDSR, which are all 256. In terms of input data, we use LR images with a batch size of 64×4 and their corresponding HR images. The learning rate is initialized to 1e-4 and is halved after every 200 cycles, and the final model is reached after 1000 cycles. We use the ADAM optimizer with parameters set to β1=0.9, β2=0.999, ∈=10 -8 The experiments were conducted using the PyTorch framework and efficiently computed on an Nvidia 3090 GPU.

[0109] (1) Ablation experiment

[0110] This embodiment conducts multiple sets of ablation experiments on the proposed multi-scale feature extraction module (MFB) to evaluate the efficiency of the relevant modules. First, we explore the necessity of the normalization operation in MFB. Although many super-resolution reconstruction networks omit the normalization operation, our MFB requires normalization to facilitate the subsequent hybrid extraction process. At the same time, we assume that the local feature fusion and the overall global feature fusion in the convolution group play corresponding roles in capturing effective features, as described in Table 1.

[0111] Table 1: Effectiveness of relevant components in MFB on the Set14 dataset when the scaling factor is 2

[0112] Modules PSNR / SSIM Pure 33.65 / 0.9172 Pure+BN 33.72 / 0.9183 Pure+LFF 33.70 / 0.9173 Pure+GLF 33.76 / 0.9189 Pure+BN+LFF+GLF(MFB) 34.04 / 0.9228

[0113] Next, we explored the number of MFBs in depth. Although the performance of deep convolutional neural networks is partly due to the depth of their stacked layers, in super-resolution tasks, deeper networks do not always lead to better results; the optimal depth is crucial. Therefore, we inserted 5 ResBlocks in the baseline at each MFB interval. The results of this exploration on the number of MFBs used are presented in Table 2.

[0114] Table 2: Results of different MFB usage in the network for each benchmark under ×4 scaling

[0115]

[0116] Finally, we selected several models and performed a comparative analysis with the proposed MSHA in terms of FLOPs, number of parameters, and running time, as shown in Table 3. Our findings show that in order to fully capture effective features, our model has an increase in computational workload and number of parameters compared to the baseline model. However, by optimizing the running time and overall performance, we achieved better results.

[0117] Table 3: Comparison of the complexity of different methods, evaluating complexity indicators and PSNR on Manga109 (×2)

[0118]

[0119] The results confirm that blindly increasing the network depth does not necessarily improve performance. In addition, this also emphasizes the view that the attention mechanism should not be overused. Improper use will not only increase computational costs and reduce model efficiency, but may also cause the model to overfit irrelevant information in the data, thereby reducing the generalization ability and overall performance of the model.

[0120] (2) Comparison with the latest technology

[0121] In order to evaluate the performance of MSHA, this example compares it with 10 state-of-the-art methods, including WMRN, MSRN, SeaNet, EDSR, RDN, SRFBN, DBPN, WDRN, MGAN, HRAN, DDistill, TPCNN, and ESRT. The datasets and visualization results used in the comparison are provided by the authors of each paper or reproduced independently.

[0122] Quantitative Evaluation: As shown in Table 4, a clear comparison of the proposed MSHA model with other state-of-the-art methods is shown. Obviously, due to the multi-scale fusion function of the MSHA model, its ability to capture global information and collect detailed textures is superior to other state-of-the-art methods. This advantage is confirmed by qualitative observations and numerical metrics. In all benchmarks, our MSHA consistently outperforms other methods in terms of PSNR and SSIM metrics at almost all scales, confirming its effectiveness in improving the quality of super-resolution images.

[0123] Qualitative evaluation: In order to verify the actual perceptual effect of MSHA processed images, this example shows the reconstruction results after applying the MSHA model and the results of other methods on each benchmark dataset. Figure 6 As shown, (a) is a schematic diagram of the image reconstruction effect of different methods in the Set5 dataset, (b) is a schematic diagram of the image reconstruction effect of different methods in the Set14 dataset, (c) is a schematic diagram of the image reconstruction effect of different methods in the B100 dataset, (d) is a schematic diagram of the image reconstruction effect of different methods in the Urban100 dataset, and (e) is a schematic diagram of the image reconstruction effect of different methods in the Manga109 dataset. Figure 6 It can be clearly seen that MSHA significantly outperforms most other models in detail reconstruction quality. This example evaluates the image reconstruction effects of different methods on five benchmark datasets, each with a scaling of ×4. The results show that most methods have difficulty achieving satisfactory results in detailed texture areas and can only provide blurry results. In contrast, the MSHA model is consistently able to generate clearer and more natural textures. This striking result further highlights the effectiveness of our multi-scale hybrid attention mechanism in image super-resolution tasks.

[0124] Table 4: Quantitative evaluation of PSNR and SSIM averages of state-of-the-art methods with scaling ratios of ×2, ×3, and ×4 on multiple benchmarks

[0125]

[0126]

[0127]

[0128] In order to address the limitations of existing single image super-resolution (SISR) methods in identifying detailed textures and processing large field of view images, this application proposes a multi-scale hybrid attention network (MSHA) model for high-quality super-resolution image reconstruction. The MSHA model uses a large receptive field at multiple scales during feature extraction to make better inferences based on neighboring information. In addition, the MSHA model prioritizes detail enhancement, significantly improving the perceptual quality of the reconstructed image.

[0129] In one embodiment, Figure 7 As shown, a high-quality super-resolution image reconstruction device is provided, comprising: a shallow feature extraction unit, a deep feature extraction unit and an image reconstruction unit, wherein:

[0130] The shallow feature extraction unit is used to input the input image into the shallow feature extraction module to obtain shallow features; the shallow feature extraction module is used to extract low-frequency information from the image using a single-layer convolution and output shallow features.

[0131] The deep feature extraction unit is used to input the shallow features into the deep feature extraction module for fine-grained feature extraction to obtain the deep features; the deep feature extraction module includes multiple residual convolution modules and multiple multi-scale feature extraction modules; the multi-scale feature extraction module is used to perform convolution processing on the input features, and then perform global feature fusion with the input features after processing by the multi-feature mixed extraction module to obtain multi-scale fusion features; the multi-feature mixed extraction module is used to perform multi-scale image feature extraction through feature extraction branches with different expansion rates, and then use the PHA module to perform mixed weighting operation, and fuse the mixed weighted results with the features input to the multi-feature mixed extraction module to obtain multi-scale fusion features;

[0132] The image reconstruction unit is used to input the deep features into the super-resolution reconstruction module to obtain a high-quality reconstructed image; the super-resolution reconstruction module is used to adjust the deep features by using different numbers of convolutional layers and sub-pixel convolutional layers to generate a high-quality reconstructed image.

[0133] In one embodiment, the deep feature extraction module includes 4 residual convolution modules and 5 multi-scale feature extraction modules; the shallow feature is the input of the first multi-scale feature extraction module, the first multi-scale feature extraction module, the first residual convolution module, the second multi-scale feature extraction module, the input of the second residual convolution module, the third multi-scale feature extraction module, the input of the third residual convolution module, the fourth multi-scale feature extraction module, the input of the fourth residual convolution module, and the fifth multi-scale feature extraction module are connected in sequence.

[0134] In one of the embodiments, the residual convolution module in the deep feature extraction module includes 8 stacked residual blocks.

[0135] In one of the embodiments, the multi-scale feature extraction module in the deep feature extraction module includes: a convolution group, a multi-feature mixed extraction module and a fusion module; in the multi-scale feature extraction module: the features input to the multi-scale feature extraction module are convolved through the convolution group to obtain convolution features; the convolution features are exchanged with the features input to the multi-scale feature extraction module through a local feature fusion mechanism to obtain local fusion features; the local fusion features are input to the multi-feature mixed extraction module to obtain multi-scale mixed features; the multi-scale mixed features and the features input to the multi-scale feature extraction module are fused through a fusion module to obtain multi-scale fusion features.

[0136] In one embodiment, the fusion module uses a multiplication operation.

[0137] In one embodiment, a multi-feature mixed extraction module includes: four feature extraction branches with different expansion rates and a parallel mixed attention module; each feature extraction branch includes a hole convolution layer; the local fusion feature is input into the multi-feature mixed extraction module to obtain a multi-scale mixed feature, including: the local fusion feature is processed by the hole convolution layer of the first feature extraction branch to obtain a first-scale image feature; the local fusion feature and the first-scale image feature are fused and then processed by the hole convolution layer of the second feature extraction branch to obtain a second-scale image feature; the local fusion feature and the second-scale image feature are fused and then processed by the hole convolution layer of the third feature extraction branch to obtain a third-scale image feature; the local fusion feature and the third-scale image feature are fused and then processed by the hole convolution layer of the fourth feature extraction branch to obtain a fourth-scale image feature; the first-scale image feature, the second-scale image feature, the third-scale image feature and the fourth-scale image feature are input into the parallel mixed attention module to obtain a mixed weighted feature; the mixed weighted feature is fused with the local fusion feature to obtain a multi-scale mixed feature.

[0138] In one embodiment, the parallel hybrid attention module in the deep feature extraction unit is used to obtain a global feature representation by global average pooling the first scale image features, the second scale image features, the third scale image features and the fourth scale image features, so as to capture the long-distance dependencies in the horizontal and vertical directions while retaining the positional relationships, and then model the correlation between channels through a 1×1 convolution operation, and after activation by a sigmoid function, obtain the global features in the horizontal and vertical directions as shown in formulas (8) and (9).

[0139] The global features in the horizontal and vertical directions are concatenated in the second channel dimension and activated by the Softmax function in the channel dimension to obtain the multi-scale channel feature weights in the horizontal and vertical directions; the multi-scale channel feature weights in the horizontal and vertical directions are normalized to obtain the normalized weights in the horizontal and vertical directions; the first-scale image features, the second-scale image features, the third-scale image features, and the fourth-scale image features are respectively multiplied element-by-element with the corresponding normalized weights, and the multiplication results are summed to obtain the mixed weighted features shown in formula (10).

[0140] For the specific definition of the high-quality super-resolution image reconstruction device, please refer to the definition of the high-quality super-resolution image reconstruction method above, which will not be repeated here. Each module in the above-mentioned high-quality super-resolution image reconstruction device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0141] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a high-quality super-resolution image reconstruction method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0142] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above method embodiment when executing the computer program.

[0144] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0145] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0146] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A high-quality super-resolution image reconstruction method, characterized in that: The method comprises: Inputting the input image into a shallow feature extraction module to obtain shallow features; the shallow feature extraction module is used to extract low-frequency information from the image using a single-layer convolution and output shallow features; The shallow features are input into the deep feature extraction module for fine-grained feature extraction to obtain deep features; the deep feature extraction module includes multiple residual convolution modules and multiple multi-scale feature extraction modules; the multi-scale feature extraction module is used to perform convolution processing on the input features and then exchange information with the input features and the local feature fusion mechanism, and then perform global feature fusion with the input features after processing by the multi-feature mixed extraction module to obtain multi-scale fusion features; the multi-feature mixed extraction module is used to perform multi-scale image feature extraction through feature extraction branches with different expansion rates, and then use the PHA module to perform mixed weighting operation, and fuse the mixed weighted results with the features input to the multi-feature mixed extraction module to obtain multi-scale fusion features; The deep features are input into a super-resolution reconstruction module to obtain a high-quality reconstructed image; the super-resolution reconstruction module is used to adjust the deep features by using different numbers of convolutional layers and sub-pixel convolutional layers to generate a high-quality reconstructed image.

2. The method according to claim 1, characterized in that The deep feature extraction module includes 4 residual convolution modules and 5 multi-scale feature extraction modules; The shallow features are the input of the first multi-scale feature extraction module, the first multi-scale feature extraction module, the first residual convolution module, the second multi-scale feature extraction module, the input of the second residual convolution module, the third multi-scale feature extraction module, the input of the third residual convolution module, the fourth multi-scale feature extraction module, the input of the fourth residual convolution module, and the fifth multi-scale feature extraction module, which are connected once.

3. The method according to claim 1, characterized in that: The residual convolution module includes 8 stacked residual blocks.

4. The method according to claim 1, characterized in that: The multi-scale feature extraction module includes: a convolution group, a multi-feature mixed extraction module and a fusion module; in the multi-scale feature extraction module: The features input to the multi-scale feature extraction module are convolved by the convolution group to obtain convolution features; The convolution feature and the feature input to the multi-scale feature extraction module are exchanged with information through a local feature fusion mechanism to obtain a local fusion feature; Inputting the local fusion feature into the multi-feature mixed extraction module to obtain a multi-scale mixed feature; The multi-scale mixed features and the features input into the multi-scale feature extraction module are fused through a fusion module to obtain multi-scale fused features.

5. The method according to claim 4, characterized in that The fusion module adopts multiplication operation.

6. The method according to claim 4, characterized in that The multi-feature hybrid extraction module includes: four feature extraction branches with different expansion rates and a parallel hybrid attention module; each feature extraction branch includes a hole convolution layer; Inputting the local fusion feature into the multi-feature mixed extraction module to obtain a multi-scale mixed feature, including: Processing the local fusion features using the hole convolution layer of the first feature extraction branch to obtain a first-scale image feature; After fusing the local fusion feature and the first-scale image feature, the feature is processed using a dilated convolutional layer of a second feature extraction branch to obtain a second-scale image feature; After fusing the local fusion feature and the second-scale image feature, the feature is processed using a dilated convolutional layer of a third feature extraction branch to obtain a third-scale image feature; After fusing the local fusion feature and the third-scale image feature, the feature is processed using a dilated convolution layer of a fourth feature extraction branch to obtain a fourth-scale image feature; Inputting the first scale image feature, the second scale image feature, the third scale image feature and the fourth scale image feature into the parallel hybrid attention module to obtain a hybrid weighted feature; The mixed weighted features are fused with the local fusion features to obtain multi-scale mixed features.

7. The method according to claim 5, characterized in that Inputting the first scale image feature, the second scale image feature, the third scale image feature and the fourth scale image feature into the parallel hybrid attention module to obtain a hybrid weighted feature, including: The first scale image features, the second scale image features, the third scale image features and the fourth scale image features are subjected to global average pooling to obtain global feature representation, capturing long-distance dependencies in the horizontal and vertical directions while retaining positional relationships. Then, the correlation between channels is modeled through a 1×1 convolution operation, and after activation through a sigmoid function, the global features in the horizontal and vertical directions are obtained as follows: X ni =sigmiod(Conv(AvgPool H (F n '1))) Y nj =sigmiod(Conv(AvgPool W (F n '1))) Among them, X ni ∈R 1×1×W is the global feature in the horizontal direction corresponding to the i-th scale image feature, i = 1, 2, ..., m, Y nj ∈R 1×H×1 is the global feature in the vertical direction corresponding to the j-th scale image feature, j = 1, 2, ..., m; F n ' i is the i-th scale image feature, m is the number of scales of image features of different scales; The global features in the horizontal and vertical directions are concatenated in the second channel dimension and activated by the Softmax function in the channel dimension to obtain multi-scale channel feature weights in the horizontal and vertical directions; Normalizing the multi-scale channel feature weights in the horizontal direction and the vertical direction to obtain normalized weights in the horizontal direction and the vertical direction; The first scale image feature, the second scale image feature, the third scale image feature and the fourth scale image feature are respectively multiplied element by element with the corresponding normalized weights, and the multiplication results are summed to obtain the mixed weighted feature: Among them, F n ' is the mixed weighted feature, W X [:,i,:]∈R 1×1×W , W y [:,i,:]∈R 1×H×1 is the normalized weight in the horizontal and vertical directions; F n is the input feature.

8. A high-quality super-resolution image reconstruction device, characterized in that: The device comprises: A shallow feature extraction unit, used for inputting the input image into a shallow feature extraction module to obtain shallow features; the shallow feature extraction module is used for extracting low-frequency information from the image using a single-layer convolution and outputting shallow features; A deep feature extraction unit is used to input the shallow features into the deep feature extraction module for fine-grained feature extraction to obtain deep features; the deep feature extraction module includes multiple residual convolution modules and multiple multi-scale feature extraction modules; the multi-scale feature extraction module is used to perform convolution processing on the input features, and then perform global feature fusion with the input features after processing by the multi-feature mixed extraction module to obtain multi-scale fusion features; the multi-feature mixed extraction module is used to perform multi-scale image feature extraction through feature extraction branches with different expansion rates, and then use the PHA module to perform mixed weighting operation, and fuse the mixed weighted results with the features input to the multi-feature mixed extraction module to obtain multi-scale fusion features; The image reconstruction unit is used to input the deep features into the super-resolution reconstruction module to obtain a high-quality reconstructed image; the super-resolution reconstruction module is used to adjust the deep features by using different numbers of convolutional layers and sub-pixel convolutional layers to generate a high-quality reconstructed image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images