An image super-resolution method based on hierarchical interactive aggregation

Abstract

The application discloses a layered interactive aggregation-based image super-resolution method, comprising the following steps: acquiring a low-resolution image and extracting shallow features; inputting the shallow features into a deep feature extraction module formed by cascading multi-level layered interactive aggregation blocks; and outputting a high-resolution image through reconstruction and up-sampling; the layered interactive aggregation block of the application adopts a residual structure, first performs multi-expansion-rate deep convolution and gate fusion through a multi-scale gate feedforward unit to extract multi-receptive-field local details; then converts the features into Token through a content-aware global aggregation unit and performs L2 normalization, dynamically groups according to the cosine similarity with the content center, performs in-group interaction and center-guided cross-attention interaction, and realizes long-distance similar content aggregation; the application significantly improves the reconstruction effect of repetitive textures, regular structures and high-frequency edges under the constraint of low parameter quantity, and has excellent precision and efficiency compromise.

Description

Technical Field

[0001] This invention belongs to the fields of digital image processing, computer vision and deep learning technology, and in particular relates to an image super-resolution method based on hierarchical interactive aggregation. Background Technology

[0002] Single-image super-resolution aims to recover a high-resolution image from a single low-resolution image, and has important application value in scenarios such as mobile terminal vision, edge device perception, video surveillance, medical imaging, and remote sensing imaging.

[0003] Existing lightweight super-resolution methods can be broadly classified into two categories: one is lightweight networks based on convolutional neural networks, which reduce the number of model parameters and computational cost through information distillation, feature reuse, or adaptive weighting; the other is to introduce Transformer or Transformer-like structures, which model long-distance dependencies through the interaction between features to improve the recovery capability of complex textures and repetitive structures.

[0004] Existing lightweight Transformer-based image super-resolution methods typically employ global interaction through fixed grid partitioning or fixed spatial partitioning to reduce the computational complexity of traditional global self-attention and achieve a balance between reconstruction accuracy and inference efficiency. However, the inventors discovered in their research that existing technologies still have at least the following shortcomings: Existing lightweight global interaction methods are mostly based on fixed windows or fixed grid divisions. Although they can effectively control computational overhead, they limit the information transfer between distant but similar regions. Especially in image scenes with repetitive textures, regular geometric structures, and strong self-similarity, remote related information cannot be fully utilized.

[0005] The local feature transformation part of existing lightweight networks usually adopts single-scale feedforward networks or single-scale deep convolutions, which have relatively limited receptive fields and make it difficult to simultaneously take into account the joint modeling of large-scale smooth regions and high-frequency detailed textures.

[0006] Existing work typically improves either local multi-scale modeling or global dependency modeling separately, lacking a unified feature extraction structure that organically couples the two under lightweight constraints. Therefore, it is difficult to simultaneously improve the ability to recover short-range structural details and the ability to model long-range content association.

[0007] Therefore, it is necessary to propose a new lightweight image super-resolution technology that can improve the comprehensive modeling ability of local multi-scale texture and cross-regional content similarity information while keeping the model complexity under control. Summary of the Invention

[0008] To address the issues in existing lightweight image super-resolution networks, such as insufficient interaction in distant content-related regions due to fixed spatial partitioning and limited multi-scale detail representation capabilities due to single-scale local feature transformation, a hierarchical interactive aggregation-based image super-resolution method is proposed. This method improves the recovery capabilities of repetitive structures, regular textures, and high-frequency edge details under lightweight computational constraints, while balancing reconstruction accuracy and inference efficiency.

[0009] To achieve the above objectives, the present invention provides the following technical solution: An image super-resolution method based on hierarchical interactive aggregation includes the following steps: S1. Obtain a low-resolution image and obtain initial shallow features through the shallow feature extraction module; S2. Input the shallow features into the deep feature extraction module, which consists of multiple cascaded hierarchical interactive aggregation blocks, to obtain deep reconstructed features; S3. Input the deep reconstruction features into the reconstruction head and upsampling module, and output a high-resolution image; Each of the hierarchical interactive aggregation blocks adopts a residual structure and performs the following processing: S21. Perform local feature preprocessing on the input features to obtain intermediate features; S22. Input the intermediate features into a multi-scale gated feedforward unit. After channel expansion, the input features are divided into multiple parallel branches. Each branch performs a depthwise convolution with a different dilation rate to extract local contextual information under different receptive fields. Then, gated fusion is performed on the outputs of multiple branches and projected back to the original channel dimension to obtain local multi-scale enhanced features. S23. Input the local multi-scale enhanced features into the content-aware global aggregation unit, convert the two-dimensional feature map into a token feature representation, and perform similarity calculation with multiple preset content centers. Based on the similarity results, assign each token to the corresponding content group, perform intra-group interaction on the tokens in each content group, and simultaneously use the content center to perform center-guided global interaction on the tokens in each group. Then, fuse the intra-group interaction results with the center-guided interaction results to obtain the content-aware global enhanced features. S24. Integrate the global enhancement features with other features in the current hierarchical interactive aggregation block through fusion convolution, and output the result of the current hierarchical interactive aggregation block; multiple hierarchical interactive aggregation blocks are concatenated to obtain deep reconstruction features.

[0010] The present invention further defines the technical solution as follows: Preferably, the multi-scale gated feedforward unit includes a 1×1 convolutional channel expansion layer, multiple parallel deep convolutional branches, a gated fusion unit, and a channel projection layer, and each deep convolutional branch is set with a different dilation rate to introduce a multi-scale receptive field under a limited increase in the number of parameters.

[0011] Preferably, the multi-scale gated feedforward unit has 3 parallel branches, and the 3×3 depth convolution dilation rates of the three branches are 1, 2, and 3, respectively, corresponding to receptive fields of 3×3, 5×5, and 7×7, respectively. The gated fusion unit is used to perform multiplicative fusion or equivalent content-related fusion on the outputs of multiple scale branches, so that the local responses of different scales can be modulated to each other, thereby enhancing the joint expression ability of smooth regions, fine textures, and repetitive structures.

[0012] Preferably, the content-aware global aggregation unit includes: The tokenization module flattens the two-dimensional spatial feature map into a one-dimensional sequence of token features, preparing for subsequent global interactions; The normalization module normalizes the token features to ensure the stability and consistency of subsequent similarity calculations. The content center storage module normalizes the token features to ensure the stability and consistency of subsequent similarity calculations. The similarity calculation module calculates the feature similarity between each token feature and each content center, providing a basis for content grouping; The grouping and allocation module assigns tokens to corresponding content groups based on the similarity between the token and the content center, so that tokens that are spatially far apart but have similar content are grouped into the same group. The group interaction module performs feature interactions within each content group, enhancing the local consistency and detailed correlation of tokens within the same group; The central guidance interaction module uses the content center as the global guide, performs global interactions constrained by the central system on each group of tokens, and establishes cross-group and long-distance content associations; The aggregation output module merges the interaction results within the group with the global interaction results guided by the center, restores the token sequence to the form of a two-dimensional spatial feature map, and outputs the global enhanced features.

[0013] Preferably, the content center in the content-aware global aggregation unit is initialized as a learnable parameter. During the training phase, it is not updated directly through gradient backpropagation, but rather updated using an exponential moving average (EMA) based on the current batch of token statistics. During the inference phase, the content center is fixed and not dynamically updated.

[0014] Preferably, the shallow feature extraction module, the deep feature extraction module, and the reconstruction head can all be implemented using a convolutional structure, and the deep feature extraction module contains N cascaded hierarchical interactive aggregation blocks, where N is a positive integer.

[0015] Preferably, the deep feature extraction module includes 4-5 cascaded hierarchical interactive aggregation blocks, with a network basic channel dimension of 64 and a window size of 8.

[0016] An image super-resolution device based on hierarchical interactive aggregation, comprising: The input module acquires and preprocesses low-resolution images, providing standard input. The shallow feature extraction module extracts basic shallow features such as edges and textures through convolution, completing the mapping from the pixel domain to the feature domain; The deep feature extraction module consists of multiple cascaded hierarchical interactive aggregation blocks, which extract local multi-scale details and cross-regional content similarity features layer by layer; The reconstruction and upsampling module performs detailed reconstruction of deep features and performs resolution upscaling through PixelShuffle; The output module outputs the final high-resolution image. The aforementioned hierarchical interactive aggregation block includes: The local feature preprocessing unit normalizes and aligns the input features. Multi-scale gated feedforward unit, multi-receptive field feature extraction and gated fusion, enhances local details; Content-aware global aggregation units allow for interactive grouping based on content similarity, breaking through the limitations of fixed windows; By fusing convolutional units, local and global features are integrated to output optimized features; Residual connection units preserve the original information and improve training stability.

[0017] Beneficial effects:

[0018] This invention achieves the organic integration of local multi-scale feature enhancement and content-aware global feature aggregation under lightweight computational constraints through a hierarchical interactive aggregation architecture. This invention employs multi-scale gated feedforward units and constructs a multi-receptive field modeling mechanism through multi-branch convolutions with different dilation rates. Under the premise of minimal parameter increment, it significantly improves the network's ability to jointly express smooth regions, edge contours, and high-frequency textures, enabling the image to simultaneously maintain structural integrity and detail clarity during reconstruction. This effectively improves the problems of insufficient receptive field and weak detail recovery ability of single-scale networks.

[0019] This invention proposes a content-aware global aggregation unit, which breaks through the spatial limitations of traditional fixed windows and fixed grids. It dynamically groups based on the similarity of feature content, enabling efficient information interaction between regions that are spatially far apart but have similar content. It makes full use of the long-range self-similarity characteristics of images, significantly improving the reconstruction effect of repetitive textures, regular geometric structures and line patterns, and solving the defect of traditional lightweight Transformers that cannot effectively model long-distance related information.

[0020] This invention employs a hierarchical interactive aggregation structure that first performs local multi-scale transformation and then global content-aware aggregation. This allows local short-range information and global long-range information to work collaboratively within a unified lightweight framework, improving reconstruction results in scenarios with complex geometric structures, repetitive lines, and regular textures. Meanwhile, the content center adopts a mechanism of EMA sliding update during the training phase and fixed update during the inference phase, which ensures the stability of content grouping and effectively controls computational overhead and inference latency, making it suitable for resource-constrained scenarios such as mobile terminals and edge devices. Attached Figure Description

[0021] Figure 1 This diagram illustrates the performance-complexity trade-off between lightweight image super-resolution methods on the Urban100×4 dataset. Figure 2 This is a schematic diagram of the overall network structure of the present invention; Figure 3 This is a schematic diagram of the structure of the hierarchical interactive aggregation block HIA of the present invention; Figure 4 This is a schematic diagram of the structure of the multi-scale gated feedforward network (MSFN) of the present invention; Figure 5 A visualization comparison of the super-resolution reconstruction results from different methods; Figure 6 A schematic diagram of the reconstruction error map visualization results for different methods; Figure 7 A schematic diagram showing the LAM visualization results of different methods on the img_004 image in the Urban100 dataset; Figure 8 The diagram shows the performance-complexity trade-off analysis of the method of this invention on the Urban100×4 dataset, where: (a) is the performance-complexity trade-off analysis at the module variant level; and (b) is the performance-complexity trade-off analysis at the model scale level. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1 This embodiment provides an image super-resolution method based on hierarchical interactive aggregation, including the following steps: A low-resolution image is acquired, and initial shallow features are extracted using a shallow feature extraction module composed of 3×3 convolutions, thus completing the mapping from the pixel domain to the feature domain. The shallow features are input into the deep feature extraction module, which consists of 5 cascaded hierarchical interactive aggregation blocks. The basic channel dimension of the network is set to 64, and the window size is 8. Each hierarchical interactive aggregation block adopts a residual structure, sequentially performing local feature preprocessing, multi-scale gated feedforward enhancement, content-aware global aggregation, fusion convolution and residual addition, and outputting enhanced deep features step by step; The deep reconstruction features output by the multi-level hierarchical interactive aggregation block are first fused by 1×1 convolution, and then fed into the reconstruction module consisting of 3×3 reconstruction convolution and PixelShuffle subpixel upsampling to complete the ×4 ratio super-resolution reconstruction. Output the final high-resolution image.

[0024] Example 2 This embodiment provides a detailed description of the workflow of the hierarchical interactive aggregation block, which is executed as follows: Normalization and channel alignment preprocessing are performed on the input features to obtain intermediate features suitable for subsequent processing; The intermediate features are input into the multi-scale gated feedforward unit, and channel expansion is performed by 1×1 convolution. The system is divided into three parallel branches along the channel dimension. Each branch is subjected to 3×3 depth convolution with different dilation rates of 1, 2 and 3 to extract 3×3, 5×5 and 7×3 multi-receptive field features. After each branch output is activated by the GELU activation function, channel-wise multiplicative gating fusion is performed, and then 1×1 convolution is projected back to the original channel dimension to obtain local multi-scale enhanced features. The local multi-scale enhanced features are input into the content-aware global aggregation unit, first flattened into a token sequence and then L2 normalized. The cosine similarity is calculated with the 8 content centers, and the content is grouped according to the maximum similarity. Perform intra-group self-attention interaction on each group of tokens, and perform center-guided cross-attention interaction with the content center as key K and value V. After fusing the two results, reshape them into the original spatial size feature map. The global enhancement features are integrated through a fusion convolution, and then added to the input features using residuals to output the final features of the current block.

[0025] Example 3 This embodiment describes the content-aware global aggregation unit, as follows: Tokenization and Feature Flattening The input two-dimensional spatial feature map is flattened into a one-dimensional token sequence according to its spatial location, completing the transformation from a grid structure to a sequence structure, and providing a unified representation for subsequent global interactions; Feature normalization processing L2 normalization is performed on the flattened token features to keep the feature vector magnitude consistent, thereby improving the stability and reliability of subsequent similarity calculations and avoiding content grouping failure due to differences in feature magnitude. Content center and similarity calculation Each unit maintains a preset number of content centers to represent typical content prototypes of the image. For each token, the cosine similarity between it and each content center is calculated, and the token is assigned to the corresponding content group according to the principle of maximum similarity, so that tokens that are spatially far apart but have similar content are grouped into the same group; Centralized global interaction For each token in a content group, query, key, and value features are constructed. Self-attention interaction is performed within the group to enhance the consistency and detailed correlation of features within the same group and improve the expression accuracy of similar content. Intra-group feature interaction Using each group's Token as the query Q, and the content center as the key K and value V, the cross-attention interaction guided by the execution center enables cross-group, long-distance global information transmission through the content center, breaking through the limitations of a fixed spatial window. Feature fusion and dimensionality restoration The interaction results within the group are weighted and fused with the global interaction results guided by the center to obtain the aggregated global enhanced features; then the token sequence is reshaped into a two-dimensional feature map according to the original spatial size to adapt to the input format of subsequent modules, thus completing the content-aware global feature aggregation process. Content Center Updates and Reasoning Initialization: The content center is initialized with learnable parameters, and the dimensions are consistent with the token features; Training phase: Instead of updating directly through gradient backpropagation, the update is based on the statistical characteristics of the current batch of tokens and uses the exponential moving average (EMA) method, with the momentum coefficient set to 0.99; Inference phase: The content center that has been trained is used permanently and is no longer dynamically updated to ensure stable content grouping and controllable inference latency; Grouping rules: Tokens are assigned to corresponding content groups based on the maximum cosine similarity with the content center, so that features that are spatially distant but have similar content are grouped into the same group for interaction.

[0026] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0027] Example 4 This embodiment, while maintaining the core structure of the invention, constructs a lighter model variant by reducing the number of hierarchical interaction aggregation blocks in the deep feature extraction module, specifically as follows: The number of hierarchical interactive aggregation blocks in the deep feature extraction module has been reduced from 5 to 4; Maintain the channel dimension of 64, the number of content centers of 8, the multi-scale branching and the expansion rate unchanged; The number of model parameters was further reduced from 1.26M to approximately 1.02M, and the inference latency was further shortened. It can still achieve a PSNR of over 26.62dB on Urban100×4 super-resolution tasks, maintaining excellent reconstruction performance under extremely low complexity, and can be directly deployed on mobile terminals and embedded devices.

[0028] Example 5 This embodiment discloses several equivalent replacement methods: The number of branches of the multi-scale gated feedforward unit can be set to 2, 3, or 4, and the expansion rate can be combined with 1 / 2 / 4, 1 / 3 / 5, etc. The number of content centers can be set to 4, 8, or 16, and the grouping granularity can be flexibly adjusted according to the computing power budget; Upsampling can be replaced by equivalent schemes such as deconvolution, bilinear interpolation plus convolution; The super-resolution ratio can be expanded to ×2, ×3, ×4, and ×8 to suit different resolution enhancement needs; The activation function can be replaced with functions such as ReLU and SiLU, which have equivalent nonlinear expressive power.

[0029] Example 6 This embodiment provides an image super-resolution device based on hierarchical interactive aggregation, including: The input module acquires and preprocesses low-resolution images, providing standard input. The shallow feature extraction module extracts basic shallow features such as edges and textures through convolution, completing the mapping from the pixel domain to the feature domain; The deep feature extraction module consists of multiple cascaded hierarchical interactive aggregation blocks, which extract local multi-scale details and cross-regional content similarity features layer by layer; The reconstruction and upsampling module performs detailed reconstruction of deep features and performs resolution upscaling through PixelShuffle; The output module outputs the final high-resolution image. The aforementioned hierarchical interactive aggregation block includes: The local feature preprocessing unit normalizes and aligns the input features. Multi-scale gated feedforward unit, multi-receptive field feature extraction and gated fusion, enhances local details; Content-aware global aggregation units allow for interactive grouping based on content similarity, breaking through the limitations of fixed windows; By fusing convolutional units, local and global features are integrated to output optimized features; Residual connection units preserve the original information and improve training stability.

[0030] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An image super-resolution method based on hierarchical interactive aggregation, characterized in that, Includes the following steps: S1. Obtain a low-resolution image and obtain initial shallow features through the shallow feature extraction module; S2. Input the shallow features into the deep feature extraction module, which consists of multiple cascaded hierarchical interactive aggregation blocks, to obtain deep reconstructed features; S3. Input the deep reconstruction features into the reconstruction head and upsampling module, and output a high-resolution image; Each of the hierarchical interactive aggregation blocks adopts a residual structure and performs the following processing: S21. Perform local feature preprocessing on the input features to obtain intermediate features; S22. Input the intermediate features into a multi-scale gated feedforward unit. After channel expansion, the input features are divided into multiple parallel branches. Each branch performs a depthwise convolution with a different dilation rate to extract local contextual information under different receptive fields. Then, gated fusion is performed on the outputs of multiple branches and projected back to the original channel dimension to obtain local multi-scale enhanced features. S23. Input the local multi-scale enhanced features into the content-aware global aggregation unit, convert the two-dimensional feature map into a token feature representation, and perform similarity calculation with multiple preset content centers. Based on the similarity results, assign each token to the corresponding content group, perform intra-group interaction on the tokens in each content group, and simultaneously use the content center to perform center-guided global interaction on the tokens in each group. Then, fuse the intra-group interaction results with the center-guided interaction results to obtain the content-aware global enhanced features. S24. Integrate the global enhancement features with other features in the current hierarchical interactive aggregation block through fusion convolution, and output the result of the current hierarchical interactive aggregation block; multiple hierarchical interactive aggregation blocks are concatenated to obtain deep reconstruction features.

2. The image super-resolution method based on hierarchical interactive aggregation according to claim 1, characterized in that, The multi-scale gated feedforward unit includes a 1×1 convolutional channel expansion layer, multiple parallel deep convolutional branches, a gated fusion unit, and a channel projection layer. Each deep convolutional branch is configured with a different dilation rate to introduce a multi-scale receptive field under a limited increase in the number of parameters.

3. The image super-resolution method based on hierarchical interactive aggregation according to claim 2, characterized in that, The multi-scale gated feedforward unit has 3 parallel branches, and the 3×3 depth convolution dilation rates of the three branches are 1, 2, and 3, respectively, corresponding to receptive fields of 3×3, 5×5, and 7×7. The gated fusion unit is used to perform multiplicative fusion or equivalent content-related fusion on the outputs of multiple scale branches, so that the local responses of different scales can be modulated to each other, thereby enhancing the joint expression ability of smooth regions, fine textures and repetitive structures.

4. The image super-resolution method based on hierarchical interactive aggregation according to claim 1, characterized in that, The content-aware global aggregation unit includes: The tokenization module flattens the two-dimensional spatial feature map into a one-dimensional sequence of token features, preparing for subsequent global interactions; The normalization module normalizes the token features to ensure the stability and consistency of subsequent similarity calculations. The content center storage module normalizes the token features to ensure the stability and consistency of subsequent similarity calculations. The similarity calculation module calculates the feature similarity between each token feature and each content center, providing a basis for content grouping; The grouping and allocation module assigns tokens to corresponding content groups based on the similarity between the token and the content center, so that tokens that are spatially far apart but have similar content are grouped into the same group. The group interaction module performs feature interactions within each content group, enhancing the local consistency and detailed correlation of tokens within the same group; The central guidance interaction module uses the content center as the global guide, performs global interactions constrained by the central system on each group of tokens, and establishes cross-group and long-distance content associations; The aggregation output module merges the interaction results within the group with the global interaction results guided by the center, restores the token sequence to the form of a two-dimensional spatial feature map, and outputs the global enhanced features.

5. The image super-resolution method based on hierarchical interactive aggregation according to claim 2, characterized in that, The content center in the content-aware global aggregation unit is initialized as a learnable parameter. During the training phase, it is not updated directly through gradient backpropagation, but is updated using exponential moving average (EMA) based on the statistics of the current batch of tokens. During the inference phase, the content center is fixed and is not dynamically updated.

6. The image super-resolution method based on hierarchical interactive aggregation according to claim 1, characterized in that, The shallow feature extraction module, the deep feature extraction module, and the reconstruction head can all be implemented using a convolutional structure, and the deep feature extraction module contains N cascaded hierarchical interactive aggregation blocks, where N is a positive integer.

7. The image super-resolution method based on hierarchical interactive aggregation according to claim 1, characterized in that, The deep feature extraction module contains 4-5 cascaded hierarchical interactive aggregation blocks, with a basic network channel dimension of 64 and a window size of 8.

8. An image super-resolution device based on hierarchical interactive aggregation, implementing the method described in claims 1-7, characterized in that: include: The input module acquires and preprocesses low-resolution images, providing standard input. The shallow feature extraction module extracts basic shallow features such as edges and textures through convolution, completing the mapping from the pixel domain to the feature domain; The deep feature extraction module consists of multiple cascaded hierarchical interactive aggregation blocks, which extract local multi-scale details and cross-regional content similarity features layer by layer; The reconstruction and upsampling module performs detailed reconstruction of deep features and performs resolution upscaling through PixelShuffle; The output module outputs the final high-resolution image. The aforementioned hierarchical interactive aggregation block includes: The local feature preprocessing unit normalizes and aligns the input features. Multi-scale gated feedforward unit, multi-receptive field feature extraction and gated fusion, enhances local details; Content-aware global aggregation units allow for interactive grouping based on content similarity, breaking through the limitations of fixed windows; By fusing convolutional units, local and global features are integrated to output optimized features; Residual connection units preserve the original information and improve training stability.

Images