Image super-resolution method and device based on adaptive semantic guidance network
By using a dictionary-based self-attention mechanism with an adaptive semantically guided network and a one-way Mamba scanning mechanism, the problems of long-distance dependency modeling and computational complexity in image super-resolution methods are solved, improving the performance and computational efficiency of image super-resolution, and achieving higher reconstruction quality and lower computational cost.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG NORMAL UNIV
- Filing Date
- 2026-01-14
- Publication Date
- 2026-05-01
AI Technical Summary
Existing image super-resolution methods have limitations in terms of long-distance pixel dependency modeling capabilities and computational complexity, especially in terms of causal modeling and computational efficiency based on the Transformer architecture and the selective state-space model (Mamba) in image super-resolution tasks.
An adaptive semantic guidance network is adopted, which introduces external prior information through a dictionary-based hybrid self-attention mechanism. It combines a one-way Mamba scanning mechanism and a multi-head self-attention mechanism to perform semantic guidance pixel rearrangement and local interaction. The model is optimized using an adaptive dictionary update strategy to generate high-resolution images.
It significantly improves image super-resolution performance, reduces computational complexity, enhances global information utilization efficiency and the model's semantic perception capabilities, achieves higher peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), and reduces the number of model parameters.
Smart Images
Figure CN121961844A_ABST
Abstract
Description
An image super-resolution method and apparatus based on an adaptive semantic guidance network Technical Field
[0001] This invention relates to the field of image processing technology, and more specifically to an image super-resolution method and apparatus based on an adaptive semantic guidance network. Background Technology
[0002] Image super-resolution (ISR) technology aims to recover high-resolution (HR) images from low-resolution (LR) images to improve the visual quality and detail of images. This technology has broad application prospects in many fields such as medical imaging, satellite image processing, and video enhancement.
[0003] Traditional image super-resolution methods mainly rely on convolutional neural networks (CNNs), but their convolution operations are inherently local, limiting their ability to model long-distance pixel dependencies.
[0004] In recent years, the Transformer architecture has been introduced into the field of image super-resolution. Although it has improved the performance of the model, its computational complexity is high, especially when processing high-resolution images. The secondary computational complexity makes it impractical for real-world applications.
[0005] Selective state-space model (Mamba), as an emerging image restoration backbone network, has shown great potential in balancing global perception and computational efficiency; however, the causal modeling limitations of Mamba and the computational complexity and redundancy issues brought about by the multi-directional scanning strategy limit its performance in image super-resolution tasks.
[0006] Therefore, overcoming the limitations of existing state-space models in causal modeling and computational efficiency in image super-resolution tasks is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0007] In view of this, the present invention provides an image super-resolution method and apparatus based on an adaptive semantic guidance network to solve some of the technical problems mentioned in the background art.
[0008] To achieve the above objectives, the present invention adopts the following technical solution: an image super-resolution method based on an adaptive semantic guidance network, comprising the following steps: S1. Input a low-resolution image and initialize an external dictionary, using a dictionary-mixed self-attention mechanism to introduce external prior information and generate a similarity map containing image semantic information; S2. Perform semantically guided pixel rearrangement on the input low-resolution image according to the similarity map generated in step S1, and use a one-way Mamba scanning mechanism to model the state space of the rearranged sequence, outputting a feature map with global dependencies; S3. Use a multi-head self-attention mechanism (SW-MHSA) based on a shift window to enhance local interactions on the input low-resolution image, obtaining locally enhanced features; S4. Add the feature results from steps S1 to S3 element-wise with the original input features to fuse the information from all processing steps, and then further extract and enhance features through a feedforward network; S5. Based on an adaptive dictionary update strategy, use the output features and similarity maps of each layer to update the dictionary of the previous layer; S6. Generate the final SR high-resolution image by performing pixel shuffling on the features after passing through multiple feature extraction networks.
[0009] Preferably, step S1 specifically includes: S11. Generating a query vector from the input image using a linear transformation matrix, and introducing an external dictionary, which is then formalized into a key vector and a value vector using a linear transformation matrix; S12. Generating a similarity graph matrix by calculating the cosine similarity between the query vector and the key vector; S13. Using the similarity graph matrix to weight the value vector to obtain an enhanced value vector.
[0010] Preferably, step S11 specifically includes:
[0011]
[0012]
[0013] Where X is the input image vector and D is the external dictionary. , and These are linear transformation matrices, used to generate query vectors. Key vector Sum value vector Step S12, similarity graph matrix for:
[0014] in, For query vector, For key vectors, For vector dimensions, To adjust the learnable parameters of the similarity value range, the softmax function is used to convert the vector into a probability distribution; step S13 specifically involves:
[0015] in, This represents matrix multiplication.
[0016] Preferably, step S2 specifically includes: S21. Applying positional encoding to the input low-resolution image features to preserve the original structural information; S22. Determining the semantic category of each pixel based on the similarity map matrix; S23. Rearranging the pixels according to the semantic category, arranging semantically similar pixels together to form a semantically consistent sequence; S24. Inputting the semantically consistent sequence into the state space model SSM for state space modeling, using reverse pixel rearrangement RSGA to reshape the semantic space sequence into a spatial feature map, and outputting a high-resolution image.
[0017] Preferably, step S24, the state-space model SSM, is as follows:
[0018]
[0019] in, For the image features at the i-th time step, Let A be the image features from the previous time step, and let A be the state transition matrix. The input is the low-resolution image information at the current time step, and C and D are the convolution kernels or other linear transformations used to extract and reconstruct image details from the feature map. This is the high-resolution image reconstructed at the i-th time step.
[0020] Preferably, in step S3, the multi-head self-attention mechanism specifically is as follows:
[0021]
[0022] in, To output a linear transformation matrix, Concat is a concatenation operation used to merge the outputs of multiple attention heads.
[0023] Preferably, in step S5, the adaptive dictionary update strategy is as follows: through an adaptive combination process, the newly generated similar graph elements are combined with the dictionary of the previous layer to form an updated dictionary for the next layer;
[0024] in, For the updated dictionary, This is a parameter between 0 and 1, used to control the ratio of merging the old and new dictionaries; The function is used to convert a vector into a probability distribution. Represents the similarity graph matrix Normalization is performed to ensure numerical stability. Let be the transpose feature matrix of the (i+1)th layer. It is the dictionary of the current layer.
[0025] An image super-resolution system based on an adaptive semantic guidance network, comprising: a data acquisition module, a DFSA module, a SGM module, an MHSA module, a feature enhancement module, an ADU module, and a pixel shuffling module; the data acquisition module is used to acquire low-resolution images; the DFSA module is used to initialize an external dictionary and introduce external prior information into the low-resolution image using a dictionary shuffling self-attention mechanism to generate a similarity map containing semantic information of the image; the SGM module is used to perform semantically guided pixel rearrangement on the input low-resolution image based on the generated similarity map, and to perform state-space modeling on the rearranged sequence using a one-way Mamba scanning mechanism. The system outputs a feature map with global dependencies; the MHSA module enhances local interactions in the input low-resolution image using the multi-head self-attention mechanism SW-MHSA based on a shift window, resulting in locally enhanced features; the feature enhancement module adds the feature results from the DFSA, SGM, and MHSA modules element-wise with the original input features to fuse all processed information, and then further extracts and enhances features through a feedforward network; the MHSA module updates the dictionary of the previous layer using the output features and similar maps of each layer based on an adaptive dictionary update strategy; and the pixel shuffling module performs a pixel shuffling operation on the features after multi-layer feature extraction networks to generate the final SR high-resolution image.
[0026] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned image super-resolution method based on an adaptive semantic guidance network.
[0027] A processing terminal includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the aforementioned image super-resolution method based on an adaptive semantic guidance network.
[0028] As can be seen from the above technical solutions, compared with the prior art, this invention discloses an image super-resolution method and apparatus based on an adaptive semantically guided network. Through a semantically guided unidirectional Mamba scanning mechanism and a dictionary-based hybrid self-attention mechanism, it significantly improves the performance of image super-resolution while reducing computational complexity. Specifically: the semantically guided pixel rearrangement and unidirectional scanning mechanism effectively alleviate the long-distance attenuation problem in the traditional Mamba model, improving the utilization efficiency of global information; the introduction of a dictionary-based hybrid self-attention mechanism integrates external prior knowledge into image analysis, further enhancing the model's semantic perception capability; an adaptive dictionary update strategy is proposed to dynamically adjust the dictionary to adapt to different input images, improving the model's generalization ability. Experimental results show that this invention achieves a better trade-off between reconstruction performance and computational efficiency on multiple public benchmark datasets. Compared with the prior art, this invention exhibits higher peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) in image super-resolution tasks, while significantly reducing the number of model parameters and computational complexity. It has broad application prospects in fields such as medical imaging, satellite image processing, and video enhancement. Attached Figure Description
[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0030] Figure 1 is a schematic diagram of an image super-resolution method based on an adaptive semantic guidance network provided by the present invention; Figure 2 is a schematic diagram of a quantitative comparison of PSNR and SSIM between the present invention and existing methods on multiple public benchmark datasets; Figure 3 is a schematic diagram of a qualitative comparison of the present invention with existing methods in terms of vision. Detailed Implementation
[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0032] This invention discloses an image super-resolution method based on an adaptive semantic guidance network, as shown in Figure 1, comprising the following steps: S1. Input a low-resolution image and initialize an external dictionary. Use the dictionary hybrid self-attention mechanism (DFSA) to introduce external prior information and generate a similarity map containing image semantic information; S2. Perform semantically guided pixel rearrangement on the input low-resolution image based on the similarity map generated in step S1, and use a one-way Mamba scanning mechanism to model the state space of the rearranged sequence, capture global dependencies, and output a feature map with global dependencies; S3. Use the multi-head self-attention mechanism (SW-MHSA) based on a shift window to enhance local interactions on the input low-resolution image, improve image super-resolution, and obtain locally enhanced features; S4. Add the feature results from steps S1 to S3 to the original input features element-wise to fuse the information from all processing steps, and then further extract and enhance features through a feedforward network; S5. Based on the adaptive dictionary update strategy (ADU), use the output features and similarity maps of each layer to update the dictionary of the previous layer; S6. After passing through multiple feature extraction networks, perform pixel shuffling to generate the final SR high-resolution image.
[0033] To further implement the above technical solution, the specific content of step S1 is as follows: S11. Generate a query vector from the input image through a linear transformation matrix, and introduce an external dictionary. Formulate the external dictionary into a key vector and a value vector through a linear transformation matrix; S12. Generate a similarity graph matrix by calculating the cosine similarity between the query vector and the key vector; S13. Use the similarity graph matrix to weight the value vector to obtain an enhanced value vector.
[0034] To further implement the above technical solution, step S11 is as follows:
[0035]
[0036]
[0037] Where X is the input image vector and D is the external dictionary. , and These are linear transformation matrices, used to generate query vectors. Key vector Sum value vector Step S12 involves calculating the similarity between the input features and the external dictionary, generating a similarity graph matrix for subsequent semantic-guided pixel rearrangement. for:
[0038] in, For query vector, For key vectors, For vector dimensions, To adjust the learnable parameters of the similarity value range, the softmax function is used to convert the vector into a probability distribution; step S13 involves using the similarity graph matrix... Weighted vector Thus, we can acquire and utilize external prior information, specifically:
[0039] in, This represents matrix multiplication.
[0040] To further implement the above technical solution, step S2 proposes a semantically guided unidirectional Mamba scanning mechanism (SGM). This mechanism reconstructs the image by semantically guiding the arrangement of the semantically linked image (SGA) before performing a unidirectional Mamba scan. Specifically: S21. Apply positional encoding to the features of the input low-resolution image to preserve the original structural information; S22. Determine the semantic category of each pixel based on the similarity graph matrix in the DFSA; S23. Rearrange the pixels according to their semantic categories, grouping semantically similar pixels together to form a semantically consistent sequence; S24. Input the semantically consistent sequence into the state-space model (SSM) for state-space modeling, and use the reverse pixel rearrangement (RSGA) to reshape the semantic space sequence into a spatial feature map, outputting a high-resolution image.
[0041] To further implement the above technical solution, step S24, the state-space model SSM, is specifically as follows:
[0042]
[0043] in, For the image features at the i-th time step, Let A be the image features from the previous time step, and let A be the state transition matrix. The input is the low-resolution image information at the current time step, and C and D are the convolution kernels or other linear transformations used to extract and reconstruct image details from the feature map. This is the high-resolution image reconstructed at the i-th time step.
[0044] To further implement the above technical solution, step S3, the multi-head self-attention mechanism, specifically is as follows:
[0045]
[0046] in, To output a linear transformation matrix, Concat is a concatenation operation used to merge the outputs of multiple attention heads, thereby enhancing local interactions and improving the performance of image super-resolution through a multi-head self-attention mechanism.
[0047] In this embodiment, step S4, the feature fusion formula is:
[0048] The feed-forward network (FFN) consists of multiple fully connected layers, used to further extract and enhance features. The specific formula is as follows: .
[0049] To further implement the above technical solution, in step S5, the adaptive dictionary update strategy is as follows: through an adaptive combination process, the newly generated similar graph elements are combined with the dictionary of the previous layer to form an updated dictionary for the next layer;
[0050] in, For the updated dictionary, This is a parameter between 0 and 1, used to control the ratio of merging the old and new dictionaries; The function is used to convert a vector into a probability distribution. Represents the similarity graph matrix Normalization is performed to ensure numerical stability. Let be the transpose feature matrix of the (i+1)th layer. It is the dictionary of the current layer.
[0051] In this embodiment, the pixel shuffling operation in step S6 is as follows: the features after passing through multiple ASGMamba blocks are aggregated using a 3×3 convolutional layer, and a residual connection is established between its output and the shallow features. The reconstruction module uses a 3×3 convolutional layer to aggregate features and uses a shuffle layer channel rearrangement to obtain the final SR output image.
[0052] In another embodiment, the proposed image super-resolution method based on an adaptive semantic guidance network is quantitatively compared with the PSNR and SSIM of some existing methods on five public benchmark sets, as shown in Figure 2. The experimental results show that the proposed method achieves a better trade-off between reconstruction performance and computational efficiency on multiple public benchmark datasets, with higher PSNR and SSIM values compared to existing methods, while using fewer parameters. In addition, the visual results of the proposed method are compared with existing methods based on CNN, ViT, and Mamba on the ×4Urban100 dataset, as shown in Figure 3. Existing methods suffer from blur artifacts and distorted lines, while the proposed method can recover images more accurately.
[0053] An image super-resolution system based on an adaptive semantic guidance network (ASN) includes the following modules: a data acquisition module, a DFSA module, a SGM module, an MHSA module, feature enhancement, an ADU module, and a pixel shuffling module. The data acquisition module acquires low-resolution images. The DFSA module initializes an external dictionary and uses a dictionary shuffling self-attention mechanism to introduce external prior information into the low-resolution image, generating a similarity graph containing semantic information. The SGM module performs semantically guided pixel rearrangement on the input low-resolution image based on the generated similarity graph and uses a one-way Mamba scanning mechanism to model the state space of the rearranged sequence. The output features a globally dependent feature map; the MHSA module enhances local interactions in the input low-resolution image using the multi-head self-attention mechanism SW-MHSA based on a shift window, resulting in locally enhanced features; feature enhancement adds the feature results from the DFSA, SGM, and MHSA modules element-wise with the original input features to fuse all processed information, and then further extracts and enhances features through a feedforward network; the MHSA module updates the dictionary of the previous layer using the output features and similarity maps of each layer based on an adaptive dictionary update strategy; and the pixel shuffling module performs a pixel shuffling operation on the features after multi-layer feature extraction networks to generate the final SR high-resolution image.
[0054] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements an image super-resolution method based on an adaptive semantic guidance network.
[0055] A processing terminal includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements an image super-resolution method based on an adaptive semantic guidance network.
[0056] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.
[0057] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An image super-resolution method based on an adaptive semantic guidance network, characterized in that, Includes the following steps: S1. Input a low-resolution image and initialize an external dictionary. Use the dictionary-mixed self-attention mechanism to introduce external prior information and generate a similarity graph containing image semantic information. S2. The input low-resolution image is semantically guided pixel rearrangement based on the similarity map generated in step S1, and the state space model of the rearranged sequence is performed using a one-way Mamba scanning mechanism to output a feature map with global dependencies; S3. The input low-resolution image is enhanced with a multi-head self-attention mechanism SW-MHSA based on a shift window to obtain locally enhanced features. S4. Add the feature results from steps S1 to S3 element-wise to the original input features to fuse the information from all processing steps, and then further extract and enhance the features through a feedforward network; S5. Based on the adaptive dictionary update strategy, use the output features and similar maps of each layer to update the dictionary of the previous layer; S6. After the features have passed through the multi-layer feature extraction network, perform pixel shuffling to generate the final SR high-resolution image.
2. The image super-resolution method based on an adaptive semantic guidance network according to claim 1, characterized in that, The specific content of step S1 is as follows: S11. Generate a query vector from the input image using a linear transformation matrix, and introduce an external dictionary. Formulate the external dictionary into a key vector and a value vector using a linear transformation matrix; S12. Generate a similarity graph matrix by calculating the cosine similarity between the query vector and the key vector; S13. Use the similarity graph matrix to weight the value vector to obtain an enhanced value vector.
3. The image super-resolution method based on an adaptive semantic guidance network according to claim 2, characterized in that, Step S11 is as follows: Where X is the input image vector and D is the external dictionary. 、 and These are linear transformation matrices, used to generate query vectors. Key vector Sum value vector Step S12, similarity graph matrix for: in, For query vector, For key vectors, For vector dimensions, To adjust the learnable parameters of the similarity value range, the softmax function is used to convert the vector into a probability distribution; step S13 specifically involves: in, This represents matrix multiplication.
4. The image super-resolution method based on an adaptive semantic guidance network according to claim 1, characterized in that, Step S2 consists of the following steps: S21. Apply positional encoding to the input low-resolution image features to preserve the original structural information; S22. Determine the semantic category of each pixel based on the similarity map matrix; S23. Rearrange the pixels according to their semantic categories, grouping semantically similar pixels together to form a semantically consistent sequence; S24. Input the semantically consistent sequence into the state space model (SSM) for state space modeling, and use reverse pixel rearrangement (RSGA) to reshape the semantic space sequence into a spatial feature map, outputting a feature map with global dependencies.
5. The image super-resolution method based on an adaptive semantic guidance network according to claim 4, characterized in that, Step S24, the state-space model SSM is as follows: in, For the image features at the i-th time step, Let A be the image features from the previous time step, and let A be the state transition matrix. The input is the low-resolution image information at the current time step, and C and D are the convolution kernels or other linear transformations used to extract and reconstruct image details from the feature map. This is the high-resolution image reconstructed at the i-th time step.
6. The image super-resolution method based on an adaptive semantic guidance network according to claim 1, characterized in that, Step S3 specifically involves: first, dividing the feature map into multiple non-overlapping local windows, and independently calculating multi-head self-attention within each window; then, cyclically shifting the windows to form new window divisions; finally, calculating multi-head self-attention again within the newly divided windows to achieve information interaction between adjacent windows; the multi-head self-attention mechanism is as follows: in, 、 、 The inputs are the query matrix, key matrix, and value matrix of this mechanism, respectively. 、 、 Let be the learnable linear projection matrices of the query, key, and value corresponding to the i-th attention head, respectively. To output a linear transformation matrix, Concat is a concatenation operation used to merge the outputs of multiple attention heads.
7. The image super-resolution method based on an adaptive semantic guidance network according to claim 1, characterized in that, Step S5, the adaptive dictionary update strategy is: through an adaptive combination process, the newly generated similar graph elements are combined with the dictionary of the previous layer to form an updated dictionary for the next layer; in, For the updated dictionary, This is a parameter between 0 and 1, used to control the ratio of merging the old and new dictionaries; The function is used to convert a vector into a probability distribution. Represents the similarity graph matrix Normalization is performed to ensure numerical stability. Let be the transpose feature matrix of the (i+1)th layer. It is the dictionary of the current layer.
8. An image super-resolution system based on an adaptive semantic guidance network, characterized in that, An image super-resolution method based on an adaptive semantic guidance network according to any one of claims 1-7, comprising: a data acquisition module, a DFSA module, a SGM module, an MHSA module, a feature enhancement module, an ADU module, and a pixel shuffling module; the data acquisition module is used to acquire low-resolution images; the DFSA module is used to initialize an external dictionary and introduce external prior information into the low-resolution image using a dictionary shuffling self-attention mechanism to generate a similarity map containing semantic information of the image; the SGM module is used to perform semantically guided pixel rearrangement on the input low-resolution image according to the generated similarity map, and to perform state space modeling on the rearranged sequence using a one-way Mamba scanning mechanism, outputting a fully semantically guided pixel rearrangement. The system consists of: a localized feature map; an MHSA module for enhancing local interactions in the input low-resolution image using a shift-window-based multi-head self-attention mechanism (SW-MHSA) to obtain locally enhanced features; a feature enhancement module for element-wise addition of the feature results from the DFSA, SGM, and MHSA modules to the original input features to fuse all processed information, followed by further feature extraction and enhancement via a feedforward network; an MHSA module for updating the dictionary of the previous layer using the output features and similarity maps of each layer based on an adaptive dictionary update strategy; and a pixel shuffling module for generating the final SR high-resolution image by performing a pixel shuffling operation on the features after multi-layer feature extraction networks.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements an image super-resolution method based on an adaptive semantic guidance network as described in any one of claims 1-7.
10. A processing terminal, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes a computer program, it implements an image super-resolution method based on an adaptive semantic guidance network as described in any one of claims 1-7.