A semantically aware token-based image super-resolution method and system

Abstract

This invention discloses an image super-resolution method and system based on semantically aware tokens, belonging to the field of computer vision technology. The method includes: generating semantic confidence and grouping information through content-aware token aggregation; decoupling the basic residual into a texture enhancement and degradation suppression guided map; achieving progressive attention focusing by combining static semantic constraint masks and sparse matrix multiplication mechanisms; introducing diffusion time-step embedding and a collaborative modulator to dynamically inject semantically guided information into a multi-step denoising process; fusing adaptive attention features and diffusion reconstruction features; and finally outputting a high-fidelity, high-resolution image. This invention achieves content-adaptive high-resolution image reconstruction through a semantically grouped guided sparse attention mechanism and semantically decoupled guided diffusion denoising collaborative modulation.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to an image super-resolution method and system based on semantically aware tokens. Background Technology

[0002] Since the successful application of Convolutional Neural Networks (CNNs) to single-image super-resolution tasks, the field has made significant progress. Early CNN methods, such as SRCNN, effectively extracted features by utilizing the local properties of convolutional kernels, but their inherent local receptive field limited the model's ability to capture long-range dependencies, posing a challenge for reconstructing complex textures and structural details in images. To compensate for this deficiency, some studies have expanded the receptive field by constructing deeper and more complex network structures, but this has resulted in a sharp increase in the number of model parameters and computational resources, limiting its deployment in practical applications.

[0003] In recent years, the Transformer model, due to its powerful global dependency modeling capabilities, has achieved breakthroughs in the field of natural language processing and has been rapidly introduced into computer vision tasks, demonstrating performance superior to CNNs. In the field of image super-resolution, the Transformer can effectively capture non-local information in images through its self-attention mechanism, but its computational complexity increases quadratically with image resolution, constituting a serious computational bottleneck.

[0004] To address this issue, researchers have proposed various sparse attention mechanisms. For example, SwinIR segments the image into non-overlapping local windows for attention computation and increases cross-window connectivity by moving the windows. While this approach improves computational efficiency to some extent, most are "content-agnostic," meaning they compute within fixed spatial regions and fail to flexibly aggregate information based on the actual similarity of image content. This limitation directly weakens the attention mechanism's ability to capture long-distance content relevance, potentially introducing irrelevant interference information into the reconstruction results. Subsequent studies attempted to achieve content awareness through feature clustering, but the iterative process in the inference phase limited the model's practical application speed. Meanwhile, diffusion models, as an emerging generative paradigm, have achieved significant success in image synthesis and have been shown to generate higher perceptual quality results in image super-resolution tasks, outperforming generative adversarial networks (GANs). However, existing diffusion super-resolution methods still face some challenges. Some methods (such as SR3) start sampling from pure Gaussian noise, a lengthy and suboptimal process. While methods like ResShift simplify the process by embedding LR information in the initial noise map, they typically apply a uniform isotropic noise level to all pixels during diffusion. This indiscriminate approach ignores the inherent characteristics of different regions in the image: for example, pixel values ​​in flat regions are already very close to their high-resolution (HR) target, while pixels in edge and textured regions exhibit greater distributional differences. Applying large noise to flat regions erases the high-fidelity details that should be preserved, while applying small noise to high-frequency regions may result in oversmoothing due to insufficient degrees of freedom in generation. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes an image super-resolution method and system based on semantically aware tokens. This method achieves high-resolution image reconstruction through an adaptive attention mechanism and multi-stage feature processing, thereby improving image detail recovery and suppressing artifact generation.

[0006] On the one hand, an image super-resolution method based on semantically aware tokens includes:

[0007] S1 performs a convolution operation on the input low-resolution image to extract shallow features and extract feature tokens from the shallow features.

[0008] S2: A pre-trained auxiliary super-resolution network is used to generate a preliminary super-resolution image from the input low-resolution image. The pixel-level absolute difference between the preliminary super-resolution image and the upsampled and aligned low-resolution image is calculated to obtain the basic structure residual map. Semantic confidence and semantic grouping information are generated through content-aware token aggregation. Then, pixel-level semantic confidence maps are generated by upsampling and smoothing the semantic confidence. The pixel-level semantic confidence maps are used to modulate the basic structure residual map through two learnable nonlinear gating functions, decoupling them into complementary texture enhancement guidance maps and degradation suppression guidance maps. The texture enhancement guidance maps and degradation suppression guidance maps are concatenated in the channel dimension to form a semantic decoupling guidance map.

[0009] The generation of the preliminary super-resolution image specifically includes: resizing and standardizing the input low-resolution image to obtain a preprocessed low-resolution image; inputting the preprocessed low-resolution image into a pre-trained auxiliary super-resolution network; enhancing the preprocessed low-resolution image through deep residual learning and feature reconstruction to obtain an enhanced low-resolution image; and performing color correction on the enhanced low-resolution image to obtain the preliminary super-resolution image.

[0010] S3 generates a static semantic constraint mask based on semantic grouping information, and calculates the semantically guided initial attention map based on shallow features and the static semantic constraint mask. In subsequent layers, the static semantic constraint mask is removed, and the sparse attention of the current layer is efficiently calculated through the sparse matrix multiplication (SMM) mechanism to obtain the initial attention map of the current layer. The initial attention map of the current layer is then obtained by performing a Hadamard product operation on the attention map output from the previous layer. The product result is normalized, and the normalized result is sparsified. The progressive focusing of the attention mechanism is realized layer by layer to obtain the first deep feature.

[0011] S4. Based on the preset current diffusion time step, the current diffusion time step embedding vector is obtained through sinusoidal position encoding and mapping of two multilayer perceptrons. The current diffusion time step embedding vector, semantic decoupling guidance map and first deep feature are used to generate a fused guidance feature map through a collaborative modulator. In each step of reverse denoising, the fused guidance feature map is concatenated with the current noise feature map on the channel and used as the input of the main denoising U-Net to guide it to perform multi-step diffusion denoising and generate the second deep feature.

[0012] S5 concatenates the first and second depth features to obtain a hybrid feature map. The hybrid feature map is then convolved to fuse the information from both features and restore the number of channels. Finally, upsampling and convolution refinement are performed to output the reconstructed high-resolution image.

[0013] Furthermore, in S2, the content-aware token aggregation method is as follows:

[0014] A set of global token centers shared across the training set is constructed and maintained. The initial value of the global token centers is obtained by performing regular regional average pooling on the shallow features of a portion of the training data. During the training process, the set of token centers is dynamically updated by exponential moving average so that it can adaptively represent the global semantic pattern. For each feature token in the shallow features, its cosine similarity with each token center is calculated. The maximum similarity is taken as the semantic confidence of the feature token. The index of the global token center with the maximum cosine similarity with the feature token is determined as the semantic grouping information.

[0015] Furthermore, in S2, the two learnable nonlinear gating functions are specifically texture enhancement gating subnetworks. and degradation suppression gating subnetwork All include It consists of a convolutional layer and a sigmoid activation function, used to compress the result to... To make it serve as a gating signal;

[0016] The texture enhancement gating subnetwork The residual map of the basic structure is modulated to generate a texture-enhanced guided map. The calculation formula is as follows:

[0017] ;

[0018] in, This represents a texture enhancement guidance map; Represents the residual diagram of the basic structure; Represents a pixel-level semantic confidence map; This indicates element-wise multiplication.

[0019] The degradation suppression gating subnetwork The residual map of the basic structure is modulated to generate a degradation suppression guidance map, and the calculation formula is as follows:

[0020] ;

[0021] in, This represents a degradation inhibition guidance diagram.

[0022] Furthermore, in S3, the static semantic constraint mask is defined as:

[0023] ;

[0024] Represents a static semantic constraint mask; This represents a learnable scalar parameter used for automatic optimization during training via backpropagation; and These are respectively represented as feature tokens and The semantic group.

[0025] Furthermore, in S3, the formula for calculating the semantically guided initial attention map based on shallow features and static semantic constraint masks is as follows:

[0026] ;

[0027] The initial attention graph represents semantic guidance; For semantic constraint mask; Represents the key vector The dimension; Representing the query matrix With transpose key matrix The product of.

[0028] Furthermore, in S3, the sparsity operation is specifically as follows:

[0029] The normalization result retains a learnable, layer-by-layer adaptive maximum attention value, while the remaining attention values ​​are set to zero, in order to achieve sparsity and focus of attention weights.

[0030] Furthermore, the specific method for calculating the number of maximum attention values ​​retained through layer-by-layer adaptive computation is as follows:

[0031] Define a learnable parameter vector ,in, Indicates the total number of floors;

[0032] The specific focus rate of the current layer is calculated using the Sigmoid function. To ensure :

[0033] ;

[0034] Use focus rate to calculate the number of attention values ​​that need to be retained in this layer. :

[0035] ;

[0036] in, This represents the number of attention points retained in the previous layer; for the first layer, it represents the number of attention points retained. Set to a fixed preset value. This represents a minimum retention threshold; This represents the floor function.

[0037] Furthermore, in S4, the cooperative modulator is specifically a learnable neural network module, including an input projection layer, a time modulation layer, a modulation block, and an output projection layer:

[0038] The input projection layer is used to concatenate the semantically decoupled guidance map with the upsampled first depth features in the channel dimension, and then project them into intermediate features through a convolution operation.

[0039] The time modulation layer is used to input the diffusion time step embedding vector into a multilayer perceptron, and the output is used for the scaling parameters of the affine transformation in the modulation block. and bias parameters ;

[0040] The modulation block is used to sequentially perform convolution, affine transformation, activation function processing and double convolution on the intermediate features, and introduces residual connections to enhance stability, outputting a refined feature map;

[0041] The output projection layer is used to adjust the channels of the refined feature map output by the modulation block through convolution, and output a multi-channel fused guided feature map that matches the number of channels of the noise feature map.

[0042] Furthermore, the calculation formula for sequentially performing affine transformations on the intermediate features is as follows:

[0043] ;

[0044] in, Indicates the scale parameter and bias parameters Features after affine transformation modulation; This represents the feature output of the first convolutional layer in the modulation block.

[0045] On the other hand, an image super-resolution system based on semantically aware tokens includes:

[0046] The token extraction module is used to perform convolution operations on the input low-resolution image, extract shallow features, and extract feature tokens from the shallow features.

[0047] The semantic decoupling guided map generation module is used to generate a preliminary super-resolution image from the input low-resolution image using a pre-trained auxiliary super-resolution network. It calculates the pixel-level absolute difference between the preliminary super-resolution image and the upsampled and aligned low-resolution image to obtain the basic structure residual map. Semantic confidence and semantic grouping information are generated through content-aware token aggregation. Then, pixel-level semantic confidence maps are generated by upsampling, aligning, and convolutionally smoothing the semantic confidence. The pixel-level semantic confidence maps are then used to modulate the basic structure residual map using two learnable nonlinear gating functions, decoupling it into complementary texture enhancement and degradation suppression guided maps. Finally, the texture enhancement and degradation suppression guided maps are concatenated along the channel dimension to form the semantic decoupling guided map.

[0048] The first deep feature acquisition module is used to generate a static semantic constraint mask based on semantic grouping information, and to calculate the semantically guided initial attention map based on shallow features and the static semantic constraint mask. In subsequent layers, the static semantic constraint mask is removed, and the sparse attention of the current layer is efficiently calculated through the sparse matrix multiplication (SMM) mechanism to obtain the initial attention map of the current layer. The initial attention map of the current layer is then obtained by performing a Hadamard product operation on the attention map output from the previous layer. The product result is normalized, and the normalized result is sparsified. The attention mechanism is progressively focused layer by layer to obtain the first deep feature.

[0049] The second deep feature acquisition module is used to obtain the current diffusion time step embedding vector based on the preset current diffusion time step through sinusoidal position encoding and mapping of two multilayer perceptrons. The current diffusion time step embedding vector, semantic decoupling guidance map and first deep features are used to generate a fused guidance feature map through a collaborative modulator. In each step of reverse denoising, the fused guidance feature map is concatenated with the current noise feature map on the channel and used as the input of the main denoising U-Net to guide it to perform multi-step diffusion denoising and generate the second deep features.

[0050] The reconstruction module is used to stitch together the first depth feature and the second depth feature to obtain a hybrid feature map, perform convolution on the hybrid feature map to fuse the information of the two and restore the number of channels, and then perform upsampling and convolution refinement to output the reconstructed high-resolution image.

[0051] The present invention adopts the above technical solution and has the following beneficial effects:

[0052] (1) This invention generates semantic confidence through content-aware token aggregation and modulates the basic structure residual map using two learnable nonlinear gating functions respectively, decoupling them into complementary texture enhancement guidance map and degradation suppression guidance map to more accurately capture and restore complex texture information and structural details in the original high-resolution image, thereby improving the quality of the reconstructed image;

[0053] (2) This invention introduces a static semantic constraint mask and combines it with the sparse matrix multiplication (SMM) mechanism. By performing the Hadamard product operation on the initial attention map of the current layer and the attention map of the previous layer, as well as subsequent normalization and sparsification operations, the attention mechanism is made adaptively adjusted and progressively focused layer by layer.

[0054] (3) The present invention utilizes the preset current diffusion time step embedding vector, semantic decoupling guidance map and first deep feature to generate a fusion guidance feature map through a collaborative modulator, which guides the main denoising U-Net network to perform multi-step diffusion denoising. This not only enhances the robustness of the model to uncertainty, but also effectively reduces noise interference, further improving the clarity and visual effect of the final output image. Attached Figure Description

[0055] Figure 1 This is a flowchart of an image super-resolution method based on semantically aware tokens according to an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the overall network architecture of an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of content-aware estimation according to an embodiment of the present invention;

[0058] Figure 4 This is a schematic diagram of the progressive focusing attention according to an embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram of coordinated modulation diffusion according to an embodiment of the present invention;

[0060] Figure 6 This is a diagram of an image super-resolution system based on semantically aware tokens, according to an embodiment of the present invention. Detailed Implementation

[0061] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0062] like Figure 1 As shown, the present invention provides an image super-resolution method based on semantically aware tokens, comprising:

[0063] S1 performs a convolution operation on the input low-resolution image to extract shallow features and extract feature tokens from the shallow features.

[0064] Specifically, such as Figure 2 The diagram shown is a schematic of the overall network architecture of an embodiment of the present invention. It adopts a dual-branch collaborative processing framework, the core of which is a content-adaptive guidance mechanism and two parallel and cooperative network flows. The entire framework can be decomposed into the following steps:

[0065] (1) Shallow feature extraction: First, the low-resolution (LR) image is subjected to shallow feature extraction through a 3×3 convolutional layer, which provides a basis for subsequent complex processing.

[0066] (2) Semantic decoupling guidance graph and dual-branch collaborative processing:

[0067] Content-Aware Estimation Module: This module receives shallow features in parallel and generates a semantic decoupling guidance map and semantic grouping information. This map precisely indicates which regions in the image need texture enhancement and which regions need noise suppression.

[0068] Progressive Focus Attention Module: Responsible for discriminative image inpainting. Similarly, this module also receives shallow features as input and uses semantic grouping information output by the content-aware estimation module for progressive focusing, efficiently capturing long-distance dependencies and outputting "first deep features".

[0069] The Coordinated Modulation Diffusion Module is responsible for generative image restoration. This module receives shallow features as conditional input and uses a coordinated modulator to fuse the semantically decoupled guided map generated by the Content-Aware Estimation Module and the first deep features and time-step information generated by the Progressive Focusing Module to generate a dynamic fused guided feature map. During diffusion denoising, this feature map is concatenated with the noise map to guide the main denoising network in restoring image details, ultimately outputting the second deep feature map.

[0070] (3) Deep Feature Fusion: The two parallel networks, the Progressive Focused Attention Module and the Diffusion Generation Module, perform deep feature fusion. Specifically, this method concatenates the outputs from the Progressive Focused Attention Module (responsible for discriminative features) and the Co-modulation Diffusion Module (responsible for generative features) along the channel dimension and fuses the information through convolution to restore the number of channels, thereby achieving lossless preservation of the complementary information processed by the two networks.

[0071] (4) Image reconstruction: Finally, the fused deep features are upsampled (pixel shuffling) and refined by convolution to reconstruct a high-quality high-resolution (HR) image.

[0072] S2: A pre-trained auxiliary super-resolution network is used to generate a preliminary super-resolution image from the input low-resolution image. The pixel-level absolute difference between the preliminary super-resolution image and the upsampled and aligned low-resolution image is calculated to obtain the basic structure residual map. Semantic confidence and semantic grouping information are generated through content-aware token aggregation. Then, pixel-level semantic confidence maps are generated by upsampling and smoothing the semantic confidence. The pixel-level semantic confidence maps are used to modulate the basic structure residual map through two learnable nonlinear gating functions, decoupling them into complementary texture enhancement guidance maps and degradation suppression guidance maps. The texture enhancement guidance maps and degradation suppression guidance maps are concatenated in the channel dimension to form a semantic decoupling guidance map.

[0073] The generation of the preliminary super-resolution image specifically includes: resizing and standardizing the input low-resolution image to obtain a preprocessed low-resolution image; inputting the preprocessed low-resolution image into a pre-trained auxiliary super-resolution network; enhancing the preprocessed low-resolution image through deep residual learning and feature reconstruction to obtain an enhanced low-resolution image; and performing color correction on the enhanced low-resolution image to obtain the preliminary super-resolution image.

[0074] Specifically, the content-aware token aggregation method is as follows:

[0075] A set of global token centers shared across datasets is constructed and maintained. The initial values ​​are obtained by performing regular region average pooling on the shallow features of a portion of the training data. During training, the set of token centers is dynamically updated using exponential moving average to adaptively represent global semantic patterns. For each feature token in the input shallow features, its cosine similarity with each global token center is calculated, and the maximum cosine similarity is taken as the semantic confidence of the feature token. The index of the global token center with the maximum cosine similarity corresponding to the feature token is determined as the semantic grouping information.

[0076] Specifically, the two learnable nonlinear gating functions are texture enhancement gating subnetworks. and degradation suppression gating subnetwork All include It consists of a convolutional layer and a sigmoid activation function, used to compress the result to... To make it serve as a gating signal;

[0077] The texture enhancement gating subnetwork The residual map of the basic structure is modulated to generate a texture-enhanced guided map. The calculation formula is as follows:

[0078] ;

[0079] in, This represents a texture enhancement guidance map; Represents the residual diagram of the basic structure; Represents a pixel-level semantic confidence map; This indicates element-wise multiplication.

[0080] The degradation suppression gating subnetwork The residual map of the basic structure is modulated to generate a degradation suppression guidance map, and the calculation formula is as follows:

[0081] ;

[0082] in, This represents a degradation inhibition guidance diagram.

[0083] For details, see Figure 3 A schematic diagram for content-aware estimation: Specifically, average pooling is performed on the regular regions to obtain the initial... A token center, To learn a set of global token centers across the training set, token centers are shared among all image tokens, and the token centers are updated using an exponential moving average (EMA) during training, enabling them to adaptively capture semantic patterns in the dataset.

[0084] For each token feature in the input shallow features Calculate its relationship with all token centers The cosine similarity is calculated and assigned to the token center with the highest similarity to form semantic grouping information.

[0085] At the same time, the maximum value of the cosine similarity mentioned above is taken as the feature-level semantic confidence of the feature token. :

[0086]

[0087] This confidence level reflects the degree of matching between the current feature and the global semantic pattern. Subsequently, the feature-level semantic confidence is upsampled, aligned, and smoothed via convolution to obtain a pixel-level semantic confidence map. This image serves as a gating signal, subsequently decoupled from the underlying residual map using a nonlinear gating function to create complementary texture enhancement and degradation suppression guidance maps, thereby achieving adaptive enhancement and suppression of different image regions. The two generated guidance maps are then concatenated along the channel dimension to form the final semantically decoupled map, which is then transmitted to the cooperative modulation diffusion for cooperative modulation.

[0088] Specifically, this embodiment employs a large-scale dataset, including a DIV2K pre-trained general super-resolution base model (such as ESRGAN, OmniSR, or SwinIR) as an auxiliary network. Lightweight supervised fine-tuning is performed for the target application scenario (such as specific degradation types or real-world images): only a small number of target domain image pairs (100–200 images) are used to fine-tune the top-level reconstruction module and upsampling layer with a low learning rate, freezing most of the backbone parameters, balancing convergence speed and generalization ability. The fine-tuned model does not participate in the final high-fidelity reconstruction, but is dedicated to generating preliminary super-resolution images with reasonable structure, reliable texture, and consistent semantics, providing a high-quality, low-biased prior starting point for subsequent semantic decoupling guidance, progressive sparse attention focusing, and diffusion collaborative refinement. This strategy inherits the powerful generalization ability of publicly available pre-trained models and effectively alleviates the domain offset problem through lightweight adaptation, achieving an organic unity of out-of-the-box usability and task customization.

[0089] S3 generates a static semantic constraint mask based on semantic grouping information, and calculates the semantically guided initial attention map based on shallow features and the static semantic constraint mask. In subsequent layers, the static semantic constraint mask is removed, and the sparse attention of the current layer is calculated through the sparse matrix multiplication (SMM) mechanism to obtain the initial attention map of the current layer. The initial attention map of the current layer is then obtained by performing a Hadamard product operation with the attention map output from the previous layer. The product result is normalized, and a sparsification operation is performed on the normalized result. This process achieves progressive focusing of the attention mechanism layer by layer, thereby obtaining the first deep feature.

[0090] Specifically, the static semantic constraint mask is defined as:

[0091] ;

[0092] Represents a static semantic constraint mask; This represents a learnable scalar parameter used for automatic optimization during training via backpropagation; and These are respectively represented as feature tokens and The semantic group.

[0093] Specifically, the formula for calculating the semantically guided initial attention map based on shallow features and static semantic constraint masks is as follows:

[0094]

[0095] The initial attention graph represents semantic guidance; For semantic constraint mask; Represents the key vector The dimension; Representing the query matrix With transpose key matrix The product of.

[0096] Specifically, the sparsity operation is as follows:

[0097] The normalization result retains a learnable, layer-by-layer adaptive maximum attention value, while the remaining attention values ​​are set to zero, in order to achieve sparsity and focus of attention weights.

[0098] Specifically, the method for calculating the maximum attention value for layer-by-layer adaptive input is as follows:

[0099] Define a learnable parameter vector ,in, Indicates the total number of floors;

[0100] The specific focus rate of the current layer is calculated using the Sigmoid function. To ensure :

[0101] ;

[0102] Use focus rate to calculate the number of attention values ​​that need to be retained in this layer. :

[0103] ;

[0104] in, This indicates the number of items retained in the previous layer. For the first layer ( ), the attention value retained Set to a fixed preset value (e.g.) ), This represents a minimum retention threshold; This represents the floor function.

[0105] For details, see Figure 4 This is a schematic diagram of progressive attention focusing, specifically a refined network based on Transformer. It receives shallow features as input and uses semantic grouping information for global constraints, ultimately generating the first deep feature. First, a static semantic constraint mask is generated based on the semantic grouping information. In the first attention layer, the semantic constraint mask is applied. By combining the query (Q) and key (K) with a dot product and overlaying a semantic mask, a semantically guided initial attention graph is generated. In this way, attention computation is strictly constrained to tokens with the same semantic grouping information, thus achieving efficient initial sparse attention computation. Starting from the second layer, for the... Each attention layer utilizes the sparse index generated by the previous layer. The initial attention map of the current layer is efficiently computed using sparse matrix multiplication (SMM). Then, a gradual focusing is achieved through a layer-by-layer transmission mechanism:

[0106]

[0107] in, It is the focus attention map of the previous layer. It is element-wise multiplication. This indicates a sparsification operation, retaining only the values ​​in each row. Set the maximum attention value, and set the rest to 0.

[0108] At the same time, to guide the next level Perform efficient SMM calculations based on the final focused attention map generated at the current layer. Calculate the sparse index :

[0109]

[0110] The index This will serve as a "hot" index map for the next layer, indicating that the next layer will only perform calculations within these active regions.

[0111] At the same time, on each layer Attention map generated in It will be used to aggregate the value matrix through SMM. Features after aggregation :

[0112]

[0113] in This represents the sparse matrix multiplication (SMM) operator, used in sparse indexing. Under the guidance of this method, we can efficiently compute sparse attention graphs. AND-value matrix The product of these features yields the aggregated features. .

[0114] Features after aggregation This is then followed by a standard feedforward network (FFN) and a residual connection, the output of which... Will be used as the next layer ( Input features.

[0115] In the last layer of the module, after passing through FFN and residual connections, the final feature is output, which is the first depth feature.

[0116] S4. Based on the preset current diffusion time step, the current diffusion time step embedding vector is obtained through sinusoidal position encoding and mapping of two multilayer perceptrons. The current diffusion time step embedding vector, semantic decoupling guidance map and first deep feature are used to generate a fused guidance feature map through a collaborative modulator. In each step of reverse denoising, the fused guidance feature map is concatenated with the current noise feature map on the channel and used as the input of the main denoising U-Net to guide it to perform multi-step diffusion denoising and generate the second deep feature.

[0117] Specifically, the cooperative modulator is a learnable neural network module, including an input projection layer, a time modulation layer, a modulation block, and an output projection layer:

[0118] The input projection layer is used to concatenate the semantically decoupled guidance map with the upsampled first depth features in the channel dimension, and then project them into intermediate features through a convolution operation.

[0119] The time modulation layer is used to input the diffusion time step embedding vector into a multilayer perceptron, and the output is used for the scaling parameters of the affine transformation in the modulation block. and bias parameters ;

[0120] The modulation block is used to sequentially perform convolution, affine transformation, activation function processing and double convolution on the intermediate features, and introduces residual connections to enhance stability, outputting a refined feature map;

[0121] The output projection layer is used to adjust the channels of the refined feature map output by the modulation block through convolution, and output a multi-channel fused guided feature map that matches the number of channels of the noise feature map.

[0122] Specifically, the calculation formula for sequentially performing affine transformations on the intermediate features is as follows:

[0123] ;

[0124] in, Indicates the scale parameter and bias parameters Features after affine transformation modulation; This represents the intermediate features extracted by the first convolutional layer in the modulation block.

[0125] For details, see Figure 5 This diagram illustrates the cooperative modulation diffusion process, representing the generative branch of the framework responsible for generative image restoration. Overall, this module implements a multi-step iterative reverse denoising diffusion process. This process starts from an initial noisy state... ;

[0126] Begin, through The next iteration (e.g.) The noise is gradually reduced to generate the final features.

[0127] In each iteration In this mode, a cooperative modulator is used. This achieves dynamic guidance. The modulator operates at each time step during the diffusion process. It obtains information from three sources to make decisions: the "semantic decoupling guidance map" from the content-aware estimation module, the "first deep feature" from the progressive focusing attention module, and the "time step embedding vector" from the diffusion process of the co-modulation diffusion module.

[0128] The "time-step embedding vector" is obtained through a time-step encoder. The time-step encoder uses sinusoidal position coding combined with two multilayer perceptron (MLP) refinement mapping methods to generate the diffused time-step embedding vector. Specifically, the time step... Encoded into a fixed-length vector using sine and cosine functions, its ... The value of each dimension is defined as:

[0129]

[0130] in, The initial embedding vector is then processed by two multilayer perceptrons (MLPs) for nonlinear mapping and dimension adjustment to obtain the time-step embedding vector. The formula is expressed as:

[0131]

[0132] This represents the weight matrix of each linear layer in a multilayer perceptron. This represents the corresponding bias vector. Represented as the SILU activation function. Cooperative modulator. The specific data flow path is as follows:

[0133] First, spatial information fusion processing is performed. The semantically decoupled guidance map and the upsampled first deep feature map are concatenated together along the channel dimension. Then, a... The convolutional layer (Proj_in) projects this high-dimensional combined feature into a unified, easily processed intermediate feature. Secondly, time information processing and spatial information processing are performed in parallel, with time steps... The vector is fed into a dedicated multilayer perceptron (MLP_t). The task of this MLP is to abstract time-step information (e.g., The MLP is parsed into a specific set of affine transformation parameters used to control subsequent network behavior. In this embodiment, the MLP outputs a set of scale parameters. and bias parameters .

[0134] Next, the core dynamic modulation is performed. The aforementioned intermediate features... It then enters a modulation block composed of a deep network structure. Inside the modulation block, the intermediate features first pass through a... Convolutional layers (Conv1) extract spatial information and output features. Then it is immediately modulated by the time modulation parameters specific to that layer. This modulation is achieved through affine transformation (FiLM):

[0135]

[0136] The modulated features are then processed by an activation function SiLU and a second... A convolutional layer (Conv2) is used, and residual connections are made with the input features of the block to ensure the stability of the information flow and gradient propagation.

[0137] Finally, multi-channel fused guidance features are generated. After deep processing of the modulation block, the resulting feature map fully integrates semantic, structural, and temporal information. This feature map is ultimately processed by a... The convolutional layer will adjust the channels to output a multi-channel "fusion-guided feature map" that matches the number of channels in the noise feature map.

[0138] this It acts as a dynamic spatial guidance map in the current iteration step. It is used to correct the conditional input of the main denoising U-Net: at each step, this... The current noisy feature map is concatenated with it along the channel dimension, and the concatenated feature map will serve as the complete input to the main denoising U-Net.

[0139] U-Net predicts noise based on the feature map, and the Co-modulation Diffusion module then uses the predicted noise to calculate the next noise map using a standard sampler. After this process is iterated a specified number of times, the final output is the output second depth feature.

[0140] Specifically, after performing diffusion denoising at all time steps, the final output is the second depth feature, which will be sent to the image reconstruction part for final fusion and reconstruction.

[0141] S5 concatenates the first and second depth features, and then... Convolution fuses the information from both sources and restores the number of channels, then upsampling and convolution refinement are performed to output a reconstructed high-resolution image.

[0142] Specifically, to verify the practical effect of the image super-resolution method based on adaptive attention proposed in this invention, experimental tests were conducted on a unified hardware platform, and multi-dimensional evaluation metrics were performed on multiple authoritative test sets, as follows:

[0143] Hardware environment: The experimental platform was equipped with an Intel(R) Core(TM) i9-12900HX CPU and 64GB of RAM. Model training and inference were both performed on an NVIDIA GeForce RTX 4060 Laptop GPU.

[0144] Software environment: The algorithm is implemented based on the deep learning framework PyTorch 2.4.1.

[0145] Training strategy: The DF2K dataset (containing DIV2K and Flickr2K) is used as the training set. The AdamW optimizer is used, with an initial learning rate of [value missing]. The input image patch size is The total number of iterations was 500,000.

[0146] For real-world scene perception quality estimation, the perception metrics on the RealSR real-world scene dataset under a 4x super-resolution task are shown in Table 1.

[0147] Table 1: Perceived quality metrics on the RealSR dataset (x4);

[0148]

[0149] In terms of fidelity evaluation of complex structures, the fidelity index test results of this invention on the Urban100 dataset, which contains dense line structures, are shown in Table 2.

[0150] Table 2: Fidelity metrics on the Urban100 dataset (x4);

[0151]

[0152] Experimental data show that this method achieves extremely low LPIPS (0.2815) and high MUSIQ (65.30) scores on the RealSR dataset, indicating that the generated images are visually clear and natural, with no obvious artifacts. Meanwhile, it maintains a high PSNR (28.18dB) on the Urban100 dataset, demonstrating that the model can accurately recover complex structures such as building lines.

[0153] like Figure 6 As shown, this embodiment also discloses an image super-resolution system based on semantically aware tokens, including:

[0154] The token extraction module 61 is used to perform convolution operations on the input low-resolution image, extract shallow features, and extract feature tokens from the shallow features.

[0155] The semantic decoupling guided map generation module 62 is used to generate a preliminary super-resolution image from the input low-resolution image using a pre-trained auxiliary super-resolution network, calculate the pixel-level absolute difference between the preliminary super-resolution image and the upsampled and aligned low-resolution image to obtain the basic structure residual map; generate semantic confidence and semantic grouping information through content-aware token aggregation, and then generate a pixel-level semantic confidence map by upsampling and convolutional smoothing of the semantic confidence; use the pixel-level semantic confidence map to modulate the basic structure residual map through two learnable nonlinear gating functions to decouple it into complementary texture enhancement guided map and degradation suppression guided map; and concatenate the texture enhancement guided map and degradation suppression guided map in the channel dimension to form the semantic decoupling guided map.

[0156] The generation of the preliminary super-resolution image specifically includes: resizing and standardizing the input low-resolution image to obtain a preprocessed low-resolution image; inputting the preprocessed low-resolution image into a pre-trained auxiliary super-resolution network; enhancing the preprocessed low-resolution image through deep residual learning and feature reconstruction to obtain an enhanced low-resolution image; and performing color correction on the enhanced low-resolution image to obtain the preliminary super-resolution image.

[0157] The first deep feature acquisition module 63 is used to generate a static semantic constraint mask based on semantic grouping information, and calculate the semantically guided initial attention map based on shallow features and the static semantic constraint mask. In subsequent layers, the static semantic constraint mask is removed, and the sparse attention of the current layer is efficiently calculated through the sparse matrix multiplication (SMM) mechanism to obtain the initial attention map of the current layer. The initial attention map of the current layer is then obtained by performing a Hadamard product operation on the attention map output from the previous layer. The product result is normalized, and the normalized result is sparsified. The attention mechanism is progressively focused layer by layer to obtain the first deep feature.

[0158] The second deep feature acquisition module 64 is used to obtain the current diffusion time step embedding vector based on the preset current diffusion time step through sinusoidal position encoding and mapping of two multilayer perceptrons. The current diffusion time step embedding vector, semantic decoupling guidance map and first deep features are used to generate a fused guidance feature map through a collaborative modulator. In each step of reverse denoising, the fused guidance feature map is concatenated with the current noise feature map on the channel and used together as the input of the main denoising U-Net to guide it to perform multi-step diffusion denoising and generate the second deep features.

[0159] The reconstruction module 65 is used to stitch together the first depth feature and the second depth feature to obtain a mixed feature map, perform convolution on the mixed feature map to fuse the information of the two and restore the number of channels, and then perform upsampling and convolution refinement to output the reconstructed high-resolution image.

[0160] A specific implementation of an image super-resolution system based on semantically aware tokens is described in this embodiment, which is the same as the image super-resolution method based on semantically aware tokens.

[0161] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A semantic-aware token based image super-resolution method, characterized in that, The method comprises the following steps: S1, performing a convolution operation on the input low-resolution image to extract a shallow feature and a feature token from the shallow feature; S2, generating a preliminary super-resolution image for the input low-resolution image by using a pre-trained auxiliary super-resolution network, calculating a pixel-level absolute difference between the preliminary super-resolution image and the low-resolution image after up-sampling alignment to obtain a basic structure residual image, generating a semantic confidence and semantic grouping information by a content-aware token aggregation method, and then generating a pixel-level semantic confidence map by up-sampling alignment and convolution smoothing processing on the semantic confidence, and modulating the basic structure residual image by using the pixel-level semantic confidence map through two learnable nonlinear gating functions to decouple into a complementary texture enhancement guidance image and a degradation suppression guidance image; and splicing the texture enhancement guidance image and the degradation suppression guidance image in the channel dimension to form a semantic decoupling guidance image; The generating of the preliminary super-resolution image specifically comprises: adjusting the size and normalizing the input low-resolution image to obtain a preprocessed low-resolution image, inputting the preprocessed low-resolution image into the pre-trained auxiliary super-resolution network, enhancing the preprocessed low-resolution image through deep residual learning and feature reconstruction to obtain an enhanced low-resolution image, and performing color correction on the enhanced low-resolution image to obtain the preliminary super-resolution image; S3, generating a static semantic constraint mask based on the semantic grouping information, and calculating an initial attention map guided by semantics based on the shallow feature and the static semantic constraint mask; in the subsequent layers, the static semantic constraint mask is removed, the initial attention map of the current layer is calculated through sparse matrix multiplication SMM mechanism, and the initial attention map of the current layer is subjected to Hadamard product operation with the attention map output by the previous layer, the product operation result is normalized, and the normalized result is sparsified, so that the attention mechanism is gradually focused layer by layer to obtain a first deep feature; S4, based on a preset current diffusion time step, a current diffusion time step embedding vector is obtained by using a sinusoidal position encoding and two multi-layer perception mappings, the current diffusion time step embedding vector, the semantic decoupling guidance image and the first deep feature are input into a collaborative modulator to generate a fusion guidance feature map, and the fusion guidance feature map is spliced with a current noise feature map in the channel at each step of reverse denoising, and is used as an input of a main denoising U-Net together with the current noise feature map to guide the main denoising U-Net to perform multi-step diffusion denoising to generate a second deep feature; S5, splicing the first deep feature and the second deep feature to obtain a hybrid feature map, performing convolution fusion on the hybrid feature map to integrate the information of the first deep feature and the second deep feature and restore the number of channels, and then performing up-sampling and convolution refining to output a reconstructed high-resolution image.

2. The semantic perception token based image super-resolution method according to claim 1, characterized in that, In S2, the content-aware token aggregation method is as follows: A set of global token centers shared across the training set is constructed and maintained, the initial value of the global token center is obtained by performing rule region average pooling on the shallow features of part of the training data, and a set of token centers is dynamically updated by exponential moving average during the training process, so that it can adaptively represent the global semantic pattern, for each feature token in the shallow feature, the cosine similarity between it and each token center is calculated, the maximum value of the similarity is taken as the semantic confidence of the feature token, and the index of the global token center with the maximum cosine similarity with the feature token is determined as the semantic grouping information.

3. The semantic perception token based image super-resolution method according to claim 1, characterized in that, In S2, the two learnable nonlinear gating functions are specifically texture enhancement gating subnetworks. and degradation suppression gating subnetwork All include It consists of a convolutional layer and a sigmoid activation function, used to compress the result to... To make it serve as a gating signal; The texture enhancement gating subnetwork The texture enhancement gating subnetwork modulates the infrastructure residual map to generate a texture enhancement guidance map, and the calculation formula is as follows: ； wherein, denotes a texture enhanced guidance map; denotes a base structure residual map; denotes a pixel-level semantic confidence map; denotes element-wise multiplication; The degradation suppression gating sub-network The infrastructure residual map is modulated to generate a degradation suppression guidance map, and the calculation formula is as follows: ； wherein represents a degeneration suppression guide map.

4. The method of claim 1, wherein, In S3, the static semantic constraint mask is defined as: ； represents a static semantic constraint mask; represents a learnable scalar parameter for automatic optimization during training by backpropagation; and respectively represent semantic groups of feature tokens and .

5. The semantic perception token based image super-resolution method according to claim 4, characterized in that, In S3, the calculation formula of the initial attention map guided by semantics based on the shallow features and the static semantic constraint mask is as follows: ； an initial attention map representing semantic guidance; a semantic constraint mask; a key vector of dimensionality; a query matrix a product of the transpose key matrix .

6. The method of claim 1, wherein, In S3, the sparsification operation is specifically: The normalized result is retained a learnable, layer-by-layer adaptive number of maximum attention values, and the remaining attention values are set to zero, so as to realize the sparsification and focusing of the attention weight.

7. The semantic perception token based image super-resolution method according to claim 6, characterized in that, The calculation method of the layer-by-layer adaptive calculation of the number of retained maximum attention values is as follows: defining a learnable parameter vector wherein, denotes the total number of layers; For the current layer, the focusing rate specific to it is calculated by a sigmoid function to ensure : ； Using a focus rate to calculate the number of attention values to retain for the current layer : ； wherein, represents the number of previous layer reserved, for the first layer, the reserved attention value is set as a fixed preset value, represents a minimum reserved threshold value; represents a floor function.

8. The method of claim 1, wherein, In S4, the cooperative modulator is specifically a learnable neural network module, including an input projection layer, a time modulation layer, a modulation block and an output projection layer: The input projection layer is used to splice the semantic decoupling guidance map and the first deep feature after upsampling in the channel dimension, and then project it into an intermediate feature through convolution operation; The temporal modulation layer is used to embed diffusion time steps into a vector input to a multi-layer perceptron, outputting scale parameters for modulating affine transformations in the modulation block and bias parameters ; The modulation block is used for convolution, affine transformation, activation function processing and secondary convolution on the intermediate feature in turn, and residual connection is introduced to enhance stability, and an refined feature map is output; The output projection layer is used to adjust the channel of the refined feature map output by the modulation block through convolution, and output a multi-channel fusion guidance feature map matching the channel number of the noise feature map.

9. The semantic perception token based image super-resolution method according to claim 8, characterized in that, The calculation formula of the affine transformation of the intermediate feature in turn is as follows: ； wherein, denotes a feature after an affine transformation modulation and a bias parameter affine transformation modulation; denotes a feature output by the first convolutional layer in the modulation block.

10. A semantic-aware token based image super-resolution system, comprising: It includes: The token extraction module is used for convolution operation on the input low-resolution image, extracting shallow features and extracting feature tokens from the shallow features; The semantic decoupling guidance map generation module is used to generate a preliminary super-resolution image from the input low-resolution image by using a pre-trained auxiliary super-resolution network, calculate the pixel-level absolute difference between the preliminary super-resolution image and the low-resolution image after upsampling alignment, and obtain a basic structure residual map; generate semantic confidence and semantic grouping information through content-aware token aggregation, and then generate a pixel-level semantic confidence map by upsampling alignment and convolution smoothing processing on the semantic confidence; the pixel-level semantic confidence map is used to modulate the basic structure residual map through two learnable nonlinear gating functions, respectively, to decouple into complementary texture enhancement guidance map and degradation suppression guidance map; the texture enhancement guidance map and the degradation suppression guidance map are spliced in the channel dimension to form a semantic decoupling guidance map; The generating the preliminary super-resolution image specifically comprises: resizing and normalizing the input low-resolution image to obtain a preprocessed low-resolution image, inputting the preprocessed low-resolution image into a pre-trained auxiliary super-resolution network, enhancing the preprocessed low-resolution image through deep residual learning and feature reconstruction to obtain an enhanced low-resolution image, and performing color correction on the enhanced low-resolution image to obtain the preliminary super-resolution image. The first deep feature acquisition module is configured to generate a static semantic constraint mask based on the semantic grouping information, calculate an initial attention map guided by semantics based on the shallow feature and the static semantic constraint mask, remove the static semantic constraint mask in a subsequent layer, efficiently calculate sparse attention of the current layer through a sparse matrix multiplication (SMM) mechanism to obtain an initial attention map of the layer, perform a Hadamard product operation on the initial attention map and an attention map output by a previous layer, normalize a result of the product operation, perform a sparsification operation on the normalized result, gradually focus the attention mechanism layer by layer, and obtain the first deep feature. The second deep feature acquisition module is configured to obtain a current diffusion time step embedding vector through sinusoidal position encoding and two multi-layer perception mappings based on a preset current diffusion time step, generate a fusion guidance feature map through a collaborative modulator based on the current diffusion time step embedding vector, the semantic decoupling guidance map, and the first deep feature, and splice the fusion guidance feature map and a current noise feature map in a channel at each step of reverse denoising to be jointly used as an input of a main denoising U-Net to guide the main denoising U-Net to perform multi-step diffusion denoising and generate the second deep feature. The reconstruction module is configured to splice the first deep feature and the second deep feature to obtain a mixed feature map, perform convolution fusion on the mixed feature map to integrate information of the first deep feature and the second deep feature and restore a channel number, perform upsampling and convolution refinement on the mixed feature map, and output a reconstructed high-resolution image.

Images