Route-aware and space-frequency cooperative mamba-driven image super-resolution method and device

Abstract

The application discloses a kind of route perception and space-frequency cooperation Mamba driven image super-resolution method and device, it is related to image processing field, in the application, the input feature of route enhancement space-frequency coordination module is input to local feature extraction module, obtain first intermediate feature, then in turn after first layer normalization layer and space-frequency cooperation Mamba module, again with first intermediate feature residual connection, obtain second intermediate feature, again in turn after second layer normalization layer and first gated convolution feedforward network, again with second intermediate feature residual connection, obtain third intermediate feature, again in turn after third layer normalization layer and difficulty perception dynamic routing module, again with third intermediate feature residual connection, obtain fourth intermediate feature, again in turn after fourth layer normalization layer and second gated convolution feedforward network, again with fourth intermediate feature residual connection, obtain the output feature of route enhancement space-frequency coordination module, solve the problem that feature representation ability is weak, and reconstruction quality is low.

Description

Technical Field

[0001] This invention relates to the field of image processing, and specifically to a route-aware and space-frequency cooperative Mamba-driven image super-resolution method and apparatus. Background Technology

[0002] Single Image Super-Resolution (SISR) is a key task in computer vision, aiming to restore high-resolution images from degraded low-resolution images. Because restoring low-resolution images to super-resolution images can extract more useful information, it is widely used in computer vision fields such as object detection, face recognition, and satellite imaging.

[0003] Mamba, a deep learning model that has emerged in recent years, has attracted widespread attention in the field of image super-resolution due to its ability to model long-range dependencies with linear complexity. MambaIR pioneered the application of Mamba to image super-resolution, alleviating the problems of local pixel forgetting and channel redundancy by introducing local convolution enhancement and channel attention, respectively. MambaIRv2 further achieves global perception of pixels outside the scan sequence through attention-enhanced state-space equations. However, existing research shows that Mamba is inferior to Transformer in local context modeling and has significant shortcomings in high-frequency defect decoupling, orientation-adaptive scanning, and multi-scale feature modeling.

[0004] Furthermore, existing methods typically apply a uniform computational intensity to the entire image, leading to redundant computations in flat regions and limited representational capabilities in complex regions due to the uniform intensity. While some works have explored content-adaptive computation, such as CAMixerSR which classifies tokens using a learnable predictor based on soft hybrid attention and convolution mechanisms, the problem of adaptively allocating computational resources according to the difficulty of region reconstruction still needs improvement. Summary of the Invention

[0005] The purpose of this application is to propose a route-aware and space-frequency cooperative Mamba-driven image super-resolution method and apparatus to address the aforementioned technical problems.

[0006] In a first aspect, the present invention provides a route-aware and space-frequency cooperative Mamba-driven image super-resolution method, comprising the following steps:

[0007] An image super-resolution model driven by route awareness and space-frequency cooperation (SFR) was constructed and trained to obtain the trained image super-resolution model. The image super-resolution model includes a first convolutional layer, several route-enhancing SFR modules, a sub-pixel convolutional layer, and a second convolutional layer connected in sequence. The output features of the first convolutional layer are concatenated with the residual output features of the last route-enhancing SFR module and then passed through the sub-pixel convolutional layer and the second convolutional layer in sequence. The route-enhancing SFR module includes a local feature extraction module, a first normalization layer, a SFR Mamba module, a second normalization layer, a first gated convolutional feedforward network, a third normalization layer, a difficulty-aware dynamic routing module, a fourth normalization layer, and a second gated convolutional feedforward network.

[0008] The low-resolution image to be reconstructed is acquired and input into a trained image super-resolution model to reconstruct the corresponding high-resolution image. The input features of the routing enhancement space-frequency coordination module are first input into the local feature extraction module to obtain the first intermediate feature. The first intermediate feature is then passed through a first normalization layer and a space-frequency coordination Mamba module, and the result is concatenated with the residual of the first intermediate feature to obtain the second intermediate feature. The second intermediate feature is then passed through a second normalization layer and a first gated convolutional feedforward network, and the result is concatenated with the residual of the second intermediate feature to obtain the third intermediate feature. The third intermediate feature is then passed through a third normalization layer and a difficulty-aware dynamic routing module, and the result is concatenated with the residual of the third intermediate feature to obtain the fourth intermediate feature. The fourth intermediate feature is then passed through a fourth normalization layer and a second gated convolutional feedforward network, and the result is concatenated with the residual of the fourth intermediate feature to obtain the output feature of the routing enhancement space-frequency coordination module.

[0009] Preferably, both the first gated convolutional feedforward network and the second gated convolutional feedforward network adopt a gated convolutional feedforward structure. The gated convolutional feedforward structure includes a first projective expansion convolutional layer, a first depthwise separable convolutional layer, a first segmentation unit, a first GELU activation function layer, and a first projective reduction convolutional layer. The input features of the gated convolutional feedforward structure are first input into the first projective expansion convolutional layer to perform a projection expansion operation along the channel to obtain the first expanded features, as shown in the following equation:

[0010] ;

[0011] in, This represents the input features of the gated convolutional feedforward structure. Indicates the first extended feature, This indicates that the convolution kernel is... The projection spreading operation performed by the convolutional layer;

[0012] The first depthwise separable convolutional layer is used to perform local feature extraction on the first extended feature. Each channel is convolved independently to obtain local features, as shown in the following formula:

[0013] ;

[0014] in, Indicates local features, Indicates the kernel size as The first depth separable convolutional layer;

[0015] The local features are input into the first segmentation unit, and average segmentation is performed along the channel dimension to obtain the first segmentation feature and the second segmentation feature; the first segmentation unit is input into the first GELU activation function layer for nonlinear transformation, and then multiplied with the second segmentation feature as a gate to obtain the gated feature;

[0016] The gated features are input into the first projection-reduced convolutional layer and projected and reduced along the channel to obtain the output features of the gated convolutional feedforward structure, as shown in the following equation:

[0017] ;

[0018] in, Indicates gating features, This indicates that the convolution kernel is... The projection reduction operation performed by the convolutional layer This represents the output characteristics of the gated convolution feedforward structure.

[0019] Preferably, the difficulty-aware dynamic routing module includes a third convolutional layer, a fifth normalization layer, a first ReLU activation function layer, a channel attention branch, a spatial attention branch, a path routing branch, and an enhancement branch. The input features of the difficulty-aware dynamic routing module first pass through the third convolutional layer to obtain a value vector. The value vector undergoes feature compression and then passes through the fifth normalization layer and the first ReLU activation function layer to obtain compressed features. The compressed features then pass through the channel attention branch and the spatial attention branch to obtain channel attention weights and spatial attention weights.

[0020] The compressed features are input into the path routing branch. The average value of the compressed features along the channel dimension is taken to obtain single-channel features. The single-channel features are then segmented into non-overlapping windows of size [missing value]. The window features, each window feature flattened to a length of The vectors are used to calculate the spatial mean and variance of each window feature to obtain the variance feature. The variance feature is then passed sequentially through a first multilayer perceptron, a second ReLU activation function layer, another multilayer perceptron, and a softmax function layer to obtain the routing probability for each window. Based on the routing probability, different windows are routed to either dense or lightweight paths. The input features of the difficulty-aware dynamic routing module are passed through a first linear layer and a second linear layer to obtain the query vector and key vector. The value vector, query vector, and key vector are then divided into non-overlapping segments of size [missing information]. The system generates dense augmented features by first generating value window vectors, query window vectors, and key window vectors; then, it performs self-attention computation on the value window vectors, query window vectors, and key window vectors corresponding to windows assigned to dense paths, resulting in dense augmented features; finally, it multiplies the value window vectors corresponding to windows assigned to dense paths element-wise with spatial attention weights, resulting in lightweight augmented features; then, it reorders the dense and lightweight augmented features output from all windows back to the original spatial layout to form rearranged features; finally, it inputs the rearranged features into a pointwise convolutional layer to obtain dimensionality-reduced features.

[0021] The dimensionality-reduced features are input into three depthwise separable convolutional branches with different kernel sizes to obtain the first shape feature, the second shape feature, and the third shape feature, respectively. The dimensionality-reduced features are added to the first shape feature, the second shape feature, and the third shape feature to obtain the cross-window enhancement feature. The cross-window enhancement feature is then multiplied element-wise with the channel attention weights after passing through the second GELU activation function layer, and then connected with the rearranged feature residual to obtain the output feature of the difficulty-aware dynamic routing module.

[0022] Preferably, the channel attention branch includes a first global average pooling layer, a fourth convolutional layer, and a first sigmoid function layer connected in sequence; the spatial attention branch includes a fifth convolutional layer and a second sigmoid function layer connected in sequence; and the depthwise separable convolutional branch includes convolutional kernels set in parallel with a size of [missing value]. The second depth separable convolutional layer, with a kernel size of The horizontal strip depth can separate convolutional layers and use convolutional kernels for Vertical stripe depth can separate convolutional layers;

[0023] Based on routing probability, different windows are assigned to either dense or lightweight paths, specifically including:

[0024] During the training phase of the image super-resolution model, Gumbel-Softmax hard sampling is used to generate window-level binary routing masks for the routing probabilities. When the value of the window-level binary routing mask is 1, the corresponding window is assigned to a dense path, and when the value of the window-level binary routing mask is 0, the corresponding window is assigned to a lightweight path.

[0025] During the inference phase of the image super-resolution model, the routing probabilities of all windows are sorted from largest to smallest, and the top M windows are assigned to dense paths, while the remaining windows are assigned to lightweight paths.

[0026] Preferably, the space-frequency collaborative Mamba module includes a second segmentation unit, a frequency domain scanning path module, a parameter generation module, a spatial domain scanning path module, and a third linear layer. The parameter generation module includes two parallelly configured first and second prior parameter generation networks. The first prior parameter generation network includes a second global average pooling layer and a sixth convolutional layer connected in sequence, and the second prior parameter generation network includes a third global average pooling layer and a seventh convolutional layer connected in sequence. The input features of the space-frequency collaborative Mamba module first pass through the second segmentation unit to obtain third and fourth segmentation features. The third segmentation features are then input into the frequency domain. The scanning path module obtains frequency domain path features. These features are then processed by a parameter generation module, which first averages the frequency domain path features along the batch dimension to obtain a batch aggregation representation. The batch aggregation representation is then processed by a first prior parameter generation network and a second prior parameter generation network to obtain a first prior parameter and a second prior parameter. These first and second prior parameters are then applied to the spatial domain scanning path module, and the fourth segmentation feature is input into the spatial domain scanning path module to obtain the spatial domain path features. The frequency domain path features and the spatial path features are then concatenated along the channel dimension and projected through a third linear layer to obtain the output features of the space-frequency cooperation Mamba module.

[0027] Preferably, in the frequency domain scanning path module, the third segmentation feature sequentially passes through the first linear extended projection layer and the third segmentation unit to obtain the fifth and sixth segmentation features; the fifth segmentation feature sequentially passes through the third depthwise separable convolutional layer and the first SiLU activation function layer to obtain the first convolutional enhancement feature; the first convolutional enhancement feature is decomposed into the first sub-band image, the second sub-band image, the third sub-band image, and the fourth sub-band image using Haar discrete wavelet transform; the Haar discrete wavelet transform uses four filters to perform convolution operations with the first convolutional enhancement feature and spatial downsampling with a scaling factor of 2, as shown in the following equation:

[0028] ;

[0029] ;

[0030] ;

[0031] ;

[0032] in, This represents the first convolutional enhancement feature. This represents the convolution operation. This indicates spatial downsampling with a scaling factor of 2. , , and There are four filters, with the following expressions:

[0033] ;

[0034] ;

[0035] ;

[0036] ;

[0037] The first, second, third, and fourth sub-band images are flattened into one-dimensional sequences, yielding first, second, third, and fourth sub-band vectors, respectively. These vectors are then subjected to a first-level projection mapping and concatenated to obtain a joint tensor, as shown in the following equation:

[0038] ;

[0039] in, , , and Let these represent the first sub-band vector, the second sub-band vector, the third sub-band vector, and the fourth sub-band vector, respectively. This indicates a dimensional projection operation. This indicates a splicing operation. Represents a joint tensor. Represents the set of real numbers. Correct the dimension for the time step. For the state parameter dimensions, B, H, and W represent the batch size, number of channels, height, and width of the low-resolution image in the training samples, respectively.

[0040] The joint tensor is decomposed into time step parameters along the channel dimension. State input matrix State output matrix Time step parameters The time-step correction matrix is ​​obtained through second-level projection mapping and the Softplus activation function. C represents the number of channels in the low-resolution image of the training samples, and L represents the length of the one-dimensional sequence. ;

[0041] The state transition matrix A and feedforward matrix D in the fixed frequency domain scanning path module;

[0042] Time step correction matrix State input matrix and state output matrix The first state space parameters are constructed, and combined with the state transition matrix A and the feedforward matrix D, a Mamba-based selective state space computation is performed to obtain the enhanced features, as shown in the following equation:

[0043] ;

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] in, This represents the input feature at time step t; This represents the hidden state at time step (t-1). When t=1, , ; Let represent the hidden state at time step t. These represent the corrected state transition matrix and the corrected state input matrix, respectively. This represents the output feature at time step t. This represents the output feature at the Lth time step. Indicates enhanced features, This indicates splicing by time step;

[0049] The enhanced features are reconstructed back to the spatial domain using the inverse Haar wavelet discretization operation, and then processed by a convolution kernel with a size of [missing value]. The fourth deep separable convolutional layer performs feature modulation to obtain frequency domain scanning features. The result of the frequency domain scanning features after passing through the sixth normalization layer is multiplied pixel-by-pixel with the result of the sixth segmentation features after passing through the second SiLU activation function layer. Then, the results are processed through the fourth linear layer for linear projection to obtain the frequency domain path features, as shown in the following formula:

[0050] ;

[0051] ;

[0052] ;

[0053] in, This represents the inverse Haar wavelet discrete transform operation. Presentation layer normalization operation, Indicates the kernel size as Depth-separable convolution operations, Represents frequency domain path characteristics, Indicates frequency domain scanning characteristics; This represents the SiLU activation function. This represents the sixth segmentation feature. This represents a linear projection operation. This represents element-wise multiplication;

[0054] In the spatial domain scanning path module, the fourth segmentation feature passes through the second linear expansion projection layer and the fifth segmentation unit in sequence to obtain the seventh segmentation feature and the eighth segmentation feature; the seventh segmentation feature passes through the fifth depthwise separable convolutional layer and the third SiLU activation function layer in sequence to obtain the second convolutional enhancement feature; the second convolutional enhancement feature passes through the single-scale scanning branch and the multi-scale scanning branch respectively;

[0055] In the single-scale scanning branch, the second convolutional enhancement feature is first flattened into a one-dimensional sequence, and the first-level projection mapping and the second-level projection mapping are used to obtain the second state space parameters. The first set of parameters of the first prior parameter is used as the result of the exponent with base e and then taking the negative number. The first set of parameters of the second prior parameter replaces the state transition matrix and the feedforward matrix respectively, and together with the second state space parameters, Mamba-based selective state space calculation is performed to obtain the single-scale enhancement feature.

[0056] The second convolutional enhancement feature is decomposed into a fifth sub-band image using Haar discrete wavelet transform. Sixth sub-band image Seventh sub-band image and the eighth sub-band image The high-frequency sixth, seventh, and eighth sub-band images are element-wise summed and normalized along the spatial dimension using the Softmax2d function to obtain an attention weight map. This map is then element-wise multiplied with the low-frequency fifth sub-band image and added to the attention weight map via residual connections to obtain the downsampling enhancement feature. Its expression is as follows:

[0057] ;

[0058] ;

[0059] in, Represents the attention weight map. This represents the Softmax2d function;

[0060] In the multi-scale scanning branch, the downsampled enhancement features are expanded into one-dimensional sequences along three different directions: reverse horizontal, vertical, and reverse vertical. The first-level projection mapping and the second-level projection mapping are used to obtain the corresponding third, fourth, and fifth state space parameters. The last three sets of parameters of the first prior are used as the result of the exponent with base e and the negative number of the second prior, respectively, to replace the state transition matrix and the feedforward matrix. They are then combined with the third, fourth, and fifth state space parameters to perform selective state space calculations based on Mamba in the three directions, resulting in multi-scale enhancement features in the three directions.

[0061] The original spatial arrangement order is restored by performing inverse scanning in each direction on the multi-scale enhancement features in three directions. Then, the scale is restored by pixel copying and upsampling to obtain upsampled features. The single-scale enhancement features and upsampled features are then fused by weighted summation to obtain multi-branch fusion output features.

[0062] The multi-branch fusion output features, after passing through the seventh normalization layer, are multiplied pixel-by-pixel with the eighth segmentation features after passing through the fourth SiLU activation function layer. Then, a linear projection operation is performed through the fifth linear layer to obtain the spatial path features. As shown in the following formula:

[0063] ;

[0064] in, This represents the eighth segmentation feature. This indicates the output characteristics of multi-branch fusion.

[0065] Preferably, the local feature extraction module includes a second projective extended convolutional layer, a third GELU activation function layer, a sixth depthwise separable convolutional layer, a fourth GELU activation function layer, a channel attention module, and a second projective reduced convolutional layer connected in sequence.

[0066] The input features of the local feature extraction module are first input into the second projection expansion convolutional layer to perform projection expansion operation along the channel to obtain the second expanded features; the second expanded features are input into the sixth depthwise separable convolutional layer for local spatial feature extraction, and then pass through the fourth GELU activation function layer for nonlinear transformation to obtain local enhanced features;

[0067] Local enhancement features are input into the channel attention module. They first pass through the fourth global average pooling layer, where global average pooling is performed along the channel dimension to compress the spatial information of each channel into a global descriptor, resulting in a channel description vector. The channel description vector then passes through a linear dimensionality reduction layer, a fifth SiLU activation function layer, a linear dimensionality increase layer, and a Sigmoid activation function layer to obtain channel weights. The channel weights are then multiplied pixel-wise with the local enhancement features to obtain the channel attention enhancement features.

[0068] The channel attention enhancement features are input into the second projection reduction convolutional layer for projection reduction along the channels, and then residually connected with the input features of the local feature extraction module to obtain the first intermediate features.

[0069] Secondly, the present invention provides a route-aware and space-frequency cooperative Mamba-driven image super-resolution device, comprising:

[0070] The model building module is configured to construct and train an image super-resolution model driven by route awareness and space-frequency cooperation (SFR), resulting in a trained image super-resolution model. The image super-resolution model includes a first convolutional layer, several route-enhancing SFR modules, a sub-pixel convolutional layer, and a second convolutional layer connected in sequence. The output features of the first convolutional layer are concatenated with the residual output features of the last route-enhancing SFR module and then passed through the sub-pixel convolutional layer and the second convolutional layer in sequence. The route-enhancing SFR module includes a local feature extraction module, a first normalization layer, a SFR Mamba module, a second normalization layer, a first gated convolutional feedforward network, a third normalization layer, a difficulty-aware dynamic routing module, a fourth normalization layer, and a second gated convolutional feedforward network.

[0071] The reconstruction module is configured to acquire the low-resolution image to be reconstructed and input it into a trained image super-resolution model to reconstruct the corresponding high-resolution image. The input features of the route enhancement space-frequency coordination module are first input into the local feature extraction module to obtain the first intermediate feature. The first intermediate feature is then passed through a first normalization layer and a space-frequency coordination Mamba module, and the result is concatenated with the residual of the first intermediate feature to obtain the second intermediate feature. The second intermediate feature is then passed through a second normalization layer and a first gated convolutional feedforward network, and the result is concatenated with the residual of the second intermediate feature to obtain the third intermediate feature. The third intermediate feature is then passed through a third normalization layer and a difficulty-aware dynamic routing module, and the result is concatenated with the residual of the third intermediate feature to obtain the fourth intermediate feature. The fourth intermediate feature is then passed through a fourth normalization layer and a second gated convolutional feedforward network, and the result is concatenated with the residual of the fourth intermediate feature to obtain the output feature of the route enhancement space-frequency coordination module.

[0072] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0073] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0074] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the implementations in the first aspect.

[0075] Compared with the prior art, the present invention has the following beneficial effects:

[0076] (1) The route-aware and space-frequency cooperative Mamba-driven image super-resolution method mentioned in this invention realizes window-level route allocation through a difficulty-aware dynamic routing module. It sends high-difficulty window routes to dense paths and low-difficulty window routes to lightweight paths, thereby achieving content-adaptive computation allocation and avoiding computational redundancy and limited representation.

[0077] (2) The route-aware and space-frequency cooperative Mamba-driven image super-resolution method mentioned in this invention uses wavelet subband decomposition to achieve high-low frequency decoupling and direction adaptive scanning in the frequency domain scanning path module through the space-frequency cooperative Mamba module. In the spatial domain scanning path module, wavelet cross-subband attention is combined to make up for the defects of multi-scale modeling. The first and second prior parameters generated by the frequency domain scanning path module guide the generation of state space parameters of the spatial domain scanning path module, realize the transfer of frequency energy prior to the spatial state model, realize the network of frequency domain and spatiotemporal domain cooperation, further improve the reconstruction quality, and thus enhance the feature representation capability of the image super-resolution model. Attached Figure Description

[0078] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0079] Figure 1 This is a flowchart illustrating the route-aware and space-frequency cooperative Mamba-driven image super-resolution method according to an embodiment of this application.

[0080] Figure 2This is a schematic diagram of the structure of the image super-resolution model based on the route-aware and space-frequency cooperative Mamba-driven image super-resolution method according to an embodiment of this application.

[0081] Figure 3 This is a schematic diagram of the structure of the route enhancement space-frequency coordination module of the Mamba-driven image super-resolution method for route awareness and space-frequency cooperation, as shown in the embodiment of this application.

[0082] Figure 4 This is a schematic diagram of the local feature extraction module of the route-aware and space-frequency cooperative Mamba-driven image super-resolution method according to an embodiment of this application;

[0083] Figure 5 This is a schematic diagram of the structure of the space-frequency cooperative Mamba module in the route-aware and space-frequency cooperative Mamba-driven image super-resolution method according to an embodiment of this application.

[0084] Figure 6 This is a schematic diagram of the spatial scanning path module of the route-aware and space-frequency cooperative Mamba-driven image super-resolution method according to an embodiment of this application.

[0085] Figure 7 This is a schematic diagram of the frequency domain scanning path module of the route-aware and space-frequency cooperative Mamba-driven image super-resolution method according to an embodiment of this application;

[0086] Figure 8 This is a schematic diagram of the gated convolutional feedforward structure of the route-aware and space-frequency cooperative Mamba-driven image super-resolution method according to an embodiment of this application.

[0087] Figure 9 This is a schematic diagram of the structure of the difficulty-aware dynamic routing module of the Mamba-driven image super-resolution method with route awareness and space-frequency cooperation, as shown in the embodiment of this application.

[0088] Figure 10 This is a schematic diagram of a route-aware and space-frequency cooperative Mamba-driven image super-resolution device according to an embodiment of this application;

[0089] Figure 11 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0090] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0091] Figure 1 This application illustrates an embodiment of a route-aware and space-frequency cooperative Mamba-driven image super-resolution method, comprising the following steps:

[0092] S1. Construct and train an image super-resolution model driven by route awareness and space-frequency cooperation Mamba to obtain a trained image super-resolution model. The image super-resolution model includes a first convolutional layer, several route-enhancing space-frequency coordination modules, a sub-pixel convolutional layer, and a second convolutional layer connected in sequence. The output features of the first convolutional layer are concatenated with the output feature residuals of the last route-enhancing space-frequency coordination module and then passed through the sub-pixel convolutional layer and the second convolutional layer in sequence. The route-enhancing space-frequency coordination module includes a local feature extraction module, a first normalization layer, a space-frequency cooperation Mamba module, a second normalization layer, a first gated convolutional feedforward network, a third normalization layer, a difficulty-aware dynamic routing module, a fourth normalization layer, and a second gated convolutional feedforward network.

[0093] For details, please refer to Figure 2 The image super-resolution model based on route awareness and space-frequency cooperation driven by Mamba proposed in the embodiments of this application consists of a first convolutional layer, several route-enhanced space-frequency coordination modules, a sub-pixel convolutional layer, and a second convolutional layer. The first and second convolutional layers use a 3×3 convolutional kernel. The input low-resolution image first passes through the first convolutional layer. The output features of the first convolutional layer pass through several route-enhanced space-frequency coordination modules in sequence. The input features of the first route-enhanced space-frequency coordination module are the output features of the first convolutional layer. The output features of the previous route-enhanced space-frequency coordination module are used as the input features of the next route-enhanced space-frequency coordination module. The output features of the last route-enhanced space-frequency coordination module are added to the output features of the first convolutional layer using a residual connection and then sequentially input to the sub-pixel convolutional layer and the second convolutional layer to finally reconstruct the corresponding high-resolution image.

[0094] refer to Figure 3Each routing enhancement space-frequency coordination module consists of a local feature extraction module, a first normalization layer, a space-frequency cooperation Mamba module (SFCM), a second normalization layer, a first gated convolutional feedforward network, a third normalization layer, a difficulty-aware dynamic routing module (DADR), a fourth normalization layer, and a second gated convolutional feedforward network. The input features of the routing enhancement space-frequency coordination module are first input to the local feature extraction module to obtain the first intermediate feature. The first intermediate feature passes through the first normalization layer and the space-frequency cooperation Mamba module in sequence, and is then connected with the residual of the first intermediate feature to obtain the second intermediate feature. The second intermediate feature passes through the second normalization layer and the first gated convolutional feedforward network in sequence, and is then connected with the residual of the second intermediate feature to obtain the third intermediate feature. The third intermediate feature passes through the third normalization layer and the difficulty-aware dynamic routing module in sequence, and is then connected with the residual of the third intermediate feature to obtain the fourth intermediate feature. The fourth intermediate feature passes through the fourth normalization layer and the second gated convolutional feedforward network in sequence, and is then connected with the residual of the fourth intermediate feature to obtain the output feature of the routing enhancement space-frequency coordination module. The specific calculation formula for the routing enhancement space-frequency coordination module is as follows:

[0095] ;

[0096] ;

[0097] ;

[0098] in, This represents the input characteristics of the routing enhancement space-frequency coordination module. This represents the function corresponding to the local feature extraction module. Indicates the first intermediate feature. Presentation layer normalization operation, The function representing the gated convolution feedforward structure, Indicates the third intermediate feature. This refers to the function corresponding to the space-frequency cooperation Mamba module. This represents the function corresponding to the difficulty-aware dynamic routing module. This indicates the output characteristics of the routing enhancement space-frequency coordination module.

[0099] This invention achieves window-level route allocation through a difficulty-aware dynamic routing module. High-difficulty window routes are routed to denser paths, while low-difficulty window routes are routed to lighter paths, enabling content-adaptive allocation and avoiding computational redundancy and limited representation. Simultaneously, a space-frequency collaborative Mamba module utilizes wavelet subband decomposition in the frequency domain scanning path module to achieve high-low frequency decoupling and adaptive direction scanning. In the spatial domain scanning path module, wavelet cross-subband attention is combined to compensate for multi-scale modeling deficiencies. Furthermore, frequency domain priors guide the generation of state-space parameters in the spatial domain scanning path module, enabling the transfer of frequency energy priors to the spatial state model. This achieves a frequency- and spatial domain collaborative network, further improving reconstruction quality and enhancing the feature representation capabilities of the image super-resolution model. The first, second, third, and fourth normalization layers all employ layer normalization (Layer Norm) operations.

[0100] In a specific embodiment, the local feature extraction module includes a second projective expanded convolutional layer, a third GELU activation function layer, a sixth depthwise separable convolutional layer, a fourth GELU activation function layer, a channel attention module, and a second projective reduced convolutional layer connected in sequence.

[0101] The input features of the local feature extraction module are first input into the second projection expansion convolutional layer to perform projection expansion operation along the channel to obtain the second expanded features; the second expanded features are input into the sixth depthwise separable convolutional layer for local spatial feature extraction, and then pass through the fourth GELU activation function layer for nonlinear transformation to obtain local enhanced features;

[0102] Local enhancement features are input into the channel attention module. They first pass through the fourth global average pooling layer, where global average pooling is performed along the channel dimension to compress the spatial information of each channel into a global descriptor, resulting in a channel description vector. The channel description vector then passes through a linear dimensionality reduction layer, a fifth SiLU activation function layer, a linear dimensionality increase layer, and a Sigmoid activation function layer to obtain channel weights. The channel weights are then multiplied pixel-wise with the local enhancement features to obtain the channel attention enhancement features.

[0103] The channel attention enhancement features are input into the second projection reduction convolutional layer for projection reduction along the channels, and then residually connected with the input features of the local feature extraction module to obtain the first intermediate features.

[0104] For details, please refer to Figure 4The local feature extraction module in the embodiments of this application consists of a second projective expansion convolutional layer, a third GELU activation function layer, a sixth depthwise separable convolutional layer, a fourth GELU activation function layer, a channel attention module, and a second projective reduction convolutional layer. The second projective expansion convolutional layer performs projection expansion using a convolution operation with a kernel size of 1×1. The sixth depthwise separable convolutional layer performs a depthwise separable convolution operation with a kernel size of 3×3, a stride of 1, and padding of 1. The second projective reduction convolutional layer performs projection reduction using a convolution operation with a kernel size of 1×1. The input features of the local feature extraction module are low-resolution images, which are sequentially passed through a second projection expansion convolutional layer, a third GELU activation function layer, a sixth depthwise separable convolutional layer, and a fourth GELU activation function layer to obtain local enhancement features. The local enhancement features are input to the channel attention module for channel attention enhancement to obtain channel attention enhancement features. The channel attention enhancement features are then passed through a second projection reduction convolutional layer for channel projection reduction. Finally, the output features of the second projection reduction convolutional layer are added to the input low-resolution image using a residual connection to obtain the first intermediate feature.

[0105] In the embodiments of this application, a channel attention module is used to adaptively recalibrate the local enhanced features. The channel attention module adopts a compressed-excitation channel attention mechanism. The channel attention module includes a fourth global average pooling layer, a linear dimensionality reduction layer, a fifth SiLU activation function layer, a linear dimensionality increase layer, and a Sigmoid activation function layer connected in sequence. The local enhanced features pass through the fourth global average pooling layer, the linear dimensionality reduction layer, the fifth SiLU activation function layer, the linear dimensionality increase layer, and the Sigmoid activation function layer in sequence to obtain channel weights. Then, the channel weights are used to recalibrate the local enhanced features channel by channel to obtain the channel attention enhanced features.

[0106] In a specific embodiment, the space-frequency cooperative Mamba module includes a second segmentation unit, a frequency domain scanning path module, a parameter generation module, a spatial domain scanning path module, and a third linear layer. The parameter generation module includes two parallelly configured first and second prior parameter generation networks. The first prior parameter generation network includes a second global average pooling layer and a sixth convolutional layer connected in sequence, and the second prior parameter generation network includes a third global average pooling layer and a seventh convolutional layer connected in sequence. The input features of the space-frequency cooperative Mamba module first pass through the second segmentation unit to obtain third and fourth segmentation features. The third segmentation features are then input to... The frequency domain scanning path module obtains frequency domain path features. These features are then processed by a parameter generation module, which first averages the frequency domain path features along the batch dimension to obtain a batch aggregation representation. The batch aggregation representation is then processed by a first prior parameter generation network and a second prior parameter generation network to obtain a first prior parameter and a second prior parameter. These first and second prior parameters are then applied to the spatial domain scanning path module, and the fourth segmentation feature is input into the spatial domain scanning path module to obtain the spatial domain path features. The frequency domain path features and the spatial path features are concatenated along the channel dimension and projected through a third linear layer to obtain the output features of the space-frequency cooperation Mamba module.

[0107] For details, please refer to Figure 5 The space-frequency cooperative Mamba module proposed in this application includes a second segmentation unit, a frequency domain scanning path module (Wave-SS2D), a parameter generation module, a spatial domain scanning path module (MS-SS2D), and a third linear layer. First, the second segmentation unit uniformly divides the input features of the space-frequency cooperative Mamba module along the channel dimension into two parts: a third segmentation feature and a fourth segmentation feature. These third and fourth segmentation features are then used as the input features of the frequency domain scanning path module and the spatial domain scanning path module, respectively. The third segmentation feature is input to the frequency domain scanning path module to obtain the frequency domain path feature. The frequency domain path feature is first averaged along the batch dimension to obtain a batch aggregation representation. This batch aggregation representation is then input to the parallel first and second prior parameter generation networks to generate the first prior parameter. Second prior parameter The first prior parameter generation network includes a second global average pooling layer and a sixth convolutional layer. The second prior parameter generation network includes a third global average pooling layer and a seventh convolutional layer, where both the sixth and seventh convolutional layers use 1×1 convolutional operations. The first prior parameter... The total number of parameters is This refers to the sum of parameters for four scanning directions: one group for single-scale branches and three groups for multi-scale branches. Each group has... One parameter. Second prior parameter. The total number of parameters is There are a total of 4 sets of parameters, each set has One parameter. First prior parameter. Second prior parameter It will be used in the airspace scan path module, and will be used in subsequent applications. As a negative number of a power with base e ( To ensure the stability of state decay. First prior parameter With the second prior parameter The fourth segmentation feature is input into the spatial domain scanning path module to generate spatial domain path features. The frequency domain path features and spatial domain path features output from the two paths are concatenated along the channel dimension and then linearly projected through the third linear layer to obtain the output features of the space-frequency cooperative Mamba module.

[0108] In a specific embodiment, in the frequency domain scanning path module, the third segmentation feature sequentially passes through the first linear extended projection layer and the third segmentation unit to obtain the fifth segmentation feature and the sixth segmentation feature; the fifth segmentation feature sequentially passes through the third depthwise separable convolutional layer and the first SiLU activation function layer to obtain the first convolutional enhancement feature; the first convolutional enhancement feature is decomposed into the first sub-band image, the second sub-band image, the third sub-band image, and the fourth sub-band image using Haar discrete wavelet transform; the Haar discrete wavelet transform uses four filters to perform convolution operations with the first convolutional enhancement feature and spatial downsampling with a scaling factor of 2, as shown in the following equation:

[0109] ;

[0110] ;

[0111] ;

[0112] ;

[0113] in, This represents the first convolutional enhancement feature. This represents the convolution operation. This indicates spatial downsampling with a scaling factor of 2. , , and There are four filters, with the following expressions:

[0114] ;

[0115] ;

[0116] ;

[0117] ;

[0118] The first, second, third, and fourth sub-band images are flattened into one-dimensional sequences, yielding first, second, third, and fourth sub-band vectors, respectively. These vectors are then subjected to a first-level projection mapping and concatenated to obtain a joint tensor, as shown in the following equation:

[0119] ;

[0120] in, , , and Let these represent the first sub-band vector, the second sub-band vector, the third sub-band vector, and the fourth sub-band vector, respectively. This indicates a dimensional projection operation. This indicates a splicing operation. Represents a joint tensor. Represents the set of real numbers. Correct the dimension for the time step. For the state parameter dimension, B, H, and W represent the batch, number of channels, height, and width of the low-resolution image in the training samples, respectively.

[0121] The joint tensor is decomposed into time step parameters along the channel dimension. State input matrix State output matrix Time step parameters The time-step correction matrix is ​​obtained through second-level projection mapping and the Softplus activation function. C represents the number of channels in the low-resolution image of the training samples, and L represents the length of the one-dimensional sequence. ;

[0122] The state transition matrix A and feedforward matrix D in the fixed frequency domain scanning path module;

[0123] Time step correction matrix State input matrix and state output matrix The first state space parameters are constructed, and combined with the state transition matrix A and the feedforward matrix D, a Mamba-based selective state space computation is performed to obtain the enhanced features, as shown in the following equation:

[0124] ;

[0125] ;

[0126] ;

[0127] ;

[0128] ;

[0129] in, This represents the input feature at time step t; This represents the hidden state at time step (t-1). When t=1, , ; Let represent the hidden state at time step t. These represent the corrected state transition matrix and the corrected state input matrix, respectively. This represents the output feature at time step t. This represents the output feature at the Lth time step. Indicates enhanced features, This indicates splicing by time step;

[0130] The enhanced features are reconstructed back to the spatial domain using the inverse Haar wavelet discretization operation, and then processed by a convolution kernel with a size of [missing value]. The fourth deep separable convolutional layer performs feature modulation to obtain frequency domain scanning features. The result of the frequency domain scanning features after passing through the sixth normalization layer is multiplied pixel-by-pixel with the result of the sixth segmentation features after passing through the second SiLU activation function layer. Then, the results are processed through the fourth linear layer for linear projection to obtain the frequency domain path features, as shown in the following formula:

[0131] ;

[0132] ;

[0133] ;

[0134] in, This represents the inverse Haar wavelet discrete transform operation. Presentation layer normalization operation, Indicates the kernel size as Depth-separable convolution operations, Represents frequency domain path characteristics, Indicates frequency domain scanning characteristics; This represents the SiLU activation function. This represents the sixth segmentation feature. This represents a linear projection operation. This represents element-wise multiplication;

[0135] refer to Figure 6In the spatial scanning path module, the fourth segmentation feature passes through the second linear expansion projection layer and the fifth segmentation unit in sequence to obtain the seventh segmentation feature and the eighth segmentation feature; the seventh segmentation feature passes through the fifth depthwise separable convolutional layer and the third SiLU activation function layer in sequence to obtain the second convolutional enhancement feature; the second convolutional enhancement feature passes through the single-scale scanning branch and the multi-scale scanning branch respectively.

[0136] In the single-scale scanning branch, the second convolutional enhancement feature is first flattened into a one-dimensional sequence, and the first-level projection mapping and the second-level projection mapping are used to obtain the second state space parameters. The first set of parameters of the first prior parameter is used as the result of the exponent with base e and then taking the negative number. The first set of parameters of the second prior parameter replaces the state transition matrix and the feedforward matrix respectively, and together with the second state space parameters, Mamba-based selective state space calculation is performed to obtain the single-scale enhancement feature.

[0137] The second convolutional enhancement feature is decomposed into a fifth sub-band image using Haar discrete wavelet transform. Sixth sub-band image Seventh sub-band image and the eighth sub-band image The high-frequency sixth, seventh, and eighth sub-band images are element-wise summed and normalized along the spatial dimension using the Softmax2d function to obtain an attention weight map. This map is then element-wise multiplied with the low-frequency fifth sub-band image and added to the attention weight map via residual connections to obtain the downsampling enhancement feature. Its expression is as follows:

[0138] ;

[0139] ;

[0140] in, Represents the attention weight map. This represents the Softmax2d function;

[0141] In the multi-scale scanning branch, the downsampled enhancement features are expanded into one-dimensional sequences along three different directions: reverse horizontal, vertical, and reverse vertical. The first-level projection mapping and the second-level projection mapping are used to obtain the corresponding third, fourth, and fifth state space parameters. The last three sets of parameters of the first prior are used as the result of the exponent with base e and the negative number of the second prior, respectively, to replace the state transition matrix and the feedforward matrix. They are then combined with the third, fourth, and fifth state space parameters to perform selective state space calculations based on Mamba in the three directions, resulting in multi-scale enhancement features in the three directions.

[0142] The original spatial arrangement order is restored by performing inverse scanning in each direction on the multi-scale enhancement features in three directions. Then, the scale is restored by pixel copying and upsampling to obtain upsampled features. The single-scale enhancement features and upsampled features are then fused by weighted summation to obtain multi-branch fusion output features.

[0143] The multi-branch fusion output features, after passing through the seventh normalization layer, are multiplied pixel-by-pixel with the eighth segmentation features after passing through the fourth SiLU activation function layer. Then, a linear projection operation is performed through the fifth linear layer to obtain the spatial path features. As shown in the following formula:

[0144] ;

[0145] in, This represents the eighth segmentation feature. This indicates the output characteristics of multi-branch fusion.

[0146] For details, please refer to Figure 7 The third segmentation feature is input into the frequency domain scanning path module. First, it undergoes a linear expansion projection operation through the first linear expansion projection layer. Then, the third segmentation unit is used to split along the channel dimension to obtain the fifth and sixth segmentation features, as shown in the following formula:

[0147] ;

[0148] in, Indicates the third segmentation feature. This represents a channel segmentation operation, which divides the input feature map into two equal parts along the channel dimension; denoted as linear extended projection operation; x represents the fifth segmentation feature, which serves as the main path input for state space scanning; z represents the sixth segmentation feature, which serves as the gating signal for the subsequent SiLU gating mechanism.

[0149] The fifth segmentation feature is input into the third depthwise separable convolutional layer, and then processed by the first SiLU activation function layer to obtain the first convolutional enhancement feature; the third depthwise separable convolutional layer uses a convolution kernel of... A depthwise separable convolution with padding of 1 and group number C is performed. Then, the Haar discrete wavelet transform is used to decompose the first convolution enhancement feature into four sub-band images, namely... , , and ,in, , , and Using low-frequency filters and three high-frequency filters , , Enhanced features with the first convolution The images are obtained by performing a convolution operation with a stride of 2. Next, the four sub-band images are independently input into the state-space model for selective horizontal scanning. Each sub-band image is first flattened into a one-dimensional sequence, and then the first state-space parameters are generated through first-level and second-level projection mapping. A learnable first-level projection matrix is ​​used in the first-level projection mapping. In a second-level projection mapping, a second-level projection matrix is ​​used. In the frequency domain scanning path module, the state transition matrix A and the feedforward parameters D are fixed, learnable parameters that are not dynamically generated depending on the input. The state transition matrix A is generated using the S4D initialization method, and the feedforward parameters D are initialized as an all-one vector. The four scanning directions share four independent sets of state transition matrices A and feedforward parameters D. Based on the first state space parameters, the state transition matrix A, and the feedforward parameters D, Mamba-based selective state space computation is performed to obtain the enhanced features. .

[0150] Finally, the enhanced features of the four sub-band images are reconstructed back into the spatial domain using the inverse Haar wavelet discrete transform (IDWT), and then processed by a convolution kernel. The fourth depthwise separable convolutional layer is used for feature modulation to obtain frequency domain scanning features. The frequency domain scanning features are then input into a six-layer normalization layer for layer normalization, and then gatedly multiplied with the result of the sixth segmentation feature z after passing through the second SiLU activation function layer. Finally, the frequency domain path features are projected through the fourth linear layer to obtain the frequency domain path features. .

[0151] In the spatial domain scanning path module, the same processing as in the frequency domain scanning path module is first applied, inputting the fourth segmentation feature into the second linear expansion projection layer and the fifth segmentation unit to obtain the seventh segmentation feature x' and the eighth segmentation feature z'. The seventh segmentation feature is then sequentially passed through the fifth depthwise separable convolutional layer and the third SiLU activation function layer to obtain the second convolutional enhancement feature. The fifth depthwise separable convolutional layer uses a convolution kernel of... A depthwise separable convolution operation with padding of 1 and grouping number C.

[0152] Then, a single-scale scanning branch is constructed. The second convolutional enhanced features are flattened into a one-dimensional sequence, and state space modeling is performed using the same operations as the frequency domain scanning path module, using the first set of first-level projection matrices. With second-level projection matrix Generate second state space parameters , , The first prior parameters generated by the frequency domain scanning path module for prior guidance The first set of parameters and the second prior parameters The first set of parameters is used to perform selective state-space computation based on Mamba to obtain single-scale enhanced features.

[0153] Next, the second convolutional enhancement feature is processed using Haar discrete wavelet transform to obtain four sub-band images. The three high-frequency sub-band images are added element-wise and normalized along the spatial dimension using the Softmax2d function to obtain the attention weight map. This map is then multiplied element-wise with the low-frequency sub-band image and added to the attention weight map through residual connection to obtain the enhanced downsampling feature.

[0154] Next, a multi-scale scanning branch is constructed. The enhanced downsampling features are unfolded into one-dimensional sequences along three different directions: reverse horizontal, vertical, and reverse vertical. State-space parameters for each direction are generated using their respective two sets of projection matrices, in conjunction with prior guidance. and The last three sets of parameters are used to perform selective state-space computation based on Mamba in three directions, resulting in multi-scale augmented features in three directions. Then, the original spatial arrangement is restored by inverse scanning of each direction's multi-scale augmented features. The output of each direction is then upsampled using a pixel copy operation to achieve scale restoration and obtain upsampled features. In one example, the pixel copy upsampling operation uses a 2×2 all-one matrix for the Kronecker product. The single-scale augmented features and the upsampled features are then fused using a learnable weighted summation to obtain multi-branch fused output features. These multi-branch fused output features are passed through a seventh normalization layer, then multiplied pixel-wise with the result of the eighth segmentation feature after passing through a fourth SiLU activation function layer. Finally, a linear projection operation is performed through a fifth linear layer to obtain spatial path features.

[0155] In a specific embodiment, both the first gated convolutional feedforward network and the second gated convolutional feedforward network adopt a gated convolutional feedforward structure. The gated convolutional feedforward structure includes a first projective expansion convolutional layer, a first depthwise separable convolutional layer, a first segmentation unit, a first GELU activation function layer, and a first projective reduction convolutional layer. The input features of the gated convolutional feedforward structure are first input into the first projective expansion convolutional layer to perform a projection expansion operation along the channel to obtain the first expanded features, as shown in the following formula:

[0156] ;

[0157] in, This represents the input features of the gated convolutional feedforward structure. Indicates the first extended feature, This indicates that the convolution kernel is... The projection spreading operation performed by the convolutional layer;

[0158] The first depthwise separable convolutional layer is used to perform local feature extraction on the first extended feature. Each channel is convolved independently to obtain local features, as shown in the following formula:

[0159] ;

[0160] in, Indicates local features, Indicates the kernel size as The first depth separable convolutional layer;

[0161] The local features are input into the first segmentation unit, and average segmentation is performed along the channel dimension to obtain the first segmentation feature and the second segmentation feature; the first segmentation unit is input into the first GELU activation function layer for nonlinear transformation, and then multiplied with the second segmentation feature as a gate to obtain the gated feature;

[0162] The gated features are input into the first projection-reduced convolutional layer and projected and reduced along the channel to obtain the output features of the gated convolutional feedforward structure, as shown in the following equation:

[0163] ;

[0164] in, Indicates gating features, This indicates that the convolution kernel is... The projection reduction operation performed by the convolutional layer This represents the output characteristics of the gated convolution feedforward structure.

[0165] For details, please refer to Figure 8 In the embodiments of this application, both the first gated convolutional feedforward network and the second gated convolutional feedforward network adopt a gated convolutional feedforward structure. The input features of the gated convolutional feedforward structure first undergo projection expansion operation along the channel in the first projection expansion convolutional layer to obtain the first expanded features. In the first projection expansion convolutional layer, the kernel size is... The convolution operation achieves projection expansion; the first expanded feature is then input into the first depthwise separable convolutional layer for local feature extraction. Each channel is convolved independently to obtain local features. The first depthwise separable convolutional layer uses a kernel size of [missing information]. The depthwise separable convolution operation is performed. The first segmentation unit divides the local features along the channel dimension into first and second segmentation features. The first segmentation feature is then non-linearly transformed using a first GELU activation function layer and multiplied with the second segmentation feature to enhance its representational power, resulting in gated features. These gated features are then input into a first projection reduction convolutional layer for projection reduction along the channel, yielding the output features of the gated convolutional feedforward structure. The first projection reduction convolutional layer uses a convolution kernel with a size of... Convolution operations are used to achieve projection reduction.

[0166] In a specific embodiment, the difficulty-aware dynamic routing module includes a third convolutional layer, a fifth normalization layer, a first ReLU activation function layer, a channel attention branch, a spatial attention branch, a path routing branch, and an enhancement branch. The input features of the difficulty-aware dynamic routing module first pass through the third convolutional layer to obtain a value vector. The value vector undergoes feature compression and then passes through the fifth normalization layer and the first ReLU activation function layer to obtain compressed features. The compressed features then pass through the channel attention branch and the spatial attention branch to obtain channel attention weights and spatial attention weights.

[0167] The compressed features are input into the path routing branch. The average value of the compressed features along the channel dimension is taken to obtain single-channel features. The single-channel features are then segmented into non-overlapping windows of size [missing value]. The window features, each window feature flattened to a length of The vectors are used to calculate the spatial mean and variance of each window feature to obtain the variance feature. The variance feature is then passed sequentially through a first multilayer perceptron, a second ReLU activation function layer, another multilayer perceptron, and a softmax function layer to obtain the routing probability for each window. Based on the routing probability, different windows are routed to either dense or lightweight paths. The input features of the difficulty-aware dynamic routing module are passed through a first linear layer and a second linear layer to obtain the query vector and key vector. The value vector, query vector, and key vector are then divided into non-overlapping segments of size [missing information]. The system generates dense augmented features by first generating value window vectors, query window vectors, and key window vectors; then, it performs self-attention computation on the value window vectors, query window vectors, and key window vectors corresponding to windows assigned to dense paths, resulting in dense augmented features; finally, it multiplies the value window vectors corresponding to windows assigned to dense paths element-wise with spatial attention weights, resulting in lightweight augmented features; then, it reorders the dense and lightweight augmented features output from all windows back to the original spatial layout to form rearranged features; finally, it inputs the rearranged features into a pointwise convolutional layer to obtain dimensionality-reduced features.

[0168] The dimensionality-reduced features are input into three depthwise separable convolutional branches with different kernel sizes to obtain the first shape feature, the second shape feature, and the third shape feature, respectively. The dimensionality-reduced features are added to the first shape feature, the second shape feature, and the third shape feature to obtain the cross-window enhancement feature. The cross-window enhancement feature is then multiplied element-wise with the channel attention weights after passing through the second GELU activation function layer, and then connected with the rearranged feature residual to obtain the output feature of the difficulty-aware dynamic routing module.

[0169] In a specific embodiment, the channel attention branch includes a first global average pooling layer, a fourth convolutional layer, and a first sigmoid function layer connected in sequence; the spatial attention branch includes a fifth convolutional layer and a second sigmoid function layer connected in sequence; and the depthwise separable convolutional branch includes convolutional kernels set in parallel with a size of [missing value]. The second depth separable convolutional layer, with a kernel size of The horizontal strip depth can separate convolutional layers and use convolutional kernels for Vertical stripe depth can separate convolutional layers;

[0170] Based on routing probability, different windows are assigned to either dense or lightweight paths, specifically including:

[0171] During the training phase of the image super-resolution model, Gumbel-Softmax hard sampling is used to generate window-level binary routing masks for the routing probabilities. When the value of the window-level binary routing mask is 1, the corresponding window is assigned to a dense path, and when the value of the window-level binary routing mask is 0, the corresponding window is assigned to a lightweight path.

[0172] During the inference phase of the image super-resolution model, the routing probabilities of all windows are sorted from largest to smallest, and the top M windows are assigned to dense paths, while the remaining windows are assigned to lightweight paths.

[0173] For details, please refer to Figure 9 The difficulty-aware dynamic routing module mentioned in the embodiments of this application consists of a third convolutional layer, a fifth normalization layer, a first ReLU activation function layer, a channel attention branch, a spatial attention branch, a path routing branch, and an enhancement branch. The input features of the difficulty-aware dynamic routing module are first processed by a convolutional kernel with a kernel size of... The third convolutional layer projects the vector V, which is then subjected to feature compression. This compressed vector V is then processed by the fifth normalization layer and the first ReLU activation function layer to obtain the compressed features. Then compress the features. Two different branches are input to generate channel attention weights. Spatial attention weights In the channel attention branch, features are compressed. First, it passes through the first global average pooling layer, then through a convolutional kernel with a size of... The fourth convolutional layer is used for dimensional expansion, and the first sigmoid function layer is used for non-linear transformation to obtain channel attention weights; in the spatial attention branch, features are compressed. From the kernel size The fifth convolutional layer is used for dimensional projection, and then the second sigmoid function layer is used for nonlinear transformation to obtain spatial attention weights.

[0174] Next, the compression feature will be... The average value along the channel dimension is used to obtain single-channel features. These single-channel features are then segmented into non-overlapping segments. There are 10 windows, each window unfolded to a length of 1... The vector. Calculate the spatial mean and variance for each window to obtain the variance characteristics. ,in The total number of windows. Variance features. The routing probability for each window is mapped using two layers of multilayer perceptrons; the first multilayer perceptron is used to reduce the dimension of the variance feature from... Furthermore, it needs to pass through a second ReLU activation function layer, using a second multilayer perceptron to reduce its dimension from... Then, the routing probability of each window is obtained by normalization through the Softmax function layer.

[0175] During the model training phase, Gumbel-Softmax hard sampling is used to generate window-level binary routing masks. Among them, The window is routed to a dense path, The windows are routed to lightweight paths. During the model inference phase, windows are sorted by routing probability, and the top M windows are routed to dense paths, while the remaining windows are routed to lightweight paths. In one example, .

[0176] Next, the input features of the difficulty-aware dynamic routing module are passed through the first linear layer. With the second linear layer to be converted into query vectors With key vector The value vector V, query vector Q, and key vector Press the same way Window segmentation yields the value window vector V n Query window vector Q n Key window vector .

[0177] Next, according to the path allocation principle, the corresponding condition calculations are performed on the routing mask:

[0178] Dense paths (i.e.) (At time), self-attention computation is performed on the value window vector, query window vector, and key window vector within the window to which the route is assigned to the dense path to obtain dense augmented features. .

[0179] Lightweight path (i.e.) (At that time), for the value window vector assigned to the lightweight path, the spatial gating mechanism is directly used, without performing attention calculation, to obtain lightweight augmented features.

[0180] Then, all the dense and lightweight enhancement features of the windows are reordered back into the original spatial layout to form a rearranged feature. Because independent window processing leads to a lack of information interaction between adjacent windows, the features are rearranged. The input to the convolution kernel size is In the pointwise convolutional layers, dimensionality reduction features are obtained. Then, the dimensionality reduction features The first shape feature is obtained by inputting it into three separable convolution branches with different depths of convolution kernels. Second shape features and third shape features In one example, three separable convolutional branches with different depths of convolutional kernels are respectively used with convolutional kernels of... Depth-separable convolution operations, using convolution kernels as The horizontal strip depth can separate convolution operations and the use of convolution kernels for The vertical strip depth can separate convolution operations.

[0181] Finally, dimensionality reduction features The cross-window enhanced feature is obtained by adding the first shape feature, the second shape feature, and the third shape feature element by element. The cross-window enhancement features are passed through the second GELU activation function layer and multiplied element-wise with the channel attention weights, and then connected with the rearranged feature residuals to obtain the output features of the difficulty-aware dynamic routing module (DADR).

[0182] S2: Obtain the low-resolution image to be reconstructed and input it into the trained image super-resolution model to reconstruct the corresponding high-resolution image. The input features of the routing enhancement space-frequency coordination module are first input into the local feature extraction module to obtain the first intermediate feature. The first intermediate feature is then passed through the first normalization layer and the space-frequency coordination Mamba module, and the result is concatenated with the residual of the first intermediate feature to obtain the second intermediate feature. The second intermediate feature is then passed through the second normalization layer and the first gated convolutional feedforward network, and the result is concatenated with the residual of the second intermediate feature to obtain the third intermediate feature. The third intermediate feature is then passed through the third normalization layer and the difficulty-aware dynamic routing module, and the result is concatenated with the residual of the third intermediate feature to obtain the fourth intermediate feature. The fourth intermediate feature is then passed through the fourth normalization layer and the second gated convolutional feedforward network, and the result is concatenated with the residual of the fourth intermediate feature to obtain the output feature of the routing enhancement space-frequency coordination module.

[0183] Specifically, the trained image super-resolution model from the embodiments of this application is deployed. The low-resolution image to be reconstructed is input into the trained image super-resolution model, and an upsampling factor, scale, is applied. The scale value is the required magnification factor, for example, scale is 2, 3, or 4. This upsampling factor, scale, participates in the reconstruction process as a scale factor in the sub-pixel convolutional layer. The specific process is as follows:

[0184] First, using a convolution kernel with a size of The first convolutional layer extracts shallow features. Its expression is as follows:

[0185] ;

[0186] in, Indicates the kernel size as The function of the first convolutional layer.

[0187] Secondly, deep features are extracted using A routing enhancement space-frequency coordination modules. Its expression is as follows:

[0188] ;

[0189] ;

[0190] in, This represents the function of the i-th routing enhancement space-frequency coordination module, where In one example, A is typically set to 5.

[0191] Finally, a sub-pixel convolutional layer with a scale factor of scale and a convolutional kernel of scale are used. The second convolutional layer for depth features Upsampling and reconstruction generate the corresponding high-resolution image. Its expression is as follows:

[0192] ;

[0193] in, The function that represents a subpixel convolutional layer.

[0194] In the embodiments of this application, the data used for training and inference are all obtained in a legal and compliant manner, and the application scenarios are also legal and compliant.

[0195] Further reference Figure 10 As an implementation of the methods shown in the above figures, this application provides an embodiment of a route-aware and space-frequency cooperative Mamba-driven image super-resolution device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0196] This application provides a route-aware and space-frequency cooperative Mamba-driven image super-resolution device, comprising:

[0197] Model building module 1 is configured to build and train an image super-resolution model driven by route awareness and space-frequency cooperation (SFR), resulting in a trained image super-resolution model. The image super-resolution model includes a first convolutional layer, several route-enhancing SFR modules, a sub-pixel convolutional layer, and a second convolutional layer connected in sequence. The output features of the first convolutional layer are concatenated with the residual output features of the last route-enhancing SFR module and then passed through the sub-pixel convolutional layer and the second convolutional layer in sequence. The route-enhancing SFR module includes a local feature extraction module, a first normalization layer, a SFR Mamba module, a second normalization layer, a first gated convolutional feedforward network, a third normalization layer, a difficulty-aware dynamic routing module, a fourth normalization layer, and a second gated convolutional feedforward network.

[0198] Reconstruction module 2 is configured to acquire the low-resolution image to be reconstructed and input it into a trained image super-resolution model to reconstruct the corresponding high-resolution image. The input features of the routing enhancement space-frequency coordination module are first input into the local feature extraction module to obtain the first intermediate feature. The first intermediate feature is then passed through a first normalization layer and a space-frequency coordination Mamba module, and the result is concatenated with the residual of the first intermediate feature to obtain the second intermediate feature. The second intermediate feature is then passed through a second normalization layer and a first gated convolutional feedforward network, and the result is concatenated with the residual of the second intermediate feature to obtain the third intermediate feature. The third intermediate feature is then passed through a third normalization layer and a difficulty-aware dynamic routing module, and the result is concatenated with the residual of the third intermediate feature to obtain the fourth intermediate feature. The fourth intermediate feature is then passed through a fourth normalization layer and a second gated convolutional feedforward network, and the result is concatenated with the residual of the fourth intermediate feature to obtain the output feature of the routing enhancement space-frequency coordination module.

[0199] Figure 11 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. For example... Figure 11 As shown, the electronic device in this embodiment includes a processor 1101 and a memory 1102; wherein the memory 1102 is used to store computer execution instructions; and the processor 1101 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0200] Alternatively, the memory 1102 can be either standalone or integrated with the processor 1101.

[0201] When the memory 1102 is set up independently, the electronic device also includes a bus 1103 for connecting the memory 1102 and the processor 1101.

[0202] This invention also provides a computer storage medium storing computer execution instructions, which, when executed by the processor 1101, implement the above method.

[0203] This invention also provides a computer program product, including a computer program that, when executed by a processor 1101, implements the above-described method.

[0204] In the embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0205] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0206] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit formed by the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0207] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor 1101 to execute some steps of the methods of the various embodiments of this application.

[0208] It should be understood that the processor 1101 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor, or the processor 1101 can be any conventional processor 1101. The steps of the method disclosed in this invention can be directly manifested as the hardware processor 1101 executing the steps, or as a combination of hardware and software modules within the processor 1101 executing the steps.

[0209] The memory 1102 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0210] Bus 1103 can be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 1103 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 1103 in the accompanying drawings of this application is not limited to only one bus 1103 or one type of bus 1103.

[0211] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0212] An exemplary storage medium is coupled to a processor 1101, enabling the processor 1101 to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor 1101. The processor 1101 and the storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor 1101 and the storage medium can exist as discrete components in an electronic device or a host device.

[0213] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0214] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A route-aware and space-frequency cooperative Mamba-driven image super-resolution method, characterized in that, Includes the following steps: An image super-resolution model driven by route awareness and spatial-frequency cooperation (SFR) is constructed and trained to obtain a trained image super-resolution model. The image super-resolution model includes a first convolutional layer, several route-enhancing SFR modules, a sub-pixel convolutional layer, and a second convolutional layer connected in sequence. The output features of the first convolutional layer are concatenated with the residual output features of the last route-enhancing SFR module and then passed sequentially through the sub-pixel convolutional layer and the second convolutional layer. The route-enhancing SFR module includes a local feature extraction module, a first normalization layer, a SFR module, a second normalization layer, a first gated convolutional feedforward network, a third normalization layer, a difficulty-aware dynamic routing module, a fourth normalization layer, and a second gated convolutional feedforward network. The SFR module includes a second segmentation unit, a frequency domain scanning path module, a parameter generation module, a spatial domain scanning path module, and a third linear layer. The parameter generation module includes two parallel first prior parameter generation networks and a second prior parameter generation network. The first prior parameter generation network includes a second global average pooling layer and a sixth convolutional layer connected in sequence. The second prior parameter generation network... The number generation network includes a third global average pooling layer and a seventh convolutional layer connected in sequence. The input features of the space-frequency cooperative Mamba module first pass through the second segmentation unit to obtain third and fourth segmentation features. The third segmentation features are input to the frequency domain scanning path module to obtain frequency domain path features. The frequency domain path features pass through the parameter generation module, where the average value of the frequency domain path features along the batch dimension is taken to obtain a batch aggregation representation. The batch aggregation representation passes through the first and second prior parameter generation networks to obtain first and second prior parameters. The first and second prior parameters are applied to the spatial domain scanning path module, and the fourth segmentation feature is input to the spatial domain scanning path module to obtain spatial path features. The frequency domain path features and the spatial path features are concatenated along the channel dimension and projected through the third linear layer to obtain the output features of the space-frequency cooperative Mamba module. The difficulty-aware dynamic routing module includes a third convolutional layer, a fifth normalization layer, a first ReLU activation function layer, a channel attention branch, a spatial attention branch, a path routing branch, and an enhancement branch. The input features of the difficulty-aware dynamic routing module first pass through the third convolutional layer to obtain a value vector. The value vector undergoes feature compression and then passes through the fifth normalization layer and the first ReLU activation function layer to obtain compressed features. The compressed features are processed through the channel attention branch and the spatial attention branch respectively to obtain the channel attention weights and the spatial attention weights; The compressed features are input into the path routing branch. The average value of the compressed features along the channel dimension is taken to obtain single-channel features. The single-channel features are then segmented into non-overlapping windows of size [missing value]. The window features, each window feature flattened to a length of The vectors are used to calculate the spatial mean and variance of each window feature to obtain the variance feature. The variance feature is then passed sequentially through a first multilayer perceptron, a second ReLU activation function layer, another multilayer perceptron, and a Softmax function layer to obtain the routing probability of each window. Based on this routing probability, different windows are assigned to dense or lightweight paths. The input features of the difficulty-aware dynamic routing module are passed through a first linear layer and a second linear layer to obtain the query vector and key vector. The value vector, query vector, and key vector are then divided into non-overlapping segments of size [missing information]. The value window vector, query window vector, and key window vector are used to calculate the value window vector, query window vector, and key window vector corresponding to the window assigned to the dense path, thereby obtaining the dense enhancement feature. The value window vector corresponding to the window assigned to the dense path is multiplied element-wise with the spatial attention weight to obtain the lightweight augmented feature; the dense augmented features and lightweight augmented features output by all windows are reordered back to the original spatial layout to form the rearranged feature; the rearranged feature is input into the pointwise convolutional layer to obtain the dimensionality reduction feature; The dimensionality reduction features are input into three depthwise separable convolutional branches with different kernel sizes to obtain a first shape feature, a second shape feature, and a third shape feature, respectively; the dimensionality reduction features are added to the first shape feature, the second shape feature, and the third shape feature to obtain the cross-window enhancement feature; The cross-window enhancement feature is multiplied element-wise by the channel attention weight after passing through the second GELU activation function layer, and then connected with the rearranged feature residual to obtain the output feature of the difficulty-aware dynamic routing module. A low-resolution image to be reconstructed is acquired and input into the trained image super-resolution model to reconstruct the corresponding high-resolution image. The input features of the routing enhancement space-frequency coordination module are first input into the local feature extraction module to obtain a first intermediate feature. The first intermediate feature is then passed through the first normalization layer and the space-frequency coordination Mamba module, and the result is concatenated with the residual of the first intermediate feature to obtain a second intermediate feature. The second intermediate feature is then passed through the second normalization layer and the first gated convolutional feedforward network, and the result is concatenated with the residual of the second intermediate feature to obtain a third intermediate feature. The third intermediate feature is then passed through the third normalization layer and the difficulty-aware dynamic routing module, and the result is concatenated with the residual of the third intermediate feature to obtain a fourth intermediate feature. The fourth intermediate feature is then passed through the fourth normalization layer and the second gated convolutional feedforward network, and the result is concatenated with the residual of the fourth intermediate feature to obtain the output feature of the routing enhancement space-frequency coordination module.

2. The route-aware and space-frequency cooperative Mamba-driven image super-resolution method according to claim 1, characterized in that, Both the first and second gated convolutional feedforward networks employ a gated convolutional feedforward structure. This structure includes a first projective expansion convolutional layer, a first depthwise separable convolutional layer, a first segmentation unit, a first GELU activation function layer, and a first projective reduction convolutional layer. The input features of this gated convolutional feedforward structure are first fed into the first projective expansion convolutional layer for projection expansion along the channels, resulting in the first expanded features, as shown in the following equation: ； in, This represents the input features of the gated convolutional feedforward structure. Indicates the first extended feature, This indicates that the convolution kernel is... The projection spreading operation performed by the convolutional layer; The first depthwise separable convolutional layer is used to perform local feature extraction on the first extended feature. Each channel is convolved independently to obtain local features, as shown in the following equation: ； in, Indicates local features, Indicates the kernel size as The first depth separable convolutional layer; The local features are input into the first segmentation unit, and average segmentation is performed along the channel dimension to obtain the first segmentation feature and the second segmentation feature; the first segmentation unit is input into the first GELU activation function layer for nonlinear transformation, and then multiplied with the second segmentation feature as a gate to obtain the gated feature; The gated features are input into the first projection-reduced convolutional layer and projection-reduced along the channel to obtain the output features of the gated convolutional feedforward structure, as shown in the following equation: ； in, Indicates gating features, This indicates that the convolution kernel is... The projection reduction operation performed by the convolutional layer This represents the output characteristics of the gated convolution feedforward structure.

3. The route-aware and space-frequency cooperative Mamba-driven image super-resolution method according to claim 1, characterized in that, The channel attention branch includes a first global average pooling layer, a fourth convolutional layer, and a first sigmoid function layer connected in sequence; the spatial attention branch includes a fifth convolutional layer and a second sigmoid function layer connected in sequence; the depthwise separable convolutional branch includes convolutional kernels set in parallel with a size of [missing information]. The second depth separable convolutional layer, with a kernel size of The horizontal strip depth can separate convolutional layers and use convolutional kernels for Vertical stripe depth can separate convolutional layers; Based on the routing probabilities, different windows are assigned to dense or lightweight paths, specifically including: During the training phase of the image super-resolution model, the routing probabilities are hard-sampled using Gumbel-Softmax to generate window-level binary routing masks. When the value of the window-level binary routing mask is 1, the corresponding window is assigned to the dense path; when the value of the window-level binary routing mask is 0, the corresponding window is assigned to the lightweight path. During the inference phase of the image super-resolution model, the routing probabilities of all windows are sorted from largest to smallest, and the top M windows are assigned to the dense path, while the remaining windows are assigned to the lightweight path.

4. The route-aware and space-frequency cooperative Mamba-driven image super-resolution method according to claim 1, characterized in that, In the frequency domain scanning path module, the third segmentation feature sequentially passes through the first linear extended projection layer and the third segmentation unit to obtain the fifth segmentation feature and the sixth segmentation feature; the fifth segmentation feature sequentially passes through the third depthwise separable convolutional layer and the first SiLU activation function layer to obtain the first convolutional enhancement feature; the first convolutional enhancement feature is decomposed into a first sub-band image, a second sub-band image, a third sub-band image, and a fourth sub-band image using Haar discrete wavelet transform; the Haar discrete wavelet transform uses four filters to perform convolution operations with the first convolutional enhancement feature and spatial downsampling with a scaling factor of 2, as shown in the following equation: ； ； ； ； in, This represents the first convolutional enhancement feature. This represents the convolution operation. This indicates spatial downsampling with a scaling factor of 2. , , and There are four filters, with the following expressions: ； ； ； ； The first, second, third, and fourth sub-band images are flattened into one-dimensional sequences to obtain first, second, third, and fourth sub-band vectors, respectively. First-level projection mapping is then performed on each of these vectors, and they are concatenated to obtain a joint tensor, as shown in the following equation: ； in, , , and Let these represent the first sub-band vector, the second sub-band vector, the third sub-band vector, and the fourth sub-band vector, respectively. This indicates a dimensional projection operation. This indicates a splicing operation. Represents a joint tensor. Represents the set of real numbers. Correct the dimension for the time step. For the state parameter dimension, B, H, and W represent the batch, number of channels, height, and width of the low-resolution image in the training samples, respectively. The joint tensor is decomposed into time step parameters along the channel dimension. State input matrix State output matrix The time step parameters The time-step correction matrix is ​​obtained through second-level projection mapping and the Softplus activation function. C represents the number of channels in the low-resolution image of the training samples, and L represents the length of the one-dimensional sequence. ; The state transition matrix A and feedforward matrix D in the frequency domain scanning path module are fixed; The time step correction matrix State input matrix and state output matrix The first state space parameters are constructed, and combined with the state transition matrix A and the feedforward matrix D, a Mamba-based selective state space computation is performed to obtain the enhanced features, as shown in the following equation: ； ； ； ； ； in, This represents the input feature at time step t; This represents the hidden state at time step (t-1). When t=1, , ; Let represent the hidden state at time step t. These represent the corrected state transition matrix and the corrected state input matrix, respectively. This represents the output feature at time step t. This represents the output feature at the Lth time step. Indicates enhanced features, This indicates splicing by time step; The enhanced features are reconstructed back into the spatial domain using an inverse Haar wavelet discretization operation, and then processed by a convolution kernel with a size of [missing value]. The fourth deep separable convolutional layer performs feature modulation to obtain frequency domain scanning features; the result of the frequency domain scanning features after passing through the sixth normalization layer is multiplied pixel-by-pixel with the result of the sixth segmentation features after passing through the second SiLU activation function layer, and then linearly projected through the fourth linear layer to obtain the frequency domain path features, as shown in the following formula: ； ； in, This represents the inverse Haar wavelet discrete transform operation. Presentation layer normalization operation, Indicates the kernel size as Depth-separable convolution operations, Represents frequency domain path characteristics, Indicates frequency domain scanning characteristics; This represents the SiLU activation function. This represents the sixth segmentation feature. This represents a linear projection operation. This represents element-wise multiplication; In the spatial scanning path module, the fourth segmentation feature passes through the second linear expansion projection layer and the fifth segmentation unit in sequence to obtain the seventh segmentation feature and the eighth segmentation feature; the seventh segmentation feature passes through the fifth depthwise separable convolutional layer and the third SiLU activation function layer in sequence to obtain the second convolutional enhancement feature; the second convolutional enhancement feature passes through the single-scale scanning branch and the multi-scale scanning branch respectively; In the single-scale scanning branch, the second convolutional enhancement feature is first flattened into a one-dimensional sequence, and the first-level projection mapping and the second-level projection mapping are used to obtain the second state space parameters. The first set of parameters of the first prior parameter is used as the result of the exponent of the first prior parameter and the negative number of the second prior parameter to replace the state transition matrix and the feedforward matrix respectively, and together with the second state space parameters, the selective state space calculation based on Mamba is performed to obtain the single-scale enhancement feature. The second convolutional enhancement feature is decomposed into a fifth sub-band image using Haar discrete wavelet transform. Sixth sub-band image Seventh sub-band image and the eighth sub-band image The high-frequency sixth, seventh, and eighth sub-band images are element-wise summed and normalized along the spatial dimension using the Softmax2d function to obtain an attention weight map. This map is then element-wise multiplied with the low-frequency fifth sub-band image and added to the attention weight map via residual connections to obtain the downsampling enhancement feature. Its expression is as follows: ； ； in, Represents the attention weight map. This represents the Softmax2d function; In the multi-scale scanning branch, the downsampling enhancement features are expanded into one-dimensional sequences along three different directions: reverse horizontal, vertical, and reverse vertical. The first-level projection mapping and the second-level projection mapping are used to obtain the corresponding third-state space parameters, fourth-state space parameters, and fifth-state space parameters. The last three sets of parameters of the first prior parameter are used as the result of the exponent with base e and the negative number, and the last three sets of parameters of the second prior parameter are used to replace the state transition matrix and the feedforward matrix, respectively. They are then combined with the third-state space parameters, fourth-state space parameters, and fifth-state space parameters to perform selective state space calculations based on Mamba in the three directions, thereby obtaining multi-scale enhancement features in the three directions. The original spatial arrangement order is restored by performing inverse scanning in each direction on the multi-scale enhancement features in the three directions. Then, the scale is restored by pixel copying upsampling operation to obtain upsampled features. The single-scale enhancement features and the upsampled features are then fused by weighted summation to obtain multi-branch fusion output features. The multi-branch fusion output feature, after passing through the seventh normalization layer, is multiplied pixel-by-pixel with the eighth segmentation feature, after passing through the fourth SiLU activation function layer. Then, a linear projection operation is performed through the fifth linear layer to obtain the spatial path feature. As shown in the following formula: ； in, This represents the eighth segmentation feature. This indicates the output characteristics of multi-branch fusion.

5. The route-aware and space-frequency cooperative Mamba-driven image super-resolution method according to claim 1, characterized in that, The local feature extraction module includes a second projective expanded convolutional layer, a third GELU activation function layer, a sixth depthwise separable convolutional layer, a fourth GELU activation function layer, a channel attention module, and a second projective reduced convolutional layer, connected in sequence. The input features of the local feature extraction module are first input into the second projection-expanded convolutional layer to perform projection-expanding operations along the channels to obtain the second expanded features; the second expanded features are then input into the sixth depthwise separable convolutional layer for local spatial feature extraction, and then subjected to nonlinear transformation by the fourth GELU activation function layer to obtain local enhanced features; The local enhancement features are input into the channel attention module. They first pass through the fourth global average pooling layer, where global average pooling is performed along the channel dimension to compress the spatial information of each channel into a global descriptor, resulting in a channel description vector. The channel description vector then passes through a linear dimensionality reduction layer, a fifth SiLU activation function layer, a linear dimensionality increase layer, and a Sigmoid activation function layer to obtain channel weights. The channel weights are then multiplied pixel-by-pixel by the local enhancement features to obtain the channel attention enhancement features. The channel attention enhancement feature is input into the second projection reduction convolutional layer to perform projection reduction operation along the channel, and then residually connected with the input feature of the local feature extraction module to obtain the first intermediate feature.

6. A route-aware and space-frequency cooperative Mamba-driven image super-resolution device, characterized in that, include: The model building module is configured to construct and train an image super-resolution model driven by route awareness and space-frequency cooperation (SFR), resulting in a trained image super-resolution model. The image super-resolution model includes a first convolutional layer, several route-enhancing SFR modules, a sub-pixel convolutional layer, and a second convolutional layer connected in sequence. The output features of the first convolutional layer are concatenated with the residual output features of the last route-enhancing SFR module and then passed sequentially through the sub-pixel convolutional layer and the second convolutional layer. The route-enhancing SFR module includes a local feature extraction module, a first normalization layer, a SFR Mamba module, a second normalization layer, a first gated convolutional feedforward network, a third normalization layer, a difficulty-aware dynamic routing module, a fourth normalization layer, and a second gated convolutional feedforward network. The SFR Mamba module includes a second segmentation unit, a frequency domain scanning path module, a parameter generation module, a spatial domain scanning path module, and a third linear layer. The parameter generation module includes two parallel first prior parameter generation networks and a second prior parameter generation network. The first prior parameter generation network includes a second global average pooling layer and a sixth convolutional layer connected in sequence. The second prior parameter generation network includes a third global average pooling layer and a seventh convolutional layer connected in sequence. The input features of the space-frequency cooperative Mamba module first pass through the second segmentation unit to obtain third and fourth segmentation features. The third segmentation features are input to the frequency domain scanning path module to obtain frequency domain path features. The frequency domain path features pass through the parameter generation module, where the average value of the frequency domain path features along the batch dimension is taken to obtain a batch aggregation representation. The batch aggregation representation passes through the first and second prior parameter generation networks to obtain first and second prior parameters. The first and second prior parameters are applied to the spatial domain scanning path module, and the fourth segmentation feature is input to the spatial domain scanning path module to obtain spatial path features. The frequency domain path features and the spatial path features are concatenated along the channel dimension and projected through the third linear layer to obtain the output features of the space-frequency cooperative Mamba module. The difficulty-aware dynamic routing module includes a third convolutional layer, a fifth normalization layer, a first ReLU activation function layer, a channel attention branch, a spatial attention branch, a path routing branch, and an enhancement branch. The input features of the difficulty-aware dynamic routing module first pass through the third convolutional layer to obtain a value vector. The value vector undergoes feature compression and then passes through the fifth normalization layer and the first ReLU activation function layer to obtain compressed features. The compressed features are processed through the channel attention branch and the spatial attention branch respectively to obtain the channel attention weights and the spatial attention weights; The compressed features are input into the path routing branch. The average value of the compressed features along the channel dimension is taken to obtain single-channel features. The single-channel features are then segmented into non-overlapping windows of size [missing value]. The window features, each window feature flattened to a length of The vectors are used to calculate the spatial mean and variance of each window feature to obtain the variance feature. The variance feature is then passed sequentially through a first multilayer perceptron, a second ReLU activation function layer, another multilayer perceptron, and a Softmax function layer to obtain the routing probability of each window. Based on this routing probability, different windows are assigned to dense or lightweight paths. The input features of the difficulty-aware dynamic routing module are passed through a first linear layer and a second linear layer to obtain the query vector and key vector. The value vector, query vector, and key vector are then divided into non-overlapping segments of size [missing information]. The value window vector, query window vector, and key window vector are used to calculate the value window vector, query window vector, and key window vector corresponding to the window assigned to the dense path, thereby obtaining the dense enhancement feature. The value window vector corresponding to the window assigned to the dense path is multiplied element-wise with the spatial attention weight to obtain the lightweight augmented feature; the dense augmented features and lightweight augmented features output by all windows are reordered back to the original spatial layout to form the rearranged feature; the rearranged feature is input into the pointwise convolutional layer to obtain the dimensionality reduction feature; The dimensionality reduction features are input into three depthwise separable convolutional branches with different kernel sizes to obtain a first shape feature, a second shape feature, and a third shape feature, respectively; the dimensionality reduction features are added to the first shape feature, the second shape feature, and the third shape feature to obtain the cross-window enhancement feature; The cross-window enhancement feature is multiplied element-wise by the channel attention weight after passing through the second GELU activation function layer, and then connected with the rearranged feature residual to obtain the output feature of the difficulty-aware dynamic routing module. The reconstruction module is configured to acquire the low-resolution image to be reconstructed and input it into the trained image super-resolution model to reconstruct the corresponding high-resolution image. The input features of the routing enhancement space-frequency coordination module are first input into the local feature extraction module to obtain a first intermediate feature. The first intermediate feature is then passed through a first normalization layer and a space-frequency coordination Mamba module, and the result is concatenated with the residual of the first intermediate feature to obtain a second intermediate feature. The second intermediate feature is then passed through a second normalization layer and a first gated convolutional feedforward network, and the result is concatenated with the residual of the second intermediate feature to obtain a third intermediate feature. The third intermediate feature is then passed through a third normalization layer and a difficulty-aware dynamic routing module, and the result is concatenated with the residual of the third intermediate feature to obtain a fourth intermediate feature. The fourth intermediate feature is then passed through a fourth normalization layer and a second gated convolutional feedforward network, and the result is concatenated with the residual of the fourth intermediate feature to obtain the output feature of the routing enhancement space-frequency coordination module.

7. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images