Image super-resolution reconstruction method of state space model based on geometric transformation enhancement

Abstract

The invention discloses an image super-resolution reconstruction method based on a geometric transformation enhanced state space model, and belongs to the technical field of image processing and computer vision. The method solves the problem of poor reconstruction quality caused by insufficient long-range dependence modeling capability, weak geometric transformation robustness and insufficient local texture detail recovery of the existing method. The method comprises the following steps: constructing a high-resolution and low-resolution image pair data set and constructing a reconstruction network; training the network by using the training set; and inputting a to-be-processed low-resolution image into the trained network to obtain a high-resolution image. The network comprises a shallow feature extraction module, a plurality of residual state space groups connected in series and a reconstruction module. Each residual state space group comprises a plurality of visual state space layers, and each visual state space layer sequentially performs geometric transformation enhancement, double-branch parallel processing, feature fusion, double residual learning and inverse geometric transformation on input. The method is used for converting a low-resolution image into a high-resolution image.

Description

Technical Field

[0001] This invention belongs to the field of image processing and computer vision technology. Background Technology

[0002] Image super-resolution reconstruction technology typically refers to recovering detailed information from low-resolution images into high-resolution ones. It has wide applications in scenarios such as security monitoring, remote sensing imaging, medical imaging, and mobile terminal imaging. In recent years, deep learning-based image super-resolution methods have made significant progress in both objective metrics and visual effects. Convolutional neural networks can mine rich local texture features by stacking multiple layers of convolution and nonlinear activation. However, limited by the receptive field of the convolution kernel, their ability to model long-distance dependencies is limited, making it difficult to fully utilize the structural associations between distant pixels.

[0003] To enhance long-range modeling capabilities, some studies have introduced self-attention or transformer structures to improve reconstruction quality through global correlation modeling. However, self-attention mechanisms typically require calculating the correlation matrix across the entire feature map, leading to rapidly increasing computational complexity and storage overhead with resolution, making them difficult to deploy on edge devices or in real-time applications. To balance performance and efficiency, some lightweight network structures have emerged, compressing model size through methods such as reducing the number of channels, pruning, or low-rank decomposition. However, these methods often sacrifice the ability to recover complex textures and edge details, easily resulting in overly smooth textures and lost details.

[0004] Recent state-space models offer a novel approach to sequence modeling. Some works have attempted to introduce one-dimensional selective scanning state-space structures into image restoration tasks, achieving long-distance dependency modeling with lower computational complexity. However, existing image super-resolution methods based on state-space models generally suffer from the following limitations: First, most methods scan feature sequences only along a fixed direction, making it difficult to simultaneously consider structural consistency in the horizontal, vertical, and different geometric transformations, resulting in insufficient robustness to geometric changes such as rotation and flipping. Second, the collaborative modeling ability between the state-space backbone and convolutional branches is limited, lacking effective fusion with prior information related to local spatial correlations, which can easily lead to insufficient reconstruction of texture details. Third, while some lightweight designs reduce the number of parameters and computational load, they lack efficient fusion mechanisms for features from different branches, resulting in insufficient cross-branch information interaction and limiting the overall network's representational capabilities.

[0005] Therefore, while keeping the number of model parameters and computational complexity under control, how to simultaneously utilize geometric transformation enhancement, orientation-selective state space modeling, and the local texture extraction capabilities of convolutional branches to achieve high-quality and high-efficiency image super-resolution reconstruction remains a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] This invention addresses the problems of poor image reconstruction quality caused by insufficient long-range dependency modeling capabilities, weak robustness to geometric transformations, and inadequate recovery of local texture details in existing super-resolution methods. A new image super-resolution reconstruction method based on a state-space model enhanced by geometric transformations is provided.

[0007] The image super-resolution reconstruction method based on a state-space model enhanced by geometric transformation as described in this invention includes:

[0008] Step 1: Construct a dataset containing high-resolution images and corresponding low-resolution images from different scenes as a training set;

[0009] A network for image super-resolution reconstruction is constructed using network mapping relationships;

[0010] Step 2: Train the image super-resolution reconstruction network using the training set to obtain the trained image super-resolution reconstruction network;

[0011] Step 3: Input the low-resolution image to be processed into the trained image super-resolution reconstruction network to obtain the corresponding high-resolution image;

[0012] The image super-resolution reconstruction network includes: a shallow feature extraction module, multiple cascaded residual state space groups, residual connection and reconstruction module;

[0013] The shallow feature extraction module is used to extract shallow features from the input low-resolution image to obtain shallow features;

[0014] The shallow features are sequentially processed through a series of residual state space groups for deep feature extraction.

[0015] Within each residual state space group, the input features are processed sequentially through multiple visual state space layers and a convolutional operation layer.

[0016] In each visual state space layer, the input features are sequentially subjected to geometric transformation enhancement, dual-branch parallel processing, feature fusion, nonlinear mapping and inverse geometric transformation of the multilayer perception mechanism to obtain the output features of that layer;

[0017] The convolutional operation layer aggregates the features after they have been processed by multiple visual state space layers;

[0018] The residual connection and reconstruction module performs residual connection between the features aggregated by convolutional features and the shallow features to obtain deep reconstruction features; then, the deep reconstruction features are rearranged and upsampled to output a high-resolution reconstructed image.

[0019] Furthermore, in this invention, the visual state space layer includes a geometric transformation enhancement module, a dual-branch parallel processing module, a feature fusion module, a multilayer perceptron module, and an inverse geometric transformation module; wherein,

[0020] The geometric transformation enhancement module is used to apply geometric transformations to the input features in the spatial dimension to obtain the enhanced features.

[0021] The geometric transformation method used by the geometric transformation enhancement module is taken from a preset set of geometric transformations; the set of geometric transformations includes six geometric transformation methods: identity transformation, horizontal flip, vertical flip, transpose, simultaneous horizontal and vertical flip, and 180-degree rotation.

[0022] In different visual state spatial layers, the geometric transformation enhancement modules cyclically select the corresponding geometric transformation mode from the geometric transformation set according to the layer index order; specifically, for the first... Each visual state space layer employs a geometric transformation method determined by the layer index. The result of taking the modulus of 6 determines that the six geometric transformation methods are used in rotation and cycle between different layers. The first layer selects the identity transformation, the second layer selects the horizontal flip, the third layer selects the vertical flip, and so on. The sixth layer selects the 180-degree rotation, the seventh layer selects the identity transformation again, and the thirteenth layer selects the identity transformation again. In this way, the six transformation methods cycle once.

[0023] The dual-branch parallel processing module is connected to the geometric transformation enhancement module and is used to divide the geometrically transformed features into a left branch and a right branch along the channel dimension for parallel processing.

[0024] The left branch uses a lightweight convolutional neural network for local feature extraction; the right branch uses a direction-selective state-space model to scan and model the feature sequence along a predetermined direction.

[0025] The feature fusion module is connected to the dual-branch parallel processing module. It is used to concatenate the features output by the left branch and the features output by the right branch in the channel dimension, and perform channel shuffling operation on the concatenated features to perform inter-branch interaction and fusion of features output by different branches to obtain fused features.

[0026] The fused features are residually connected with the input features to obtain a first residual connection feature, which is then output to the multilayer perceptron module.

[0027] The multilayer perceptron module is used to perform nonlinear mapping on the fused features after residual connection in the channel domain, enhance the feature representation, and obtain the enhanced features;

[0028] The enhanced feature is re-concatenated with the first residual connection feature to obtain the second residual connection feature, and the second residual connection feature is output to the inverse geometric transformation module.

[0029] The inverse geometric transformation module is used to apply an inverse geometric transformation corresponding to the geometric transformation enhancement module to the second residual connection feature, restore the feature to the original spatial coordinate system, and obtain the final output feature of the visual state space layer.

[0030] Furthermore, in this invention, the shallow feature extraction module is constructed using a channel expansion and shuffling strategy, specifically: the input low-resolution image is processed... The expanded image is obtained by copying the channel dimensions by a factor of 1. The extended image Perform within the channel dimension according to the preset number of groups. Channel shuffling of scaling factor to obtain shuffled features ; and then the characteristics after mixing and washing Apply 3×3 convolution operation To obtain initial shallow features ;

[0031]

[0032] in, This represents the input low-resolution image. This represents a 3×3 convolution operation.

[0033] Furthermore, in this invention, the formula for extracting deep features by sequentially passing the shallow features through a series of residual state space groups is as follows:

[0034]

[0035] in, This indicates the number of sets of cascaded residual state-space groups. Indicates deep features, This indicates a cascade operation. Indicates the first A set of residual state spaces, .

[0036] Specifically, initial shallow features Input to the first residual state space group For any ; A set of residual state spaces (in, Its input is the previous residual state space set. The output, finally, the first The output of each residual state space set is determined as the deep feature. .

[0037] Furthermore, in this invention, the formula for obtaining the corresponding high-resolution image in step three is:

[0038]

[0039] in, The high-resolution image output by the network is used to super-resolution reconstruct images of the trained images.

[0040] Furthermore, in this invention, the left branch uses a lightweight convolutional neural network for local feature extraction as follows:

[0041] The input features are sequentially passed through the first batch of normalization layers. First 3×3 convolutional layer, second batch normalization layer, first ReLU activation function The process involves a second 3×3 convolutional layer, a third batch normalization layer, a second ReLU activation function, a 1×1 convolutional layer, and a third ReLU activation function to extract local spatial texture and detail information. The number of channels in all convolutional layers is the same as or matches the number of channels in the left branch. The output of the left branch is:

[0042]

[0043] in, This indicates the output of the left branch. This indicates the characteristics of the branch input in the dual-branch parallel processing module. Indicates activation function, Indicates normalization operation, This represents a 1×1 convolution operation.

[0044] Furthermore, in this invention, the method for the right branch to scan and model the feature sequence along a predetermined direction using a direction-selective state-space model is as follows:

[0045] The orientation-selective state-space model scans and models the feature sequence along the horizontal direction of the image in odd-numbered visual state-space layers;

[0046] The feature sequence is scanned and modeled along the vertical direction of the image in the even-numbered visual state space layer;

[0047] The long-range dependencies between sequences are modeled using state-space equations, and the output of the model is obtained as follows: :

[0048]

[0049]

[0050]

[0051] in, This indicates the output of the left branch. This indicates the characteristics of the branch input in the dual-branch parallel processing module. Presentation layer normalization operation, Indicates a linear layer. This represents a 3×3 depthwise separable convolution. It is an activation function. Indicates priors with local spatial correlation Unidirectional state space scanning modeling.

[0052] Furthermore, in this invention, before the direction-selective state-space model scans and models the feature sequence along a predetermined direction, it also includes a process of calculating the prior local spatial correlation, the process specifically including:

[0053] A 5×5 depthwise separable convolution is applied to the features input to the orientation-selective state space model to form a local correlation enhancement branch. Local correlation information is extracted and then fused with the output projection features of the orientation-selective state space model in a channel-by-channel manner before being output back to the orientation-selective state space model. The local correlation information is as follows:

[0054]

[0055] in, This represents the Sigmoid activation function. The features input to the direction-selective state-space model represent the characteristics. This indicates the application of a 5×5 depth separable convolution operation. This represents the local correlation information extracted through convolution.

[0056] Furthermore, in this invention, the process by which the feature fusion module obtains the fused features is as follows:

[0057] Output features of the left branch With right branch output characteristics The splicing is performed along the channel dimension to obtain the splicing features. Perform channel shuffling operation on the splicing features. The number of groups in the channel shuffling corresponds to the number of dual branches, so that the features output by different branches are rearranged between each group, realizing the inter-branch interaction and fusion of the features output by different branches;

[0058]

[0059] in, This represents the fused features of the residual state space group output.

[0060] Furthermore, in this invention, the first residual connection feature is:

[0061]

[0062] in, This is the first residual connectivity feature. The first scaling factor, As a feature of fusion, For input features.

[0063] Furthermore, in this invention, the enhanced features are:

[0064]

[0065] in, Indicates the enhanced features, This represents a multi-layer perceptron. This represents the second scaling factor.

[0066] This invention first introduces multiple geometric transformations and their corresponding inverse geometric transformations before and after the visual state space layer, enabling the network to explicitly access feature representations of different poses during training and inference, thus enhancing the model's robustness to geometric changes such as flipping and rotation. Then, it employs a parallel structure of lightweight convolutional branches and orientation-selective state space branches. On one hand, convolution efficiently extracts local texture and edge information; on the other hand, it captures long-distance dependencies through an alternating horizontal and vertical scanning state space model, balancing local and global modeling capabilities while maintaining manageable complexity. Next, it achieves deep fusion between branches through channel stitching and channel shuffling, and combines local correlation priors and a dual residual scaling mechanism to effectively improve the sufficiency and stability of feature representation. Finally, the entire network adopts a shallow feature extraction with channel expansion and a pixel rearrangement upsampling structure, significantly reducing the number of parameters and computational load while ensuring reconstruction quality, making it suitable for efficient image super-resolution reconstruction in edge devices and real-time application scenarios. Attached Figure Description

[0067] Figure 1 This is a flowchart of the method described in this invention;

[0068] Figure 2 This is a schematic diagram of signal transmission in the visual state space layer, where c represents the feature after geometric transformation. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0070] Specific implementation method one: Refer to Figure 1 and 2 This embodiment specifically describes the image super-resolution reconstruction method based on a state-space model enhanced by geometric transformation, which includes:

[0071] Step 1: Construct a dataset containing high-resolution images and corresponding low-resolution images from different scenes as a training set;

[0072] A network for image super-resolution reconstruction is constructed using network mapping relationships;

[0073] Step 2: Train the image super-resolution reconstruction network using the training set to obtain the trained image super-resolution reconstruction network;

[0074] Step 3: Input the low-resolution image to be processed into the trained image super-resolution reconstruction network to obtain the corresponding high-resolution image;

[0075] The image super-resolution reconstruction network includes: a shallow feature extraction module, multiple cascaded residual state space groups, residual connection and reconstruction module;

[0076] The shallow feature extraction module is used to extract shallow features from the input low-resolution image to obtain shallow features;

[0077] The shallow features are sequentially processed through a series of residual state space groups for deep feature extraction.

[0078] Within each residual state space group, the input features are processed sequentially through multiple visual state space layers and a convolutional operation layer.

[0079] In each visual state space layer, the input features are sequentially subjected to geometric transformation enhancement, dual-branch parallel processing, feature fusion, nonlinear mapping and inverse geometric transformation of the multilayer perception mechanism to obtain the output features of that layer;

[0080] The convolutional operation layer aggregates the features after they have been processed by multiple visual state space layers;

[0081] The residual connection and reconstruction module performs residual connection between the features aggregated by convolutional features and the shallow features to obtain deep reconstruction features; then, the deep reconstruction features are rearranged and upsampled to output a high-resolution reconstructed image.

[0082] Furthermore, in this embodiment, the visual state space layer includes a geometric transformation enhancement module, a dual-branch parallel processing module, a feature fusion module, a multilayer perceptron module, and an inverse geometric transformation module; wherein,

[0083] The geometric transformation enhancement module is used to apply geometric transformations to the input features in the spatial dimension to obtain the enhanced features.

[0084] The geometric transformation includes at least one of the following: identity transformation, horizontal flip, vertical flip, transpose, simultaneous horizontal and vertical flip, and 180-degree rotation.

[0085] The geometric transformation types of the geometric transformation enhancement modules in different visual state spatial layers are selected in rotation from the transformation set according to the layer index;

[0086] Specifically, for the i-th visual state space layer, the geometric transformation method adopted is determined by the result of taking the layer index i modulo 6, so that the six geometric transformation methods are used in turn and cycled between different layers.

[0087] The dual-branch parallel processing module is connected to the geometric transformation enhancement module and is used to divide the geometrically transformed features into a left branch and a right branch along the channel dimension for parallel processing.

[0088] The left branch uses a lightweight convolutional neural network for local feature extraction; the right branch uses a direction-selective state-space model to scan and model the feature sequence along a predetermined direction.

[0089] The feature fusion module is connected to the dual-branch parallel processing module. It is used to concatenate the features output by the left branch and the features output by the right branch in the channel dimension, and perform channel shuffling operation on the concatenated features to perform inter-branch interaction and fusion of features output by different branches to obtain fused features.

[0090] The fused features are residually connected with the input features to obtain a first residual connection feature, which is then output to the multilayer perceptron module.

[0091] The multilayer perceptron module is used to perform nonlinear mapping on the fused features after residual connection in the channel domain, enhance the feature representation, and obtain the enhanced features;

[0092] The enhanced feature is re-concatenated with the first residual connection feature to obtain the second residual connection feature, and the second residual connection feature is output to the inverse geometric transformation module.

[0093] The inverse geometric transformation module is used to apply an inverse geometric transformation corresponding to the geometric transformation enhancement module to the second residual connection feature, restore the feature to the original spatial coordinate system, and obtain the final output feature of the visual state space layer.

[0094] Furthermore, in this embodiment, the shallow feature extraction module is constructed using a channel expansion and shuffling strategy, specifically: the input low-resolution image is processed... The expanded image is obtained by copying the channel dimensions by a factor of 1. ,

[0095] The extended image Perform within the channel dimension according to the preset number of groups. Channel shuffling of scaling factor to obtain shuffled features ; and then the characteristics after mixing and washing Apply 3×3 convolution operation To obtain initial shallow features ;

[0096]

[0097] in, This represents the input low-resolution image. This represents a 3×3 convolution operation;

[0098] Furthermore, in this embodiment, the formula for extracting deep features by sequentially passing the shallow features through a series of residual state space groups is as follows:

[0099]

[0100] in, This indicates the number of sets of cascaded residual state-space groups. Indicates deep features, This indicates a cascade operation. Indicates the first A set of residual state spaces, .

[0101] Specifically, initial shallow features Input to the first residual state space group For any ; A set of residual state spaces (in, Its input is the previous residual state space set. The output, finally, the first The output of each residual state space set is determined as the deep feature. .

[0102] Furthermore, in this embodiment, the formula for obtaining the corresponding high-resolution image in step three is:

[0103]

[0104] in, The high-resolution image output by the network is used to super-resolution reconstruct images of the trained images.

[0105] Furthermore, in this embodiment, the left branch uses a lightweight convolutional neural network for local feature extraction as follows:

[0106] The input features are sequentially passed through the first batch of normalization layers. First 3×3 convolutional layer, second batch normalization layer, first ReLU activation function The process involves a second 3×3 convolutional layer, a third batch normalization layer, a second ReLU activation function, a 1×1 convolutional layer, and a third ReLU activation function to extract local spatial texture and detail information. The number of channels in all convolutional layers is the same as or matches the number of channels in the left branch. The output of the left branch is:

[0107]

[0108] in, This indicates the output of the left branch. This indicates the characteristics of the branch input in the dual-branch parallel processing module. Indicates activation function, Indicates normalization operation, This represents a 1×1 convolution operation.

[0109] Furthermore, in this embodiment, the method for the right branch to scan and model the feature sequence along a predetermined direction using a direction-selective state-space model is as follows:

[0110] The orientation-selective state-space model scans and models the feature sequence along the horizontal direction of the image in odd-numbered visual state-space layers;

[0111] The feature sequence is scanned and modeled along the vertical direction of the image in the even-numbered visual state space layer;

[0112] The long-range dependencies between sequences are modeled using state-space equations, and the output of the model is obtained as follows: :

[0113]

[0114]

[0115]

[0116] in, This indicates the output of the left branch. This indicates the characteristics of the branch input in the dual-branch parallel processing module. Presentation layer normalization operation, Indicates a linear layer. This represents a 3×3 depthwise separable convolution. It is an activation function. Indicates priors with local spatial correlation Unidirectional state space scanning modeling.

[0117] Furthermore, in this embodiment, before the direction-selective state-space model scans and models the feature sequence along a predetermined direction, it also includes a process of calculating the prior local spatial correlation, the process specifically including:

[0118] A 5×5 depthwise separable convolution is applied to the features input to the orientation-selective state space model to form a local correlation enhancement branch. Local correlation information is extracted and then fused with the output projection features of the orientation-selective state space model in a channel-by-channel manner before being output back to the orientation-selective state space model. The local correlation information is as follows:

[0119]

[0120] in, This represents the Sigmoid activation function. The features input to the direction-selective state-space model represent the characteristics. This indicates the application of a 5×5 depth separable convolution operation. This represents the local correlation information extracted through convolution.

[0121] Furthermore, in this embodiment, the process by which the feature fusion module obtains the fused features is as follows:

[0122] Output features of the left branch With right branch output characteristics The splicing is performed along the channel dimension to obtain the splicing features. Perform channel shuffling operation on the splicing features. The number of groups in the channel shuffling corresponds to the number of dual branches, so that the features output by different branches are rearranged between each group, realizing the inter-branch interaction and fusion of the features output by different branches;

[0123]

[0124] in, This represents the fused features of the residual state space group output.

[0125] Furthermore, in this embodiment, the first residual connection feature is:

[0126]

[0127] in, First residual connectivity feature, The first scaling factor, As a feature of fusion, For input features.

[0128] Furthermore, in this embodiment, the enhanced feature is as follows:

[0129]

[0130] in, Indicates the enhanced features, This represents a multi-layer perceptron. This represents the second scaling factor.

[0131] In this invention, the high-resolution image (HR) refers to the original, realistic image containing rich details and clear textures, which is the ultimate goal of model restoration. The low-resolution image (LR) refers to an image obtained by degrading the HR image in a specific way, resulting in a loss of detail, and is the input to the model; the two are relative terms.

[0132] This implementation primarily performs channel expansion and shallow feature extraction on input low-resolution images. Shallow features are input into a concatenated residual state space group. A geometric transformation enhancement module applies various spatial geometric transformations such as flipping, transposing, and rotating to the features. Lightweight convolutional branches extract local texture details, while direction-selective state space branches alternately scan along the horizontal and vertical directions to model long-range dependencies. Cross-branch information interaction is achieved through channel stitching and channel shuffling. A dual residual structure with a learnable scaling factor is introduced between local convolutional enhancement and the multilayer perceptron to stabilize training and improve expressive power. After progressive refinement through multi-level residual state space groups, the images are added to the shallow feature residuals via convolutional aggregation and reconstructed using pixel rearrangement and upsampling. During the training phase, pixel-domain reconstruction loss and other methods are used for end-to-end optimization of the network. This scheme maintains controllable parameter count and computational complexity while considering both local texture modeling and global structure modeling capabilities. It exhibits strong robustness to geometric changes and is suitable for image super-resolution tasks with high reconstruction quality requirements, such as security monitoring, remote sensing imaging, and medical imaging. Specifically, this embodiment includes:

[0133] S1. Construct an image training set and build an image super-resolution reconstruction network;

[0134] The image super-resolution reconstruction network includes a shallow feature extraction module, multiple cascaded residual state space groups, residual connections, and a reconstruction module.

[0135] S1.1 The shallow feature extraction module performs shallow feature extraction on the input low-resolution image to obtain shallow features;

[0136] The shallow feature extraction module is constructed using a channel expansion and shuffling strategy, specifically: the input low-resolution image is processed... Multiplier of channel dimensions The expanded image is obtained, and then the expanded image is grouped into preset number of groups along the channel dimension. Channel shuffling of scale factor To enhance cross-channel information interaction, a 3×3 convolution operation is then applied to the shuffled features. To obtain the initial shallow feature representation :

[0137]

[0138] in, This represents the input low-resolution image.

[0139] The shallow features are input into multiple cascaded residual state space sets to extract the deep features. :

[0140]

[0141] in, The number of residual state space sets in series is characterized by, .

[0142] Deep features are processed through convolutional operations and aggregated with shallow features to form the backbone output features, ultimately resulting in a super-resolution image output by the network. It can be represented as:

[0143]

[0144] S1.2. The shallow features are used as input and sequentially fed into multiple cascaded residual state space groups. Each residual state space group includes several visual state space layers and a convolution operation. Each visual state space layer includes:

[0145] S1.2.1, Geometric Transformation Enhancement Module, applies multiple geometric transformations to the input features in the spatial dimension, the geometric transformations including at least one of identity transformation, horizontal flip, vertical flip, transpose, simultaneous horizontal and vertical flip, and 180-degree rotation;

[0146] The transformation set of the geometric transformation enhancement module It includes all or some of the following six basic transformation operations: identity transformation, horizontal flip, vertical flip, matrix transpose, simultaneous horizontal and vertical flip, and 180-degree rotation. The geometric transformation types of different layers in the network are selected in rotation from the transformation set according to the layer index, and the geometric transformation used by each layer is restored to the original spatial structure through the corresponding inverse geometric transformation after completing the bi-branch processing and feature fusion.

[0147]

[0148] S1.2.2, the dual-branch parallel processing module divides the geometrically transformed features into a left branch and a right branch along the channel dimension. The left branch uses a convolutional neural network for local feature extraction, while the right branch uses a direction-selective state-space model to scan and model the feature sequence along a predetermined direction.

[0149] The left branch convolutional neural network of the dual-branch parallel processing module adopts a lightweight structure with depthwise separable convolutions. The left branch sequentially includes: a first batch of normalization layers. First 3×3 convolutional layer, second batch normalization layer, first ReLU activation function The system consists of a second 3×3 convolutional layer, a third batch normalization layer, a second ReLU activation function, a 1×1 convolutional layer, and a third ReLU activation function. The number of channels in each convolutional layer is the same as or matches the number of channels in the left branch, so as to achieve efficient extraction of local spatial texture and detail information.

[0150]

[0151] in, This indicates the output of the left branch. This indicates the characteristics of branch inputs in a dual-branch parallel processing module.

[0152] The right branch of the dual-branch parallel processing module employs a direction-selective state-space model. This model alternates between horizontal and vertical scanning in different visual state-space layers: feature sequences are scanned and modeled along the horizontal direction of the image in odd-numbered visual state-space layers, and along the vertical direction of the image in even-numbered visual state-space layers. Long-range dependencies between sequences are modeled using state-space equations, thereby achieving directional adaptive representation of image structural information. For input features... have:

[0153]

[0154]

[0155]

[0156] in, This indicates the output of the left branch. This indicates the characteristics of the branch input in the dual-branch parallel processing module. Presentation layer normalization operation, Indicates a linear layer. This represents a 3×3 depthwise separable convolution. It is an activation function. Indicates priors with local spatial correlation Unidirectional state space scanning modeling.

[0157] The directional selective state-space model incorporates local spatial correlation priors during computation, including: before calculating the state-space backbone, it performs a priori analysis on the input features. Apply 5×5 depth-separable convolution A local correlation enhancement branch is formed, which incorporates the local correlation information extracted through convolution. By fusing the output projection features in the state space model in a channel-by-channel manner, the parameterization process of the state transition matrix is ​​modulated, thereby enhancing the sensitivity to local texture information while maintaining the ability to model long-range dependencies.

[0158]

[0159] in, This represents the Sigmoid activation function.

[0160] S1.2.3 Feature fusion module: It splices the features output by the left and right branches in the channel dimension, and performs information exchange between different branches through channel shuffling to obtain fused features;

[0161] The fusion process of the feature fusion module includes: converting the left branch output features... With right branch output characteristics The splicing is performed along the channel dimension to obtain the splicing features. Perform channel shuffling operation on the splicing features. The number of groups in the channel shuffling corresponds to the number of dual branches, so that features from different branches are rearranged among the groups, thereby realizing cross-branch information interaction and fusion.

[0162]

[0163] S1.2.4 Inverse geometric transformation module: The fused features are nonlinearly mapped to the channel domain by the multilayer perceptron module and then an inverse geometric transformation corresponding to the geometric transformation enhancement module is applied to restore the features to the original spatial coordinate system;

[0164] S1.3. Perform convolutional feature aggregation on the features processed by multiple residual state space groups, and superimpose them with shallow features through residual connections to obtain deep reconstructed features.

[0165] S1.4 The reconstruction module uses a pixel rearrangement upsampling structure to upsample the deep reconstruction features and outputs a high-resolution reconstructed image;

[0166] S2. Based on the image training set, train and optimize the parameters in the image super-resolution reconstruction network;

[0167] S3. In the inference stage, the low-resolution image to be processed is input into the trained image super-resolution reconstruction network, and the corresponding high-resolution reconstruction result is obtained through the network mapping relationship.

[0168] Each visual state space layer, based on geometric transformation, bi-branch parallel processing, and feature fusion, further employs a dual residual learning mechanism, including: a first residual connection is set between the fused features and the input features processed by the geometric transformation enhancement module, through a learnable first scaling factor. The residual term is scaled and then added to the input features; the second residual connection is set between the input information and the multilayer perceptron module. Between these points, the multilayer perceptron module performs a nonlinear mapping of the channel domain, using a learnable second scaling factor. Scaling is performed and the residuals are added to the input features, thereby enabling multi-level residual learning across modules within each visual state space layer.

[0169]

[0170]

[0171] in, This represents a multi-layer perceptron.

[0172] All experiments in this application were performed on a unified hardware platform, using a server equipped with an NVIDIA RTX 2080Ti GPU. To comprehensively evaluate the reconstruction quality, we used two metrics: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM), both calculated on the luminance channel (Y channel) of the YCbCr color space. Experiments were conducted on five publicly available datasets at a ×4 magnification, and the results are shown in Table 1. Our method achieved optimal results in both PSNR and SSIM metrics.

[0173] Table 1

[0174]

[0175] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. An image super-resolution reconstruction method based on a state-space model enhanced by geometric transformation, characterized in that, include: Step 1: Construct a dataset containing high-resolution images and corresponding low-resolution images from different scenes as a training set; A network for image super-resolution reconstruction is constructed using network mapping relationships; Step 2: Train the image super-resolution reconstruction network using the training set to obtain the trained image super-resolution reconstruction network; Step 3: Input the low-resolution image to be processed into the trained image super-resolution reconstruction network to obtain the corresponding high-resolution image; The image super-resolution reconstruction network includes: a shallow feature extraction module, multiple cascaded residual state space groups, and a residual connection and reconstruction module; The shallow feature extraction module is used to extract shallow features from the input low-resolution image to obtain shallow features; The shallow features are sequentially processed through a series of residual state space groups for deep feature extraction. Within each residual state space group, the input features are processed sequentially through multiple visual state space layers and a convolutional operation layer. In each visual state space layer, the input features are sequentially subjected to geometric transformation enhancement, dual-branch parallel processing, feature fusion, nonlinear mapping and inverse geometric transformation of the multilayer perception mechanism to obtain the output features of that layer; The convolutional operation layer aggregates the features after they have been processed by multiple visual state space layers; The residual connection and reconstruction module performs residual connection between the features aggregated by convolutional features and the shallow features to obtain deep reconstruction features; then, the deep reconstruction features are rearranged and upsampled to output a high-resolution image.

2. The image super-resolution reconstruction method based on a state-space model enhanced by geometric transformation according to claim 1, characterized in that, The visual state space layer includes a geometric transformation enhancement module, a dual-branch parallel processing module, a feature fusion module, a multilayer perceptron module, and an inverse geometric transformation module; wherein... The geometric transformation enhancement module is used to apply geometric transformations to the input features in the spatial dimension to obtain the enhanced features. The geometric transformation method used by the geometric transformation enhancement module is selected from a preset set of geometric transformations; the set of geometric transformations includes identity transformation, horizontal flip, vertical flip, transpose, simultaneous horizontal and vertical flip, and 180-degree rotation geometric transformation. The geometric transformation enhancement modules in all visual state space layers sequentially and cyclically select geometric transformation methods from the geometric transformation set according to the layer index order. The dual-branch parallel processing module is connected to the geometric transformation enhancement module and is used to divide the geometrically transformed features into a left branch and a right branch along the channel dimension for parallel processing. The left branch uses a lightweight convolutional neural network for local feature extraction; the right branch uses a direction-selective state-space model to scan and model the feature sequence along a predetermined direction. The feature fusion module is connected to the dual-branch parallel processing module. It is used to concatenate the features output by the left branch and the features output by the right branch in the channel dimension, and perform channel shuffling operation on the concatenated features to perform inter-branch interaction and fusion of features output by different branches to obtain fused features. The fused features are residually connected with the input features to obtain a first residual connection feature, which is then output to the multilayer perceptron module. The multilayer perceptron module is used to perform nonlinear mapping on the fused features after residual connection in the channel domain, enhance the feature representation, and obtain the enhanced features; The enhanced feature is re-concatenated with the first residual connection feature to obtain the second residual connection feature, and the second residual connection feature is output to the inverse geometric transformation module. The inverse geometric transformation module is used to apply an inverse geometric transformation corresponding to the geometric transformation enhancement module to the second residual connection feature, restore the feature to the original spatial coordinate system, and obtain the final output feature of the visual state space layer.

3. The image super-resolution reconstruction method based on a state-space model enhanced by geometric transformation according to claim 1, characterized in that, The shallow feature extraction module is constructed using a channel expansion and shuffling strategy. Specifically, it performs channel expansion and shuffling on the input low-resolution image. The expanded image is obtained by copying the channel dimensions by a factor of 1. The extended image Perform within the channel dimension according to the preset number of groups. Channel shuffling of scaling factor to obtain shuffled features ; and then the characteristics after mixing and washing Apply 3×3 convolution operation To obtain initial shallow features ; in, This represents the input low-resolution image. This represents a 3×3 convolution operation.

4. The image super-resolution reconstruction method based on a state-space model enhanced by geometric transformation according to claim 1, characterized in that, The formula for extracting deep features from the shallow features by sequentially passing them through a series of residual state space groups is as follows: in, This indicates the number of sets of cascaded residual state-space groups. Indicates deep features, This indicates a cascade operation. Indicates the first A set of residual state spaces, .

5. The image super-resolution reconstruction method based on a state-space model enhanced by geometric transformation according to claim 1, characterized in that, In step three, the formula for obtaining the corresponding high-resolution image is: in, The high-resolution image output by the network is used to super-resolution reconstruct images of the trained images.

6. The image super-resolution reconstruction method based on a state-space model enhanced by geometric transformation according to claim 2, characterized in that, The left branch uses a lightweight convolutional neural network for local feature extraction as follows: The input features are sequentially passed through the first batch of normalization layers. First 3×3 convolutional layer, second batch normalization layer, first ReLU activation function The process involves a second 3×3 convolutional layer, a third batch normalization layer, a second ReLU activation function, a 1×1 convolutional layer, and a third ReLU activation function to extract local spatial texture and detail information. The number of channels in all convolutional layers is the same as or matches the number of channels in the left branch. The output of the left branch is: in, This indicates the output of the left branch. This indicates the characteristics of the branch input in the dual-branch parallel processing module. Indicates activation function, Indicates normalization operation, This represents a 1×1 convolution operation.

7. The image super-resolution reconstruction method based on a state-space model enhanced by geometric transformation according to claim 2, characterized in that, The method for modeling the right branch by scanning the feature sequence along a predetermined direction using a direction-selective state-space model is as follows: The orientation-selective state-space model scans and models the feature sequence along the horizontal direction of the image in odd-numbered visual state-space layers; The feature sequence is scanned and modeled along the vertical direction of the image in the even-numbered visual state space layer; The long-range dependencies between sequences are modeled using state-space equations, and the output of the model is obtained as follows: : in, This indicates the output of the left branch. This indicates the characteristics of the branch input in the dual-branch parallel processing module. Presentation layer normalization operation, Indicates a linear layer. This represents a 3×3 depthwise separable convolution. It is an activation function. Indicates priors with local spatial correlation Unidirectional state space scanning modeling.

8. The image super-resolution reconstruction method based on a state-space model enhanced by geometric transformation according to claim 7, characterized in that, Before the direction-selective state-space model scans and models the feature sequence along a predetermined direction, it also includes a process of calculating the prior correlation of local space, which specifically includes: A 5×5 depthwise separable convolution is applied to the features input to the orientation-selective state space model to form a local correlation enhancement branch. Local correlation information is extracted and then fused with the output projection features of the orientation-selective state space model in a channel-by-channel manner before being output back to the orientation-selective state space model. The local correlation information is as follows: in, This represents the Sigmoid activation function. This represents the features input to the direction-selective state-space model. This indicates the application of a 5×5 depth separable convolution operation. This represents the local correlation information extracted through convolution.

9. The image super-resolution reconstruction method based on a state-space model enhanced by geometric transformation according to claim 2, characterized in that, The process by which the feature fusion module obtains the fused features is as follows: Output features of the left branch With right branch output characteristics The splicing is performed along the channel dimension to obtain the splicing features. Perform channel shuffling operation on the splicing features. The number of groups in the channel shuffling corresponds to the number of dual branches, so that the features output by different branches are rearranged between each group, realizing the inter-branch interaction and fusion of the features output by different branches; in, This represents the fused features of the residual state space group output.

10. The image super-resolution reconstruction method based on a state-space model enhanced by geometric transformation according to claim 9, characterized in that, The enhanced features are: in, This is the first residual connectivity feature. Indicates the enhanced features, This represents a multi-layer perceptron. This represents the second scaling factor.

Images