Panoramic image super-resolution reconstruction method and system based on improved mamba

Abstract

The application provides a panoramic image super-resolution reconstruction method and system based on improved Mamba, and the method comprises the following steps: shallow layer features are extracted by using convolution operation; the shallow layer features are input into a deep layer feature extraction module, and deep layer features are extracted by using a feature processing unit integrating three collaborative mechanisms; and after the shallow layer features and the deep layer features are fused, upsampling reconstruction is performed; the application realizes spatial domain local geometric perception enhancement through a geometric distortion perception window attention mechanism; the geometric distortion modulation Mamba mechanism uses a geometric distortion degree graph to perform affine transformation on the features, dynamically modulates key parameters of a state space model, and realizes spatial domain global distortion adaptive sequence modeling; the block DCT domain scanning Mamba mechanism constructs frequency-based ordered association between feature blocks through a specific scanning path in the frequency domain, and realizes frequency domain structured sequence modeling; through the collaboration of the three mechanisms, the application effectively improves the feature representation capability and reconstruction quality of the model for panoramic images.

Description

Technical Field

[0001] This invention relates to the field of computer image processing technology, and in particular to a panoramic image super-resolution reconstruction method and system based on an improved Mamba. Background Technology

[0002] Panoramic image super-resolution technology is crucial for enhancing the visual immersion and information accuracy in applications such as virtual reality, augmented reality, and panoramic navigation. This technology aims to recover high-resolution details from low-resolution panoramic images, and its output quality directly impacts the downstream user experience. However, for ease of storage and transmission, the equidistant cylindrical projection (ERP) format commonly used for panoramic images introduces severe, non-uniform latitude-dependent geometric distortion when mapping spherical information to a plane, primarily manifested as stretching near the poles. This inherent distortion, strongly correlated with spatial location, violates the translation invariance assumption of standard image models, making it difficult to directly apply super-resolution models designed for ordinary images, resulting in poor reconstruction results. Therefore, enabling the model to perceive and adapt to this geometric structure is a prerequisite for achieving high-quality panoramic reconstruction.

[0003] To address these challenges, technological approaches have continuously evolved. Early deep learning-based methods, such as LAU-Net, attempted to solve the problem of uneven pixel density by segmenting images by latitude and adaptively adjusting the reconstruction scale, but these methods were prone to introducing inter-block artifacts. With the emergence of the Transformer architecture, its powerful long-range dependency modeling capabilities provided a new paradigm for solving geometric distortion problems. Representative works, such as OSRT, designed a window attention mechanism that is aware of geometric distortion. It cleverly uses the area stretching ratio of the ERP along the latitude direction as a priori for geometric distortion, dynamically generating attention key-value pairs and sampling offsets for deformable convolutions, enabling the model to adaptively adjust the receptive field range in different distortion regions. Another work, BPOSR, explored a dual-projection attention mechanism based on ERP and hexahedral projection (CMP). However, the global self-attention mechanism of the Transformer has quadratic computational complexity. Although window attention can significantly reduce computational overhead, its local window partitioning inevitably fragments the global context of the ERP panorama, limiting the model's coherent understanding of the overall panoramic geometry.

[0004] In recent years, state-space models, particularly the Mamba architecture, have provided a new paradigm for efficiently processing long-sequence data due to their linear computational complexity and ability to model long-range dependencies. In the vision domain, Mamba serializes two-dimensional images through a multi-directional scanning strategy to capture global context. Furthermore, various variants of the Mamba architecture have been extensively explored in different tasks. However, applying such Mamba architectures to panoramic image super-resolution faces the following challenges:

[0005] First, existing state-space models struggle to adapt to the inherent non-uniform geometric distortions of panoramic images. In panoramic super-resolution tasks, the latitude-dependent geometric distortions introduced by the ERP format require models to possess spatially adaptive feature modeling capabilities. However, the current parameterization and sequence modeling processes of visual Mamba and its variants primarily rely on image content, lacking explicit perception and response mechanisms for the degree of distortion corresponding to pixel locations. This makes it difficult for them to perform adaptive feature representation and reconstruction in different distortion regions.

[0006] Second, Mamba's sequence modeling mechanism is insufficient in preserving local details. Mamba serializes 2D images through a scanning strategy, which, while efficiently capturing global dependencies, inherently weakens its ability to coherently model local spatial structures and high-frequency details. Since the core of super-resolution tasks lies in detail recovery, how to compensate for Mamba's weakness in local fine-grained perception becomes a crucial issue.

[0007] Third, current research on combining Mamba with frequency domain transformation is insufficient. Related studies are still in their early stages and have not yet fully explored the synergistic potential between frequency domain features. In panoramic image super-resolution tasks, how to construct a sequence mechanism that can effectively model the complex relationships between frequency domain features to improve reconstruction quality remains an unsolved problem. Summary of the Invention

[0008] In view of the above, the main objective of this invention is to propose a panoramic image super-resolution reconstruction method and system based on improved Mamba, so as to solve the above-mentioned technical problems.

[0009] This invention proposes a panoramic image super-resolution reconstruction method based on an improved Mamba, the method comprising the following steps:

[0010] Step 1: Construct a reconstruction model based on the distortion information generation module, shallow feature extraction module, deep feature extraction module, feature fusion module, and upsampling reconstruction module;

[0011] Step 2: Input the low-resolution panoramic image into the distortion information generation module. Based on the mapping relationship between the equidistant cylindrical projection format of the low-resolution panoramic image and the spherical coordinates, calculate the area stretching ratio before and after projection to determine the degree of geometric distortion corresponding to the spatial position of each pixel.

[0012] Step 3: Input the low-resolution panoramic image into the shallow feature extraction module and use convolution operation to extract shallow features;

[0013] Step 4: Input the shallow features into the deep feature extraction module, and use the cascaded feature processing units to extract deep features based on the geometric distortion map to obtain deep features; wherein, the feature processing unit combines the geometric distortion map and uses the geometric distortion perception window attention mechanism and the geometric distortion modulation Mamba mechanism to realize spatial local geometric perception enhancement and spatial global distortion modulation sequence modeling, respectively; on the other hand, it realizes frequency domain structured sequence modeling through the block DCT domain scanning Mamba mechanism that does not depend on the geometric distortion map.

[0014] Step 5: Use the feature fusion module to fuse the shallow features and deep features to obtain the fused features;

[0015] Step 6: Input the fused features into the upsampling reconstruction module for upsampling reconstruction to obtain the super-resolution reconstructed panoramic image; construct a loss function based on the super-resolution reconstructed panoramic image and the real panoramic image, and use the loss function to optimize the reconstruction model to obtain the optimized reconstruction model; input the low-resolution panoramic image into the optimized reconstruction model to obtain the final super-resolution reconstructed panoramic image.

[0016] This invention also proposes a panoramic image super-resolution reconstruction system based on an improved Mamba, the system comprising:

[0017] Build modules are used for:

[0018] The reconstruction model is composed of a distortion information generation module, a shallow feature extraction module, a deep feature extraction module, a feature fusion module, and an upsampling reconstruction module.

[0019] The information generation module is used for:

[0020] The low-resolution panoramic image is input into the distortion information generation module. Based on the mapping relationship between the equidistant cylindrical projection format of the low-resolution panoramic image and the spherical coordinates, the area stretching ratio before and after projection is calculated to determine the degree of geometric distortion corresponding to the spatial position of each pixel.

[0021] The first extraction module is used for:

[0022] The low-resolution panoramic image is input into the shallow feature extraction module, and shallow features are extracted using convolution operations.

[0023] The second extraction module is used for:

[0024] Shallow features are input into the deep feature extraction module, and deep features are extracted based on the geometric distortion map using cascaded feature processing units. The feature processing units combine the geometric distortion map with the geometric distortion perception window attention mechanism and the geometric distortion modulation Mamba mechanism to achieve spatial local geometric perception enhancement and spatial global distortion modulation sequence modeling, respectively. On the other hand, they achieve frequency domain structured sequence modeling through the block DCT domain scanning Mamba mechanism, which is independent of the geometric distortion map.

[0025] The fusion module is used for:

[0026] The feature fusion module is used to fuse shallow features and deep features to obtain fused features;

[0027] The rebuild module is used for:

[0028] The fused features are input into the upsampling reconstruction module for upsampling reconstruction to obtain the super-resolution reconstructed panoramic image. A loss function is constructed based on the super-resolution reconstructed panoramic image and the real panoramic image. The reconstruction model is optimized using the loss function to obtain the optimized reconstruction model. The low-resolution panoramic image is input into the optimized reconstruction model to obtain the final super-resolution reconstructed panoramic image.

[0029] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0030] 1. This invention proposes geometric distortion modulation Mamba blocks and block-based DCT domain scanning Mamba blocks, and integrates them with existing geometric distortion sensing window attention mechanisms. This architecture achieves effective synergy among spatial local geometric adaptation, spatial global distortion modulation, and frequency domain structured feature modeling.

[0031] 2. This invention achieves explicit adaptive modeling of latitude-related distortion in panoramic images using the Mamba model in the spatial domain. By modulating the Mamba block with geometric distortion, the prior latitude-related distortion is transformed into a dynamic modulation signal for key parameters of the state-space model. This enables the model to adaptively adjust the feature evolution process according to the degree of distortion in different regions, thereby effectively improving its feature representation and reconstruction accuracy in non-uniform distortion regions.

[0032] 3. This invention scans the Mamba block in a segmented DCT domain, designing specific intra-block and cross-block scanning paths, and actively constructs ordered associations between different feature blocks in the frequency domain. This enables the model to not only follow the energy pattern from low to high frequency during sequence inference, but also to perform sequence modeling based on this pre-constructed structured association, thereby more systematically and efficiently integrating global structure and local details, and improving the reconstruction quality.

[0033] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the steps of the panoramic image super-resolution reconstruction method based on the improved Mamba proposed in this invention.

[0035] Figure 2 This is a general framework diagram of the panoramic image super-resolution reconstruction method based on the improved Mamba proposed in this invention.

[0036] Figure 3 This is a schematic diagram of the geometric distortion modulation Mamba unit of the panoramic image super-resolution reconstruction method based on the improved Mamba proposed in this invention.

[0037] Figure 4 This is a schematic diagram of the structure of a block-based DCT domain scanning Mamba cell based on the improved Mamba panoramic image super-resolution reconstruction method proposed in this invention.

[0038] Figure 5 This is a schematic diagram illustrating the scanning path generation of the panoramic image super-resolution reconstruction method based on the improved Mamba proposed in this invention.

[0039] Figure 6 This is a schematic diagram of the panoramic image super-resolution reconstruction system based on the improved Mamba proposed in this invention. Detailed Implementation

[0040] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0041] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to provide some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0042] Please see Figure 1 This invention proposes a panoramic image super-resolution reconstruction method based on an improved Mamba, which includes the following steps:

[0043] Step 1: The reconstruction model is constructed based on the distortion information generation module, shallow feature extraction module, deep feature extraction module, feature fusion module and upsampling reconstruction module.

[0044] Step 2: Input the low-resolution panoramic image into the distortion information generation module. Based on the mapping relationship between the equidistant cylindrical projection format of the low-resolution panoramic image and the spherical coordinates, calculate the area stretching ratio before and after projection to determine the degree of geometric distortion corresponding to the spatial position of each pixel.

[0045] In step 2, based on the mapping relationship between the equidistant cylindrical projection format of the low-resolution panoramic image and spherical coordinates, the area stretching ratio before and after projection is calculated to determine the degree of geometric distortion corresponding to the spatial position of each pixel. The corresponding relationship in this process is as follows:

[0046] ;

[0047] in, A graph representing the degree of geometric distortion. This represents the height index of a low-resolution panoramic image. Represents the cosine function. Indicates the height of a low-resolution panoramic image. It represents pi (π).

[0048] Step 3: Input the low-resolution panoramic image into the shallow feature extraction module and use convolution operation to extract shallow features.

[0049] Specifically, in step 3, a 3×3 convolutional layer is used to extract and upscale the input image to obtain shallow features. The number of output channels for this operation can be set as needed, such as 156 dimensions.

[0050] Step 4: Input the shallow features into the deep feature extraction module, and use the cascaded feature processing units to extract deep features based on the geometric distortion map to obtain deep features. The feature processing unit combines the geometric distortion map with the geometric distortion perception window attention mechanism and the geometric distortion modulation Mamba mechanism to realize spatial local geometric perception enhancement and spatial global distortion modulation sequence modeling, respectively. On the other hand, it realizes frequency domain structured sequence modeling through the block DCT domain scanning Mamba mechanism that does not depend on the geometric distortion map.

[0051] Specifically, in this embodiment, the deep feature extraction module consists of nine feature processing units with identical structures connected in series. The specific structure of each feature processing unit is as follows: a first geometric distortion sensing window attention block, a geometric distortion modulation Mamba block, a second geometric distortion sensing window attention block, and a block DCT domain scanning Mamba block are connected in sequence. The output of the unit is then processed by a convolutional layer and added to the input features of the unit through residual connections to obtain the output features of the unit. The first geometric distortion sensing window attention block and the second geometric distortion sensing window attention block have the same structure.

[0052] Please see Figure 2 In step 4, the shallow features are input into the deep feature extraction module, and deep features are extracted based on the geometric distortion map using cascaded feature processing units to obtain deep features. Specifically, this includes the following steps:

[0053] S401. The shallow features are used as the input features of the first feature processing unit in the deep feature extraction module and input into the first geometric distortion perception window attention block. Local geometric perception and enhancement processing is performed on the input features according to the geometric distortion degree map to obtain the output features of the first geometric distortion perception window attention block. The feature processing unit further includes: a geometric distortion modulation Mamba block, a second geometric distortion perception window attention block, a block-based DCT domain scanning Mamba block, and a convolutional layer. The first geometric distortion perception window attention block, the geometric distortion modulation Mamba block, the second geometric distortion perception window attention block, the block-based DCT domain scanning Mamba block, and the convolutional layer are connected sequentially. The first geometric distortion perception window attention block and the second geometric distortion perception window attention block have the same structure. Each feature processing unit has the same structure.

[0054] S402. Input the output features of the attention block of the first geometric distortion perception window and the geometric distortion degree map into the geometric distortion modulation Mamba block. Perform an affine transformation on the input features based on the geometric distortion degree map and modulate the parameters of the state space model to complete the spatial global adaptive sequence modeling and obtain the output features of the geometric distortion modulation Mamba block.

[0055] S403. Input the output features of the geometric distortion modulation Mamba block and the geometric distortion degree map into the attention block of the second geometric distortion perception window, and then perform local geometric perception and enhancement processing on the input features according to the geometric distortion degree map to obtain the output features of the attention block of the second geometric distortion perception window.

[0056] S404. Input the output features of the attention block of the second geometric distortion sensing window into the block-based DCT domain scanning Mamba block, perform block-based discrete cosine transform, and perform scanning and sequence modeling in the frequency domain according to a preset structured order to obtain the output features of the block-based DCT domain scanning Mamba block.

[0057] S405. The output features of the block-based DCT domain scan Mamba block are input into the convolutional layer for processing. The output of the convolutional layer is added to the input features of the current feature processing unit through residual connections to obtain the output features of the current feature processing unit.

[0058] S406. Input the output features of the current feature processing unit into the next feature processing unit in an iterative manner, and repeat steps S401 to S405 to obtain the output features of the next feature processing unit.

[0059] S407. Take the output feature of the next feature processing unit as input and input it into the last feature processing unit. Repeat steps S401 to S405 to obtain the output feature of the last feature processing unit and take the output feature of the last feature processing unit as the deep feature.

[0060] The shallow features are used as the input features of the first feature processing unit in the deep feature extraction module and input into the attention block of the first geometric distortion perception window. The input features are then subjected to local geometric perception and enhancement processing based on the geometric distortion degree map to obtain the output features of the attention block of the first geometric distortion perception window. The specific steps include the following:

[0061] Shallow features are used as input features and input into the first geometric distortion perception window attention block of the first feature processing unit in the deep feature extraction module. Normalization processing is then performed to obtain the features after the first layer normalization processing. The first geometric distortion perception window attention block includes a geometric distortion perception window attention unit and a first feedforward network layer.

[0062] The features after the first layer of normalization are input into the geometric distortion perception window attention unit for processing, and the output features of the geometric distortion perception window attention unit are obtained.

[0063] The output features of the attention unit of the geometric distortion sensing window are connected with the input features of the attention block of the first geometric distortion sensing window to obtain the features after the first residual connection.

[0064] The features after the first residual connection are subjected to a second layer of normalization to obtain the features after the second layer of normalization.

[0065] The features after the second layer of normalization are input into the first feedforward network layer for feedforward processing to obtain the output features of the first feedforward network layer.

[0066] The output features of the first feedforward network layer are combined with the features after the first residual connection to form a second residual connection, resulting in the output of the attention block of the first geometric distortion perception window.

[0067] Specifically, the features after the first layer of normalization are input into the geometric distortion perception window attention unit for processing, resulting in the output features of the geometric distortion perception window attention unit, including:

[0068] The features after the first layer of normalization are input into the attention unit of the geometric distortion perception window, which is divided into non-overlapping windows of a preset size. The features in each non-overlapping window are linearly projected to obtain the query vector. The size of the non-overlapping window is 8×8.

[0069] Window conditions representing relative positions within non-overlapping windows are obtained and concatenated with a geometric distortion degree map along the channel dimension to obtain concatenated features. The concatenated features are then processed using a lightweight network in the attention block of the first geometric distortion-aware window to generate adaptive position offsets in the height and width directions of each position of the input features of the attention unit of the geometric distortion-aware window. The lightweight network consists of three linear layers.

[0070] Based on the adaptive position offset of each position of the input feature of the geometric distortion sensing window attention unit in the height and width directions, bilinear sampling is performed on the input feature of the geometric distortion sensing window attention unit to obtain the feature after position offset.

[0071] The features after positional offset are divided into non-overlapping windows of a preset size, and the features within each window are linearly projected to generate key vectors and value vectors; the size of the non-overlapping window is 8×8.

[0072] Window self-attention is calculated using query vectors, key vectors, and value vectors to obtain the output features of the attention unit of the geometric distortion-aware window. The relationships between these parameters in the window self-attention calculation process are as follows:

[0073] ;

[0074] in, This indicates the calculation of window self-attention. Represents the query vector. Represents the key vector. Represents a value vector. Represents the normalized exponential function, Represents the transpose of the key vector , Represents the query vector and key vector Dimensions This indicates the relative position offset term.

[0075] The output features of the attention block of the first geometric distortion perception window and the geometric distortion degree map are input into the geometric distortion modulation Mamba block. Based on the geometric distortion degree map, an affine transformation is performed on the input features, and the parameters of the state space model are modulated to complete the spatial global adaptive sequence modeling, thereby obtaining the output features of the geometric distortion modulation Mamba block. The specific steps include the following:

[0076] The output features of the geometric distortion perception window attention block and the geometric distortion degree map are input into the geometric distortion modulation Mamba block, and layer normalization is performed to obtain the features after the third layer normalization. The geometric distortion modulation Mamba block includes: a geometric distortion modulation Mamba unit and a second feedforward network layer; the geometric distortion modulation Mamba unit includes an affine transformation module, a first branch, and a second branch; both the first and second branches are composed of linear projection, a 3×3 depth convolutional layer, and a SiLU activation function layer connected sequentially; the affine transformation module consists of an affine transformation coefficient generation network and affine transformation operations; the affine transformation coefficient generation network consists of the innermost convolutional layer, the first leaky rectified linear unit activation function, the middle convolutional layer, the second leaky rectified linear unit activation function, and the outermost convolutional layer;

[0077] The features after the third layer normalization process are input into the geometric distortion modulation Mamba unit for processing to obtain the output features of the geometric distortion modulation Mamba unit.

[0078] The output features of the geometrically distorted modulation Mamba unit are joined with the input features of the geometrically distorted modulation Mamba block by a third residual connection to obtain the features after the third residual connection.

[0079] The features after the third residual connection are processed by layer normalization and the second feedforward network layer to obtain the output features of the second feedforward network layer.

[0080] The output features of the second feedforward network layer are combined with the features after the third residual connection to form a fourth residual connection, resulting in the output features of the geometric distortion modulation Mamba block.

[0081] Please see Figure 3 The normalized features are input into a geometric distortion modulation (MDM) Mamba unit for processing to obtain the output features of the MBM unit, including:

[0082] The geometric distortion map is input into the affine transformation module of the geometric distortion modulation Mamba unit. It sequentially passes through the innermost convolutional layer, the first leaky rectified linear unit (LER) activation function, the middle convolutional layer, the second LRR activation function, and the outermost convolutional layer, resulting in the outermost convolutional layer feature. Channel segmentation is then performed on this outermost convolutional layer feature to obtain the channel-segmented output feature. This channel-segmented output feature is then split into two tensors, which are used as two affine transformation coefficients. and The layer-normalized features are input into the geometric distortion modulation Mamba unit, based on two affine transform coefficients. and An affine transformation is performed on the input features of the geometrically distorted modulation Mamba unit. This is done by multiplying one of the two affine transformation coefficients element-wise with the input features of the geometrically distorted modulation Mamba unit and adding the other of the two affine transformation coefficients to obtain the affine transformed features.

[0083] The input features of the geometric distortion modulated Mamba unit are input into the first branch, and then processed sequentially through linear projection, a 3×3 depth convolutional layer, and SiLU activation function to obtain the input branch features.

[0084] The features after affine transformation are input into the second branch, and then processed sequentially through linear projection, a 3×3 depth convolutional layer, and SiLU activation function to obtain the modulated branch features.

[0085] S4021. Flatten the input branch features into a one-dimensional sequence, which serves as the input sequence for the state-space model. Based on the input sequence, generate the output matrix of the state-space model through a linear transformation. Simultaneously, flatten the modulation branch features into a one-dimensional sequence, and dynamically generate the input matrix and discretization step size of the state-space model through a linear transformation based on the one-dimensional sequence of the modulation branch features. Based on the input sequence, output matrix, input matrix, discretization step size, and two preset learnable parameters (state matrix and direct transition matrix) of the state-space model, perform state-space modeling through the latent states.

[0086] The input branch features and modulation branch features are flattened into one-dimensional sequences along four directions: horizontal forward, horizontal reverse, vertical forward, and vertical reverse, using a four-directional scanning mechanism. For each scanning direction, step S4021 is repeated to obtain the output features in the four directions. The output features in the four directions are rearranged into two-dimensional features and then fused element by element to obtain the spatial state modeling features.

[0087] The input features of the geometrically distorted modulation Mamba unit are sequentially processed by linear projection and SiLU activation function to generate a gated signal; the spatial state modeling features are sequentially processed by layer normalization, element-wise multiplication with the gated signal, and linear projection to obtain the output features of the geometrically distorted modulation Mamba unit.

[0088] The geometric distortion map is input into the affine transformation module of the geometric distortion modulation Mamba unit. It sequentially passes through the innermost convolutional layer, the first leaky rectified linear unit (LER) activation function, the middle convolutional layer, the second LRR activation function, and the outermost convolutional layer, resulting in the outermost convolutional layer feature. Channel segmentation is then performed on this outermost convolutional layer feature to obtain the channel-segmented output feature. This channel-segmented output feature is then split into two tensors, which are used as two affine transformation coefficients. and The layer-normalized features are input into the geometric distortion modulation Mamba unit, based on two affine transform coefficients. and Affine transformation is performed on the input features of the geometrically distorted modulation (MDM) Mamba unit. This is achieved by element-wise multiplying one of the two affine transformation coefficients with the input features of the MBM unit and then adding the result to the other of the two affine transformation coefficients. The relationships involved in this process are as follows:

[0089] ;

[0090] in, This represents the affine transformation coefficient generation network; This indicates a channel splitting operation. This indicates a 1×1 convolution process. This represents the activation function of the leaky rectifier linear unit. Represents the characteristics after affine transformation. This indicates element-wise multiplication. This indicates element-wise addition. This represents the input characteristics of the geometric distortion modulation Mamba unit;

[0091] In the process of inputting the geometric distortion modulated Mamba unit's input features into the first branch, and then sequentially processing them through linear projection, a 3×3 depthwise convolutional layer, and SiLU activation function to obtain the input branch features, the corresponding relationships in the process are as follows:

[0092] ;

[0093] in, Represents the input branch features, This represents the SiLU activation function. This indicates a 3×3 depthwise convolution process. This indicates linear projection processing;

[0094] In the process of inputting the affine transformation features into the second branch, and then sequentially processing them through linear projection, a 3×3 depth convolutional layer, and SiLU activation function to obtain the modulated branch features, the corresponding relationships in the process are as follows:

[0095] ;

[0096] in, Indicates modulation branch characteristics;

[0097] In the steps of flattening the input branch features into a one-dimensional sequence as the input sequence of the state-space model, and generating the output matrix of the state-space model through a linear transformation based on the input sequence; simultaneously flattening the modulation branch features into a one-dimensional sequence, and dynamically generating the input matrix and discretization step size of the state-space model through a linear transformation based on the one-dimensional sequence of the modulation branch features; and performing state-space modeling through latent states based on the input sequence, output matrix, input matrix, discretization step size, and two preset learnable parameters (state matrix and direct transition matrix), the corresponding relationships in the process are as follows:

[0098] ;

[0099] in, Representing the potential state Regarding time The derivative of , i.e., the rate of change of the potential state; This represents the first preset learnable parameter, namely the state matrix, which describes how the system state evolves over time. The HiPPO matrix is ​​used for initialization to enhance the model's ability to remember long-term historical information. Indicates a potential state. The input matrix describes how the input signal affects the system state and is dynamically generated from the modulation branch characteristics. Represents a one-dimensional input sequence. Output feature sequence, The output matrix represents how the system state is transformed into the output signal, and is dynamically generated from the input branch features. This represents the second preset learnable parameter, namely the direct transition matrix, which is initially set to a matrix of all 1s.

[0100] It should be noted that, in order to process the discretely sampled input sequence, the zero-order hold method is adopted. The two continuous parameters are transformed into two discrete parameters using a dynamically generated discretization step size based on the modulation branch features. This process is defined by the following relationship:

[0101] ;

[0102] in, This represents the discretized state matrix, i.e., the first discrete parameter; This represents the natural exponential function. This represents the discretization step size dynamically generated by the modulation branch features. This represents the discretized input matrix, i.e., the second discrete parameter; Represents the identity matrix.

[0103] The process of sequential modeling of the state-space model of a discrete system is defined by the following recursive relation:

[0104] ;

[0105] in, Represents the potential state of a discrete system. The value corresponding to each element Indicates the element index of the sequence. The first element of the input sequence of the discrete system One element, The output feature after the state-space model is modeled as a sequence is represented by the first... The values ​​corresponding to each element;

[0106] In the steps of processing the input features of the geometrically distorted modulation Mamba unit sequentially through linear projection and SiLU activation function to generate a gated signal, and processing the spatial state modeling features sequentially through layer normalization, element-wise multiplication with the gated signal, and linear projection to obtain the output features of the geometrically distorted modulation Mamba unit, the corresponding relationships in the process are as follows:

[0107] ;

[0108] in, Indicates a gating signal. This represents the output characteristics of the geometric distortion modulation Mamba unit. Presentation layer normalization processing.

[0109] The output features of the attention block of the second geometric distortion sensing window are input into the block-based DCT domain scanning Mamba block, and block-based discrete cosine transform is performed. Scanning and sequence modeling are then performed in the frequency domain according to a preset structured order to obtain the output features of the block-based DCT domain scanning Mamba block. The specific steps include the following:

[0110] The output features of the second geometric distortion modulation Mamba unit are input into the block-based DCT domain scanning Mamba block for layer normalization to obtain the fourth layer normalized features. The block-based DCT domain scanning Mamba block includes a block-based DCT domain scanning Mamba unit and a third feedforward network layer. Each block-based DCT domain scanning Mamba unit includes a spatial domain feature mapping network and a dual-mode scanning frequency domain modeling submodule. The spatial domain feature mapping network is composed of linear projection, a 3×3 depth convolutional layer, and a SiLU activation function connected sequentially. The dual-mode scanning frequency domain modeling submodule includes an intra-block priority continuous scanning mode and a same-frequency cross-block priority continuous scanning mode.

[0111] The normalized features of the fourth layer are input into the block DCT domain scan Mamba cell for processing to obtain the output features of the block DCT domain scan Mamba cell.

[0112] The output features of the block-based DCT domain scan Mamba cell are combined with the input features of the block-based DCT domain scan Mamba block to form a fifth residual connection, resulting in the features after the fifth residual connection.

[0113] The features after the fifth residual connection are processed by layer normalization and then by the third feedforward network layer to obtain the output features of the third feedforward network layer.

[0114] The output features of the third feedforward network layer are combined with the features after the fifth residual connection to form a sixth residual connection, so as to obtain the output features of the block-based DCT domain scanning Mamba block.

[0115] Please see Figure 4 The fourth-layer normalized features are input into the block-based DCT domain scanning Mamba cells for processing to obtain the output features of the block-based DCT domain scanning Mamba cells, including:

[0116] The input features of the block-based DCT domain scanning Mamba unit are processed by linear projection, 3×3 depth convolutional layer and SiLU activation function of the spatial domain feature mapping network to obtain spatial domain mapping features.

[0117] The spatial domain mapping features are divided into non-overlapping sub-blocks of a preset size, and a two-dimensional discrete cosine transform operation is performed on each non-overlapping sub-block to obtain two-dimensional frequency domain features; wherein, the size of the non-overlapping sub-block is 8×8.

[0118] The two-dimensional frequency domain feature is divided into two frequency domain sub-features along the channel dimension, each with half the number of channels of the original two-dimensional frequency domain feature. These are denoted as the first frequency domain sub-feature and the second frequency domain sub-feature, respectively.

[0119] Based on the intra-block priority continuous scanning mode, four scanning paths are generated according to four predefined inter-block traversal orders for the first frequency domain sub-feature. According to the order defined for each path, the first frequency domain sub-feature is flattened into four one-dimensional feature sequences. Based on the four one-dimensional feature sequences of the intra-block priority continuous scanning mode, the state space model parameters of the corresponding scanning path of the intra-block priority continuous scanning mode are generated through linear transformation. The four predefined inter-block traversal orders are: row-by-row traversal from top left to bottom right, column-by-column traversal from top left to bottom right, row-by-row traversal from bottom right to top left, and column-by-column traversal from bottom right to top left.

[0120] The rules for the block-priority continuous scanning mode are as follows: S4041, scan all frequency domain coefficients within a sub-block in a two-dimensional zigzag order from low frequency to high frequency; the total number of frequency domain coefficients is 64.

[0121] After completing the intra-block scan of the current sub-block, jump to the next sub-block according to the current inter-block traversal order, and repeat the intra-block scan process of step S4041 until all sub-blocks have been scanned.

[0122] For the second frequency domain sub-feature, based on the same-frequency cross-block priority continuous scanning mode, four scanning paths are generated according to the four predefined traversal orders between sub-blocks. According to the order defined for each path, the second frequency domain sub-feature is flattened into four one-dimensional feature sequences. Based on the four one-dimensional feature sequences of the same-frequency cross-block priority continuous scanning mode, the state space model parameters of the corresponding scanning path of the same-frequency cross-block priority continuous scanning mode are generated through linear transformation.

[0123] The rule for the same-frequency cross-block priority continuous scanning mode is as follows: S4042, select a frequency component in sequence according to the two-dimensional zigzag order from low frequency to high frequency, and scan the coefficients corresponding to the current frequency component in all sub-blocks in sequence according to the current sub-block traversal order.

[0124] After completing the scanning of the current frequency component in all sub-blocks, move to the next frequency component and repeat the scanning process of step S4042 until all frequency components have been scanned; the total number of frequency components is 64.

[0125] The four one-dimensional feature sequences generated according to the intra-block priority continuous scanning mode, along with the state space model parameters of the corresponding scanning path of the intra-block priority continuous scanning mode, are input into the state space model for parallel modeling to obtain the four output sequences of the four scanning paths corresponding to the intra-block priority continuous scanning mode. Through a rearrangement operation, the four output sequences of the four scanning paths corresponding to the intra-block priority continuous scanning mode are mapped back to the original two-dimensional frequency domain sub-feature layout according to their respective scanning paths, and then fused by element-wise addition to obtain the first fused feature.

[0126] The four one-dimensional feature sequences of the same-frequency cross-block priority continuous scanning mode, along with the state space model parameters of the corresponding scanning path of the same-frequency cross-block priority continuous scanning mode, are input into the state space model for parallel modeling to obtain the four output sequences of the four scanning paths corresponding to the same-frequency cross-block priority continuous scanning mode. Through a rearrangement operation, the four output sequences of the four scanning paths corresponding to the same-frequency cross-block priority continuous scanning mode are mapped back to the original two-dimensional frequency domain sub-feature layout according to their respective scanning paths, and then fused by element-wise addition to obtain the second fused feature.

[0127] The first fusion feature and the second fusion feature are concatenated along the channel dimension to obtain the combined feature, which is used as the frequency domain feature after sequence modeling.

[0128] After modeling the sequence, the frequency domain features are sequentially subjected to layer normalization and block-based inverse two-dimensional discrete cosine transform operations to be converted back to the spatial domain, thus obtaining the spatial domain reconstructed features.

[0129] The input features of the block-based DCT domain scanning Mamba cell are sequentially processed by linear projection and SiLU activation function to generate a gated signal. The spatial reconstruction features are then multiplied element-wise with the gated signal, followed by linear projection to obtain the output features of the block-based DCT domain scanning Mamba cell.

[0130] The input features of the Mamba unit are scanned in the block DCT domain and processed through linear projection, 3×3 depth convolutional layers, and SiLU activation function of the spatial domain feature mapping network to obtain the spatial domain mapped features. The corresponding relationship in the process is as follows:

[0131] ;

[0132] in, Represents spatial mapping characteristics, This represents the input features of the Mamba cell in a block-based DCT domain scan;

[0133] In the steps of dividing the spatial domain mapping features into non-overlapping sub-blocks of a preset size and performing a two-dimensional discrete cosine transform operation on each non-overlapping sub-block to obtain two-dimensional frequency domain features, the corresponding process relationships are as follows:

[0134] ;

[0135] in, Representing two-dimensional frequency domain characteristics, This represents the two-dimensional discrete cosine transform operation. Indicates block operation;

[0136] In the step of dividing the two-dimensional frequency domain feature into two frequency domain sub-features along the channel dimension, each with half the number of channels of the original two-dimensional frequency domain feature, denoted as the first frequency domain sub-feature and the second frequency domain sub-feature respectively, the corresponding relationship in the process is as follows:

[0137] ;

[0138] in, Represents the first frequency domain sub-feature. Indicates the second frequency domain sub-feature;

[0139] In the process of inputting four one-dimensional feature sequences generated according to the intra-block priority continuous scanning mode, along with the state space model parameters of the corresponding scanning paths of the intra-block priority continuous scanning mode, into the state space model for parallel modeling, four output sequences corresponding to the four scanning paths of the intra-block priority continuous scanning mode are obtained. Through a rearrangement operation, the four output sequences corresponding to the four scanning paths of the intra-block priority continuous scanning mode are mapped back to the original two-dimensional frequency domain sub-feature layout according to their respective scanning paths, and then fused by element-wise addition to obtain the first fused feature, the corresponding relationship in the process is as follows:

[0140] ;

[0141] in, Indicates the first sequential scan mode within the block. The output sequence of the state-space model corresponding to each scan path. Representing a state-space model, Indicates the first sequential scan mode within the block. One-dimensional feature sequences corresponding to each scan path. , , These represent the block-priority continuous scan modes, respectively. One-dimensional feature sequence corresponding to each scanning path Discretization step size, input matrix, and output matrix of the state-space model generated by linear transformation. , These represent the block-priority continuous scan modes, respectively. Two learnable parameters of the state-space model corresponding to each scan path. Indicates the first fusion feature, Indicates a rearrangement operation;

[0142] It should be noted that there are two learnable parameters. , These correspond to the state matrix and direct transition matrix of the state-space model, respectively.

[0143] In the step of parallel modeling, four one-dimensional feature sequences based on the same-frequency cross-block priority continuous scanning mode and the state space model parameters of the corresponding scanning paths of the same-frequency cross-block priority continuous scanning mode are input into the state space model to obtain four output sequences corresponding to the four scanning paths of the same-frequency cross-block priority continuous scanning mode. Then, through a rearrangement operation, the four output sequences corresponding to the four scanning paths of the same-frequency cross-block priority continuous scanning mode are mapped back to the original two-dimensional frequency domain sub-feature layout according to their respective scanning paths. Finally, they are fused by element-wise addition to obtain the second fused feature. The corresponding relationship in this process is as follows:

[0144] ;

[0145] in, Indicates the same frequency cross-block priority continuous scan mode. The output sequence of the state-space model corresponding to each scan path. Indicates the same frequency cross-block priority continuous scan mode. One-dimensional feature sequences corresponding to each scan path. , , These represent the first and second consecutive scan modes with the same frequency and cross-block priority. One-dimensional feature sequence corresponding to each scanning path Discretization step size, input matrix, and output matrix of the state-space model generated by linear transformation. , These represent the first and second consecutive scan modes with the same frequency and cross-block priority. Two learnable parameters of the state-space model corresponding to each scan path. Indicates the second fusion feature;

[0146] In the step of concatenating the first fused feature and the second fused feature along the channel dimension to obtain the combined feature, the corresponding relationship in the process is as follows:

[0147] ;

[0148] in, Represents combined features, Indicates a splicing operation;

[0149] In the steps of processing the input features of the block-based DCT domain scanning Mamba cell by sequentially performing linear projection and SiLU activation function processing to generate a gated signal, and then performing element-wise multiplication of the spatial domain reconstructed features with the gated signal followed by linear projection processing to obtain the output features of the block-based DCT domain scanning Mamba cell, the corresponding relationships in the process are as follows:

[0150] ;

[0151] in, Indicates the input features of block DCT domain scanning Mamba cells The generated gating signal, This represents the output characteristics of the Mamba cell in a block-based DCT domain scan. This represents the inverse two-dimensional discrete cosine transform operation.

[0152] It should be noted that, in Figure 3 middle, Indicates the flattening operation. This represents a linear transformation operation. This represents the SiLU activation function. This represents a one-dimensional feature sequence flattened from the modulation branch features; in Figure 4 middle, This indicates a channel splitting operation. This indicates a rearrangement operation. This indicates a channel splicing operation.

[0153] For further details, please refer to Figure 5 For ease of understanding, a schematic diagram of an embodiment of scan path generation is shown. In this embodiment, the feature map size is 8×8, the sub-block size is set to 4×4, and the number of sub-blocks is 2×2=4 (labeled as sub-blocks A, B, C, and D respectively). Each sub-block contains 16 frequency domain coefficients after DCT transformation (taking sub-block A as an example, labeled A1 to A16, where A1 is the DC coefficient, and A2 to A16 are the AC coefficients arranged in a zigzag order), corresponding to 16 frequency components. Figure 5 The upper part shows the process of generating four scan paths in the intra-block priority continuous scan mode; the lower part shows the process of generating four scan paths in the same-frequency cross-block priority continuous scan mode by schematically aggregating the same frequency components in different sub-blocks.

[0154] Step 5: Use the feature fusion module to fuse shallow features and deep features to obtain fused features.

[0155] In step 5, specifically, the deep features are input into a convolutional layer through the feature fusion module to refine the features and obtain refined deep features. Then, the refined deep features are added to the shallow features through residual connections to obtain fused features.

[0156] Step 6: Input the fused features into the upsampling reconstruction module for upsampling reconstruction to obtain the super-resolution reconstructed panoramic image; construct a loss function based on the super-resolution reconstructed panoramic image and the real panoramic image, and use the loss function to optimize the reconstruction model to obtain the optimized reconstruction model; input the low-resolution panoramic image into the optimized reconstruction model to obtain the final super-resolution reconstructed panoramic image.

[0157] In step 6, a loss function is constructed based on the super-resolution reconstructed panoramic image and the real panoramic image. The corresponding relationship in this process is as follows:

[0158] ;

[0159] in, Represents the loss function. This represents a panoramic image reconstructed using super-resolution technology. Represents a true panoramic image. This represents the L1 norm.

[0160] Please see Figure 6 The present invention also proposes a panoramic image super-resolution reconstruction system based on an improved Mamba, the system comprising:

[0161] Build modules are used for:

[0162] The reconstruction model is composed of a distortion information generation module, a shallow feature extraction module, a deep feature extraction module, a feature fusion module, and an upsampling reconstruction module.

[0163] The information generation module is used for:

[0164] The low-resolution panoramic image is input into the distortion information generation module. Based on the mapping relationship between the equidistant cylindrical projection format of the low-resolution panoramic image and the spherical coordinates, the area stretching ratio before and after projection is calculated to determine the degree of geometric distortion corresponding to the spatial position of each pixel.

[0165] The first extraction module is used for:

[0166] The low-resolution panoramic image is input into the shallow feature extraction module, and shallow features are extracted using convolution operations.

[0167] The second extraction module is used for:

[0168] Shallow features are input into the deep feature extraction module, and deep features are extracted based on the geometric distortion map using cascaded feature processing units. The feature processing units combine the geometric distortion map with the geometric distortion perception window attention mechanism and the geometric distortion modulation Mamba mechanism to achieve spatial local geometric perception enhancement and spatial global distortion modulation sequence modeling, respectively. On the other hand, they achieve frequency domain structured sequence modeling through the block DCT domain scanning Mamba mechanism, which is independent of the geometric distortion map.

[0169] The fusion module is used for:

[0170] The feature fusion module is used to fuse shallow features and deep features to obtain fused features;

[0171] The rebuild module is used for:

[0172] The fused features are input into the upsampling reconstruction module for upsampling reconstruction to obtain the super-resolution reconstructed panoramic image. A loss function is constructed based on the super-resolution reconstructed panoramic image and the real panoramic image. The reconstruction model is optimized using the loss function to obtain the optimized reconstruction model. The low-resolution panoramic image is input into the optimized reconstruction model to obtain the final super-resolution reconstructed panoramic image.

[0173] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0174] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0175] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A panoramic image super-resolution reconstruction method based on an improved Mamba, characterized in that, The method includes the following steps: Step 1: Construct a reconstruction model based on the distortion information generation module, shallow feature extraction module, deep feature extraction module, feature fusion module, and upsampling reconstruction module; Step 2: Input the low-resolution panoramic image into the distortion information generation module. Based on the mapping relationship between the equidistant cylindrical projection format of the low-resolution panoramic image and the spherical coordinates, calculate the area stretching ratio before and after projection to determine the degree of geometric distortion corresponding to the spatial position of each pixel. Step 3: Input the low-resolution panoramic image into the shallow feature extraction module and use convolution operation to extract shallow features; Step 4: Input the shallow features into the deep feature extraction module, and use the cascaded feature processing units to extract deep features based on the geometric distortion map to obtain deep features; wherein, the feature processing unit combines the geometric distortion map and uses the geometric distortion perception window attention mechanism and the geometric distortion modulation Mamba mechanism to realize spatial local geometric perception enhancement and spatial global distortion modulation sequence modeling, respectively; on the other hand, it realizes frequency domain structured sequence modeling through the block DCT domain scanning Mamba mechanism that does not depend on the geometric distortion map. Step 5: Use the feature fusion module to fuse the shallow features and deep features to obtain the fused features; Step 6: Input the fused features into the upsampling reconstruction module to perform upsampling reconstruction and obtain the super-resolution reconstructed panoramic image; construct a loss function based on the super-resolution reconstructed panoramic image and the real panoramic image, and use the loss function to optimize the reconstruction model to obtain the optimized reconstruction model; input the low-resolution panoramic image into the optimized reconstruction model to obtain the final super-resolution reconstructed panoramic image. In step 4, the shallow features are input into the deep feature extraction module, and the deep features are extracted based on the geometric distortion map using cascaded feature processing units. This process includes the following steps: S401. Shallow features are used as input features to the first feature processing unit in the deep feature extraction module and input into the first geometric distortion sensing window attention block. Local geometric sensing and enhancement processing is performed on the input features according to the geometric distortion degree map to obtain the output features of the first geometric distortion sensing window attention block. The feature processing unit further includes: a geometric distortion modulation Mamba block, a second geometric distortion sensing window attention block, a block-based DCT domain scanning Mamba block, and a convolutional layer. The first geometric distortion sensing window attention block, the geometric distortion modulation Mamba block, the second geometric distortion sensing window attention block, the block-based DCT domain scanning Mamba block, and the convolutional layer are connected in sequence. The first geometric distortion sensing window attention block and the second geometric distortion sensing window attention block have the same structure. Each feature processing unit has the same structure. The block-based DCT domain scanning Mamba block is used to: perform block-based discrete cosine transform and perform scanning and sequence modeling in the frequency domain according to a preset structured order.

2. The panoramic image super-resolution reconstruction method based on improved Mamba according to claim 1, characterized in that, In step 2, based on the mapping relationship between the equidistant cylindrical projection format of the low-resolution panoramic image and spherical coordinates, the area stretching ratio before and after projection is calculated to determine the degree of geometric distortion corresponding to the spatial position of each pixel. The corresponding relationship in this process is as follows: ； in, A graph representing the degree of geometric distortion. This represents the height index of a low-resolution panoramic image. Represents the cosine function. Indicates the height of a low-resolution panoramic image. It represents pi (π).

3. The panoramic image super-resolution reconstruction method based on improved Mamba according to claim 2, characterized in that, In step 4, the shallow features are input into the deep feature extraction module, and deep features are extracted based on the geometric distortion map using cascaded feature processing units to obtain deep features. Specifically, the following steps are also included: S402. Input the output features of the attention block of the first geometric distortion perception window and the geometric distortion degree map into the geometric distortion modulation Mamba block. Perform an affine transformation on the input features based on the geometric distortion degree map and modulate the parameters of the state space model to complete the spatial global adaptive sequence modeling and obtain the output features of the geometric distortion modulation Mamba block. S403. Input the output features of the geometric distortion modulation Mamba block and the geometric distortion degree map into the attention block of the second geometric distortion perception window, and then perform local geometric perception and enhancement processing on the input features according to the geometric distortion degree map to obtain the output features of the attention block of the second geometric distortion perception window. S404. Input the output features of the attention block of the second geometric distortion sensing window into the block-based DCT domain scanning Mamba block, perform block-based discrete cosine transform, and perform scanning and sequence modeling in the frequency domain according to a preset structured order to obtain the output features of the block-based DCT domain scanning Mamba block. S405. The output features of the block-based DCT domain scan Mamba block are input into the convolutional layer for processing. The output of the convolutional layer is added to the input features of the current feature processing unit through residual connections to obtain the output features of the current feature processing unit. S406. Input the output features of the current feature processing unit into the next feature processing unit in an iterative manner, and repeat steps S401 to S405 to obtain the output features of the next feature processing unit. S407. Take the output feature of the next feature processing unit as input and input it into the last feature processing unit. Repeat steps S401 to S405 to obtain the output feature of the last feature processing unit and take the output feature of the last feature processing unit as the deep feature.

4. The panoramic image super-resolution reconstruction method based on improved Mamba according to claim 3, characterized in that, The shallow features are used as the input features of the first feature processing unit in the deep feature extraction module and input into the attention block of the first geometric distortion perception window. The input features are then subjected to local geometric perception and enhancement processing based on the geometric distortion degree map to obtain the output features of the attention block of the first geometric distortion perception window. The specific steps include the following: Shallow features are used as input features and input into the first geometric distortion perception window attention block of the first feature processing unit in the deep feature extraction module. Normalization processing is then performed to obtain the features after the first layer normalization processing. The first geometric distortion perception window attention block includes a geometric distortion perception window attention unit and a first feedforward network layer. The features after the first layer of normalization are input into the geometric distortion perception window attention unit for processing, and the output features of the geometric distortion perception window attention unit are obtained. The output features of the attention unit of the geometric distortion sensing window are connected with the input features of the attention block of the first geometric distortion sensing window to obtain the features after the first residual connection. The features after the first residual connection are subjected to a second layer of normalization to obtain the features after the second layer of normalization. The features after the second layer of normalization are input into the first feedforward network layer for feedforward processing to obtain the output features of the first feedforward network layer. The output features of the first feedforward network layer are combined with the features after the first residual connection to form a second residual connection, thereby obtaining the output features of the attention block of the first geometric distortion perception window. The process involves inputting the normalized features from the first layer into the geometric distortion perception window attention unit for further processing, resulting in the output features of the geometric distortion perception window attention unit. This includes the following sub-steps: The features after the first layer of normalization are input into the attention unit of the geometric distortion perception window, which is divided into non-overlapping windows of a preset size. The features in each non-overlapping window are linearly projected to obtain the query vector. Window conditions representing relative positions within non-overlapping windows are obtained and concatenated with a geometric distortion degree map along the channel dimension to obtain concatenated features. The concatenated features are then processed using a lightweight network in the attention block of the first geometric distortion-aware window to generate adaptive position offsets in the height and width directions of each position of the input features of the attention unit of the geometric distortion-aware window. The lightweight network consists of three linear layers. Based on the adaptive position offset of each position of the input feature of the geometric distortion sensing window attention unit in the height and width directions, bilinear sampling is performed on the input feature of the geometric distortion sensing window attention unit to obtain the feature after position offset. The features after positional offset are divided into non-overlapping windows of a preset size, and the features within each window are linearly projected to generate key vectors and value vectors. Window self-attention is calculated using query vectors, key vectors, and value vectors to obtain the output features of the attention unit of the geometric distortion-aware window. The relationships between these parameters in the window self-attention calculation process are as follows: ； in, This indicates the calculation of window self-attention. Represents the query vector. Represents the key vector. Represents a value vector. Represents the normalized exponential function, Represents the transpose of the key vector , Represents the query vector and key vector Dimensions This indicates the relative position offset term.

5. The panoramic image super-resolution reconstruction method based on improved Mamba according to claim 4, characterized in that, The output features of the attention block of the first geometric distortion perception window and the geometric distortion degree map are input into the geometric distortion modulation Mamba block. Based on the geometric distortion degree map, an affine transformation is performed on the input features, and the parameters of the state space model are modulated to complete the spatial global adaptive sequence modeling, thereby obtaining the output features of the geometric distortion modulation Mamba block. The specific steps include the following: The output features of the attention block of the first geometric distortion perception window and the geometric distortion degree map are input into the geometric distortion modulation Mamba block, and layer normalization is performed to obtain the features after the third layer normalization. The geometric distortion modulation Mamba block includes: a geometric distortion modulation Mamba unit and a second feedforward network layer; the geometric distortion modulation Mamba unit includes an affine transformation module, a first branch, and a second branch; both the first and second branches are composed of linear projection, a 3×3 depth convolutional layer, and a SiLU activation function layer connected sequentially; the affine transformation module consists of an affine transformation coefficient generation network and affine transformation operations; the affine transformation coefficient generation network consists of the innermost convolutional layer, the first leaky rectified linear unit activation function, the middle convolutional layer, the second leaky rectified linear unit activation function, and the outermost convolutional layer; The features after the third layer normalization process are input into the geometric distortion modulation Mamba unit for processing to obtain the output features of the geometric distortion modulation Mamba unit. The output features of the geometrically distorted modulation Mamba unit are joined with the input features of the geometrically distorted modulation Mamba block by a third residual connection to obtain the features after the third residual connection. The features after the third residual connection are processed by layer normalization and the second feedforward network layer to obtain the output features of the second feedforward network layer. The output features of the second feedforward network layer are combined with the features after the third residual connection to form a fourth residual connection, resulting in the output features of the geometric distortion modulation Mamba block. The process involves inputting the layer-normalized features into a geometric distortion modulation (MDM) Mamba unit for processing to obtain the output features of the MBM unit, including the following sub-steps: The geometric distortion map is input into the affine transformation module, and then processed sequentially through the innermost convolutional layer, the first leaky rectified linear unit (LER) activation function, the middle convolutional layer, the second LRR activation function, and the outermost convolutional layer. This yields the outermost convolutional layer features. Channel segmentation is then performed on these features to obtain the segmented output features. These segmented output features are then split into two tensors, which are used as two affine transformation coefficients. and The layer-normalized features are input into the geometric distortion modulation Mamba unit, based on two affine transform coefficients. and An affine transformation is performed on the input features of the geometrically distorted modulation Mamba unit. This is done by multiplying one of the two affine transformation coefficients element-wise with the input features of the geometrically distorted modulation Mamba unit and adding the other of the two affine transformation coefficients to obtain the affine transformed features. The input features of the geometric distortion modulated Mamba unit are input into the first branch, and then processed sequentially through linear projection, a 3×3 depth convolutional layer, and SiLU activation function to obtain the input branch features. The features after affine transformation are input into the second branch, and then processed sequentially through linear projection, a 3×3 depth convolutional layer, and SiLU activation function to obtain the modulated branch features. S4021. Flatten the input branch features into a one-dimensional sequence, use it as the input sequence of the state space model, and generate the output matrix of the state space model through a linear transformation based on the input sequence; simultaneously flatten the modulation branch features into a one-dimensional sequence, and dynamically generate the input matrix and discretization step size of the state space model through a linear transformation based on the one-dimensional sequence of the modulation branch features; perform state space modeling through latent states based on the input sequence, output matrix, input matrix, discretization step size, and two preset learnable parameters of the state space model. The input branch features and modulation branch features are flattened into one-dimensional sequences along four directions: horizontal forward, horizontal reverse, vertical forward, and vertical reverse, using a four-directional scanning mechanism. For each scanning direction, step S4021 is repeated to obtain the output features in the four directions. The output features in the four directions are rearranged into two-dimensional features and then fused element by element to obtain the spatial state modeling features. The input features of the geometrically distorted modulation Mamba unit are sequentially processed by linear projection and SiLU activation function to generate a gated signal; the spatial state modeling features are sequentially processed by layer normalization, element-wise multiplication with the gated signal, and linear projection to obtain the output features of the geometrically distorted modulation Mamba unit.

6. The panoramic image super-resolution reconstruction method based on improved Mamba according to claim 5, characterized in that, The geometric distortion map is input into the affine transformation module of the geometric distortion modulation Mamba unit. It sequentially passes through the innermost convolutional layer, the first leaky rectified linear unit (LER) activation function, the middle convolutional layer, the second LRR activation function, and the outermost convolutional layer, resulting in the outermost convolutional layer feature. Channel segmentation is then performed on this outermost convolutional layer feature to obtain the channel-segmented output feature. This channel-segmented output feature is then split into two tensors, which are used as two affine transformation coefficients. and The layer-normalized features are input into the geometric distortion modulation Mamba unit, based on two affine transform coefficients. and Affine transformation is performed on the input features of the geometrically distorted modulation (MDM) Mamba unit. This is achieved by element-wise multiplying one of the two affine transformation coefficients with the input features of the MBM unit and then adding the result to the other of the two affine transformation coefficients. The relationships involved in this process are as follows: ； in, This represents the affine transformation coefficient generation network; This indicates a channel splitting operation. This indicates a 1×1 convolution process. This represents the activation function of the leaky rectifier linear unit. Represents the characteristics after affine transformation. This indicates element-wise multiplication. This indicates element-wise addition. This represents the input characteristics of the geometric distortion modulation Mamba unit; In the process of inputting the geometric distortion modulated Mamba unit's input features into the first branch, and then sequentially processing them through linear projection, a 3×3 depthwise convolutional layer, and SiLU activation function to obtain the input branch features, the corresponding relationships in the process are as follows: ； in, Represents the input branch features, This represents the SiLU activation function. This indicates a 3×3 depthwise convolution process. This indicates linear projection processing; In the process of inputting the affine transformation features into the second branch, and then sequentially processing them through linear projection, a 3×3 depth convolutional layer, and SiLU activation function to obtain the modulated branch features, the corresponding relationships in the process are as follows: ； in, Indicates modulation branch characteristics; In the steps of flattening the input branch features into a one-dimensional sequence as the input sequence of the state-space model, and generating the output matrix of the state-space model through a linear transformation based on the input sequence; simultaneously flattening the modulation branch features into a one-dimensional sequence, and dynamically generating the input matrix and discretization step size of the state-space model through a linear transformation based on the one-dimensional sequence of the modulation branch features; and performing state-space modeling through the latent state based on the input sequence, output matrix, input matrix, discretization step size, and two preset learnable parameters, the corresponding relationships in the process are as follows: ； in, Representing the potential state Regarding time The derivative, This represents the first preset learnable parameter. Indicates a potential state. Represents the input matrix, Represents a one-dimensional input sequence. Output feature sequence, Indicates the output matrix. This indicates the second preset learnable parameter; In the steps of processing the input features of the geometrically distorted modulation Mamba unit sequentially through linear projection and SiLU activation function to generate a gated signal, and processing the spatial state modeling features sequentially through layer normalization, element-wise multiplication with the gated signal, and linear projection to obtain the output features of the geometrically distorted modulation Mamba unit, the corresponding relationships in the process are as follows: ； in, Indicates a gating signal. This represents the output characteristics of the geometric distortion modulation Mamba unit. Presentation layer normalization processing.

7. The panoramic image super-resolution reconstruction method based on improved Mamba according to claim 6, characterized in that, The output features of the attention block of the second geometric distortion sensing window are input into the block-based DCT domain scanning Mamba block, and block-based discrete cosine transform is performed. Scanning and sequence modeling are then performed in the frequency domain according to a preset structured order to obtain the output features of the block-based DCT domain scanning Mamba block. The specific steps include the following: The output features of the second geometric distortion modulation Mamba unit are input into the block-based DCT domain scanning Mamba block for layer normalization to obtain the fourth layer normalized features. The block-based DCT domain scanning Mamba block includes a block-based DCT domain scanning Mamba unit and a third feedforward network layer. Each block-based DCT domain scanning Mamba unit includes a spatial domain feature mapping network and a dual-mode scanning frequency domain modeling submodule. The spatial domain feature mapping network is composed of linear projection, a 3×3 depth convolutional layer, and a SiLU activation function connected sequentially. The dual-mode scanning frequency domain modeling submodule includes an intra-block priority continuous scanning mode and a same-frequency cross-block priority continuous scanning mode. The normalized features of the fourth layer are input into the block DCT domain scan Mamba cell for processing to obtain the output features of the block DCT domain scan Mamba cell. The output features of the block-based DCT domain scan Mamba cell are combined with the input features of the block-based DCT domain scan Mamba block to form a fifth residual connection, resulting in the features after the fifth residual connection. The features after the fifth residual connection are processed by layer normalization and then by the third feedforward network layer to obtain the output features of the third feedforward network layer. The output features of the third feedforward network layer are combined with the features after the fifth residual connection to form a sixth residual connection, so as to obtain the output features of the block-based DCT domain scanning Mamba block. Specifically, the normalized features of the fourth layer are input into the block-based DCT domain scanning Mamba cells for processing to obtain the output features of the block-based DCT domain scanning Mamba cells, including: The input features of the block-based DCT domain scanning Mamba unit are processed by linear projection, 3×3 depth convolutional layer and SiLU activation function of the spatial domain feature mapping network to obtain spatial domain mapping features. The spatial domain mapping features are divided into non-overlapping sub-blocks of a preset size, and a two-dimensional discrete cosine transform operation is performed on each non-overlapping sub-block to obtain two-dimensional frequency domain features. The two-dimensional frequency domain feature is divided into two frequency domain sub-features along the channel dimension, each with half the number of channels of the original two-dimensional frequency domain feature. These are denoted as the first frequency domain sub-feature and the second frequency domain sub-feature, respectively. Based on the intra-block priority continuous scanning mode, four scanning paths are generated according to four predefined inter-block traversal orders for the first frequency domain sub-feature. According to the order defined for each path, the first frequency domain sub-feature is flattened into four one-dimensional feature sequences. Based on the four one-dimensional feature sequences of the intra-block priority continuous scanning mode, the state space model parameters of the corresponding scanning path of the intra-block priority continuous scanning mode are generated by linear transformation. The rule for the block-priority continuous scanning mode is: S4041, scan all frequency domain coefficients in a two-dimensional zigzag order from low frequency to high frequency. After completing the intra-block scan of the current sub-block, jump to the next sub-block according to the current inter-block traversal order, and repeat the intra-block scan process of step S4041 until all sub-blocks have been scanned. For the second frequency domain sub-feature, based on the same-frequency cross-block priority continuous scanning mode, four scanning paths are generated according to the four predefined traversal orders between sub-blocks. According to the order defined for each path, the second frequency domain sub-feature is flattened into four one-dimensional feature sequences. Based on the four one-dimensional feature sequences of the same-frequency cross-block priority continuous scanning mode, the state space model parameters of the corresponding scanning path of the same-frequency cross-block priority continuous scanning mode are generated through linear transformation. The rule for the same-frequency cross-block priority continuous scanning mode is as follows: S4042, select a frequency component in sequence according to the two-dimensional zigzag order from low frequency to high frequency, and scan the coefficients corresponding to the current frequency component in all sub-blocks in sequence according to the current sub-block traversal order. After completing the scanning of the current frequency component in all sub-blocks, move to the next frequency component and repeat the scanning process of step S4042 until all frequency components have been scanned. The four one-dimensional feature sequences generated according to the intra-block priority continuous scanning mode, along with the state space model parameters of the corresponding scanning path of the intra-block priority continuous scanning mode, are input into the state space model for parallel modeling to obtain the four output sequences of the four scanning paths corresponding to the intra-block priority continuous scanning mode. Through a rearrangement operation, the four output sequences of the four scanning paths corresponding to the intra-block priority continuous scanning mode are mapped back to the original two-dimensional frequency domain sub-feature layout according to their respective scanning paths, and then fused by element-wise addition to obtain the first fused feature. Based on the four one-dimensional feature sequences of the same-frequency cross-block priority continuous scanning mode, and the state space model parameters of the corresponding scanning path of the same-frequency cross-block priority continuous scanning mode, the parameters are input into the state space model for parallel modeling to obtain the four output sequences of the four scanning paths corresponding to the same-frequency cross-block priority continuous scanning mode. The four output sequences of the four scanning paths corresponding to the same-frequency cross-block priority continuous scanning mode are mapped back to the original two-dimensional frequency domain sub-feature layout according to their respective scanning paths through a rearrangement operation, and then fused by element-wise addition to obtain the second fused feature. The first fusion feature and the second fusion feature are concatenated along the channel dimension to obtain the combined feature, which is used as the frequency domain feature after sequence modeling. After modeling the sequence, the frequency domain features are sequentially subjected to layer normalization and block-based inverse two-dimensional discrete cosine transform operations to be converted back to the spatial domain, thus obtaining the spatial domain reconstructed features. The input features of the block-based DCT domain scanning Mamba cell are sequentially processed by linear projection and SiLU activation function to generate a gated signal. The spatial reconstruction features are then multiplied element-wise with the gated signal, followed by linear projection to obtain the output features of the block-based DCT domain scanning Mamba cell.

8. The panoramic image super-resolution reconstruction method based on improved Mamba according to claim 7, characterized in that, The input features of the Mamba unit are scanned in the block DCT domain and processed through linear projection, 3×3 depth convolutional layers, and SiLU activation function of the spatial domain feature mapping network to obtain the spatial domain mapped features. The corresponding relationship in the process is as follows: ； in, Represents spatial mapping characteristics, This represents the input features of the Mamba cell in a block-based DCT domain scan; In the steps of dividing the spatial domain mapping features into non-overlapping sub-blocks of a preset size and performing a two-dimensional discrete cosine transform operation on each non-overlapping sub-block to obtain two-dimensional frequency domain features, the corresponding process relationships are as follows: ； in, Representing two-dimensional frequency domain characteristics, This represents the two-dimensional discrete cosine transform operation. Indicates block operation; In the step of dividing the two-dimensional frequency domain feature into two frequency domain sub-features along the channel dimension, each with half the number of channels of the original two-dimensional frequency domain feature, denoted as the first frequency domain sub-feature and the second frequency domain sub-feature respectively, the corresponding relationship in the process is as follows: ； in, Represents the first frequency domain sub-feature. Indicates the second frequency domain sub-feature; In the process of generating four one-dimensional feature sequences based on the intra-block priority continuous scanning mode, and inputting the state space model parameters of the corresponding scanning paths of the intra-block priority continuous scanning mode into the state space model for parallel modeling, four output sequences corresponding to the four scanning paths of the intra-block priority continuous scanning mode are obtained. Through a rearrangement operation, the four output sequences corresponding to the four scanning paths of the intra-block priority continuous scanning mode are mapped back to the original two-dimensional frequency domain sub-feature layout according to their respective scanning paths, and then fused by element-wise addition to obtain the first fused feature, the corresponding relationship in the process is as follows: ； in, Indicates the first sequential scan mode within the block. The output sequence of the state-space model corresponding to each scan path. Representing a state-space model, Indicates the first sequential scan mode within the block. One-dimensional feature sequences corresponding to each scan path. , , These represent the block-priority continuous scan modes, respectively. One-dimensional feature sequence corresponding to each scanning path Discretization step size, input matrix, and output matrix of the state-space model generated by linear transformation. , These represent the block-priority continuous scan modes, respectively. Two learnable parameters of the state-space model corresponding to each scan path. Indicates the first fusion feature, Indicates a rearrangement operation; In the step of parallel modeling, four one-dimensional feature sequences based on the same-frequency cross-block priority continuous scanning mode and the state space model parameters of the corresponding scanning paths of the same-frequency cross-block priority continuous scanning mode are input into the state space model to obtain four output sequences corresponding to the four scanning paths of the same-frequency cross-block priority continuous scanning mode. Then, through a rearrangement operation, the four output sequences corresponding to the four scanning paths of the same-frequency cross-block priority continuous scanning mode are mapped back to the original two-dimensional frequency domain sub-feature layout according to their respective scanning paths. Finally, they are fused by element-wise addition to obtain the second fused feature. The corresponding relationship in this process is as follows: ； in, Indicates the same frequency cross-block priority continuous scan mode. The output sequence of the state-space model corresponding to each scan path. Indicates the same frequency cross-block priority continuous scan mode. One-dimensional feature sequences corresponding to each scan path. , , These represent the first and second consecutive scan modes with the same frequency and cross-block priority. One-dimensional feature sequence corresponding to each scanning path Discretization step size, input matrix, and output matrix of the state-space model generated by linear transformation. , These represent the first and second consecutive scan modes with the same frequency and cross-block priority. Two learnable parameters of the state-space model corresponding to each scan path. Indicates the second fusion feature; In the step of concatenating the first fused feature and the second fused feature along the channel dimension to obtain the combined feature, the corresponding relationship in the process is as follows: ； in, Represents combined features, Indicates a splicing operation; In the steps of processing the input features of the block-based DCT domain scanning Mamba cell by sequentially performing linear projection and SiLU activation function processing to generate a gated signal, and then performing element-wise multiplication of the spatial domain reconstructed features with the gated signal followed by linear projection processing to obtain the output features of the block-based DCT domain scanning Mamba cell, the corresponding relationships in the process are as follows: ； in, Indicates the input features of block DCT domain scanning Mamba cells The generated gating signal, This represents the output characteristics of the Mamba cell in a block-based DCT domain scan. This represents the inverse two-dimensional discrete cosine transform operation.

9. The panoramic image super-resolution reconstruction method based on improved Mamba according to claim 8, characterized in that, In step 6, a loss function is constructed based on the super-resolution reconstructed panoramic image and the real panoramic image. The corresponding relationship in the process is as follows: ； in, Represents the loss function. This represents a panoramic image reconstructed using super-resolution technology. Represents a true panoramic image. This represents the L1 norm.

10. A panoramic image super-resolution reconstruction system based on an improved Mamba, characterized in that, The system employs the panoramic image super-resolution reconstruction method based on the improved Mamba as described in any one of claims 1-9, and the system comprises: Build modules are used for: The reconstruction model is composed of a distortion information generation module, a shallow feature extraction module, a deep feature extraction module, a feature fusion module, and an upsampling reconstruction module. The information generation module is used for: The low-resolution panoramic image is input into the distortion information generation module. Based on the mapping relationship between the equidistant cylindrical projection format of the low-resolution panoramic image and the spherical coordinates, the area stretching ratio before and after projection is calculated to determine the degree of geometric distortion corresponding to the spatial position of each pixel. The first extraction module is used for: The low-resolution panoramic image is input into the shallow feature extraction module, and shallow features are extracted using convolution operations. The second extraction module is used for: Shallow features are input into the deep feature extraction module, and deep features are extracted based on the geometric distortion map using cascaded feature processing units. The feature processing units combine the geometric distortion map with the geometric distortion perception window attention mechanism and the geometric distortion modulation Mamba mechanism to achieve spatial local geometric perception enhancement and spatial global distortion modulation sequence modeling, respectively. On the other hand, they achieve frequency domain structured sequence modeling through the block DCT domain scanning Mamba mechanism, which is independent of the geometric distortion map. The fusion module is used for: The feature fusion module is used to fuse shallow features and deep features to obtain fused features; The rebuild module is used for: The fused features are input into the upsampling reconstruction module for upsampling reconstruction to obtain the super-resolution reconstructed panoramic image. A loss function is constructed based on the super-resolution reconstructed panoramic image and the real panoramic image. The reconstruction model is optimized using the loss function to obtain the optimized reconstruction model. The low-resolution panoramic image is input into the optimized reconstruction model to obtain the final super-resolution reconstructed panoramic image.

Images