Mamba network optimization method and device based on rotation equivariant depth image prior

Abstract

The application discloses a method and device for optimizing a Mamba network based on a rotation equivariant depth image prior, which comprises the following steps: converting original image data into a two-dimensional image tensor and dividing the two-dimensional image tensor into a training set and a test set; converting the two-dimensional image tensor into sequence data through a two-dimensional path scanning strategy of four-angle rotation equivariance; constructing a data modeling expression of the sequence data by using a Mamba network; constructing a visual Mamba module with four-angle rotation equivariance based on the data modeling expression; integrating the visual Mamba module to construct a visual state space model network, and training the visual state space model network by using the training set until a training stop condition is reached; and inputting the test set into the trained visual state space model network to obtain a processing result of an image task. The method solves the problem that the existing Mamba network causes insufficient accuracy and robustness of a visual processing task when an image is rotated. The accuracy and robustness of visual processing tasks such as target detection and semantic segmentation are significantly improved.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a method and apparatus for optimizing Mamba networks based on rotationally variable depth image priors. Background Technology

[0002] The Vision Transformer (VTR) is a mainstream model architecture in deep learning. Leveraging its powerful self-attention mechanism, it effectively captures non-local self-similarity within the global receptive field, overcoming the limitations of the local receptive field in traditional convolutional networks. It is widely used in various complex visual processing tasks. However, the inherent characteristics of VTRs lead to significant bottlenecks in large-scale image processing scenarios. Specifically, the computational complexity of VTRs increases quadratically with image size, making it difficult to efficiently process high-resolution images. Furthermore, its self-attention mechanism lacks spatial awareness, requiring manually designed positional encodings to supplement spatial information, often ignoring inherent spatial adjacency priors such as texture and shape.

[0003] Against this backdrop, the Mamba Network was proposed as a novel architecture following the state-space model paradigm. This architecture unifies the state-space model with the attention mechanism through the theory of structured state-space duality. In its implementation, the Mamba Network simplifies the state transition matrix into a structured mask and combines linear attention with the mask through the Hadamard product, forming a structured mask attention mechanism with spatial adjacency awareness. This design enables it to achieve linear computational complexity while maintaining the global receptive field, demonstrating excellent performance in natural language processing tasks. It has been rapidly extended to various vision-related fields, including high-level vision tasks (image classification, object detection, semantic segmentation), low-level vision tasks (super-resolution, image denoising, image deraining), image generation, video analysis, point cloud analysis, and remote sensing image processing, becoming a current research hotspot in the field of deep learning.

[0004] While the Mamba Network excels in various visual tasks, its core state-space model lacks rotational equivariance. Existing Mamba Networks employ a four-way scanning mechanism to traverse image patches in a 2D image. When the 2D image undergoes rotational transformations, the traversal order of these patches is directly altered, leading to a significant performance degradation when dealing with image rotation. Furthermore, the lack of rotational equivariance prevents the capture of local rotational similarities in 2D images, resulting in inaccurate characterization of image details in visual processing tasks and impacting the accuracy of downstream tasks. This remains a major bottleneck for the further widespread application of the Mamba Network. Summary of the Invention

[0005] This application provides a method and apparatus for optimizing Mamba networks based on rotationally variable depth image priors, which solves the problem that existing Mamba networks suffer from insufficient accuracy and robustness in visual processing tasks when images are rotated.

[0006] In a first aspect, embodiments of this application provide a Mamba network optimization method based on rotationally equivariant depth image priors, comprising: converting raw image data from different visual processing tasks into two-dimensional image tensors and dividing them into training and testing sets; converting the two-dimensional image tensors in the training and testing sets into sequence data using a two-dimensional path scanning strategy with four-angle rotation equivariance; constructing a data modeling expression for the sequence data using the Mamba network; constructing a visual Mamba module with four-angle rotation equivariance based on the data modeling expression; integrating the visual Mamba module to construct a visual state space model network, and training the visual state space model network using the training set until the training stopping condition is met; and inputting the test set into the trained visual state space model network to obtain the processing results of the image task.

[0007] In conjunction with the first aspect, in one possible implementation, before converting the raw image data from different visual processing tasks into two-dimensional image tensors, the method further includes: cleaning the raw image data to remove invalid image data; and / or, normalizing the resolution and number of channels of the raw image data; and / or, performing data augmentation on the raw image data to enhance data diversity.

[0008] In conjunction with the first aspect, in one possible implementation, the conversion of the two-dimensional image tensors in the training set and the test set into sequence data using a two-dimensional path scanning strategy with four-angle rotational equivariance includes: constructing four mutually rotationally symmetric scanning paths; using the four scanning paths to perform block scanning on the two-dimensional image tensor to divide the two-dimensional image tensor into multiple image blocks, and arranging them sequentially to obtain four independent rotated image block groups; and converting the four rotated image block groups into one-dimensional sequence data respectively.

[0009] In conjunction with the first aspect, in one possible implementation, the construction of the Mamba network's data modeling expression for sequence data includes: constructing a recurrent expression for sequence data processing based on the state-space model paradigm of the Mamba network; converting the recurrent expression into an attention expression containing a structured mask matrix to associate the state-space model with the attention mechanism; wherein the values ​​of the elements in the structured mask matrix are determined by the product of decay factors.

[0010] In conjunction with the first aspect, in one possible implementation, the recursive expression is as follows: ;in, ; The attention expression is as follows: ;in, ; In the formula, Indicates the first position of the output sequence One token, Indicates the first There are 10 hidden variables, where the initial hidden variables are set as a zero vector. , , Represents the learnable network parameters, where, Indicates the attenuation factor. Indicates the input projection matrix. Indicates the output projection matrix. This represents the first of the input sequence data. One token, This represents the number of feature channels of the token. Indicates the dimension of the latent variable. This represents the output feature tensor. This represents the learnable network parameter matrix. This indicates the transpose operation. Represents a structured mask matrix. Indicates the length of the sequence data. This represents the input sequence data. Indicates the position index. This indicates the column number of the elements in the structured mask matrix. It represents the Hadamardi (or Hadama) stack.

[0011] In conjunction with the first aspect, in one possible implementation, constructing a visual Mamba module with four-angle rotation equivariance based on the data modeling expression includes: reconstructing the network parameters of the Mamba network based on the four-angle rotation equivariance; using a rotationally equivariant linear layer as the feature extraction layer of the Mamba network to calculate the network parameters in the Mamba network under the condition of sharing network parameters; transforming the remaining network layers in the Mamba network into network layers with rotation equivariance to ensure that the output of each network layer changes synchronously with the rotation / rotation of the input feature map, thereby obtaining a visual Mamba module with four-angle rotation equivariance.

[0012] In conjunction with the first aspect, in one possible implementation, the integration of the visual mamba modules to construct a visual state space model network includes: cascading a pre-feature extraction layer, four visual mamba layers, a downsampling layer, and an output layer in sequence to obtain a visual state space model network; wherein, the visual mamba layer includes multiple visual mamba modules connected in sequence.

[0013] Secondly, embodiments of this application provide a Mamba network optimization device based on rotationally equivariant depth image priors, comprising: a data processing module for converting raw image data from different visual processing tasks into two-dimensional image tensors and dividing them into training and testing sets; an image conversion module for converting the two-dimensional image tensors in the training and testing sets into sequence data using a two-dimensional path scanning strategy with four-angle rotation equivariance; a data modeling module for constructing a data modeling expression for the sequence data using the Mamba network; a module construction module for constructing a visual Mamba module with four-angle rotation equivariance based on the data modeling expression; a network construction and training module for integrating the visual Mamba module to construct a visual state space model network and training the visual state space model network using the training set until the training stopping condition is met; and a testing module for inputting the test set into the trained visual state space model network to obtain the processing results of the image task.

[0014] Thirdly, embodiments of this application provide an apparatus comprising: a processor; a memory for storing processor-executable instructions; wherein, when the processor executes the executable instructions, it implements the method as described in the first aspect or any possible implementation of the first aspect.

[0015] Fourthly, embodiments of this application provide a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium including storage for storing a computer program or instructions that, when executed, cause the method described in the first aspect or any possible implementation of the first aspect to be implemented.

[0016] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: This application's embodiments convert raw image data into a two-dimensional image tensor, unifying the input data format for different visual tasks and eliminating differences in resolution, number of channels, and other dimensions of the raw image. This provides a standardized input foundation for subsequent path scanning with four-angle rotation equivariance. By converting the two-dimensional image tensor into sequential data through a two-dimensional path scanning strategy with four-angle rotation equivariance, the image rotation structure information can be preserved while adapting to the processing requirements of the Mamba network for sequential input. This ensures that the traversal order of image blocks remains consistent with the rotation transformation, avoiding information loss or misalignment caused by rotation. Constructing the data modeling expression for the Mamba network can incorporate the cyclic characteristics of the state-space model. The combination of parallel advantages of the attention mechanism and the visual Mamba module provides a rigorous mathematical framework for its rotational equivariance design. Based on this expression, the visual Mamba module achieves the core characteristic of synchronous changes in output with input rotation by reconstructing network parameters, introducing rotationally equivariant linear layers, and transforming other network layers into rotationally equivariant layers. This effectively overcomes the technical bottleneck of traditional Mamba networks lacking rotational equivariance. By integrating the visual state-space model network constructed from the visual Mamba module, and through pre-feature extraction, multi-visual Mamba layer cascading, downsampling, and collaborative work with the output layer, it can efficiently process high-resolution images while accurately capturing local rotational similarities. This solves the problem of insufficient accuracy and robustness in visual processing tasks caused by image rotation in existing Mamba networks. Consequently, it significantly improves the accuracy and robustness of visual processing tasks such as object detection and semantic segmentation, providing a feasible solution for the widespread application of Mamba networks in complex visual fields such as remote sensing image processing and medical image analysis. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of a Mamba network optimization method based on rotationally variable depth image priors provided in an embodiment of this application; Figure 2 The existing state-space model based on Mamba networks provided in the embodiments of this application employs a four-way scanning mechanism; Figure 3 This application provides a two-dimensional path scanning strategy with four-angle rotational equivariance. Figure 4 The visual Mamba module with four-angle rotational equivariance provided in the embodiments of this application; Figure 5This is a schematic diagram of the structure of the Mamba network optimization device based on rotationally variable depth image prior provided in the embodiments of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0020] The following description of some technologies involved in the embodiments of this application is provided to aid understanding and should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, some descriptions of well-known functions and structures are omitted in the following description.

[0021] Figure 1 This is a flowchart of the Mamba network optimization method based on rotationally variable depth image prior provided in this application embodiment, including steps 101 to 106. Wherein, Figure 1 This is merely one execution order shown in the embodiments of this application and does not represent the only execution order of the Mamba network optimization method based on rotationally variable depth image priors. Where the final result can be achieved, Figure 1 The steps shown can be performed in parallel or in reverse order.

[0022] Step 101: Convert the raw image data from different visual processing tasks into two-dimensional image tensors and divide them into training and testing sets. In this embodiment, before converting the raw image data from different visual processing tasks into two-dimensional image tensors, the method further includes: cleaning the raw image data to remove invalid image data; and / or, standardizing the resolution and number of channels of the raw image data; and / or, performing data augmentation on the raw image data to enhance data diversity.

[0023] Specifically, public or private raw image data corresponding to different visual processing tasks (classification, segmentation, detection, etc.) are collected, and the basic attributes of the raw image data (resolution, number of channels, annotation format, etc.) are recorded. The raw image data is then cleaned to remove blurry, incorrectly labeled, corrupted, or duplicate invalid image data. Further, during the data cleaning process, the variance of the raw image data can be calculated using the Laplacian variance method. Blurry image data with a variance value below a preset threshold (e.g., 100) is marked as invalid; raw image data containing large areas of salt-and-pepper noise (ratio > 10%) or Gaussian noise (signal-to-noise ratio < 20dB) is also marked as invalid; and extremely small raw image data with a resolution lower than 128×128 is marked as invalid, ensuring that the retained raw image data is valid.

[0024] It can also standardize the resolution and number of channels of the original image data, and use bilinear interpolation or nearest neighbor interpolation algorithms to uniformly adjust the cleaned original image data to a preset resolution (e.g., 256×256 or 512×512); for images with inconsistent number of channels (e.g., RGB three-channel, grayscale single-channel, RGBA four-channel), they are uniformly converted to the standard three-channel RGB format (grayscale images are expanded to three channels through channel copying, and RGBA images have their alpha channel removed while retaining the RGB components), ensuring the dimensionality consistency of all original image data.

[0025] Data augmentation operations can also be performed on the original image data, including random horizontal flipping (probability set to 0.5), random vertical flipping (probability set to 0.2), random small-angle rotation (rotation angle range from -15° to 15°, after rotation, mirror filling or black pixel filling is used to maintain the resolution of the original image data), random scaling (scaling factor range from 0.8 to 1.2, after scaling, cropping or padding to the original resolution), random color dithering (adjusting brightness ±10%, contrast ±10%, saturation ±10%, hue ±5°), and random Gaussian noise addition (noise standard deviation range from 0.01 to 0.05). Through these diverse augmentation operations, a large number of variant data are generated while preserving the core features of the original image data, significantly expanding the data distribution range of the original image data and improving the adaptability and generalization performance of subsequent models to different visual deformations.

[0026] After data preprocessing, the original image data is converted into a two-dimensional image tensor (e.g., Batch×3×224×224) suitable for input to the model. Then, the processed two-dimensional image tensor is divided into a training set and a test set according to a preset ratio (e.g., 7:3 or 8:2). The training set is used for learning network parameters, and the test set is used to evaluate the final performance of the trained visual state space model network. During the division process, it is ensured that the sample distribution of the training set and the test set is consistent to avoid distortion of evaluation results due to data distribution deviation.

[0027] Step 102: Convert the two-dimensional image tensors in the training and test sets into sequence data using a two-dimensional path scanning strategy with four-angle rotational equivariance. In this embodiment, four mutually rotationally symmetric scanning paths are constructed. The two-dimensional image tensors are then scanned in blocks using these four scanning paths to divide the two-dimensional image tensors into multiple image blocks, which are then arranged sequentially to obtain four independent rotated image block groups. The four rotated image block groups are then converted into one-dimensional sequence data.

[0028] Specifically, four rotationally symmetrical scan paths are constructed, with the center of the 2D image tensor as the rotation axis and rotation angles of 0°, 90°, 180°, and 270°, respectively. These four scan paths satisfy the condition that only the sequence alternates after rotation, while the internal token order remains unchanged. The 0° rotation scan path starts from the top left corner of the 2D image tensor and scans in row-major order. The 90° rotation scan path corresponds to a column-major scan (from top right to bottom left), consistent with the row-major scan result after a 90° counter-clockwise rotation. The 180° rotation scan path corresponds to a scan after a 180° rotation, and the 270° rotation scan path corresponds to a scan after a 270° rotation. Furthermore, the rotational equivariance of the scan paths can be verified. Specifically, rotations of 0°, 90°, 180°, and 270° are performed on the test 2D image tensor to verify that the sequence data after scanning only alternates between the four scan paths, while the internal token order remains unchanged.

[0029] Then, four mutually rotationally symmetric scan paths are used to scan all the 2D image tensors in blocks. Each scan path divides the 2D image tensor into blocks, generating four independent groups of image blocks after scanning. Each group of image blocks is then arranged sequentially into a one-dimensional sequence of data, ensuring that the dimension and token format of each sequence of data match the input requirements. ,in This represents the first of the input sequence data. One token, This indicates the number of feature channels of the token.

[0030] For example, if the image patch size is 8×8 and the dimension is 3, each image patch is flattened into a one-dimensional token with a size of 1×192, meaning the token has 192 = 8×8×3 feature channels. The flattened image patches are then arranged sequentially as one-dimensional sequence data, with a sequence data length N = H×W / (8×8), where H and W are the height and width of the two-dimensional image tensor, respectively. The sequence data corresponding to the four scan paths has a format of 4×N×C.

[0031] like Figure 2 As shown, to enhance the receptive field and contextual semantic understanding capabilities of existing Mamba networks, they employ a four-directional scanning mechanism to process two-dimensional images, obtaining image patch sequences suitable for state-space models. The red arrows in the figure represent the scanning path and direction; four scans of the two-dimensional image yield four image patch sequences. However, the image patch sequences obtained by this scanning strategy lack rotational equivariance. Figure 2 As shown, the image patch sequence obtained by scanning a 2D image after a 90° rotation is completely scrambled compared to the scan result before rotation, indicating that the scanning method is overly sensitive to rotation. This drawback is that the algorithm's performance drops significantly when dealing with visual processing tasks such as classification, especially with rotated images. Furthermore, the local rotational similarity in 2D images cannot be well learned and represented when performing tasks such as denoising and semantic segmentation.

[0032] like Figure 3 As shown, in order to solve the problem that the existing four-way scanning mechanism of Mamba Networks does not have rotational equivariance, a two-dimensional path scanning strategy with four-angle rotational equivariance suitable for Mamba Networks was invented. Figure 3 The red arrows in the diagram represent the scan path and direction. Following the four-item mechanism, this process still involves four scans, with the four scan paths rotating symmetrically around each other. For example... Figure 3 As shown, after rotating a 2D image by 90°, the resulting sequence of image patches differs from the original sequence only in the order of the four sequences. The order of image patches within a single sequence remains unchanged. This scanning strategy exhibits equivariance to rotation in four directions. It can provide more stable performance when facing image processing tasks such as image classification and semantic segmentation.

[0033] Step 103: Construct the data modeling expression for sequence data using the Mamba Network. In this embodiment, based on the state-space model paradigm of the Mamba Network, a recurrent expression for sequence data processing is constructed; the recurrent expression is then converted into an attention expression containing a structured mask matrix to associate the state-space model with the attention mechanism; wherein, the values ​​of the elements in the structured mask matrix are determined by the product of decay factors.

[0034] Specifically, the core variables for Mamba networks to process sequence data are identified, including the input... Latent variables Attenuation factor (etc.) Define the dimensions, value space, and physical meaning of each core variable. Then, based on the first-order linear transition principle of the Mamba Network's State-Space Model (SSM), derive the formulas for updating latent variables and generating output tokens, thus obtaining the recursive expression. Finally, through matrix expansion, transform the recursive expression into an attention form for the entire sequence, thus obtaining the attention expression.

[0035] Furthermore, the recursive expression is as follows: ;in, ; The attention expression is as follows: ;in, ; In the formula, Indicates the first position of the output sequence One token, Indicates the first There are 10 hidden variables, where the initial hidden variables are set as a zero vector. , , Represents the learnable network parameters, where, Indicates the attenuation factor. Indicates the input projection matrix. Indicates the output projection matrix. This represents the first of the input sequence data. One token, This represents the number of feature channels of the token. Indicates the dimension of the latent variable. This represents the output feature tensor. This represents the learnable network parameter matrix. This indicates the transpose operation. Represents a structured mask matrix. Indicates the length of the sequence data. This represents the input sequence data. Indicates the position index. This indicates the column number of the elements in the structured mask matrix. It represents the Hadamardi (or Hadama) stack.

[0036] By utilizing the structured mask matrix of Mamba Networks, the problem of traditional learnable positional encoding being difficult to explicitly embed into the network can be effectively solved. Based on this technology, Mamba Networks have achieved state-of-the-art performance in a large number of natural language processing tasks.

[0037] Step 104: Construct a visual Mamba module with four-angle rotation equivariance based on the data modeling expression. In this embodiment, the network parameters of the Mamba network are reconstructed based on four-angle rotation equivariance; a rotationally equivariant linear layer is used as the feature extraction layer of the Mamba network to calculate the network parameters in the Mamba network under the condition of sharing network parameters; the remaining network layers in the Mamba network are transformed into network layers with rotation equivariance to ensure that the output of each network layer changes synchronously with the rotation / rotation of the input feature map, thus obtaining a visual Mamba module with four-angle rotation equivariance.

[0038] Specifically, extract the network parameters of the existing Mamba network. , , The dimensions of each network parameter are adjusted to adapt them to the sequence data. This involves expanding the original network parameters to four dimensions, corresponding to four rotation angles, while preserving the learnable characteristics of each parameter. Furthermore, the network parameter branches at the four rotation angles share the basic weights, with differences achieved only through cyclic shifts (e.g., shifting the weight matrix rows to the right).

[0039] An equivariant linear layer (EQ-Linear) is used as the feature extraction layer in the Mamba network to compute network parameters while sharing network parameters. The remaining layers in the Mamba network, such as convolutional layers, fully connected layers, normalization layers, and Dropout layers, are replaced with rotationally equivariant convolutional layers, fully connected layers, normalized layers, and Dropout layers, which possess rotational equivariance. This gives the reconstructed visual Mamba module four-angle rotational equivariance, meaning that when the input feature map is rotated or rotated, the output of the visual Mamba module also rotates or rotates.

[0040] Among them, the rotationally equivariant linear layer is a rotationally equivariant linear transformation unit designed for quadruple-rotated copy features in a group convolution framework. It achieves strict equivariance by employing grouped weight sharing and cyclic shifting mechanisms. The rotationally equivariant linear layer transforms the input features... according to Dimensional organization, Indicates the sequence length. Indicates channel dimension, Indicates the number of rotational isomorphisms (in a four-angle rotational isomorphism, ... The specific calculation process can be expressed as follows: , In the formula, This represents the geometric transformation group consisting of all rotationally equivalent angles, including... A rotational constant angle, and All The elements in the array serve as indices of the input and output features in the group dimension, where I represents the index of the filter's rotation angle and J represents the channel index of the feature map F. This indicates that the input features are expressed in the group dimension by Slices of indexes This indicates that the output features are expressed in the group dimension by Slices of indexes This represents the learnable weight matrix. Indicates the index of the matrix.

[0041] The above summation formula is equivalent to constructing a large weight matrix. Its structure is as follows: , In the formula, rows and columns correspond to geometric transformation groups. elements in ( (Assuming identity elements, i.e., no transformations are performed) Each matrix block Each of these is an independently learnable weight matrix, where each row is the result of a rightward cyclic shift of the previous row. Finally, from Output features are obtained by splicing .

[0042] like Figure 4As shown, this paper establishes a visual Mamba module with four-angle rotational equivariance. In the diagram, EQLinear represents the rotationally equivariant linear layer, the core linear transformation layer for achieving rotational equivariance. It ensures synchronous changes in output after input rotation through grouped weight sharing and cyclic shifting. EQ Cross Merge represents the rotationally equivariant cross-merge layer, used to fuse features from different branches under rotational equivariance constraints. 1D Selective Scan (Mamba1) represents a one-dimensional selective scan (Mamba1 version), the core module of the Mamba network, achieving long-sequence dependency modeling through selective scanning. EQ SS2D Block represents a rotationally equivariant state space two-dimensional block, a visual state space module integrating rotational equivariance characteristics. EQ MLP represents a rotationally equivariant multilayer perceptron, a stack of fully connected layers adapted to rotational equivariance constraints. Norm represents a normalization layer, usually referring to layer normalization (LayerNorm) or batch normalization (BatchNorm), used for stable training. EQ Cross Scan represents a rotationally equivariant cross-scan layer, performing multi-path scanning of features under rotational equivariance constraints. EQConv represents a rotationally equivariant convolutional layer, achieving rotational equivariance of the convolution operation through rotationally symmetric filters. SiLU represents a Sigmoid weighted linear unit, an activation function. nn.Parameter represents a learnable parameter class in PyTorch, used to define network parameters that need to be updated through training. X represents the input sequence data, Y represents the output feature tensor, and Batch, C, A, D, and W4 represent the learnable parameters of the Mamba network. Batch stands for batch size, C represents the number of feature channels in a token, and N represents the sequence length.

[0043] The Visual Mamba module is a rotationally equivariant visual state-space modeling unit. It expands input features into four rotational copies along the rotation group direction and introduces equivariant linear projection, equivariant depthwise separable convolution, and equivariant selective scanning mechanisms during state-space sequence modeling. This ensures that any spatial rotation operation maintains a strict group homomorphic mapping between the input and output feature maps. The Visual Mamba module employs a grouped four-fold parameter sharing strategy in the channel dimension, completing forward propagation in a five-dimensional tensor of Batch×C×4×H×W, where 4 represents the number of rotational copies, Batch represents the number of samples processed simultaneously in one propagation (i.e., batch), and C represents the number of feature channels of the token. All network layer weights are constructed using an out×4, in×4 grouping structure. Finally, equivariant normalization and residual connections achieve seamless embedding with standard visual backbone networks. This reduces the number of parameters while endowing the overall network with strict equivariance and robustness to 0°, 90°, 180°, and 270° planar rotations, making it suitable for highly stable visual recognition and detection tasks.

[0044] Step 105: Integrate the visual mamba modules to construct a visual state space model network, and train the visual state space model network using the training set until the training stopping condition is met. In this embodiment, the pre-feature extraction layer, four visual mamba layers, downsampling layer, and output layer are cascaded sequentially to obtain the visual state space model network; wherein, the visual mamba layer includes multiple visual mamba modules connected in sequence.

[0045] Specifically, a visual state space model network with four-angle rotation equivariance is constructed based on the visual Mamba module to extract effective semantic features from the input image. The constructed visual state space model network with four-angle rotation equivariance includes a pre-feature extraction layer, four visual Mamba layers with four-angle rotation equivariance, a downsampling layer, and an output layer.

[0046] Furthermore, the pre-feature extraction layer includes two isovariant convolutional layers (with kernel size of [missing information]). ), activation function layer and batch normalization layer, to process the input image (resolution is Downsampling and extraction yielded a feature map (resolution: The visual mamba layer with four-angle rotation equivariance includes multiple visual mamba modules with four-angle rotation equivariance, and rotationally equivariant convolutional layers (with kernel size of [missing information]). The network consists of an equivariant feedforward layer, an equivariant rotational position encoder, and an equivariant fully connected layer. The downsampling layer contains a rotationally equivariant convolutional layer (with a kernel size of [missing information - likely a value]). Step size is The output layer varies depending on the image processing task.

[0047] Among them, rotationally equivariant convolutional layers can be divided into input layer equivariant convolution, intermediate layer equivariant convolution, and output layer equivariant convolution. The input image is defined. The size is , For dimension The feature map is defined the same as that in the rotationally equivariant linear layer. Indicates the number of rotational isomorphisms (in a four-angle rotational isomorphism, ... ), , , Let represent the rotationally equivariant convolution filters of the input layer, intermediate layer, and output layer, respectively. Then, the rotationally equivariant convolution of the input layer can be expressed as: , In the formula, S represents the geometric transformation group consisting of all rotationally equivalent angles, containing t rotationally equivalent angles. and All The elements in the array serve as indices of the input and output features in the group dimension, where I represents the index of the filter's rotation angle and J represents the channel index of the feature map F. Represents equivariant convolution. Indicates the corresponding rotational constant angle The filter, i.e., the filter for the input image Using filters with different rotational angles for... Two-dimensional convolution operations are performed to obtain a dimension of Feature map .

[0048] Intermediate layer rotational isovariant convolution can be represented as: , In the formula, Representation of feature map of The corresponding rotational angle of the channel The filter, This indicates the output channel based on feature map F. and rotational angle The result obtained after cyclically shifting the feature map channels. This represents the geometric transformation group consisting of all rotationally equivalent angles, including... Each rotation is a uniform angle, i.e., the feature map obtained by uniform convolution of the input layer. Each channel is performed separately Convolution and summation of filters with different rotational angles yields a new dimension. Feature map .

[0049] The output layer rotational isovariant convolution can be represented as: , In the formula, Representation of feature map passage , Representation of feature map of The filter corresponding to each channel. That is, performing convolution operation on each channel of the feature map F and summing the results to obtain the final output.

[0050] By reconstructing the state space models VMamba and MambaIR of the Mamba network used for visual processing tasks, the scanning strategy and Mamba network modules within the state space model are replaced with a two-dimensional path scanning strategy with four-angle rotation equivariance and a visual Mamba module with four-angle rotation equivariance. Furthermore, the remaining convolutional network layers, fully connected layers, normalization layers, and Dropout layers are replaced with rotationally equivariant convolutional network layers, rotationally equivariant fully connected layers, rotationally equivariant normalization layers, and rotationally equivariant Dropout layers, thus establishing a visual state space model network with four-angle rotation equivariance suitable for various visual processing tasks. This network achieves state-of-the-art performance in various experiments and significantly reduces the number of parameters compared to the original Mamba network.

[0051] Step 106: Input the test set into the trained visual state space model network to obtain the processing result of the image task. In this embodiment, for different visual processing tasks, the pre-processed training set is input into the constructed visual state space model network. The set loss function is calculated based on the output of the visual state space model network and the annotation of the two-dimensional image tensors in the training set. Then, the gradient descent algorithm is used to update the network parameters of the visual state space model network and optimize the loss function until a preset number of iterations (e.g., 300 times) is reached, and the training step is terminated, thereby obtaining the optimized visual state space model network based on path mask attention.

[0052] In the image classification task, the classification performance of the proposed visual state-space model network (denoted as EQ-VMamba-T) is evaluated using the ImageNet100 dataset (a widely used subset of ImageNet-1K). The ImageNet-100 dataset consists of 100 object classes, each with corresponding training and validation images. Following the training settings of VMamba (visual state-space model), all models (or networks) were trained from scratch for 300 epochs using the AdamW optimizer, with a total batch size of 1024, an initial learning rate of 0.001, and a weight decay rate of 0.05%. To stabilize training, an exponential moving average (EMA) of the network parameters, consistent with VMamba, was used. During training, all input images were resized to 224px × 224px, and standard data augmentation strategies were employed, including color dithering, auto-enlargement, random erasure, blending, and CutMix (a data augmentation strategy). Experimental results are shown in Table 1.

[0053] Table 1 shows the experiments conducted on the ImageNet-100 dataset with five lightweight versions of the Mamba network-based state-space model for image classification. Param. indicates the number of parameters. Top-1 indicates the accuracy of the model (or network) in predicting the most probable class for a single test image, which is exactly equal to the image's true label. Top-5 indicates a relaxed judgment condition, where the model (or network) is considered to have correct accuracy as long as the true label appears in the top five most probable classes in its output. Bold text indicates the best result.

[0054] Table 1

[0055] As can be seen from the metrics in Table 1, the visual state space model network of this application has significantly better classification performance and more prominent advantages. Moreover, it has a significantly reduced number of parameters compared with models of the same level, and has higher training and running efficiency.

[0056] In the semantic segmentation task experiments, four natural image segmentation datasets (ADE20K, VOC2012, Cityscapes, COCO-Stuff) and two remote sensing image segmentation datasets (LoveDA, Potsdam) were used to evaluate the semantic segmentation performance of the visual state-space model network. MMSegmentation was used as the training framework, with training settings mirroring those provided by VMamba. The decoder network UpperNet used by VMamba was reconstructed into a decoder network EQ-UperNet with four-angle rotation equivariance. The experimental results are shown in Tables 2 and 3.

[0057] Table 2 shows the tests of the visual state-space model network and a simplified version of VMamba on four natural image segmentation datasets for semantic segmentation tasks. Param. indicates the number of parameters, and the evaluation metric is MIoU (mean intersection-union ratio). Bold text indicates the best results.

[0058] Table 2

[0059] Table 3 shows our tests of the visual state-space model network and a simplified version of VMamba on three remote sensing image segmentation datasets for semantic segmentation tasks. Param. represents the number of model parameters, and the evaluation metric is MIoU (%). Bold text indicates the best results.

[0060] Table 3

[0061] According to Tables 2 and 3, it can be seen that this application achieves better results than VMamba on multiple test sets, and the number of parameters is reduced by nearly 70%, which greatly reduces the computational burden.

[0062] In the super-resolution experiments, the performance of the visual state-space model network for classic image super-resolution was evaluated using a standard benchmark with bicubic degradation. Following the experimental setup of MambaIR, the model was trained on the DIV2K and Flickr2K datasets and evaluated on the Set5, Set14, B100, Urban100, and Manga109 datasets. For ×3 and ×4 super-resolution, the model was initialized with pre-trained weights from a ×2 model. During training, the same data augmentation, AdamW optimizer, batch size, number of training iterations, and L1 loss function as those used in MambaIR were employed. The input block size for training was fixed. The initial learning rate is set to... The training mileage is halved upon reaching a predefined training mileage. The experimental results are shown in Tables 4, 5, and 6.

[0063] Table 4 shows our 2x super-resolution tests on 4 test models across 5 common datasets. Param. indicates the number of model parameters, PSNR represents the peak signal-to-noise ratio calculated based on mean squared error, reflecting the similarity between the reconstructed image and the reference high-resolution image, SSIM represents the structural similarity index, and bold indicates the best result.

[0064] Table 4

[0065] Table 5 shows the super-resolution tests of the four test models on five general datasets at a multiplier of 3. Param. represents the number of model parameters, PSNR represents the mean squared error between the reconstructed image and the reference high-resolution image, and SSIM represents the structural similarity index. Bold text indicates the best results.

[0066] Table 5

[0067] Table 6 shows the super-resolution tests of the four test models on five common datasets at a multiplier of 4. Param. represents the number of model parameters, PSNR represents the mean squared error between the reconstructed image and the reference high-resolution image, SSIM represents the structural similarity index, and bold indicates the best result.

[0068] Table 6

[0069] Based on Tables 4, 5 and 6, it can be seen that this application demonstrates excellent performance on super-resolution tasks of multiple datasets at different scales, and achieves a significant reduction in the number of parameters.

[0070] In the Gaussian color image denoising experiments, DIV2K, Flickr2K, BSD500, and WEB were used as training datasets. Test datasets for Gaussian color image denoising included BSD68, Kodak24, McMaster, and Urban100. The experimental setup followed Mambair's setup. The experimental results are shown in Table 7.

[0071] Table 7 shows the Gaussian color denoising experiments conducted on eight test models on three common datasets with noise levels of 15, 25, and 50. Param. represents the number of model parameters. The noise level is indicated by PSNR, the experimental metric is the peak signal-to-noise ratio, and bold indicates the best result.

[0072] Table 7

[0073] As can be seen from Table 7, this application achieved good performance in tests with noise levels of 15, 25, and 50.

[0074] This application embeds rotational equivariance into the state-space model of Mamba networks. Through comprehensive validation and evaluation, this application verifies the effectiveness of embedding rotationally equivariant image priors into neural networks for Mamba networks. Since equivariant convolutions can share similar rotationally symmetric structures, the method in this application can achieve superior results in visual processing tasks such as image classification, denoising, and semantic segmentation with fewer parameters. By comparing with current mainstream image processing network models, this application demonstrates better image processing performance and lower parameter count on different settings and datasets, showcasing its superior performance and good generalization ability.

[0075] While this application provides the method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in this embodiment is merely one possible execution order among many and does not represent the only execution order. In actual device or client product execution, the methods shown in this embodiment or the accompanying drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0076] like Figure 5As shown in the figure, this application embodiment also provides a Mamba network optimization device 500 based on rotationally variable depth image priors. The device includes: a data processing module 501, an image conversion module 502, a data modeling module 503, a module construction module 504, a network construction and training module 505, and a testing module 506, as detailed below.

[0077] The data processing module 501 is used to convert raw image data from different vision processing tasks into two-dimensional image tensors and divide them into training and testing sets.

[0078] The image conversion module 502 is used to convert two-dimensional image tensors in the training set and the test set into sequence data through a two-dimensional path scanning strategy with four-angle rotational equivariance.

[0079] The data modeling module 503 is used to construct the data modeling expression of the Mamba network for sequence data.

[0080] Module 504 is used to construct a visual mamba module with four-angle rotational isovariance based on data modeling expressions.

[0081] The network construction and training module 505 is used to integrate the visual mamba module to build a visual state space model network and train the visual state space model network using the training set until the training stopping condition is met.

[0082] The test module 506 is used to input the test set into the trained visual state space model network to obtain the processing results of the image task.

[0083] Some modules in the apparatus described in this application can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0084] The apparatus or module described in the above embodiments can be implemented by a computer chip or physical entity, or by a product with a certain function. For ease of description, the above apparatus is described by dividing it into various modules according to their functions. When implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0085] The methods, apparatus, or modules described in this application can be implemented in a computer-readable program code manner. The controller can be implemented in any suitable manner, such as a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of a memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code manner, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included within it for implementing various functions can also be considered as structures within the hardware component. Alternatively, the device used to implement various functions can be viewed as either a software module that implements the method or a structure within a hardware component.

[0086] This application also provides an apparatus, the apparatus comprising: a processor; a memory for storing processor-executable instructions; wherein, when the processor executes the executable instructions, it implements the method described in this application.

[0087] This application also provides a non-volatile computer-readable storage medium storing a computer program or instructions thereon, which, when executed, enables the method described in this application embodiment to be implemented.

[0088] Furthermore, in the various embodiments of this application, each functional module can be integrated into one processing module, or each module can exist independently, or two or more modules can be integrated into one module.

[0089] The aforementioned storage media include, but are not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), Cache, Hard Disk Drive (HDD), or Memory Card. The memory can be used to store computer program instructions.

[0090] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, or it can be embodied in the process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0091] The various embodiments described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. All or part of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0092] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of this application.

Claims

1. A Mamba network optimization method based on rotationally variable depth image priors, characterized in that, include: Raw image data from different visual processing tasks are converted into two-dimensional image tensors and divided into training and testing sets. The two-dimensional image tensors in the training set and the test set are converted into sequence data using a two-dimensional path scanning strategy with four-angle rotational equivariance. Constructing a data modeling expression for sequence data using a Mamba network; A visual Mamba module with four-angle rotational isovariance is constructed based on the data modeling expression; The visual mamba module is integrated to construct a visual state space model network, and the visual state space model network is trained using the training set until the training stopping condition is met. The test set is input into the trained visual state space model network to obtain the processing results of the image task.

2. The method according to claim 1, characterized in that, Before converting the raw image data from different visual processing tasks into two-dimensional image tensors, the process also includes: Clean the raw image data to remove invalid image data; and / or, Standardize the resolution and number of channels of the raw image data; and / or, Data augmentation is performed on the original image data to enhance data diversity.

3. The method according to claim 1, characterized in that, The two-dimensional path scanning strategy, which uses four-angle rotational isomorphism to convert the two-dimensional image tensors in the training set and the test set into sequence data, includes: Four mutually rotationally symmetrical scanning paths are constructed, and the two-dimensional image tensor is scanned in blocks using the four scanning paths to divide the two-dimensional image tensor into multiple image blocks, which are then arranged sequentially to obtain four independent rotating image block groups. The four groups of rotated image blocks are each converted into one-dimensional sequence data.

4. The method according to claim 1, characterized in that, The data modeling expression for sequence data constructed by the Mamba network includes: Based on the state-space model paradigm of Mamba networks, a cyclic recursive expression for sequence data processing is constructed. The recursive expression is transformed into an attention expression containing a structured mask matrix to associate the state-space model with the attention mechanism; wherein the value of each element in the structured mask matrix is ​​determined by the product of decay factors.

5. The method according to claim 4, characterized in that, The recursive expression is as follows: ;in, ; The attention expression is as follows: ;in, ; In the formula, Indicates the first position of the output sequence One token, Indicates the first There are 10 hidden variables, where the initial hidden variables are set as a zero vector. , , Represents the learnable network parameters, where, Indicates the attenuation factor. Indicates the input projection matrix. Indicates the output projection matrix. This represents the first of the input sequence data. One token, This represents the number of feature channels of the token. Indicates the dimension of the latent variable. This represents the output feature tensor. This represents the learnable network parameter matrix. This indicates the transpose operation. Represents a structured mask matrix. Indicates the length of the sequence data. This represents the input sequence data. Indicates the position index. This indicates the column number of the elements in the structured mask matrix. It represents the Hadamardi (or Hadama) stack.

6. The method according to claim 1, characterized in that, The construction of a visual Mamba module with four-angle rotational equivariance based on the data modeling expression includes: The network parameters of the Mamba network are reconstructed based on the four-angle rotation isovariability. Rotationally equivariant linear layers are used as feature extraction layers in the Mamba network to compute network parameters in the Mamba network while sharing network parameters. The remaining network layers in the Mamba network are transformed into network layers with rotational equivariance to ensure that the output of each network layer changes synchronously with the rotation / rotation of the input feature map, resulting in a visual Mamba module with four-angle rotational equivariance.

7. The method according to claim 1, characterized in that, The integration of the visual Mamba module to construct a visual state-space model network includes: The visual state space model network is obtained by cascading the pre-feature extraction layer, four visual Mamba layers, downsampling layer, and output layer in sequence; wherein, the visual Mamba layer includes multiple visual Mamba modules connected in sequence.

8. A Mamba network optimization device based on rotationally variable depth image prior for implementing the method of any one of claims 1-7, characterized in that, include: The data processing module is used to convert raw image data from different vision processing tasks into two-dimensional image tensors and divide them into training and testing sets. The image conversion module is used to convert the two-dimensional image tensors in the training set and the test set into sequence data through a two-dimensional path scanning strategy with four-angle rotational equivariance. The data modeling module is used to construct data modeling expressions for sequence data using the Mamba network; The module construction module is used to construct a visual Mamba module with four-angle rotational equivariance based on the data modeling expression; The network construction and training module is used to integrate the visual mamba module to construct a visual state space model network, and to train the visual state space model network using the training set until the training stopping condition is met. The testing module is used to input the test set into the trained visual state space model network to obtain the processing results of the image task.

9. An apparatus for performing a Mamba network optimization method based on rotationally variable depth image priors, characterized in that, include: processor; Memory used to store processor-executable instructions; When the processor executes the executable instructions, it implements the method as described in any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium, characterized in that, Includes storage of computer programs or instructions that, when executed, cause the method as described in any one of claims 1 to 7 to be implemented.

Images