An image classification method based on improved Mamba

Abstract

The invention discloses a picture classification method based on improved Mamba, comprising obtaining an input image to be classified; generating a feature block matrix; extracting and obtaining four downsampled bidirectional feature maps of different scales layer by layer; gradually restoring the image resolution to obtain the upsampled bidirectional feature maps corresponding to the four different scales; gradually restoring a high-resolution feature map with the same resolution as the input image; a multi-layer perception module performs feature mapping on the high-resolution feature map according to the category feature blocks to generate a probability distribution of each category; classifying the input image to be classified according to the probability distribution of each category, and outputting an image classification result; the invention extracts bidirectional feature maps of different scales layer by layer through a progressive downsampling operation of an encoder, thereby retaining the global context information of the image and effectively extracting local detail features, thereby ensuring the integrity and expression ability of multi-scale features.

Description

Technical Field

[0001] The present invention relates to a picture classification method, in particular to a picture classification method based on improved Mamba. Background Art

[0002] With the rapid development of deep learning, the Transformer model has achieved great success in the field of natural language processing (NLP) and has gradually expanded to the field of computer vision, especially in image classification tasks. However, the traditional Vision Transformer (ViT) relies on a multi-head self-attention mechanism, and when processing high-resolution images, the computational complexity grows exponentially with the image size. In addition, the design of ViT does not have built-in inductive biases that are useful for visual tasks, such as translation invariance, like traditional convolutional neural networks, resulting in a high demand for large amounts of training data and complex data augmentation techniques. The patent document with patent publication number CN110443842B discloses a depth map prediction method based on perspective fusion, establishes a multi-perspective fusion depth map prediction network model, and predicts the depth information of a single-perspective color image.

[0003] However, the above patents and the prior art still have the problem of difficulty in capturing global semantic information due to high computational complexity. In order to solve these problems, the prior art proposes a state space model (SSM) called Mamba, which effectively captures global semantic information by reducing computational complexity. A key advantage of Mamba is that it has linear computational complexity, which means that even when processing larger or higher resolution images, its computational cost will not rise sharply.

[0004] However, the state-space model based on Mamba does not propose bidirectional feature extraction for corresponding image feature extraction, nor does it perform deep feature extraction on feature maps of multiple scales. The extracted features have not undergone feature conversion and abstraction of a multi-layer perceptron, making it difficult to reduce computational complexity while efficiently extracting image features. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides an image classification method based on an improved Mamba, which solves the technical problems raised in the background technology through bidirectional feature maps of multiple different scales.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0007] A picture classification method based on improved Mamba includes the following steps:

[0008] S1. Obtain an input image to be classified;

[0009] S2, performing preprocessing on the input image to be classified to generate a feature block matrix;

[0010] S3, inputting the feature block matrix into the encoder, performing a step-by-step downsampling operation on it, extracting layer by layer and obtaining four downsampled bidirectional feature maps of different scales;

[0011] S4, inputting the four downsampled bidirectional feature maps of different scales into the decoder, and gradually restoring the image resolution through layer-by-layer upsampling operations to obtain upsampled bidirectional feature maps corresponding to four different scales;

[0012] S5, performing point-by-point convolution fusion on the upsampled bidirectional feature map and the downsampled bidirectional feature map of the corresponding scale, and ensuring the transmission of feature information through residual connection, and gradually restoring a high-resolution feature map with the same resolution as the input image;

[0013] S6, inputting the high-resolution feature map into a multi-layer perception module, which performs feature mapping on the high-resolution feature map according to the category feature block to generate a probability distribution for each category;

[0014] S7. Classify the input image to be classified according to the probability distribution of each category, and output the image classification result.

[0015] In some embodiments, preprocessing is performed on the input image to be classified to generate a feature block matrix, including:

[0016] S2-1, receiving the input image to be classified;

[0017] Among them, the image to be classified is RGB image of size; H and W represent the image height and width respectively, and C represents the number of channels;

[0018] S2-2, evenly dividing the image to be classified into a number of pixel area blocks;

[0019] S2-3, flattening the pixel area block into a one-dimensional vector;

[0020] S2-4, projecting the flattened one-dimensional vector into a vector space with a fixed dimension D to generate a first feature block;

[0021] The calculation formula for each first feature block is:

[0022] in, represents the i-th first feature block, W represents the learnable weight matrix, represents the i-th one-dimensional vector after flattening;

[0023] S2-5, adding a position code to each first feature block to generate a second feature block;

[0024] S2-6, inserting an additional category feature block at the beginning of a sequence composed of a plurality of second feature blocks, and representing the sequence containing the category feature blocks in a matrix form as the feature block matrix;

[0025] The expression of the feature block matrix is:

[0026] Among them, T represents the feature block matrix, which contains a category feature block and N second feature blocks.

[0027] In some embodiments, the feature block matrix is ​​input to an encoder, a step-by-step downsampling operation is performed on it, and four downsampled bidirectional feature maps of different scales are extracted layer by layer to obtain the following:

[0028] S3-1, normalizing the second feature block in the feature block matrix, and mapping its dimensions to two vector spaces by linear projection to generate feature vectors X and Z;

[0029] S3-2, performing forward feature modeling between the feature vector X and the feature vector Z to obtain a forward feature vector;

[0030] S3-3, inverting the feature vector X, simulating the reverse modeling of the input image, and obtaining a reverse feature vector;

[0031] S3-4, fusing the forward feature vector and the reverse feature vector to generate a third bidirectional feature;

[0032] S3-5, inputting the third bidirectional feature into a multi-layer perception module, performing nonlinear feature transformation, and generating a downsampled bidirectional feature map;

[0033] S3-6. After repeating S3-1 to S3-5 four times, the resolution of the reverse feature map is halved step by step to generate four downsampled bidirectional feature maps of different scales.

[0034] In some of the embodiments, forward feature modeling is performed between the feature vector X and the feature vector Z to obtain a forward feature vector, including:

[0035] S3-2-1, performing one-dimensional convolution on the feature vector X to extract a local feature vector;

[0036] S3-2-2, substituting the extracted local feature vector into the continuous state update equation of the state space model, and calculating its evolution parameter matrix A, input parameter matrix B and output parameter matrix C;

[0037] Among them, the matrix elements of the evolution parameter matrix A, the input parameter matrix B and the output parameter matrix C are the continuous evolution parameters, input parameters and output parameters respectively;

[0038] The continuous state update equation is as follows:

[0039]

[0040] In the continuous state update equation:

[0041] is the hidden state vector The time derivative of , which represents the rate of change of the hidden state vector over time;

[0042] is the hidden state vector, representing the state at time point t; where is an N-dimensional vector, , R is the real number field, N is the dimension;

[0043] is a local eigenvector, representing the input signal at time t; the local eigenvector is a 1-dimensional vector, that is,

[0044] is the output vector, which represents the output signal at time t; the output vector is a 1-dimensional vector, that is,

[0045] A is the evolution parameter matrix, which is used to describe the evolution of the hidden state vector over time; The matrix of

[0046] B is the input parameter matrix, which is used to transform the input local graph feature vector Projected into the hidden state vector space; it is The matrix of

[0047] C is the output parameter matrix used to transform the hidden state vector Projected into the output space; it is The matrix of .

[0048] S3-2-3. Discretize the continuous evolution parameters and input parameters to obtain a discrete evolution matrix containing the discretized parameters. and the discrete input matrix ;

[0049] S3-2-4, according to the discrete evolution matrix , discrete input matrix And the output parameter matrix C, perform state space modeling, and generate the forward eigenvector.

[0050] In some of the embodiments, according to the discrete evolution matrix , discrete input matrix And the output parameter matrix C, perform state space modeling, generate the forward eigenvector, including:

[0051] S3-2-4-1. Substitute the local feature vector into the discrete state update equation to obtain the updated hidden state vector;

[0052] The expression of the discrete state update equation is:

[0053]

[0054] represents the updated hidden state vector;

[0055] S3-2-4-2. Use the output parameter matrix C to map the updated hidden state vector to generate an initial forward feature vector;

[0056] S3-2-4-3. Use a nonlinear activation transformation on the initial forward feature vector to generate the forward feature vector.

[0057] In some embodiments, the forward feature vector and the reverse feature vector are fused to generate a third bidirectional feature, including:

[0058] S3-4-1, performing positive and negative feature fusion on the forward feature vector and the reverse feature vector to obtain a first bidirectional feature;

[0059] S3-4-2, performing linear projection on the fused first bidirectional feature to obtain a second bidirectional feature;

[0060] S3-4-3. Perform residual connection on the second bidirectional feature after linear projection and the second feature block input by the encoder to generate the third bidirectional feature.

[0061] In some embodiments, the four downsampled bidirectional feature maps of different scales are input to the decoder, and the image resolution is gradually restored through layer-by-layer upsampling operations to obtain upsampled bidirectional feature maps corresponding to four different scales, including:

[0062] S4-1, receiving four downsampled bidirectional feature maps that are gradually transferred from large to small scales;

[0063] S4-2. Starting from the lowest scale, perform deconvolution upsampling on the downsampled bidirectional feature map of each scale, double its resolution, and generate an upsampled bidirectional feature map.

[0064] In some of the embodiments, the upsampled bidirectional feature map is convolved and fused point by point with the downsampled bidirectional feature map of the corresponding scale, and the feature information is transmitted through residual connection to gradually restore a high-resolution feature map with the same resolution as the input image, including:

[0065] S5-1, after obtaining the upsampled bidirectional feature map each time, extract the downsampled bidirectional feature map output by the encoder corresponding to the current scale;

[0066] S5-2, performing point-by-point convolution feature aggregation on the upsampled bidirectional feature map and the downsampled bidirectional feature map output by the encoder of the corresponding scale to generate a point-by-point convolution aggregation feature map;

[0067] S5-3, adding the point-by-point convolution aggregation feature map and the upsampled bidirectional feature map through residual connection to generate the high-resolution feature map;

[0068] S5-4, using the high-resolution feature map generated in step S5-3 as the up-sampling input of the next scale, repeatedly performing up-sampling and point-by-point convolution feature fusion until the highest scale is processed and a high-resolution feature map with the same resolution as the input image is restored;

[0069] In some of the embodiments, the high-resolution feature map is input into a multi-layer perception module, and the multi-layer perception module performs feature mapping on the high-resolution feature map according to the category feature block to generate a probability distribution for each category;

[0070] S6-1, obtaining the high-resolution feature map;

[0071] S6-2, flattening the high-resolution feature map into a one-dimensional vector after feature extraction;

[0072] S6-3, inputting the one-dimensional vector after feature extraction into a multi-layer perception module, performing feature mapping, and extracting the final feature representation in the category feature block;

[0073] S6-4. Use the Softmax activation function to map the final feature representation in the category feature block into a category probability distribution.

[0074] In some embodiments, classifying the input image to be classified according to the probability distribution of each category and outputting the image classification result includes:

[0075] S7-1, obtain the category probability distribution output by the multi-layer perception module;

[0076] S7-2, perform a maximum operation on the category probability distribution to obtain a category index;

[0077] S7-3. Select the category index corresponding to the maximum probability value as the image classification result.

[0078] The present invention provides an image classification method based on improved Mamba, which has the following beneficial effects:

[0079] The present invention extracts bidirectional feature maps of different scales layer by layer through the progressive downsampling operation of the encoder, which not only retains the global context information of the image, but also effectively extracts local detail features, ensuring the integrity and expression ability of multi-scale features.

[0080] Specifically, the state space model (SSM) is used to model the feature vectors, effectively capturing the forward dependency of the input image. The feature vectors are inverted to simulate the reverse dependency, supplement the deficiencies in the forward features, and improve the comprehensiveness of feature expression. The forward and reverse features are fused to generate bidirectional features, effectively solving the problem of incomplete information when modeling unidirectional features.

[0081] Furthermore, in the decoder, the low-resolution bidirectional feature map is gradually restored to a high-resolution feature map through layer-by-layer upsampling, and point-by-point convolution fusion is performed with the downsampled bidirectional feature map of the corresponding scale at each layer, and the residual connection ensures the continuous transmission of information. This layer-by-layer restoration and multi-scale fusion ensures that the high-level features have strong semantic expression capabilities, while combining the underlying detail features to improve the accuracy of image classification.

[0082] In addition, the present invention fully utilizes the semantic expression ability of high-resolution feature maps through feature mapping and category prediction of the MLP classifier to achieve high-precision image classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 A schematic diagram of the steps of an image classification method based on improved Mamba according to the present invention;

[0084] Figure 2 A schematic diagram of a process for generating a forward feature vector according to the present invention;

[0085] Figure 3 This is a structural block diagram of a picture classification system based on improved Mamba according to the present invention; DETAILED DESCRIPTION

[0086] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0087] First, the prior art and related concepts involved in the embodiments of the present invention are described:

[0088] Multi-Layer Perceptron (MLP): A feed-forward neural network consisting of multiple fully connected layers (i.e., neurons in each layer are connected to all neurons in the next layer). Each neuron usually contains an activation function (such as ReLU, Sigmoid, or Tanh) to introduce nonlinear characteristics. The basic structure of MLP includes an input layer, one or more hidden layers, and an output layer. Mamba: A variant of the state space model (SSM) for efficient feature modeling, commonly used in tasks such as time series modeling and image modeling.

[0089] State Space Model: (SSM) is a mathematical model used to describe the evolution of the internal state of a dynamic system over time.

[0090] See also Figure 1-2 The present invention provides an image classification method based on improved Mamba, comprising the following steps:

[0091] S1. Obtain an input image to be classified;

[0092] S2, performing preprocessing on the input image to be classified to generate a feature block matrix;

[0093] Wherein, step S2 specifically includes:

[0094] S2-1, receiving the input image to be classified;

[0095] Among them, the image to be classified is Size RGB image; H and W represent the image height and width respectively, C represents the number of channels; the number of channels is generally 3, representing the three color channels of red, green, and blue.

[0096] S2-2, evenly dividing the image to be classified into a number of pixel area blocks; extracting local area features from the original image to be classified by evenly dividing the input image into pixel area blocks (patches) of fixed size.

[0097] S2-3, flattening the pixel area block into a one-dimensional vector;

[0098] For example, if the size of each pixel region block is , then the length of the flattened one-dimensional vector is .

[0099] S2-4, projecting the flattened one-dimensional vector into a vector space with a fixed dimension D to generate a first feature block;

[0100] The calculation formula for each first feature block is:

[0101] in, represents the i-th first feature block, W represents the learnable weight matrix, represents the i-th one-dimensional vector after flattening;

[0102] Exemplarily, the learnable weight matrix W performs a linear transformation on each flattened one-dimensional vector to map it from the original space to a vector space of a fixed dimension D, thereby generating a first feature block.

[0103] The expression of the learnable weight matrix is:

[0104] W represents the learnable weight matrix, represents the length of the flattened one-dimensional vector (the number of pixels in each pixel area block), D represents the fixed output vector space dimension, and R represents the real number field.

[0105] The features are unified to the same feature dimension through linear transformation of the weight matrix W to facilitate subsequent feature processing.

[0106] S2-5, adding a position code to each first feature block to generate a second feature block;

[0107] Among them, the position encoding is generated by static encoding or learnable parameters, and its generation process can be expressed as:

[0108] represents the i-th second feature block, represents the i-th position code;

[0109] S2-6, inserting an additional category feature block at the beginning of a sequence composed of a plurality of second feature blocks, and representing the sequence containing the category feature blocks in a matrix form as the feature block matrix;

[0110] The expression of the feature block matrix is:

[0111] Among them, T represents the feature block matrix, which contains a category feature block and N second feature blocks.

[0112] Specifically, this embodiment provides a unified and optimized feature input format for the encoder stage by converting the input image to be classified into a feature block matrix with spatial and category information, laying the foundation for subsequent feature extraction and classification tasks while ensuring that the global semantics and local features of the input image can be preserved.

[0113] S3, inputting the feature block matrix into the encoder, performing a step-by-step downsampling operation on it, extracting layer by layer and obtaining four downsampled bidirectional feature maps of different scales;

[0114] Wherein, step S3 specifically includes:

[0115] S3-1, normalizing the second feature block in the feature block matrix, and mapping its dimensions to two vector spaces by linear projection to generate feature vectors X and Z;

[0116] S3-2, performing forward feature modeling between the feature vector X and the feature vector Z to obtain a forward feature vector;

[0117] S3-3, inverting the feature vector X, simulating the reverse modeling of the input image, and obtaining a reverse feature vector;

[0118] S3-4, fusing the forward feature vector and the reverse feature vector to generate a third bidirectional feature;

[0119] S3-5, inputting the third bidirectional feature into a multi-layer perception module, performing nonlinear feature transformation, and generating a downsampled bidirectional feature map;

[0120] S3-6. After repeating S3-1 to S3-5 four times, the resolution of the reverse feature map is halved step by step to generate four downsampled bidirectional feature maps of different scales.

[0121] Specifically, this embodiment extracts multi-scale features of the image at each layer by encoding the feature block matrix and using a step-by-step downsampling operation. The encoding process gradually normalizes and projects the input feature block matrix into different vector spaces, and uses forward and reverse modeling to achieve bidirectional feature capture. Forward modeling captures the forward dependency information of the image, while reverse modeling supplements the backward dependency information, and enhances the global feature expression capability through bidirectional feature fusion. At the same time, the multi-layer perception module further performs nonlinear transformation on the fused features to generate a multi-scale downsampled bidirectional feature map. In each downsampling, the feature resolution is reduced step by step, and the semantic level of feature expression is gradually enhanced.

[0122] Through the step-by-step downsampling operation, the global and local information expression capabilities of the input image are retained while reducing the computational complexity. The combination of feature bidirectional modeling and nonlinear transformation enhances the ability to capture complex patterns and semantic information, providing technical support for the ultimate realization of image classification.

[0123] Furthermore, step S3-2 specifically includes:

[0124] S3-2-1, performing one-dimensional convolution on the feature vector X to extract a local feature vector;

[0125] Specifically, step S3-2-1 one-dimensional convolution refers to performing one-dimensional convolution on the feature vector X along the X-axis (row direction) to extract relational features in the local space; wherein the convolution kernel size is an adjustable parameter used to define the size of the local receptive field; and the local feature vector is output through the convolution operation.

[0126] S3-2-2, substituting the extracted local feature vector into the continuous state update equation of the state space model, and calculating its evolution parameter matrix A, input parameter matrix B and output parameter matrix C;

[0127] Among them, the matrix elements of the evolution parameter matrix A, the input parameter matrix B and the output parameter matrix C are the continuous evolution parameters, input parameters and output parameters respectively;

[0128] Specifically, the state space model SSM maps a one-dimensional function or sequence to an output through a hidden state vector. Using A as the evolution parameter matrix, B as the input parameter matrix, and C as the projection parameter matrix, its continuous state update equation is as follows:

[0129]

[0130] In the continuous state update equation:

[0131] is the hidden state vector The time derivative of , which represents the rate of change of the hidden state vector over time;

[0132] is the hidden state vector, representing the state at time point t; where is an N-dimensional vector, , R is the real number field, N is the dimension;

[0133] is a local eigenvector, representing the input signal at time t; the local eigenvector is a 1-dimensional vector, that is,

[0134] is the output vector, which represents the output signal at time t; the output vector is a 1-dimensional vector, that is,

[0135] A is the evolution parameter matrix, which is used to describe the evolution of the hidden state vector over time; The matrix of

[0136] B is the input parameter matrix, which is used to transform the input local graph feature vector Projected into the hidden state vector space; it is The matrix of

[0137] C is the output parameter matrix used to transform the hidden state vector Projected into the output space; it is The matrix of .

[0138] Further, Indicates that in the continuous state update equation, the hidden state vector Changes over time; it consists of two parts:

[0139] Represents the hidden state vector natural evolution in the absence of external input;

[0140] Represents the local feature vector of the external input For the hidden state vector The impact of

[0141] Indicates that in the continuous state update equation, the output vector and the hidden state vector The equation converts the hidden state vector into Project to the output space and get the output vector.

[0142] In summary, the state space model describes the evolution of the internal state of the system and the relationship between input and output. The evolution parameter matrix A describes the natural evolution of the state, the input parameter matrix B describes the impact of the input on the state; and the output parameter matrix C describes the projection of the state to the output.

[0143] S3-2-3, discretizing the continuous evolution parameters and input parameters to obtain a discrete evolution matrix A containing the discretized parameters and a discrete input matrix B;

[0144] Specifically, since the original state space model SSM calculates continuous values ​​related to time, but the system often calculates discrete values ​​consisting of 0-1, a time scale parameter is introduced to convert the continuous parameter matrices A and B into discrete parameter matrices through discretization rules.

[0145] Specifically, the discretization method may select the zero-order hold ZOH (Zero-Order Hold, ZOH) method for discretization, such as the following discretization equation:

[0146]

[0147] In the discretized equation:

[0148] is the discrete evolution parameter matrix, representing the discrete time step How does the hidden state vector transfer from one time point to the next?

[0149] is the time step In, the exponential function of the evolution parameter matrix A, where represents the time step. Exponential function Describes the system in time The internal state evolution;

[0150] is the discrete input matrix, representing the time step How the input vector affects the change of the state vector;

[0151] is the time step The inverse matrix of the internal evolution parameter matrix A, which is used to calculate the time step The discretized form of the internal input parameter matrix B;

[0152] Represents the exponential matrix Subtract the identity matrix I. The identity matrix I is a diagonal matrix with all 1s on the diagonal and all 0s elsewhere.

[0153] is the continuous input matrix B multiplied by the time step , which represents the influence of the input on the state within the time step.

[0154] Represents matrix multiplication

[0155] In the discretized equation, is the discrete evolution parameter matrix The calculation equation represents the continuous evolution parameter matrix A at the time step The discretized form of . Exponential function Describes the state at time The evolution within.

[0156] In the discretized equation, Represents the continuous input matrix B at time step The discretized form of .

[0157] The specific steps are as follows:

[0158] First calculate , and obtain the discrete evolution parameter matrix in time Evolution within;

[0159] Then calculate , we get the difference between the discrete evolution parameter matrix and the identity matrix;

[0160] Then calculate , get the inverse matrix;

[0161] Finally, the above results are compared with Multiply to get the discrete input matrix ;

[0162] In summary, the continuous state space model is converted to a discrete state space model for processing in a computer. The discretization process ensures that the model is continuous in discrete time steps. Accuracy and stability within.

[0163] S3-2-4, according to the discrete evolution matrix , discrete input matrix And the output parameter matrix C, perform state space modeling, and generate the forward eigenvector.

[0164] Furthermore, step S3-2-4 specifically includes:

[0165] S3-2-4-1. Substitute the local feature vector into the discrete state update equation to obtain the updated hidden state vector;

[0166] The expression of the discrete state update equation is:

[0167]

[0168] represents the updated hidden state vector;

[0169] S3-2-4-2. Use the output parameter matrix C to map the updated hidden state vector to generate an initial forward feature vector;

[0170] S3-2-4-3. Use a nonlinear activation transformation on the initial forward feature vector to generate the forward feature vector.

[0171] In summary, through the discretization matrix calculation and update of the state space model, the hidden state vector is effectively evolved within the time step, and through the output matrix mapping and nonlinear activation processing, a forward feature vector with expressive ability is generated.

[0172] Furthermore, step S3-4 specifically includes:

[0173] S3-4-1, performing positive and negative feature fusion on the forward feature vector and the reverse feature vector to obtain a first bidirectional feature;

[0174] S3-4-2, performing linear projection on the fused first bidirectional feature to obtain a second bidirectional feature;

[0175] S3-4-3. Perform residual connection on the second bidirectional feature after linear projection and the second feature block input by the encoder to generate the third bidirectional feature.

[0176] This embodiment fuses forward and reverse features and combines linear projection with residual connection operations to generate a third bidirectional feature that can better express the contextual information of the input data and effectively retain the key details of the original feature block in the feature transformation, thereby improving the integrity of the feature representation.

[0177] The picture classification method also includes:

[0178] S4, inputting the four downsampled bidirectional feature maps of different scales into the decoder, and gradually restoring the image resolution through layer-by-layer upsampling operations to obtain upsampled bidirectional feature maps corresponding to four different scales;

[0179] Wherein, step S4 specifically includes:

[0180] S4-1, receiving four downsampled bidirectional feature maps that are gradually transferred from large to small scales;

[0181] Specifically, the downsampled bidirectional feature maps of four different scales can be expressed as:

[0182]

[0183] S4-2. Starting from the lowest scale, perform deconvolution upsampling on the downsampled bidirectional feature map of each scale, double its resolution, and generate an upsampled bidirectional feature map.

[0184] Exemplarily, the deconvolution may be an upsampling method such as transposed convolution or bilinear interpolation.

[0185] Through layer-by-layer upsampling operations, the decoder can gradually restore the resolution of the input image, while gradually introducing multi-scale feature information to ensure that the image's detail information and global semantics are consistent during high-resolution reconstruction, providing high-quality upsampled feature maps for subsequent feature fusion and final classification.

[0186] The picture classification method also includes:

[0187] S5, performing point-by-point convolution fusion on the upsampled bidirectional feature map and the downsampled bidirectional feature map of the corresponding scale, and ensuring the transmission of feature information through residual connection, and gradually restoring a high-resolution feature map with the same resolution as the input image;

[0188] Furthermore, step S5 specifically includes:

[0189] S5-1. After each upsampled bidirectional feature map is obtained, a downsampled bidirectional feature map output by the encoder corresponding to the current scale is extracted; the downsampled bidirectional feature map provides multi-scale semantic information in the encoding stage, which is used to be fused with the current feature map in the upsampling stage.

[0190] S5-2. Aggregate the upsampled bidirectional feature map with the downsampled bidirectional feature map output by the encoder of the corresponding scale through point-by-point convolution to generate a point-by-point convolution aggregated feature map. Point-by-point convolution is used for aggregation and feature mapping of feature channels, which can fuse multi-scale features while maintaining spatial resolution to generate a point-by-point convolution aggregated feature map.

[0191] S5-3, adding the point-by-point convolution aggregation feature map and the upsampled bidirectional feature map through residual connection to generate the high-resolution feature map;

[0192] S5-4. Use the high-resolution feature map generated in step S5-3 as the upsampling input for the next scale, and repeat upsampling and point-by-point convolution feature fusion until the highest scale is processed and a high-resolution feature map with the same resolution as the input image is restored. This process proceeds upward step by step, gradually restoring the image resolution from the lowest scale until the highest scale is processed, and finally a high-resolution feature map with the same resolution as the input image is generated.

[0193] Specifically, the execution logic can be summarized as follows:

[0194] Level 1 (lowest scale):

[0195] Upsampled feature map + downsampled feature map → convolution aggregation → residual connection → high-resolution feature map.

[0196] Level 2:

[0197] The high-resolution feature map output by the first stage is taken as input and the upsampling and fusion operations are repeated.

[0198] …

[0199] Highest level (original input scale):

[0200] Perform final upsampling and feature fusion to obtain the final high-resolution feature map.

[0201] Through layer-by-layer upsampling and point-by-point convolution fusion, as well as the feature transfer mechanism of residual connection, combined with the multi-scale features of the encoder, the spatial resolution of the input image is gradually restored, and the semantic and detail information of different scales is retained, ensuring that the generated high-resolution feature map has high-quality expressiveness.

[0202] The picture classification method also includes:

[0203] S6, inputting the high-resolution feature map into a multi-layer perception module, which performs feature mapping on the high-resolution feature map according to the category feature block to generate a probability distribution for each category;

[0204] Wherein, step S6 specifically includes:

[0205] S6-1, obtaining the high-resolution feature map;

[0206] S6-2, flattening the high-resolution feature map into a one-dimensional vector after feature extraction;

[0207] S6-3, inputting the one-dimensional vector after feature extraction into a multi-layer perception module, performing feature mapping, and extracting the final feature representation in the category feature block;

[0208] S6-4. Use the Softmax activation function to map the final feature representation in the category feature block into a category probability distribution.

[0209] This embodiment achieves accurate probability prediction of different categories by inputting high-resolution feature maps into the multi-layer perception module and gradually processing them through flattening, feature mapping and Softmax normalization. The multi-layer perception module can extract and learn key category information in high-dimensional feature representation, while the Softmax activation function ensures the interpretability of the output results and provides accurate category probability distribution for subsequent image classification tasks.

[0210] The picture classification method also includes:

[0211] S7. Classify the input image to be classified according to the probability distribution of each category, and output the image classification result.

[0212] Wherein, step S7 specifically includes:

[0213] S7-1. Obtain the category probability distribution output by the multi-layer perception module; the category probability distribution is a vector in which each element represents the probability value of the image belonging to the corresponding category, and the sum of all category probabilities is 1.

[0214] S7-2. Perform a maximum operation on the category probability distribution to obtain the category index; the maximum operation ensures that the category with the highest probability is selected, reflecting the category label to which the image is most likely to belong

[0215] S7-3. Select the category index corresponding to the maximum probability value as the image classification result.

[0216] Through the above steps, the present invention determines the final category label of the input image based on the category probability distribution output by the multi-layer perception module. The maximum value operation ensures the uniqueness and certainty of the classification result, thereby outputting the image category that best matches the feature expression.

[0217] Specifically, this embodiment obtains an input image to be classified and performs preprocessing on the input image to generate a feature block matrix, thereby providing a basic data structure for feature extraction and classification; further, the encoder downsamples layer by layer to extract four bidirectional feature maps of different scales, and upsamples layer by layer in the decoder to restore the resolution, and combines the feature fusion process of the encoder and decoder to ultimately generate a high-resolution feature map; finally, the multi-layer perception module maps the high-resolution feature map to the category probability distribution and outputs the final classification result.

[0218] Embodiment 2: The technical solution of Embodiment 2 differs from that of Embodiment 1 in that a classification system for executing an improved Mamba-based image classification method is disclosed, the classification system comprising: an initialization block, a first Eamba block, a second Eamba block, a third Eamba block and a fourth Eamba block; wherein the initialization block is used to perform preprocessing on the input image to be classified to generate a feature block matrix; and the first Eamba block, the second Eamba block, the third Eamba block and the fourth Eamba block are used to obtain bidirectional feature maps after upsampling corresponding to four different scales.

[0219] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.) means.

[0220] The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., DVD ), or semiconductor media. The semiconductor media may be a solid state drive.

[0221] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division of a waterway underwater terrain change analysis system and method. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0222] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A picture classification method based on improved Mamba, characterized in that: include: S1. Obtain an input image to be classified; S2, performing preprocessing on the input image to be classified to generate a feature block matrix; S3, inputting the feature block matrix into the encoder, performing a step-by-step downsampling operation on it, extracting layer by layer and obtaining four downsampled bidirectional feature maps of different scales; S4, inputting the four downsampled bidirectional feature maps of different scales into the decoder, and gradually restoring the image resolution through layer-by-layer upsampling operations to obtain upsampled bidirectional feature maps corresponding to four different scales; S5, performing point-by-point convolution fusion on the upsampled bidirectional feature map and the downsampled bidirectional feature map of the corresponding scale, and ensuring the transmission of feature information through residual connection, and gradually restoring a high-resolution feature map with the same resolution as the input image; S6, inputting the high-resolution feature map into a multi-layer perception module, which performs feature mapping on the high-resolution feature map according to the category feature block to generate a probability distribution for each category; S7. Classify the input image to be classified according to the probability distribution of each category, and output the image classification result.

2. The image classification method based on improved Mamba according to claim 1, characterized in that: Preprocessing the input image to be classified to generate a feature block matrix, including: S2-1, receiving the input image to be classified; Among them, the image to be classified is RGB image of size; H and W represent the image height and width respectively, and C represents the number of channels; S2-2, evenly dividing the image to be classified into a number of pixel area blocks; S2-3, flattening the pixel area block into a one-dimensional vector; S2-4, projecting the flattened one-dimensional vector into a vector space with a fixed dimension D to generate a first feature block; The calculation formula for each first feature block is: in, represents the i-th first feature block, W represents the learnable weight matrix, represents the i-th one-dimensional vector after flattening; S2-5, adding a position code to each first feature block to generate a second feature block; S2-6, inserting an additional category feature block at the beginning of a sequence composed of a plurality of second feature blocks, and representing the sequence containing the category feature blocks in a matrix form as the feature block matrix; The expression of the feature block matrix is: Among them, T represents the feature block matrix, which contains a category feature block and N second feature blocks.

3. The image classification method based on improved Mamba according to claim 1, characterized in that: The feature block matrix is ​​input into the encoder, and a step-by-step downsampling operation is performed on it, and four downsampled bidirectional feature maps of different scales are extracted layer by layer, including: S3-1, normalizing the second feature block in the feature block matrix, and mapping its dimensions to two vector spaces by linear projection to generate feature vectors X and Z; S3-2, performing forward feature modeling between the feature vector X and the feature vector Z to obtain a forward feature vector; S3-3, inverting the feature vector X, simulating the reverse modeling of the input image, and obtaining a reverse feature vector; S3-4, fusing the forward feature vector and the reverse feature vector to generate a third bidirectional feature; S3-5, inputting the third bidirectional feature into a multi-layer perception module, performing nonlinear feature transformation, and generating a downsampled bidirectional feature map; S3-6. After repeating S3-1 to S3-5 four times, the resolution of the reverse feature map is halved step by step to generate four downsampled bidirectional feature maps of different scales.

4. The image classification method based on improved Mamba according to claim 3, characterized in that: Perform forward feature modeling between feature vector X and feature vector Z to obtain a forward feature vector, including: S3-2-1, performing one-dimensional convolution on the feature vector X to extract a local feature vector; S3-2-2, substituting the extracted local feature vector into the continuous state update equation of the state space model, and calculating its evolution parameter matrix A, input parameter matrix B and output parameter matrix C; Among them, the matrix elements of the evolution parameter matrix A, the input parameter matrix B and the output parameter matrix C are the continuous evolution parameters, input parameters and output parameters respectively; The continuous state update equation is as follows: In the continuous state update equation: is the hidden state vector The time derivative of , which represents the rate of change of the hidden state vector over time; is the hidden state vector, representing the state at time point t; where is an N-dimensional vector, , R is the real number field, N is the dimension; is a local eigenvector, representing the input signal at time t; is the output vector, representing the output signal at time t; A is the evolution parameter matrix, which is used to describe the evolution of the hidden state vector over time; B is the input parameter matrix, which is used to transform the input local graph feature vector Projection into hidden state vector space; C is the output parameter matrix used to transform the hidden state vector Project to output space; S3-2-3. Discretize the continuous evolution parameters and input parameters to obtain a discrete evolution matrix containing the discretized parameters. and the discrete input matrix ; S3-2-4, according to the discrete evolution matrix , discrete input matrix And the output parameter matrix C, perform state space modeling, and generate the forward eigenvector.

5. The image classification method based on improved Mamba according to claim 3, characterized in that: According to the discrete evolution matrix , discrete input matrix And the output parameter matrix C, perform state space modeling, generate the forward eigenvector, including: S3-2-4-1. Substitute the local feature vector into the discrete state update equation to obtain the updated hidden state vector; The expression of the discrete state update equation is: represents the updated hidden state vector; S3-2-4-2. Use the output parameter matrix C to map the updated hidden state vector to generate an initial forward feature vector; S3-2-4-3. Use a nonlinear activation transformation on the initial forward feature vector to generate the forward feature vector.

6. The image classification method based on improved Mamba according to claim 3, characterized in that: The forward feature vector and the reverse feature vector are fused to generate the third bidirectional feature, including: S3-4-1, performing positive and negative feature fusion on the forward feature vector and the reverse feature vector to obtain a first bidirectional feature; S3-4-2, performing linear projection on the fused first bidirectional feature to obtain a second bidirectional feature; S3-4-3. Perform residual connection on the second bidirectional feature after linear projection and the second feature block input by the encoder to generate the third bidirectional feature.

7. The image classification method based on improved Mamba according to claim 1, characterized in that: The four downsampled bidirectional feature maps of different scales are input to the decoder, and the image resolution is gradually restored through layer-by-layer upsampling operations to obtain the upsampled bidirectional feature maps corresponding to the four different scales, including: S4-1, receiving four downsampled bidirectional feature maps that are gradually transferred from large to small scales; S4-2. Starting from the lowest scale, perform deconvolution upsampling on the downsampled bidirectional feature map of each scale, double its resolution, and generate an upsampled bidirectional feature map.

8. The image classification method based on improved Mamba according to claim 1, characterized in that: The upsampled bidirectional feature map is convolved and fused with the downsampled bidirectional feature map of the corresponding scale point by point, and the feature information is transmitted through residual connection to gradually restore the high-resolution feature map with the same resolution as the input image, including: S5-1, after obtaining the upsampled bidirectional feature map each time, extract the downsampled bidirectional feature map output by the encoder corresponding to the current scale; S5-2, performing point-by-point convolution feature aggregation on the upsampled bidirectional feature map and the downsampled bidirectional feature map output by the encoder of the corresponding scale to generate a point-by-point convolution aggregated feature map; S5-3, adding the point-by-point convolution aggregation feature map and the upsampled bidirectional feature map through residual connection to generate the high-resolution feature map; S5-4. Use the high-resolution feature map generated in step S5-3 as the upsampling input of the next scale, and repeat the upsampling and point-by-point convolution feature fusion until the highest scale is processed and a high-resolution feature map with the same resolution as the input image is restored.

9. The image classification method based on improved Mamba according to claim 1, characterized in that: The high-resolution feature map is input into a multi-layer perception module, and the multi-layer perception module performs feature mapping on the high-resolution feature map according to the category feature block to generate a probability distribution of each category; S6-1, obtaining the high-resolution feature map; S6-2, flattening the high-resolution feature map into a one-dimensional vector after feature extraction; S6-3, inputting the one-dimensional vector after feature extraction into a multi-layer perception module, performing feature mapping, and extracting the final feature representation in the category feature block; S6-4. Use the Softmax activation function to map the final feature representation in the category feature block into a category probability distribution.

10. The image classification method based on improved Mamba according to claim 1, characterized in that: According to the probability distribution of each category, the input image to be classified is classified, and the image classification result is output, including: S7-1, obtain the category probability distribution output by the multi-layer perception module; S7-2, perform a maximum operation on the category probability distribution to obtain a category index; S7-3. Select the category index corresponding to the maximum probability value as the image classification result.

Images