A texture image classification method based on a Vision Mamba double-branch cross-fusion network

Abstract

This invention proposes a texture image classification method based on a Vision Mamba two-branch cross-fusion network. The method involves inputting the texture image to be classified into a trained Vision Mamba two-branch cross-fusion network model to obtain the corresponding classification result. The Vision Mamba two-branch cross-fusion network model is trained through the following steps: obtaining a texture image dataset; training the Vision Mamba two-branch cross-fusion network model using the texture image dataset; the Vision Mamba two-branch cross-fusion network model includes a feature extractor, a spatial domain branch, a frequency domain branch, a cross-attention feature fusion module, and a classifier. The feature extractor extracts high-level feature maps of the texture image; the spatial and frequency domain branches process the high-level feature maps to obtain spatial and frequency feature maps; the cross-attention feature fusion module fuses the spatial and frequency feature maps to obtain a fused feature map; and the classifier determines the category of the texture image. This invention can effectively predict the category of texture images.

Description

Technical Field

[0001] This invention belongs to the field of digital image analysis and image classification technology, and specifically relates to a texture image classification method based on the Vision Mamba dual-branch cross-fusion network. Background Technology

[0002] Texture, as a ubiquitous visual element in nature, is a key information carrier for identifying objects, analyzing scenes, and distinguishing materials. For example, in high-value applications such as remote sensing image analysis, medical image processing, and industrial defect detection, the accurate classification and recognition of texture images are crucial.

[0003] However, achieving efficient and accurate recognition and classification of texture images faces a series of inherent technical challenges. First, texture images possess complex randomness and internal structure, leading to significant differences in local appearance and global distribution even among similar textures. Second, environmental factors such as changes in illumination, viewing angle, and non-rigid deformation further exacerbate texture variability. Therefore, an ideal texture image recognition and classification method must strike a balance between high discriminative power and robustness to changes in environmental factors.

[0004] When using digital methods for texture image recognition and classification, traditional, hand-designed texture descriptors struggle to effectively distinguish complex or fine-grained texture patterns, exhibiting a bottleneck in discriminative power of feature representation. In recent years, deep learning, particularly convolutional neural networks (CNNs), has significantly improved texture classification accuracy due to its powerful data-adaptive feature learning capabilities. However, standard CNN architectures primarily focus on extracting spatial domain features. While CNNs can capture local structures, they often neglect frequency domain information crucial for capturing periodic texture patterns. Furthermore, traditional CNN structures and their subsequent simple global pooling operations struggle to effectively extract spatially invariant feature representations when processing texture images, often losing rich details of spatial layout.

[0005] To better capture long-range dependencies and global contextual information in images, the Visual Transformer (ViT) model and its self-attention mechanism have been introduced into the field of visual tasks, which were originally dominated by CNNs, and have attracted much attention due to their excellent global modeling capabilities. By dividing the image into blocks and calculating the attention between blocks, the ViT model can overcome the limitations of the local receptive field of CNNs. However, the core of the ViT model—the self-attention mechanism—has an inherent quadratic computational complexity. This makes the computational and memory requirements of the ViT model increase significantly when processing high-resolution or large-scale texture data, severely limiting its practical application in scenarios requiring real-time or high throughput. Therefore, exploring a new architecture that can maintain the ability to model long-range dependencies while achieving higher computational efficiency has become a key problem that urgently needs to be solved in the field of texture recognition. Summary of the Invention

[0006] To address at least one of the problems existing in the prior art, this invention provides a texture image classification method based on the Vision Mamba dual-branch cross-fusion network, which can effectively predict the category of texture images.

[0007] To achieve the objectives of this invention, a texture image classification method based on the Vision Mamba two-branch cross-fusion network is proposed. The texture image to be classified is input into the trained Vision Mamba two-branch cross-fusion network model to obtain the corresponding classification result. The Vision Mamba two-branch cross-fusion network model is trained through the following steps: Obtain a texture image dataset, wherein the texture image dataset contains texture image data of two or more categories, and the texture image data includes texture images and corresponding category labels; A Vision Mamba dual-branch cross-fusion network model is constructed, which includes a feature extractor, a spatial domain branch, a frequency domain branch, a cross-attention feature fusion module, and a classifier. The feature extractor is used to extract high-level feature maps of the texture image. The spatial domain branch and the frequency domain branch are used to simultaneously process the high-level feature maps in the spatial domain and the frequency domain, respectively, to obtain spatial feature maps and frequency feature maps. The cross-attention feature fusion module is used to fuse the spatial feature map and the frequency feature map through a bidirectional cross-attention mechanism to obtain a fused feature map. The classifier is used to determine the category of the texture image based on the fused feature map. The Vision Mamba dual-branch cross-fusion network model was trained using a texture image dataset to optimize the model parameters.

[0008] The high-level feature map is processed in the spatial domain and frequency domain respectively by parallel connection of spatial domain branches and frequency domain branches.

[0009] The present invention also provides a texture image classification device based on the Vision Mamba dual-branch cross-fusion network.

[0010] The present invention also provides a computer device.

[0011] The present invention also provides a computer-readable storage medium.

[0012] Compared with existing technologies, the present invention has the following features and advantages: (1) The texture image classification method proposed in this invention is based on the Vision Mamba dual-branch cross-fusion network. Through the dual-branch cross-fusion network architecture based on Vision Mamba (ViM), the method achieves comprehensive and collaborative extraction and utilization of global long-range dependence, local spatial complexity and frequency domain structure information of texture images. The spatial domain branch refines the spatial characteristics of texture through multi-scale feature extraction, local differential box counting and Gaussian soft histogram statistics. The frequency domain branch explicitly enhances the frequency domain structure of texture by means of Haar wavelet transform and sub-band independent processing. Then, the two domain features are deeply fused and complemented by a bidirectional cross-attention mechanism. Thus, the classification accuracy of texture images is significantly improved while maintaining linear computational complexity. It also shows excellent robustness to complex environmental factors such as deformation, illumination change and scale change, enabling the model to quickly output prediction results. It is suitable for high-value application scenarios rich in texture, such as medical image analysis, remote sensing image classification, industrial defect detection and geological material identification.

[0013] (2) This invention proposes an image classification method that fully utilizes the global long-range dependence, local spatial complexity and frequency domain structure information of texture for the classification of texture images in real scenes. This method can predict the category of texture-rich images such as lesion areas in medical images, scene images, geological material images, and remote sensing images. It is robust to complex environmental changes such as deformation, illumination changes and scale changes, and is suitable for real-world scenarios.

[0014] (3) The Vision Mamba dual-branch cross-fusion network model proposed in this invention can embed the ability to describe various statistical features of texture images (including long-range dependency, spatial complexity, and frequency domain decomposition) end-to-end into the training and learning process of the network. It can fully utilize the knowledge of the dataset, automatically learn and optimize the feature extractor, spatial and frequency domain branch structures, and hyperparameters in the bidirectional cross-attention fusion module. Compared with other deep learning methods, the introduction of the spatial domain branch for fine characterization of local differential features and the explicit enhancement of structural information by the frequency domain branch enable the model to effectively classify texture image categories in various real-world scenarios and achieve higher classification accuracy.

[0015] (4) Compared with traditional texture image classification algorithms or ViT-based methods, this invention has lower time complexity. By adopting the ViM architecture as a feature extractor, it maintains strong global long-range dependency modeling capabilities and avoids the efficiency bottleneck caused by the quadratic computational complexity of traditional ViT models. The network model trained by this invention can quickly obtain prediction results, while traditional methods need to calculate local features based on determined local feature descriptors, which requires higher time complexity. Attached Figure Description

[0016] Figure 1 This is an overall flowchart of an embodiment of the present invention.

[0017] Figure 2 This is a diagram of the Vision Mamba dual-branch cross-fusion network structure in an embodiment of the present invention.

[0018] Figure 3 This is a structural diagram of the multi-layer feature aggregation module in an embodiment of the present invention.

[0019] Figure 4 This is a spatial domain branch structure diagram in an embodiment of the present invention.

[0020] Figure 5 This is a frequency domain branch structure diagram in an embodiment of the present invention.

[0021] Figure 6 This is a schematic diagram of the texture image classification method in an embodiment of the present invention. Detailed Implementation

[0022] The details of the present invention can be more clearly understood by referring to the accompanying drawings and the description of specific embodiments. However, the specific embodiments of the present invention described herein are for illustrative purposes only and should not be construed as limiting the invention in any way. Under the teachings of this invention, those skilled in the art can conceive of any possible modifications based on the invention, and these should all be considered to fall within the scope of the invention.

[0023] like Figure 6 As shown in the figure, an embodiment of the present invention provides a texture image classification method based on a Vision Mamba dual-branch cross-fusion network, which includes the following steps: Step 1: Obtain the texture image dataset. The texture image dataset contains texture image data of two or more categories. The texture image data includes texture images and their corresponding category labels. Step 2: Construct the Vision Mamba dual-branch cross-fusion network model. The Vision Mamba dual-branch cross-fusion network model extracts the high-level feature map of the input texture image, performs spatial domain processing and frequency domain processing on the high-level feature map simultaneously to obtain spatial feature map and frequency feature map respectively; fuse the spatial feature map and frequency feature map through a bidirectional cross-attention mechanism, and determine the category of the texture image based on the fusion result. Step 3: Train the Vision Mamba dual-branch cross-fusion network model using a texture image dataset to optimize the model parameters; Step 4: Input the texture image to be classified into the trained Vision Mamba dual-branch cross-fusion network model to obtain the corresponding classification result.

[0024] In one embodiment of the present invention, an end-to-end Vision Mamba (Selective Structured State Space Model for Computer Vision Tasks) dual-branch cross-fusion network model is constructed. The Vision Mamba dual-branch cross-fusion network model includes a feature extractor, a spatial domain branch, a frequency domain branch, a cross-attention feature fusion module, and a fully connected classifier. The model takes a three-channel RGB texture image as input and uses the high-level feature map output by the feature extractor based on the ViM pre-trained model as the basis of the entire model. These high-level feature maps are then fed into the spatial domain branch and the frequency domain branch in parallel and independently for processing. After processing by the two branches, feature maps of the same size are output and then coupled into the subsequent cross-attention feature fusion module to achieve deep feature fusion. Finally, the fully connected layer and the Softmax function are used to map the prediction probability vectors of the same number of classes as the training dataset, where the index corresponding to the vector element with the higher probability is the predicted class.

[0025] In one embodiment of the present invention, the texture image dataset is divided into non-overlapping segments according to a preset ratio: The training set is used to train the Vision Mamba two-branch cross-fusion network model; The test set is used to evaluate the performance of the trained Vision Mamba two-branch cross-fusion network model.

[0026] Specifically, the training set is dedicated to end-to-end training of the constructed Vision Mamba two-branch cross-fusion network model. Through iterative optimization of model parameters, the Vision Mamba two-branch cross-fusion network model learns the mapping relationship from image features to class labels. The test set is retained independently of the training process. After model training is complete, it is used as input to the trained Vision Mamba two-branch cross-fusion network model to objectively evaluate its performance metrics such as classification accuracy, thereby verifying the generalization ability and final effect of the Vision Mamba two-branch cross-fusion network model. By systematically dividing the texture image dataset into training and test sets, the objectivity and effectiveness of the model learning process and the final performance evaluation are ensured, truly reflecting the classification ability of the trained Vision Mamba two-branch cross-fusion network model on unseen texture images and effectively avoiding overfitting.

[0027] In one embodiment of the present invention, the texture image dataset is a large number of multi-class (two or more) texture image datasets to be trained, containing multiple texture image samples of corresponding classes and their corresponding class labels.

[0028] In one embodiment of the invention, a publicly available dataset of natural scene texture images with a large data size was selected for the training dataset, which was captured by a mobile phone device camera.

[0029] In one embodiment of the present invention, the texture image is preprocessed, the preprocessing including: Adjust the size of the texture images to the first preset size; Randomly crop out image blocks of a second preset size from the adjusted texture image; Perform random horizontal flipping operations on image patches with preset probabilities; The pixel values ​​of each image block are normalized to a preset range.

[0030] Specifically, images are first randomly selected from the training set. To improve the network's robustness to different images, the present invention also resizes the input images and unifies them to the same size, i.e., a first preset size.

[0031] Preferably, the first preset size is 256×256.

[0032] Preferably, the texture image of the first preset size is then randomly cropped to obtain an image block of the second preset size.

[0033] Preferably, the second preset size block is 224×224.

[0034] Preferably, the preset probability is 50%, that is, the image is randomly horizontally flipped with a 50% probability.

[0035] Preferably, the preset numerical range is [0,1], and the image pixel values ​​are normalized to [0,1] before being input into the network for calculation.

[0036] In the foregoing embodiments, the above series of preprocessing operations are executed sequentially. Image resizing and random cropping together expand the diversity of training samples. Random horizontal flipping enhances the model's adaptability to changes in texture direction by introducing mirror transformation. Pixel value normalization unifies data distribution and accelerates network convergence.

[0037] In this embodiment of the invention, preprocessing of texture images effectively expands the diversity of training data, simulating scale, position, and orientation variations that may occur in real-world scenes. This helps improve the adaptability and robustness of subsequent network models to these common interference factors. Simultaneously, normalization ensures that the input data remains within a stable numerical range, which is beneficial for stable convergence and improved generalization performance during model training.

[0038] In one embodiment of the present invention, when training the Vision Mamba two-branch cross-fusion network model, the texture images of the training set are input into the network model in batches to obtain the predicted class probability vectors. Then, the one-hot encoded vectors corresponding to the true classes of the predicted vectors are compared, and the cross-entropy loss function is calculated. The gradient is calculated based on the cross-entropy loss function, and the model parameters are updated using gradient backpropagation and gradient descent. Through multiple iterations, the network model can learn model parameters that result in increasingly accurate classification.

[0039] In one embodiment of the present invention, the high-level feature map of the input texture image is extracted by a feature extractor based on the VisionMamba architecture in the VisionMamba dual-branch cross-fusion network model. The execution steps include: Convert the texture image into sequence features and perform positional encoding; The sequence features are processed by multiple isomorphic Mamba modules to extract high-level features of long-range dependencies. High-level features of long-range dependencies output by multiple isomorphic Mamba modules are integrated to generate a high-level feature map.

[0040] Specifically, feature extraction is performed based on the Vision Mamba (ViM) architecture. First, the input texture image is transformed into sequential features through patch embedding. These sequential features then flow through 24 isomorphic Mamba modules (MambaBlocks). These Mamba modules efficiently extract high-level features with long-range dependencies through linear mapping layers, one-dimensional convolutional layers, and forward and backward spatial state models. The forward and backward spatial state model branches essentially perform forward and backward spatial state scans; the forward spatial state model scans the sequence in the forward direction, while the backward spatial state model performs a backward scan.

[0041] The forward and backward state space models are essentially bidirectional state space scanning mechanisms. Specifically, the forward state space model performs a forward traversal scan of the feature sequence from beginning to end, while the backward state space model performs a backward traversal scan of the feature sequence from end to beginning. Both routes are embedded within the Mamba module structure. This model is existing technology and will not be elaborated upon here.

[0042] In an optional example, using the ViM-S (Vision Mamba-Small) architecture, the backbone network of the feature extractor consists of 24 Mamba modules. The input texture image is first processed through patch embedding and positional encoding, and then features are extracted through the backbone network to obtain multi-level feature maps. These multi-level feature maps are then input into the multi-level feature aggregation module in the feature extractor. The multi-level feature aggregation module integrates the multi-level feature maps output from the backbone network to output a high-level feature map. .

[0043] The multi-layer feature aggregation module first uses an independent adapter, then utilizes a channel attention mechanism to generate spatially varying weights, and performs a weighted summation of multi-layer features in the texture image after unification. Finally, the multi-layer feature aggregation module outputs a high-level feature map with high semantic density and fixed size, serving as a high-quality input for subsequent spatial and frequency domain branches.

[0044] In this embodiment of the invention, the feature extraction process can effectively capture global long-range dependency information in texture images. By systematically integrating multi-level features, high-level feature maps with high semantic density and fixed size are generated, providing a high-quality and standardized feature input foundation for subsequent dual-branch structures. This significantly enhances the model's ability to represent the intrinsic structure of textures, ensures the integrity and stability of feature transmission during classification, and improves the reliability of overall classification decisions.

[0045] In one embodiment of the present invention, the high-level feature map is processed in the spatial domain and frequency domain respectively by parallel connected spatial domain branches and frequency domain branches.

[0046] Specifically, the high-level feature map generated by the feature extractor is simultaneously and independently input into two parallel branches: a spatial domain branch and a frequency domain branch. The spatial domain branch receives and processes the high-level feature map, focusing on extracting and enhancing the spatial domain features of the image, such as local details and spatial complexity. The frequency domain branch also receives and processes the high-level feature map, focusing on analyzing and enhancing the frequency domain features of the image, such as the frequency domain structure information of different sub-bands. The two branches work in parallel, outputting processed spatial and frequency feature maps respectively. By setting up parallel connected spatial and frequency domain branches, the model can simultaneously analyze and enhance the same high-level feature map in both spatial and frequency dimensions.

[0047] In this embodiment of the invention, the spatial domain branching of the spatial domain processing of the high-level feature map includes: Multi-level spatial feature extraction is performed on the high-level feature map through multiple feature layers; The extracted multi-level spatial features are aggregated, and local differential box counting is performed on the aggregated spatial features to generate a feature map containing local spatial complexity. The feature map containing local spatial complexity is converted into soft histogram features and residually fused with the high-level feature map to output a spatial feature map.

[0048] In one embodiment of the present invention, the spatial domain branch acts as a multi-scale spatial feature extractor. It simulates hierarchical feature extraction through multiple feature layers, sequentially passing the input features through multiple sequentially connected feature layers. Each feature layer performs convolution, normalization, and nonlinear activation operations on the input features, thereby completing multi-level spatial feature extraction and obtaining a series of spatial features at different levels. These multi-level spatial features are then aggregated into a unified spatial feature map. Subsequently, a local differential box counting operation is performed on this aggregated spatial feature map to approximate local differential box counting. By calculating its statistical characteristics under different local windows, a feature map that can quantify the local spatial complexity of the image is generated, thus extracting a feature map containing local spatial complexity. Finally, this feature map containing local spatial complexity is converted into soft histogram features using a soft histogram statistical method based on a Gaussian kernel function. The soft histogram features are then concatenated with the high-level feature map of the original input spatial domain branch (residual fusion, i.e., addition) using Gaussian kernel function-based soft histogram statistics to obtain an enhanced spatial feature map. The entire process ensures the effective integration and expression of features at different levels in the spatial domain, enhancing the model's ability to recognize texture details.

[0049] In this embodiment of the invention, the frequency domain branching of the high-level feature map includes the following frequency domain processing: Perform discrete wavelet transform on the high-level feature map to decompose it into multiple frequency sub-bands; Feature enhancement is performed separately for each frequency sub-band; The enhanced frequency sub-bands are reconstructed using wavelet reconstruction, and the reconstruction results are fused with the high-level feature map using residual fusion to output the frequency feature map.

[0050] In one embodiment of the invention, the frequency domain branch is intended to enhance the frequency information of the features using discrete wavelet transform, and the input high-level feature map is decomposed into four sub-bands using a Haar wavelet filter: low-frequency sub-band (LL), vertical high-frequency sub-band (LH), horizontal high-frequency sub-band (HL), and diagonal high-frequency sub-band (HH).

[0051] Subsequently, an independent subband processing module is applied to each subband for feature enhancement. That is, for each frequency subband obtained by decomposition, a subband processing module with the same structure but independent parameters is applied to process it. Each subband processing module contains a sequence of operations such as convolution, normalization and activation functions to enhance the feature representation of the subband.

[0052] Finally, the enhanced subbands are spliced ​​together and the features are reconstructed through deconvolution. The features are then fused with the original input using residuals to obtain the frequency feature map.

[0053] In this embodiment of the invention, spatial feature maps and frequency feature maps are fused through a bidirectional cross-attention mechanism, including: The spatial feature map and the frequency feature map are subjected to feature projection and spatial flattening processes, respectively. Use a multi-head attention mechanism and enhance the frequency feature map with spatial information; Use a multi-head attention mechanism to enhance the spatial feature map with frequency information; The enhanced frequency feature map and the enhanced spatial feature map are spliced ​​together to obtain the fused feature map.

[0054] In one embodiment of the present invention, firstly, the spatial feature map and the frequency feature map are subjected to feature projection and spatial flattening processes, respectively, to convert them into a format suitable for attention calculation. Next, a standard multi-head attention mechanism is employed, using the processed spatial feature map as the information source, to perform cross-attention calculation on the processed frequency feature map, thereby using spatial information to enhance the frequency feature map. Then, the standard multi-head attention mechanism is again employed, using the enhanced frequency feature map as the information source, to perform cross-attention calculation on the processed spatial feature map, thereby using frequency information to enhance the spatial feature map. Finally, the frequency feature map enhanced with spatial information is concatenated with the spatial feature map enhanced with frequency information to obtain a final fused feature map combining the complementary information of both domains. Through the above-described bidirectional cross-attention mechanism, deep interaction and information complementarity between spatial domain features and frequency domain features can be achieved.

[0055] In this embodiment of the invention, determining the category of the texture image based on the fusion result includes: The fused feature map is input into a fully connected classifier, which includes at least one fully connected layer and a Softmax activation function layer. The softmax activation function layer outputs a class probability vector that is equal to the number of classes in the texture image dataset, and the class corresponding to the position with the highest probability is used as the prediction result.

[0056] In one embodiment of the present invention, the fused feature map is first transformed and dimensionality reduced by a fully connected layer, and then processed by a Softmax activation function layer. Each element of the class probability vector output by the Softmax activation function layer represents the probability that the input image belongs to the corresponding class. Finally, the class index corresponding to the element with the largest value in the probability vector is used as the model's predicted classification result for the texture image. By processing the fused feature map using a fully connected classifier including a fully connected layer and a Softmax activation function layer, the high-dimensional, deeply fused features can be effectively mapped to a specific class space. The fully connected layer is responsible for key feature integration and dimensionality transformation, while the Softmax activation function layer converts the final output into a normalized probability distribution, thereby providing a clear and quantitative confidence level for each candidate class, and determining the final predicted class based on the highest confidence level, achieving a direct and effective mapping from features to class determination.

[0057] In this embodiment of the invention, the Vision Mamba dual-branch cross-fusion network model is trained using a texture image dataset, including: The texture images in the training set are input into the Vision Mamba dual-branch cross-fusion network model in batches to obtain the class probability vector of each texture image. The class probability vector is compared with the encoded vector corresponding to the true class label of the texture image, and the cross-entropy loss function is calculated. The gradient is calculated based on the cross-entropy loss function, and the model parameters are updated using gradient backpropagation and gradient descent.

[0058] In one embodiment of the present invention, the training stages proceed sequentially according to the data processing and parameter optimization process. The working principle is to adjust the model parameters by comparing the difference between the predicted results and the true labels. Specifically, the Vision Mamba dual-branch cross-fusion network model, after feature extraction, dual-branch processing, feature fusion, and classifier calculation, outputs a category probability vector corresponding to each image. The category probability vector is compared with the encoding vector corresponding to the true category label of the image, and the prediction deviation is quantified by calculating the cross-entropy loss function of the two. Based on the obtained cross-entropy loss function value, the gradient of each trainable parameter of the model is calculated. The gradient backpropagation algorithm is used to backpropagate the gradient information layer by layer to each module of the Vision Mamba dual-branch cross-fusion network model, and then the model parameters are iteratively updated by the gradient descent algorithm. The above process is repeated until the preset training rounds are completed.

[0059] Among them, the encoding vector corresponding to the true category label of the texture image can be a one-hot encoding vector.

[0060] In one embodiment of the present invention, the model is trained iteratively multiple times using the Vision Mamba dual-branch cross-fusion network model, enabling the model to learn model parameters that result in increasingly accurate classification.

[0061] In one embodiment of the present invention, during gradient backpropagation, the gradient is nonlinearly modulated by the ELU activation function. When the input value of the activation function is negative, its output gradient is a non-zero positive value based on the exponential function, so as to ensure that the gradient signal from the cross-entropy loss function can be effectively backpropagated.

[0062] By modulating the gradient using the characteristics of the ELU activation function during the backpropagation process of model training, the vanishing gradient or signal interruption problem that may be caused by negative activation function inputs is effectively mitigated. This ensures that even when the network is deep or some neurons are inactive, the gradient calculated by the loss function can still be smoothly backpropagated to the network front end, thereby stabilizing and promoting the continuous and effective updating of all parameters in the entire Vision Mamba two-branch cross-fusion network model, which helps the model converge faster and learn better feature representations.

[0063] In one embodiment of the present invention, the expression for the ELU activation function is: ,in, As a preset constant, This represents the input value to the activation function. When the input value... When the value is greater than 0, the output is It itself; when the input value When less than or equal to 0, the output is .

[0064] This invention proposes a texture image classification method based on the Vision Mamba dual-branch cross-fusion network. The Vision Mamba dual-branch cross-fusion network model enables parallel processing of the high-level feature maps of the input texture image in both the spatial and frequency domains. Furthermore, a bidirectional cross-attention mechanism is used to deeply fuse the resulting spatial and frequency feature maps. During training, this method automatically learns and optimizes network parameters end-to-end, thereby achieving the collaborative extraction and utilization of global long-range dependencies, local spatial complexity, and frequency domain structural information in texture images. Ultimately, it achieves higher texture image classification accuracy with lower computational complexity, significantly improving the accuracy of texture image classification. It exhibits good robustness to deformation, illumination changes, and scale changes, and possesses efficient computational and inference capabilities. It is applicable to texture-rich image classification scenarios such as medical images, remote sensing images, scene images, and geological material images.

[0065] In one embodiment of the present invention, a specific example is used to illustrate the detailed implementation process of the texture image classification method based on the Vision Mamba dual-branch cross-fusion network of the present invention.

[0066] like Figure 1 As shown, this embodiment illustrates a texture image classification method based on a Vision Mamba dual-branch cross-fusion network. The main steps of this method are as follows: Step 1: Data preparation, including a database of texture images and their category labels for training and testing.

[0067] Step 2: Training data partitioning and preprocessing. The texture image database is divided into training and test sets, and preprocessed to serve as input for the network model.

[0068] Step 3: Train the Vision Mamba dual-branch cross-fusion network model and save the trained network parameters; Step 4: Load the saved Vision Mamba dual-branch cross-fusion network model and perform validation and testing on the validation and test datasets.

[0069] Based on the above example steps, the method can be implemented in the following details: Step 1: Data Preparation (1) Select and download the texture image dataset collected in real-world scenarios and label the categories.

[0070] Step 2: Data Preprocessing (1) Divide the texture images into training and testing sets, and take the images and their corresponding category labels as a group; (2) Preprocessing: Each group of images is resized and cropped to a size of 224×224, and then randomly flipped horizontally with a 50% probability and normalized.

[0071] Step 3: Network Structure and Training 3.1 Establish the network structure, such as... Figure 2 As shown, the network model consists of five parts: a feature extractor based on a Vim pre-trained model, a spatial domain branch, a frequency domain branch, a cross-attention feature fusion module, and a fully connected classifier. The high-level feature maps output by the Vim pre-trained model feature extractor are processed in parallel and independently into the spatial and frequency domain branches, respectively. The resulting outputs are feature maps of consistent size, which then enter the subsequent cross-attention feature fusion module. Specifically, the network model is as follows: (1) Feature extractor based on Vim pre-trained model (taking ViM-S as an example): Patch Embedding: A two-dimensional convolutional layer with a kernel size of 16×16, 384 kernels, stride=16, padding=0, input channels C=3, and output embedding dimension D=384. Position Embedding: Shape is 1×196×384; Backbone network (MambaBlock×24, MB1-MB24): Isomorphic from layer 1 to layer 24 Root Mean Square Normalization (RMSNorm): 384 dimensions; Linear mapping layer 1: Dimension 384 → 1536; One-dimensional convolutional layer: kernel size is 4, number of kernels is 768; Forward spatial state model branch: Dimension 768→56→768; Backward spatial state model branch: Dimension 768→56→768; Linear mapping layer 2: Dimension 768 → 384; Multi-layer feature aggregation module; (2-1) Spatial domain branching; (2-2) Frequency domain branching; (3) Cross-attention feature fusion module; (4) Fully connected classifier: Fully connected layer 1: 256 input features, 128 output features; ELU activation function; Fully connected layer 2: The number of input features is 128, and the number of output features is the number of categories; SoftMax layer.

[0072] 3.2 Implementation and Calculation Process of the Multi-Level Feature Aggregation Module like Figure 3 As shown, the multi-layer feature aggregation module aims to integrate the previously obtained multi-layer information to achieve a unified feature representation. It also generates a high-semantic-density, fixed-dimensional feature map by introducing a channel attention-based mechanism, providing high-quality input for the subsequent dual-branch approach. In this embodiment of the invention, the feature extractor outputs a feature map containing... List of feature maps For each feature map Using a separate feature adapter sequence( The adaptive feature map is obtained by processing the convolutional layer → batch normalization layer → ReLU activation layer. :

[0073] The resulting adaptation feature map All of them need to be interpolated to the first layer feature map For the same space dimensions, A uniformly sized adaptation feature map Then average the values ​​of all the adapted feature maps Input to attention sequence (convolutional layer → Activation → Convolutional layer In, and perform on the output. Activation, normalization along the channel dimension, generating spatially varying attention weights. :

[0074] Then, attention weights and the adaptation feature map after size standardization Perform a weighted summation and pass it through a final fused convolutional block. (Convolutional layer → Batch normalization →) activation, The final output of the multi-layer feature aggregation module is obtained, namely the high-level feature map. :

[0075] It is the first Attention weights obtained from each channel It is the first Adaptation feature map of each channel output.

[0076] 3.3 Calculation process for spatial domain branching like Figure 4 As shown, the spatial domain branch is a multi-scale spatial feature extractor designed to enhance the spatial representation capability of features based on convolutional operations and spatial attention mechanisms. In this embodiment, the feature map after multi-layer feature aggregation... Input feature map for spatial domain branch First, save the original input as: This is used for the final residual fusion, and then the feature maps are... The data is sequentially passed through N preset feature layers, each consisting of a sequence of 3×3 convolutional layers, batch normalization, and ReLU activation functions, to simulate the extraction of hierarchical features.

[0077]

[0078]

[0079]

[0080] The output is obtained by directly passing the input through a layer module. The input is the output obtained after passing through two layer modules. Depend on It is obtained through a layer module, and subsequently follows... This recursion is then complete.

[0081] For what was obtained The list is aggregated into a single feature map using a multi-level aggregation method. Then, through local differential box counting, multiple box filters of different sizes are used to obtain a feature map containing local spatial complexity. First, define... For feature map In batch ,aisle ,coordinate The value at that location, For Centered on, with side length as A local window, using a mean-based ( ) and standard deviation ( The approximate method is used to calculate the local differential box dimension. :

[0082]

[0083] In batch ,aisle coordinates The position, with The standard deviation of all eigenvalues ​​of a window with side length ; It is a coordinate. It is a window. It involves iterating through the points within this window; In batch ,aisle of The position, with The feature value of the window with side length; In batch ,aisle of The position, with The mean of the feature values ​​of the window with side length ; Noise term; batch ,aisle On the feature map, with ( Centered on, with side length as The dimension of the local differential box of the local window.

[0084] All channels and all scales Feature maps are concatenated along the channel dimension to obtain local complexity features. Then, local complexity features Normalization is performed within each channel, and for each normalized eigenvalue... and the center of each interval Instead of traditional hard partitioning, a Gaussian kernel function is used to calculate the smooth probability that a pixel belongs to the interval. Then smooth the probabilities of each bin. Histogram statistics are performed by summing all weights along the spatial dimension to obtain the histogram counts for each channel and each interval. Then, L1 normalization is performed on the histogram to obtain the soft histogram features. and compare it with the original input. Residual fusion was performed, and the spatial feature map after spatial domain branch extraction and enhancement was finally obtained. .

[0085]

[0086]

[0087]

[0088] It is a hyperparameter of the Gaussian kernel, which is generally taken as the bin width and its multiples. It is the first in the feature map The ( ) channel, the ( ) The feature value of the spatial location belongs to the feature value of the soft histogram. Gaussian weights for each bin; For the first The first channel, the first The summation of Gaussian weights for each bin; Yes Sum the H values ​​of all sub-boxes under the channel. It is to traverse all channels The boxes are divided into smaller containers.

[0089] 3.4 Calculation process for frequency domain branching like Figure 5 As shown, frequency domain analysis aims to decompose the feature map into different frequency sub-bands using Discrete Wavelet Transform (DWT) and then process and reconstruct these sub-bands using Inverse Wavelet Transform (IWT) to enhance the frequency information of the features. In this embodiment of the invention, the original input is first saved as: Four 2×2 Haar wavelet filters are used as depthwise separable convolution kernels to simultaneously filter and downsample the preprocessed input features X, decomposing them into four subbands: LL (low frequency), LH (vertical high frequency), HL (horizontal high frequency), and HH (diagonal high frequency). An independent subband processing module is then applied to each subband. Perform feature enhancement, in In the module, the input sub-band features They passed in turn. Sequence processing, here The module processes the four sub-band feature maps independently, that is... The modules have the same structure, but each module has its own independent parameters. The transformed features are then compared with the original input sub-band features. Perform residual connections to obtain the enhanced output. :

[0090] in

[0091] Then bring these four new children By concatenating the features along the channel dimension and performing deconvolution (using an inverse filter corresponding to DWT (Discrete Wavelet Transform) as a depthwise separable transpose convolution kernel), the feature maps are upsampled to their original size to obtain the reconstructed features. Reconstructed features After a final 3×3 convolutional block and combined with the original input Residual fusion is performed to obtain the final frequency feature map. .

[0092] 3.5 Implementation and computation process of the cross-attention feature fusion module The cross-attention feature fusion module is designed to receive the outputs of the spatial domain branch and the frequency domain branch, and to perform information fusion in a bidirectional cross-attention manner. In this embodiment of the invention, the spatial feature map is first processed... and frequency feature map Preprocessing is performed, including feature projection and spatial flattening. Then, the standard multi-head attention (MHA) mechanism is used to enhance the frequency features with spatial information. Subsequently, the frequency features are enhanced with the same MHA mechanism. Finally, the two enhanced feature maps are concatenated to obtain the final output that combines complementary information from the spatial and frequency domains.

[0093] 3.6 ELU activation function, the expression for the ELU activation function is as follows: It combines the advantages of ReLU, which can accelerate model convergence and solve the "neuron death" problem of ReLU in the negative value region.

[0094] Step 4: Model Testing (1) Read the test image data in the test set of the dataset and preprocess it according to the preprocessing method of the training set; (2) Input the test image into the loaded, trained Vision Mamba dual-branch cross-fusion network model to obtain the predicted class probability vector, and calculate the class corresponding to the position with the maximum probability value. (3) Compare the predicted categories with the true category labels and calculate the prediction accuracy.

[0095] In this embodiment, as shown in Table 1, the results obtained by the method in this embodiment are significantly better than those of other existing methods.

[0096] Table 1 Comparison of results between the method of the present invention embodiment and other methods.

[0097] In one embodiment of the present invention, a texture image classification device based on a Vision Mamba dual-branch cross-fusion network is provided to implement the method described in the foregoing embodiments. The device includes the following modules: The training module is used to train the Vision Mamba two-branch cross-fusion network model; The classification module is used to input the texture image to be classified into the trained Vision Mamba dual-branch cross-fusion network model and obtain the corresponding classification result. The training module includes the following sub-modules: The dataset acquisition submodule is used to acquire a texture image dataset, which contains texture image data of two or more categories, and the texture image data includes texture images and corresponding category labels. The model construction submodule is used to construct the VisionMamba dual-branch cross-fusion network model. The VisionMamba dual-branch cross-fusion network model includes a feature extractor, a spatial domain branch, a frequency domain branch, a cross-attention feature fusion module, and a classifier. The feature extractor is used to extract high-level feature maps of the texture image. The spatial domain branch and the frequency domain branch are used to simultaneously process the high-level feature maps in the spatial domain and the frequency domain, respectively, to obtain spatial feature maps and frequency feature maps. The cross-attention feature fusion module is used to fuse the spatial feature map and the frequency feature map through a bidirectional cross-attention mechanism to obtain a fused feature map. The classifier is used to determine the category of the texture image based on the fused feature map. The training submodule is used to train the Vision Mamba dual-branch cross-fusion network model using a texture image dataset to optimize the model parameters.

[0098] In one embodiment of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the methods described in the foregoing embodiments.

[0099] In one embodiment of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the foregoing embodiments.

[0100] The device, equipment, and medium described herein have the same technical effects as those achieved by the methods described in the foregoing embodiments.

[0101] The detailed explanations of the above embodiments are intended only to explain the present invention so as to facilitate a better understanding of the present invention. However, these descriptions should not be construed as limiting the present invention for any reason. In particular, the various features described in different embodiments can be arbitrarily combined with each other to form other embodiments. Unless there is an explicit description to the contrary, these features should be understood to be applicable to any embodiment, and not limited to the described embodiments.

Claims

1. A texture image classification method based on a Vision Mamba dual-branch cross-fusion network, characterized in that, The texture image to be classified is input into the trained Vision Mamba two-branch cross-fusion network model to obtain the corresponding classification result. The Vision Mamba two-branch cross-fusion network model is trained through the following steps: Obtain a texture image dataset, wherein the texture image dataset contains texture image data of two or more categories, and the texture image data includes texture images and corresponding category labels; A Vision Mamba dual-branch cross-fusion network model is constructed, which includes a feature extractor, a spatial domain branch, a frequency domain branch, a cross-attention feature fusion module, and a classifier. The feature extractor is used to extract high-level feature maps of the texture image. The spatial domain branch and the frequency domain branch are used to simultaneously process the high-level feature maps in the spatial domain and the frequency domain, respectively, to obtain spatial feature maps and frequency feature maps. The cross-attention feature fusion module is used to fuse the spatial feature map and the frequency feature map through a bidirectional cross-attention mechanism to obtain a fused feature map. The classifier is used to determine the category of the texture image based on the fused feature map. The Vision Mamba dual-branch cross-fusion network model was trained using a texture image dataset to optimize the model parameters.

2. The texture image classification method based on the Vision Mamba dual-branch cross-fusion network according to claim 1, characterized in that, Divide the texture image dataset into a training set and a test set, and use the texture image dataset to train the Vision Mamba dual-branch cross-fusion network model, including: Each texture image in the training set is input into the Vision Mamba dual-branch cross-fusion network model in batches, and the class probability vector of each texture image is output. The category probability vector is compared with the encoded vector corresponding to the true category label of the texture image, and the cross-entropy loss function is calculated. The gradient is calculated based on the cross-entropy loss function, and the model parameters are updated using gradient backpropagation and gradient descent; and / or, During gradient backpropagation, the gradient is nonlinearly modulated using the ELU activation function. When the input value of the activation function is negative, its output gradient is a non-zero positive value based on an exponential function, ensuring that the gradient signal from the cross-entropy loss function can be effectively backpropagated.

3. The texture image classification method based on the Vision Mamba dual-branch cross-fusion network according to claim 1, characterized in that, The feature extractor is configured to perform the following operations: The texture image is converted into sequence features and then positionally encoded. The sequence features are processed by multiple isomorphic Mamba modules to extract high-level features of long-range dependencies. High-level features of long-range dependencies output by multiple isomorphic Mamba modules are integrated to generate a high-level feature map.

4. The texture image classification method based on the Vision Mamba dual-branch cross-fusion network according to claim 1, characterized in that, The spatial domain branch performs spatial domain processing on high-level feature maps, including: Multi-level spatial feature extraction is performed on the high-level feature map through multiple feature layers; The extracted multi-level spatial features are aggregated, and local differential box counting is performed on the aggregated spatial features to generate a feature map containing local spatial complexity. The feature map containing local spatial complexity is converted into soft histogram features and residually fused with the high-level feature map to output a spatial feature map.

5. The texture image classification method based on the Vision Mamba dual-branch cross-fusion network according to claim 1, characterized in that, The frequency domain branch performs frequency domain processing on high-level feature maps, including: Perform discrete wavelet transform on the high-level feature map to decompose the high-level feature map into multiple frequency sub-bands; Feature enhancement is performed separately for each frequency sub-band; The enhanced frequency sub-bands are reconstructed using wavelet reconstruction, and the reconstruction results are fused with the high-level feature map using residual fusion to output the frequency feature map.

6. The texture image classification method based on the Vision Mamba dual-branch cross-fusion network according to claim 1, characterized in that, The spatial feature map and the frequency feature map are fused through a bidirectional cross-attention mechanism, including: The spatial feature map and the frequency feature map are respectively subjected to feature projection and spatial flattening processes; A multi-head attention mechanism is used to enhance the frequency feature map with spatial information. The spatial feature map is enhanced using a multi-head attention mechanism and frequency information. The enhanced frequency feature map and the enhanced spatial feature map are spliced ​​together to obtain a fused feature map.

7. The texture image classification method based on the Vision Mamba dual-branch cross-fusion network according to any one of claims 1-6, characterized in that, The classification of the texture image is determined based on the fused feature map, including: The fused feature map is input into a full classifier, which includes at least one fully connected layer and a Softmax activation function layer. The softmax activation function layer outputs a category probability vector that is equal to the number of categories in the texture image dataset, and the category corresponding to the position with the highest probability is used as the prediction result.

8. A texture image classification device based on a Vision Mamba dual-branch cross-fusion network, characterized in that, For implementing the method according to any one of claims 1-7, the apparatus comprises the following modules: The training module is used to train the Vision Mamba two-branch cross-fusion network model; The classification module is used to input the texture image to be classified into the trained Vision Mamba dual-branch cross-fusion network model and obtain the corresponding classification result. The training module includes the following sub-modules: The dataset acquisition submodule is used to acquire a texture image dataset, which contains texture image data of two or more categories, and the texture image data includes texture images and corresponding category labels. The model construction submodule is used to construct the Vision Mamba dual-branch cross-fusion network model. The Vision Mamba dual-branch cross-fusion network model includes a feature extractor, a spatial domain branch, a frequency domain branch, a cross-attention feature fusion module, and a classifier. The feature extractor is used to extract high-level feature maps of the texture image. The spatial domain branch and the frequency domain branch are used to simultaneously process the high-level feature maps in the spatial domain and the frequency domain, respectively, to obtain spatial feature maps and frequency feature maps. The cross-attention feature fusion module is used to fuse the spatial feature map and the frequency feature map through a bidirectional cross-attention mechanism to obtain a fused feature map. The classifier is used to determine the category of the texture image based on the fused feature map. The training submodule is used to train the Vision Mamba dual-branch cross-fusion network model using a texture image dataset to optimize the model parameters.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

Images