Multi-dimensional structure attention model and network structure design method based on same

Abstract

The invention discloses a multi-dimensional structure attention model and a mobile network structure design method based on the same, a Vim structure is used for reference, a '1 + 3' architecture is adopted to embed multi-dimensional information into channel attention, and the model is called a 'multi-dimensional fusion VIM structure-like attention mechanism structure'. Different from the traditional channel attention that a feature tensor is simplified into a single feature vector through two-dimensional global pooling, the attention mechanism divides the input into two main branches. The left branch retains original information for subsequent feature screening, and the right branch divides channel attention into three one-dimensional feature coding processes, which aggregate features in one channel direction and two spatial directions, respectively. This approach simultaneously captures remote dependencies and preserves precise location information. Then, the obtained feature map is combined with the left branch to generate an attention map with direction perception, position sensitivity and channel perception.

Description

Technical Field

[0001] The present invention belongs to the field of computer image processing and attention mechanism, and specifically relates to a multidimensional structural attention model and a mobile network structure design method based on the model. Background Art

[0002] Attention mechanisms are key techniques in deep learning that enable models to focus on specific parts of input data when making predictions. By selectively emphasizing the most relevant parts of the input, these mechanisms significantly improve the performance of models on a wide range of tasks, including natural language processing (NLP) and computer vision (CV). They allow models to dynamically adjust their attention based on the importance of the input, thereby reducing the influence of low-relevance information. As a result, attention mechanisms have attracted great interest in the field of computer vision research. Traditional channel attention methods usually involve strategies to aggregate spatial information and then reweight channels accordingly. This design draws on the ideas of Mamba and combines the results of MambaOut to redefine multi-dimensional attention as a "1+3" branch module and design a new attention module.

[0003] In recent years, attention mechanisms have made significant progress in deep learning, especially in the field of NLP, and have become an indispensable tool for technological progress. Various attention mechanisms have been proposed and applied to different tasks, showing excellent performance and versatility. Soft Attention focuses on the key parts of the sequence by calculating the context vector as a weighted sum, enabling the model to assign different weights to different parts of the input. Hard Attention uses discontinuous selection to focus on a single input element; although there are challenges in training, it performs well in specific tasks.

[0004] The self-attention mechanism is a core component of the Transformer model, which enables the model to consider all positions in the sequence simultaneously and effectively capture long-range dependencies. Multi-head attention enhances the expressiveness of the model by independently calculating attention in multiple subspaces and aggregating the results.

[0005] In addition, hierarchical attention processes hierarchical data by calculating low-level and high-level attention, allowing the model to focus on salient information across multiple layers. Convolutional attention takes advantage of convolutional neural networks and is very suitable for processing images or time series data. Channel-wise convolution is one of the most effective ways to manage model complexity.

[0006] Existing attention mechanisms, such as the Squeeze Excitation (SE) module and the Convolutional Block Attention Module (CBAM), have made significant progress in enhancing feature representation. Although the SE module is simple and computationally efficient, it is limited by channel attention and cannot capture spatial information. CBAM improves on this by applying channel and spatial attention sequentially, but it does not fully exploit the interdependencies between these dimensions and may increase computational complexity. Compared with SE and CBAM, coordinate attention (CA) embeds direction-specific information into channel attention along the spatial dimension and selects a suitable channel dimensionality reduction ratio to achieve comparable performance to SE attention. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a multi-dimensional structural attention model and a mobile network structure design method based on the model, which solves the problems in the prior art of complex calculation of the attention mechanism and low utilization of the interdependence between dimensions.

[0008] The present invention adopts the following technical solutions to solve the above technical problems:

[0009] A multi-dimensional structural attention model comprises a feature data processing module and a dual-branch module of a "1+3" structure; wherein the feature data processing module normalizes the input feature data; the dual-branch module of the "1+3" structure comprises a first branch consisting of a Sigmoid activation layer, and a second branch consisting of channel dimension, width dimension and height dimension feature extraction modules; the normalized feature data are respectively input into the first branch and the second branch, the data processed by the first branch are respectively multiplied with the data processed by the three parts of the second branch, and the multiplication results are then added to obtain tensor data as output.

[0010] The specific implementation process of the feature data processing module is as follows:

[0011] The feature data X′ p The input expansion layer and a depth-wise separable convolutional layer pass through a convolutional layer with an expansion factor of t and a feature scale of 1×1 to obtain an expansion layer with an output channel number t times the number of input channels; the feature scale of the depth-wise separable convolutional layer is 3×3, the step size is 1 or 2, and the number of groups is equal to the number of input channels of this layer, resulting in χ(C×H×W).

[0012] The χ(C×H×W) passes through the first branch and inputs a Sigmoid activation layer to obtain χ l .

[0013] The processing process of the second branch includes the following steps:

[0014] Step 1: Rotate the χ(C×H×W) feature data 90 degrees counterclockwise along the height axis to obtain a W×H×C feature data.

[0015] Step 2: Reduce through DR-pooling layer The size is 2×H×C, resulting in a tensor of

[0016] Step 3: Input a convolution kernel with a feature scale of 7×7 to obtain

[0017] Step 4: After batch normalization, a 1×H×C tensor is obtained through a sigmoid (σ) activation layer;

[0018] Step 5: Rotate 90 degrees clockwise along the height axis to restore the tensor to C×H×1 Operate along the width axis to get a C×1×W tensor Without rotation, we get a 1×H×W tensor

[0019] By l Respectively Multiply, then add the three and divide by 3, and perform residual connection with χ(C×H×W) to obtain the output of the model.

[0020] The output of the model is expressed by the following formula:

[0021]

[0022] The mobile network structure design method based on the multi-dimensional structural attention model includes the following steps:

[0023] Step S1, data preprocessing, preprocessing the image data according to the quality of the existing data set;

[0024] Step S2, data dimensionality reduction, input the image data into a convolution layer, a batch normalization layer and an activation layer to initialize the data;

[0025] Step S3, inverse residual convolution structure and feature fusion, input the initialized data into the multi-layer convolution layer of the inverse residual result for feature extraction and multi-dimensional information encoding for three-dimensional feature information fusion; input the extracted features into two layers of convolution layers and one layer of multi-dimensional fusion convolution of the inverse residual connection for information fusion of the correlation between local image features and global image features;

[0026] Step S4, feature dimension reduction and full connection layer operation, input the information fused features into the adaptive mean pooling layer for feature dimension reduction, and input the dimension reduced features into the full connection layer to obtain the final category probability output;

[0027] Step S5, model training: during the network training process, the network model parameters are adjusted according to the loss function through the AdamW optimization algorithm to obtain the final network global optimal solution parameters;

[0028] Step S6: After the network model training is completed, it can be used in actual image processing classification detection tasks.

[0029] The specific process of preprocessing the image data in step S1 is as follows:

[0030] Step S11, randomly cropping the image to a specified size;

[0031] Step S12, randomly flipping the image horizontally;

[0032] Step S13, converting the flipped image into a PyTorch tensor;

[0033] Step S14, normalize the tensor using the mean and standard deviation values ​​of the commonly used pre-trained model to obtain image data X.

[0034] The specific process of step S2 is as follows:

[0035] Step S21, the image data X obtained by preprocessing is input into the convolution layer for convolution operation to obtain 32 feature data X′ whose image scale is reduced to one quarter, wherein the convolution layer includes 32 two-dimensional convolution kernels with a feature scale of 3×3 and a convolution step size of 2;

[0036] Step S22: batch normalize the feature data X′ and input it into a ReLu6 activation layer to obtain feature data X′ p .

[0037] The step S4 specifically includes the following steps:

[0038] Step S41, inputting the multi-dimensional feature fusion data y into the adaptive average pooling layer;

[0039] Step S42, the multi-dimensional feature fusion data y is subjected to a random inactivation with a ratio of 0.2, and then input into a fully connected layer, wherein the number of output neurons of the fully connected layer is set to be the number of categories of the target classification task, to obtain the final category probability output.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. This method significantly improves classification and detection accuracy, and is plug-and-play and suitable for lightweight models on mobile devices with minimal additional computational overhead.

[0042] 2. This paper proposes a multi-dimensional Vision-Mamba structural attention mechanism for lightweight networks. It adopts the structure of Vim and draws on MambaOut to remove the structured spatial model in Vim; it integrates spatial attention, channel attention, channel pooling, dimensionality reduction pooling (DR-pooling), and convolution operations across multiple dimensions in a unified framework. Compared with existing methods, this strategy not only reduces computational redundancy, but also captures interdependencies in a more comprehensive and efficient way, thereby providing a more effective feature extraction mechanism.

[0043] 3. The multi-dimensional feature fusion method based on the Vision-Mamba structure is a very effective method to improve the feature extraction ability of image classification and recognition. The attention module proposed in the present invention has only 0.0026M parameters, while achieving performance equivalent to or better than SE and CBAM, improving by 0.9% and 0.2% on the CIFAR-100 dataset based on MobileNetV2 and ShuffleNetV2 respectively, and improving by 1.1 on the MS-COCO dataset based on YoloV5s, mAP (0.5:0.95). A new method is provided for the attention mechanism in feature extraction of image classification and recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The network calculation process of the method of the present invention and its corresponding actual network structure diagram.

[0045] Figure 2 This is the core attention mechanism calculation process of the method of the present invention.

[0046] Figure 3 This is a comparison chart between the method of the present invention and other methods based on the model MobileNetV2 and the Cifar100 dataset.

[0047] Figure 4 is a comparison chart between network accuracy and training throughput.

[0048] Figure 5 It is a diagram of the ablation experiment results of the present invention. DETAILED DESCRIPTION

[0049] The structure and working process of the present invention will be further described below in conjunction with the accompanying drawings.

[0050] The proposed multi-dimensional vim-like structure attention module (MDVA) integrates spatial attention, channel attention, channel pooling, dimensionality reduction pooling (DR-pooling), and convolution operations across multiple dimensions in a unified framework. Compared with existing methods, this strategy not only reduces computational redundancy, but also captures interdependencies in a more comprehensive and efficient way, thereby providing a more effective feature extraction mechanism.

[0051] A multi-dimensional structural attention model comprises a feature data processing module and a dual-branch module of a "1+3" structure; wherein the feature data processing module normalizes the input feature data; the dual-branch module of the "1+3" structure comprises a first branch consisting of a Sigmoid activation layer, and a second branch consisting of channel dimension, width dimension and height dimension feature extraction modules; the normalized feature data are respectively input into the first branch and the second branch, the data processed by the first branch are respectively multiplied with the data processed by the three parts of the second branch, and the multiplication results are then added to obtain tensor data as output.

[0052] The mobile network structure design method based on the multi-dimensional structural attention model includes the following steps:

[0053] Step S1, data preprocessing, preprocessing the image data according to the quality of the existing data set;

[0054] Step S2, data dimensionality reduction, input the image data into a convolution layer, a batch normalization layer and an activation layer to initialize the data;

[0055] Step S3, inverse residual convolution structure and feature fusion, input the initialized data into the multi-layer convolution layer of the inverse residual result for feature extraction and multi-dimensional information encoding for three-dimensional feature information fusion; input the extracted features into two layers of convolution layers and one layer of multi-dimensional fusion convolution of the inverse residual connection for information fusion of the correlation between local image features and global image features;

[0056] Step S4, feature dimension reduction and full connection layer operation, input the information fused features into the adaptive mean pooling layer for feature dimension reduction, and input the dimension reduced features into the full connection layer to obtain the final category probability output;

[0057] Step S5, model training: during the network training process, the network model parameters are adjusted according to the loss function through the AdamW optimization algorithm to obtain the final network global optimal solution parameters;

[0058] Step S6: After the network model training is completed, it can be used in actual image processing classification detection tasks.

[0059] Specific embodiments, such as Figures 1 to 5 As shown,

[0060] The dataset CIFAR-100 used in this embodiment is a dataset for machine learning and computer vision, containing 60,000 32x32 pixel color images. It has 100 classes, each with 600 images, divided into 50,000 training images and 10,000 test images. These 100 classes are divided into 20 superclasses, each with 5 classes. For example, the superclass "vehicle" includes bicycles, buses, motorcycles, pickup trucks, and trains. The implementation steps of the method are as follows:

[0061] Step S1, preprocessing the image data according to the quality of the existing data set, the specific implementation steps are as follows:

[0062] Step S11, randomly cropping the image to a specified size;

[0063] Step S12, randomly flipping the image horizontally;

[0064] Step S13, converting the image into a PyTorch tensor;

[0065] Step S14, normalizing the tensor using the mean and standard deviation values ​​of a commonly used pre-trained model;

[0066] Step S2, input the image data into a convolution layer, a batch normalization layer and an activation layer to initialize the data. The specific implementation steps are as follows:

[0067] Step S21, preprocessing the image to obtain image data X, and inputting the image data into the convolution layer (101), wherein (101) uses 32 two-dimensional convolution kernels with a feature scale of 3×3 and a convolution step of 2 to perform convolution operation on the input data, and obtains 32 feature data X′ with the image scale reduced to one quarter;

[0068] Step S22: batch normalize the feature data X′ and input it into a ReLu6 activation layer to obtain feature data X′ p ;

[0069] Step S3, input the initialized data into the multi-layer convolution layer of the inverse residual result for feature extraction and multi-dimensional information encoding for three-dimensional feature information fusion; input the extracted features into two layers of convolution layers and one layer of multi-dimensional fusion convolution connected by the inverse residual for information fusion of the correlation between local image features and global image features. The specific implementation steps are as follows:

[0070] Step S31, combining two convolutional layers with inverted residual connections and one multi-dimensional fusion convolutional module, using the output of the previous module as the input of the next module, and repeating 7 times;

[0071] Further, step S31 includes:

[0072] Step S311: feature data X′ p The input expansion layer and a depth-wise separable convolution layer pass through a convolution layer with an expansion factor of t (t is a variable parameter) and a feature scale of 1×1. The number of output channels is t times the number of input channels, which is called an expansion layer; the feature scale of the depth-wise separable convolution layer is 3×3, the step size is 1 or 2, and the number of groups is equal to the number of input channels of the layer, and χ is obtained;

[0073] Step S312: The feature data χ is passed through a double branch of a "1+3" structure. First, χ passes through the left branch and inputs a Sigmoid activation layer to obtain χ l ;

[0074] Step S313, χ(C×H×W) passes through the right branch, and rotates the χ feature data 90 degrees counterclockwise along the height axis to obtain a (W×H×C) Then the DR-pooling layer is used to reduce The size of the tensor is (2×H×C). Then input a convolution kernel with a feature scale of 7×7 to obtain Then batch normalization is performed and the tensor (1×H×C) is obtained through the sigmoid (σ) activation layer, and finally the tensor (C×H×1) is restored by rotating 90 degrees clockwise along the height axis. The same operation is performed along the width axis to obtain a tensor of (C×1×W) The same operation without rotation gives a tensor of (1×H×W)

[0075] Step S314, the x generated by step 312 l respectively generated in step 313 Multiply, then add the three together and divide by 3;

[0076] Step S315, the results generated by steps 312, 313, and 314 are connected with x in step 311 by residual connection to obtain y, which can be expressed as:

[0077]

[0078] Step S4: Input the information fused features into the adaptive mean pooling layer for feature dimensionality reduction. Input the reduced dimensionality features into the fully connected layer to obtain the final category probability output. The specific implementation steps are as follows:

[0079] Step S41, inputting the multi-dimensional feature fusion data y into an adaptive average pooling layer;

[0080] Step S42, the multi-dimensional feature fusion data y is subjected to random inactivation with a ratio of 0.2, and then input into a fully connected layer, wherein the number of output neurons of the fully connected layer is set to be the number of categories of the target classification task, to obtain the final category probability output.

[0081] Step S5: During the network training process, the network is adjusted using the AdamW optimization algorithm according to the loss function.

[0082] Model parameters, get the final network global optimal solution parameters, the specific implementation steps are as follows:

[0083] Step S51, selecting a cross entropy loss function as the network training loss function to evaluate the gap between the model prediction result and the actual result;

[0084] Step S52, calculate the derivative of the gap according to the loss function, use the AdamW optimization algorithm to iteratively update the network training parameters, and obtain the global optimal parameter result of the fitting.

[0085] Step S6, after the network model training is completed, it can be used in actual image processing classification detection tasks.

[0086] Based on RGB image data, the present invention completes image classification through data preprocessing, image data dimensionality reduction, multi-scale local feature extraction, multi-scale information fusion and fully connected layer mapping, and simply and quickly classifies various items. It integrates spatial attention, channel attention, channel pooling, dimensionality reduction pooling (DR-pooling), and convolution operations across multiple dimensions in a unified framework. Compared with existing methods, this strategy not only reduces computational redundancy, but also captures interdependencies in a more comprehensive and efficient way, thereby providing a more effective feature enhancement mechanism.

[0087] To further validate the effectiveness of our approach, experiments were conducted in which various attention strategies were integrated into the standard YOLOv5s backbone and the same default settings were applied in all implementations to evaluate their impact on the object detection performance. The results are summarized in the table.

[0088] The baseline model YOLOv5s has a mean average precision (mAP) of 37.2 with 7.23 million parameters and 16.5 million FLOPs. Augmenting the baseline model using different attention mechanisms yields significant improvements.

[0089] Combined with the (SE) module, the number of parameters increases slightly to 7.27 million, FLOPs increases to 16.6 million, and the mAP is 37.5, an improvement of 0.3.

[0090] Adding (CBAM) further improves mAP to 37.7 with similar parameter and FLOP increments. The (CA) mechanism improves mAP more significantly to 38.0 while keeping almost the same parameters and FLOP counts.

[0091] Finally, we introduce the Multi-Dimensional Class-Dimensional Attention (MDVA) mechanism and achieve the highest mAP 38.3, again with no additional parameters or FLOPs cost compared to the baseline.

[0092] These results demonstrate that incorporating the attention mechanism can effectively improve the detection performance of YOLOv5s on the COCO dataset without significantly affecting the computational efficiency.

[0093] Based on the data set MS COCO and the model YOLOv5s, the comparison between the method of the present invention and other methods is shown in the following table:

[0094] Table.Results on MS COCO dataset.

[0095]

[0096] The present invention discloses a multi-dimensional fusion attention mechanism mobile network structure design based on Vim-like structure, which draws on the Vim structure and adopts the "1+3" architecture to embed multi-dimensional information into channel attention, which is called "multi-dimensional fusion VIM-like structure attention mechanism structure". Different from the traditional channel attention that simplifies the feature tensor into a single feature vector through two-dimensional global pooling, this attention mechanism divides the input into two main branches. The left branch retains the original information for subsequent feature screening, and the right branch divides the channel attention into three one-dimensional feature encoding processes, which aggregate features in one channel direction and two spatial directions respectively. This method captures long-range dependencies and retains precise location information at the same time. Then, the resulting feature map is combined with the left branch to generate an attention map with direction awareness, position sensitivity and channel awareness.

[0097] The multi-dimensional Vim-like attention module proposed in this design has a simple structure and can be seamlessly integrated into classic mobile networks such as MobileNetV2 and ShuffleNetV2, with extremely low computational overhead. Experimental results show that the attention module has a small number of parameters, is suitable for mobile networks, and performs well on the CIFAR-100 dataset. Compared with the existing attention mechanism, this method not only improves the model performance, but also effectively handles the interaction problem of position information and channel information while maintaining low computational cost.

[0098] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Claims

1. A multi-dimensional structural attention model, characterized by: It includes a feature data processing module and a dual-branch module with a "1+3" structure; wherein the feature data processing module normalizes the input feature data; the dual-branch module with a "1+3" structure includes a first branch composed of a Sigmoid activation layer, and a second branch composed of channel dimension, width dimension and height dimension feature extraction modules; the normalized feature data are input to the first branch and the second branch respectively, the data processed by the first branch are multiplied with the data processed by the three parts of the second branch respectively, and the multiplication results are added to obtain tensor data as output.

2. The multidimensional structural attention model according to claim 1, characterized in that: Feature data The specific implementation process of the processing module is as follows: The feature data X p ′Input expansion layer and a depth-wise separable convolutional layer, through a convolutional layer with an expansion factor of t and a feature scale of 1×1, obtain an expansion layer with an output channel number t times the number of input channels; the feature scale of the depth-wise separable convolutional layer is 3×3, the step size is 1 or 2, and the number of groups is equal to the number of input channels of this layer, and we get χ(C×H×W).

3. The multidimensional structural attention model according to claim 2, characterized in that: The χ(C×H×W) passes through the first branch and inputs a Sigmoid activation layer to obtain χ l .

4. The multidimensional structural attention model according to claim 3, characterized in that: The processing process of the second branch includes the following steps: Step 1: Rotate the χ(C×H×W) feature data 90 degrees counterclockwise along the height axis to obtain a W×H×C feature data. Step 2: Reduce through DR-pooling layer The size of the tensor is 2×H×C. Step 3: Input a convolution kernel with a feature scale of 7×7 to obtain Step 4: After batch normalization, a 1×H×C tensor is obtained through a sigmoid (σ) activation layer; Step 5: Rotate 90 degrees clockwise along the height axis to restore the tensor to C×H×1 Operate along the width axis to get a C×1×W tensor Without rotation, we get a 1×H×W tensor 5. The multidimensional structural attention model according to claim 4, characterized in that: By l Respectively Multiply, then add the three and divide by 3, and perform residual connection with χ(C×H×W) to obtain the output of the model.

6. The multidimensional structural attention model according to claim 5, characterized in that: The output of the model is expressed by the following formula:

7. A mobile network structure design method based on a multi-dimensional structural attention model, characterized by: The steps include: Step S1, data preprocessing, preprocessing the image data according to the quality of the existing data set; Step S2, data dimensionality reduction, input the image data into a convolution layer, a batch normalization layer and an activation layer to initialize the data; Step S3, inverse residual convolution structure and feature fusion, input the initialized data into the multi-layer convolution layer of the inverse residual result for feature extraction and multi-dimensional information encoding for three-dimensional feature information fusion; input the extracted features into two layers of convolution layers and one layer of multi-dimensional fusion convolution of the inverse residual connection for information fusion of the correlation between local image features and global image features; Step S4, feature dimension reduction and full connection layer operation, input the information fused features into the adaptive mean pooling layer for feature dimension reduction, and input the dimension reduced features into the full connection layer to obtain the final category probability output; Step S5, model training: during the network training process, the network model parameters are adjusted according to the loss function through the AdamW optimization algorithm to obtain the final network global optimal solution parameters; Step S6: After the network model training is completed, it can be used in actual image processing classification detection tasks.

8. The method for designing a mobile network structure based on a multi-dimensional structural attention model according to claim 7, characterized in that: The specific process of preprocessing the image data in step S1 is as follows: Step S11, randomly cropping the image to a specified size; Step S12, randomly flipping the image horizontally; Step S13, converting the flipped image into a PyTorch tensor; Step S14, normalize the tensor using the mean and standard deviation values ​​of the commonly used pre-trained model to obtain image data X.

9. The method for designing a mobile network structure based on a multi-dimensional structural attention model according to claim 8, characterized in that: The specific process of step S2 is as follows: Step S21, the image data X obtained by preprocessing is input into the convolution layer for convolution operation to obtain 32 feature data X′ whose image scale is reduced to one quarter, wherein the convolution layer includes 32 two-dimensional convolution kernels with a feature scale of 3×3 and a convolution step size of 2; Step S22: batch normalize the feature data X′ and input it into a ReLu6 activation layer to obtain feature data X′ p .

10. The method for designing a mobile network structure based on a multi-dimensional structural attention model according to claim 8, characterized in that: The step S4 specifically includes the following steps: Step S41, inputting the multi-dimensional feature fusion data y into the adaptive average pooling layer; Step S42, the multi-dimensional feature fusion data y is subjected to a random inactivation with a ratio of 0.2, and then input into a fully connected layer, wherein the number of output neurons of the fully connected layer is set to be the number of categories of the target classification task, to obtain the final category probability output.

Images