A vehicle classification method based on multi-branch local attention network
By introducing multi-branch local attention networks into vehicle classification, combining channel and spatial attention modules, the problem of insufficient feature recognition capabilities in vehicle classification is solved, and more accurate feature extraction and classification effects are achieved.
Patent Information
- Application Number
- CN202110881344.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-02
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-08-02
AI Technical Summary
The existing convolutional neural network methods have insufficient feature recognition capabilities in vehicle classification and cannot effectively distinguish key information from redundant information, resulting in insufficient feature extraction, especially in environmental changes and occlusion.
Using a vehicle classification method based on multi-branch local attention network, the channel-based convolutional attention module and space-based local attention module are introduced into the ResNet-50 model, and the branch fusion is combined with the parallel method to enhance feature extraction capabilities. Avoid information loss caused by multiple pooling and convolution.
It improves the accuracy of vehicle classification, enhances the expressiveness of features, and can more accurately obtain different neighborhood information in the feature map, reduces the loss of important information, and improves the feature recognition ability of the model.
Smart Images

Figure CN113610144B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning and computer vision technology, and in particular to a vehicle classification method based on a multi-branch local attention network. Background Art
[0002] In recent years, intelligent transportation systems (ITS) have developed rapidly. With the development of computer vision and deep learning, it has provided an opportunity for more effective application of intelligent transportation systems. Computer vision uses computers to simulate human visual ability, extract information from images of objective objects, process and understand them, and finally apply them to real life, such as vehicle classification.
[0003] Traditional vision-based vehicle classification methods usually rely on manual features, such as color histogram (CH), texture descriptor (TD), GIST) or representations generated by encoding local features such as BoVW, IFK, SPM, etc., which are time-consuming and labor-intensive, have poor generalization ability, and are easily affected by environmental changes and occlusion. With the rapid development of deep learning theory and practice, object detection and classification based on deep learning has entered a new stage. Different from traditional feature extraction algorithms, convolutional neural networks have certain invariance to geometric transformations, deformations, and illumination, which can overcome the difficulties of changing the appearance of vehicles, solve occlusion problems, and can adaptively construct feature descriptions driven by training data, with greater flexibility and comprehensive capabilities.
[0004] The existing convolutional neural network method still has shortcomings in that it does not clearly distinguish between key information and redundant information, and the model's feature recognition ability is still not strong enough. During the multi-layer convolution and pooling, a lot of important information is lost, resulting in the extracted features not being able to represent the target well. Summary of the invention
[0005] The purpose of the present invention is to overcome the shortcomings of the prior art. To achieve the above purpose, a vehicle classification method based on a multi-branch local attention network is adopted to solve the problems raised in the above background technology.
[0006] A vehicle classification method based on a multi-branch local attention network, the specific steps include:
[0007] Obtain vehicle images to be classified and divide them into training sets and test sets;
[0008] Inputting the training set of vehicle images into a vehicle classification model that constructs a multi-branch local attention network for training, wherein the multi-branch local attention network includes a channel-based convolutional attention module and a space-based local attention module;
[0009] According to the trained vehicle classification model, the test set of vehicle images is classified and predicted to obtain the classification results.
[0010] As a further solution of the present invention: the specific steps of obtaining the vehicle images to be classified and dividing them into a training set and a test set include:
[0011] Obtain vehicle scene images and divide them into training sets and test sets, with the ratio of training sets to test sets being 4:1;
[0012] The training set includes M training pictures X={X 1 ,X 2 ,…,X m ,…,X M}, the training set includes N training pictures Y={Y 1 ,Y 2 ,…,Y n ,…,Y N}, where X m represents the mth training picture, Y n Indicates the nth test picture;
[0013] Label the images in the training set and test set Figure 1 Mark one by one.
[0014] As a further solution of the present invention: the training set of vehicle images is input into a vehicle classification model for constructing a multi-branch local attention network for training, and the multi-branch local attention network includes a channel-based convolutional attention module and a space-based local attention module. The specific steps include:
[0015] Construct a multi-branch local attention network structure based on the ResNet-50 model;
[0016] The channel-based convolutional attention module is introduced as the first branch;
[0017] The spatial-based local attention module is introduced as the second branch;
[0018] The two branches are merged using a parallel method, and the fusion formula is:
[0019] F′=σ(M C (F)×M S (F))×F;
[0020] Among them, M c (F) represents channel attention, M s (F) represents spatial attention, σ(*) is the sigmoid activation function;
[0021] The multi-branch local attention network structure is embedded into each bottleneck layer of ResNet-50 to obtain a vehicle classification model based on the multi-branch local attention network.
[0022] As a further solution of the present invention: the specific steps of introducing the channel-based convolutional attention module as the first branch include:
[0023] Global average pooling and global maximum pooling operations are used to aggregate the spatial information of feature maps, namely and
[0024] Using the 2×1 convolution method, and Combine;
[0025] Finally, a multi-layer perceptron is added to learn the final channel attention feature map M c (F), the formula is:
[0026]
[0027] Among them, W 0 ∈R (C / r)×C , W 1 ∈R C , W 0 and W 1 is the weight of the multilayer perceptron MLP, f 2×1 Represents a convolution operation with a filter size of 2×1.
[0028] As a further solution of the present invention: the specific steps of introducing the space-based local attention module as the second branch include:
[0029] Apply local spatial max pooling and average pooling in parallel to all channels of the original feature map, with kernel and stride equal to ε;
[0030] Two compressed spatial attention descriptors F are generated by aggregating features in all ε-neighborhoods of F. M and F A ;
[0031] F A =AvgPool kernel,stride=ε (F);
[0032] F M =MaxPool kernel,stride=ε (F);
[0033] Use global maximum pooling and global average pooling to tighten F along the channel direction M and F A Generate Descriptor and and connect them;
[0034] Then add 3×3 dilated convolution and nearest neighbor interpolation operations in sequence to obtain the feature map of spatial attention;
[0035] M S (F) = σ(f nearest (f([F max (F M );F avg (F A )])));
[0036] Among them, MaxPool(*) and AvgPool(*) represent the local maximum pooling and average pooling operations with kernel and stride ε in the spatial domain, and F max (*) and F avg (*) represents the maximum pooling and average pooling operations along the channel direction, f(*) represents a 3×3 dilated convolution, and f nearest (*) is the nearest neighbor interpolation upsampling operator, and σ(*) is the sigmoid activation function.
[0037] As a further solution of the present invention: the specific steps of performing classification prediction on the test set of vehicle images according to the trained vehicle classification model to obtain the classification results include:
[0038] Set the loss function, adjust the size of the vehicle images in the training set, and then input them into the multi-branch local attention network for training to obtain the trained network model;
[0039] The fully connected layer of the extracted network of the trained network model is used to classify and predict the vehicle images in the test set using the Softmax function to obtain the classification results.
[0040] Compared with the prior art, the present invention has the following technical effects:
[0041] By adopting the above technical solution, we obtain vehicle images and then build a vehicle classification model based on the RestNet-50 model with a multi-branch local attention network, which includes the introduction of a channel-based convolutional attention module and a space-based local attention module, and then extract the network's fully connected layer to use the softmax function for vehicle image prediction and classification. The above method can more accurately obtain information from different neighborhoods in the feature map, enhance the expressiveness of the features, and improve the accuracy of vehicle classification. At the same time, it avoids multiple pooling and convolution, and a large amount of important information is lost, resulting in the problem that the extracted features cannot well represent the target. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] The specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings:
[0043] Figure 1 A schematic diagram of the steps of a vehicle classification method according to some embodiments disclosed in the present application;
[0044] Figure 2 A schematic diagram of a network model flow chart of some embodiments disclosed in this application;
[0045] Figure 3 A schematic diagram of the network structure of some embodiments disclosed in this application;
[0046] Figure 4 A schematic diagram of a channel-based convolutional attention module for some embodiments disclosed in this application;
[0047] Figure 5 A schematic diagram of a spatial-based local attention module according to some embodiments disclosed in this application;
[0048] Figure 6 Schematic diagram of a CNN unit integrated with a multi-branch attention structure of some embodiments disclosed in this application. DETAILED DESCRIPTION
[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0050] Please refer to Figure 1 and Figure 2 In an embodiment of the present invention, a vehicle classification method based on a multi-branch local attention network comprises the following specific steps:
[0051] S1. Obtain the vehicle images to be classified and divide them into training set and test set. The specific steps include:
[0052] Obtain images of the required vehicle in a specific scene and divide them into training and test sets, with a ratio of 4:1.
[0053] The training set includes M training pictures X={X 1 ,X 2 ,…,X m ,…,X M}, the training set includes N training pictures Y={Y 1 ,Y 2 ,…,Y n ,…,Y N}, where X mrepresents the mth training picture, Y n Indicates the nth test picture;
[0054] Label the images in the training set and test set Figure 1 Mark one by one.
[0055] S2. Input the training set of vehicle images into a vehicle classification model that constructs a multi-branch local attention network for training. The multi-branch local attention network includes a channel-based convolutional attention module and a space-based local attention module. The specific steps include:
[0056] S21. Construct a multi-branch local attention network structure based on the ResNet-50 model; specifically, Figure 3 As shown in the figure, the framework structure diagram of the vehicle image classification model combines the classic CNN model with the attention mechanism, and sets the output dimension of the classification model to the total number of vehicle categories. The convolutional neural network ResNet-50 is used as the backbone network of the classification model to extract the original feature map from the vehicle image.
[0057] S22. A channel-based convolutional attention module is introduced as the first branch to calculate the weights of the feature map channels, and the feature map after updating the channel attention is obtained based on the calculated weights of the feature map channels and the original feature map.
[0058] In the specific implementation of step S22, Figure 4 As shown, the specific steps of introducing the channel-based convolutional attention module as the first branch include:
[0059] First, global average pooling and global maximum pooling operations are used to aggregate the spatial information of the feature map, that is, to obtain two 1×1×c channels and They represent average pooling and maximum pooling respectively.
[0060] Then use the 2×1 convolution method to and Combine;
[0061] Finally, a multi-layer perceptron is added to learn the final channel attention feature map M c (F), in order to reduce parameter overhead, the size of the hidden layer is set to C / r, where r is the compression ratio. Specifically, it is found through experiments that setting the compression ratio r to 16 can obtain better results. The specific formula is:
[0062]
[0063] Among them, W 0 ∈R (C / r)×C , W1 ∈R C , W 0 and W 1 is the weight of the multilayer perceptron MLP, f 2×1 Represents a convolution operation with a filter size of 2×1.
[0064] S23. A space-based local attention module is introduced as the second branch to calculate the similarity between each feature in the feature map and other features, and obtain the feature map after updating the spatial attention.
[0065] In the specific implementation of step S23, Figure 5 As shown, the specific steps of introducing the space-based local attention module as the second branch include:
[0066] Local similarity (spatial local pooling) is used to achieve spatial attention, and the similarity calculation of non-adjacent image regions is achieved by calculating the similarity between each feature in the feature map and other features.
[0067] Apply local spatial max pooling and average pooling in parallel to all channels of the original feature map, with kernel and stride equal to ε;
[0068] Two compressed spatial attention descriptors F are generated by aggregating features in all ε-neighborhoods of F. M and F A ;
[0069] F A =AvgPool kernel,stride=ε (F);
[0070] F M =MaxPool kernel,stride=ε (F);
[0071] Use global maximum pooling and global average pooling to tighten F along the channel direction M and F A Generate descriptors and and Each feature represents the maximum pooled feature and average pooled feature across channels. These feature descriptors are concatenated to reduce computational cost;
[0072] In order to reduce the feature loss caused by multi-step pooling operations, a 3×3 dilated convolution is added to further learn features and improve the nonlinear representation ability of the spatial attention descriptor. Finally, upsampling is performed through nearest neighbor interpolation to obtain the spatial attention feature map M of the same scale as the original input image. S (F);
[0073] M S (F) = σ(fnearest (f([F max (F M );F avg (F A )」)));
[0074] Among them, MaxPool(*) and AvgPool(*) represent the local maximum pooling and average pooling operations with kernel and stride ε in the spatial domain, and F max (*) and F avg (*) represents the maximum pooling and average pooling operations along the channel direction, f(*) represents a 3×3 dilated convolution, and f nearest (*) is the nearest neighbor interpolation upsampling operator, and σ(*) is the sigmoid activation function.
[0075] S24, the two branches are merged using a parallel method, such as Figure 6 As shown, the feature map after updating the channel attention outputted above and the feature map after updating the spatial attention outputted above are connected in parallel and embedded into each layer of the backbone network;
[0076] The fusion formula is:
[0077] F′=σ(M C (F)×M S (F))×F;
[0078] Among them, M c (F) represents channel attention, M s (F) represents spatial attention, and σ(*) is the sigmoid activation function; the parallel method has fewer activation functions than the sequential method, resulting in a larger representation range and stronger feature extraction ability.
[0079] The multi-branch local attention network structure is embedded into each bottleneck layer of ResNet-50 to obtain a vehicle classification model based on the multi-branch local attention network.
[0080] S3. Classify and predict the test set of vehicle images according to the trained vehicle classification model to obtain the classification results. The specific steps include:
[0081] S31, setting a loss function, adjusting the size of the vehicle image in the training set and inputting it into a multi-branch local attention network for training, and obtaining a trained network model;
[0082] Specifically, the size of the vehicle image is adjusted to 224 pixels×224 pixels, and then the pixels of the image are normalized and used as the input of the classification model. The loss function is a cross entropy loss function.
[0083] During training, the parameters of the CNN model pre-trained by ImageNet are loaded, but the parameters of the classification model are not frozen. Fine-tuning is performed on this basis, for example, gradient descent is used to fine-tune all parameters of the backbone network.
[0084] S32. Use the fully connected layer of the extracted network of the trained network model to classify and predict the vehicle images in the test set using the Softmax function to obtain the classification results.
[0085] In some specific embodiments, the backbone network in step S2 is ResNet-50, and the backbone network can also be other networks, such as AlexNet, VGGNet, GoogLeNet or DenseNet.
[0086] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents, and all should be included within the scope of protection of the present invention.
Claims
1. A vehicle classification method based on a multi-branch local attention network, characterized in that: The specific steps include: Obtain vehicle images to be classified and divide them into training sets and test sets; The training set of vehicle images is input into a vehicle classification model that constructs a multi-branch local attention network for training. The multi-branch local attention network includes a channel-based convolutional attention module and a space-based local attention module. The specific steps include: Construct a multi-branch local attention network structure based on the ResNet-50 model; The channel-based convolutional attention module is introduced as the first branch, and its specific steps include: Global average pooling and global maximum pooling operations are used to aggregate the spatial information of feature maps, namely and Using the 2×1 convolution method, and Combine; Finally, a multi-layer perceptron is added to learn the final channel attention feature map M c (F), the formula is: Among them, W0∈R (C / r)×C , W1∈R C , W0 and W1 are the weights of the multilayer perceptron MLP, f 2×1 represents a convolution operation with a filter size of 2×1; The spatial-based local attention module is introduced as the second branch, and its specific steps include: Apply local spatial max pooling and average pooling in parallel to all channels of the original feature map, with kernel and stride equal to ε; Two compressed spatial attention descriptors F are generated by aggregating features in all ε-neighborhoods of F. M and F A ; F A =AvgPool kernel,stride=ε (F); F M =MaxPool kernel,stride=ε (F); Use global maximum pooling and global average pooling to tighten F along the channel direction M and F A Generate descriptors and and and connect them; Then add 3×3 dilated convolution and nearest neighbor interpolation operations in sequence to obtain the feature map of spatial attention; M S (F)=σ(f nearest (f([F max (F M );F avg (F A )]))); Among them, MaxPool(*) and AvgPool(*) represent the local maximum pooling and average pooling operations with kernel and stride ε in the spatial domain, and F max (*) and F avg (*) represents the maximum pooling and average pooling operations along the channel direction, f(*) represents a 3×3 dilated convolution, and f nearest (*) is the nearest neighbor interpolation upsampling operator, σ(*) is the sigmoid activation function; The two branches are merged using a parallel method, and the fusion formula is: F′=σ(M C (F)×M S (F))×F; Among them, M c (F) represents channel attention, M s (F) represents spatial attention, σ(*) is the sigmoid activation function; The multi-branch local attention network structure is embedded into each bottleneck layer of ResNet-50 to obtain a vehicle classification model based on the multi-branch local attention network. According to the trained vehicle classification model, the test set of vehicle images is classified and predicted to obtain the classification results.
2. According to claim 1, a vehicle classification method based on a multi-branch local attention network is characterized in that: The specific steps of obtaining the vehicle images to be classified and dividing them into a training set and a test set include: Obtain vehicle scene images and divide them into training sets and test sets, with the ratio of training sets to test sets being 4:1; The training set includes M training pictures X={X1,X2,…,X m ,…,X M }, the training set includes N training pictures Y = {Y1, Y2, ..., Y n ,…,Y N }, where X m represents the mth training picture, Y n Represents the nth test picture; The images in the training set and the test set are labeled one-to-one with the original images.
3. The vehicle classification method based on a multi-branch local attention network according to claim 1 is characterized in that: The specific steps of performing classification prediction on the test set of vehicle images according to the trained vehicle classification model to obtain the classification results include: Set the loss function, adjust the size of the vehicle images in the training set, and then input them into the multi-branch local attention network for training to obtain the trained network model; The fully connected layer of the extracted network of the trained network model is used to classify and predict the vehicle images in the test set using the Softmax function to obtain the classification results.
