A fine-grained vehicle model recognition method based on pixel-level spatial perception attention

Abstract

The present invention belongs to the field of vehicle recognition technology and relates to a fine-grained vehicle model recognition method based on pixel-level spatial perception attention. First, a pixel-level spatial perception attention module is established to extract finer attention areas. The spatial perception attention module includes three parts: feature capture, feature fusion, and feature excitation. Second, the spatial perception attention module established in the previous step is inserted into a convolutional neural network to form a convolutional neural network including the spatial perception attention module. The fine-grained vehicle model recognition method based on the pixel-level spatial perception attention module proposed in the present invention improves the performance of the neural network algorithm through feature capture, feature fusion, and feature excitation, and increases the accuracy of fine-grained vehicle model recognition to more than 90%.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle recognition and relates to a fine-grained vehicle model recognition method based on pixel-level spatial perception attention. Background Art

[0002] In recent years, with the rapid development of computer vision and intelligent transportation technologies, fine-grained vehicle type recognition (FVGA) has attracted widespread attention. FVGA is an important and representative research branch of fine-grained object recognition. Unlike coarse-grained vehicle type recognition (VGR) for cars, trucks, buses, etc., FVGA aims to identify more refined attributes of a vehicle based on its appearance, including manufacturer, specific model, and year of production. Because it can provide more detailed vehicle attribute information, FVGA has enormous application potential in various fields, such as intelligent transportation, security monitoring, and parking access control systems. Furthermore, FGVR technology can improve the accuracy and efficiency of image-based vehicle retrieval, thereby opening up a wider range of application scenarios. Currently, most mainstream vehicle websites use keyword-based vehicle search. However, this approach may only be suitable for consumers familiar with car brands. For those unfamiliar with car brands, it can be difficult to find the vehicle they like. Therefore, using FGVR technology for precise search can significantly improve the user experience. Overall, FGVR has great potential in various applications and is a promising research topic. Fine-grained vehicle recognition is a technology that distinguishes vehicles at a finer level based on their visual appearance. It focuses more on the subtle visual differences between vehicles. Figure 1 As shown in the figure, the upper left model (a) is the "Audi S6 Sedan 2011" and the lower left model (c) is the "Audi S5 Coupe 2012." Although both belong to the same brand and have similar appearances, they are from different models, which highlights the necessity of fine-grained classification based on local discriminative regions. Similarly, the upper right model (b) is the "Bentley Continental GT Coupe 2012" and the lower right model (d) is the "Bentley Continental GT Coupe 2007." Although they belong to the same brand and have similar appearances, detailed visual information is required for correct classification. In addition, there are many other types of challenges, such as those caused by the background environment, shooting angle, lighting, and other factors, which affect the visual appearance of the vehicle and increase the difficulty of recognition. Therefore, fine-grained vehicle recognition needs to focus on capturing more refined local features to distinguish subcategories.

[0003] However, traditional fine-grained vehicle model recognition algorithms rely on extracting image features such as color, texture, gradient, shape, and spatial relationships to roughly locate candidate target vehicles. Classifiers such as SVM and Adaboost are then used to identify the vehicle model. However, these handcrafted features often fail to capture subtle differences between vehicle models, making it difficult to construct a sufficiently robust image representation, resulting in consistently poor recognition performance.

[0004] In recent years, convolutional neural networks (CNNs) have demonstrated significant advantages over competing methods in numerous visual tasks, including image classification, object detection, and image segmentation. Currently, fine-grained vehicle model recognition algorithms based on CNNs fall into three main categories: those based on local region detection, those based on attention mechanisms, and those based on local feature learning. Local region detection-based fine-grained vehicle model recognition algorithms rely on additional information, such as manually annotated vehicle bounding boxes and component annotations, and are therefore not widely used. However, fine-grained vehicle model recognition algorithms based on attention mechanisms and local feature learning aim to accurately classify various vehicle models using only class labels, and are currently a research trend. Attention-based methods utilize different attention strategies to derive a data distribution of attention weights. These methods employ a "dynamic weighting" approach to enable the CNN to automatically focus on subtle differences between vehicles, thereby achieving fine-grained vehicle model recognition.

[0005] While current fine-grained vehicle model recognition algorithms based on attention mechanisms use different methods to obtain weight distributions, they are all constructed based on raw feature maps extracted by deep convolutional neural networks. Under the supervision of class labels, raw feature maps typically focus on foreground objects. However, for fine-grained vehicle targets, focusing solely on foreground vehicles is insufficient to distinguish between different models. Therefore, a vehicle model recognition method with better recognition performance is needed to address this issue. Summary of the Invention

[0006] The technical solution adopted by the present invention to solve the technical problem is: a fine-grained vehicle model recognition method based on pixel-level spatial perception attention, comprising the following steps:

[0007] Step 1: Establish a pixel-level spatial-aware attention module that can extract finer attention regions. The spatial-aware attention module consists of three parts: feature capture, feature fusion, and feature excitation. The spatial-aware attention module learns an attention mask at the same spatial scale as the input image. This not only effectively distinguishes the foreground and background features of the target, but also further highlights the local discriminant region, improving the network's ability to recognize fine-grained categories.

[0008] Step 2: Insert the spatial perception attention module established in step 1 into the convolutional neural network to form a convolutional neural network containing the spatial perception attention module;

[0009] The feature capture is completed by selecting feature maps of different convolution stages from the backbone network for preprocessing to reduce feature redundancy and standardize the size of feature maps. The feature capture includes the following steps:

[0010] Step S1-1: Capture the features of the deep convolutional neural network and reduce the dimension to obtain a feature map;

[0011] Step S1-2: upsampling the feature map after dimensionality reduction;

[0012] Step S1-3: Reduce the dimension of the low-level feature map to obtain the output feature map;

[0013] The feature fusion completes the fusion of high-level and low-level features to obtain pixel-level spatial perception features. The feature fusion includes the following steps:

[0014] Step S2-1: Complete the splicing of high-level features and low-level features; the high-level features F obtained by the capture operation h ′ and low-level features F l ' Perform channel-level splicing to obtain features

[0015] Step S2-2: Use convolution blocks to fuse high-level and low-level features;

[0016] Step S2-3: Use Dropout to reduce the coupling between high-level and low-level features;

[0017] Step S2-4: Output feature map; after a 1×1 convolution, the feature channel is reduced to the number of vehicle types N, that is, the output feature map Where H and W are the height and width of the input image

[0018] The feature excitation completes the excitation of spatial perception features and obtains spatial perception attention weights; the feature excitation includes the following steps:

[0019] Step S3-1: Obtain spatial perception features; upsample the output feature map F′ obtained in step S2-4 to obtain spatial perception features F′ up ;

[0020] Step S3-2: Obtain spatial weights.

[0021] Preferably, in step S1-1, feature capture needs to capture high-level features and low-level features of the deep convolutional neural network; capture high-level features of the deep convolutional neural network and low-level features Use 1×1 convolution to first perform high-level feature F h Perform dimensionality reduction to obtain feature maps Thereby reducing the computational complexity of the network; where H, W, and C represent the height, width, and number of channels of the feature map respectively;

[0022] As the convolutional neural network continues to deepen, the resolution of the extracted features continues to decrease, and the target information contained in the feature map becomes more abstract and contains more image semantic information; although the information contained in the low-level feature map is not sufficient for image recognition, it contains rich edge information that helps to locate the target pixel points; therefore, the first step is to capture the high-level and low-level features of the deep convolutional neural network.

[0023] More preferably, in step S1-2, the feature map after dimensionality reduction is upsampled to obtain the same feature map F as the low-level feature map F. l Same width and height (H h ′=H l , W h ′=W l )’s output feature map

[0024] More preferably, in step S1-2, deconvolution is used to learn the parameters of the transposed matrix through network forward and backward propagation, and the weight parameters of the transposed matrix are initialized using a bilinear interpolation method to better restore the activation responses of each location in the feature map.

[0025] More preferably, in step S1-3, 1×1 convolution is used to perform the low-level feature F l Perform channel dimensionality reduction to obtain the output feature map Because the information of low-level features is only to provide more detailed context information for the generation of spatial perception attention, the number of channels after dimensionality reduction is set to C l ' is less than C l ,It has been verified that the fused feature map preserves more high-level semantic information.

[0026] Preferably, in step S2-2, the feature F is passed through two 3×3 convolutional layers to fully fuse the channel and spatial information of the high and low layers; each 3×3 convolutional layer includes a 3×3 convolution, a batch normalization layer and a ReLU activation layer.

[0027] More preferably, in step S2-3, feature F passes through two 3×3 convolutional layers and then a Dropout layer, and neurons are discarded with a certain probability during network training, thereby reducing the coupling between high- and low-level features and increasing the robustness of network feature extraction.

[0028] Preferably, in step S3-2, the formula is used:

[0029] s=σ(δ(ωF′ up )) (1)

[0030] The spatial perception feature F′ up After a 1×1 convolution, the dimension is reduced to a two-dimensional feature vector with a channel number of 1. Then, a ReLU activation function and a sigmoid function are used to obtain the spatial weight s between [0, 1] of the input image; where: δ is the ReLU activation function and σ is the sigmoid function.

[0031] The beneficial effects of the present invention are:

[0032] The fine-grained vehicle model recognition method based on the pixel-level spatial perception attention module proposed in the present invention improves the performance of the neural network algorithm through feature capture, feature fusion and feature excitation, and increases the accuracy of fine-grained vehicle model recognition to more than 90%. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a visual difference display diagram of different car models based on a fine-grained car model recognition method based on pixel-level spatial perception attention;

[0034] Figure 2 This is a block diagram of the pixel-level spatial perception attention module;

[0035] Figure 3 This is a block diagram of a fine-grained vehicle model recognition algorithm based on a pixel-level spatial perception attention module;

[0036] Figure 4 This is a flowchart of the feature fusion process. DETAILED DESCRIPTION

[0037] The following will provide a clear and complete description of the relevant technologies in the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0038] refer to Figures 1 to 4 , a fine-grained vehicle model recognition method based on pixel-level spatial perception attention, including:

[0039] Step 1: A pixel-level spatial awareness attention module (SAM) is proposed to extract more detailed attention areas. This module consists of three parts: feature capture, feature fusion, and feature excitation. The module block diagram is as follows: Figure 2 shown.

[0040] This module learns attention masks at the same spatial scale as the input image, which can not only effectively distinguish the foreground and background features of the target, but also further highlight the local discriminant area and improve the network's recognition ability for fine-grained categories.

[0041] Step 2: Insert SAM into a variety of classic convolutional neural networks to form a new convolutional neural network containing a spatial attention module. The flow chart of the algorithm is as follows Figure 3 shown.

[0042] In this algorithm, the pixel-level spatial perception attention module SAM consists of three parts: feature capture, feature fusion, and feature excitation:

[0043] Feature capture: This step involves preprocessing the feature maps from the backbone network at different convolutional stages to reduce feature redundancy and normalize the feature map size. This step is divided into three steps.

[0044] S1-1: Capturing the features of deep convolutional neural networks and reducing the dimensionality to obtain feature maps

[0045] As convolutional neural networks deepen, the resolution of extracted features decreases, and the target information contained in the feature maps becomes more abstract, encompassing more image semantics. While low-level feature maps may not contain sufficient information for image recognition, they do contain rich edge information that aids in localizing target pixels. Therefore, the first step is to capture both high-level and low-level features required for deep convolutional neural networks.

[0046] Capturing high-level features of deep convolutional neural networks and low-level features Use 1×1 convolution to first perform high-level feature F h Perform dimensionality reduction to obtain feature maps This reduces the computational effort of the network. (In the above symbolic expressions, H, W, and C represent the height, width, and number of channels of the feature map, respectively; the same applies below.)

[0047] S1-2: Upsampling the feature map after dimensionality reduction

[0048] Perform upsampling on the feature map after dimensionality reduction to obtain the same low-level feature map F l Same width and height (H h ′=H l , W h ′=W l )’s output feature map

[0049] Common upsampling methods involve interpolation and amplification via linear or bilinear transformation functions. This method uses deconvolution to learn the parameters of the transposed matrix through forward and backward propagation of the network. Bilinear interpolation is then used to initialize the weight parameters of the transposed matrix to better restore the activation responses at each location in the feature map.

[0050] S1-3: Reduce the dimension of the low-level feature map to obtain the output feature map

[0051] Use 1×1 convolution to perform the low-level feature F l Perform channel dimensionality reduction to obtain the output feature map Because the information of low-level features is only to provide more detailed context information for the generation of spatial perception attention, the number of channels after dimensionality reduction is set to C′ l Less than C l ,It has been verified that the fused feature map preserves more high-level semantic information.

[0052] Feature fusion: This step completes the fusion of high-level and low-level features to obtain pixel-level spatial perception features. It includes 4 steps, such as Figure 4 shown.

[0053] S2-1 completes the splicing of high-level features and low-level features

[0054] The high-level features F obtained by the capture operation h ′ and low-level features F l ' Perform channel-level splicing to obtain features

[0055] S2-2: Using convolutional blocks to fuse high-level and low-level features

[0056] The feature F is passed through two 3×3 convolutional layers to fully integrate the channel and spatial information of the high and low layers. Each 3×3 convolutional layer contains a 3×3 convolution, a batch normalization layer (BN) and a ReLU activation layer, such as Figure 4 shown.

[0057] S2-3: Use Dropout to reduce the coupling of high- and low-level features

[0058] After feature F passes through two 3×3 convolutional layers, it passes through a Dropout layer. During the network training process, neurons are discarded with a certain probability to reduce the coupling between high-level and low-level features and increase the robustness of the network's feature extraction.

[0059] S2-4: Output feature map

[0060] After a 1×1 convolution, the feature channel is reduced to the number of vehicle categories N, that is, the output feature map Where H and W are the height and width of the input image.

[0061] Feature excitation: This step excites spatial perception features and obtains spatial perception attention weights. This is done in two steps.

[0062] S3-1: Acquiring spatial perception features

[0063] Upsample the output feature map F′ obtained in S2-4 to obtain the spatial perception feature F′ up .

[0064] S3-2: Obtaining spatial weights

[0065] Using formula (1), the spatial perception feature F′ up After a 1×1 convolution, the image is reduced to a two-dimensional feature vector with one channel. It is then activated by a ReLU and a sigmoid function to obtain the spatial weight s between [0, 1] of the input image. Here, δ is the ReLU activation function and σ is the sigmoid function.

[0066] s=σ(δ(ωF′ up )) (1).

[0067] Example

[0068] This example applies the proposed SAM module to three classic deep CNN networks, VGG19, InceptionV1, and ResNet50, to construct novel networks, named VGG19-SAM, InceptionV1-SAM, and ResNet50-SAM. Experiments compared the vehicle model recognition accuracy of the original neural networks, with and without the SAM module, and with and without data augmentation. The comparisons were made using the Stanford Cars dataset, a fine-grained vehicle image dataset, and using VGG19-GAP, a more robust network, instead of VGG19.

[0069]

[0070] Table 1 Ablation experiment results based on the Stanford Cars dataset

[0071] The experimental results are shown in Table 1 above. "SAM" indicates the insertion of a pixel-level perceptual attention module, and "Aug." indicates the use of data augmentation for model training, where "√" indicates the application of the corresponding module and "×" indicates the non-application of the corresponding module. From the results, it can be seen that the proposed fine-grained vehicle model recognition method based on the pixel-level spatial perceptual attention module SAM improves the performance of the three networks, confirming the effectiveness and applicability of the method. It can be observed that before adding the data augmentation operation, the method proposed in this article improves the performance of the three models by 2.4% (VGG19-GAP), 1.7% (InceptionV1) and 1.3% (ResNet50), respectively. After data augmentation, it also brings considerable performance improvements, which are 1.1% (VGG19-GAP), 1.5% (InceptionV1) and 0.7% (ResNet50), respectively.

[0072] The performance of the new network constructed by the proposed SAM module is compared with the baseline results and recent research work of others, as shown in Table 2 below.

[0073]

[0074] Table 2 Performance comparison of recognition accuracy on the CompCars web-nature dataset

[0075] The classification accuracy of the ResNet50-SAM method proposed in this embodiment is 97.9%, the highest among all the methods.

[0076] In summary, the fine-grained vehicle model recognition method based on the pixel-level spatial perception attention module proposed in the present invention improves the performance of the neural network algorithm through feature capture, feature fusion and feature excitation, and increases the accuracy of fine-grained vehicle model recognition to more than 90%. Therefore, the present invention has broad application prospects.

[0077] It should be emphasized that the above are only preferred embodiments of the present invention and do not limit the present invention in any form. Any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A fine-grained vehicle model recognition method based on pixel-level spatial perception attention, characterized by: The steps include: Step 1: Establish a pixel-level spatial-aware attention module that can extract finer attention regions. The spatial-aware attention module consists of three parts: feature capture, feature fusion, and feature excitation. The spatial-aware attention module learns an attention mask at the same spatial scale as the input image. Step 2: Insert the spatial perception attention module established in step 1 into the convolutional neural network to form a convolutional neural network containing the spatial perception attention module; The feature capture is used to complete the preprocessing of feature maps selected from different convolution stages of the backbone network to reduce feature redundancy and standardize the size of feature maps. The feature capture includes the following steps: Step S1-1: Capture the features of the deep convolutional neural network and reduce the dimension to obtain a feature map; Step S1-2: upsampling the feature map after dimensionality reduction; Step S1-3: Reduce the dimension of the low-level feature map to obtain the output feature map; The feature fusion is used to complete the fusion of high-level and low-level features to obtain pixel-level spatial perception features. The feature fusion includes the following steps: Step S2-1: complete the splicing of high-level features and low-level features; Step S2-2: Use convolution blocks to fuse high-level and low-level features; Step S2-3: Use Dropout to reduce the coupling between high-level and low-level features; Step S2-4: output feature map; The feature excitation is used to complete the excitation of spatial perception features and obtain spatial perception attention weights; the feature excitation includes the following steps: Step S3-1: obtaining spatial perception features; Step S3-2: Obtain spatial weights.

2. The fine-grained vehicle model recognition method based on pixel-level spatial perception attention according to claim 1 is characterized in that: In step S1-1, feature capture needs to capture high-level features and low-level features of the deep convolutional neural network; capture high-level features of the deep convolutional neural network and low-level features Use 1×1 convolution to first perform high-level feature F h Perform dimensionality reduction to obtain feature maps Thereby reducing the computational complexity of the network; where H, W, and C represent the height, width, and number of channels of the feature map, respectively.

3. The fine-grained vehicle model recognition method based on pixel-level spatial perception attention according to claim 2 is characterized in that: In step S1-2, the feature map after dimensionality reduction is upsampled to obtain the same feature map F as the low-level feature map F. l Output feature maps with the same width and height 4. The fine-grained vehicle model recognition method based on pixel-level spatial perception attention according to claim 3 is characterized in that: In step S1-2, deconvolution is used to learn the parameters of the transposed matrix through network forward and backward propagation, and the weight parameters of the transposed matrix are initialized using the bilinear interpolation method to better restore the activation responses of each location in the feature map.

5. The fine-grained vehicle model recognition method based on pixel-level spatial perception attention according to claim 3 is characterized in that: In step S1-3, 1×1 convolution is used to transform the low-level features F l Perform channel dimensionality reduction to obtain the output feature map 6. The fine-grained vehicle model recognition method based on pixel-level spatial perception attention according to claim 1 is characterized in that: In step S2-2, the feature F passes through two 3×3 convolutional layers to fully fuse the channel and spatial information of the high and low layers; each 3×3 convolutional layer contains a 3×3 convolution, a batch normalization layer and a ReLU activation layer.

7. The fine-grained vehicle model recognition method based on pixel-level spatial perception attention according to claim 6 is characterized in that: In step S2-3, feature F passes through two 3×3 convolutional layers and then a Dropout layer. During the network training process, neurons are discarded with a certain probability to reduce the coupling between high-level and low-level features and increase the robustness of network feature extraction.

8. The fine-grained vehicle model recognition method based on pixel-level spatial perception attention according to claim 1 is characterized in that: In step S3-2, the formula is used: The spatial perception feature F u ' p After a 1×1 convolution, the dimension is reduced to a two-dimensional feature vector with a channel number of 1. Then, a ReLU activation function and a sigmoid function are used to obtain the spatial weight s between [0, 1] of the input image; where: δ is the ReLU activation function and σ is the sigmoid function.

Images