Vehicle target detection method based on attribute information and large basic model

Abstract

This invention discloses a vehicle target detection method based on attribute information and a large-scale base model. The method involves a backbone network acquiring the original features of an image; a region proposal module generating candidate regions; a region of interest alignment module aligning the candidate regions with the original feature map to obtain visual features; cropping candidate region images from the original image based on the candidate regions; extracting features from each candidate region image using a visual feature enhancement module to obtain enhanced features; concatenating the visual features with the enhanced features to obtain the final visual representation; and using a loss constraint to make the visual and semantic representations more similar, thereby aligning the visual and semantic representations to compensate for semantic differences. This invention enhances the original image features extracted by the backbone network through a visual feature enhancement module, fuses and unifies the attribute vector groups of candidate regions through an attribute representation learning module, and finally uses a contrastive learning method to align the visual and semantic representations to compensate for semantic differences.

Description

Technical Field

[0001] The present invention relates to vehicle target detection and deep learning technology, and in particular to a vehicle target detection method based on attribute information and a large basic model. Background Art

[0002] Vehicle target detection aims to detect the location and category of vehicle targets from a given input image. Currently, many target detectors can be used for vehicle detection, such as Mask RCNN, YoLoV8, and ViTDet. Although good performance can be achieved using these detectors, the detection performance of these detectors is still unsatisfactory in challenging scenarios such as darkness, occlusion, and strong light. Some researchers use multimodal data to introduce additional information to solve these problems. However, the popularity of multimodal devices is still low, which limits the scope of use of their methods. Other researchers use basic models pre-trained on large-scale data to solve these problems, but the general pre-trained basic models are generally pre-trained on conventional datasets, and the results on vehicle target detection tasks are suboptimal. At the same time, traditional vehicle detection algorithms mainly rely on feature representations obtained through large visual models, while ignoring the semantic differences between the semantic label information of the vehicle and the visual representation, which also limits the performance of vehicle target detection.

[0003] At present, the object detection methods based on deep learning can be roughly divided into two directions, namely one-stage detectors and two-stage detectors. For example, Dai Xiyang et al. proposed an end-to-end object detection model based on the Transformer architecture. First, the image features are extracted through a convolutional neural network, these features are straightened and position encoded, then the target query and sequence features are input into the Transformer encoder to generate a set of prediction boxes, and finally the predictions are paired with the real targets through the Hungarian algorithm to obtain the detection results. For example, Kaiming He et al. proposed a deep learning model for object detection and instance segmentation. First, the image features are extracted through the backbone network, and a region proposal module is used to generate several candidate regions from the input image. Then, the feature maps of the candidate regions are aligned through the region of interest alignment module. Finally, the aligned features are input into the prediction head to predict the category, position and segmentation mask of the target. The multi-task loss function is used to optimize the model parameters during the training process.

[0004] However, the above-mentioned existing target detection methods are mainly based on general pre-trained basic models. Such basic models lack vehicle-related data in the pre-training stage, so the results on vehicle detection tasks are suboptimal. In addition, the existing target detection methods mainly rely on large pure visual basic models, ignoring the semantic differences between the semantic label information of the target and the visual representation. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to solve the deficiencies in the prior art and provide a vehicle target detection method based on attribute information and a large basic model, in order to solve the problem that the existing detection algorithm ignores the semantic difference between the target semantic information and the visual representation, and the general pre-trained basic model has limited effect on the vehicle target detection task. The present invention proposes a vehicle target detection framework based on attribute information and a large basic model, aiming to achieve more accurate vehicle target detection.

[0006] Technical solution: A vehicle target detection method based on attribute information and a large basic model of the present invention comprises the following steps:

[0007] Step 1: Get the original image and input it into the backbone network (ResNet50), and extract the original feature map from the backbone network;

[0008] Step 2: For the original feature map obtained in step 1, first use the region proposal module to generate a set of candidate regions, and then use the region of interest alignment module to pool the position coordinates of the candidate regions in the corresponding regions of the original feature map, and pool them into a fixed size to obtain the visual feature F. r ;

[0009] For a set of candidate regions generated by the region proposal module, the candidate region images are cropped from the original image to obtain a set of candidate region images;

[0010] Step 3: For the candidate region image set obtained in step 2, the visual feature enhancement module is used to extract the features of each candidate region image, and then the original feature map obtained in step 1 is enhanced to obtain the enhanced features corresponding to the candidate region image.

[0011] Step 4: Use the attribute representation learning module to obtain the attribute representation of the candidate region. The specific method is as follows:

[0012] Step 4.1: Use the large language model T5 to encode the attribute words of the vehicle to obtain text features;

[0013] Step 4.2: Input the enhanced features and text features into the pre-trained attribute prediction network to predict the attribute information corresponding to each candidate region image, that is, the predicted attribute of the candidate region;

[0014] Step 4.3: Convert the attribute features of each candidate region into a unified semantic attribute representation V through a gated recurrent unit a , i.e., the attribute vector;

[0015] Step 5: Transform the visual feature F r With enhanced features Splice along the channel to obtain the final visual representation F v , the final visual representation F v Input to the classification head and regression head to obtain the detection results;

[0016] Based on the idea of ​​contrastive learning, visual features and semantic attribute representations are aligned to make up for the semantic differences and further improve the performance of vehicle target detection tasks. To make up for the semantic differences between visual representations and semantic labels,

[0017] Then, the final visual representation F is calculated for each candidate region v and semantic attribute representation V a The cosine embedding loss between them constrains the visual representation and the semantic representation to be more similar, thereby aligning the visual representation and the semantic representation to make up for the semantic difference.

[0018] Furthermore, the visual feature enhancement module includes a pre-trained vehicle basic model encoder and a linear projection layer. After the image in the candidate area image set is input into the visual feature enhancement module,

[0019] First, each candidate region image is deformed into I∈R 224×224×3 , and I∈R 224×224×3 Divide into 196 non-overlapping blocks P i ∈R 16×16×3 After the 196 blocks are input to the visual feature enhancement module, the P i Projection as Token Embedding Where i refers to the sequence number of the block, i∈{1,2,…,196};

[0020] Then, since the encoder parameters of the pre-trained vehicle base model are frozen, a set of learnable tokens K∈R is introduced for fine-tuning 8×768 , thus achieving more effective fine-tuning, after integrating the global semantic tokens and learnable tokens, the tokens are embedded into P emb ∈R 205×768 Input to the pre-trained vehicle base model encoder and introduce the corresponding position encoding The pre-trained vehicle base model encoder includes 12 Transformer blocks, each of which includes layer normalization, multi-head self-attention, and multi-layer perceptron; here, P emb Including multiple and global semantic tokens and learnable tokens;

[0021] Finally, the pre-trained vehicle base model encoder outputs the feature F c ∈R 205×768 , the feature F c ∈R205×768 Projected through a 256-dimensional linear projection layer

[0022] Furthermore, in step 4, there are six groups of attribute words for the vehicle, including color, model, displacement, number of doors, number of seats and maximum speed, with a total of 47 attribute tags.

[0023] Furthermore, the specific method of step 4 is:

[0024] Input the attribute words with attribute labels into the pre-trained large language model to obtain the text embedding of the attribute labels Embed the resulting text Splicing to get F a ∈R 47×768 ; Then F a ∈R 47×768 Enhanced features output from the encoder with the pre-trained vehicle base model After connection, it is input into the attribute prediction module; here, j refers to the serial number of the attribute label;

[0025] Then, the concatenated features are input into a layer of Transformer fusion block for fusion, and the text tokens in the output features are input into the attribute prediction head to obtain the predicted attributes of the candidate region; based on the predicted attributes, the attribute label with the highest probability is selected from each attribute group, and the corresponding text embedding is extracted Get the attribute vector A = [a 1 ,a 2 ,a 3 ,a 4 ,a 5 ,a 6 ];

[0026] In order to obtain a unified semantic attribute representation, a gated recurrent unit module is used to fuse the attribute vector A = [a 1 ,a 2 ,a 3 ,a 4 ,a 5 ,a 6 ], hiding each attribute embedding as an element in the input sequence, and thus using the gated recurrent unit module to effectively integrate these embeddings into a unified visual representation V a ∈R 1×256 .

[0027] Furthermore, to obtain a unified semantic attribute representation, we use a gated recurrent unit module to fuse attribute embeddings. The gated recurrent unit is a recurrent neural network that contains two main gates, namely the update gate and the reset gate. Specifically, the reset gate is used to calculate the candidate hidden state, and the update gate is used to retain important information from the previous hidden state and the candidate hidden state to obtain the current hidden state. The gated recurrent unit module is very effective in capturing dependencies in long sequences. We regard each attribute embedding as an element in the input sequence, and thus use the gated recurrent unit module to effectively integrate these embeddings into a unified visual representation V. a ∈R 1×256 Furthermore, the specific method of the visual text contrast learning is:

[0028] First, the visual representation is projected to the same dimension as the text representation through a linear projection layer, and then the visual representation and text representation are L2 normalized respectively. Finally, for each candidate region, the normalized visual representation and text representation are used to calculate the cosine embedding loss;

[0029] The visual-text contrastive learning process is constrained by the following formula:

[0030]

[0031] Among them, CEL is the abbreviation of cosine embedding loss function, F v is the visual representation, V a is a text representation, refers to the normalized visual representation, It refers to the normalized text representation, N refers to the number of images processed in a batch; t refers to the sequence number of the image to be processed.

[0032] Beneficial effects: The vehicle target detection method based on attribute information and a large base model of the present invention enhances the original candidate region features by introducing a large vehicle base model encoder, and designs an attribute representation learning module to compensate for semantic differences by aligning visual features and semantic representations through comparative learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the overall network framework of the present invention;

[0034] Figure 2 This is a schematic diagram of the attribute representation learning module of the present invention. DETAILED DESCRIPTION

[0035] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0036] like Figure 1 and Figure 2 As shown, the vehicle target detection method based on attribute information and a large basic model implemented in this embodiment includes the following steps:

[0037] Step 1: Get the original image and input it into the backbone network (ResNet50), and extract the original feature map from the backbone network;

[0038] Step 2: For the original feature map obtained in step 1, first use the region proposal module to generate a set of candidate regions, and then use the region of interest alignment module to pool the position coordinates of the candidate regions in the corresponding regions in the original feature map obtained in step 1, and pool them into a fixed size (i.e., aligned) to obtain the visual feature F r ;

[0039] For a set of candidate regions generated by the region proposal module, the candidate region images are cropped from the original image to obtain a set of candidate region images;

[0040] Step 3: For the candidate region image set obtained in step 2, the visual feature enhancement module is used to extract the features of each candidate region image, and then the original feature map obtained in step 1 is enhanced to obtain the enhanced features corresponding to the candidate region image.

[0041] Step 4: Use the attribute representation learning module to obtain the attribute representation of the candidate region. The specific method is as follows:

[0042] Step 4.1: Use the large language model T5 to encode the attribute words of the vehicle to obtain text features;

[0043] Step 4.2: Input the enhanced features and text features into the pre-trained attribute prediction network to predict the attribute information corresponding to each candidate region image, that is, the predicted attribute of the candidate region;

[0044] Step 4.3: Convert the attribute features of each candidate region into a unified semantic attribute representation V through a gated recurrent unit a , i.e., the attribute vector;

[0045] Step 5: Transform the visual feature F r With enhanced features Splice along the channel to obtain the final visual representation F v , the final visual representation F v Input to the classification head and regression head to obtain the detection results;

[0046] Then, the final visual representation F is calculated for each candidate region v and semantic attribute representation V aThe cosine embedding loss between them constrains the visual representation and the semantic representation to be more similar, thereby aligning the visual representation and the semantic representation to make up for the semantic difference.

[0047] This embodiment adopts the PyTorch framework. The overall network structure of the present invention includes three parts, namely, a visual feature enhancement module, an attribute representation learning module, and a visual text contrast learning module.

[0048] The visual feature enhancement module of this embodiment includes a pre-trained vehicle basic model encoder and a linear projection layer. After the image in the candidate area image set is input into the visual feature enhancement module,

[0049] First, each candidate region image is deformed into I∈R 224×224×3 , and I∈R 224×224×3 Divide into 196 non-overlapping blocks P i ∈R 16×16×3 After the 196 blocks are input to the visual feature enhancement module, the P i Projection as Token Embedding Where i refers to the sequence number of the block, i∈{1,2,…,196};

[0050] Then, a set of learnable tokens K∈R is introduced for fine-tuning 8×768 , thus achieving more effective fine-tuning, after integrating the global semantic tokens and learnable tokens, the tokens are embedded into P emb ∈R 205×768 Input to the pre-trained vehicle base model encoder and introduce the corresponding position encoding The pre-trained vehicle base model encoder includes 12 Transformer blocks, each of which includes layer normalization, multi-head self-attention, and multi-layer perceptron; here, P emb Including multiple and global semantic tokens and learnable tokens;

[0051] Finally, the pre-trained vehicle base model encoder outputs the feature F c ∈R 205×768 , the feature F c ∈R 205×768 Projected through a 256-dimensional linear projection layer

[0052] The main contents of the attribute representation learning module are as follows:

[0053] In this embodiment, there are six groups of vehicle attribute words, including color, model, displacement, number of doors, number of seats and maximum speed, with a total of 47 attribute tags.

[0054] First, input the attribute words with attribute labels into the pre-trained large language model to obtain the text embedding of the attribute labels. Embed the resulting text Splicing to get F a ∈R 47×768 ; Then F a ∈R 47×768 Enhanced features output from the encoder with the pre-trained vehicle base model After connection, it is input into the attribute prediction module, where j refers to the serial number of the attribute label;

[0055] Then, the concatenated features are input into a layer of Transformer fusion block for fusion (here the Transformer fusion block allows unlimited interaction between the input visual text features, and deeply models the correlation within and across modalities. The output features will contain rich visual text information, which is conducive to subsequent attribute prediction). The text tokens in the output features are input into the attribute prediction head to obtain the predicted attributes of the candidate region; based on the predicted attributes, the attribute label with the highest probability is selected from each attribute group, and the corresponding text embedding is extracted. Get the attribute vector A = [a 1 ,a 2 ,a 3 ,a 4 ,a 5 ,a 6 ];

[0056] Next, a gated recurrent unit is used to fuse the attribute vector A = [a 1 ,a 2 ,a 3 ,a 4 ,a 5 ,a 6 ]Get V a ∈R 1×256 .

[0057] The gated recurrent unit includes an update gate and a reset gate. The reset gate calculates a candidate hidden state, and the update gate retains important information from a previous hidden state and the candidate hidden state to obtain a current hidden state.

[0058] The specific method of visual text contrast learning in this embodiment is:

[0059] First, the visual representation is projected to the same dimension as the text representation through a linear projection layer, and then the visual representation and text representation are L2 normalized respectively. Finally, for each candidate region, the normalized visual representation and text representation are used to calculate the cosine embedding loss;

[0060] The visual-text contrastive learning process is constrained by the following formula:

[0061]

[0062] Among them, CEL is the abbreviation of cosine embedding loss function, F v is the visual representation, V a is a text representation, refers to the normalized visual representation, It refers to the normalized text representation, N refers to the number of images processed in a batch; t refers to the sequence number of the image to be processed.

[0063] This application compares the performance of the proposed algorithm with the most advanced detection algorithms on three vehicle target detection datasets to evaluate the proposed algorithm and verify the effectiveness of the proposed method (Tables 1, 2, 3, and 4). This application selects the most commonly used evaluation indicators for target detection tasks for quantitative performance evaluation, namely AP [0.5:0.95] , A.P. 0.5 , A.P. 0.75 Among them, AP [0.5:0.95] It represents the average precision under multiple values ​​of intersection-over-union ratio threshold from 0.5 to 0.95 with an interval of 0.05, and then the average value, AP, is calculated. 0.5 and AP 0.75 They represent the average precision calculated when the intersection-over-union ratio threshold is 0.5 or 0.75 respectively. From the results, it can be seen that the method proposed in this application is applied to two vehicle target detection datasets, and the results are significantly improved compared with other target detection methods.

[0064] Table 1 Comparison of experimental results with other detectors on the Cityscapes dataset

[0065]

[0066] Table 2 Comparison of experimental results with other detectors on the CoCo2017 dataset

[0067]

[0068] Table 3 Comparison of experimental results on the Cityscapes dataset with different pre-trained models

[0069]

[0070] Table 4 Comparison of experimental results on the CoCo2017 dataset with different pre-trained models

[0071]

[0072] The visual feature enhancement module of the present invention has the following advantages: through the pre-trained large vehicle base model encoder, the image features of the candidate area are extracted to enhance the image features extracted by the original backbone network. In the attribute representation learning module of the present invention, the attribute embedding is encoded using the pre-trained large language model T5, and the attribute information of each candidate area is predicted through the pre-trained attribute prediction module, and the attribute vector group of each candidate area is fused into a unified attribute representation using a gated recurrent unit. Finally, a visual text contrast learning strategy is adopted: using the idea of ​​contrastive learning, the visual representation enhanced by the visual feature enhancement module and the attribute representation generated by the attribute representation learning module are used to calculate the cosine embedding loss of the visual representation and attribute representation of each candidate area, and the loss constrains the visual representation and the semantic representation to be more similar, thereby aligning the visual representation and the semantic representation to make up for the semantic difference.

Claims

1. A vehicle target detection method based on attribute information and a large basic model, characterized in that: The following steps are involved: Step 1: Get the original image and input it into the backbone network, and extract the original feature map from the backbone network; Step 2: For the original feature map obtained in step 1, first use the region proposal module to generate a set of candidate regions, and then use the region of interest alignment module to pool the position coordinates of the candidate regions in the corresponding regions of the original feature map, and pool them into a fixed size to obtain the visual feature F. r ; For a set of candidate regions generated by the region proposal module, the candidate region images are cropped from the original image to obtain a set of candidate region images; Step 3: For the candidate region image set obtained in step 2, the visual feature enhancement module is used to extract the features of each candidate region image, and then the original feature map obtained in step 1 is enhanced to obtain the enhanced features corresponding to the candidate region image. Step 4: Use the attribute representation learning module to obtain the attribute representation of the candidate region. The specific method is as follows: Input the attribute words with attribute labels into the pre-trained large language model to obtain the text embedding of the attribute labels Embed the resulting text Splice to get F a ∈R 47×768 ; Then F a ∈R 47×768 Enhanced features output from the encoder with the pre-trained vehicle base model After connection, it is input into the attribute prediction module; here, j refers to the serial number of the attribute label; Then, the concatenated features are input into a layer of Transformer fusion block for fusion, and the text tokens in the output features are input into the attribute prediction head to obtain the predicted attributes of the candidate region; based on the predicted attributes, the attribute label with the highest probability is selected from each attribute group, and the corresponding text embedding is extracted Get attribute vector A = [a1, a2, a3, a4, a5, a6]; Next, a gated recurrent unit is used to fuse the attribute vector A = [a1, a2, a3, a4, a5, a6] to obtain V a ∈R 1×256 ; Step 5: Transform the visual feature F r With enhanced features Splice along the channel to obtain the final visual representation F v , the final visual representation F v Input to the classification head and regression head to obtain the detection results; then, calculate the final visual representation F of each candidate region v and semantic attribute representation V a The cosine embedding loss between them constrains the visual representation and the semantic representation to be more similar, thereby aligning the visual representation and the semantic representation to make up for the semantic difference; The visual feature enhancement module includes a pre-trained vehicle basic model encoder and a linear projection layer. After the image in the candidate area image set is input into the visual feature enhancement module, First, each candidate region image is deformed into I∈R 224×224×3 , and I∈R 224×224×3 Divide into 196 non-overlapping blocks P i ∈R 16×16×3 After the 196 blocks are input to the visual feature enhancement module, the P i Projection as Token Embedding Among them, i refers to the sequence number of the block, i∈{1,2,…,196}； Then, a set of learnable tokens K∈R is introduced for fine-tuning 8×768 , thus achieving more effective fine-tuning, after integrating the global semantic tokens and learnable tokens, the tokens are embedded into P emb ∈R 205×768 Input to the pre-trained vehicle base model encoder and introduce the corresponding position encoding The pre-trained vehicle base model encoder includes 12 Transformer blocks, each of which includes layer normalization, multi-head self-attention, and multi-layer perceptron; here, P emb Including multiple and global semantic tokens and learnable tokens; Finally, the pre-trained vehicle base model encoder outputs the feature F c ∈R 205×768 , the feature F c ∈R 205×768 Projected through a 256-dimensional linear projection layer 2. The vehicle target detection method based on attribute information and a large basic model according to claim 1, characterized in that: In step 4, there are six groups of attribute words for the vehicle, including color, model, displacement, number of doors, number of seats and maximum speed, with a total of 47 attribute tags.

3. The vehicle target detection method based on attribute information and large-scale basic model according to claim 1, characterized in that: The gated recurrent unit includes an update gate and a reset gate, the reset gate calculates a candidate hidden state, and the update gate retains important information from a previous hidden state and the candidate hidden state to obtain a current hidden state.

4. The vehicle target detection method based on attribute information and large-scale basic model according to claim 1, characterized in that: The specific method of the visual text contrast learning is: First, the visual representation is projected to the same dimension as the text representation through a linear projection layer, and then the visual representation and text representation are L2 normalized respectively. Finally, for each candidate region, the normalized visual representation and text representation are used to calculate the cosine embedding loss; The visual-text contrastive learning process is constrained by the following formula: Among them, CEL is the abbreviation of cosine embedding loss function, F v is the visual representation, V a is a text representation, refers to the normalized visual representation, It refers to the normalized text representation, N refers to the number of images processed in a batch; t refers to the sequence number of the image to be processed.

Images