A vehicle re-identification method from text based on multi-scale and multi-view feature alignment

Abstract

A text vehicle re-identification method based on multi-scale and multi-view feature alignment belongs to the field of deep learning technology and solves the existing vehicle re-identification perspective challenge problem; constructs a two-stream feature learning network based on the residual network ResNet-50 and the convolutional neural network; introduces the vehicle perspective mask in the visual encoder to extract the corresponding perspective features, and extracts multi-scale features from different layers of the encoder; uses the text convolutional network to construct multi-view and multi-scale branches in the text encoder, and uses the aligned visual features to supervise the learning of the corresponding branches; in order to effectively align the various branches, a masked bidirectional matching loss is proposed to mask unpaired samples under the same vehicle and consider text-to-image and image-to-text bidirectional retrieval; by introducing the perspective mask, the features of different vehicle perspectives are effectively decoupled, and the multi-scale matching clues between image and text are mined; the masked bidirectional matching loss ensures training stability while accelerating model convergence.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology and relates to a text vehicle re-identification method based on multi-scale and multi-view feature alignment. Background Art

[0002] Vehicle re-identification (VRI) aims to retrieve matching vehicles for a given query. Currently, VRI technology primarily relies on visual features, such as vehicle color, license plate information, and model. However, due to the high similarity between vehicles and significant viewpoint variations, these visual features often lack sufficient discriminability. Therefore, more effective VRI methods are needed.

[0003] Text-to-image retrieval, as an emerging information retrieval technology, holds enormous potential. It can achieve vehicle recognition by combining textual information with visual features, making re-identification results more accurate and reliable. Existing text-to-image retrieval methods are primarily categorized into two types based on how they process images: image-text matching methods leveraging detection networks and local feature extraction methods leveraging prior knowledge.

[0004] The first category of methods uses a detection network to extract object features in the image, thereby mining fine-grained matching clues between text and images. For example, the paper "Proceedings of the European conference on computer vision" (Kuang-Huei Lee et al., Stacked Cross Attention for Image-Text Matching. 2018: 201-216.) proposes using image regions and words in sentences as context to discover complete potential alignments and infer image-text similarity. First, for each image region, attention is paid to the words in the sentence, and each image region is compared with the attention information in the sentence to determine the importance of the image region. Similarly, in the proposed text-image formula, attention is paid to the image region of each word, and then a decision is made to give more or less attention to each word, making image-text matching more interpretable. The second category of methods uses prior knowledge to construct local feature branches. For example, the invention patent application document "Text Pedestrian Retrieval Method Based on Text Dynamically Guided Visual Feature Extraction" with an application publication date of August 6, 2021 and application publication number CN113221680A proposes a text pedestrian retrieval framework based on global and local feature matching. In terms of images, the cropped pedestrian images are horizontally segmented, and the resulting image blocks are used as local features of the pedestrian images. Simultaneously, the text feature extraction module uses a classification network to predict the probability that a word semantically belongs to each image block, mining fine-grained matching clues.

[0005] While the first category of methods can extract displayed image object blocks and mine fine-grained image-text matching clues, the retrieved vehicle images are fine-grained and cropped, resulting in low resolution. Therefore, reusing the detection network to extract object information from cropped vehicle images is difficult. The second category of methods mostly relies on prior knowledge of pedestrian images, where the image blocks to which each component belongs are essentially determined after horizontal cropping. However, for vehicle images, horizontal segmentation cannot align vehicle components with the cut blocks due to differences in viewing angles. Summary of the Invention

[0006] The purpose of this invention is to solve the existing vehicle re-identification perspective challenge by introducing a vehicle perspective mask.

[0007] The present invention solves the above technical problems through the following technical solutions:

[0008] A vehicle re-identification method based on multi-scale and multi-view feature alignment from text includes the following steps:

[0009] Step 1: Use ResNet-50 residual network to extract image feature maps and extract visual multi-scale features of the input vehicle image;

[0010] Step 2: Obtain the perspective features of the vehicle image by introducing the vehicle perspective mask into the middle layer of the ResNet-50 residual network;

[0011] Step 3: Use the vehicle view mask to generate the similarity weight of the view branch;

[0012] Step 4: Use BERT and text convolutional network to extract multi-scale and multi-view vehicle text features;

[0013] Step 5: Use mask bidirectional matching loss to perform multi-scale, multi-view, cross-modal feature alignment;

[0014] Step 6: Calculate multi-scale and multi-view features for testing.

[0015] The present invention is based on a text vehicle re-identification framework for multi-scale and multi-view feature alignment, aiming to constrain the alignment between modalities in multi-scale and multi-view subspaces; construct a two-stream feature learning network based on the residual network ResNet-50 and the convolutional neural network; introduce a vehicle view mask in the visual encoder to extract the corresponding view features, and extract multi-scale features from different layers of the encoder; at the same time, use a text convolutional network in the text encoder to construct multi-view and multi-scale branches, and use the aligned visual features to supervise the learning of the corresponding branches; in order to effectively align the various branches, a masked bidirectional matching loss is proposed to mask unpaired samples under the same vehicle and consider text-to-image and image-to-text bidirectional retrieval. The present invention effectively decouples the features of different vehicle views by introducing the view mask, and mines multi-scale matching clues between image and text; in addition, the present invention also proposes a masked bidirectional matching loss to ensure training stability while accelerating model convergence.

[0016] Furthermore, the method described in step 1 of extracting the image feature map using the ResNet-50 residual network to extract the visual multi-scale features of the input vehicle image is as follows: the ResNet-50 residual network contains five layers, each layer consists of a residual block and a pooling layer; the outputs of different layers of the ResNet-50 residual network are used as visual low-level features and visual high-level features; in the training stage, the training data is Where N represents the number of image-text pairs that match each other and belong to the same identity; given a vehicle image I∈R 224×224×3 , then the high-level image features obtained in the fifth stage of the residual network ResNet are i high =GMP(Layer 1-5 (I)), the low-level features obtained in the fourth stage are i low =GMP(Layer 1-4 (I)), where i high ∈R 2048 ,i low ∈R 1024 .

[0017] Furthermore, the method described in step 2 for obtaining the perspective features of the vehicle image by introducing the vehicle perspective mask into the middle layer of the ResNet-50 residual network is as follows: the feature map output by the fourth layer of the ResNet-50 residual network is defined as the middle layer feature I mid =Layer 1-4 (I) and introduce the trained vehicle view mask {M fro ,M sid ,M rea}, thereby obtaining the corresponding viewing angle feature i v =GMP(Layer4(M fro *Imid )),v∈{fro,sid,rea}; then use the view feature aggregation to obtain the composite view feature i com =RMP(Concat(i fro ,i rea ,i sid )), where i fro ,i sid ,i rea ,i com ∈R 2048 .

[0018] Furthermore, the method of using the vehicle view mask in step 3 to generate the similarity weights of the view branches is as follows: the proportion of the image area occupied by the three views of the vehicle changes with the angle of the camera and the driving process of the vehicle, and the weights of the three branches are normalized using the vehicle view mask to achieve dynamic aggregation of the similarity scores of the view branches: [w fro ,w sid ,w rea ]=Softmax(GAP(Concat(M fro ,M sid ,M rea ))).

[0019] Furthermore, the method described in step 4 for extracting multi-scale, multi-perspective vehicle text features using BERT and a text convolutional network is as follows: in the text branch, the BERT model widely used in natural language processing is used to convert the text description into a token sequence and extract a word vector; a fixed value L is set to control the sentence length, and in the process of converting the text description into a token sequence, a zero padding operation is performed for sequences less than L in length; for sequences longer than L, the first L tokens are taken; a fixed-length token sequence is obtained; and these are then input into the BERT model to obtain a text feature representation T = BERT(T token ),T∈R L×D ; In order to extract the feature representation of each branch corresponding to the text description, firstly, the dimension of the word vector is changed from T∈R L×D Expand to T * ∈R 1×L×D , which is then processed by the subsequent convolutional neural network; next, a convolutional layer and batch norm operation are used to obtain the low-level features of the text T mid =ReLU(Bn(Conv(T * ))); Then, by aggregating the low-level features of the text, the low-level feature vector t of the text is obtained low =RMP(T mid ); Finally, a modular network composed of convolution and residual is used to obtain multi-scale and multi-view text features corresponding to the vehicle image: s=RMP(Mobular s (T mid )),s∈{fro,sid,rea,com,low,high}.

[0020] Furthermore, the method for multi-scale, multi-view, cross-modal feature alignment using mask bidirectional matching loss described in step 5 is specifically as follows:

[0021] Simultaneously optimize the two retrieval directions of text to image and image to text and propose a mask bidirectional matching loss:

[0022]

[0023] Among them, U i represents unpaired samples belonging to the same vehicle, and τ is the temperature hyperparameter used to adjust the distribution of the loss function;

[0024] Optimize multiple branches simultaneously using masked bidirectional matching loss:

[0025] Loss=

[0026] L mb (t mid ,i mid )+L mb (t hig ,i hig )+w fro *L mb (t fro ,i fro )+w sid *L mb (t sid ,i sid )+w rea *L mb (t rea ,i rea ).

[0027] Furthermore, the method for calculating the multi-scale and multi-view features for testing described in step 6 is specifically as follows: given a text description as a query, the cosine similarity between the multi-scale and multi-view features extracted from the text description and the multi-scale and multi-view features of each vehicle image in the image library is aggregated to obtain the similarity between the text description and the image, and the image with the highest similarity is the result of the vehicle retrieval in this round.

[0028] The advantages of the present invention are:

[0029] 1) We design a novel framework based on multi-scale and multi-view feature alignment to perform fine-grained text-based vehicle retrieval by aligning vehicle images and text in a multi-scale and multi-view feature space;

[0030] 2) A masked bidirectional matching loss is designed to ensure training stability while accelerating model convergence;

[0031] 3) In the field of text description vehicle retrieval, a view mask is introduced to extract vehicle view features and effectively decouple vehicle view information;

[0032] 4) Simultaneously constrain the multi-scale feature alignment between text and image, and mine multi-scale matching clues between image and text. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is an overall network framework diagram of a text vehicle re-identification method based on multi-scale and multi-view feature alignment according to the first embodiment of the present invention;

[0034] Figure 2 It is a masked bidirectional loss map of a text vehicle re-identification method based on multi-scale and multi-view feature alignment in embodiment 1 of the present invention. DETAILED DESCRIPTION

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0036] The technical solution of the present invention is further described below with reference to the accompanying drawings and specific embodiments:

[0037] Example 1

[0038] like Figure 1 As shown in the figure, the text-to-image vehicle re-identification method based on multi-scale and multi-view feature alignment consists of three parts: multi-scale and multi-view visual feature extraction, text feature extraction, and cross-modal feature alignment.

[0039] (1) To effectively mine multi-scale matching clues between vehicle text and images, the present invention simultaneously aligns the encoder outputs at multiple layers. Furthermore, to decouple the features of vehicle images from multiple perspectives, a vehicle perspective segmentation mask is introduced in the middle layer of the encoder.

[0040] (2) In order to obtain multi-scale and multi-view feature representations of text descriptions, a modular network is used to extract text features of the corresponding branches. Specifically, a text convolutional network with the same structural parameters that do not share features is used to extract features of the corresponding branches, and the aligned image multi-view and multi-scale features are used as supervision for the text convolutional network.

[0041] (3) In order to effectively constrain the alignment of features in each branch, a masked bidirectional matching loss is proposed. Vehicle images from different perspectives of the same vehicle have significant appearance differences; vehicle images from similar perspectives of the same vehicle have similar appearances. Instead of treating all such samples as positive or negative, these samples are masked to ensure the stability of network training. At the same time, to accelerate model convergence, both text-to-image and image-to-text retrieval are considered during training.

[0042] The specific steps are as follows:

[0043] 1. Extract visual multi-scale features of input vehicle images

[0044] The present invention uses ResNet-50 residual network to extract image feature maps. ResNet-50 residual network contains five layers, each layer consists of residual blocks and pooling layers. The outputs of different layers of ResNet-50 residual network are used as low-level visual features and high-level visual features. In the training stage, assuming that a batch of training data is Where N represents the number of image-text pairs that match each other and belong to the same identity. Given a vehicle image I∈R 224×224×3 , then the high-level image features obtained in the fifth stage of the residual network ResNet are i high =GMP(Layer 1-5 (I)), the low-level features obtained in the fourth stage are i low =GMP(Layer 1-4 (I)), where i high ∈R 2048 ,i low ∈R 1024 .

[0045] 2. By introducing the vehicle view mask in the middle layer of the ResNet-50 residual network, the view features of the vehicle image are obtained

[0046] The feature map output by the fourth layer of the ResNet-50 residual network is defined as the intermediate layer feature I mid =Layer 1-4 (I) and introduce the trained vehicle view mask {M fro ,M sid ,M rea}, thereby obtaining the corresponding viewing angle feature i v =GMP(Layer4(M fro *I mid )),v∈{fro,sid,rea}; then use the view feature aggregation to obtain the composite view feature i com =RMP(Concat(i fro ,irea ,i sid )), where i fro ,i sid ,i rea ,i com ∈R 2048 .

[0047] 3. Use the vehicle view mask to generate the similarity weight of the view branch

[0048] The proportion of the image area occupied by the three vehicle perspectives changes with the camera angle and the vehicle's driving process. The vehicle perspective mask is used to normalize the weights of the three branches to achieve dynamic aggregation of the similarity scores of the perspective branches: [w fro ,w sid ,w rea ]=Softmax(GAP(Concat(M fro ,M sid ,M rea ))).

[0049] 4. Use BERT and text convolutional networks to extract multi-scale and multi-perspective vehicle text features

[0050] In the text branch, the BERT model, which is widely used in natural language processing, is used to convert text descriptions into token sequences and extract word vectors. To facilitate subsequent processing, a fixed value L is set to control the sentence length. In the process of converting text descriptions into token sequences, zero padding is performed for sequences less than L in length; for sequences longer than L, the first L tokens are taken. In this way, a fixed-length token sequence is obtained; these are then input into the BERT model to obtain the text feature representation T = BERT(T token ),T∈R L×D .

[0051] In order to extract the feature representation of each branch corresponding to the text description, we first reduce the dimension of the word vector from T∈R L×D Expand to T * ∈R 1×L×D , which can be processed by the subsequent convolutional neural network. Next, a convolutional layer and batch norm operation are used to obtain the low-level features of the text T mid =ReLU(Bn(Conv(T * ))); Then, by aggregating the low-level features of the text, the low-level feature vector t of the text is obtained low =RMP(T mid ); Finally, a modular network composed of convolution and residual is used to obtain multi-scale and multi-view text features corresponding to the vehicle image: s =RMP(Modular s (Tmid )),s∈{fro,sid,rea,com,low,high}.

[0052] 5. Use mask bidirectional matching loss for multi-scale multi-view cross-modal feature alignment

[0053] Vehicles have similar appearances under similar viewing angles, but their appearances may be very different under different viewing angles. This makes it difficult to define the matching relationship between images of the same vehicle under different viewing angles. Figure 2 As shown in the figure, this embodiment designs a new positive and negative sample division strategy. For the samples of the same vehicle from different perspectives that are difficult to define, these samples are masked during the training process to ensure the stability of model training. At the same time, to further accelerate model convergence, a masked bidirectional matching loss is proposed to optimize both text-to-image and image-to-text retrieval directions:

[0054]

[0055] Among them, U i Represents samples belonging to the same vehicle but not paired, and τ is a temperature hyperparameter used to adjust the distribution of the loss function. Multiple branches are optimized simultaneously using masked bidirectional matching loss:

[0056] Loss = L mb (t mid ,i mid )+L mb (t hig ,ih ig )+w fro *L mb (t fro ,i fro )+w sid *L mb (t sid ,i sid )+w re□ *L mb (t rea ,i rea )

[0057] 6. Calculate multi-scale and multi-view features for testing.

[0058] Given a text description as a query, the cosine similarity between the multi-scale and multi-view features extracted from the text description and the multi-scale and multi-view features of each vehicle image in the image library is aggregated to obtain the similarity between the text description and the image. The image with the highest similarity is the result of the vehicle retrieval in this round.

[0059] This paper introduces a vehicle view mask to solve the existing vehicle re-identification perspective challenge, and achieves robust re-identification effect on vehicle text image data of any perspective. Using the PyTorch framework, deep learning technology is applied to the field of natural language pedestrian retrieval, and a multi-scale multi-view cross-modal alignment network (MCANet) is proposed, which pursues the alignment of multi-scale and multi-view features to reduce the modal distance and intra-class distance between image and text. Figure 1 As shown. Specifically, the present invention first constructs a new dual-stream feature learning network to extract image and text features respectively, pushes the samples of the two modalities closer in the feature space, and performs feature alignment. In the visual branch, multi-scale features and multi-view features are introduced. Specifically, the network middle layer features are used as low-level features, and the vehicle perspective mask is introduced in the middle layer to obtain the features corresponding to the vehicle perspective. In the text branch, BERT and text convolutional network are used to extract multi-scale and multi-view features corresponding to the image. Finally, the mask bidirectional matching loss is used to constrain the cross-modal feature alignment of the multi-scale and multi-view branches.

[0060] 7. Test verification

[0061] The effectiveness of our proposed method is verified by comparing its retrieval performance with several feasible retrieval algorithms on the only currently available text-to-image vehicle re-identification dataset, T2I VeRi (Table 1). We employ the popular rank-n retrieval and mean average precision (mAP) of retrieval algorithms. We compare state-of-the-art text-to-image person re-identification methods with methods pre-trained on visual language with additional data. The results demonstrate that our proposed method significantly improves the performance of text-to-image vehicle re-identification.

[0062] Table 1 Retrieval performance comparison verification table

[0063]

[0064]

[0065] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A vehicle re-identification method based on multi-scale and multi-view feature alignment, characterized by: The following steps are involved: Step 1: Use ResNet-50 residual network to extract image feature maps and extract visual multi-scale features of the input vehicle image; Step 2: The vehicle view mask is introduced into the middle layer of the ResNet-50 residual network to obtain the view features of the vehicle image. The specific method is as follows: the ResNet-50 residual network contains five layers, each layer consists of a residual block and a pooling layer; the outputs of different layers of the ResNet-50 residual network are used as low-level visual features and high-level visual features; During the training phase, the training data is , where N represents the number of image-text pairs that match each other and belong to the same identity; given a vehicle image , then the high-level image features obtained in the fifth stage of the residual network ResNet are , the low-level features obtained in the fourth stage are ,in ; Step 3: Use the vehicle view mask to generate the similarity weight of the view branch. The specific method is: define the feature map output by the fourth layer of the ResNet-50 residual network as the intermediate layer feature , and introduce the trained vehicle view mask , thereby obtaining the corresponding viewing angle features ; Then use the view feature aggregation to obtain the composite view feature ,in ; Step 4: Use BERT and a text convolutional network to extract multi-scale, multi-perspective vehicle text features. The specific method is as follows: In the text branch, the BERT model, which is widely used in natural language processing, is used to convert text descriptions into token sequences and extract word vectors; A fixed value L is set to control the sentence length. In the process of converting text descriptions into token sequences, zero padding is performed for sequences with a length less than L; zero padding is performed for sequences with a length greater than L. L sequence, take the first L tokens; Get a fixed-length token sequence; then input them into the BERT model to get text feature representation ; In order to extract the feature representation of each branch corresponding to the text description, first the dimension of the word vector is changed from Expand to , which is then processed by the subsequent convolutional neural network; next, a convolutional layer and batch norm operation are used to obtain the low-level features of the text ; Then, by aggregating the low-level features of the text, the low-level feature vector of the text is obtained Finally, a modular network composed of convolution and residual is used to obtain multi-scale and multi-view text features corresponding to vehicle images: ; Step 5: Use mask bidirectional matching loss to perform multi-scale, multi-view, cross-modal feature alignment; Step 6: Calculate multi-scale and multi-view features for testing.

2. The method for vehicle re-identification based on text based on multi-scale and multi-view feature alignment according to claim 1, characterized in that: The method described in step 3 for generating similarity weights of view branches using the vehicle view mask is as follows: the proportion of the image area occupied by the three vehicle views changes with the camera angle and the vehicle's driving process. The vehicle view mask is used to normalize the weights of the three branches to achieve dynamic aggregation of the similarity scores of the view branches: .

3. The method for vehicle re-identification based on text based on multi-scale and multi-view feature alignment according to claim 1, characterized in that: The method described in step 5 for multi-scale, multi-view, cross-modal feature alignment using mask bidirectional matching loss is as follows: Simultaneously optimize the two retrieval directions of text to image and image to text and propose a mask bidirectional matching loss: in, represents unpaired samples belonging to the same vehicle, is the temperature hyperparameter used to adjust the distribution of the loss function; Optimize multiple branches simultaneously using masked bidirectional matching loss: 。 4. The method for vehicle re-identification based on text based on multi-scale and multi-view feature alignment according to claim 3, characterized in that: The method for calculating the multi-scale and multi-view features for testing described in step 6 is specifically as follows: given a text description as a query, the cosine similarity between the multi-scale and multi-view features extracted from the text description and the multi-scale and multi-view features of each vehicle image in the image library is aggregated to obtain the similarity between the text description and the image. The image with the highest similarity is the result of the current round of vehicle retrieval.

Images