A semantic knowledge-guided vehicle re-identification method

Abstract

The present invention belongs to the field of vehicle re-identification, and provides a vehicle re-identification method guided by semantic knowledge, including: (1) constructing a vehicle image prompt text generation network, training an image encoder, a text encoder, and a conditional encoding network through cross entropy loss, and outputting a prompt text vector of the vehicle image; (2) constructing a vehicle fine-grained feature extraction network with text semantic alignment, and outputting a feature vector of the vehicle image through multi-granularity global feature learning, cross-modal cross-granularity feature alignment, and joint supervision training using classification loss and cross entropy loss; (3) using the vehicle fine-grained feature extraction network with text semantic alignment trained in step (2), respectively extracting features of a query vehicle image and a vehicle image to be retrieved, and using cosine distance to sort feature similarity to obtain retrieved vehicle image results. The vehicle re-identification method guided by multi-semantic knowledge of the present invention effectively improves the accuracy and robustness of vehicle re-identification.

Description

Technical Field

[0001] The present invention belongs to the field of vehicle re-identification, and in particular relates to a vehicle re-identification method guided by semantic knowledge. Background Art

[0002] Vehicle re-identification, also known as cross-border tracking, has become an important means to make up for the limitations of license plate recognition verification. Current vehicle re-identification methods mainly focus on specific data sets from the perspective of visual interpretation, which is difficult to meet the actual application needs in open scenarios. The challenges faced by vehicle re-identification mainly include: 1) inter-class similarity, vehicles of the same model are highly similar; 2) intra-class differences, the same vehicle photographed from different perspectives has large appearance differences; 3) In actual applications, there may be a lack of specific vehicle images, only relevant text descriptions. Summary of the invention

[0003] In view of the above problems existing in the prior art, the present invention proposes a vehicle re-identification method guided by multi-semantic knowledge, which effectively improves the accuracy and robustness of vehicle re-identification.

[0004] To achieve the above purpose, the present invention adopts the following technical solution.

[0005] The present invention provides a vehicle re-identification method guided by multi-semantic knowledge, comprising the following steps:

[0006] (1) Construct a vehicle image prompt text generation network, train the image encoder, text encoder, and conditional encoding network through cross entropy loss, and output the prompt text vector of the vehicle image;

[0007] (2) Construct a text semantically aligned vehicle fine-grained feature extraction network, learn multi-granular global features, align cross-modal and cross-granular features, and use classification loss and cross-entropy loss for joint supervision training to output the feature vector of the vehicle image.

[0008] (3) Using the text semantic alignment vehicle fine-grained feature extraction network trained in step (2), extract features of the query vehicle image and the vehicle image to be retrieved, respectively, and use cosine distance to sort feature similarity to obtain retrieved vehicle image results;

[0009] In the above technical solution, the step (1) includes the following sub-steps:

[0010] (1-1) Use the pre-trained CLIP model as the text encoder and image encoder, and use the image encoder to extract image features after inputting the image;

[0011] (1-2) Construct an instance conditional encoder to transform the encoded image feature vector into a conditional vector. The instance conditional encoder contains two bottleneck layers, each of which is composed of a convolutional layer, a BatchNorm layer, and a ReLU layer. The conditional vector is added to the input prompt text vector and input into the text encoder to obtain a classification weight vector representing visual semantics.

[0012] (1-3) Train the vehicle image prompt text generation network, freeze the network parameters of the image encoder and text encoder, calculate the cross entropy loss, and perform backpropagation to update the network parameters of the instance condition encoder.

[0013] In the above technical solution, the step (2) includes the following sub-steps:

[0014] (2-1) Construct a multi-granularity global feature learning module, which includes two parts: discriminative token selection and multi-granularity global feature learning.

[0015] (2-1-1) Discriminant Token selection. The output Class Token is obtained by weighting all local Tokens. The weight represents the correlation between local area features and global features. Specifically, the cosine similarity matrix of the Class Token and all local Tokens is calculated, and the local Tokens with the top 2K correlations are selected to form a new Token sequence. The tokens in it are arranged in descending order of correlation.

[0016] (2-1-2) Multi-granularity global feature learning: construct two network branches to use the selected Token sequence to learn global features of different granularities. The first branch inputs the first K Tokens to learn high-level semantic features, and the second branch inputs the remaining K Tokens to learn mid-level semantic features. Each branch consists of a multi-head self-attention block (MHSA), Layer Norm, a multi-head convolutional attention block (MHCA), and a multi-layer perceptron (MLP). The new Token sequence selected in step (2-1-1) is divided into a high-level sequence and a mid-level sequence, and the Class Token is appended to the high-level sequence and the mid-level sequence. The final global features of the vehicle image include low-level semantic features, mid-level semantic features, and high-level semantic features.

[0017] (2-2) Cross-modal and cross-granular feature alignment, establishing semantic alignment of vehicle images-words and region images-sentences.

[0018] (2-2-1) Cross-modal and cross-granularity similarity calculation, which calculates the similarity between the entire image and the words in the description text, and the similarity between the local area of ​​the image and the entire description text;

[0019] (2-2-2) Aggregate the final similarity of the image and the description text according to the similarity across granularities. During aggregation, use the Softmax function to generate different weights for each similarity.

[0020] (2-3) Cross-modal fine-grained feature alignment: use the selected first K image tokens and text tokens to perform fine-grained feature alignment.

[0021] (2-3-1) Calculate the cosine similarity of the first K image tokens and text tokens. For each image / text token, select the first K text / image tokens with the highest similarity and perform average pooling to obtain a new feature pair;

[0022] (2-3-2) Calculate the similarity of each feature pair to perform fine-grained feature alignment and output fine-grained similarity;

[0023] (2-4) Model training, using cross entropy loss and batch hard sample loss function (Batch Hard TripletLoss) to simultaneously supervise the multi-granularity global feature learning of text and image, summing the cross-modal and cross-granular similarity in step (2-2-2) and the cross-modal fine-grained similarity in step (2-3-2), and using the triplet loss function to supervise the summed cross-modal similarity.

[0024] In the above technical solution, the step (3) includes the following sub-steps:

[0025] (3-1) Input all vehicle images to be retrieved into the text semantic alignment vehicle fine-grained feature extraction network trained in step (2) to obtain feature vectors of all images, and input the query vehicle image into the text semantic alignment vehicle fine-grained feature extraction network trained in step (2) to obtain the feature vector of the query image;

[0026] (3-2) Calculate the cosine distance between the query vehicle image feature and all the vehicle image features to be retrieved, sort them from large to small according to the cosine distance, and obtain the retrieval result of the query vehicle.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. The introduction of semantic knowledge constraints helps to improve the generalization ability of the model.

[0029] 2. We designed multi-granularity global feature learning, cross-modal and cross-granularity feature alignment, and cross-modal fine-grained feature alignment, which aligned the visual features of the image and the semantic features of the text description at different granularities, which can better represent the local information of the vehicle and improve the accuracy of re-identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a diagram of the semantic knowledge-guided vehicle re-identification network architecture of an embodiment of the present invention.

[0031] Figure 2 The present invention is implemented in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0033] like Figure 2 As shown, a semantic knowledge-guided vehicle re-identification method provided by an embodiment of the present invention includes the following steps:

[0034] (1) Construct a vehicle image prompt text generation network, train the image encoder, text encoder, and conditional encoding network through cross entropy loss, and output the prompt text vector of the vehicle image;

[0035] (2) Construct a text semantically aligned vehicle fine-grained feature extraction network, learn multi-granular global features, align cross-modal and cross-granular features, and use classification loss and cross-entropy loss for joint supervision training to output the feature vector of the vehicle image.

[0036] (3) Using the text semantically aligned vehicle fine-grained feature extraction network trained in step (2), the features of the query vehicle image and the vehicle image to be retrieved are extracted respectively, and the cosine distance is used to sort the feature similarity to obtain the result of the retrieved vehicle.

[0037] Figure 1 The network architecture diagram of the semantic knowledge-guided vehicle re-identification according to an embodiment of the present invention includes: a vehicle image prompt text generation network; and a vehicle fine-grained feature extraction network with text semantic alignment.

[0038] Next, this specification describes the identification method in detail.

[0039] The step (1) specifically comprises the following steps:

[0040] (1-1) The pre-trained CLIP model is used as the text encoder and image encoder. After the image is input, the feature extracted by the image encoder is represented as f, and the prompt text vector input to the text encoder is represented as:

[0041] t=[V] 1 [V] 2 ...[V] M [CLASS]

[0042] Among them, [V] m(m∈{1,......M}) is a 512-dimensional word vector, and M is the prompt text length parameter.

[0043] (1-2) Construct an instance conditional encoder to transform the encoded image feature vector into a conditional vector. The instance conditional encoder contains two bottleneck layers, each of which consists of a convolutional layer, a BatchNorm layer, and a ReLU layer. The conditional vector is added to the input prompt text vector:

[0044] [V] i =[V] i +θ

[0045] θ=h(f)

[0046] Where i∈{1,...M}. After adding the prompt text vector t to the conditional vector, the prompt text vector t is input into the text encoder to obtain a classification weight vector representing visual semantics. The prediction probability is calculated as follows:

[0047]

[0048] Where K is the number of vehicle instances.

[0049] (1-3) Train the vehicle image prompt text generation network, freeze the network parameters of the image encoder and the text encoder, calculate the cross entropy loss, and perform back propagation to update the network parameters of the instance conditional encoder. For example, using the training set of the Veri-776 dataset, randomly select 16 images for each vehicle, a total of 9216 images, and generate 576 vehicle identity prompt text vectors.

[0050] The step (2) specifically comprises the following steps:

[0051] (2-1) Construct a multi-granularity global feature learning module. This module includes two parts: discriminative token selection and multi-granularity global feature learning.

[0052] (2-1-1) Discriminant Token selection. The output Class Token is obtained by weighting all local Tokens, and the weight represents the correlation between local area features and global features. Specifically, the cosine similarity matrix A∈R between the Class Token and all local Tokens is calculated. (1+N)×(1+N) , the correlation between Class Token and local Token is m = A[0,1:]∈R N , select the top 2K local tokens with the highest correlation to form a new token sequence. The selected image and text token sequences are represented as:

[0053]

[0054]

[0055] All tokens are arranged in descending order according to their relevance. img and K text Indicates the number of tokens selected for image and text respectively.

[0056] (2-1-2) Multi-granularity global feature learning. Construct two network branches to use the selected token sequence to learn global features of different granularities. The first branch inputs the first K tokens to learn high-level semantic features, and the second branch inputs the remaining K tokens to learn mid-level semantic features. Each branch consists of a multi-head self-attention block (MHSA), Layer Norm, a multi-head convolutional attention block (MHCA), and a multi-layer perceptron (MLP). Specifically, F Sel High-level sequence and middle sequence And append the Class Token to the high-level sequence and the middle-level sequence, expressed as [CLS] h and [CLS] m The learning process of multi-granularity global features is as follows:

[0057]

[0058]

[0059]

[0060] Select the learned [CLS] h Features as high-level semantic features F h , [CLS] m Features as mid-level semantic features F m The original Class Token combines all local area features and represents low-level semantic features, expressed as F l The final global features of the vehicle image, including low-level semantic features, mid-level semantic features and high-level semantic features, are expressed as F = {F l ,F m ,F h};

[0061] (2-2) Cross-modal and cross-granular feature alignment to establish semantic alignment of vehicle image-word and region image-sentence.

[0062] (2-2-1) Similarity calculation across modalities and granularities. Similarity is calculated using the following formula:

[0063] S global-word =(T Sel Fl ) T

[0064] S local-text =F Sel T l

[0065] in, Represents the similarity between the entire image and the words in the description text, Indicates the similarity between the local area of ​​the image and the entire description text.

[0066] (2-2-2) The final similarity between the image and the description text is aggregated based on the similarity across granularities. During aggregation, the Softmax function is used to generate different weights for each similarity. The calculation process is as follows:

[0067]

[0068]

[0069]

[0070] Among them, S global-word(1,i) Represents the similarity matrix S global-word The value of row 1, column i in Represents the similarity across modalities and granularities.

[0071] (2-3) Cross-modal fine-grained feature alignment, using the selected first K image tokens and text tokens ( and ) to perform fine-grained feature alignment.

[0072] (2-3-1) Calculation and For each image / text token, select the top K text / image tokens with the highest similarity and perform average pooling to obtain a new feature pair. The specific calculation method is as follows:

[0073]

[0074] Where i∈{1,...,K img}. Generate K img Region image-word matching pairs and K text Word-region image matching

[0075] (2-3-2) Calculate the similarity of each feature pair to perform fine-grained feature alignment. The calculation method is as follows:

[0076]

[0077] (2-4) Model training, using cross entropy loss and batch difficult sample loss function to simultaneously supervise the multi-granularity global feature learning of text and image, and replace the and step (2-3-2) The sum is performed and a triplet loss function is used to supervise the summed cross-modal similarity.

[0078] The step (3) specifically comprises the following steps:

[0079] (3-1) All vehicle images to be retrieved are input into the text semantically aligned vehicle fine-grained feature extraction network trained in step (2) to obtain feature vectors of all images, and the query vehicle image is input into the text semantically aligned vehicle fine-grained feature extraction network trained in step (2) to obtain the feature vector of the query image.

[0080] (3-2) Calculate the cosine distance between the query vehicle image feature and all the vehicle image features to be retrieved, sort them from large to small according to the cosine distance, and obtain the retrieval result of the query vehicle.

[0081] The contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0082] Although the present invention is described in detail and embodiments are given above, the present invention and applicable embodiments are not limited thereto, and those skilled in the art can make various modifications based on the principles of the present invention, and can also apply some methods in the present invention to other systems. Therefore, all modifications made according to the principles of the present invention should be understood to fall within the protection scope of the present invention.

Claims

1. A semantic knowledge-guided vehicle re-identification method, characterized in that: The following steps are involved: (1) Construct a vehicle image prompt text generation network, train the image encoder, text encoder, and conditional encoding network through cross entropy loss, and output the prompt text vector of the vehicle image; specifically, the following sub-steps are included: (1-1) Use the pre-trained CLIP model as the text encoder and image encoder, and use the image encoder to extract image features after inputting the image; (1-2) Construct an instance conditional encoder to transform the encoded image feature vector into a conditional vector. The instance conditional encoder contains two bottleneck layers, each of which is composed of a convolutional layer, a BatchNorm layer, and a ReLU layer. The conditional vector is added to the input prompt text vector and input into the text encoder to obtain a classification weight vector representing visual semantics. (1-3) Train the vehicle image prompt text generation network, freeze the network parameters of the image encoder and text encoder, calculate the cross entropy loss, and perform backpropagation to update the network parameters of the instance condition encoder; (2) Construct a text semantically aligned vehicle fine-grained feature extraction network, learn multi-granular global features, align cross-modal and cross-granular features, and use classification loss and cross-entropy loss for joint supervision training to output the feature vector of the vehicle image. The specific steps include the following: (2-1) Construct a multi-granularity global feature learning module, which includes two parts: discriminative token selection and multi-granularity global feature learning. (2-1-1) Discriminant Token selection. The output Class Token is obtained by weighting all local Tokens. The weight represents the correlation between local area features and global features. Specifically, the cosine similarity matrix of the Class Token and all local Tokens is calculated, and the local Tokens with the top 2K correlations are selected to form a new Token sequence. The tokens in it are arranged in descending order of correlation. (2-1-2) Multi-granularity global feature learning: construct two network branches to use the selected Token sequence to learn global features of different granularities. The first branch inputs the first K Tokens to learn high-level semantic features, and the second branch inputs the remaining K Tokens to learn mid-level semantic features. Each branch consists of a multi-head self-attention block, a Layer Norm, a multi-head convolutional attention block, and a multi-layer perceptron. The new Token sequence selected in step (2-1-1) is divided into a high-level sequence and a mid-level sequence, and the Class Token is appended to the high-level sequence and the mid-level sequence. The final global features of the vehicle image include low-level semantic features, mid-level semantic features, and high-level semantic features. (2-2) Cross-modal and cross-granular feature alignment, establishing semantic alignment of vehicle images-words and region images-sentences. (2-2-1) Cross-modal and cross-granularity similarity calculation, which calculates the similarity between the entire image and the words in the description text, and the similarity between the local area of ​​the image and the entire description text; (2-2-2) Aggregate the final similarity of the image and the description text according to the similarity across granularities. During aggregation, use the Softmax function to generate different weights for each similarity. (2-3) Cross-modal fine-grained feature alignment: Use the first K selected image tokens and text tokens for fine-grained feature alignment. (2-3-1) Calculate the cosine similarity of the first K image tokens and text tokens, select the first K text / image tokens with the highest similarity for each image / text token, and perform average pooling to obtain a new feature pair; (2-3-2) Calculate the similarity of each feature pair to perform fine-grained feature alignment and output fine-grained similarity; (2-4) Model training, using cross entropy loss and batch difficult sample loss function to simultaneously supervise the multi-granularity global feature learning of text and image, summing the cross-modal and cross-granular similarity in step (2-2-2) and the cross-modal fine-grained similarity in step (2-3-2), and using the triplet loss function to supervise the summed cross-modal similarity; (3) Using the text semantic alignment vehicle fine-grained feature extraction network trained in step (2), extract features of the query vehicle image and the vehicle image to be retrieved, respectively, and use cosine distance to sort feature similarity to obtain the result of retrieving vehicles; specifically, the following sub-steps are included: (3-1) Input all vehicle images to be retrieved into the text semantic alignment vehicle fine-grained feature extraction network trained in step (2) to obtain feature vectors of all images, and input the query vehicle image into the text semantic alignment vehicle fine-grained feature extraction network trained in step (2) to obtain the feature vector of the query image; (3-2) Calculate the cosine distance between the query vehicle image feature and all the vehicle image features to be retrieved, sort them from large to small according to the cosine distance, and obtain the retrieval result of the query vehicle.

Images