Vehicle re-identification method, device, equipment and product based on multi-modal pre-training

Abstract

The invention discloses a vehicle re-identification method, device, equipment and product based on multi-modal pre-training. The method comprises the following steps: acquiring vehicle data of a target vehicle; wherein the vehicle data at least comprises one of vehicle image data or text description information of the target vehicle; performing feature extraction on the vehicle data by adopting a trained multi-modal vehicle re-identification model to obtain final features of the vehicle data, and outputting information of the target vehicle according to the final features; wherein the trained multi-modal vehicle re-identification model comprises a pre-trained visual converter model, a pre-trained text graph model and a visual converter main network. According to the method, multiple query modes and text information embedding can be supported, capturing of image text information and understanding of image semantics are improved, the robustness and discrimination of extracted features are enhanced through pre-training, and therefore the vehicle re-recognition performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of vehicle identification technology, and in particular to a vehicle re-identification method, device, equipment and product based on multimodal pre-training. Background Art

[0002] Vehicle re-identification technology is an important research direction in the intelligent transportation system. Its purpose is to quickly identify and match vehicles with the same identity under different surveillance camera views. It can be used for suspect vehicle tracking, highway toll collection, vehicle management, etc., replacing the traditional manual retrieval method, which can save a lot of manpower and material resources. How to extract discriminative and robust features is the optimization direction of vehicle re-identification. With the development of deep learning, neural network-based methods have gradually become a popular choice for vehicle re-identification. It mainly extracts vehicle features through neural networks and then measures distance in feature space. However, there are still three problems with neural network-based vehicle re-identification methods: First, it does not support multiple query methods such as intelligent text query, only relies on image data, and lacks understanding of image semantics and text description information; second, it does not fully consider multimodal data such as camera type and perspective, and it is difficult to adapt to complex traffic scenes; third, it directly uses the ImageNet pre-trained network, the network structure and training method are relatively simple, and it is not optimized for specific tasks, resulting in a large dependence of the model on the scale of specific task data sets, and insufficient adaptability and universality. Summary of the invention

[0003] The present invention provides a vehicle re-identification method, device, equipment and product based on multimodal pre-training, which supports multiple query modes such as text query and text information embedding, improves the capture of image text information and the understanding of image semantics, improves the understanding of different modalities through multimodal information embedding to adapt to complex traffic scenes, and enhances the robustness and discriminability of extracted features through pre-training, thereby improving the performance of vehicle re-identification.

[0004] In order to achieve the above object, an embodiment of the present invention provides a vehicle re-identification method based on multimodal pre-training, comprising: Acquire vehicle data of a target vehicle; wherein the vehicle data includes at least one of vehicle image data or text description information of the target vehicle; A trained multimodal vehicle re-identification model is used to extract features from the vehicle data to obtain final features of the vehicle data, and information of the target vehicle is output according to the final features; wherein the trained multimodal vehicle re-identification model includes a pre-trained visual transformer model, a pre-trained text graph model and a visual transformer main network.

[0005] As an improvement of the above solution, if the vehicle data includes the vehicle image data and text description information, the trained multimodal vehicle re-identification model is used to extract features from the vehicle data to obtain final features of the vehicle data, and the information of the target vehicle is output according to the final features, including: Using a pre-trained visual transformer model to identify the vehicle data, and obtaining overall vehicle features and detailed vehicle features of the vehicle data; wherein the pre-trained visual transformer model includes a first visual transformer model and a second visual transformer model; Using a pre-trained text-based graph model to generate a word embedding vector of the text description information and a vehicle image feature from the text description information; Performing feature splicing on the overall features of the vehicle and the detailed features of the vehicle, and integrating the word embedding vector into the spliced ​​vehicle features through a cross self-attention mechanism to obtain a feature representation of the vehicle data; The feature representation and the vehicle image feature are subjected to feature fusion by using a visual transformer main network to obtain final features of the vehicle data, and information of the target vehicle is output according to the final features.

[0006] As an improvement of the above solution, the vehicle data is identified by using a pre-trained visual transformer model to obtain the overall vehicle features and detailed vehicle features of the vehicle data, including: Acquire multimodal information of the target vehicle according to the text description information, acquire a feature sequence of the vehicle image data according to the vehicle image data, and convert the multimodal information and the feature sequence into an image feature representation; wherein the multimodal information includes a viewing angle, a camera type, and a camera type; Using a first visual transformer model to identify the image feature representation to obtain the overall vehicle features of the vehicle data; The second visual transformer model is used to identify the image feature representation through image mask reconstruction to obtain vehicle detail features of the vehicle data.

[0007] As an improvement of the above solution, the method of using a pre-trained text graph model to generate a word embedding vector of the text description information and a vehicle image feature from the text description information includes: Inputting the text description information into the pre-trained text graph model, and obtaining a word embedding vector according to the text description information; Generate vehicle latent space features of the text description information according to the word embedding vector and preset noise; The vehicle image features of the text description information are obtained by decoding the vehicle latent space features.

[0008] As an improvement of the above solution, before using the trained multimodal vehicle re-identification model, the method further includes: A multimodal vehicle re-identification model is constructed, and the multimodal vehicle re-identification model is pre-trained to obtain a trained multimodal vehicle re-identification model.

[0009] As an improvement of the above solution, the multimodal vehicle re-identification model is constructed, and the multimodal vehicle re-identification model is pre-trained to obtain a trained multimodal vehicle re-identification model, including: Constructing a multimodal vehicle re-identification model; wherein the multimodal vehicle re-identification model includes a visual transformer model, a text graph model and a visual transformer main network; Acquire sample vehicle image data and a vehicle re-identification dataset for each vehicle type in the target area; pre-process the sample vehicle image data and the vehicle re-identification dataset to obtain a sample training set and a pre-training dataset; Pre-training the visual transformer model using the vehicle image data in the pre-training data set to obtain a pre-trained visual transformer model; pre-training the Vincent graph model using the pre-training data set to obtain a pre-trained Vincent graph model; The sample training set is used to fine-tune the visual transformer main network through a pre-trained visual transformer model and the pre-trained text graph model to obtain a trained multi-modal vehicle re-identification model.

[0010] In order to achieve the above object, an embodiment of the present invention provides a vehicle re-identification device based on multimodal pre-training, comprising: A vehicle data acquisition module, used to acquire vehicle data of a target vehicle; wherein the vehicle data includes at least one of vehicle image data or text description information of the target vehicle; The vehicle information output module is used to use the trained multimodal vehicle re-identification model to extract features from the vehicle data to obtain final features of the vehicle data, and output the information of the target vehicle according to the final features; wherein the trained multimodal vehicle re-identification model includes a pre-trained visual transformer model, a pre-trained text graph model and a visual transformer main network.

[0011] As an improvement of the above solution, the multi-modal vehicle re-identification device 10 further includes: The recognition model training module is used to construct a multimodal vehicle re-identification model, pre-train the multimodal vehicle re-identification model, and obtain a trained multimodal vehicle re-identification model.

[0012] In order to achieve the above-mentioned purpose, an embodiment of the present invention provides a vehicle re-identification device based on multimodal pre-training, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the above-mentioned vehicle re-identification method based on multimodal pre-training when executing the computer program.

[0013] To achieve the above objectives, an embodiment of the present invention further provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the steps of the above-mentioned vehicle re-identification method based on multimodal pre-training.

[0014] Compared with the prior art, the vehicle re-identification method, device, equipment and product based on multimodal pre-training disclosed in the embodiment of the present invention obtains the vehicle data of the target vehicle; wherein the vehicle data at least includes one of the vehicle image data or text description information of the target vehicle; uses the trained multimodal vehicle re-identification model to extract features from the vehicle data to obtain the final features of the vehicle data, and outputs the information of the target vehicle according to the final features; wherein the trained multimodal vehicle re-identification model includes a pre-trained visual transformer model, a pre-trained text graph model and a visual transformer main network. It can support multiple query methods such as text query and text information embedding, improve the capture of image text information and the understanding of image semantics, improve the understanding of different modalities through multimodal information embedding to adapt to complex traffic scenes, and enhance the robustness and discriminability of extracted features through pre-training, thereby improving the performance of vehicle re-identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of a vehicle re-identification method based on multimodal pre-training provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of a multimodal information module conversion process provided by an embodiment of the present invention; Figure 3 is a schematic diagram of a pre-training process of a visual transformer model provided by an embodiment of the present invention; Figure 4 It is a pre-training process diagram of a cultural graph model provided by an embodiment of the present invention; Figure 5 A schematic diagram of a trained multimodal vehicle re-identification model framework provided by an embodiment of the present invention; Figure 6 is a structural schematic diagram of a vehicle re-identification device based on multimodal pre-training provided by an embodiment of the present invention; Figure 7It is a structural block diagram of a vehicle re-identification device based on multimodal pre-training provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] It should be noted that the terms "comprises" and "specifically" and any variations of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0018] See also Figure 1 , Figure 1 1 is a flow chart of a vehicle re-identification method based on multimodal pre-training provided by an embodiment of the present invention. The vehicle re-identification method based on multimodal pre-training includes: S1, obtaining vehicle data of a target vehicle; wherein the vehicle data includes at least one of vehicle image data or text description information of the target vehicle; S2, using a trained multimodal vehicle re-identification model to extract features from the vehicle data to obtain final features of the vehicle data, and output information of the target vehicle based on the final features; wherein the trained multimodal vehicle re-identification model includes a pre-trained visual transformer model, a pre-trained text graph model and a visual transformer main network.

[0019] Exemplarily, the vehicle re-identification method based on multimodal pre-training described in the embodiment of the present invention can be implemented by a vehicle re-identification server, and the vehicle re-identification server can interact with the target user and the traffic data system. The vehicle re-identification server obtains the vehicle data (such as vehicle image data or text description information) of the target vehicle from the traffic data system; if the vehicle data is vehicle image data, the vehicle image data is input into a trained multimodal vehicle re-identification model, and multimodal information (such as viewing angle and camera type, etc.) is obtained according to the vehicle image data, and the vehicle image data and multimodal information are identified using a pre-trained visual transformer model to obtain the overall vehicle features and vehicle detail features of the vehicle data; a pre-trained text graph model is used to generate word embedding vectors and vehicle image features of the multimodal information from the multimodal information; feature splicing is performed on the overall vehicle features and vehicle detail features, and the word embedding is transformed into a word embedding vector through a cross self-attention mechanism. The method integrates the spliced ​​vehicle features into the data to obtain the feature representation of the vehicle data; uses the visual transformer main network to fuse the feature representation and the vehicle image features to obtain the final features of the vehicle data, and outputs the information of the target vehicle according to the final features; it can support multiple query methods such as image query and text description query, as well as text information embedding function, to improve the model's capture of image text information and understanding of image semantics, and through the design of a cross self-attention mechanism, the text information embedding is fused with the image information with multimodal information to enhance the model's understanding of different modalities and reduce intra-class differences; at the same time, the pre-training is used to enhance the model's ability to extract robust and discriminative features, thereby improving the performance of vehicle re-identification.

[0020] Further, after executing step S1, if the vehicle data includes the vehicle image data and text description information, step S2 includes: S21, using a pre-trained visual transformer model to identify the vehicle data to obtain overall vehicle features and detailed vehicle features of the vehicle data; wherein the pre-trained visual transformer model includes a first visual transformer model and a second visual transformer model; S22, using a pre-trained text-based graph model to generate a word embedding vector of the text description information and a vehicle image feature from the text description information; S23, performing feature concatenation on the overall vehicle feature and the detailed vehicle feature, and integrating the word embedding vector into the concatenated vehicle feature through a cross self-attention mechanism to obtain a feature representation of the vehicle data; S24, using a visual transformer main network to perform feature fusion on the feature representation and the vehicle image feature to obtain final features of the vehicle data, and output information of the target vehicle according to the final features.

[0021] Specifically, the step S21 includes: S211, acquiring multimodal information of the target vehicle according to the text description information, acquiring a feature sequence of the vehicle image data according to the vehicle image data, and converting the multimodal information and the feature sequence into an image feature representation; wherein the multimodal information includes a viewing angle, a camera type, and a camera type; S212, using a first visual transformer model to identify the image feature representation to obtain overall vehicle features of the vehicle data; S213, using a second visual transformer model to identify the image feature representation through image mask reconstruction to obtain vehicle detail features of the vehicle data.

[0022] Exemplarily, in step S211, firstly, the multimodal information of the target vehicle (the viewing angle of the vehicle, the type of camera, etc.) is obtained according to the text description information, and at the same time, the vehicle image data is passed through a convolutional layer to obtain a feature sequence of the vehicle image data, and then the multimodal information module is used to convert the multimodal information and the feature sequence into an image feature representation, such as Figure 2 As shown, Figure 2 is a schematic diagram of a multimodal information module conversion process provided by an embodiment of the present invention. Taking the camera category information in the multimodal information as an example, assuming that there is camera categories (multimodal information embedding), creating a learnable multimodal information embedding matrix based on the camera category , whose size is ,in is the dimension of the trained multimodal vehicle re-identification model. Then for a camera category Vehicle image data, read its corresponding feature sequence (primary features) and multimodal embedding vectors , embedding the multimodal vector and position embedding vector (where the total number of position embedding vectors is m, which is determined by the number of feature sequences) and added to the initialized feature sequence In this paper, we obtain the image feature representation with multimodal information. as follows: , The same is true for other multimodal data. Then the image features with multimodal information are represented Input them into the pre-trained visual transformer model to obtain the overall characteristics of the vehicle and vehicle details The multimodal information embedding module in the embodiment of the present invention converts various multimodal information such as viewing angle and camera type into embedding vectors and adds them to the image feature sequence, so that the model can better learn the characteristics of the same vehicle in different modes during training, thereby reducing the characteristic deviation of similar vehicles caused by changes in factors such as viewing angle.

[0023] More specifically, the step S212 includes: Inputting the image feature representation into a first visual transformer model, dividing the image feature representation into a plurality of feature blocks, and obtaining a sequence representation corresponding to each feature block through linear mapping; Extracting a feature sequence corresponding to each feature block through an encoder according to the sequence representation and the position sequence of the image feature representation; Passing all feature sequences through a fully connected layer to obtain overall vehicle features of the vehicle data, wherein the overall vehicle features include scale information, vehicle location features, and vehicle category features of the vehicle data; Exemplarily, the image feature representation is divided into several feature blocks, and then the feature blocks are input into the linear mapping layer to obtain a sequence representation, which is added to the position sequence and input into the encoder of the visual Transformer to extract the feature sequence. After obtaining the feature sequence of the vehicle data, all feature sequences (e.g., K-dimensional feature sequences) are input into the first fully connected layer, and the vehicle position features and scale information (a 4-dimensional bounding box coordinate (x, y, h, w)) of the vehicle data are output, where x represents the horizontal coordinate of the center pixel of the bounding box, y represents the vertical coordinate of the center pixel of the bounding box, h represents the height of the bounding box, and w represents the width of the bounding box. In the second fully connected layer, the K-dimensional feature sequence is input into the fully connected layer 2, and the vehicle category feature (C-dimensional classification score) is output.

[0024] Specifically, the step S22 includes: S221, inputting the text description information into the pre-trained text graph model, and obtaining a word embedding vector according to the text description information; S222, generating a vehicle latent space feature of the text description information according to the word embedding vector and preset noise; S223, obtaining vehicle image features of the text description information according to the vehicle latent space feature decoding.

[0025] It is understandable that the preset noise can be obtained by training a pre-trained text-image model. The embodiment of the present invention supports text description query and text information embedding, thereby improving the model's capture of image text information and understanding of image semantics.

[0026] Exemplarily, in step S23, the overall vehicle features extracted by the pre-trained visual transformer model are and vehicle details Splice them together to get the fusion feature representation as follows: , Then, the word embedding vector y is added to the fusion feature through the cross self-attention mechanism to obtain a feature representation with more comprehensive and rich information. , the formula is as follows: , , , , In the formula, are query vector, key vector and value vector respectively, are learnable projection matrices corresponding to the query vector, key vector and value vector respectively; is the dimension of the key vector, The transpose symbol.

[0027] Exemplarily, in step S24, the feature representation The vehicle image features are input into a multi-layer visual Transformer encoder to obtain the final features of the vehicle data, and the vehicle retrieval results are obtained by sorting the final features from small to large according to the Euclidean distance matrix.

[0028] Further, after executing step S1, if the vehicle data only includes the vehicle image data, then step S2 includes: S201, using a pre-trained visual transformer model to identify the vehicle image data to obtain vehicle overall features and vehicle detail features of the vehicle image data; wherein the pre-trained visual transformer model includes a first visual transformer model and a second visual transformer model; S202, obtaining multimodal information of the vehicle image data, and using a pre-trained text-based graph model to generate a word embedding vector of the vehicle image data and a vehicle image feature from the multimodal information; S203, performing feature splicing on the overall vehicle features and the detailed vehicle features, and integrating the word embedding vector into the spliced ​​vehicle features through a cross self-attention mechanism to obtain a feature representation of the vehicle image data; S204, using a visual transformer main network to perform feature fusion on the feature representation and the vehicle image feature to obtain final features of the vehicle image data, and output information of the target vehicle according to the final features.

[0029] It is worth noting that the recognition process of the embodiment of the present invention has an additional step of obtaining multimodal information of the vehicle image data, and the remaining steps are the same as the recognition process of the vehicle data including the vehicle image data and text description information.

[0030] Further, after executing step S1, if the vehicle data only includes the text description information, step S2 includes: Using a pre-trained text-to-image model to generate vehicle image features of the text description information from the text description information; The visual transformer main network is used to perform feature fusion on the vehicle image features to obtain final features of the text description information, and the information of the target vehicle is output according to the final features.

[0031] Furthermore, before using the trained multimodal vehicle re-identification model, the method further includes: S101, constructing a multimodal vehicle re-identification model, and pre-training the multimodal vehicle re-identification model to obtain a trained multimodal vehicle re-identification model.

[0032] Specifically, the step S01 includes: S011, constructing a multimodal vehicle re-identification model; wherein the multimodal vehicle re-identification model includes a visual transformer model, a text graph model and a visual transformer main network; S012, obtaining sample vehicle image data and a vehicle re-identification data set for each vehicle type in the target area; preprocessing the sample vehicle image data and the vehicle re-identification data set to obtain a sample training set and a pre-training data set; S013, using the vehicle image data in the pre-training data set to pre-train the visual transformer model to obtain a pre-trained visual transformer model; using the pre-training data set to pre-train the Vincent graph model to obtain a pre-trained Vincent graph model; S014, using the sample training set through a pre-trained visual transformer model and the pre-trained text graph model, fine-tuning the visual transformer main network to obtain a trained multimodal vehicle re-identification model.

[0033] Exemplarily, in step S012, firstly, vehicle images captured by surveillance cameras on real urban roads within a certain range are collected, and it is ensured that each vehicle with a specific identity appears multiple times in different scenes and perspectives and that the vehicle types evenly cover trucks, buses, cars and other models. In the process of data collection, attention should be paid to the diversity of data, including changes in different brands, models, colors, perspectives, sizes, etc., and text descriptions should be added to each vehicle. Then, the collected vehicle image data is stored and annotated. For vehicle images with the same identity, their unique corresponding identity identifiers, i.e., IDs, are uniformly annotated. Then, for each vehicle image, its shooting perspective, camera number, and the bounding box coordinates containing the vehicle target are annotated; finally, the collected data set is divided into a training set, a query set, and a test set (library set), and it is ensured that the data in each set has similar distribution and characteristics. The above constructed data set is used as a data set in a simulated specific traffic task and participates in the training and testing of the multimodal vehicle re-identification model. In addition, the pre-training dataset uses the VeRi-776 vehicle re-identification dataset, which contains a collection of 776 vehicle images of different identities captured by 20 surveillance cameras in an urban area of ​​1 square kilometer for 24 consecutive hours. Through data enhancement methods such as Gaussian blur, color jitter, horizontal flip, border filling, image cropping, image standardization, random erasing, etc., a series of transformations and expansions are performed on the original data to generate new training samples to increase the diversity and quantity of the dataset, improve the generalization ability and robustness of the multimodal vehicle re-identification model, and reduce the risk of overfitting.

[0034] Exemplarily, in step S013, if Figure 3 As shown, Figure 3It is a schematic diagram of a visual transformer model pre-training process provided by an embodiment of the present invention; the pre-training process of the visual transformer model includes the pre-training of a first visual transformer model and the pre-training of a second visual transformer model, and the visual transformer model is pre-trained to obtain a pre-trained visual transformer model (pre-trained visual Transformer model). The pre-training of the first visual transformer model is a multi-stage pre-training of visual Transformer target detection. First, the first sub-branch, that is, the branch where the fully connected layer 1 is located, is frozen, and the second sub-branch, that is, the branch where the fully connected layer 2 is located, is used on the pre-training data set to perform vehicle category classification prediction, and the feature extraction network (visual Transformer encoder) and the parameters of the fully connected layer 2 are preliminarily trained so that the model first learns a feature representation with stronger expressive power for classification. Then freeze the parameters of the feature extraction network and the second branch, perform bounding box regression prediction training on the first sub-branch, and update the parameters of the fully connected layer 1, which can improve the training efficiency. At the same time, the bounding box prediction network is mainly responsible for learning the location and scale information of the target, while the feature extraction network is mainly responsible for extracting the visual features of the target. Therefore, freezing the feature extraction network can help maintain the stability of the feature representation and reduce the changes to the feature representation during training, which is conducive to improving the stability and robustness of the first visual transformer model. Finally, open all the parameters of the network for training again, which is conducive to the first visual transformer model to fully understand the location and scale information of the target and combine it with the vehicle features. The total loss function of the first visual transformer model includes the sum of the classification loss and the bounding box regression prediction loss.

[0035] Among them, the classification loss The calculation is as follows: , Bounding box regression prediction loss The calculation is as follows: , Total loss The formula is as follows: , In the formula, represents the prediction of the first visual transformer model Category Vehicle Images The probability distribution of is obtained by normalizing the classification scores through the softmax layer. Indicates the The true category distribution of the category, in the form of a one-hot vector, that is, the value corresponding to the true category is 1, the value of other categories is 0, and n is the total number of categories; represents the Euclidean distance between two coordinates, that is, the sum of the squared differences of the coordinates, represents the bounding box coordinates predicted by the first visual transformer model; where, Represents the horizontal coordinate of the center pixel of the bounding box, Represents the ordinate of the center pixel of the bounding box, represents the height of the bounding box, Indicates the width of the bounding box; represents the ground-truth bounding box coordinates of the annotation, where Represents the horizontal coordinate of the center pixel of the true bounding box, Represents the ordinate of the center pixel of the true bounding box, represents the height of the true bounding box, Indicates the width of the ground-truth bounding box.

[0036] The embodiments of the present invention can improve the network's ability to perceive and understand the position and bounding box of the target by learning vehicle image positioning, so that the pre-trained first visual transformer model can well understand the impact of changes in the position of vehicles with the same identity on the features, thereby improving the translation invariance of the first visual transformer model.

[0037] like Figure 3 As shown in the figure, the pre-training of the second visual transformer model is the unsupervised pre-training of the visual transformer. Through unsupervised pre-training, the model can initially master the knowledge in the traffic field and be better applied to specific tasks in the traffic field. The input vehicle image is first randomly masked and divided into several blocks. Then each image block is input into the linear mapping layer to obtain a sequence representation. The sequence is added to the position sequence and input into the encoder of the visual transformer to extract the feature representation of the image. The encoded representation output by the encoder is then received by the decoder network and decoded into an image similar to the original image block. The goal of training the autoencoder is to minimize the difference between the decoded image and the original image. The mean square error (MSE) is usually used as the loss function. This loss function measures the reconstruction ability of the decoder, that is, the network is able to decode the feature encoding representation into an image as similar as possible to the original image block. The loss function of the second visual transformer model The calculation formula is as follows: , In the formula, represents the total number of samples of vehicle images, It is The vehicle image reconstructed by the decoder, For vehicle images The corresponding original vehicle image.

[0038] The embodiment of the present invention generates a vehicle image through unsupervised pre-training to realize mask image reconstruction. The second visual transformer model can learn more detailed feature information, strengthen the modeling of dependencies between different sequences, and deepen the model's attention to local features by adding masks, thereby combining supervised and unsupervised training methods.

[0039] like Figure 4 As shown, Figure 4 This is a diagram of a pre-training process of a Wensheng graph model provided by an embodiment of the present invention; the pre-training of the Wensheng graph model mainly trains the Denoise module (denoising module), namely U-Net. The U-Net noise prediction network training process is as follows: first, the diffusion process, the purpose of which is to gradually generate a disordered image through T steps. The specific operation process is for a clean original image , mapped to the latent space through the encoder, and the latent space representation of the image is obtained , in step 1, a Gaussian random noise is sampled , and then the random noise join in The noisy image is obtained Repeat this process until step T to get the final random image Then input the text description corresponding to the vehicle image, and convert the text information into a word embedding vector through a CLIP text encoder. Then repeat T rounds, each round embedding the text word vector v, step number s, latent space random representation The image generation module first outputs the noise prediction , then Subtract the noise prediction for this round to get the generated image for:

[0040] The goal of U-Net network optimization is to make the noise prediction generated by it in step s The target noise added in the sth step during the diffusion process As close as possible, the loss function of the Wensheng graph model for: .

[0041] The embodiment of the present invention can support multiple query methods such as text query and text information embedding by pre-training the text-based graph model, improve the model's capture of image text information and understanding of image semantics, and support diversified query input.

[0042] Exemplarily, in step S014, Figure 5 As shown, Figure 5 A schematic diagram of a trained multimodal vehicle re-identification model framework provided in an embodiment of the present invention; the sample training set is used to obtain vehicle overall features, vehicle detail features, word embedding vectors and vehicle image features through a pre-trained visual transformer model (multimodal information embedding module and two pre-trained ViTs) and the pre-trained text graph model, a cross self-attention fusion module is used to perform feature splicing on the vehicle overall features and vehicle detail features, and the word embedding vector is integrated into the spliced ​​vehicle features through a cross self-attention mechanism to obtain a feature representation of the vehicle data, and the feature representation and vehicle image features are input into a multi-layer visual Transformer encoder to obtain the final features of the vehicle. For the final feature of the vehicle, on the one hand, the feature is input into the fully connected layer for classification to obtain the classification loss (also known as the cross entropy loss). On the other hand, the feature is used as an anchor point to form a triplet with other features of the same batch. The positive example uses the feature with the closest Euclidean distance to the feature, and the negative example uses the feature of the same batch with the farthest Euclidean distance to the feature. Then the triplet loss function is calculated, the classification loss and the triplet loss are added to obtain the total loss, and the model weights are updated by reverse gradient propagation until the training reaches the specified round. The main network of the visual transformer is fine-tuned to obtain a trained multimodal vehicle re-identification model.

[0043] The trained multimodal vehicle re-identification model of the embodiment of the present invention supports multiple query combinations such as text description information query. Text query and text information embedding are supported through the text-to-image task, which improves the model's ability to capture image text information and understand image semantics. The model's understanding of different modalities is improved by designing a multimodal information embedding module, reducing intra-class differences, and enhancing the model's ability to extract robust and discriminative features through multi-task pre-training.

[0044] The embodiment of the present invention discloses a vehicle re-identification method based on multimodal pre-training, which obtains vehicle data of a target vehicle; wherein the vehicle data includes at least one of the vehicle image data or text description information of the target vehicle; uses a trained multimodal vehicle re-identification model to extract features from the vehicle data to obtain the final features of the vehicle data, and outputs information of the target vehicle based on the final features; wherein the trained multimodal vehicle re-identification model includes a pre-trained visual transformer model, a pre-trained text graph model, and a visual transformer main network. It can support multiple query methods such as text query and text information embedding, improve the capture of image text information and the understanding of image semantics, enhance the understanding of different modalities through multimodal information embedding to adapt to complex traffic scenes, and enhance the robustness and discriminability of extracted features through pre-training, thereby improving the performance of vehicle re-identification.

[0045] See also Figure 6 , Figure 6 1 is a schematic diagram of a vehicle re-identification device 10 based on multimodal pre-training provided by an embodiment of the present invention. The vehicle re-identification device 10 based on multimodal pre-training includes: The vehicle data acquisition module 11 is used to acquire vehicle data of a target vehicle; wherein the vehicle data at least includes one of vehicle image data or text description information of the target vehicle; The vehicle information output module 12 is used to use the trained multimodal vehicle re-identification model to extract features from the vehicle data to obtain final features of the vehicle data, and output the information of the target vehicle according to the final features; wherein the trained multimodal vehicle re-identification model includes a pre-trained visual transformer model, a pre-trained text graph model and a visual transformer main network.

[0046] Furthermore, the vehicle re-identification device 10 based on multimodal pre-training further includes: The recognition model training module is used to construct a multimodal vehicle re-identification model, pre-train the multimodal vehicle re-identification model, and obtain a trained multimodal vehicle re-identification model.

[0047] A vehicle re-identification device 10 based on multimodal pre-training provided in an embodiment of the present invention can implement all the processes of the vehicle re-identification method based on multimodal pre-training in the above-mentioned embodiment. The functions of each module in the device and the technical effects achieved are respectively the same as those of the vehicle re-identification method based on multimodal pre-training in the above-mentioned embodiment, and are not repeated here.

[0048] See also Figure 7 , Figure 7 1 is a schematic diagram of the structure of a vehicle re-identification device 20 based on multimodal pre-training provided in an embodiment of the present invention. The vehicle re-identification device 20 based on multimodal pre-training of this embodiment includes: a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, the steps in the above-mentioned vehicle re-identification method embodiment based on multimodal pre-training are implemented. Alternatively, when the processor 21 executes the computer program, the functions of each module in the above-mentioned vehicle re-identification device embodiment based on multimodal pre-training are implemented.

[0049] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory 22 and executed by the processor 21 to implement the present invention. The one or more modules may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the vehicle re-identification device 20 based on multimodal pre-training.

[0050] The vehicle re-identification device 20 based on multimodal pre-training can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The vehicle re-identification device 20 based on multimodal pre-training can include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art can understand that the schematic diagram is only an example of the vehicle re-identification device 20 based on multimodal pre-training, and does not constitute a limitation on the vehicle re-identification device 20 based on multimodal pre-training, and can include more or less components than shown in the figure, or combine certain components, or different components. For example, the vehicle re-identification device 20 based on multimodal pre-training can also include input and output devices, network access devices, buses, etc.

[0051] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 21 is the control center of the vehicle re-identification device 20 based on multimodal pre-training, and uses various interfaces and lines to connect various parts of the vehicle re-identification device 20 based on multimodal pre-training.

[0052] The memory 22 can be used to store the computer program and / or module. The processor 21 realizes various functions of the vehicle re-identification device 20 based on multimodal pre-training by running or executing the computer program and / or module stored in the memory 22 and calling the data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0053] Wherein, if the module integrated in the vehicle re-identification device 20 based on multimodal pre-training is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor 21, the steps of the above-mentioned method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunication signals.

[0054] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0055] In addition, an embodiment of the present invention further provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the vehicle re-identification method based on multimodal pre-training of the above embodiment.

[0056] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A vehicle re-identification method based on multimodal pre-training, characterized in that: include: Acquire vehicle data of a target vehicle; wherein the vehicle data includes at least one of vehicle image data or text description information of the target vehicle; A trained multimodal vehicle re-identification model is used to extract features from the vehicle data to obtain final features of the vehicle data, and information of the target vehicle is output according to the final features; wherein the trained multimodal vehicle re-identification model includes a pre-trained visual transformer model, a pre-trained text graph model and a visual transformer main network.

2. The vehicle re-identification method based on multimodal pre-training according to claim 1, characterized in that: If the vehicle data includes the vehicle image data and text description information, the trained multimodal vehicle re-identification model is used to extract features from the vehicle data to obtain final features of the vehicle data, and information of the target vehicle is output according to the final features, including: Using a pre-trained visual transformer model to identify the vehicle data, and obtaining overall vehicle features and detailed vehicle features of the vehicle data; wherein the pre-trained visual transformer model includes a first visual transformer model and a second visual transformer model; Using a pre-trained text-based graph model to generate a word embedding vector of the text description information and a vehicle image feature from the text description information; Performing feature splicing on the overall features of the vehicle and the detailed features of the vehicle, and integrating the word embedding vector into the spliced ​​vehicle features through a cross self-attention mechanism to obtain a feature representation of the vehicle data; The feature representation and the vehicle image feature are subjected to feature fusion by using a visual transformer main network to obtain final features of the vehicle data, and information of the target vehicle is output according to the final features.

3. The vehicle re-identification method based on multimodal pre-training according to claim 2, characterized in that: The vehicle data is identified by using a pre-trained visual transformer model to obtain overall vehicle features and detailed vehicle features of the vehicle data, including: Acquire multimodal information of the target vehicle according to the text description information, acquire a feature sequence of the vehicle image data according to the vehicle image data, and convert the multimodal information and the feature sequence into an image feature representation; wherein the multimodal information includes a viewing angle, a camera type, and a camera type; Using a first visual transformer model to identify the image feature representation to obtain the overall vehicle features of the vehicle data; The second visual transformer model is used to identify the image feature representation through image mask reconstruction to obtain vehicle detail features of the vehicle data.

4. The vehicle re-identification method based on multimodal pre-training as claimed in claim 2, characterized in that: The method of using the pre-trained text-based graph model to generate a word embedding vector and a vehicle image feature of the text description information includes: Inputting the text description information into the pre-trained text graph model, and obtaining a word embedding vector according to the text description information; Generate vehicle latent space features of the text description information according to the word embedding vector and preset noise; The vehicle image features of the text description information are obtained by decoding the vehicle latent space features.

5. The vehicle re-identification method based on multimodal pre-training according to claim 1, characterized in that: Before using the trained multimodal vehicle re-identification model, the method further includes: A multimodal vehicle re-identification model is constructed, and the multimodal vehicle re-identification model is pre-trained to obtain a trained multimodal vehicle re-identification model.

6. The vehicle re-identification method based on multimodal pre-training according to claim 5, characterized in that: The constructing of the multimodal vehicle re-identification model and pre-training the multimodal vehicle re-identification model to obtain a trained multimodal vehicle re-identification model includes: Constructing a multimodal vehicle re-identification model; wherein the multimodal vehicle re-identification model includes a visual transformer model, a text graph model and a visual transformer main network; Acquire sample vehicle image data and a vehicle re-identification dataset for each vehicle type in the target area; pre-process the sample vehicle image data and the vehicle re-identification dataset to obtain a sample training set and a pre-training dataset; Pre-training the visual transformer model using the vehicle image data in the pre-training data set to obtain a pre-trained visual transformer model; pre-training the Vincent graph model using the pre-training data set to obtain a pre-trained Vincent graph model; The sample training set is used to fine-tune the visual transformer main network through a pre-trained visual transformer model and the pre-trained text graph model to obtain a trained multi-modal vehicle re-identification model.

7. A vehicle re-identification device based on multimodal pre-training, characterized in that: include: A vehicle data acquisition module, used to acquire vehicle data of a target vehicle; wherein the vehicle data includes at least one of vehicle image data or text description information of the target vehicle; The vehicle information output module is used to use the trained multimodal vehicle re-identification model to extract features from the vehicle data to obtain final features of the vehicle data, and output the information of the target vehicle according to the final features; wherein the trained multimodal vehicle re-identification model includes a pre-trained visual transformer model, a pre-trained text graph model and a visual transformer main network.

8. The vehicle re-identification device based on multimodal pre-training as claimed in claim 7, characterized in that: include: The recognition model training module is used to construct a multimodal vehicle re-identification model, pre-train the multimodal vehicle re-identification model, and obtain a trained multimodal vehicle re-identification model.

9. A vehicle re-identification device based on multimodal pre-training, characterized in that: The invention comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the vehicle re-identification method based on multimodal pre-training as described in any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the vehicle re-identification method based on multimodal pre-training as described in any one of claims 1 to 6.

Images