Image-text cross-modal vehicle retrieval model training method in vehicle-dense scenes

Abstract

The present invention relates to the field of computer technology, and provides a method for training an image-text cross-modal vehicle retrieval model in a vehicle-dense scene. The method solves the problem in the prior art that the vehicle retrieval model cannot accurately understand the main content of the image, and improves the retrieval accuracy. The scheme includes obtaining a training sample set, performing feature extraction on image samples and text samples, and obtaining a visual-text joint embedding; calculating the semantic features of the image samples; calculating the feature similarity between the image samples and the text samples based on cosine similarity to obtain global and local feature similarities, integrating the global and local feature similarities through a learnable network to obtain the final inter-modal similarity; calculating the semantic similarity between image samples of traffic scenes; and updating the weights according to a loss function, wherein the loss function includes a triple loss and an image semantic loss. The present invention is used for image-text cross-modal vehicle retrieval in a vehicle-dense scene.

Description

Technical Field

[0001] The present invention relates to the field of computer technology and provides an image-text cross-modal vehicle retrieval model training method in a vehicle-intensive scene. Background Art

[0002] With the rapid development of digitalization and informatization, graphic and text data are constantly generated on social media, monitoring points, e-commerce and other platforms, and the development of intelligent transportation systems is gradually tending to improve multimodal information processing and comparison capabilities. As a basic technology in cross-modal retrieval, graphic and text matching aims to accurately match related text and images, and plays a vital role in traffic scenarios, such as vehicle retrieval and target vehicle re-identification. At present, the graphic and text matching model for traffic scenes cannot accurately focus on the main semantics in the image, and thus cannot extract image features that are consistent with the main semantics of the picture.

[0003] In summary, the existing traffic scene image-text cross-modal vehicle retrieval has the following shortcomings:

[0004] 1) The existing vehicle retrieval model only updates the network parameters of the retrieval model based on the difference between the inter-modal similarity and the actual value, without considering whether these image features reflect the main semantics of the picture;

[0005] 2) The triplet loss commonly used in cross-modal image-text matching tasks cannot accurately measure whether the similarity between the main semantics of the image and the text semantics is consistent with the actual value.

[0006] The shortcomings of the above-mentioned prior art will affect the application of the image-text matching model in traffic scenes with dense vehicles, making it impossible to efficiently retrieve vehicles that meet the text description, which brings great inconvenience to the intelligent transportation system. Therefore, the present invention proposes an image-text cross-modal vehicle retrieval training method in vehicle-dense scenes. Summary of the invention

[0007] The present invention aims to solve the problem that the existing traffic scene image-text cross-modal vehicle retrieval model cannot accurately understand the main content of the image in vehicle-dense scenes, resulting in low retrieval accuracy.

[0008] To achieve the above purpose, the present invention adopts the following technical solutions:

[0009] The present invention provides an image-text cross-modal vehicle retrieval model training method in a vehicle-intensive scene, comprising:

[0010] S1: obtaining a training sample set, wherein the training sample set includes frame image data of densely packed vehicles in a plurality of traffic flow monitoring videos and their corresponding description texts, i.e., image samples and text samples, respectively performing feature extraction on the image samples and the text samples, and obtaining a visual-text joint embedding;

[0011] S2: Calculate the TFIDF vector of the description text corresponding to the image sample to obtain the semantic features of the image sample;

[0012] S3: Calculate the feature similarity between image samples and text samples based on cosine similarity to obtain global and local feature similarities. Integrate the global and local feature similarities through a learnable network to obtain the final inter-modal similarity.

[0013] S4: Calculate the semantic similarity between image samples of traffic scenes;

[0014] S5: weights updated according to a loss function, wherein the loss function includes triplet loss and image semantic loss.

[0015] In the above method, in S1, the detailed description of feature extraction is as follows:

[0016] S11: Obtain the embedded representation of the image, where an image feature encoder is built based on the deep convolutional neural network Faster RCNN to extract the original image features of the image sample to be processed; a network is built based on the self-attention mechanism to transform the original features of the image sample to obtain the global image features ;

[0017] S12: Obtain the embedded representation of the text, where a text feature encoder is constructed based on the bidirectional gated recurrent unit structure Bi-GRU to extract the text features of the text to be processed and obtain the global text features ;

[0018] S13: Obtain visual-text joint embedding, where a network is built based on the cross-attention mechanism to capture the fine-grained correspondence between image regions and text words, and the global image features are respectively and global text features Converted into local image features containing more image-text correspondence information and local text features , and obtain the joint visual-text embedding.

[0019] In the above method, in S2, the steps of extracting image semantic features are as follows:

[0020] S21: The input data of the image semantic feature calculation method is the description text corresponding to the image sample, wherein the TFIDF vector of the description text is calculated based on the word frequency-inverse document frequency, and the TFIDF vector is used to represent the semantic feature of the image sample;

[0021] S22: The calculation process of the word frequency-inverse document rate method is as follows:

[0022]

[0023] in For words In the description text The number of occurrences in yes The total number of words contained in is the total number of words contained in the text set in the training set, The text set contains words The total number of entries of texts, where the text samples constitute the text set;

[0024] S23: The semantic feature dimension of the image sample is equal to the total number of different words contained in the text set. Each word corresponds to an attribute of the image semantic feature vector. If there is a word in the text, in the image sample semantic feature vector, the attribute corresponding to the word is the TFIDF value of the word and the other positions in the image sample semantic feature vector are 0. In this way, the semantic features of the image can be obtained.

[0025] The method for calculating the feature similarity between the image sample and the text sample in S3 in the above method is described in detail as follows:

[0026] S31: Calculate the feature similarity between images and text based on cosine similarity. The formula is as follows:

[0027]

[0028] in, and are the feature vectors of image samples and text samples respectively;

[0029] The feature similarity between the image and text is divided into two parts, namely global feature similarity and local feature similarity. The specific strategy is as follows:

[0030] S311: The global feature similarity is calculated from the global image feature and the global text feature, reflecting the feature similarity between the modalities at a macro level, and is recorded as ,in and Calculated from S11 and S12;

[0031] S312: The local feature similarity is calculated from the local image feature and the local text feature, reflecting the feature similarity between the image region and the word, denoted as ,in , is calculated by S13, where The meaning is the local image feature vector of the nth image, The meaning of is the local text feature vector of the nth image;

[0032] S32: Build a learnable network based on the fully connected layer, integrate the global and local feature similarities obtained in S31 into the final inter-modal similarity, and obtain the inter-modal feature similarity vector , the formula is as follows:

[0033]

[0034] in, is a learnable weight vector, is the sigmoid function, is the similarity obtained from S31, is the batch normalization function, which calculates the normalized weights , and then calculate again to get the feature similarity vector between modalities .

[0035] In the above method, the semantic similarity between image samples of traffic scenes in S4 includes the following steps:

[0036] S41: Use the CIDEr index to evaluate the semantic similarity between traffic scene images and obtain the semantic similarity. The semantic similarity matrix C is constructed, and the specific formula is as follows:

[0037]

[0038] in, is an image sample of a traffic scene and image samples The semantic similarity between is the number of entries of description text corresponding to the image sample, is an image No. Description text, is the TFIDF vector of the text obtained by S22 and S23, represents the first i Image samples.

[0039] The loss function in S5 in the above method is specifically described as follows:

[0040] S51: Update the weights in the network according to the value of the loss function. The loss function includes two parts, namely, triplet loss and image semantic loss. The triplet loss is used to measure the difference between the similarity between modalities and the actual value. The formula is as follows:

[0041]

[0042] in is the boundary parameter, where is the positive sample set of images, is the description text positive sample set, is the set of negative image samples, is a description text negative sample set, where It is the inter-modal similarity matrix obtained by calculating the sample image feature vectors and text feature vectors in the image set and text set using the calculation method in s3. , where s1, s2, , sn are all calculated by the inter-modal similarity equation in S32, s1 represents the inter-modal similarity feature vector between the first image in the image positive sample set and its corresponding description text positive sample;

[0043] S52: The image semantic loss is used to measure the difference between the feature similarity matrix and the image semantic similarity matrix. The specific strategy is as follows:

[0044] S521: Based on the sigmoid function, the similarity value matrix is ​​converted into a similarity ranking matrix, that is, the feature similarity matrix S and the semantic similarity matrix C are sorted to obtain the similarity ranking matrix. The sigmoid function adds a temperature parameter The ranking value is closer to the real ranking value, while ensuring the effectiveness of the gradient. The formula is as follows:

[0045]

[0046] in, is the similarity value matrix to be sorted, ,in Represents the similarity matrix i Row, No. j Column elements, is the batch size during training, is the sigmoid function with the temperature parameter added. The feature similarity matrix and semantic similarity matrix after sorting are and ;

[0047] S522: The input data of image semantic loss is and , the formula is as follows:

[0048]

[0049] in, is the batch size during training, and is the sorting matrix obtained from S521 and The value of the corresponding position;

[0050] S53: The loss function is ultimately composed of two loss functions, and the formula is as follows:

[0051]

[0052] in, and Used to balance the two losses, and .

[0053] Because the present invention adopts the above technical means, it has the following beneficial effects:

[0054] 1. Visual-text joint embedding: Through the deep convolutional neural network Faster RCNN and the self-attention mechanism, combined with the bidirectional gated recurrent unit Bi-GRU, the features of the image and text are extracted to obtain the visual-text joint embedding. This method can more effectively capture the fine-grained correspondence between image regions and text words, and improve the accuracy of retrieval.

[0055] 2. Image semantic feature calculation: The semantic features of the text are calculated using the term frequency-inverse document frequency (TFIDF) method to represent the semantic features of the image. This method takes into account the problem that the model is difficult to extract effective image features in vehicle-intensive scenes, making the model better adapted to the intelligent transportation system.

[0056] 3. Feature similarity calculation: Calculate the feature similarity between the image and the text based on cosine similarity, including global feature similarity and local feature similarity. The similarity is integrated through the fully connected layer and the sigmoid function to obtain the final inter-modal similarity.

[0057] 4. Semantic similarity calculation: The CIDEr index is used to evaluate the semantic similarity between traffic scene images, thereby establishing semantic connections between images.

[0058] 5. Loss function design: The loss function includes triplet loss and image semantic loss, which are used to measure the difference between the inter-modal similarity and the actual value, as well as the difference between the feature similarity matrix and the image semantic similarity matrix. Through this design, the model can pay more attention to the main semantic content of the image.

[0059] 6. Performance improvement: By introducing the image semantic loss function, the present invention solves the problem in the prior art that the vehicle retrieval model cannot accurately understand the main content of the image, and improves the retrieval accuracy of the model in vehicle-dense scenes.

[0060] In summary, this invention significantly improves the accuracy and efficiency of image-text cross-modal vehicle retrieval in vehicle-dense scenes through innovative visual-text joint embedding methods, image semantic feature calculation, feature and semantic similarity calculation, and designed loss function, which is of great significance to the development of intelligent transportation systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A block diagram of an image-text cross-modal vehicle retrieval training method in a vehicle-dense scene provided for the implementation of the present invention;

[0062] Figure 2 A schematic diagram of the structure of an image-text cross-modal vehicle retrieval training method in a vehicle-dense scene provided for the implementation of the present invention;

[0063] Figure 3 Schematic diagram of traffic scene image and description text of target vehicle.

[0064] Figure 4 A table of key fields describing the text;

[0065] Figure 5 Pseudocode for testing the vehicle retrieval model training method. DETAILED DESCRIPTION

[0066] The following is a detailed description of the embodiments of the present invention. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, modifications or equivalent substitutions made to the present invention should all be included in the scope of the claims of the present invention.

[0067] In addition, in order to better illustrate the present invention, numerous specific details are given in the following specific embodiments. It will be understood by those skilled in the art that the present invention can also be implemented without these specific details.

[0068] The present invention provides an image-text cross-modal vehicle retrieval model training method in a vehicle-intensive scene, comprising:

[0069] S1: obtaining a training sample set, wherein the training sample set includes frame image data of densely packed vehicles in a plurality of traffic flow monitoring videos and their corresponding description texts, i.e., image samples and text samples, respectively performing feature extraction on the image samples and the text samples, and obtaining a visual-text joint embedding;

[0070] S2: Calculate the TFIDF vector of the description text corresponding to the image sample to obtain the semantic features of the image sample;

[0071] S3: Calculate the feature similarity between image samples and text samples based on cosine similarity to obtain global and local feature similarities. Integrate the global and local feature similarities through a learnable network to obtain the final inter-modal similarity.

[0072] S4: Calculate the semantic similarity between image samples of traffic scenes;

[0073] S5: weights updated according to a loss function, wherein the loss function includes triplet loss and image semantic loss.

[0074] In the above method, in S1, the detailed description of feature extraction is as follows:

[0075] S11: Obtain the embedded representation of the image, where an image feature encoder is built based on the deep convolutional neural network Faster RCNN to extract the original image features of the image sample to be processed; a network is built based on the self-attention mechanism to transform the original features of the image sample to obtain the global image features ;

[0076] S12: Obtain the embedded representation of the text, where a text feature encoder is constructed based on the bidirectional gated recurrent unit structure Bi-GRU to extract the text features of the text to be processed and obtain the global text features ;

[0077] S13: Obtain visual-text joint embedding, where a network is built based on the cross-attention mechanism to capture the fine-grained correspondence between image regions and text words, and the global image features are respectively and global text features Converted into local image features containing more image-text correspondence information and local text features , and obtain the joint visual-text embedding.

[0078] In the above method, in S2, the steps of extracting image semantic features are as follows:

[0079] S21: The input data of the image semantic feature calculation method is the description text corresponding to the image sample, wherein the TFIDF vector of the description text is calculated based on the word frequency-inverse document frequency, and the TFIDF vector is used to represent the semantic feature of the image sample;

[0080] S22: The calculation process of the word frequency-inverse document rate method is as follows:

[0081]

[0082] in For words In the description text The number of occurrences in yes The total number of words contained in is the total number of words contained in the text set in the training set, The text set contains words The total number of entries of texts, where the text samples constitute the text set;

[0083] S23: The semantic feature dimension of the image sample is equal to the total number of different words contained in the text set. Each word corresponds to an attribute of the image semantic feature vector. If there is a word in the text, in the image sample semantic feature vector, the attribute corresponding to the word is the TFIDF value of the word and the other positions in the image sample semantic feature vector are 0. In this way, the semantic features of the image can be obtained.

[0084] The method for calculating the feature similarity between the image sample and the text sample in S3 in the above method is described in detail as follows:

[0085] S31: Calculate the feature similarity between images and text based on cosine similarity. The formula is as follows:

[0086]

[0087] in, and are the feature vectors of image samples and text samples respectively;

[0088] The feature similarity between the image and text is divided into two parts, namely global feature similarity and local feature similarity. The specific strategy is as follows:

[0089] S311: The global feature similarity is calculated from the global image feature and the global text feature, reflecting the feature similarity between the modalities at a macro level, and is recorded as ,in and Calculated from S11 and S12;

[0090] S312: The local feature similarity is calculated from the local image feature and the local text feature, reflecting the feature similarity between the image region and the word, denoted as ,in , is calculated by S13, where The meaning is the local image feature vector of the nth image, The meaning of is the local text feature vector of the nth image;

[0091] S32: Build a learnable network based on the fully connected layer, integrate the global and local feature similarities obtained in S31 into the final inter-modal similarity, and obtain the inter-modal feature similarity vector , the formula is as follows:

[0092]

[0093] in, is a learnable weight vector, is the sigmoid function, is the similarity obtained from S31, is the batch normalization function, which calculates the normalized weights , and then calculate again to get the feature similarity vector between modalities .

[0094] In the above method, the semantic similarity between image samples of traffic scenes in S4 includes the following steps:

[0095] S41: Use the CIDEr index to evaluate the semantic similarity between traffic scene images and obtain the semantic similarity. The semantic similarity matrix C is constructed, and the specific formula is as follows:

[0096]

[0097] in, is an image sample of a traffic scene and image samples The semantic similarity between is the number of entries of description text corresponding to the image sample, is an image No. Description text, is the TFIDF vector of the text obtained by S22 and S23, represents the first i Image samples.

[0098] The loss function in S5 in the above method is specifically described as follows:

[0099] S51: Update the weights in the network according to the value of the loss function. The loss function includes two parts, namely, triplet loss and image semantic loss. The triplet loss is used to measure the difference between the similarity between modalities and the actual value. The formula is as follows:

[0100]

[0101] in is the boundary parameter, where is the positive sample set of images, is the description text positive sample set, is the set of negative image samples, is a description text negative sample set, where It is the inter-modal similarity matrix obtained by calculating the sample image feature vectors and text feature vectors in the image set and text set using the calculation method in s3. , where s1, s2, , sn are all calculated by the inter-modal similarity equation in S32, s1 represents the inter-modal similarity feature vector between the first image in the image positive sample set and its corresponding description text positive sample;

[0102] S52: The image semantic loss is used to measure the difference between the feature similarity matrix and the image semantic similarity matrix. The specific strategy is as follows:

[0103] S521: Based on the sigmoid function, the similarity value matrix is ​​converted into a similarity ranking matrix, that is, the feature similarity matrix S and the semantic similarity matrix C are sorted to obtain the similarity ranking matrix. The sigmoid function adds a temperature parameter The ranking value is closer to the real ranking value, while ensuring the effectiveness of the gradient. The formula is as follows:

[0104]

[0105] in, is the similarity value matrix to be sorted, ,in Represents the similarity matrix i Row, No. j Column elements, is the batch size during training, is the sigmoid function with the temperature parameter added. The feature similarity matrix and semantic similarity matrix after sorting are and ;

[0106] S522: The input data of image semantic loss is and , the formula is as follows:

[0107]

[0108] in, is the batch size during training, and is the sorting matrix obtained from S521 and The value of the corresponding position;

[0109] S53: The loss function is ultimately composed of two loss functions, and the formula is as follows:

[0110]

[0111] in, and Used to balance the two losses, and . Example

[0112] 1. Experimental dataset processing instructions

[0113] The public dataset FindVehicle is used for training and testing. 177,000 images are used for training, 5,000 images are used for verification, and 5,000 images are used for testing. The data format example is as follows Figure 3 As shown, the key fields of the description text are as follows Figure 4 shown.

[0114] 2. Graphics and text encoder description

[0115] Based on the pre-trained Faster RCNN network, image features are extracted and converted into visual embeddings according to the steps described above. The dimension of the obtained visual modality embedding is 2048. The text is numerically processed based on the Bi-GRU model and the text is divided into A set of words is taken and digitized, mapping the words into vectors with a dimension of 2048.

[0116] 3. Test description

[0117] The present invention discloses an image-text cross-modal vehicle retrieval training method in vehicle-dense scenes. The core of the present invention is to introduce an image semantic loss function into the training process of the retrieval model, supplement the limitations of the original triplet loss, align the feature similarity between modalities with the semantic similarity within the modality, so that the vehicle retrieval model can accurately focus on the main semantic content in the image, and extract it as the main feature embedding of the image to match the text. This method improves the retrieval accuracy of the retrieval model in vehicle-dense scenes, making it easier for the model to focus on vehicles in the image that match the text. This verification test is mainly used to simulate the application scenario of retrieving related vehicles based on descriptive text in vehicle-dense scenes, so as to verify whether the present invention can achieve better performance than traditional training methods, that is, higher accuracy.

[0118] 1. Functional testing and result analysis

[0119] This verification test is mainly used to simulate and test whether the present invention can accurately retrieve relevant vehicles based on the description text in a vehicle-intensive scene. The pseudo code for the vehicle retrieval model training method is as follows: Figure 5 shown.

[0120] Through the above verification test, it can be verified that the present invention can improve the retrieval ability of the cross-modal retrieval model. Specifically, according to the steps of the above algorithm, a network structure based on the SGRAF model is first initialized, and the number of training rounds is set to 25 rounds. During the training process, the retrieval model is first used to calculate the feature embedding of the training set image and text, and the semantic embedding (semantic feature) of the image is calculated according to the TFIDF algorithm. Then the feature similarity between the modalities is calculated based on the feature embedding of the image and text, and the semantic similarity within the modality is calculated based on the semantic embedding of the image. Then the traditional loss triplet loss function of the cross-modal task is calculated, and the image semantic loss proposed by the present invention is calculated. Update the network parameters according to the two loss values, and update the model. After 25 rounds of training, the model effect is tested on the test set, and the trained retrieval model is used to calculate the feature similarity between the test set image and text, and compared with the true similarity, and the recall rate of the model is statistically calculated.

[0121] 2. Performance test and result analysis

[0122] The recall rate Recall@K reflects the accuracy of the retrieval model. The matching results of the model are sorted from large to small according to the similarity score. Recall@K reflects the ratio of successfully matching the correct samples in the first K matching results. Therefore, Recall@1 represents the ability of the model to accurately retrieve the correct samples. The calculation formula is shown below.

[0123]

[0124] in, represents the number of samples in the set, is the test set, Set to The experimental simulation retrieval model is used for vehicle retrieval after 25 rounds of training. Five different test data sets are used, each containing 1000 sample data. The relevant images are retrieved based on the text, and the recall rates Recall@1, Recall@5 and Recall@10 of the model are calculated. The simulation experimental results are shown in Table 2.

[0125] Table 2: Model test recall results

[0126] Recall@1 Recall@5 Recall@10 Test Dataset 1 0.649 0.905 0.952 Test Dataset 2 0.618 0.887 0.941 Test Dataset 3 0.639 0.901 0.950 Test Dataset 4 0.616 0.885 0.940 Test Dataset 5 0.635 0.902 0.950

[0127] 3. Test Summary

[0128] This section conducts functional and performance tests on the present invention. Through a series of experimental comparisons, the rationality and effectiveness of the vehicle retrieval model training method in vehicle-dense scenes are verified, indicating that this algorithm can well solve the problem that the traditional vehicle retrieval model cannot accurately understand the main content of the image.

Claims

1. A method for training an image-text cross-modal vehicle retrieval model in a dense vehicle scene, characterized in that: include: S1: obtaining a training sample set, wherein the training sample set includes frame image data of densely packed vehicles in a plurality of traffic flow monitoring videos and their corresponding description texts, i.e., image samples and text samples, respectively performing feature extraction on the image samples and the text samples, and obtaining a visual-text joint embedding; S2: Calculate the TFIDF vector of the description text corresponding to the image sample to obtain the semantic features of the image sample; S3: Calculate the feature similarity between image samples and text samples based on cosine similarity to obtain global and local feature similarities. Integrate the global and local feature similarities through a learnable network to obtain the final inter-modal similarity. The feature similarity calculation method between image samples and text samples in S3 is described in detail as follows: S31: Calculate the feature similarity between images and text based on cosine similarity. The formula is as follows: in, and are the feature vectors of image samples and text samples respectively; The feature similarity between the image and text is divided into two parts, namely global feature similarity and local feature similarity. The specific strategy is as follows: S311: The global feature similarity is calculated from the global image feature and the global text feature, reflecting the feature similarity between the modalities at a macro level, and is recorded as ,in and Calculated from S11 and S12; S312: The local feature similarity is calculated from the local image feature and the local text feature, reflecting the feature similarity between the image region and the word, denoted as ,in , is calculated by S13, where The meaning is the local image feature vector of the nth image, The meaning of is the local text feature vector of the nth image; S32: Build a learnable network based on the fully connected layer, integrate the global and local feature similarities obtained in S31 into the final inter-modal similarity, and obtain the inter-modal feature similarity vector , the formula is as follows: in, is a learnable weight vector, is the sigmoid function, is the similarity obtained from S31, is the batch normalization function, which calculates the normalized weights , and then calculate again to get the feature similarity vector between modalities ; S4: Calculate the semantic similarity between image samples of traffic scenes; S5: weights updated according to a loss function, wherein the loss function includes a triplet loss and an image semantic loss; The loss function in S5 is described as follows: S51: Update the weights in the network according to the value of the loss function. The loss function includes two parts, namely, triplet loss and image semantic loss. The triplet loss is used to measure the difference between the similarity between modalities and the actual value. The formula is as follows: in is the boundary parameter, where is the positive sample set of images, is the positive sample set of description text, is the set of negative image samples, is a description text negative sample set, where It is the inter-modal similarity matrix obtained by calculating the sample image feature vectors and text feature vectors in the image set and text set using the calculation method in s3. , where s1, s2, , sn are all calculated by the inter-modal similarity equation in S32, s1 represents the inter-modal similarity feature vector between the first image in the image positive sample set and its corresponding description text positive sample; S52: The image semantic loss is used to measure the difference between the feature similarity matrix and the image semantic similarity matrix. The specific strategy is as follows: S521: Based on the sigmoid function, the similarity value matrix is ​​converted into a similarity ranking matrix, that is, the feature similarity matrix S and the semantic similarity matrix C are sorted to obtain the similarity ranking matrix. The sigmoid function adds a temperature parameter The ranking value is closer to the real ranking value, while ensuring the effectiveness of the gradient. The formula is as follows: in, is the similarity value matrix to be sorted, ,in Represents the similarity matrix i Row, No. j Column elements, is the batch size during training, is the sigmoid function with the temperature parameter added. The feature similarity matrix and semantic similarity matrix after sorting are and ; S522: The input data of image semantic loss is and , the formula is as follows: in, is the batch size during training, and is the sorting matrix obtained from S521 and The value of the corresponding position; S53: The loss function is ultimately composed of two loss functions, and the formula is as follows: in, and Used to balance the two losses, and .

2. The image-text cross-modal vehicle retrieval training method in a vehicle-intensive scene according to claim 1, characterized in that: In S1, the detailed description of feature extraction is as follows: S11: Obtain the embedded representation of the image, where an image feature encoder is built based on the deep convolutional neural network Faster RCNN to extract the original image features of the image sample to be processed; a network is built based on the self-attention mechanism to transform the original features of the image sample to obtain the global image features ; S12: Obtain the embedded representation of the text, where a text feature encoder is constructed based on the bidirectional gated recurrent unit structure Bi-GRU to extract the text features of the text to be processed and obtain the global text features ; S13: Obtain visual-text joint embedding, where a network is built based on the cross-attention mechanism to capture the fine-grained correspondence between image regions and text words, and the global image features are respectively and global text features Converted into local image features containing more image-text correspondence information and local text features , and obtain the joint visual-text embedding.

3. The image-text cross-modal vehicle retrieval training method in a vehicle-intensive scene according to claim 1, characterized in that: In S2, the steps for extracting image semantic features are as follows: S21: The input data of the image semantic feature calculation method is the description text corresponding to the image sample, wherein the TFIDF vector of the description text is calculated based on the word frequency-inverse document frequency, and the TFIDF vector is used to represent the semantic feature of the image sample; S22: The calculation process of the word frequency-inverse document rate method is as follows: in For words In the description text The number of occurrences in yes The total number of words contained in is the total number of words contained in the text set in the training set, The text set contains words The total number of entries of texts, where the text samples constitute the text set; S23: The semantic feature dimension of the image sample is equal to the total number of different words contained in the text set. Each word corresponds to an attribute of the image semantic feature vector. If there is a word in the text, in the image sample semantic feature vector, the attribute corresponding to the word is the TFIDF value of the word and the other positions in the image sample semantic feature vector are 0. In this way, the semantic features of the image can be obtained.

4. The image-text cross-modal vehicle retrieval training method in a vehicle-intensive scene according to claim 1, characterized in that: The semantic similarity between image samples of traffic scenes in S4 includes the following steps: S41: Use the CIDEr index to evaluate the semantic similarity between traffic scene images and obtain the semantic similarity. The semantic similarity matrix C is constructed, and the specific formula is as follows: in, is an image sample of a traffic scene and image samples The semantic similarity between is the number of entries of description text corresponding to the image sample, is an image No. Description text, is the TFIDF vector of the text obtained by S22 and S23, represents the first i Image samples.

Images