Image-text cross-modal vehicle retrieval model training method in vehicle dense scene

By using deep convolutional neural network and self-attention mechanism in traffic scenarios, visual-text joint embedding is extracted, and combined with TFIDF and cosine similarity calculations, image semantic loss function is introduced, which solves the inaccurate understanding of vehicle retrieval models in the prior art. The problem of the main content of the image improves the retrieval accuracy and efficiency in vehicle-intensive scenarios.

CN118968516AActive Publication Date: 2024-11-15UNIV OF ELECTRONICS SCI & TECH OF CHINA

Patent Information

Application Number
CN202410942443.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-15
Estimated Expiration
2044-07-15

AI Technical Summary

Technical Problem

The existing traffic scene image-text cross-modal vehicle retrieval model cannot accurately understand the main content of the image, resulting in low retrieval accuracy and triple loss cannot accurately measure the semantic similarity between the image and the text.

Method used

Using the image-text cross-modal vehicle retrieval model training method, feature extraction is performed by obtaining the training sample set, the similarity between image semantic features and modalities is calculated, and vision-text joint embedding is extracted using deep convolutional neural network and self-attention mechanism. , combine TFIDF and cosine similarity to calculate feature similarity, and introduce image semantic loss function to update network weights.

Benefits of technology

It significantly improves the accuracy and efficiency of image-text cross-modal vehicle retrieval in vehicle-intensive scenarios, enables the model to better pay attention to the main semantic content of the image, and improves the application capabilities of smart transportation systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968516A_ABST
    Figure CN118968516A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of computers, and provides an image-text cross-modal vehicle retrieval model training method in a vehicle dense scene. The method solves the problem that the vehicle retrieval model in the prior art cannot accurately understand the main content of the image, and improves the retrieval accuracy. The scheme comprises the following steps: acquiring a training sample set, and performing feature extraction on an image sample and a text sample to obtain vision-text joint embedding; calculating semantic features of the image samples; calculating feature similarity between the image sample and the text sample based on cosine similarity to obtain global and local feature similarity, and integrating the global and local feature similarity through a learnable network to obtain final inter-modal similarity; calculating the semantic similarity between the image samples of the traffic scene; and updating the weight according to a loss function, wherein the loss function comprises triple loss and image semantic loss. The method is used for image-text cross-modal vehicle retrieval in a vehicle dense scene.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology and provides an image-text cross-modal vehicle retrieval model training method in a vehicle-intensive scene. Background Art

[0002] With the rapid development of digitalization and informatization, graphic and text data are constantly generated on social media, monitoring points, e-commerce and other platforms, and the development of intelligent transportation systems is gradually tending to improve multimodal information processing and comparison capabilities. As a basic technology in cross-modal retrieval, graphic and text matching aims to accurately match related text and images, and plays a vital role in traffic scenarios, such as vehicle retrieval and target vehicle re-identification. At present, the graphic and text matching model for traffic scenes cannot accurately focus on the main semantics in the image, and thus cannot extract image features that are consistent with the main semantics of the picture.

[0003] In summary, the existing traffic scene image-text cross-modal vehicle retrieval has the following shortcomings:

[0004] 1) The existing vehicle retrieval model only updates the network parameters of the retrieval model based on the difference between the inter-modal similarity and the actual value, without considering whether these image features reflect the main semantics of the picture;

[0005] 2) The triplet loss commonly used in cross-modal image-text matching tasks cannot accurately measure whether the similarity between the main semantics of the image and the text semantics is consistent with the actual value.

[0006] The shortcomings of the above-mentioned prior art will affect the application of the image-text matching model in traffic scenes with dense vehicles, making it impossible to efficiently retrieve vehicles that meet the text description, which brings great inconvenience to the intelligent transportation system. Therefore, the present invention proposes an image-text cross-modal vehicle retrieval training method in vehicle-dense scenes. Summary of the invention

[0007] The present invention aims to solve the problem that the existing traffic scene image-text cross-modal vehicle retrieval model cannot accurately understand the main content of the image in vehicle-dense scenes, resulting in low retrieval accuracy.

[0008] To achieve the above purpose, the present invention adopts the following technical solutions:

[0009] The present invention provides an image-text cross-modal vehicle retrieval model training method in a vehicle-intensive scene, comprising:

[0010] S1: obtaining a training sample set, wherein the training sample set includes frame image data of densely packed vehicles in a plurality of traffic flow monitoring videos and their corresponding description texts, i.e., image samples and text samples, respectively performing feature extraction on the image samples and the text samples, and obtaining a visual-text joint embedding;

[0011] S2: Calculate the TFIDF vector of the description text corresponding to the image sample to obtain the semantic features of the image sample;

[0012] S3: Calculate the feature similarity between image samples and text samples based on cosine similarity to obtain global and local feature similarities. Integrate the global and local feature similarities through a learnable network to obtain the final inter-modal similarity.

[0013] S4: Calculate the semantic similarity between image samples of traffic scenes;

[0014] S5: weights updated according to a loss function, wherein the loss function includes triplet loss and image semantic loss.

[0015] In the above method, in S1, the detailed description of feature extraction is as follows:

[0016] S11: Obtain the embedded representation of the image, where an image feature encoder is built based on the deep convolutional neural network Faster RCNN to extract the original image features of the image sample to be processed; a network is built based on the self-attention mechanism to transform the original features of the image sample to obtain the global image features ;

[0017] S12: Obtain the embedded representation of the text, where a text feature encoder is constructed based on the bidirectional gated recurrent unit structure Bi-GRU to extract the text features of the text to be processed and obtain the global text features ;

[0018] S13: Obtain visual-text joint embedding, where a network is built based on the cross-attention mechanism to capture the fine-grained correspondence between image regions and text words, and the global image features are respectively and global text features Convert it into local image features v and local text features t that contain more image-text correspondence information, and obtain the visual-text joint embedding.

[0019] In the above method, in S2, the steps of extracting image semantic features are as follows:

[0020] S21: The input data of the image semantic feature calculation method is the description text corresponding to the image sample, wherein the TFIDF vector of the description text is calculated based on the term frequency-inverse document frequency, and the TFIDF vector is used to represent the semantic feature of the image sample;

[0021] S22: The calculation process of the word frequency-inverse document rate method is as follows:

[0022] TFIDF x =TF×IDF

[0023]

[0024] Where n x is the word x in the description text T j The number of occurrences in ,∑ y n y It is T j is the total number of words contained in the text set in the training set, |T| is the total number of words contained in the text set in the training set, |{k:x∈T k}| is the total number of entries in the text set containing word x, where the text samples constitute the text set;

[0025] S23: The semantic feature dimension of the image sample is equal to the total number of different words contained in the text set. Each word corresponds to an attribute of the image semantic feature vector. If there is a word in the text, in the image sample semantic feature vector, the attribute corresponding to the word is the TFIDF value of the word and the other positions in the image sample semantic feature vector are 0. In this way, the semantic features of the image can be obtained.

[0026] The method for calculating the feature similarity between the image sample and the text sample in S3 in the above method is described in detail

[0027] as follows:

[0028] S31: Calculate the feature similarity between images and text based on cosine similarity. The formula is as follows:

[0029]

[0030] Among them, x and y are the feature vectors of image samples and text samples respectively;

[0031] The feature similarity between the image and text is divided into two parts, namely global feature similarity and local feature similarity. The specific strategy is as follows:

[0032] S311: The global feature similarity is calculated from the global image feature and the global text feature, reflecting the feature similarity between the modalities at a macro level, and is recorded as in and Calculated from S11 and S12;

[0033] S312: The local feature similarity is calculated from the local image feature and the local text feature, reflecting the feature similarity between the image region and the word, denoted as Where v = {v 1 , v 2 , …, v n}, t={t 1 , t 2 ,…,t n} is calculated by S13, where v n The meaning is the local image feature vector of the nth image, t n The meaning of is the local text feature vector of the nth image;

[0034] S32: Build a learnable network based on the fully connected layer, integrate the global and local feature similarities obtained in S31 into the final inter-modal similarity, and obtain the inter-modal feature similarity vector s, the formula is as follows:

[0035]

[0036] Where W is a learnable weight vector, δ(·) is the sigmoid function, is the similarity obtained by S31, BN(·) is the batch normalization function, and the normalized weight m is obtained by calculation e , and then calculate again to get the inter-modal feature similarity vector s.

[0037] In the above method, the semantic similarity between image samples of traffic scenes in S4 includes the following steps:

[0038] S41: Use the CIDEr index to evaluate the semantic similarity between traffic scene images and obtain semantic similarity

[0039] degree, determined by the semantic similarity c ij The semantic similarity matrix C is constructed, and the specific formula is as follows:

[0040]

[0041] Among them, c ij is the semantic similarity between image sample i and image sample j of the traffic scene, Q is the number of entries of the description text corresponding to the image sample, T ip is image I i The pth description text, g(T ip ) is the TFIDF vector of the text obtained by S22 and S23, I i represents the,th image sample in the image sample set.

[0042] The loss function in S5 in the above method is specifically described as follows:

[0043] S51: Update the weights in the network according to the value of the loss function. The loss function includes two parts, namely, triplet loss and image semantic loss. The triplet loss is used to measure the difference between the similarity between modalities and the actual value. The formula is as follows:

[0044] L 1 =∑ (I,T) [λ-s(I,T)+s(I,T - )] + +[λ-s(I,T)+D(I - , T)] +

[0045] Where λ is the boundary parameter, where is the image positive sample set, T is the description text positive sample set, I - is the image negative sample set, T - is a negative sample set of description texts, where S(·) is an inter-modal similarity matrix obtained by applying the calculation method in s3 to the sample image feature vectors and text feature vectors in the image set and the text set, S(I,T)={s1,s2,…,sn}, where s1,s2,…,sn are all calculated by the inter-modal similarity equation in S32, and s1 represents the inter-modal similarity feature vector between the first image in the image positive sample set and its corresponding positive description text sample;

[0046] S52: The image semantic loss is used to measure the difference between the feature similarity matrix and the image semantic similarity matrix. The specific strategy is as follows:

[0047] S521: Based on the sigmoid function, the similarity value matrix is ​​converted into a similarity ranking matrix, that is, the feature similarity matrix S and the semantic similarity matrix C are sorted to obtain a similarity ranking matrix. The sigmoid function adds a temperature parameter τ to make the ranking value closer to the actual ranking value, while ensuring the effectiveness of the gradient. The formula is as follows:

[0048]

[0049] Where M is the similarity matrix to be sorted, M = (m ij ) n×n , where m ij represents the i-th row and j-th column element of the similarity value matrix, n is the batch size in the training process, Ω(x) is the sigmoid function with a temperature parameter added, and the feature similarity matrix and semantic similarity matrix after sorting are S R and C R ;

[0050] S522: The input data of image semantic loss is S R and C R , the formula is as follows:

[0051]

[0052] Where n is the batch size during training, sr ij and cr ij is the sorting matrix S obtained from S521 R and C R The value of the corresponding position;

[0053] S53: The loss function is ultimately composed of two loss functions, and the formula is as follows:

[0054]

[0055] in, and Used to balance the two losses, and

[0056] Because the present invention adopts the above technical means, it has the following beneficial effects:

[0057] 1. Visual-text joint embedding: Through the deep convolutional neural network Faster RCNN and the self-attention mechanism, combined with the bidirectional gated recurrent unit Bi-GRU, the features of the image and text are extracted to obtain the visual-text joint embedding. This method can more effectively capture the fine-grained correspondence between image regions and text words, and improve the accuracy of retrieval.

[0058] 2. Image semantic feature calculation: The semantic features of the text are calculated using the term frequency-inverse document frequency (TFIDF) method to represent the semantic features of the image. This method takes into account the problem that the model is difficult to extract effective image features in vehicle-intensive scenes, making the model better adapted to the intelligent transportation system.

[0059] 3. Feature similarity calculation: Calculate the feature similarity between the image and the text based on cosine similarity, including global feature similarity and local feature similarity. The similarity is integrated through the fully connected layer and sigmoid function to obtain the final inter-modal similarity.

[0060] 4. Semantic similarity calculation: Use the CIDEr indicator to evaluate the semantic similarity between traffic scene images, thereby establishing semantic connections between images.

[0061] 5. Loss function design: The loss function includes triplet loss and image semantic loss, which are used to measure the difference between the inter-modal similarity and the actual value, as well as the difference between the feature similarity matrix and the image semantic similarity matrix. Through this design, the model can pay more attention to the main semantic content of the image.

[0062] 6. Performance improvement: By introducing the image semantic loss function, the present invention solves the problem in the prior art that the vehicle retrieval model cannot accurately understand the main content of the image, and improves the retrieval accuracy of the model in vehicle-dense scenes.

[0063] In summary, this invention significantly improves the accuracy and efficiency of image-text cross-modal vehicle retrieval in vehicle-dense scenes through innovative visual-text joint embedding methods, image semantic feature calculation, feature and semantic similarity calculation, and designed loss function, which is of great significance to the development of intelligent transportation systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A block diagram of an image-text cross-modal vehicle retrieval training method in a vehicle-dense scene provided for the implementation of the present invention;

[0065] Figure 2 A schematic diagram of the structure of an image-text cross-modal vehicle retrieval training method in a vehicle-dense scene provided for the implementation of the present invention;

[0066] Figure 3 Schematic diagram of traffic scene image and description text of target vehicle.

[0067] Figure 4 A table of key fields describing the text;

[0068] Figure 5 Pseudocode for testing the vehicle retrieval model training method. DETAILED DESCRIPTION

[0069] The following is a detailed description of the embodiments of the present invention. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, modifications or equivalent substitutions made to the present invention should all be included in the scope of the claims of the present invention.

[0070] In addition, in order to better illustrate the present invention, numerous specific details are given in the following specific embodiments. It will be understood by those skilled in the art that the present invention can also be implemented without these specific details.

[0071] The present invention provides an image-text cross-modal vehicle retrieval model training method in a vehicle-intensive scene, comprising:

[0072] S1: obtaining a training sample set, wherein the training sample set includes frame image data of densely packed vehicles in a plurality of traffic flow monitoring videos and their corresponding description texts, i.e., image samples and text samples, respectively performing feature extraction on the image samples and the text samples, and obtaining a visual-text joint embedding;

[0073] S2: Calculate the TFIDF vector of the description text corresponding to the image sample to obtain the semantic features of the image sample;

[0074] S3: Calculate the feature similarity between image samples and text samples based on cosine similarity to obtain global and local feature similarities. Integrate the global and local feature similarities through a learnable network to obtain the final inter-modal similarity.

[0075] S4: Calculate the semantic similarity between image samples of traffic scenes;

[0076] S5: weights updated according to a loss function, wherein the loss function includes triplet loss and image semantic loss.

[0077] In the above method, in S1, the detailed description of feature extraction is as follows:

[0078] S11: Obtain the embedded representation of the image, where an image feature encoder is built based on the deep convolutional neural network Faster RCNN to extract the original image features of the image sample to be processed; a network is built based on the self-attention mechanism to transform the original features of the image sample to obtain the global image features

[0079] S12: Obtain the embedded representation of the text, where a text feature encoder is constructed based on the bidirectional gated recurrent unit structure Bi-GRU to extract the text features of the text to be processed and obtain the global text features

[0080] S13: Obtain visual-text joint embedding, where a network is built based on the cross-attention mechanism to capture the fine-grained correspondence between image regions and text words, and the global image features are respectively and global text features Convert it into local image features v and local text features t that contain more image-text correspondence information, and obtain the visual-text joint embedding.

[0081] In the above method, in S2, the steps of extracting image semantic features are as follows:

[0082] S21: The input data of the image semantic feature calculation method is the description text corresponding to the image sample, wherein the TFIDF vector of the description text is calculated based on the term frequency-inverse document frequency, and the TFIDF vector is used to represent the semantic feature of the image sample;

[0083] S22: The calculation process of the word frequency-inverse document rate method is as follows:

[0084] TFIDF x =TF×IDF

[0085]

[0086] Where n x is the word x in the description text T j The number of occurrences in ,∑ y n y It is T j is the total number of words contained in the text set in the training set, |T| is the total number of words contained in the text set in the training set, |{k:x∈T k}| is the total number of entries in the text set containing word x, where the text samples constitute the text set;

[0087] S23: The semantic feature dimension of the image sample is equal to the total number of different words contained in the text set. Each word corresponds to an attribute of the image semantic feature vector. If there is a word in the text, in the image sample semantic feature vector, the attribute corresponding to the word is the TFIDF value of the word and the other positions in the image sample semantic feature vector are 0. In this way, the semantic features of the image can be obtained.

[0088] The method for calculating the feature similarity between the image sample and the text sample in S3 in the above method is described in detail

[0089] as follows:

[0090] S31: Calculate the feature similarity between images and text based on cosine similarity. The formula is as follows:

[0091]

[0092] Among them, x and y are the feature vectors of image samples and text samples respectively;

[0093] The feature similarity between the image and text is divided into two parts, namely global feature similarity and local feature similarity. The specific strategy is as follows:

[0094] S311: The global feature similarity is calculated from the global image feature and the global text feature, reflecting the feature similarity between the modalities at a macro level, and is recorded as in and Calculated from S11 and S12;

[0095] S312: The local feature similarity is calculated from the local image feature and the local text feature, reflecting the feature similarity between the image region and the word, denoted as Where v = {v 1 , v 2 , …, v n}, t={t 1 , t 2 ,…,t n} is calculated by S13, where v n The meaning is the local image feature vector of the nth image, t n The meaning of is the local text feature vector of the nth image;

[0096] S32: Build a learnable network based on the fully connected layer, integrate the global and local feature similarities obtained in S31 into the final inter-modal similarity, and obtain the inter-modal feature similarity vector s, the formula is as follows:

[0097]

[0098] Where W is a learnable weight vector, δ(·) is the sigmoid function, is the similarity obtained by S31, BN(·) is the batch normalization function, and the normalized weight m is obtained by calculation e , and then calculate again to get the inter-modal feature similarity vector s.

[0099] In the above method, the semantic similarity between image samples of traffic scenes in S4 includes the following steps:

[0100] S41: Use the cIDEr indicator to evaluate the semantic similarity between traffic scene images and obtain semantic similarity

[0101] degree, determined by the semantic similarity c ij The semantic similarity matrix C is constructed, and the specific formula is as follows:

[0102]

[0103] Among them, c ij is the semantic similarity between image sample i and image sample j of the traffic scene, Q is the number of entries of the description text corresponding to the image sample, T ip is image I i The pth description text, g(T ip ) is the TFIDF vector of the text obtained by S22 and S23, I i represents the i-th image sample in the image sample set.

[0104] The loss function in S5 in the above method is specifically described as follows:

[0105] S51: Update the weights in the network according to the value of the loss function. The loss function includes two parts, namely, triplet loss and image semantic loss. The triplet loss is used to measure the difference between the similarity between modalities and the actual value. The formula is as follows:

[0106] L 1 =∑ (I,T) [λ-s(I,T)+s(I,T - )] + +[λ-s(I,T)+S(I - , T)] +

[0107] Where λ is the boundary parameter, where is the image positive sample set, T is the description text positive sample set, I - is the image negative sample set, T - is a negative sample set of description texts, where S(·) is an inter-modal similarity matrix obtained by applying the calculation method in s3 to the sample image feature vectors and text feature vectors in the image set and the text set, S(I,T)={s1,s2,…,sn}, where s1,s2,…,sn are all calculated by the inter-modal similarity equation in S32, and s1 represents the inter-modal similarity feature vector between the first image in the image positive sample set and its corresponding positive description text sample;

[0108] S52: The image semantic loss is used to measure the difference between the feature similarity matrix and the image semantic similarity matrix. The specific strategy is as follows:

[0109] S521: Based on the sigmoid function, the similarity value matrix is ​​converted into a similarity ranking matrix, that is, the feature similarity matrix S and the semantic similarity matrix C are sorted to obtain a similarity ranking matrix. The sigmoid function adds a temperature parameter τ to make the ranking value closer to the actual ranking value, while ensuring the effectiveness of the gradient. The formula is as follows:

[0110]

[0111] Where M is the similarity matrix to be sorted, M = (m ij ) n×n , where m ij represents the i-th row and j-th column element of the similarity value matrix, n is the batch size in the training process, Ω(x) is the sigmoid function with a temperature parameter added, and the feature similarity matrix and semantic similarity matrix after sorting are S R and C R ;

[0112] S522: The input data of image semantic loss is S R and C R , the formula is as follows:

[0113]

[0114] Where n is the batch size during training, sri j andcri j The sorting matrices SR and C are obtained from S521 R The value of the corresponding position;

[0115] S53: The loss function is ultimately composed of two loss functions, and the formula is as follows:

[0116]

[0117] in, and Used to balance the two losses, and

[0118] Example

[0119] 1. Experimental dataset processing instructions

[0120] The public dataset FindVehicle is used for training and testing. 177,000 images are used for training, 5,000 images are used for verification, and 5,000 images are used for testing. The data format example is as follows Figure 3 As shown, the key fields of the description text are as follows Figure 4 shown.

[0121] 2. Graphics encoder description

[0122] Image features are extracted based on the pre-trained Faster RCNN network and converted into visual embedding according to the steps described, and the dimension of the obtained visual modality embedding is 2048. The text is digitized based on the Bi-GRU model, the text is divided into a group of words of length L, and the words are digitized to map the words into vectors of dimension 2048.

[0123] 3. Test description

[0124] The present invention discloses an image-text cross-modal vehicle retrieval training method in vehicle-dense scenes. The core of the present invention is to introduce an image semantic loss function into the training process of the retrieval model, supplement the limitations of the original triplet loss, align the feature similarity between modalities with the semantic similarity within the modality, so that the vehicle retrieval model can accurately focus on the main semantic content in the image, and extract it as the main feature embedding of the image to match the text. This method improves the retrieval accuracy of the retrieval model in vehicle-dense scenes, making it easier for the model to focus on vehicles in the image that match the text. This verification test is mainly used to simulate the application scenario of retrieving related vehicles based on descriptive text in vehicle-dense scenes, so as to verify whether the present invention can achieve better performance than traditional training methods, that is, higher accuracy.

[0125] 1. Functional testing and result analysis

[0126] This verification test is mainly used to simulate and test whether the present invention can accurately retrieve relevant vehicles based on the description text in a vehicle-intensive scene. The pseudo code for the vehicle retrieval model training method is as follows: Figure 5 shown.

[0127] Through the above verification test, it can be verified that the present invention can improve the retrieval ability of the cross-modal retrieval model. Specifically, according to the steps of the above algorithm, a network structure based on the SGRAF model is first initialized, and the number of training rounds is set to 25 rounds. During the training process, the retrieval model is first used to calculate the feature embedding of the training set image and text, and the semantic embedding (semantic feature) of the image is calculated according to the TFIDF algorithm. Then the feature similarity between the modalities is calculated based on the feature embedding of the image and text, and the semantic similarity within the modality is calculated based on the semantic embedding of the image. Then the traditional loss triplet loss function of the cross-modal task is calculated, and the image semantic loss proposed by the present invention is calculated. Update the network parameters according to the two loss values, and update the model. After 25 rounds of training, the model effect is tested on the test set, and the trained retrieval model is used to calculate the feature similarity between the test set image and text, and compared with the true similarity, and the recall rate of the model is statistically calculated.

[0128] 2. Performance test and result analysis

[0129] The recall rate Recall@K reflects the accuracy of the retrieval model. The matching results of the model are sorted from large to small according to the similarity score. Recall@K reflects the ratio of successfully matching the correct samples in the first K matching results. Therefore, Recall@1 represents the ability of the model to accurately retrieve the correct samples. The calculation formula is shown below.

[0130]

[0131] Among them, |·| represents the number of samples in the set, D test The retrieval model is used for vehicle retrieval after 25 rounds of training. Five different test data sets are used, each containing 1000 sample data. The relevant images are retrieved based on the text, and the recall rates Recall@1, Recall@5 and Recall@10 of the model are calculated. The simulation experimental results are shown in Table 2.

[0132] Table 2: Model test recall results

[0133] Recall@1 Recall@5 Recall@10 Test Dataset 1 0.649 0.905 0.952 Test Dataset 2 0.618 0.887 0.941 Test Dataset 3 0.639 0.901 0.950 Test Dataset 4 0.616 0.885 0.940 Test Dataset 5 0.635 0.902 0.950

[0134] 3. Test Summary

[0135] This section conducts functional and performance tests on the present invention. Through a series of experimental comparisons, the rationality and effectiveness of the vehicle retrieval model training method in vehicle-dense scenes are verified, indicating that this algorithm can well solve the problem that the traditional vehicle retrieval model cannot accurately understand the main content of the image.

Claims

1. A method for training an image-text cross-modal vehicle retrieval model in a dense vehicle scene, characterized in that: include: S1: obtaining a training sample set, wherein the training sample set includes frame image data of densely packed vehicles in a plurality of traffic flow monitoring videos and their corresponding description texts, i.e., image samples and text samples, respectively performing feature extraction on the image samples and the text samples, and obtaining a visual-text joint embedding; S2: Calculate the TFIDF vector of the description text corresponding to the image sample to obtain the semantic features of the image sample; S3: Calculate the feature similarity between image samples and text samples based on cosine similarity to obtain global and local feature similarities. Integrate the global and local feature similarities through a learnable network to obtain the final inter-modal similarity. S4: Calculate the semantic similarity between image samples of traffic scenes; S5: weights updated according to a loss function, wherein the loss function includes triplet loss and image semantic loss.

2. The image-text cross-modal vehicle retrieval training method in a vehicle-intensive scene according to claim 1, characterized in that: In S1, the detailed description of feature extraction is as follows: S11: Obtain the embedded representation of the image, where an image feature encoder is built based on the deep convolutional neural network Faster RCNN to extract the original image features of the image sample to be processed; a network is built based on the self-attention mechanism to transform the original features of the image sample to obtain the global image features S12: Obtain the embedded representation of the text, where a text feature encoder is constructed based on the bidirectional gated recurrent unit structure Bi-GRU to extract the text features of the text to be processed and obtain the global text features S13: Obtain visual-text joint embedding, where a network is built based on the cross-attention mechanism to capture the fine-grained correspondence between image regions and text words, and the global image features are respectively and global text features Convert it into local image features v and local text features t that contain more image-text correspondence information, and obtain the visual-text joint embedding.

3. The image-text cross-modal vehicle retrieval training method in a vehicle-intensive scene according to claim 1, characterized in that: In S2, the steps for extracting image semantic features are as follows: S21: The input data of the image semantic feature calculation method is the description text corresponding to the image sample, wherein the TFIDF vector of the description text is calculated based on the term frequency-inverse document frequency, and the TFIDF vector is used to represent the semantic feature of the image sample; S22: The calculation process of the word frequency-inverse document rate method is as follows: TFIDF x =TF×IDF where n x is the word x in the description text T j The number of occurrences in ,∑ y n y It is T j is the total number of words contained in the text set in the training set, |T| is the total number of words contained in the text set in the training set, |{k:x∈T k }| is the total number of entries in the text set containing word x, where the text samples constitute the text set; S23: The semantic feature dimension of the image sample is equal to the total number of different words contained in the text set. Each word corresponds to an attribute of the image semantic feature vector. If there is a word in the text, in the image sample semantic feature vector, the attribute corresponding to the word is the TFIDF value of the word and the other positions in the image sample semantic feature vector are 0. In this way, the semantic features of the image can be obtained.

4. The image-text cross-modal vehicle retrieval training method in a vehicle-intensive scene according to claim 1, characterized in that: The feature similarity calculation method between image samples and text samples in S3 is described in detail as follows: S31: Calculate the feature similarity between images and text based on cosine similarity. The formula is as follows: Among them, x and y are the feature vectors of image samples and text samples respectively; The feature similarity between the image and text is divided into two parts, namely global feature similarity and local feature similarity. The specific strategy is as follows: S311: The global feature similarity is calculated from the global image feature and the global text feature, reflecting the feature similarity between the modalities at a macro level, and is recorded as in and Calculated from S11 and S12; S312: The local feature similarity is calculated from the local image feature and the local text feature, reflecting the feature similarity between the image region and the word, denoted as where v = {v1, v2, ..., v n }, t={t1, t2, …, t n } is calculated by S13, where v n The meaning is the local image feature vector of the nth image, t n The meaning of is the local text feature vector of the nth image; S32: Build a learnable network based on the fully connected layer, integrate the global and local feature similarities obtained in S31 into the final inter-modal similarity, and obtain the inter-modal feature similarity vector s, the formula is as follows: Where W is a learnable weight vector, δ(·) is the sigmoid function, is the similarity obtained by S31, BN(·) is the batch normalization function, and the normalized weight m is obtained by calculation e , and then calculate again to get the inter-modal feature similarity vector s.

5. The image-text cross-modal vehicle retrieval training method in a vehicle-intensive scene according to claim 1, characterized in that: The semantic similarity between image samples of traffic scenes in S4 includes the following steps: S41: Use the CIDEr index to evaluate the semantic similarity between traffic scene images and obtain the semantic similarity. ij The semantic similarity matrix C is constructed, and the specific formula is as follows: Among them, c ij is the semantic similarity between image sample i and image sample j of the traffic scene, Q is the number of entries of the description text corresponding to the image sample, T ip is image I i The pth description text, g(T ip ) is the TFIDF vector of the text obtained by S22 and S23, I i represents the i-th image sample in the image sample set.

6. The image-text cross-modal vehicle retrieval training method in a vehicle-intensive scene according to claim 1, characterized in that: The loss function in S5 is described as follows: S51: Update the weights in the network according to the value of the loss function. The loss function includes two parts, namely, triplet loss and image semantic loss. The triplet loss is used to measure the difference between the similarity between modalities and the actual value. The formula is as follows: L1=∑ (I,T) [λ-s(I,T)+S(I,T - )] + +[λ-S(I,T)+S(I - ,T)] + Where λ is the boundary parameter, where is the image positive sample set, T is the description text positive sample set, I - is the image negative sample set, T - is the description text negative sample set, where S(·) is the inter-modal similarity matrix obtained by applying the calculation method in s3 to the sample image feature vectors and text feature vectors in the image set and the text set, S(I,T)={s1,s2,…,sn}, where s1,s2,…,sn are all calculated by the inter-modal similarity equation in S32, and s1 represents the inter-modal similarity feature vector between the first image in the image positive sample set and its corresponding description text positive sample; S52: The image semantic loss is used to measure the difference between the feature similarity matrix and the image semantic similarity matrix. The specific strategy is as follows: S521: Based on the sigmoid function, the similarity value matrix is ​​converted into a similarity ranking matrix, that is, the feature similarity matrix S and the semantic similarity matrix C are sorted to obtain a similarity ranking matrix. The sigmoid function adds a temperature parameter τ to make the ranking value closer to the actual ranking value, while ensuring the effectiveness of the gradient. The formula is as follows: Where M is the similarity matrix to be sorted, M = (m ij ) n×n , where m ij represents the i-th row and j-th column element of the similarity value matrix, n is the batch size in the training process, Ω(x) is the sigmoid function with a temperature parameter added, and the feature similarity matrix and semantic similarity matrix after sorting are S R and C R ; S522: The input data of image semantic loss is S R and C R , the formula is as follows: Where n is the batch size during training, sr ij and cr ij is the sorting matrix S obtained from S521 R and C R The value of the corresponding position; S53: The loss function is ultimately composed of two loss functions, and the formula is as follows: in, and Used to balance the two losses, and

Citation Information

Patent Citations

  • Method and apparatus for detecting barrier based on monocular vision

    CN101408978A

  • System and method for image search through images of multi-task portal vehicles

    CN108171136A

  • A neural network-based cross-modal information retrieval method and device

    CN109840287A

  • Text-based vehicle image fine-grained retrieval system

    CN110889003A

  • Dam defect image text cross-modal retrieval method and model

    CN113220919A

Cited By

  • Multi-modal large model training data acquisition method and system

    CN119380144A

  • Video description generation method and device, electronic equipment and storage medium

    CN119562141A

  • Global and component fine-grained recipe retrieval method using segmentation and negative matching

    CN119917690A

  • Entity reasoning and distillation cross-modal retrieval method based on scene text heterogeneous prompt

    CN120144804A

  • Zero-training vehicle re-identification method based on visual language large model

    CN120220085A