Vehicle image-text retrieval method and system based on multistage semantic graph alignment and attribute enhancement

Abstract

The invention belongs to the technical field of intelligent traffic, and relates to a vehicle image-text retrieval method and system based on multistage semantic graph alignment and attribute enhancement. According to the method, fusion visual embedding is obtained according to the vehicle video, and fusion text embedding is obtained according to the text data; obtaining an updated visual attribute embedding vector and an updated text attribute embedding vector based on fusion visual embedding and fusion text embedding; performing semantic matching on the updated visual attribute embedding vector and the updated text attribute embedding vector, and aligning vehicle cross-modal semantic attributes from three different levels by utilizing multi-granularity semantics to obtain final visual attribute embedding and final text attribute embedding; and according to the final visual attribute embedding and the final text attribute embedding, obtaining the similarity between the vehicle video and the artificial text description, wherein the vehicle video with the highest similarity is a vehicle video retrieval result. According to the method, the target vehicle can be quickly and accurately positioned, and higher robustness and higher retrieval success rate are shown.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation technology and relates to a vehicle image and text retrieval method and system based on multi-level semantic graph alignment and attribute enhancement. Background Technology

[0002] Vehicle retrieval is a hot research area in smart city research, referring to the process of finding the trajectory or footage of a target vehicle from calibrated urban cameras. As a technology, vehicle retrieval is widely used in smart cities and intelligent transportation systems, for example, in tasks such as finding illegal or hit-and-run vehicles in urban surveillance, tracking specific vehicles in public places, and analyzing traffic video data.

[0003] Vehicle image-text retrieval methods based on multi-level semantic graph alignment and attribute enhancement play a crucial role in vehicle retrieval in smart cities. The task of vehicle image-text retrieval based on multi-level semantic graph alignment and attribute enhancement involves retrieving corresponding vehicle trajectory segments from a massive database of vehicle trajectory video clips, given natural language. However, existing technologies for vehicle retrieval based on natural language lack efficient feature fusion mechanisms between local and global features within a modality. This leads to vehicles of different identities exhibiting small inter-class differences. Furthermore, existing models often neglect the complementary fusion of spatial attribute information from visual features and semantic information from textual features, resulting in semantic information drift and a lack of robustness in learning vehicle attributes. Summary of the Invention

[0004] The purpose of this invention is to provide a vehicle image and text retrieval method and system based on multi-level semantic graph alignment and attribute enhancement, so as to solve the technical problem that vehicle retrieval methods lack an efficient feature fusion mechanism between local and global features within a modality.

[0005] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, the present invention provides a vehicle image-text retrieval method based on multi-level semantic graph alignment and attribute enhancement, comprising the following steps: Obtain vehicle videos and human-written text descriptions of the vehicle videos, and preprocess the human-written text descriptions to obtain text data; Text attribute parsing is performed on text data to obtain vehicle attribute information; The fused visual embedding is obtained from the vehicle video, and the fused text embedding is obtained from the text data. The updated visual attribute embedding vector and the updated text attribute embedding vector are obtained based on the fusion of visual embedding and fusion of text embedding; Semantic matching is performed on the updated visual attribute embedding vector and the updated text attribute embedding vector. Multi-granular semantics are used to align the vehicle's cross-modal semantic attributes from three different levels to obtain the final visual attribute embedding and the final text attribute embedding. The similarity between vehicle videos and human-written text descriptions is obtained based on the final visual attribute embedding and the final text attribute embedding. The vehicle video with the highest similarity is the vehicle video retrieval result.

[0006] Furthermore, the preprocessing of the artificial text description to obtain text data is as follows: Each piece of human-written text was broken down into different words using the Stanza tool; The Levenshtein distance algorithm is used to identify and correct incorrect words. Each word is annotated with its part of speech using part-of-speech tagging.

[0007] Furthermore, the process of parsing the text data to obtain vehicle attribute information is as follows: Perform statistical analysis on the text to extract local and global vehicle attributes. Apply component syntactic analysis tools to analyze the local and global vehicle attributes and generate attribute labels based on the analysis results. The attribute tags include: color ( ),size( ),type( ), vehicle motion ( Traffic scenarios ), surrounding vehicles ( ) and additional information ( ); Post-processing of attribute tags involves checking the attribute tags of each text data. For attribute tags that are ambiguous or have the same meaning, the attribute tag that appears most frequently is selected as the final vehicle attribute tag.

[0008] Furthermore, the process of obtaining fused visual embeddings based on vehicle video and fused text embeddings based on text data is as follows: Vehicle videos are uniformly sampled and cropped frame by frame to obtain local vehicle images. Based on these local images, a convolutional neural network (CNN) and a multilayer perceptron are used to perform local visual embedding to obtain local visual embedding results. A Video Swin Transformer model is then used to perform global visual embedding to obtain global visual embedding results. An F-Encoder structure is applied to fuse the local and global visual embedding results. The initial fused visual embedding is obtained through a self-attention layer and then through a cross-attention layer and learnable parameters to obtain the final fused visual embedding. Local text embeddings are extracted from text data using a word embedding model and a multilayer perceptron; the text data is input into a large language model to obtain global text embeddings; the local text embeddings and global text embeddings are fused using an F-Encoder structure, and then passed through a self-attention layer and a cross-attention layer in sequence with learnable parameters to obtain the fused text embedding.

[0009] Furthermore, the method for obtaining the updated visual attribute embedding vector is as follows: The fused visual embedding is taken as input and a visual attribute feature map is obtained through a fully connected layer and three one-dimensional convolutional layers. Based on the fused text embedding, a visual attribute attention map is learned through a fully connected layer and two one-dimensional convolutional layers. Then, the visual attribute feature map is compressed into the visual attribute attention map through channel convolution operation to obtain the visual attribute embedding. Finally, the visual attribute embedding is fused with the fused visual embedding to obtain the updated visual attribute embedding vector.

[0010] Furthermore, the method for obtaining the updated text attribute embedding vector is as follows: The fused text embedding is taken as input, and a text attribute feature map is obtained through a fully connected layer and three one-dimensional convolutional layers. Based on the fused visual embedding, a text attribute attention map is learned through a fully connected layer and two one-dimensional convolutional layers. Then, the text attribute feature map is compressed into the text attribute attention map through channel convolution operation to obtain the text attribute embedding. Finally, the text attribute embedding is fused with the fused text embedding to obtain the updated text attribute embedding vector.

[0011] Furthermore, the updated visual attribute embedding vector and the updated text attribute embedding vector are semantically matched, and the vehicle's cross-modal semantic attributes are aligned from three different levels using multi-granularity semantics to obtain the final visual attribute embedding and the final text attribute embedding, as detailed below: Obtain several visual attribute vectors from the updated visual attribute embedding vectors, construct an undirected fully connected visual attribute graph from the several visual attribute vectors, calculate the visual similarity matrix between several different visual attribute graphs, aggregate the word-level visual attribute nodes in each visual attribute graph through the similarity weighting method to obtain the phrase-level visual attribute nodes, and obtain the sentence-level visual attribute nodes based on the phrase-level visual attribute nodes. Obtain several text attribute vectors from the updated text attribute embedding vectors, construct an undirected fully connected text attribute graph from the several text attribute vectors, calculate the text similarity matrix between several different text attribute graphs, aggregate the word-level text attribute nodes in each text attribute graph through the similarity weighting method to obtain the phrase-level text attribute nodes, and obtain the sentence-level text attribute nodes based on the phrase-level text attribute nodes. Semantic alignment of vehicle attributes with corresponding relationships is performed between the visual attribute map and the text attribute map at three levels: word level, phrase level, and sentence level. The vehicle attributes with corresponding relationships include word-level visual attribute nodes and word-level text attribute nodes, phrase-level visual attribute nodes and phrase-level text attribute nodes, and sentence-level visual attribute nodes and sentence-level text attribute nodes. By leveraging high-level semantic information to facilitate the learning of word-level visual attribute nodes, we merge sentence-level and phrase-level visual attribute nodes into a semantic composite graph. Then, we use a graph attention mechanism to propagate the attribute information from the high-level semantic composite graph back to the word-level attribute nodes, resulting in the final visual attribute embedding and the final text attribute embedding.

[0012] Furthermore, the specific formula for calculating the visual similarity matrix is ​​as follows:

[0013] The specific formula for calculating the text similarity matrix is ​​as follows:

[0014] in, Represents the first in the visual attribute diagram Node and the Visual similarity between nodes Represents the first in the text attribute graph Node and the Text similarity between nodes It is a visual attribute node Embedded vector, It is a visual attribute node Embedded vector, It is a text attribute node Embedded vector, It is a text attribute node Embedded vector, For transpose operation, This is an adjustable parameter.

[0015] Furthermore, the similarity between the vehicle video and the artificial text description is obtained based on the final visual attribute embedding and the final text attribute embedding. The vehicle video with the highest similarity is the vehicle video retrieval result, as detailed below: Define a modality classifier to perform modality classification by applying a binary cross-entropy classification loss to a set of input images or text, with the goal of minimizing the adversarial loss; Construct a cross-modal learner that includes identity loss, circle loss, and attribute classification loss; The learnable parameters are optimized using an adversarial learning network based on modality classifiers and cross-modality learners. The final visual attribute embedding and the final text attribute embedding are obtained based on the optimized learnable parameters. The similarity between vehicle videos and human-written text descriptions is obtained based on the final visual attribute embedding and the final text attribute embedding. The vehicle video with the highest similarity is the vehicle video retrieval result.

[0016] Secondly, the present invention provides a vehicle image and text retrieval system based on multi-level semantic graph alignment and attribute enhancement, comprising: The data acquisition module is used to acquire vehicle videos and human-written text descriptions of the vehicle videos, and to preprocess the human-written text descriptions to obtain text data. The vehicle attribute information acquisition module is used to parse text data to obtain vehicle attribute information. The fusion embedding module is used to obtain fused visual embeddings based on learnable parameters and vehicle video, and to obtain fused text embeddings based on learnable parameters and text data. The attribute embedding vector acquisition module is used to obtain updated visual attribute embedding vectors and updated text attribute embedding vectors based on fused visual embedding and fused text embedding; The multi-level semantic graph alignment module is used to perform semantic matching on the updated visual attribute embedding vector and the updated text attribute embedding vector. It uses multi-granularity semantics to align the vehicle's cross-modal semantic attributes from three different levels to obtain the final visual attribute embedding and the final text attribute embedding. The search result acquisition module is used to obtain the similarity between vehicle videos and human-written text descriptions based on the final visual attribute embedding and the final text attribute embedding. The vehicle video with the highest similarity is the vehicle video search result.

[0017] Compared with the prior art, the present invention has the following beneficial effects: This invention acquires vehicle videos and their accompanying text descriptions. The text descriptions are preprocessed to obtain high-quality data, preventing retrieval errors caused by poor text quality. Text attribute parsing is then performed to extract vehicle attribute information, making the retrieval process more accurate and targeted, thus improving efficiency. A fused visual embedding is obtained from the vehicle videos, and a fused text embedding is obtained from the text data, addressing the issue of shallow fusion of local and global features. Updated visual and text attribute embedding vectors are obtained based on the fused visual and text embeddings, enhancing feature robustness and expressiveness, and reducing retrieval errors caused by insufficient single-modal features. Semantic matching of the updated visual and text attribute embedding vectors is performed, aligning cross-modal semantic attributes of the vehicle at three different levels using multi-granularity semantics, resulting in final visual and text attribute embeddings. Through attribute aggregation and interaction, the significant gap between visual and textual data is effectively bridged, enhancing cross-modal semantic consistency. The similarity between vehicle videos and human-written text descriptions is obtained based on the final visual attribute embedding and the final text attribute embedding. The vehicle video with the highest similarity is selected as the vehicle video retrieval result. This method can quickly and accurately find target vehicle trajectory segments that match a given natural language description from a massive database of vehicle trajectory video clips. This invention achieves deep fusion and effective alignment of vehicle images and text data by employing multi-level semantic graph alignment and attribute enhancement techniques. This enables rapid and accurate location of target vehicles, exhibiting stronger robustness and a higher retrieval success rate, significantly improving the accuracy and efficiency of vehicle retrieval.

[0018] The system of this invention includes: a data acquisition module, a vehicle attribute information acquisition module, a fusion embedding module, an attribute embedding vector acquisition module, a multi-level semantic graph alignment module, and a retrieval result acquisition module. The data acquisition module acquires vehicle videos and artificial text descriptions of the vehicle videos, and preprocesses the artificial text descriptions to obtain text data. The vehicle attribute information acquisition module parses the text data to obtain vehicle attribute information. The fusion embedding module acquires fused visual embeddings based on learnable parameters and vehicle videos, and fused text embeddings based on learnable parameters and text data. The attribute embedding vector acquisition module acquires updated visual attribute embedding vectors and updated text attribute embedding vectors based on the fused visual embeddings and fused text embeddings. The multi-level semantic graph alignment module performs semantic matching on the updated visual attribute embedding vectors and updated text attribute embedding vectors, aligning vehicle cross-modal semantic attributes at three different levels using multi-granularity semantics to obtain the final visual attribute embedding and the final text attribute embedding. The retrieval result acquisition module acquires the similarity between the vehicle videos and artificial text descriptions based on the final visual attribute embedding and the final text attribute embedding, with the vehicle video having the highest similarity being the vehicle video retrieval result. The various modules work together to achieve deep fusion and effective alignment of vehicle images and text data, significantly improving the accuracy and efficiency of vehicle retrieval. In vehicle retrieval tasks within complex scenarios, this invention can quickly and accurately locate target vehicles, demonstrating stronger robustness and a higher retrieval success rate. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the vehicle trajectory retrieval model of the present invention.

[0020] Figure 2 This is a schematic diagram of the visual embedding and fusion F-Encoder of the present invention.

[0021] Figure 3 This is a schematic diagram of the text embedding and fusion F-Encoder of the present invention.

[0022] Figure 4 This is a schematic diagram of the multi-level semantic graph alignment framework of the present invention.

[0023] Figure 5 This is a flowchart of the method of the present invention.

[0024] Figure 6 This is a system module diagram of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," etc., in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] The present invention will now be described in further detail with reference to the accompanying drawings: Example 1: See Figure 5 This invention discloses a vehicle image and text retrieval method based on multi-level semantic graph alignment and attribute enhancement, comprising the following steps: S1. Obtain vehicle videos and their accompanying text descriptions. Preprocess the text descriptions to obtain text data. Preprocessing removes noise and irrelevant characters from the text, standardizes the text format, and makes the text data more standardized and accurate. This provides a high-quality data source for accurate subsequent parsing of text attributes and avoids deviations in subsequent searches due to text data quality issues.

[0028] In this embodiment of the invention, the preprocessing of the artificial text description to obtain text data is as follows: Each piece of human-written text was broken down into different words using the Stanza tool; The Levenshtein distance algorithm is used to identify and correct incorrect words. Each word is annotated with its part of speech using part-of-speech tagging.

[0029] S2. Perform text attribute parsing on the text data to obtain vehicle attribute information. Vehicle attribute information is an important basis for subsequent vehicle retrieval. It helps to transform natural language descriptions into feature information that computers can understand and process, making the retrieval process more accurate and targeted, and improving retrieval efficiency.

[0030] In this embodiment of the invention, the process of parsing text attributes from text data to obtain vehicle attribute information is as follows: Perform statistical analysis on the text to extract local and global vehicle attributes. Apply component syntactic analysis tools to analyze the local and global vehicle attributes and generate attribute labels based on the analysis results. The attribute tags include: color ( ),size( ),type( ), vehicle motion ( Traffic scenarios ), surrounding vehicles ( ) and additional information ( ).

[0031] Post-processing of attribute tags involves checking the attribute tags of each text data. For attribute tags that are ambiguous or have the same meaning, the attribute tag that appears most frequently is selected as the final vehicle attribute tag.

[0032] S3. Obtain fused visual embeddings from vehicle videos and fused text embeddings from text data to solve the problem of shallow fusion of local and global features; See Figure 2 and Figure 3 In this embodiment of the invention, the step of obtaining the fused visual embedding based on vehicle video and the fused text embedding based on text data is as follows: Vehicle videos are uniformly sampled and cropped frame by frame to obtain local vehicle images. Based on these local images, a convolutional neural network (CNN) and a multilayer perceptron are used to perform local visual embedding to obtain local visual embedding results. A Video Swin Transformer model is then used to perform global visual embedding to obtain global visual embedding results. An F-Encoder structure is applied to fuse the local and global visual embedding results. The initial fused visual embedding is obtained through a self-attention layer and then through a cross-attention layer and learnable parameters to obtain the final fused visual embedding. Local text embeddings are extracted from text data using a word embedding model and a multilayer perceptron; the text data is input into a large language model to obtain global text embeddings; the local text embeddings and global text embeddings are fused using an F-Encoder structure, and then passed through a self-attention layer and a cross-attention layer in sequence with learnable parameters to obtain the fused text embedding.

[0033] Among them, the F-Encoder structure refers to an encoder architecture with hierarchical processing capabilities and multiple feature interaction mechanisms; The Video Swin Transformer model is a video processing model based on the Transformer architecture. It extends the Swin Transformer (a Transformer model for image processing) to the video domain and achieves efficient modeling of the spatiotemporal features of video by introducing a time dimension and a 3D sliding window mechanism.

[0034] S4. Obtaining updated visual attribute embedding vectors and updated text attribute embedding vectors based on the fusion of visual embedding and fusion of text embedding is beneficial to enhancing the robustness and expressiveness of features and reducing retrieval errors caused by insufficient single-modal features.

[0035] In this embodiment of the invention, the method for obtaining the updated visual attribute embedding vector is as follows: The fused visual embedding is taken as input and a visual attribute feature map is obtained through a fully connected layer and three one-dimensional convolutional layers. Based on the fused text embedding, a visual attribute attention map is learned through a fully connected layer and two one-dimensional convolutional layers. Then, the visual attribute feature map is compressed into the visual attribute attention map through channel convolution operation to obtain the visual attribute embedding. Finally, the visual attribute embedding is fused with the fused visual embedding to obtain the updated visual attribute embedding vector.

[0036] In this embodiment of the invention, the method for obtaining the updated text attribute embedding vector is as follows: The fused text embedding is taken as input, and a text attribute feature map is obtained through a fully connected layer and three one-dimensional convolutional layers. Based on the fused visual embedding, a text attribute attention map is learned through a fully connected layer and two one-dimensional convolutional layers. Then, the text attribute feature map is compressed into the text attribute attention map through channel convolution operation to obtain the text attribute embedding. Finally, the text attribute embedding is fused with the fused text embedding to obtain the updated text attribute embedding vector.

[0037] S5. Semantic matching is performed on the updated visual attribute embedding vector and the updated text attribute embedding vector. Multi-granularity semantics is used to align the vehicle's cross-modal semantic attributes from three different levels to obtain the final visual attribute embedding and the final text attribute embedding. Through attribute aggregation and interaction, the significant gap between visual and text is effectively bridged, and cross-modal semantic consistency is enhanced. In this embodiment of the invention, the updated visual attribute embedding vector and the updated text attribute embedding vector are semantically matched, and the vehicle's cross-modal semantic attributes are aligned from three different levels using multi-granularity semantics to obtain the final visual attribute embedding and the final text attribute embedding, as detailed below: Obtain several visual attribute vectors from the updated visual attribute embedding vectors, construct an undirected fully connected visual attribute graph from the several visual attribute vectors, calculate the visual similarity matrix between several different visual attribute graphs, aggregate the word-level visual attribute nodes in each visual attribute graph through the similarity weighting method to obtain the phrase-level visual attribute nodes, and obtain the sentence-level visual attribute nodes based on the phrase-level visual attribute nodes. Obtain several text attribute vectors from the updated text attribute embedding vectors, construct an undirected fully connected text attribute graph from the several text attribute vectors, calculate the text similarity matrix between several different text attribute graphs, aggregate the word-level text attribute nodes in each text attribute graph through the similarity weighting method to obtain the phrase-level text attribute nodes, and obtain the sentence-level text attribute nodes based on the phrase-level text attribute nodes. Semantic alignment of vehicle attributes with corresponding relationships is performed between the visual attribute map and the text attribute map at three levels: word level, phrase level, and sentence level. The vehicle attributes with corresponding relationships include word-level visual attribute nodes and word-level text attribute nodes, phrase-level visual attribute nodes and phrase-level text attribute nodes, and sentence-level visual attribute nodes and sentence-level text attribute nodes. By leveraging high-level semantic information to facilitate the learning of word-level visual attribute nodes, we merge sentence-level and phrase-level visual attribute nodes into a semantic composite graph. Then, we use a graph attention mechanism to propagate the attribute information from the high-level semantic composite graph back to the word-level attribute nodes, resulting in the final visual attribute embedding and the final text attribute embedding.

[0038] In this embodiment of the invention, the formula for calculating the visual similarity matrix is ​​as follows:

[0039] The specific formula for calculating the text similarity matrix is ​​as follows:

[0040] in, Represents the first in the visual attribute diagram Node and the Visual similarity between nodes Represents the first in the text attribute graph Node and the Text similarity between nodes It is a visual attribute node Embedded vector, It is a visual attribute node Embedded vector, It is a text attribute node Embedded vector, It is a text attribute node Embedded vector, For transpose operation, This is an adjustable parameter.

[0041] S6. Based on the final visual attribute embedding and the final text attribute embedding, the similarity between the vehicle video and the human-written text description is obtained. The vehicle video with the highest similarity is selected as the vehicle video retrieval result. Using the vehicle video with the highest similarity as the retrieval result meets the target requirements of vehicle retrieval. It can quickly and accurately find the target vehicle trajectory segment that matches the given natural language description from a massive vehicle trajectory video clip library, providing effective technical support for vehicle monitoring, tracking and other tasks in smart cities.

[0042] In this embodiment of the invention, the similarity between the vehicle video and the artificial text description is obtained based on the final visual attribute embedding and the final text attribute embedding. The vehicle video with the highest similarity is the vehicle video retrieval result, as detailed below: Define a modality classifier to perform modality classification by applying a binary cross-entropy classification loss to a set of input images or text, with the goal of minimizing the adversarial loss; Construct a cross-modal learner that includes identity loss, circle loss, and attribute classification loss; The learnable parameters are optimized using an adversarial learning network based on modality classifiers and cross-modality learners. The final visual attribute embedding and the final text attribute embedding are obtained based on the optimized learnable parameters. The similarity between vehicle videos and human-written text descriptions is obtained based on the final visual attribute embedding and the final text attribute embedding. The vehicle video with the highest similarity is the vehicle video retrieval result.

[0043] This invention achieves deep fusion and effective alignment of vehicle images and text data by employing multi-level semantic graph alignment and attribute enhancement techniques, significantly improving the accuracy and efficiency of vehicle retrieval. Compared with existing technologies, this invention strengthens the fusion of multimodal features through an F-Encoder structure, enhances the representation capability of vehicle attributes through a collaborative attention mechanism, and accurately captures cross-modal semantic associations at different granularities through a multi-level semantic graph alignment module. Finally, it further optimizes model performance through adversarial learning, enabling rapid and accurate location of target vehicles in complex vehicle retrieval tasks, exhibiting stronger robustness and a higher retrieval success rate.

[0044] Based on the above method, this invention also discloses a vehicle image and text retrieval system based on multi-level semantic graph alignment and attribute enhancement, see [link to relevant documentation]. Figure 6 ,include: The data acquisition module is used to acquire vehicle videos and human-written text descriptions of the vehicle videos, and to preprocess the human-written text descriptions to obtain text data. The vehicle attribute information acquisition module is used to parse text data to obtain vehicle attribute information. The fusion embedding module is used to obtain fused visual embeddings based on learnable parameters and vehicle video, and to obtain fused text embeddings based on learnable parameters and text data. The attribute embedding vector acquisition module is used to obtain updated visual attribute embedding vectors and updated text attribute embedding vectors based on fused visual embedding and fused text embedding; The multi-level semantic graph alignment module is used to perform semantic matching on the updated visual attribute embedding vector and the updated text attribute embedding vector. It uses multi-granularity semantics to align the vehicle's cross-modal semantic attributes from three different levels to obtain the final visual attribute embedding and the final text attribute embedding. The search result acquisition module is used to obtain the similarity between vehicle videos and human-written text descriptions based on the final visual attribute embedding and the final text attribute embedding. The vehicle video with the highest similarity is the vehicle video search result.

[0045] The various modules of this invention work together to achieve deep fusion and effective alignment of vehicle images and text data, significantly improving the accuracy and efficiency of vehicle retrieval. In vehicle retrieval tasks within complex scenarios, this invention can quickly and accurately locate target vehicles, demonstrating stronger robustness and a higher retrieval success rate.

[0046] Example 2: See Figure 1 This embodiment discloses a vehicle image and text retrieval method based on multi-level semantic graph alignment and attribute enhancement, including the following steps: S1. Construct a dataset containing vehicle videos collected from different cameras in the city and related human-written text descriptions for vehicle retrieval, as follows: The text data is preprocessed by first using the Stanza tool to break down each original text into different words, then using the Levenshtein distance algorithm to identify and correct erroneous words, and finally using part-of-speech tagging to annotate the part of speech of each word in the vehicle text description. S2. Perform text attribute parsing to obtain possible vehicle attribute information in the text data. The specific steps are as follows: S2-1. Perform component syntactic analysis on the text description, perform statistical analysis on the text, extract local and global vehicle attributes, apply component syntactic analysis tools to analyze the text query of vehicle attribute information, and present the analysis results in the form of a hierarchical nested tree structure. As a preferred embodiment of the present invention, in S2-1, seven attribute labels are generated for the trajectory of each vehicle: color ( ),size( ),type( ), vehicle motion ( Traffic scenarios ), surrounding vehicles ( ) and additional information ( ).

[0047] S2-2. Post-processing of attribute labels for text descriptions: Re-examine the labels of each trajectory description text. For different descriptions that are ambiguous or have the same meaning, select the attribute with the highest frequency as the final vehicle attribute label. S3, see also Figure 2 and Figure 3 , Figure 2 This is a schematic diagram of the visual embedding and fusion F-Encoder of the present invention. Figure 3 This is a schematic diagram of the text embedding and fusion F-Encoder of the present invention. Based on the text attribute parsing in S2, a visual-text embedding and fusion module is constructed. It utilizes preprocessed data to extract and fuse local and global features from both visual and textual modalities. The specific steps are as follows: S3-1, Local and Global Visual Embedding Fusion: First, each vehicle trajectory segment is uniformly sampled and cropped frame by frame to obtain a local vehicle image. , For the first Images of individual vehicles were used, and then a multilayer perceptron with a convolutional neural network as its backbone was used to perform local visual embedding to obtain the local visual embedding result. , Indicates visual embedding. This represents local embedding. Global visual embedding is obtained using the Video Swin Transformer model. , Indicates visual embedding. The global embedding is represented, and then an F-Encoder structure is applied to fuse the local visual embedding results. With global visual embedding results The two are processed through a self-attention layer to obtain an initial fused visual embedding. The fused visual embedding is then obtained by passing through a cross-attention layer. ; As a preferred technical solution of the present invention, in S3-1, uniform sampling is performed from the video. Frame vehicle partial image Input it into ResNet101 to obtain the local visual embedding results. ,in and It is a partial image of the vehicle. The height and width, where c is the embedding dimension, which is 1024. For a vector space, the specific embedding generation process is as follows: Where MLP(·) is a multilayer perceptron, and its calculation formula is: In the above function, It is a fully connected linear function, and Res(·) represents a stack of 3 layers of ResNet101. It is a rectified linear unit activation function, and the extracted global features are: ,in It's the frame rate. and These are the height and width of the input frame. This is the number of channels in the first stage of the hidden layer, denoted as 96; S3-2, Local and global text embedding fusion: using a word embedding model and a multilayer perceptron to extract local text embeddings. All vehicle-related text descriptions are input into a large language model to obtain global text embeddings. Then, an F-Encoder structure is used to fuse local text embeddings. and global text embedding The fused text embedding is obtained by sequentially passing through a self-attention layer and a cross-attention layer. ; As a preferred technical solution of the present invention, in step S3-2, Word2Vec and a multilayer perceptron are used to obtain the local text embedding of the vehicle. The specific generation process is as follows: ,in, It is a set of keywords for a specific part of the vehicle's text. It is the Word2Vec text embedding model, and the global text embedding extracted by the large language model is... .

[0048] As a preferred embodiment of the present invention, in S3-1 and S3-2, local features and global features are fused through an F-Encoder structure. Taking visual embedding as an example, the self-attention layer calculation method for local visual embedding is as follows:

[0049]

[0050]

[0051] in, , and These are learnable parameters; Indicates the first Output of the head; This is a result of the self-attention layer; Indicates a connection operation; express The query variable; express Key variables; express Value variables; Indicates the activation function; Indicates the transpose operation; Represents the level of linear transformation; The calculation process for the cross-attention layer is similar, as shown below:

[0052]

[0053]

[0054] in, , and For network parameters; Indicates the first Output of size; integration of visual embedding It is the final output of the cross-attention layer; The F-Encoder for text fusion feature extraction does not share parameters with the visual fusion feature encoder, and they are executed simultaneously.

[0055] S4, based on the fusion visual embedding in S3 and fusion text embedding A collaborative attention attribute enhancement module is constructed. This module utilizes attention information from another modality during the vehicle attribute learning process to promote co-learning and enhance the representation of vehicle attributes. The specific steps are as follows: S4-1, Visual Attribute Enhancement Section, which integrates visual embeddings As input, visual attribute feature maps are obtained by passing through a fully connected layer and three one-dimensional convolutional layers. In the integration of text embedding Based on this, visual attribute attention maps are learned through a fully connected layer and two one-dimensional convolutional layers. Then the visual attribute feature map Compressed into a visual attribute attention map through channel convolution operation. Visual attribute embedding is obtained from Finally, embed the visual attributes. With fusion visual embedding The fusion yields an updated visual attribute embedding vector. ; S4-2, Text Attribute Enhancement Section, will integrate text embedding As input, text attribute feature maps are obtained by passing the data through a fully connected layer and three one-dimensional convolutional layers. In the fusion of visual embedding Based on this, a text attribute attention map is learned through a fully connected layer and two one-dimensional convolutional layers. Then, the text attribute feature map Compressed into a text attribute attention map through channel convolution operations. Text attribute embedding obtained from Finally, embed the text attributes. With fused text embedding The fusion yields an updated text attribute embedding vector. ; As a preferred embodiment of the present invention, the visual attribute enhancement process and the text attribute enhancement process in S4-1 and S4-2 are performed in the same way. The following example uses the visual attribute enhancement part, where the visual attribute feature map is... Learning visual attribute attention maps ,in The channel convolution operation will be derived from seven predefined vehicle attributes. Compression into visual attribute embedding Each element is calculated as follows:

[0056] in, Indicates the first The visual attribute embedding of the first attribute One element, This represents the embedding vector for each visual attribute; by concatenating the seven attribute vectors, we obtain the visual attribute embedding vector. Then and The fusion yields an updated visual attribute embedding vector. It contains N visual attributes, and each visual attribute vector is... Similarly, N text attribute vectors It can also be obtained;

[0057] in and This is a proportional parameter; and They are and The weight matrix; the text attention attribute embedding is obtained using the same method. Using seven attributes ( To predict based on tags and The value of each attribute.

[0058] S5, see also Figure 4 This is a schematic diagram of the multi-level semantic graph alignment framework of the present invention, which aims to embed the updated visual attributes extracted in S4 into the vector. and updated text attribute embedding vector Semantic matching is performed, and a multi-level semantic graph alignment module is constructed to align the visual and textual attributes of vehicles at three different levels: word level, phrase level, and sentence level. The specific steps are as follows: S5-1, Attribute Aggregation and Semantic Graph Construction: Construct undirected fully connected visual attribute graphs from N visual attribute vectors and N text attribute vectors respectively. and text attribute graph To reveal implicit semantic dependencies, first, the similarity matrix between N different visual (textual) attributes is calculated. and Then, the visual attribute nodes in each visual (text) attribute graph are aggregated using a similarity-weighted method. ( The information allows for the aggregation of visual (text) attributes from the word level to the phrase level, and then to the sentence level. As a preferred embodiment of the present invention, in S5-1, the similarity matrix between attributes is calculated. and ;

[0059]

[0060] in, and Representing visual attribute diagrams and text attribute graph The Middle Node and the Similarity between nodes It is a visual attribute node Embedded vector, It is a text attribute node The embedding vectors for each visual attribute node and text attribute node are calculated as follows:

[0061]

[0062] in, and They represent the first Embedding of visual and text attribute nodes at the word level. Represents the concatenation of vectors, and the visual update function. and text update function Able to store word-level attribute nodes These are aggregated to form phrase-level attribute nodes. Phras-level attribute points can be further aggregated to form sentence-level attribute nodes. The hierarchical aggregation process is as follows:

[0063]

[0064] in, and These refer to word-level and phrase-level visual attribute maps, respectively. and These refer to word-level and phrase-level text attribute graphs, respectively. These are attribute nodes for the entire sentence level visual representation. These are attribute nodes at the sentence-level of the text.

[0065] S5-2, Multi-level alignment between patterns: Semantic alignment of vehicle attributes with corresponding relationships between visual and text attribute graphs at three levels: word level, phrase level, and sentence level. This includes two processes: alignment from visual attribute nodes to text attribute nodes and bidirectional alignment from text attribute nodes to visual attribute nodes. As a preferred embodiment of the present invention, in S5-2, the alignment process from each visual attribute node to the text attribute node at the word level is as follows:

[0066] in, and These represent the first word on the word-level visual attribute map. The first visual attribute node and the first text attribute graph Each text attribute node; , and These are learnable parameters; weights Indicates the first The visual attribute node to the first The text attribute node in the _ ... Attention value of each attention head; It is the total number of attention heads; vector It is the first in the visual attribute diagram Attention-updated attribute embeddings for each node keep the correspondence between visual and text information consistent. These operations are also performed in phrase-level and sentence-level attribute graphs.

[0067] S5-3, Intramodal Semantic Backtracking: High-level semantic information is used to facilitate the learning of word-level nodes. Sentence-level and phrase-level visual attribute nodes are merged into a semantic composite graph. A graph attention mechanism is used to propagate attribute information from the high-level semantic composite graph back to the word-level attribute nodes, resulting in the final visual attribute embedding. and the final text attribute embedding ; S6. Construct an adversarial learning module to optimize the learnable parameters learned in steps S3-S5 above. First, define a modality classifier. Applying a binary cross-entropy classification loss to a set of inputs (images or text) to achieve modality classification, the goal is to minimize the adversarial loss. Then construct a model that includes identity loss. Circular loss and attribute classification loss The cross-modal learner uses an adversarial learning network based on modality classifiers and cross-modal learners to optimize learnable parameters. The optimized model then sorts each text description and vehicle video according to similarity, ultimately returning the most similar vehicle text and video pair to achieve real-world vehicle video retrieval.

[0068] As a preferred embodiment of the present invention, in S6, the modality classifier The losses in the fight Defined as ,in It is the modal label of the true value of each input. Output the modal probability for each input; the attribute classification loss in the cross-modal learner is:

[0069]

[0070] in, =7 indicates seven vehicle attributes. and These are the loss functions for visual attribute embedding and text attribute embedding, respectively. Indicates the number of samples. This is the number of tag categories for each attribute; and These are the first in visual embedding and text embedding. The first attribute The probability of the i-th category, the i-th The first attribute The true values ​​for each category are respectively represented as: and Introducing cross-entropy function to the identity loss ,in, Represents the true value. Indicates the parameters of the classifier; It is the quantity input in each batch. This represents the probability of obtaining each identity; the circular loss achieves this by penalizing the similarity of negative sample pairs while encouraging the similarity of positive sample pairs: ,in and These represent the similarity scores of positive and negative pairs, respectively. It is the inner margin of the class. It is the margin between classes. It is a predefined parameter, the penalty coefficient. and The definition is:

[0071] in, and Set as ; The overall loss function of a cross-modal learner can be expressed as:

[0072] in, , and These are learnable hyperparameters. These represent the learnable parameters of the visual branch in the model. This represents the learnable parameters of the text branches in the model. The learnable parameters of the cross-entropy function in identity loss are represented; The optimization of the model is similar to a mini-maximum game involving two concurrent sub-processes:

[0073] in, This represents the estimated values ​​of the learnable parameters of the visual branch in the model. This represents the estimated values ​​of the learnable parameters for the text branch in the model. This represents an estimate of the learnable parameters of the cross-entropy function in identity loss. This represents the variable value that minimizes the objective function. Indicating resistance to loss, Modal classifier Learnable parameters of adversarial loss in the model. This represents the value of the variable that maximizes the objective function.

[0074] The F-Encoder module of this invention solves the problem of shallow fusion in multimodal feature embedding; at the same time, it uses a multi-level semantic graph alignment module to solve the alignment problem between visual and textual modalities in vehicle trajectory retrieval tasks.

[0075] Example 3: The performance of the vehicle image-text retrieval method and system based on multi-level semantic graph alignment and attribute enhancement in this invention on the CityFlow-NL dataset is shown in the table below:

[0076] The beneficial effects of this invention are: This vehicle image-text retrieval method based on multi-level semantic graph alignment and attribute enhancement achieves deep fusion and effective alignment of vehicle images and text data, significantly improving the accuracy and efficiency of vehicle retrieval. Compared with existing technologies, this method strengthens the fusion of multimodal features through an F-Encoder structure, enhances the representation ability of vehicle attributes through a collaborative attention mechanism, and accurately captures cross-modal semantic associations at different granularities through a multi-level semantic graph alignment module. Finally, adversarial learning is used to further optimize model performance, enabling the rapid and accurate location of target vehicles in complex vehicle retrieval tasks, demonstrating stronger robustness and a higher retrieval success rate.

[0077] An electronic device includes: a processor; a memory for storing computer program instructions; and steps for implementing a method for automatically synchronizing historical data in a real-time database when executing the computer program.

[0078] A storage medium storing computer program instructions, which, when loaded and executed by a processor, enable the processor to perform a method for automatically synchronizing historical data with a real-time database.

[0079] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0081] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0082] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0083] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A vehicle image-text retrieval method based on multi-level semantic graph alignment and attribute enhancement, characterized in that, Includes the following steps: Obtain vehicle videos and human-written text descriptions of the vehicle videos, and preprocess the human-written text descriptions to obtain text data; Text attribute parsing is performed on text data to obtain vehicle attribute information; The fused visual embedding is obtained from the vehicle video, and the fused text embedding is obtained from the text data. The updated visual attribute embedding vector and the updated text attribute embedding vector are obtained based on the fusion of visual embedding and fusion of text embedding; Semantic matching is performed on the updated visual attribute embedding vector and the updated text attribute embedding vector. Multi-granular semantics are used to align the vehicle's cross-modal semantic attributes from three different levels to obtain the final visual attribute embedding and the final text attribute embedding. The similarity between vehicle videos and human-written text descriptions is obtained based on the final visual attribute embedding and the final text attribute embedding. The vehicle video with the highest similarity is the vehicle video retrieval result.

2. The vehicle image-text retrieval method based on multi-level semantic graph alignment and attribute enhancement according to claim 1, characterized in that, The preprocessing of the artificial text description to obtain text data is as follows: Each piece of human-written text was broken down into different words using the Stanza tool; The Levenshtein distance algorithm is used to identify and correct incorrect words. Each word is annotated with its part of speech using part-of-speech tagging.

3. The vehicle image-text retrieval method based on multi-level semantic graph alignment and attribute enhancement according to claim 1, characterized in that, The process of parsing text attributes from text data to obtain vehicle attribute information is as follows: Perform statistical analysis on the text to extract local and global vehicle attributes. Apply component syntactic analysis tools to analyze the local and global vehicle attributes and generate attribute labels based on the analysis results. The attribute tags include: color ( ),size( ),type( ), vehicle motion ( Traffic scenarios ), surrounding vehicles ( ) and additional information ( ); Post-processing of attribute tags involves checking the attribute tags of each text data. For attribute tags that are ambiguous or have the same meaning, the attribute tag that appears most frequently is selected as the final vehicle attribute tag.

4. The vehicle image-text retrieval method based on multi-level semantic graph alignment and attribute enhancement according to claim 1, characterized in that, The process of obtaining fused visual embeddings from vehicle video and fused text embeddings from text data is as follows: Vehicle videos are uniformly sampled and cropped frame by frame to obtain local vehicle images. Based on these local images, a convolutional neural network (CNN) and a multilayer perceptron are used to perform local visual embedding to obtain local visual embedding results. A Video Swin Transformer model is then used to perform global visual embedding to obtain global visual embedding results. An F-Encoder structure is applied to fuse the local and global visual embedding results. The initial fused visual embedding is obtained through a self-attention layer and then through a cross-attention layer and learnable parameters to obtain the final fused visual embedding. Local text embeddings are extracted from text data using a word embedding model and a multilayer perceptron; the text data is input into a large language model to obtain global text embeddings; the local text embeddings and global text embeddings are fused using an F-Encoder structure, and then passed through a self-attention layer and a cross-attention layer in sequence with learnable parameters to obtain the fused text embedding.

5. The vehicle image-text retrieval method based on multi-level semantic graph alignment and attribute enhancement according to claim 1, characterized in that, The updated visual attribute embedding vector is obtained as follows: The fused visual embedding is taken as input and a visual attribute feature map is obtained through a fully connected layer and three one-dimensional convolutional layers. Based on the fused text embedding, a visual attribute attention map is learned through a fully connected layer and two one-dimensional convolutional layers. Then, the visual attribute feature map is compressed into the visual attribute attention map through channel convolution operation to obtain the visual attribute embedding. Finally, the visual attribute embedding is fused with the fused visual embedding to obtain the updated visual attribute embedding vector.

6. The vehicle image-text retrieval method based on multi-level semantic graph alignment and attribute enhancement according to claim 1, characterized in that, The updated text attribute embedding vector is obtained as follows: The fused text embedding is taken as input, and a text attribute feature map is obtained through a fully connected layer and three one-dimensional convolutional layers. Based on the fused visual embedding, a text attribute attention map is learned through a fully connected layer and two one-dimensional convolutional layers. Then, the text attribute feature map is compressed into the text attribute attention map through channel convolution operation to obtain the text attribute embedding. Finally, the text attribute embedding is fused with the fused text embedding to obtain the updated text attribute embedding vector.

7. The vehicle image-text retrieval method based on multi-level semantic graph alignment and attribute enhancement according to claim 1, characterized in that, The updated visual attribute embedding vector and the updated text attribute embedding vector are semantically matched, and the vehicle's cross-modal semantic attributes are aligned at three different levels using multi-granularity semantics to obtain the final visual attribute embedding and the final text attribute embedding, as detailed below: Obtain several visual attribute vectors from the updated visual attribute embedding vectors, construct an undirected fully connected visual attribute graph from the several visual attribute vectors, calculate the visual similarity matrix between several different visual attribute graphs, aggregate the word-level visual attribute nodes in each visual attribute graph through the similarity weighting method to obtain the phrase-level visual attribute nodes, and obtain the sentence-level visual attribute nodes based on the phrase-level visual attribute nodes. Obtain several text attribute vectors from the updated text attribute embedding vectors, construct an undirected fully connected text attribute graph from the several text attribute vectors, calculate the text similarity matrix between several different text attribute graphs, aggregate the word-level text attribute nodes in each text attribute graph through the similarity weighting method to obtain the phrase-level text attribute nodes, and obtain the sentence-level text attribute nodes based on the phrase-level text attribute nodes. Semantic alignment of vehicle attributes with corresponding relationships is performed between the visual attribute map and the text attribute map at three levels: word level, phrase level, and sentence level. The vehicle attributes with corresponding relationships include word-level visual attribute nodes and word-level text attribute nodes, phrase-level visual attribute nodes and phrase-level text attribute nodes, and sentence-level visual attribute nodes and sentence-level text attribute nodes. By leveraging high-level semantic information to facilitate the learning of word-level visual attribute nodes, we merge sentence-level and phrase-level visual attribute nodes into a semantic composite graph. Then, we use a graph attention mechanism to propagate the attribute information from the high-level semantic composite graph back to the word-level attribute nodes, resulting in the final visual attribute embedding and the final text attribute embedding.

8. The vehicle image-text retrieval method based on multi-level semantic graph alignment and attribute enhancement according to claim 7, characterized in that, The formula for calculating the visual similarity matrix is ​​as follows: The specific formula for calculating the text similarity matrix is ​​as follows: in, Represents the first in the visual attribute diagram Node and the Visual similarity between nodes Represents the first in the text attribute graph Node and the Text similarity between nodes It is a visual attribute node Embedded vector, It is a visual attribute node Embedded vector, It is a text attribute node Embedded vector, It is a text attribute node Embedded vector, For transpose operation, This is an adjustable parameter.

9. The vehicle image-text retrieval method based on multi-level semantic graph alignment and attribute enhancement according to claim 1, characterized in that, The similarity between the vehicle video and the human-written text description is obtained based on the final visual attribute embedding and the final text attribute embedding. The vehicle video with the highest similarity is the vehicle video retrieval result, as detailed below: Define a modality classifier to perform modality classification by applying a binary cross-entropy classification loss to a set of input images or text, with the goal of minimizing the adversarial loss; Construct a cross-modal learner that includes identity loss, circle loss, and attribute classification loss; The learnable parameters are optimized using an adversarial learning network based on modality classifiers and cross-modality learners. The final visual attribute embedding and the final text attribute embedding are obtained based on the optimized learnable parameters. The similarity between vehicle videos and human-written text descriptions is obtained based on the final visual attribute embedding and the final text attribute embedding. The vehicle video with the highest similarity is the vehicle video retrieval result.

10. A vehicle image-text retrieval system based on multi-level semantic graph alignment and attribute enhancement, characterized in that, include: The data acquisition module is used to acquire vehicle videos and human-written text descriptions of the vehicle videos, and to preprocess the human-written text descriptions to obtain text data. The vehicle attribute information acquisition module is used to parse text data to obtain vehicle attribute information. The fusion embedding module is used to obtain fused visual embeddings based on learnable parameters and vehicle video, and to obtain fused text embeddings based on learnable parameters and text data. The attribute embedding vector acquisition module is used to obtain updated visual attribute embedding vectors and updated text attribute embedding vectors based on fused visual embedding and fused text embedding; The multi-level semantic graph alignment module is used to perform semantic matching on the updated visual attribute embedding vector and the updated text attribute embedding vector. It uses multi-granularity semantics to align the vehicle's cross-modal semantic attributes from three different levels to obtain the final visual attribute embedding and the final text attribute embedding. The search result acquisition module is used to obtain the similarity between vehicle videos and human-written text descriptions based on the final visual attribute embedding and the final text attribute embedding. The vehicle video with the highest similarity is the vehicle video search result.

Images