Intelligent network connection automobile text video retrieval method based on semantic communication

Abstract

The invention relates to an intelligent network connection automobile text video retrieval method based on semantic communication, and belongs to the technical field of intelligent network connection automobiles. The method integrates a vehicle terminal, an edge server and a cloud server, and comprises the following steps: processing a search request initiated by a user at the vehicle terminal through a multi-semantic interaction encoder module, and extracting semantic information; encoding the extracted semantic information into a symbol stream, transmitting the symbol stream to an edge server, and decoding and recovering the semantic information by the edge server; semantic information extraction is carried out on videos in a database of the cloud server in the cloud server, and the semantic information of the videos is transmitted to an edge server; and the edge server embeds the text semantic information and the visual feature information into the same dimension to perform similarity calculation, completes fine-grained matching, and returns a matching result to the vehicle terminal. According to the method, the semantic recovery precision of the search request is improved by utilizing the historical request, and differentiated weights are given to lexical items in the video semantics and the text semantics, so that the text video retrieval precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent network-connected vehicles and relates to a text video retrieval method for intelligent network-connected vehicles based on semantic communication. Background Art

[0002] With the development of information and communication technology, the Internet of Vehicles connects vehicles, drivers and other intelligent devices to the Internet, realizes information interaction and data sharing between vehicles and other intelligent entities, and provides intelligent services for drivers and passengers. Traditional in-vehicle systems mainly present driving information, while modern intelligent in-vehicle systems provide a wider range of driving functions, such as navigation positioning, video playback, etc. through advanced human-computer interaction interfaces, significantly improving the service experience of drivers and passengers in the Internet of Vehicles. With the development of vehicle display technology, multimedia technology, and in-vehicle operating systems, the demand for diversified video content among drivers and passengers is increasing. How to search and locate video content of interest to drivers and passengers from the massive video content space faces severe challenges.

[0003] Text video retrieval technology can improve the intelligence level of drivers and passengers in obtaining interesting video content and improve their retrieval experience. At present, there are some text video retrieval schemes. For example, Radford et al. proposed the CLIP model in "Learning transferable visual models from natural language supervision". The large-scale contrast language image pre-training technology is used to make the vector distance between images and texts describing the same content closer, while the vector distance between images and texts describing different contents is farther, thereby promoting the model to learn the semantic association between images and texts, making significant progress in text video retrieval (TVR). Luo et al. processed the video into a series of frames in "Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning", and embedded the video frames and text into a shared semantic space. Different aggregation methods were proposed for processing and integrating video frame features, which can effectively combine multi-frame features into a video feature vector, and compare and learn with the text feature vector, which significantly improves the retrieval performance. However, the above schemes do not consider the application of actual scenarios. Since the query text entered by the user is easily interfered by the signal during the actual transmission process, it will affect the final retrieval effect. The introduction of semantic communication can solve this problem well. Xie et al. first proposed a specific solution to introduce deep learning into semantic communication systems in "Deep learning enabled semantic communication systems". By only extracting the user's true intention information and being able to reconstruct the sender's "intention" or "semantic content" at the receiving end, the entire system only transmits meaningful semantic information, thereby ignoring or compressing unimportant data, greatly reducing the communication bandwidth requirements and improving the robustness of the system.

[0004] Due to the high mobility of vehicles, strong interference in communication channels and the limitations of weak network coverage signals, it is very easy for the search requests submitted by vehicle network drivers and passengers to be distorted during the transmission process, resulting in limited retrieval accuracy. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a text video retrieval method for intelligent connected vehicles based on semantic communication. The three-layer architecture of vehicle terminal, edge server and cloud server is integrated to achieve efficient and accurate text video retrieval. The user initiates a video search request through the vehicle-mounted interactive system. The query semantic features are encoded and transmitted to the roadside unit through a wireless channel, and the text is restored using a decoder. The roadside unit and the cloud server respectively extract semantic features through a cross-modal fine-grained matching model, and perform matching calculations in the roadside unit to identify the most relevant video, and the results are fed back to the user. This method combines the real-time response advantages of edge computing with the powerful computing power of the cloud, optimizes channel characteristics for complex communication scenarios of the Internet of Vehicles, uses historical search records to improve the accuracy of semantic recovery under low signal-to-noise ratios, and gives query terms differentiated weights through cross-modal fine-grained semantic matching to improve retrieval accuracy.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A text and video retrieval method for intelligent connected vehicles based on semantic communication, which integrates a three-layer architecture of a vehicle terminal, an edge server, and a cloud server, and includes the following steps:

[0008] S1. Processing the search request initiated by the user through the multi-semantic interaction encoder module at the vehicle terminal and extracting semantic information;

[0009] S2, encoding the extracted semantic information into a symbol stream and transmitting it to the edge server, and decoding and restoring the semantic information at the edge server;

[0010] S3, extracting semantic information from videos in its database in the cloud server, and transmitting the extracted video semantic information to the edge server;

[0011] S4, the edge server embeds the text semantic information and visual feature information into the same dimension to perform similarity calculation and complete fine-grained matching;

[0012] S5. The above process is trained using symmetric cross entropy loss to maximize the similarity between the query and the matched videos and minimize the similarity between the query and other videos.

[0013] Furthermore, the multi-semantic interaction encoder module of the vehicle terminal in step S1 specifically includes the following steps:

[0014] S11, query statement of driver and passenger s={s 1 ,s 2 .....s N} Input the multi-semantic interaction encoder module, which introduces historical query sentences h={h 1 ,h 2......h N}, semantically encode the query statement s and the historical query statement h, and extract semantic information from them as follows:

[0015]

[0016] in, The learnable parameters are The text semantic encoder is used to extract the semantic information of query text and historical text. The text semantic encoder has multiple Transformer encoder layers; i and h i Represent the i-th word in the current query sentence and the historical query respectively;

[0017] S12. Perform a cross-attention mechanism on the semantic information of the query text and the historical text, taking the semantic information of the historical text as Q, the query text as K, and V to obtain an enhanced text representation z based on the semantic information of the historical text e :

[0018]

[0019] Among them, d h is the dimension of the attention head, W Q , W K , W V are the linear projection matrices for query, key, and value, respectively.

[0020] Further, in step S2, it specifically includes:

[0021] S21. In the channel encoder of the vehicle terminal, a dense layer with different units is used to convert the semantic information into transmitted symbols. After the cross attention, the enhanced text semantic information is obtained through channel coding into a symbol stream x:

[0022] x=C α (z e )

[0023] Among them, C α (·) represents the channel encoder with parameter set α;

[0024] The signal y received by the edge server is expressed as:

[0025] y=gx+n

[0026] Considering two channels, additive white Gaussian noise channel and Rayleigh fading channel, g is the Rayleigh fading coefficient, n is the channel noise, both obey circularly symmetric complex Gaussian distribution, that is, represents the variance of the distribution;

[0027] The semantic information recovered after decoding by the channel decoder at the edge server is expressed as

[0028]

[0029] in, represents a channel decoder with parameter set δ;

[0030] S22, decoding the obtained semantic information through a semantic decoder in the edge server, wherein the semantic decoder includes multiple Transformer decoder layers, and the initial input identifier of the semantic decoder is E cls , the embedding representation of the first word is generated as E(x 1 ), the input for generating the nth word is the embedding representation sequence of the first n-1 words, expressed as:

[0031]

[0032] For current input The self-attention calculation is represented as S t :

[0033]

[0034] Let self-attention be S t Cross-attention feature representation with channel encoder output Perform cross attention calculation and generate cross attention representation C t :

[0035]

[0036] Finally, the self-attention output S t and the cross attention output C t Concatenate, calculate the probability distribution of each word through the linear layer and the Softmax layer, and finally restore the original query sentence to be expressed as

[0037] Let the semantic decoder be represented as After the above operations, the semantic information obtained is expressed as The final recovered sentence is expressed as:

[0038]

[0039] The entire semantic communication process is trained through the cross entropy (CE) loss function to minimize the transmission query text s and the reconstructed text The difference between is expressed as:

[0040]

[0041] in, is the cross entropy loss of sentence similarity, q(s i ) is the word s i The actual probability of appearing in the transmitted sentence s, p(s i ) is s i The predicted probability of appearing in the estimated sentence s.

[0042] Further, in step S3, the cloud server uses the transformation difference of the frame embedding between adjacent time frames to reflect the motion action changes, so as to obtain an enhanced visual representation:

[0043]

[0044] Where P is the position embedding, and is the feature embedding of two adjacent frames, ζ is the sigmoid function, δ is one layer of self-attention, and v d The final visual representation is obtained by encoding and normalizing the differences with long-term relationships between all fragments and then inserting the differential enhancement tokens between each adjacent frame:

[0045]

[0046] Where T represents the time embedding.

[0047] Further, in step S4, the edge server performs query text semantic information Recorded as Project the text embedding into Embed video frames Projection and Among them, L and M are the size of the model dimension, L p is the size of the projection dimension, then:

[0048] S41. Take the query text features as Q and the visual features as K and V, and embed them into the same dimension using the normalization layer LN and the projection matrix:

[0049]

[0050] Among them, W Q , W K , W V is the projection matrix L×L in R p , LN(·) represents the normalization layer;

[0051] S42, the text feature z t With visual features V f For cross-modal interaction:

[0052]

[0053] Apply weights based on the attention between different video frames Get the video representation based on the query text:

[0054]

[0055] The final text visual representation is output through the fully connected layers FC and LN:

[0056]

[0057] S43. Enhanced frame visual representation with temporal difference sequence v f Pooling is Similarly, the visual feature z v As Q, the text feature t u As K, V, perform the above operation:

[0058]

[0059] Apply weights get:

[0060]

[0061] Through the fully connected layers FC and LN, the text representation based on the frame visual features is obtained:

[0062]

[0063] Define the similarity function s between the query text and different frames q-f It is expressed as:

[0064]

[0065] The similarity between the video and different words v-w It is expressed as:

[0066]

[0067] The correlation between the visual representation and the query representation is calculated using the average value, and the final similarity function is:

[0068] s(v i ,t j )=(s q-f +s v-w ) / 2

[0069] Among them, v i ,t jdenote the i-th visual representation and the j-th query representation respectively.

[0070] Further, in step S5, a symmetric cross entropy loss is used to maximize the similarity between the query and the matched videos and minimize the similarity between other pairs:

[0071] Given B pairs of video-historical query text pairs or video-existing text pairs, the model needs to generate and optimize B×B similarities and use symmetric cross entropy loss on similarity scores to train the model parameters:

[0072]

[0073] loss The loss of video to text and text-to-video loss The matching module is trained by adopting the above objective function, and then the video content is searched and matched according to the search request text submitted by the driver and the passenger.

[0074] The beneficial effects of the present invention are:

[0075] The invention introduces semantic communication to effectively solve the problems of reliable data transmission and effective retrieval of text and video under low signal-to-noise ratio in intelligent connected vehicle scenarios, and performs semantic mining based on user query history interests. Through multi-semantic interaction and cross-modal fine-grained matching, the video retrieval accuracy can be effectively improved, thereby improving the user experience.

[0076] This architecture introduces the idea of ​​edge computing in a hierarchical feature analysis manner. It responds to the search requests of drivers and passengers in real time through roadside units, caches the semantic features of videos in the cloud, and realizes the semantic relevance matching of text and video. This design not only greatly reduces the computing and communication burden of the cloud, but also improves the system response speed. In addition, the architecture uses the powerful computing power of cloud servers to train multi-semantic interactive codecs and cross-modal fine-grained matching models offline, and distributes them to edge servers and vehicle terminals, which enhances the accuracy of search semantic extraction and improves the accuracy and efficiency of text video retrieval, thereby significantly optimizing the service experience of drivers and passengers.

[0077] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0079] Figure 1 It is a schematic diagram of the overall architecture of the text and video retrieval method of an intelligent connected vehicle based on semantic communication of the present invention;

[0080] Figure 2 It is a schematic diagram of the overall process of the text and video retrieval method of an intelligent connected vehicle based on semantic communication of the present invention;

[0081] Figure 3 It is a schematic diagram of the flow of the multi-semantic interactive codec module of the present invention;

[0082] Figure 4 It is a schematic diagram of the process of cross-modal fine-grained feature matching of the present invention. DETAILED DESCRIPTION

[0083] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0084] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0085] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0086] See also Figure 1 to Figure 4, which is a text video retrieval method for intelligent connected cars based on semantic communication.

[0087] Example

[0088] like Figure 1 As shown, the text video retrieval method of an intelligent connected car based on semantic communication of the present invention comprises the following steps:

[0089] The user sends a video search request through the vehicle-mounted interactive system. The vehicle-mounted interactive system uses a multi-semantic interactive encoder to extract query semantic features and transmits them to the roadside unit through a wireless channel. The roadside unit uses a multi-semantic interactive decoder to restore the query text.

[0090] The roadside unit extracts semantic information from the recovered query text using a cross-modal fine-grained matching model. At the same time, the cloud server also extracts semantic information from the videos in its database using a cross-modal fine-grained matching model and transmits it to the roadside unit.

[0091] The roadside unit accepts the semantic features of both and performs matching calculations to identify the video most relevant to the query. The matching results are sent back to the driver and passengers on the vehicle through the roadside unit to complete the text video retrieval task.

[0092] This method is implemented based on a three-layer architecture of vehicle terminal layer, edge server layer, and cloud server layer. It fully integrates the advantages of the roadside unit server side in edge computing, which is close to the vehicle terminal for real-time response, and the powerful computing power and storage characteristics of the cloud. It considers the channel characteristics of complex communication scenarios in the Internet of Vehicles and uses historical search records to improve the semantic recovery accuracy of search requests under low signal-to-noise ratio. Finally, a cross-modal fine-grained semantic matching method is proposed to assign differentiated weights to candidate video content and terms in query statements to improve the accuracy of text video retrieval.

[0093] In more detail, Figure 2 As shown, the specific steps are:

[0094] Step 1: The driver initiates a video search request through the vehicle interaction system in the intelligent connected car. The system then uses the multi-semantic interaction encoder pre-trained in the cloud to extract the semantic features of the query sentence and transmits these features to the roadside unit through a wireless channel. After receiving the features, the roadside unit uses its trained multi-semantic interaction decoder to restore the original query text.

[0095] Step 1.1: Figure 3As shown in the multi-semantic interactive codec model, the first step is to obtain the search request of the driver and passenger. The present invention extracts semantic features from the search request of the driver and passenger, introduces the enhanced semantic features obtained by integrating the user's historical search records, trains them using a channel encoder, restores the user's rich semantic features through a channel decoder, and finally restores the original search text through a semantic decoder. Specifically, the following steps are included:

[0096] The input of the multi-semantic interaction encoder module is the query sentence s = {s 1 ,s 2 ……s N}, in order to obtain enhanced text representation, we introduce historical query sentences h={h 1 ,h 2 ......h N}, where s i and h i Represent the i-th word in the current query sentence and the historical query respectively. First, semantic encoding is performed on the query sentence s and the historical query sentence h, and semantic information is extracted from the two as follows:

[0097]

[0098] in, The learnable parameters are The text semantic encoder is used to extract the semantic information of query text and historical text, including multiple Transformer encoder layers.

[0099] Then, the semantic information of the query text and the historical text is subjected to a cross-attention mechanism, and the historical text semantic information is taken as Q, the query text as K, and V to obtain an enhanced text representation z based on the semantic information of the historical text e

[0100]

[0101] Among them, d h is the dimension of the attention head, W Q , W K , W V They are the linear projection matrices of query, key and value respectively, and then pass through the normalization layer and feedforward layer to get the final text representation z e .

[0102] Step 1.2: Let C α (·) represents a channel encoder with parameter set α, which uses dense layers with different units to convert semantic information into transmitted symbols. The enhanced text semantic information obtained after cross attention is channel encoded into a symbol stream x:

[0103] x=C α (z e )

[0104] Let y be the signal received by the receiver, which can be expressed as:

[0105] y=gx+n

[0106] Considering two channels, additive white Gaussian noise channel and Rayleigh fading channel, g is the Rayleigh fading coefficient, n is the channel noise, both obey circularly symmetric complex Gaussian distribution, that is, Represents the variance of the distribution.

[0107] At the receiving end, Denotes the channel decoder with parameter set δ, and the semantic information recovered after decoding is expressed as

[0108]

[0109] Step 1.3: The semantic decoder includes multiple Transformer decoder layers. The initial input identifier of the semantic decoder is E cls , the embedding representation of the first word is generated as E(x 1 ), the input for generating the nth word is the embedding representation sequence of the first n-1 words, expressed as

[0110]

[0111] For current input The self-attention calculation is represented as S t

[0112]

[0113] Let self-attention be S t Cross-attention feature representation with channel encoder output Perform cross attention calculation and generate cross attention representation C t

[0114]

[0115] Finally, the self-attention output S t and the cross attention output C t Concatenate, calculate the probability distribution of each word through the linear layer and the Softmax layer, and finally restore the original query sentence to be expressed as Let the semantic decoder be represented as After the above operations, the semantic information obtained is expressed as Then the final recovered sentence can be expressed as:

[0116]

[0117] The entire semantic communication process is trained through the cross entropy (CE) loss function to minimize the transmission query text s and the reconstructed text The difference between can be expressed as:

[0118]

[0119] in, is the cross entropy loss of sentence similarity, q(s i ) is the word s i The actual probability of appearing in the transmitted sentence s, p(s i ) is s i The predicted probability of appearing in the estimated sentence s. By minimizing the loss function, the model makes full use of historical query information, ensures the semantic accuracy of text transmission, and provides rich query semantic information for cross-modal fine-grained matching in the following text.

[0120] Step 2: If Figure 4 As shown in the schematic diagram of the cross-modal fine-grained semantic matching process, the present invention first extracts semantic information from the restored query request, and at the same time the cloud extracts semantic information from the video, and the two respectively perform video frame-query semantic interaction and video-query word interaction, and finally maximizes the similarity between the query and the matched video through the symmetric cross entropy loss, and minimizes the similarity between other pairs, thereby improving the accuracy of the retrieval model and improving the video retrieval experience of the driver and passengers. Specifically, the following steps are included:

[0121] Step 2.2: Let t i and v j They represent the input query text and video data of the cross-modal fine-grained semantic matching module respectively. The semantic information extracted from the query text recovered at the receiving end is represented as t u , the semantic information extracted from the video is represented as V f For query text Semantic information u Expressed as The text semantic encoder in CLIP is applied to generate text representation. The semantic encoder is extracted as a sequence by comparing the language-image pre-training (CLIP) encoder. The text encoder in CLIP is applied to generate text representation. It is a 12-layer 512-wide model with 8 attention heads. After CLIP, the activation token of the highest layer of transformer is regarded as the feature representation of text information.

[0122] To obtain the video representation, frames are first extracted from the video clip and then encoded by a video semantic encoder. In this paper, ViT-B / 32 with 12 layers and a patch size of 32 is used as the video encoder. The video representation is converted into an image representation using the pre-trained CLIP (ViT-B / 32) as the backbone. First, non-overlapping image patches are extracted through a visual transformer (VIT), and the interactions between each patch of the input image are modeled using the transformer architecture to obtain the final representation. After ViT and CLIP, the output of the token is used as the image representation. For the input frame sequence of the video The generated representation is Therefore, the above operation can be expressed as:

[0123]

[0124] in, and The learnable parameters are The text semantic encoder and learnable parameters are Video semantic encoder.

[0125] Step 3: Since the continuous frames contain content displacements that reflect actual actions, the transformation difference of the frame embedding between adjacent timestamps in Transfer-CLIP is used to describe the motion changes. There is a temporal correlation between frames. The present invention uses a temporal difference block to enhance the interactivity between adjacent frames and uses the frame difference as a weight to enhance the weight of the time channel to highlight the action information. By obtaining a differentially enhanced frame video representation and performing cross-modal fine-grained matching with the query statement, some unimportant frames and words can be better learned and filtered out, so that the model can pay more attention to and understand the query intentions of the driver and passengers.

[0126] Step 3.1: Since adjacent time frames reflect the semantic changes of actual actions in the video content, the transformation difference of frame embeddings between adjacent time frames is used to reflect the changes in motion actions, so as to obtain an enhanced visual representation.

[0127]

[0128] Where P is the position embedding, and is the feature embedding of two adjacent frames, ζ is the sigmoid function, δ is one layer of self-attention, and v d It is a differential enhancement token. It encodes and normalizes the differences with long-term relationships between all segments to represent the global temporal representation and capture subtle changes. Then, the differential enhancement token is inserted between each adjacent frame to obtain the final visual representation:

[0129]

[0130] For query text semantic information Recorded as Project the text embedding into Embed video frames Projection and Where L and M are the dimensions of the model, L p Is the size of the projection dimension. Take the query text features as Q, the visual features as K, V, and use the normalization layer LN and the projection matrix to embed them into the same dimension:

[0131]

[0132] W Q , W K , W V is the projection matrix L×L in R p

[0133] Step 3.2: With the visual feature V f For cross-modal interaction:

[0134]

[0135] Apply weights based on the attention between different video frames Get the video representation based on the query text:

[0136]

[0137] The final text visual representation is output through the fully connected layers FC and LN:

[0138]

[0139] Step 3.3: Since the importance of different words in the search text varies, the present invention obtains the importance weights of different words, and important words are given higher scores. By enhancing the frame visual representation v with the time difference sequence f Pooling is Similarly, the visual feature z v As Q, the text feature t u As K, V, perform the above operation:

[0140]

[0141] Apply weights get:

[0142]

[0143] Through the fully connected layers FC and LN, the text representation based on the frame visual features is obtained:

[0144]

[0145] Define the similarity function s between the query text and different frames q-f It is expressed as:

[0146]

[0147] The similarity between the video and different words v-w It is expressed as:

[0148]

[0149] The correlation between the visual representation and the query representation is calculated using the average value, and the final similarity function is:

[0150] s(v i ,t j )=(s q-f +s v-w ) / 2

[0151] Step 4: Use symmetric cross entropy loss to maximize the similarity between the query and the matching video, and minimize the similarity between other pairs. Given B pairs (video-historical query text pairs) or (video-existing text pairs), the model needs to generate and optimize B×B similarities. Use symmetric cross entropy loss on these similarity scores to train the parameters of the model.

[0152]

[0153] loss The loss of video to text and text-to-video loss By using the above objective function to train the matching module, the video content can be searched and matched according to the search request text submitted by the driver and the passenger.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A text and video retrieval method for intelligent connected vehicles based on semantic communication, characterized by: The method integrates the three-layer architecture of vehicle terminal, edge server and cloud server, and includes the following steps: S1. Processing the search request initiated by the user through the multi-semantic interaction encoder module at the vehicle terminal and extracting semantic information; S2, encoding the extracted semantic information into a symbol stream and transmitting it to the edge server, and decoding and restoring the semantic information at the edge server; S3, extracting semantic information from videos in its database in the cloud server, and transmitting the extracted video semantic information to the edge server; S4, the edge server embeds the text semantic information and visual feature information into the same dimension to perform similarity calculation and complete fine-grained matching; S5. The above process is trained using symmetric cross entropy loss to maximize the similarity between the query and the matched videos and minimize the similarity between the query and other videos.

2. The method for text and video retrieval of an intelligent connected vehicle based on semantic communication according to claim 1, characterized in that: The multi-semantic interaction encoder module of the vehicle terminal in step S1 specifically includes the following steps: S11, query statement of driver and passenger s={s1,s2......s N } Input the multi-semantic interaction encoder module, which introduces historical query sentences h={h1,h2...h N }, semantically encode the query statement s and the historical query statement h, and extract semantic information from them as follows: in, The learnable parameters are The text semantic encoder is used to extract the semantic information of query text and historical text. The text semantic encoder has multiple Transformer encoder layers; i and h i Represent the i-th word in the current query sentence and the historical query respectively; S12. Perform a cross-attention mechanism on the semantic information of the query text and the historical text, taking the semantic information of the historical text as Q, the query text as K, and V to obtain an enhanced text representation z based on the semantic information of the historical text e : Among them, d h is the dimension of the attention head, W Q , W K , W V are the linear projection matrices for query, key, and value, respectively.

3. The method for text and video retrieval of an intelligent connected vehicle based on semantic communication according to claim 2, characterized in that: In step S2, it specifically includes: S21. In the channel encoder of the vehicle terminal, a dense layer with different units is used to convert the semantic information into transmitted symbols. After the cross attention, the enhanced text semantic information is obtained through channel coding into a symbol stream x: x=C α (z e ) Among them, C α (·) represents the channel encoder with parameter set α; The signal y received by the edge server is expressed as: y=gx+n Considering two channels, additive white Gaussian noise channel and Rayleigh fading channel, g is the Rayleigh fading coefficient, n is the channel noise, both obey circularly symmetric complex Gaussian distribution, that is, represents the variance of the distribution; The semantic information recovered after decoding by the channel decoder at the edge server is expressed as in, represents a channel decoder with parameter set δ; S22, decoding the obtained semantic information through a semantic decoder in the edge server, wherein the semantic decoder includes multiple Transformer decoder layers, and the initial input identifier of the semantic decoder is E cls , the embedding representation of the first word is generated as E(x1), and the input for generating the nth word is the embedding representation sequence of the first n-1 words, expressed as: For current input The self-attention calculation is represented as S t : Let self-attention be S t Cross-attention feature representation with channel encoder output Perform cross attention calculation and generate cross attention representation C t : Finally, the self-attention output S t and the cross attention output C t Concatenate, calculate the probability distribution of each word through the linear layer and the Softmax layer, and finally restore the original query sentence to be expressed as Let the semantic decoder be represented as After the above operations, the semantic information obtained is expressed as The final recovered sentence is expressed as: The entire semantic communication process is trained through the cross entropy (CE) loss function to minimize the transmission query text s and the reconstructed text The difference between is expressed as: in, is the cross entropy loss of sentence similarity, q(s i ) is the word s i The actual probability of appearing in the transmitted sentence s, p(s i ) is s i The predicted probability of appearing in the estimated sentence s.

4. The method for text and video retrieval of an intelligent connected vehicle based on semantic communication according to claim 3, characterized in that: In step S3, the cloud server uses the transformation difference of frame embedding between adjacent time frames to reflect the change of motion action, so as to obtain an enhanced visual representation: Where P is the position embedding, and is the feature embedding of two adjacent frames, ζ is the sigmoid function, δ is one layer of self-attention, and v d The final visual representation is obtained by encoding and normalizing the differences with long-term relationships between all fragments and then inserting the differential enhancement tokens between each adjacent frame: Where T represents the time embedding.

5. The method for text and video retrieval of an intelligent connected vehicle based on semantic communication according to claim 4, characterized in that: In step S4, the edge server performs semantic analysis on the query text. Recorded as Project the text embedding into Embed video frames Projection and Among them, L and M are the size of the model dimension, L p is the size of the projection dimension, then: S41. Take the query text features as Q and the visual features as K and V, and embed them into the same dimension using the normalization layer LN and the projection matrix: Among them, W Q , W K , W V is the projection matrix L×L in R p , LN(·) represents the normalization layer; S42, the text feature z t With visual features V f For cross-modal interaction: Apply weights based on the attention between different video frames Get the video representation based on the query text: The final text visual representation is output through the fully connected layers FC and LN: S43. Enhanced frame visual representation with temporal difference sequence v f Pooling is Similarly, the visual feature z v As Q, the text feature t u As K, V, perform the above operation: Apply weights get: Through the fully connected layers FC and LN, the text representation based on the frame visual features is obtained: Define the similarity function s between the query text and different frames q-f It is expressed as: The similarity between the video and different words v-w It is expressed as: The correlation between the visual representation and the query representation is calculated using the average value, and the final similarity function is: s(v i ,t j )=(s q-f +s v-w ) / 2 Among them, v i ,t j denote the i-th visual representation and the j-th query representation respectively; The edge server determines the video matching result based on the similarity function and returns the matching video to the vehicle terminal.

6. The method for text and video retrieval of an intelligent connected vehicle based on semantic communication according to claim 5, characterized in that: In step S5, a symmetric cross-entropy loss is used to maximize the similarity between the query and the matched videos and minimize the similarity between other pairs: Given B pairs of video-historical query text pairs or video-existing text pairs, the model needs to generate and optimize B×B similarities and use symmetric cross entropy loss on similarity scores to train the model parameters: loss The loss of video to text and text-to-video loss The matching module is trained by adopting the above objective function, and then the video content is searched and matched according to the search request text submitted by the driver and the passenger.

Images