A spatio-temporal feature importance-aware text video retrieval method

Abstract

The application relates to a text video retrieval method with space-time feature importance perception, and belongs to the technical field of text video retrieval. The method specifically comprises the following steps: performing feature extraction on each frame in a video and a text description respectively, selecting key frames in the video through an importance perception mechanism, and maintaining semantic information of a query text in a transmission process by using a source channel joint coding method based on semantic communication. Image block features of each frame are aggregated into object prototypes, the objects are matched with phrases in the text description, and spatial information in the video frames is perceived. Different objects are combined into different event prototypes, the event prototypes are matched with the entire text description, time information between the frames is utilized to improve fine-grained matching between the text and the video, space-time feature processing of video content is optimized, feature importance is evaluated, efficient retrieval in large-scale video data can be realized, redundant calculation is reduced, and video retrieval precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of text video retrieval, and relates to a text video retrieval method based on the perception of spatiotemporal feature importance. Background Art

[0002] Text-video retrieval, a key task in multimodal learning, aims to achieve semantic matching between text and video. With the rapid development of internet video platforms and short video applications, users are increasingly demanding accurate and fast video retrieval. Text-video retrieval technology can help users quickly find semantically relevant video content by entering a text query, significantly improving the efficiency of video content acquisition. However, despite its enormous potential, text-video retrieval technology still faces many challenges. In particular, the inherent modality differences between text and video, as two different modalities of information, make direct text-video matching very complex. This modality gap, often referred to as the "modality gap," complicates the process of associating multimodal data. Text and video have unique expression forms in the language and vision domains, respectively. Text describes semantic information through text, while video conveys information through visual and audio elements. To achieve accurate text-video matching, it is necessary to overcome the semantic gap between modalities so that the model can effectively understand and integrate multimodal information from text and video, thereby improving retrieval accuracy and efficiency.

[0003] From cross-modal pre-training models to text-video retrieval (TVR) tasks, research has gradually evolved into a variety of methods for semantic alignment between text and video. In "Expectation maximization contrastive learning for compact video-and language representations," Jin et al. attempt to globally align sentences and videos by encoding text or video data into a single embedding space. However, these methods ignore the diversity of video content. In particular, when a video contains multiple keyframes or spatiotemporal regions, a single global alignment is insufficient to capture the rich visual information. To address this issue, Liu et al. introduced frame-level or segment-level matching in "Ts2-net: Token shift and selection transformer for text-video retrieval" to account for the varying importance and diversity of video content. These methods encode videos as a single semantic content and perform a one-to-one matching with text, but this approach still struggles to cope with the complex scenes and details in videos. Videos are rich in visual elements, while text descriptions are typically only relevant to specific spatiotemporal portions of the video. Therefore, simple global matching cannot effectively capture fine-grained semantic alignment. In "Text-adaptive multiple visual prototype matching for video-text retrieval", Lin et al. proposed to solve the dynamic matching problem by dividing the video into multiple semantic events so that different text descriptions can be adaptively matched with event prototypes. However, existing event prototypes still find it difficult to fully capture the relationship between fine-grained objects and content words, and often lack perception of local details. Summary of the Invention

[0004] In light of this, the present invention aims to provide a text-video retrieval method that is aware of the importance of spatiotemporal features. This method emphasizes fine-grained local spatial matching to construct an effective event-to-sentence matching method, ensuring that the relationship between important objects in the video and their textual descriptions is accurately captured. To this end, the generation of event prototypes should be based on the semantic information of important objects. As the event prototypes are gradually constructed, the model is guided to learn local information, thereby enhancing the perception of details and matching accuracy.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A text-video retrieval method based on the importance of spatiotemporal features, comprising the following steps:

[0007] S1. Extract features from the input video and query text, extracting different frame features, global frame features, and text features;

[0008] S2. Using a source-channel joint coding method based on semantic communication to transmit the semantic information of the query text;

[0009] S3, for different frames in the video features, extract key frames using the importance perception mechanism;

[0010] S4, generating object prototypes based on the image block features in each key frame, selecting object features with semantic information and matching them with phrases in the semantic information of the query text;

[0011] S5. Aggregate different objects into frame events and use the temporal association between frames to match frame time with event sentences;

[0012] S6, using the symmetric cross entropy loss function to optimize the prototype matching within the batch,

[0013] Furthermore, in step S1, the CLIP pre-trained model is used to extract features from the videos and text descriptions in the dataset:

[0014]

[0015] in is the global frame feature, Represents the features of the 1st to Lth video frames, K is the number of image blocks; Represents the features of the 1st to Mth words, Y S and Y E is the global text feature at the beginning and end of the text, M and D are the number of words and dimensions respectively, and L is the number of frames.

[0016] Furthermore, in step S2, the query text transmission process is as follows:

[0017] S21, transmit a user input query Y with M words T , the transmitter will input the query text Y T The complex symbol vector e after mapping to the joint source channel JSCC encoder E is expressed as:

[0018]

[0019] Where s is the dimension of e, is a set of complex numbers, θ0 is the parameter of JSCC encoder E;

[0020] S22, the encoded symbol vector e is transmitted through the noisy additive white Gaussian noise (AWGN) channel, which is expressed as:

[0021]

[0022] Where N is the channel noise, the noise is given by Sampling is obtained; σ 2 is the noise power, is a complex Gaussian distribution;

[0023] S23, the receiver maps e′ to the reconstructed feature Y through the JSCC decoder D T ′, which is expressed as:

[0024] Y T ′=D(e′,τ)=D(E(Y T ,θ0)+N,τ)

[0025] The reconstructed feature Y T ′ is the original query text feature Y T The estimated value of , τ is the parameter of the joint source channel decoder D;

[0026] S24, finally use Y T With Y T The feature difference of ′ is used as the loss function to train the joint source channel codec network, which is expressed as:

[0027]

[0028] By training the loss function Make the semantic feature Y after transmission T ′ and Y T The loss between them is minimal, thus ensuring the transmission of query semantic information.

[0029] Furthermore, in step S3, the process of extracting the video frame feature information is as follows:

[0030] S31, extracting the video frame feature information X i , as the input for semantic weight calculation, is expressed as:

[0031]

[0032] where f L represents the Lth video frame feature, θ1 represents the fixed parameters of the feature extraction network, Q, H, and L represent the width, height, and number of video frame features, respectively;

[0033] The feature semantic weights are obtained using the gradient-based semantic importance weight module:

[0034]

[0035] in To represent the video frame feature fL The combined impact on all perception results x; the superscript "c" represents the perception result index;

[0036] S32. Calculate importance weight The average of all perception results:

[0037]

[0038] w L (L∈{1,2,...,L}) represents the L-th video frame feature map f L The semantic importance of

[0039] S33. Map the weight to a preset magnitude:

[0040] W′=r×SoftMax(τ×W)

[0041] Where τ is a hyperparameter and r is a constant; the semantic weight W′ is applied to the semantic level loss function, and the hyperparameter τ controls the distribution of the semantic weight W′;

[0042] S34, represent each frame feature in the video f l Apply weight W′ to calculate weighted similarity score S L :

[0043]

[0044] where f C is the global feature of the entire video;

[0045] S35. Finally, the first n frames are selected as key frames based on the weighted similarity score:

[0046]

[0047] Furthermore, in step S4, image block-object and word-phrase aggregation is performed in the spatial dimension. By generating spatial objects, key local details of the video are revealed. Subsequently, a fine-grained prototype alignment mechanism is introduced to accurately match spatial objects with text phrases, which includes:

[0048] S41, aggregate image block features into object prototypes to represent fine-grained spatial information, and use two FC layers and ReLU functions to predict sparse weights Object prototype Expressed as:

[0049]

[0050] where N o is the number of object prototypes, K is the number of image blocks;

[0051] S42, calculate the maximum similarity between the object and the phrase prototype in each frame, and obtain the overall similarity standard s by summing the matching scores of all target phrases op :

[0052]

[0053] where N o is the number of object prototypes, N p is the number of phrase prototypes, and L is the number of video frames.

[0054] Furthermore, in step S5, key fine-grained objects and dynamic time changes are integrated into different event prototypes to complete event sentence matching, which includes the following steps:

[0055] S51. Design a frame decoder to convert all object prototypes Merge into frame-level prototype middle:

[0056]

[0057] in, is the frame query, K o and V o For the object prototype P o Features after linear transformation;

[0058] S52, introduce shielded attention mechanism, attention mask for:

[0059]

[0060] S53, add frame prototype and the original global features of the corresponding frames To enhance the robustness of the model:

[0061]

[0062] S54, develop dynamic event decoder, learn P f The relationship between frames in the event prototype is obtained Among them, dynamic attention is expressed as:

[0063]

[0064] Where, For event query, K f 、V f For the framework prototype P f Linear transformation characteristics of ;

[0065] S55, by directly using the global text to represent y E As a sentence prototype, and event prototype P e For alignment, the event sentence prototype matching process is expressed as:

[0066]

[0067] Among them, s es is the similarity score after the two are matched, Represents the prototype of the i-th event, N e Indicates the number of event prototypes.

[0068] Furthermore, in step S6, a symmetric cross-entropy loss function is used to optimize the prototype matching within the batch, considering the corresponding text-video pair as a positive example and all other paired combinations in the batch as negative examples, then:

[0069]

[0070] Where S op and S es are the similarity matrices of the object phrase and event sentence prototype matching when the batch size is B, is the loss function for text-to-video retrieval, is the loss function of the video-to-text retrieval process, and:

[0071]

[0072] Where σ is a learnable hyperparameter, S ij or S ji is the similarity score between the two modalities.

[0073] The beneficial effects of the present invention are:

[0074] The present invention designs a text-video retrieval method based on the importance perception of spatiotemporal features. Through the combination of object phrase matching and event sentence matching, it can accurately capture the fine-grained semantic relationship between video and text, and improve the matching accuracy. Secondly, the importance perception mechanism effectively weights spatiotemporal features, filters redundant information, and ensures that the retrieval process focuses on key information related to the task. It adopts a joint source channel coding method based on semantic communication to ensure the transmission of query semantics, optimize feature utilization and improve retrieval efficiency. In addition, the system enhances its adaptability to complex video content and diversified text descriptions by dynamically learning event prototypes, and improves the flexibility and accuracy of matching. At the same time, it reduces redundant calculations, making the system run more efficiently. This method is widely applicable to application scenarios such as short video platforms, intelligent recommendation systems, and intelligent monitoring, improves user experience and information acquisition efficiency, has strong cross-modal retrieval capabilities, and has broad application prospects in multimodal information retrieval and intelligent systems in the future.

[0075] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0077] Figure 1 Schematic diagram of the overall process of the text video retrieval method of the present invention;

[0078] Figure 2 Schematic diagram of the overall architecture of the text video retrieval method of the present invention;

[0079] Figure 3 Schematic diagram of the process of fine-grained matching of spatiotemporal features of the present invention. DETAILED DESCRIPTION

[0080] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0081] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0082] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0083] See also Figures 1 to 3 , which is a text-video retrieval method based on the importance of spatiotemporal features.

[0084] Example

[0085] A text-video retrieval method based on the importance of spatiotemporal features. Figure 1 As shown, the method includes: extracting features from each frame and text description in the video respectively, selecting key frames in the video through an importance perception mechanism, and maintaining the semantic information of the query text during transmission by using a source-channel joint coding method based on semantic communication. Aggregating the image block features of each frame into an object prototype, matching the object with a phrase in the text description, and perceiving the spatial information in the video frame. By combining different objects into different event prototypes, matching the event prototype with the entire text description, and utilizing the associated time information between frames to improve the fine-grained matching between text and video, the accuracy is improved. By optimizing the spatiotemporal feature processing and feature importance evaluation of video content, the present invention can achieve efficient retrieval in large-scale video data, reduce redundant calculations, and improve video retrieval accuracy.

[0086] In this embodiment, Figure 2 As shown, the method specifically includes the following steps:

[0087] Step 1: Given a video and its corresponding T text descriptions The goal of text-video retrieval is to learn a similarity function to efficiently measure the similarity between different modalities. Through this function, all videos in the dataset can be sorted according to the similarity score based on the text query. First, for a given input video v i , extract sequence features from L frames, use CLIP pre-training model, for each frame in is the global frame feature, K is the number of image blocks. For each query text t i , add [SOT] and [EOT] tags as the start and end of the query text. The output text features can be defined as where Y S and Y E is the global text feature at the beginning and end of the text, M and D are the number of words and dimension respectively.

[0088] Step 2: The text and video retrieval task needs to consider some environmental interference factors in real scenes. If the surrounding signal interference is serious, the user will face significant challenges in transmitting semantic information. Therefore, the present invention adopts a joint source channel coding method based on semantic communication to ensure the stability of the user sending semantic information to the receiving end. Specifically, first transmit a user input query Y with M words T , the transmitter will input the query text Y T The complex symbol vector e after mapping to the JSCC (Joint Source Channel Coding) encoder E can be expressed as:

[0089]

[0090] Where s is the dimension of e, is a set of complex numbers, and θ0 is the parameter of the JSCC encoder E.

[0091] Step 2.1: The encoded symbol vector e is transmitted through the noisy AWGN channel and can be expressed as:

[0092]

[0093] Where N is the channel noise, the noise is given by Sampling is obtained; σ 2 is the noise power, is a complex Gaussian distribution.

[0094] Step 2.2: The receiver maps e′ to the reconstructed feature Y through the JSCC decoder D T ′, can be expressed as

[0095] Y T ′=D(e′,τ)=D(E(Y T,θ0)+N,τ)

[0096] The reconstructed feature Y T ′ is the original query text feature Y T The estimated value of τ is the parameter of JSCC decoder D.

[0097] Step 2.3: Using Y T With Y T The feature difference of ′ is used as the loss function to train the joint source channel codec network, which can be expressed as:

[0098]

[0099] By training the loss function Make the semantic feature Y after transmission T ′ and Y T The loss between them is minimal, thus ensuring the transmission of query semantic information.

[0100] Step 3: Given that different video frame features have different importance, an importance perception mechanism is adopted to screen out features that are meaningful for cross-modal reasoning and filter out those that are not important. First, the feature semantic weight is obtained through the gradient-based semantic importance weight module to measure the semantic importance. Specifically, first, the extracted video frame feature information X i , as the input for semantic weight calculation, can be expressed as

[0101]

[0102] where f L represents the Lth video frame feature, θ1 represents the fixed parameters of the feature extraction network, Q, H, and L represent the width, height, and number of video frame features, respectively.

[0103] Step 3.1: Use the gradient of the perception result relative to the feature-level information to quantify the semantic importance, the c-th perception result x c To the Lth feature map f L The gradient of can be written as The average value of the width Q and height H dimensions can be expressed as:

[0104]

[0105] in To represent the video frame feature f L The combined impact of all perception results x is calculated The average value of all perception results can be expressed as:

[0106]

[0107] Then the semantic weight vector W={w1,w2,...,w L}, and w L (L∈{1,2,…,L}) represents the L-th video frame feature map f L semantic importance.

[0108] Step 3.2: Since the value of W is too small, it cannot be used directly in the loss function as it may lead to slow convergence. Therefore, using the parameters τ and r, map W to W′:

[0109] W′=r×SoftMax(τ×W)

[0110] Where τ is a hyperparameter and r is a constant. The semantic weight W′ is applied to the semantic-level loss function. The hyperparameter τ controls the distribution of the semantic weight W′, while the constant r keeps the final semantic-level loss at a reasonable level without affecting gradient updates.

[0111] For each frame in the video, the feature representation f l Apply weight W′ to calculate weighted similarity score S L :

[0112]

[0113] where f C is the global feature of the entire video.

[0114] Finally, the first n frames are selected as key frames based on the weighted similarity score:

[0115]

[0116] Step 4: In text-video matching, text usually subjectively describes an event consisting of different objects, actions, and temporal activities. These events run through the entire video sequence, resulting in partial matching between the video and text. To solve this problem, we first perform image block-object and word-phrase aggregation in the spatial dimension, and generate spatial objects to reveal the key local details of the video. Then, we introduce a fine-grained prototype alignment mechanism to accurately match spatial objects with text phrases, fully mining and utilizing the spatial information of the video. Figure 3 As shown, it specifically includes the following steps:

[0117] Step 4.1: For the entire video feature, first aggregate the image block features into object prototypes to represent fine-grained spatial information, such as people, objects, and certain areas in the video. During the object prototype generation process, some background features may interfere with cross-modal alignment. Therefore, object-level prototypes are generated in a sparse aggregation manner and redundant information is filtered out. Two FC layers and ReLU functions are used to predict sparse weights. where N o is the number of object prototypes, and K is the number of image patches.

[0118] This prevents the object prototype from being affected by the redundant image patch background. Spatial prototype generation can be defined as:

[0119]

[0120] For text, we also use the spatial object prototype generation mechanism and design a similar network structure to aggregate word features to generate phrase prototypes Important spatial information is explored through fine-grained image patch features and word features, and then the model is optimized through spatial object-phrase matching.

[0121] Step 4.2: Calculate the maximum similarity of the object phrase prototypes within each frame. This associates the most similar phrase prototype with each object prototype, reflecting fine-grained matching across modalities. Subsequently, by processing the similarity matrix of the multi-frame target, the maximum similarity score across the frame sequence is extracted to generate an object-phrase matching probability with higher confidence. Finally, by summing all target phrase matching scores, the overall similarity criterion s is obtained. op , the interactive process of object phrase matching is defined as:

[0122]

[0123] where N o is the number of object prototypes, N p is the number of phrase prototypes, and L is the number of video frames.

[0124] Step 5: In the event-sentence matching method, key fine-grained objects and dynamic temporal changes are incorporated into different event prototypes. First, dynamic temporal matching is modeled and a progressive object-event prototype aggregation strategy is proposed to reveal the semantic diversity of the video. Subsequently, a dynamic prototype matching mechanism is introduced to more effectively capture semantic associations across time. The core idea is to gradually aggregate spatial object prototypes into frame-level prototypes, then generate diverse event prototypes through inter-frame interactions, thereby comprehensively capturing the semantic details and dynamic changes of the video.

[0125] Step 5.1: First, a frame decoder is designed to convert all object prototypes into Merge into frame-level prototype In order to learn the object relationship between different frames, a masked attention mechanism is introduced:

[0126]

[0127] in, is the frame query (randomly initialized learnable feature vector), K o and V o For the object prototype P o Features after linear transformation. Attention mask for:

[0128]

[0129] Add frame prototype and the original global features of the corresponding frames To enhance the robustness of the model:

[0130]

[0131] Step 5.2: Develop dynamic event decoder and learn P f The relationship between frames in the event prototype is obtained To illustrate the rich information of the video. Dynamic attention is expressed as:

[0132]

[0133] Where, For event query, K f 、V f is the frame prototype P f Linear transformation characteristics.

[0134] Step 5.3: Each event query can adaptively focus on the video frame prototype, and the synergy of multiple queries implicitly ensures the diversity of event semantics. In order to take into account the fact that the same video may correspond to multiple textual semantic descriptions, by directly using the global text representation y E As a sentence prototype, and event prototype P e Alignment is performed to improve the flexibility and accuracy of cross-modal matching. The event sentence prototype matching process can be expressed as:

[0135]

[0136] Step 6: A symmetric cross-entropy loss function is used to optimize the prototype matching within the batch, treating the corresponding text-video pairs as positive examples and all other paired combinations in the batch as negative examples, thereby enhancing the model's discriminative ability for matching accuracy.

[0137]

[0138] S op and S es are the similarity matrices of the object phrase and event sentence prototype matching when the batch size is B,

[0139]

[0140] Where σ is a learnable hyperparameter, S ij or S ji is the similarity score between the two modalities, is the loss function for text-to-video retrieval, is the loss function for the video-to-text retrieval process.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A text-video retrieval method based on the importance of spatiotemporal features, characterized by: The method comprises the following steps: S1. Extract features from the input video and query text, extracting different frame features, global frame features, and text features; S2. Using a source-channel joint coding method based on semantic communication to transmit the semantic information of the query text; S3, for different frames in the video features, extract key frames using the importance perception mechanism; In step S3, the process of extracting the video frame feature information is as follows: S31, extracting the video frame feature information , as the input for semantic weight calculation, is expressed as: in represents the Lth video frame feature, represents the fixed parameters of the feature extraction network, Represent the width, height and number of video frame features respectively; The semantic importance is quantified using the gradient of the perception result relative to the feature-level information. The c-th perception result To the Lth feature map The gradient of , using the gradient-based semantic importance weight module to obtain the feature semantic weight: in To represent the video frame features For all perception results The combined impact of ” indicates the perception result index; S32. Calculate importance weight The average of all perception results: Represents the Lth video frame feature map The semantic importance of S33. Map the weight to a preset magnitude: Where τ is a hyperparameter, r is a constant; semantic weight Applicable to semantic level loss function, the hyperparameter τ controls the semantic weight distribution of S34. Feature representation of each frame in the video Apply weights Calculating weighted similarity scores : in is the global feature of the entire video; S35. Finally, the first n frames are selected as key frames based on the weighted similarity score: S4, generating object prototypes based on the image block features in each key frame, selecting object features with semantic information and matching them with phrases in the semantic information of the query text; S5. Aggregate different objects into frame events and use the temporal association between frames to match frame time with event sentences; S6. Use the symmetric cross entropy loss function to optimize the prototype matching within the batch and use it for text video retrieval after training.

2. The text-video retrieval method based on temporal-spatial feature importance perception according to claim 1, characterized in that: In step S1, the CLIP pre-trained model is used to extract features from the videos and text descriptions in the dataset: in is the global frame feature, Represents the features of the 1st to Lth video frames, is the number of image blocks; Represents the features of the 1st to Mth words, and are global text features for the start and end of text, and are the number of words and dimensions respectively, is the number of frames.

3. The text-video retrieval method based on temporal-spatial feature importance perception according to claim 2, characterized in that: In step S2, the query text transmission process is as follows: S21, transmit a user input query with M words , the transmitter will enter the query text The complex symbol vector e after mapping to the joint source channel JSCC encoder E is expressed as: Where s is the dimension of e, is a complex set, are the parameters of JSCC encoder E; S22, the encoded symbol vector e is transmitted through the noisy additive white Gaussian noise (AWGN) channel, which is expressed as: in is the channel noise, which is given by Sampling obtained; is the noise power, is a complex Gaussian distribution; S23, receiver through JSCC decoder DC mapping To reconstruct features , which is expressed as: The reconstruction features is the original query text feature The estimated value of are the parameters of the joint source-channel decoder DC; S24, last use and The feature level difference is used as the loss function to train the joint source channel codec network, which is expressed as: By training the loss function The semantic features after transmission and The loss between them is minimal, thus ensuring the transmission of query semantic information.

4. The text-video retrieval method based on temporal-spatial feature importance perception according to claim 3, characterized in that: In step S4, it includes: S41, aggregate image block features into object prototypes to represent fine-grained spatial information, and use two FC layers and ReLU functions to predict sparse weights , object prototype Expressed as: in is the number of object prototypes, K is the number of image blocks; S42, calculate the maximum similarity between the object and the phrase prototype in each frame, and obtain the overall similarity standard by summing up the matching scores of all target phrases : in is the number of object prototypes, is the number of phrase prototypes, The number of video frames.

5. The text-video retrieval method based on temporal-spatial feature importance perception according to claim 4, characterized in that: In step S5, it includes the following steps: S51. Design a frame decoder to convert all object prototypes Merge into frame-level prototype middle: in, For frame query, and For object prototype Features after linear transformation; S52, introduce shielded attention mechanism, attention mask for: S53, add frame prototype and the original global features of the corresponding frames To enhance the robustness of the model: S54, develop dynamic event decoder, learn The relationship between frames in the event prototype is obtained , where dynamic attention is expressed as: Where, For event query, 、 For the framework prototype Linear transformation characteristics of ; S55, by directly using the global text representation As sentence prototypes, and event prototypes For alignment, the event sentence prototype matching process is expressed as: in, is the similarity score after the two are matched, represents the prototype of the i-th event, Indicates the number of event prototypes.

6. The text-video retrieval method based on temporal-spatial feature importance perception according to claim 5, characterized in that: In step S6, a symmetric cross-entropy loss function is used to optimize the prototype matching within the batch, considering the corresponding text-video pairs as positive examples and all other paired combinations in the batch as negative examples, then: In the formula and are the similarity matrices of the object phrase and event sentence prototype matching when the batch size is B, is the loss function for text-to-video retrieval, is the loss function of the video-to-text retrieval process, and: in is a learnable hyperparameter, or is the similarity score between the two modalities.

Images