Text video retrieval method for perceiving importance of spatio-temporal characteristics

Abstract

The invention relates to a text video retrieval method for perceiving importance of spatio-temporal characteristics, and belongs to the technical field of text video retrieval. The method specifically comprises the steps that feature extraction is conducted on each frame and text description in a video, key frames in the video are selected through an importance perception mechanism, and semantic information of a query text in the transmission process is kept through an information source channel joint coding method based on semantic communication. And aggregating image block features of each frame into an object prototype, matching an object with phrases in text description, and sensing spatial information in a video frame. According to the method, different objects are combined into different event prototypes, the event prototypes are matched with the whole text description, the fine-grained matching between the text and the video is improved by utilizing the association time information between frames, and the content of the video is optimized by optimizing the spatial-temporal feature processing and feature importance evaluation of the video content. Efficient retrieval can be achieved in large-scale video data, redundant calculation is reduced, and video retrieval precision is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of text video retrieval, and relates to a text video retrieval method based on temporal and spatial feature importance perception. Background Art

[0002] As an important task of multimodal learning, text-video retrieval aims to achieve semantic matching between text and video. With the rapid development of Internet video platforms and short video applications, users have an increasing demand for accurate and fast video retrieval. Text-video retrieval technology can help users quickly find semantically related video content by entering text queries, greatly improving the efficiency of video content acquisition. However, despite its huge application potential, text-video retrieval technology still faces many challenges. In particular, as two different modal information, the essential modal difference between text and video makes it very complicated to directly match text and video. This difference between modalities, often referred to as the "modal gap", increases the difficulty of associating multimodal data. Text and video have unique expressions in the language and visual fields, respectively. Text describes semantic information through text, while video conveys information through visual and sound elements. To achieve accurate text-video matching, it is necessary to overcome the semantic gap between modalities so that the model can effectively understand and integrate multimodal information from text and video, thereby improving the accuracy and efficiency of retrieval.

[0003] From cross-modal pre-training models to text-video retrieval (TVR) tasks, research has gradually evolved a variety of methods to achieve semantic matching between text and video. Jin et al. in "Expectation maximization contrastive learning for compact video-and language representations." attempted to align sentences and videos at a global level by encoding text or video data into a single embedding space. However, these methods ignore the diversity of video content, especially when the video contains multiple key frames or spatiotemporal regions, a single global alignment is not enough to capture the rich visual information in the video. To address this problem, Liu et al. introduced frame-level or segment-level matching in "Ts2-net: Token shift and selection transformer for text-video retrieval" to consider the different importance and diversity of video content. These methods encode videos into a single semantic content and match them one-to-one with text, but this method still has difficulty dealing with complex scenes and details in videos. The visual elements in the video are very rich, while the text description is usually only related to a specific spatiotemporal part of the video. Therefore, simple global matching cannot effectively capture fine-grained semantic alignment. Lin et al. proposed in "Text-adaptive multiple visual prototype matching for video-text retrieval" to solve the dynamic matching problem by dividing the video into multiple semantic events so that different text descriptions can be adaptively matched with event prototypes. However, existing event prototypes still find it difficult to fully capture the relationship between fine-grained objects and content words, and often lack perception of local details. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a text-video retrieval method based on the perception of spatiotemporal feature importance. It emphasizes fine-grained local spatial matching by constructing an effective event-sentence matching method to ensure that the relationship between important objects in the video and the text description is accurately captured. To this end, the generation of event prototypes should be based on the semantic information of important objects, and when gradually constructing event prototypes, the model should be guided to learn local information, thereby enhancing the perception of details and matching accuracy.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A text and video retrieval method based on the importance of spatiotemporal features, the method comprising the following steps:

[0007] S1. Extract features from the input video and query text, extract different frame features, global frame features and text features;

[0008] S2, using a source-channel joint coding method based on semantic communication to transmit the semantic information of the query text;

[0009] S3, for different frames of video features, extract key frames using importance perception mechanism;

[0010] S4, generating an object prototype for the image block features in each key frame, selecting object features with semantic information and matching them with phrases in the semantic information of the query text;

[0011] S5, aggregate different objects into frame events, and use the time association between frames to match the frame time with the event sentence;

[0012] S6, using the symmetric cross entropy loss function to optimize the prototype matching within the batch.

[0013] Furthermore, in step S1, the CLIP pre-trained model is used to extract features from the videos and text descriptions in the dataset:

[0014]

[0015] in is the global frame feature, Represents the features of the 1st to Lth video frames, K is the number of image blocks; Represents the features of the 1st to Mth words, Y S and Y E is the global text feature at the beginning and end of the text, M and D are the number of words and dimensions respectively, and L is the number of frames.

[0016] Further, in step S2, the transmission process of the query text is:

[0017] S21, transmitting a user input query Y with M words T , the transmitter will input the query text Y T The complex valued symbol vector e after being mapped to the joint source channel JSCC encoder E is expressed as:

[0018]

[0019] Where s is the dimension of e, is a set of complex numbers, θ0 is the parameter of JSCC encoder E;

[0020] S22, the encoded symbol vector e is transmitted through the noisy additive white Gaussian noise AWGN channel, which is expressed as:

[0021]

[0022] Where N is the channel noise, the noise is given by Sampling obtained; σ 2 is the noise power, is a complex Gaussian distribution;

[0023] S23, the receiver maps e′ to the reconstructed feature Y through the JSCC decoder D T ′, which is expressed as:

[0024] Y T ′=D(e′,τ)=D(E(Y T ,θ0)+N,τ)

[0025] The reconstructed feature Y T ′ is the original query text feature Y T The estimated value of , τ is the parameter of the joint source channel decoder D;

[0026] S24, finally use Y T With Y T The feature difference of ′ is used as the loss function to train the joint source channel codec network, which is expressed as:

[0027]

[0028] Through the training loss function Let the semantic feature Y after transmission T ′ and Y T The loss between them is minimal, thus ensuring the transmission of query semantic information.

[0029] Further, in step S3, the process of extracting the video frame feature information is as follows:

[0030] S31, extracting the video frame feature information X i , as the input for semantic weight calculation, is expressed as:

[0031]

[0032] where f L represents the Lth video frame feature, θ1 represents the fixed parameters of the feature extraction network, Q, H, and L represent the width, height, and number of video frame features, respectively;

[0033] The feature semantic weights are obtained using the gradient-based semantic importance weight module:

[0034]

[0035] in To represent the video frame feature fL The combined impact on all perception results x; the superscript "c" represents the perception result index;

[0036] S32. Calculate importance weights The average of all perception results:

[0037]

[0038] w L (L∈{1,2,...,L}) represents the Lth video frame feature map f L The semantic importance of

[0039] S33, mapping the weight to a preset magnitude:

[0040] W′=r×SoftMax(τ×W)

[0041] Where τ is a hyperparameter and r is a constant; the semantic weight W′ is applicable to the semantic level loss function, and the hyperparameter τ controls the distribution of the semantic weight W′;

[0042] S34, represent each frame feature in the video f l Apply weight W′ to calculate weighted similarity score S L :

[0043]

[0044] where f C is the global feature of the entire video;

[0045] S35. Finally, the first n frames are selected as key frames according to the weighted similarity score:

[0046]

[0047] Furthermore, in step S4, image block-object and word-phrase aggregation is performed in the spatial dimension, and key local details of the video are revealed by generating spatial objects. Then, a fine-grained prototype alignment mechanism is introduced to accurately match spatial objects with text phrases, which includes:

[0048] S41, aggregate image patch features into object prototypes to represent fine-grained spatial information, and use two FC layers and ReLU functions to predict sparse weights Object prototype It is expressed as:

[0049]

[0050] Where N o is the number of object prototypes, K is the number of image patches;

[0051] S42, calculate the maximum similarity between the object and the phrase prototype in each frame, and obtain the overall similarity standard s by summing up the matching scores of all target phrases op :

[0052]

[0053] Where N o is the number of object prototypes, N p is the number of phrase prototypes, and L is the number of video frames.

[0054] Further, in step S5, key fine-grained objects and dynamic time changes are integrated into different event prototypes to complete event sentence matching, which includes the following steps:

[0055] S51. Design a frame decoder to convert all object prototypes Merge to frame-level prototype middle:

[0056]

[0057] in, is the frame query, K o and V o For the object prototype P o Features after linear transformation;

[0058] S52, introduce shielded attention mechanism, attention mask for:

[0059]

[0060] S53, add frame prototype and the original global features of the corresponding frames To enhance the robustness of the model:

[0061]

[0062] S54, develop dynamic event decoder, learn P f The relationship between frames in the event prototype Among them, dynamic attention is expressed as:

[0063]

[0064] In the formula, is the event query, K f 、V f For the framework prototype P f Linear transformation characteristics of ;

[0065] S55, by directly using the global text to represent y E As a sentence prototype, and event prototype P e Alignment is performed, and the event sentence prototype matching process is expressed as:

[0066]

[0067] Among them, s es is the similarity score after the two are matched, represents the i-th event prototype, N e Indicates the number of event prototypes.

[0068] Furthermore, in step S6, a symmetric cross entropy loss function is used to optimize the prototype matching within the batch, and the corresponding text-video pairs are regarded as positive examples, while all other paired combinations in the batch are regarded as negative examples, then:

[0069]

[0070] Where S op and S es They are the similarity matrices of the object phrase and event sentence prototype matching when the batch size is B, is the loss function for text-to-video retrieval, is the loss function of the video to text retrieval process, and:

[0071]

[0072] Where σ is a learnable hyperparameter, S ij or S ji is the similarity score between the two modalities.

[0073] The beneficial effects of the present invention are:

[0074] The present invention designs a text-video retrieval method based on the importance perception of spatiotemporal features. Through the combination of object phrase matching and event sentence matching, it can accurately capture the fine-grained semantic relationship between video and text and improve the matching accuracy. Secondly, the importance perception mechanism effectively weights spatiotemporal features, filters redundant information, ensures that the key information related to the task is focused on during the retrieval process, and adopts a joint source channel coding method based on semantic communication to ensure the transmission of query semantics, optimize feature utilization and improve retrieval efficiency. In addition, the system enhances the adaptability to complex video content and diversified text descriptions by dynamically learning event prototypes, and improves the flexibility and accuracy of matching. At the same time, redundant calculations are reduced, making the system run more efficiently. This method is widely applicable to application scenarios such as short video platforms, intelligent recommendation systems, and intelligent monitoring, improves user experience and information acquisition efficiency, has strong cross-modal retrieval capabilities, and has broad application prospects in multimodal information retrieval and intelligent systems in the future.

[0075] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0077] Figure 1 It is a schematic diagram of the overall process of the text video retrieval method of the present invention;

[0078] Figure 2 It is a schematic diagram of the overall architecture of the text video retrieval method of the present invention;

[0079] Figure 3 It is a schematic diagram of the process of fine-grained matching of spatiotemporal features of the present invention. DETAILED DESCRIPTION

[0080] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0081] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0082] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0083] See also Figure 1 to Figure 3 , which is a text-video retrieval method based on the importance of spatiotemporal features.

[0084] Example

[0085] A text-video retrieval method based on the importance of spatiotemporal features. Figure 1 As shown, the method includes: extracting features from each frame and text description in the video, selecting key frames in the video through an importance perception mechanism, and maintaining the semantic information of the query text during transmission using a source-channel joint encoding method based on semantic communication. Aggregating the image block features of each frame into an object prototype, matching the object with phrases in the text description, and perceiving the spatial information in the video frame. By combining different objects into different event prototypes, matching the event prototype with the entire text description, and utilizing the associated time information between frames to improve the fine-grained matching between text and video, the accuracy is improved. The present invention can achieve efficient retrieval in large-scale video data, reduce redundant calculations, and improve video retrieval accuracy by optimizing the spatiotemporal feature processing and feature importance evaluation of video content.

[0086] In this embodiment, Figure 2 As shown, the method specifically comprises the following steps:

[0087] Step 1: Given a set of V videos and its corresponding T text descriptions The goal of text-to-video retrieval is to learn a similarity function to efficiently measure the similarity between different modalities. Through this function, all videos in the dataset can be sorted according to the similarity scores based on the text query. First, for a given input video v i , extract sequence features from L frames, use the CLIP pre-training model, and for each frame in is the global frame feature, K is the number of image blocks. For each query text t i , add [SOT] and [EOT] tags as the start and end of the query text. The output text features can be defined as where Y S and Y E is the global text feature at the beginning and end of the text, M and D are the number of words and dimension respectively.

[0088] Step 2: The text and video retrieval task needs to consider some environmental interference factors in real scenes. If the surrounding signal interference is serious, the user will face major challenges in transmitting semantic information. Therefore, the present invention adopts a joint source channel coding method based on semantic communication to ensure the stability of the user sending semantic information to the receiving end. Specifically, first transmit a user input query Y with M words T , the transmitter will input the query text Y T The complex valued symbol vector e after being mapped to the JSCC (Joint Source Channel Coding) encoder E can be expressed as:

[0089]

[0090] Where s is the dimension of e, is a set of complex numbers, and θ0 is the parameter of the JSCC encoder E.

[0091] Step 2.1: The encoded symbol vector e is transmitted through a noisy AWGN channel and can be expressed as:

[0092]

[0093] Where N is the channel noise, the noise is given by Sampling obtained; σ 2 is the noise power, is a complex Gaussian distribution.

[0094] Step 2.2: The receiver maps e′ to the reconstructed feature Y through the JSCC decoder D T ′, can be expressed as

[0095] Y T ′=D(e′,τ)=D(E(Y T,θ0)+N,τ)

[0096] The reconstructed feature Y T ′ is the original query text feature Y T is the estimated value of , and τ is the parameter of the JSCC decoder D.

[0097] Step 2.3: Use Y T With Y T The feature difference of ′ is used as the loss function to train the joint source channel codec network, which can be expressed as:

[0098]

[0099] Through the training loss function Let the semantic feature Y after transmission T ′ and Y T The loss between them is minimal, thus ensuring the transmission of query semantic information.

[0100] Step 3: Given that different video frame features have different importance, an importance perception mechanism is used to filter out features that are meaningful for cross-modal reasoning and filter out unimportant features. First, the feature semantic weight is obtained through a gradient-based semantic importance weight module to measure the importance of semantics. Specifically, first, the extracted video frame feature information X i , as the input for semantic weight calculation, can be expressed as

[0101]

[0102] where f L represents the Lth video frame feature, θ1 represents the fixed parameters of the feature extraction network, and Q, H, and L represent the width, height, and number of video frame features, respectively.

[0103] Step 3.1: Use the gradient of the perception result with respect to the feature-level information to quantify the semantic importance. The c-th perception result x c To the Lth feature map f L The gradient of can be written as The average value of the width Q and height H dimensions can be expressed as:

[0104]

[0105] in To represent the video frame feature f L The combined impact of all perceived results x is calculated The average value of all perception results can be expressed as:

[0106]

[0107] Then the semantic weight vector W = {w1,w2,...,w L}, and w L (L∈{1,2,…,L}) represents the Lth video frame feature map f L semantic importance.

[0108] Step 3.2: Since the value of W is too small, it cannot be directly used in the loss function as it may lead to slow convergence. Therefore, using the parameters τ and r, W is mapped to W′:

[0109] W′=r×SoftMax(τ×W)

[0110] Where τ is a hyperparameter and r is a constant. The semantic weight W′ is applicable to the semantic-level loss function. The hyperparameter τ can control the distribution of the semantic weight W′, and the constant r can make the final semantic-level loss value at a reasonable level without affecting the gradient update.

[0111] For each frame in the video, the feature representation f l Apply weight W′ to calculate weighted similarity score S L :

[0112]

[0113] where f C is the global feature of the entire video.

[0114] Finally, the first n frames are selected as key frames based on the weighted similarity score:

[0115]

[0116] Step 4: In text-video matching, the text usually subjectively describes an event consisting of different objects, actions, and time activities. These events run through the entire video sequence, resulting in partial matching between the video and the text. To solve this problem, we first aggregate image blocks-objects and word-phrases in the spatial dimension, generate spatial objects, and reveal the key local details of the video. Then, we introduce a fine-grained prototype alignment mechanism to accurately match spatial objects with text phrases, and fully mine and utilize the spatial information of the video. Figure 3 As shown, it specifically includes the following steps:

[0117] Step 4.1: For the entire video feature, first aggregate the image block features into object prototypes to represent fine-grained spatial information, such as people, objects, and certain areas in the video. During the object prototype generation process, some background features may interfere with cross-modal alignment. Therefore, object-level prototypes are generated in a sparse aggregation manner and redundant information is filtered out. Two FC layers and ReLU functions are used to predict sparse weights. Where N o is the number of object prototypes, and K is the number of image patches.

[0118] This prevents the object prototype from being affected by the redundant image patch background. Spatial prototyping can be defined as:

[0119]

[0120] For text, we also use the spatial object prototype generation mechanism and design a similar network structure to aggregate word features to generate phrase prototypes. Important spatial information is explored through fine-grained image patch features and word features, and then the model is optimized through spatial object-phrase matching.

[0121] Step 4.2: Calculate the maximum similarity of the object phrase prototypes within each frame. This associates the most similar phrase prototype with each object prototype, reflecting the fine-grained matching across modalities. Subsequently, by processing the similarity matrix of the multi-frame target, the maximum similarity score across the frame sequence is extracted to generate the object-phrase matching probability with higher confidence. Finally, by summing all the target phrase matching scores, the overall similarity criterion s is obtained. op , the interactive process of object phrase matching is defined as:

[0122]

[0123] Where N o is the number of object prototypes, N p is the number of phrase prototypes, and L is the number of video frames.

[0124] Step 5: In the event sentence matching method, key fine-grained objects and dynamic time changes are integrated into different event prototypes. First, dynamic time matching is modeled and a progressive object-event prototype aggregation strategy is proposed to reveal the semantic diversity of the video. Then a dynamic prototype matching mechanism is introduced to more effectively capture the semantic associations across time. The core idea is to gradually aggregate spatial object prototypes into frame-level prototypes, and then generate diversified event prototypes through inter-frame interactions, so as to fully capture the semantic details and dynamic changes of the video.

[0125] Step 5.1: First, a frame decoder is designed to convert all object prototypes Merge to frame-level prototype In order to learn the object relationship between different frames, a masked attention mechanism is introduced:

[0126]

[0127] in, is the frame query (a randomly initialized learnable feature vector), K o and V o For the object prototype P o Features after linear transformation. Attention mask for:

[0128]

[0129] Add frame prototype and the original global features of the corresponding frames To enhance the robustness of the model:

[0130]

[0131] Step 5.2: Develop a dynamic event decoder and learn P f The relationship between frames in the event prototype To illustrate the rich information of the video. Dynamic attention is expressed as:

[0132]

[0133] In the formula, is the event query, K f 、V f is the frame prototype P f Linear transformation characteristics.

[0134] Step 5.3: Each event query can adaptively focus on the video frame prototype, and the synergy of multiple queries implicitly ensures the diversity of event semantics. In order to consider the fact that the same video may correspond to multiple textual semantic descriptions, we directly use the global text representation y E As a sentence prototype, and event prototype P e Alignment is performed to improve the flexibility and accuracy of cross-modal matching. The event sentence prototype matching process can be expressed as:

[0135]

[0136] Step 6: A symmetric cross-entropy loss function is used to optimize the prototype matching within the batch, considering the corresponding text-video pairs as positive examples and all other paired combinations in the batch as negative examples, thereby enhancing the model's ability to distinguish matching accuracy.

[0137]

[0138] S op and S es They are the similarity matrices of the object phrase and event sentence prototype matching when the batch size is B,

[0139]

[0140] Where σ is a learnable hyperparameter, S ij or S ji is the similarity score between the two modalities, is the loss function for text-to-video retrieval, is the loss function for the video to text retrieval process.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A text-video retrieval method based on temporal-spatial feature importance perception, characterized in that: The method comprises the following steps: S1. Extract features from the input video and query text, extract different frame features, global frame features and text features; S2, using a source-channel joint coding method based on semantic communication to transmit the semantic information of the query text; S3, for different frames of video features, extract key frames using importance perception mechanism; S4, generating an object prototype for the image block features in each key frame, selecting object features with semantic information and matching them with phrases in the semantic information of the query text; S5, aggregate different objects into frame events, and use the time association between frames to match the frame time with the event sentence; S6. Use the symmetric cross entropy loss function to optimize the prototype matching within the batch and use it for text video retrieval after training.

2. The text-video retrieval method based on temporal-spatial feature importance perception according to claim 1, characterized in that: In step S1, the CLIP pre-trained model is used to extract features from the videos and text descriptions in the dataset: in is the global frame feature, Represents the features of the 1st to Lth video frames, K is the number of image blocks; Represents the features of the 1st to Mth words, Y S and Y E is the global text feature at the beginning and end of the text, M and D are the number of words and dimensions respectively, and L is the number of frames.

3. The text-video retrieval method based on temporal-spatial feature importance perception according to claim 2, characterized in that: In step S2, the query text transmission process is: S21, transmitting a user input query Y with M words T , the transmitter will input the query text Y T The complex valued symbol vector e after being mapped to the joint source channel JSCC encoder E is expressed as: Where s is the dimension of e, is a set of complex numbers, θ0 is the parameter of JSCC encoder E; S22, the encoded symbol vector e is transmitted through the noisy additive white Gaussian noise AWGN channel, which is expressed as: Where N is the channel noise, the noise is given by Sampling obtained; σ 2 is the noise power, is a complex Gaussian distribution; S23, the receiver maps e′ to the reconstructed feature Y through the JSCC decoder D T ′, which is expressed as: Y T ′=D(e′,τ)=D(E(Y T ,θ0)+N,τ) The reconstructed feature Y T ′ is the original query text feature Y T The estimated value of , τ is the parameter of the joint source channel decoder D; S24, finally use Y T With Y T The feature difference of ′ is used as the loss function to train the joint source channel codec network, which is expressed as: Through the training loss function Let the semantic feature Y after transmission T ′ and Y T The loss between them is minimal, thus ensuring the transmission of query semantic information.

4. The text-video retrieval method based on temporal-spatial feature importance perception according to claim 3, characterized in that: In step S3, the process of extracting the feature information of the video frame is as follows: S31, extracting the video frame feature information X i , as the input for semantic weight calculation, is expressed as: where f L represents the Lth video frame feature, θ1 represents the fixed parameters of the feature extraction network, Q, H, and L represent the width, height, and number of video frame features, respectively; The feature semantic weights are obtained using the gradient-based semantic importance weight module: in To represent the video frame feature f L The combined impact on all perception results x; the superscript "c" represents the perception result index; S32. Calculate importance weights The average of all perception results: w L (L∈{1,2,...,L}) represents the Lth video frame feature map f L The semantic importance of S33, mapping the weight to a preset magnitude: W′=r×SoftMax(τ×W) Where τ is a hyperparameter and r is a constant; the semantic weight W′ is applicable to the semantic level loss function, and the hyperparameter τ controls the distribution of the semantic weight W′; S34, represent each frame feature in the video f l Apply weight W′ to calculate the weighted similarity score S L : where f C is the global feature of the entire video; S35. Finally, the first n frames are selected as key frames according to the weighted similarity score:

5. The text-video retrieval method based on temporal-spatial feature importance perception according to claim 4, characterized in that: In step S4, image blocks-objects and word-phrases are aggregated in the spatial dimension to reveal key local details of the video by generating spatial objects. Then, a fine-grained prototype alignment mechanism is introduced to accurately match spatial objects with text phrases, which includes: S41, aggregate image patch features into object prototypes to represent fine-grained spatial information, and use two FC layers and ReLU functions to predict sparse weights Object prototype It is expressed as: Where N o is the number of object prototypes, K is the number of image patches; S42, calculate the maximum similarity between the object and the phrase prototype in each frame, and obtain the overall similarity standard s by summing up the matching scores of all target phrases op : Where N o is the number of object prototypes, N p is the number of phrase prototypes, and L is the number of video frames.

6. The text-video retrieval method based on temporal-spatial feature importance perception according to claim 5, characterized in that: In step S5, key fine-grained objects and dynamic time changes are integrated into different event prototypes to complete event sentence matching, which includes the following steps: S51. Design a frame decoder to convert all object prototypes Merge to frame-level prototype middle: in, is the frame query, K o and V o For the object prototype P o Features after linear transformation; S52, introduce shielded attention mechanism, attention mask for: S53, add frame prototype and the original global features of the corresponding frames To enhance the robustness of the model: S54, develop dynamic event decoder, learn P f The relationship between frames in the event prototype Among them, dynamic attention is expressed as: In the formula, is the event query, K f 、V f For the framework prototype P f Linear transformation characteristics of ; S55, by directly using the global text to represent y E As a sentence prototype, and event prototype P e Alignment is performed, and the event sentence prototype matching process is expressed as: Among them, s es is the similarity score after the two are matched, represents the i-th event prototype, N e Indicates the number of event prototypes.

7. The text-video retrieval method based on temporal-spatial feature importance perception according to claim 6, characterized in that: In step S6, a symmetric cross entropy loss function is used to optimize the prototype matching within the batch, considering the corresponding text-video pairs as positive examples and all other paired combinations in the batch as negative examples, then: Where S op and S es They are the similarity matrices of the object phrase and event sentence prototype matching when the batch size is B, is the loss function for text-to-video retrieval, is the loss function of the video to text retrieval process, and: Where σ is a learnable hyperparameter, S ij or S ji is the similarity score between the two modalities.

Images