Cross-modal video clip retrieval method based on dual sequence diagrams

Abstract

The invention relates to a cross-modal video clip retrieval method based on a dual sequence diagram, which comprises the following steps of: giving an unedited video with any length and a text description, inputting the video and the text description into a cross-modal video clip retrieval model for processing, and finally obtaining a fusion activation diagram of the video and the text description, the video clip which is most matched with the text description is obtained by fusing the activation graph. The invention innovatively provides a semantic enhanced dual time sequence graph framework for text video clip retrieval. According to the method, visual content and semantic information are decoupled, and an enhanced visual appearance graph and an enhanced semantic graph are respectively constructed, so that the time sequence dependency relationship among video clips is more accurately captured. By means of the decoupling mode, the model can better distinguish fragments with similar visual content but different semantics.

Description

Technical Field

[0001] The present invention belongs to the technical field of video understanding and multimodal learning, and particularly relates to a cross-modal video clip retrieval method based on dual temporal graphs. Background Art

[0002] At the intersection of vision and language, accurately retrieving specific clips from massive unedited videos based on text descriptions poses a core challenge. The difficulty of this task stems from the intricate temporal dependencies in video content, which not only incorporates rich visual details but also embeds multi-level semantic connotations. The high degree of interweaving of the two greatly increases the accuracy requirements of retrieval. The key difficulties that need to be overcome in the cross-modal video clip retrieval task include: 1) parsing and tracking complex and changing time series events in the video to ensure accurate recognition of continuous actions and their sequence; 2) the need to cope with the close interweaving of visual and semantic information to enhance the model's ability to distinguish between visually similar but semantically distinct video clips; 3) at the same time, given the wide variation in video clip lengths, it is necessary to develop efficient algorithms to effectively represent and capture the temporal dependencies between these clips, which further increases the complexity and depth of technical implementation. However, existing methods ignore the semantic dependencies between multiple clips and lack global understanding, resulting in limited performance in complex scenarios. Specifically, the two-stage cross-modal video clip retrieval method processes candidate clips independently, ignores the temporal dependencies between clips, lacks global understanding, and has a fixed and inflexible window size; the one-stage cross-modal video clip retrieval method usually encodes visual and semantic information in a coupled manner, making it difficult to distinguish visually similar but semantically different clips. Summary of the Invention

[0003] The purpose of the present invention is to provide a cross-modal video clip retrieval method based on dual temporal graph in order to overcome the defects of the above-mentioned prior art.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] The present invention provides a cross-modal video clip retrieval method based on a dual temporal graph, constructs a cross-modal video clip retrieval model, and performs video clip retrieval based on the model, specifically comprising the following steps:

[0006] Step S1: Given an unedited video of arbitrary length and a text description, the video and text description are input into the cross-modal video clip retrieval model;

[0007] Step S2: The retrieval model extracts the language features of the text description through the GloVe algorithm and the bidirectional long short-term memory network model;

[0008] Step S3: The unedited video is sampled to obtain multiple video clips, which are processed using an I3D model and two parallel maximum pooling-convolution-linear rectifier activation networks to obtain visual representations and semantic representations corresponding to the multiple video clips;

[0009] Step S4: generating an appearance timing diagram using the visual representation and generating a semantic timing diagram using the semantic representation;

[0010] Step S5: using the action perception mechanism to process the appearance timing graph and the semantic timing graph, thereby improving the discrimination between the appearance timing graph and the semantic timing graph, and obtaining the action perception appearance timing graph and the action perception semantic timing graph;

[0011] Step S6: fusing the text description language features obtained in step S2 with the action-aware appearance timing graph and the action-aware semantic timing graph, respectively, to obtain a text-aware appearance timing graph and a text-aware semantic timing graph;

[0012] Step S7: Input the text-aware appearance timing graph and the text-aware semantic timing graph into two L convolutional layers with a convolution kernel size of K, respectively. Then, pass them through the Sigmoid activation function to generate an appearance activation map and a semantic activation map, respectively. The appearance activation map and the semantic activation map are averaged to obtain a fused activation map.

[0013] Step S8: Select the candidate index with the maximum effective score in the fused activation map, and output the video segment corresponding to the candidate index as the video segment that best matches the text description.

[0014] Furthermore, the step S3 specifically includes the following steps: sampling the video at a fixed interval to obtain N initial video segments The N initial video clips obtained by sampling Input into the I3D model for processing, and extract the I3D features corresponding to N initial video clips The extracted I3D features Input the maximum pooling-convolution-linear rectification activation network for processing to obtain the visual representation F a ∈R N×d , the extracted I3D features Input the maximum pooling-convolution-linear rectification activation network for processing to obtain the semantic representation F s ∈R N×d ,The network parameters of the two maximum pooling-convolution-linear rectifier activation networks are different due to the parameter updates during model training.

[0015] Furthermore, the step S4 of generating an appearance timing diagram using the visual representation specifically includes the following steps:

[0016] Step S41: constructing a set of candidate action clips based on the N initial video clips sampled in step S3;

[0017] Step S42: Calculating the visual features and semantic features of each candidate action segment in the candidate action segment set through a maximum pooling operation;

[0018] Step S43: constructing a visual timing graph and a semantic timing graph according to the visual feature values ​​and semantic features of each candidate action segment calculated in step S42.

[0019] Furthermore, the step S41 specifically includes the following steps:

[0020] Determine whether the number N of initial video segments sampled in step S3 is greater than a first preset value. If not, enumerate all possible initial video segment sequences as candidate action segments, and use all candidate action segments as a candidate action segment set. If N is greater than the first preset value, enumerate all possible initial video segment sequences, and the initial video sequence includes multiple initial video segments V. be (v b …v e ), when the initial video clip sequence meets the following conditions, the initial video clip sequence is used as a candidate action clip, the conditions are:

[0021] bmodg=0&((eg ′ )modg=0)

[0022] Among them, b and e are the subscript indexes of the initial video segment, g and g ′ is defined as:

[0023]

[0024] in, All candidate action segments that meet the conditions are taken as the candidate action segment set.

[0025] Furthermore, the step S42 of calculating the visual features and semantic features of each candidate action segment in the candidate action segment set by the maximum pooling operation includes the following steps:

[0026] The candidate action segments V are pooled by the maximum pooling operation be (v b …v e ) for each initial video segment v i The corresponding visual representation F ai To perform fusion, the fusion formula is:

[0027] f a[b,e] =maxpooling(F ab ,…,F ae)

[0028] Among them, f a[b,e] is a candidate action segment V be Visual features, maxpooling represents the maximum pooling operation, F ab is the initial video segment v b The corresponding visual representation, F ae is the initial video segment v e corresponding visual representations;

[0029] The candidate action segments V are pooled by the maximum pooling operation be (v b …v e ) for each initial video segment v i The corresponding semantic representation F si To perform fusion, the fusion formula is:

[0030] f s[b,e] =maxpooling(F sb ,…,F se )

[0031] Among them, f s[b,e] is a candidate action segment V be The semantic features of maxpooling are the maximum pooling operations, F sb is the initial video segment v b The corresponding semantic representation, F se is the initial video segment v e The corresponding semantic representation.

[0032] Furthermore, the step S43 specifically includes the following steps: traversing each candidate action segment V in the candidate action segment set be , the candidate action segment V be The visual features f a[b,e] Fill in the appearance timing diagram of If b>e, then Fill with 0; traverse each candidate action segment V in the candidate action segment set be , the candidate action segment V be The semantic feature f s[b,e] Fill in the semantic sequence diagram of If b>e, then Filled with 0.

[0033] Furthermore, the processing method of the appearance timing diagram and the semantic timing diagram in step S5 is the same, and the same method is used. The specific steps of step S5 include:

[0034] Step S51: Average timing diagram The value of the first dimension, Q s ∈R 1×N×d , average timing diagram The value of the second dimension is Q e ∈R N×1×d ;

[0035] Step S52: Q s ∈R 1×N×d and Q e ∈R N×1×d Input a convolution layer and a Sigmoid layer to get the attention score Q s ′ ∈R 1×N×1 and Q ′ e ∈R 1×N×1 ;

[0036] Step S53: Q s ′ With Q ′ e As the weight of the original time series diagram The weighted calculation method is:

[0037]

[0038] in, It is an element-by-element multiplication based on the broadcasting mechanism in Python;

[0039] Use the above steps to analyze the appearance timing diagram Semantic Sequence Diagram Processing is performed to obtain the action perception appearance timing diagram and action-aware semantic timing diagrams

[0040] Furthermore, the fusion formula in step S6 is:

[0041]

[0042] in, represents a tensor whose elements are all 1, represents element-by-element multiplication, ⊙ represents matrix multiplication, ||·|| F represents the Frobenius norm, F q Describe the language characteristics of the text, is the text-aware appearance timing diagram, is a text-aware semantic temporal diagram, and T is a transposition symbol.

[0043] Furthermore, the step S7 specifically includes: converting the text perception appearance sequence diagram Input convolution kernel size K into L convolution layers, and pass through a Sigmoid activation function to generate the appearance activation map P a ∈R N×N×1 , the text-aware semantic sequence diagram Input convolution kernel size K into L convolution layers, and after a Sigmoid activation function, generate semantic activation map P s ∈R N×N×1 , P a and P s Average and get the fusion activation map P, the calculation formula is:

[0044]

[0045] Furthermore, the overall loss function of the cross-modal video clip retrieval model is:

[0046]

[0047]

[0048] Among them, λ is a hyperparameter that balances the importance of different loss terms, and p i Respectively represent P a 、P s and the corresponding values ​​in P, B represents the number of finite candidate actions, M is the number of valid candidate actions, It is the temperature parameter, which is set to 0.1 by default. a The score is greater than the threshold σ a , G p Activate the image P for the current appearance a The score is greater than the threshold σ a The corresponding fragment features, G n Activate the image P for the current appearance a The score is less than the threshold σ b The corresponding fragment features, G a G p The central feature, u i is the intersection-over-union ratio of the matched video segment and the real segment, where the real segment is the video segment corresponding to the text description, t max and t min is the preset threshold.

[0049] Compared with the prior art, the present invention has the following advantages:

[0050] (1) This paper innovatively proposes a semantically enhanced dual temporal graph framework for text-based video segment retrieval. This method decouples visual content from semantic information, constructing enhanced visual appearance graphs and semantic graphs, respectively, to more accurately capture the temporal dependencies between video segments. This decoupling approach enables the model to better distinguish segments with similar visual content but different semantics.

[0051] (2) This paper innovatively proposes a novel moment-aware mechanism to enhance the traditional temporal proximity graph, enabling it to perceive the importance of each moment. This mechanism helps the model better capture key segments in the video and improves the accuracy of video segment retrieval;

[0052] (3) To encourage semantically similar retrieval segments corresponding to the same text query, we design a semantic-aware contrastive loss. This loss function guides the learning of semantic graphs by bringing together the normalized embeddings of identical queries while simultaneously pushing apart the embeddings of different queries. This loss function effectively utilizes the weak supervisory signal provided by the visual appearance graph and improves the learning effect of semantic graphs. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A model graph for cross-modal video clip retrieval based on the present invention. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0055] This embodiment provides a cross-modal video clip retrieval method based on dual temporal graph. The overall model framework is as follows: Figure 1 As shown, this can be achieved by following the steps below:

[0056] Step S1: Given an uncut video of any length and a text description The goal of the video segment retrieval task is to find the segment in the video that best matches the text description. i represents the i-th frame of the video, l v is the length of the video, s i represents the i-th word in the text, l s Indicates the length of the text;

[0057] Step S2: Extract the language features F of the text description S through the GloVe algorithm and the bidirectional long short-term memory network model (Long Short-Term Memory, LSTM) q ∈R d ;

[0058] Step S3: For the video V, first split it into non-overlapping video segments, where each segment contains T consecutive frames, and then adopt a fixed-interval sampling strategy to obtain N video segments. The N video clips are processed separately through the I3D model to extract the corresponding I3D features of the N video clips. Then, the I3D features of N video clips are used to extract the corresponding visual representation F using a "Max Pooling-Convolution-Linear Rectification Activation (ReLU)" operation. a ∈R N×d ; Then, the I3D features of N video clips are used to extract the corresponding semantic representation F using a "Max Pooling-Convolution-Linear Rectification Activation (ReLU)" operation s ∈R N×d .

[0059] Step S4: Using visual representation F a Generate an appearance sequence diagram Using semantic representation F s Generate a semantic sequence diagram

[0060] Preferably, step S4 is a dual timing graph proposed in this patent, which aims to decouple visual appearance features and semantic information. We construct the two timing graphs in the same way, but the neural network parameters for constructing the timing graphs are updated during the model training process. Therefore, even if they are constructed in the same way, the features contained in the final appearance timing graph and the semantic timing graph are different. Step S4 includes the following sub-steps:

[0061] Step S41: The core idea of ​​constructing the time sequence diagram is to densely sample the time sequence regions with short time spans and sparsely sample the regions with long time spans as candidate actions. Specifically, when N≤16, all possible video clip sequences are enumerated as candidate action clips, where the candidate action clips are the video clips that the text description may correspond to. When N>16, when the video clip sequence V be (v b …v e ) from v b to v e An action is selected as a candidate action when the following conditions are met:

[0062] bmodg=0&((eg ′ )modg=0)

[0063] Among them, b and e are the subscript indexes of the initial video segment, g and g ′ is defined as:

[0064]

[0065] in, All visual candidate action segments that meet the conditions are taken as the visual candidate action segment set.

[0066] Step S42: Construct a timing diagram for the candidate action set output in step S41 and Specifically, for each candidate action, the candidate action fragment V is pooled by the maximum pooling operation. be (v b …v e ) for each initial video segment v i The corresponding visual representation F ai To perform fusion, the fusion formula is:

[0067] f a[b,e] =maxpooling(F ab ,…,F ae )

[0068] Among them, f a[b,e] is a candidate action segment V be Visual features, maxpooling represents the maximum pooling operation, F ab is the initial video segment v b The corresponding visual representation, F ae is the initial video segment v e corresponding visual representations;

[0069] The candidate action segments V are pooled by the maximum pooling operation be (v b …v e ) for each initial video segment v i The corresponding semantic representation F si To perform fusion, the fusion formula is:

[0070] f s[b,e] =maxpooling(F sb ,…,F se )

[0071] Among them, f s[b,e] is a candidate action segment V beThe semantic features of maxpooling are the maximum pooling operations, F sb is the initial video segment v b The corresponding semantic representation, F se is the initial video segment v e The corresponding semantic representation.

[0072] Step S43: Based on the features generated for all candidate actions in S42, construct an appearance sequence diagram and semantic sequence diagrams Note that the two sequence diagrams are constructed in the same way. The first two dimensions of the sequence diagram represent the start and end indexes of the candidate actions, respectively. For example, Indicates a a to v b Note that the index range here needs to be guaranteed, that is, a≤b. Otherwise, it is considered an illegal action interval and is filled with zero elements in the timing diagram.

[0073] Step S5: Utilize a novel motion perception mechanism to improve and The discriminative power of the action perception appearance sequence diagram is generated and action-aware semantic timing diagrams

[0074] Preferably, step S5 is an action perception mechanism proposed in this patent, which aims to highlight the importance of each action and includes the following sub-steps:

[0075] Step S51: For the timing diagram and The same method is used to improve the discriminability of features. express, Each element in Then it means that from the fragment v i to v j To evaluate the possibility of each segment as the start and end moment of an action, this patent proposes to average The values ​​of the first and second dimensions are obtained s ∈R 1×N×d and Q e ∈R N×1×d , as a measure of the likelihood of the start and end moments.

[0076] Step S52: The Q generated in step S51 s and Q e Input into a convolution layer and a Sigmoid layer to get the attention score Q s′ ∈R 1×N×1 and Q ′ e ∈R 1×N×1 .

[0077] Step S53: Q s ′ With Q ′ e As the weight of the original time series diagram The weighted calculation is as follows:

[0078]

[0079] in, Represents element-by-element multiplication based on the broadcasting mechanism in Python.

[0080] Step S6: Language feature F q Separate and action-aware appearance timing diagram Action-aware semantic timing diagram Fusion is performed to obtain the text perception appearance timing diagram and text-aware semantic timing diagrams The specific operations are as follows:

[0081]

[0082] in, represents a tensor whose elements are all 1, represents element-by-element multiplication, ⊙ represents matrix multiplication, ||·|| F represents the Frobenius norm, T is the transpose symbol, is the text-aware appearance timing diagram, It is a text-aware semantic timing diagram.

[0083] Step S7: Text-aware appearance timing diagram The input is fed into L convolution layers with a convolution kernel size of K, and then passes through a Sigmoid activation function to generate an appearance activation map P. a ∈R N×N×1 , the text-aware semantic sequence diagram The input is fed into L convolution layers with a convolution kernel size of K, and then passes through a Sigmoid activation function to generate a semantic activation map P. s ∈R N×N×1 Finally, the average P a and P s The value of fused activation map P is obtained:

[0084]

[0085] Each score in P represents the degree of match between its corresponding video segment and the text description.

[0086] Step S8: Select the candidate index with the maximum valid score in P and output its corresponding video content as the video clip that best matches the search text S.

[0087] Step S9: In order to supervise the entire model training process, first calculate the Intersection over Union (IoU) u of each candidate action region and the true region i , where the true region is the video clip to which the text describes its exact correspondence. Then, two predefined thresholds t max and t min To scale u i , the specific method is as follows:

[0088]

[0089] Here is the y obtained after scaling i is used as the true value to supervise the training of the model. Specifically, for the static appearance timing graph P generated in step S7 a and semantic sequence diagram P s , using the cross entropy loss function as the learning criterion:

[0090]

[0091] in, and p i Respectively represent P a , P s and the corresponding value in P, B represents the number of finite candidate actions.

[0092] Step S10: In order to learn the semantic temporal graph, this patent believes that video clips retrieved from the same text should have similar semantic features. Therefore, a semantic-aware contrast loss is proposed, which encourages the features of video clips corresponding to the same text description to be pulled closer, and the features of video clips corresponding to different text descriptions to be pushed further away. To achieve this goal, the knowledge from the visual appearance temporal graph is distilled into the semantic branch. Specifically, if the knowledge from the appearance temporal graph P a The score is greater than the threshold σ a , then the corresponding segment features are taken as positive samples G p , if the score is lower than the threshold σ b , then the corresponding fragment features are taken as negative samples G n The central features of all positive samples serve as anchor points G a . Then, the following loss function is used to optimize it:

[0093]

[0094] Where M is the number of valid candidate actions, is the temperature parameter, which is set to 0.1 by default.

[0095] Step S11: The loss function of the model as a whole is:

[0096]

[0097] Among them, λ is a hyperparameter used to balance the importance of different loss terms.

[0098] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0099] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A cross-modal video clip retrieval method based on dual temporal graph, characterized in that: Construct a cross-modal video clip retrieval model and perform video clip retrieval based on the model. The specific steps include: Step S1: Given an uncut video of arbitrary length and a text description, the video and the text description are input into a cross-modal video clip retrieval model; Step S2: The retrieval model extracts the language features of the text description through the GloVe algorithm and the bidirectional long short-term memory network model; Step S3: The uncut video is sampled to obtain multiple video clips, and the video clips are processed by an I3D model and two parallel maximum pooling-convolution-linear rectification activation networks to obtain visual representations and semantic representations corresponding to the multiple video clips; Step S4: generating an appearance timing diagram using the visual representation, and generating a semantic timing diagram using the semantic representation; Step S5: using the action perception mechanism to process the appearance timing graph and the semantic timing graph, improving the discrimination of the appearance timing graph and the semantic timing graph, and obtaining the action perception appearance timing graph and the action perception semantic timing graph; Step S6: fusing the text description language features obtained in step S2 with the action-aware appearance timing graph and the action-aware semantic timing graph, respectively, to obtain a text-aware appearance timing graph and a text-aware semantic timing graph; Step S7: input the text-aware appearance timing graph and the text-aware semantic timing graph into two L convolutional layers with a convolution kernel size of K, respectively, and then pass through a Sigmoid activation function to generate an appearance activation map and a semantic activation map, respectively, and obtain a fused activation map by averaging the appearance activation map and the semantic activation map; Step S8: Select the candidate index with the maximum effective score in the fused activation map, and output the video segment corresponding to the candidate index as the video segment that best matches the text description.

2. According to claim 1, a cross-modal video clip retrieval method based on dual temporal graph is characterized in that: The step S3 specifically comprises the following steps: The video is sampled at fixed intervals to obtain N initial video clips The N initial video clips obtained by sampling Input into the I3D model for processing, and extract the I3D features corresponding to the N initial video clips The extracted I3D features Input the maximum pooling-convolution-linear rectification activation network for processing to obtain the visual representation F a ∈R N×d ; The extracted I3D features Input the maximum pooling-convolution-linear rectification activation network for processing to obtain the semantic representation F s ∈R N×d ; The two maximum pooling-convolution-linear rectification activation networks have different network parameters due to parameter updates during model training.

3. The cross-modal video clip retrieval method based on dual temporal graph according to claim 1 or 2, characterized in that: The step S4 uses the visual representation to generate the appearance timing diagram, which specifically includes the following steps: Step S41: constructing a candidate action segment set according to the N initial video segments sampled in step S3; Step S42: calculating the visual features and semantic features of each candidate action segment in the candidate action segment set through a maximum pooling operation; Step S43: constructing a visual timing diagram and a semantic timing diagram according to the visual feature values ​​and semantic features of each candidate action segment calculated in step S42.

4. The cross-modal video clip retrieval method based on dual temporal graph according to claim 3, characterized in that: The step S41 specifically includes the following steps: Determine whether the number N of initial video segments sampled in step S3 is greater than a first preset value; if not, enumerate all possible initial video segment sequences as candidate action segments, and use all candidate action segments as a candidate action segment set; If N is greater than a first preset value, all possible initial video segment sequences are enumerated, where the initial video sequence includes a plurality of initial video segments V be (v b …v e ), when the initial video clip sequence meets the following conditions, the initial video clip sequence is used as a candidate action clip, the conditions are: b mod g=0&((eg′)mod g=0) Among them, b and e are the subscript indices of the initial video segment, g and g ′ is defined as: g＝2 k-1 in, All candidate action segments that meet the conditions are taken as candidate action segment sets.

5. The cross-modal video clip retrieval method based on dual temporal graph according to claim 3, characterized in that: Calculating the visual features and semantic features of each candidate action segment in the candidate action segment set by the maximum pooling operation in step S42 includes the following steps: The candidate action segments V are pooled by the maximum pooling operation. be (v b …v e ) for each initial video segment v i The corresponding visual representation F ai The fusion formula is: f a[b,e] =maxpooling(F ab ,…,f ae ) Among them, f a[b,e] is the candidate action segment V be The visual features of maxpooling are the maximum pooling operations, F ab is the initial video segment v b The corresponding visual representation, F ae is the initial video segment v e corresponding visual representation; The candidate action segments V are pooled by the maximum pooling operation. be (v b …v e ) for each initial video segment v i The corresponding semantic representation F si The fusion formula is: f s[b,e] =maxpooling(F sb ,…,F se ) Among them, f s[b,e] is the candidate action segment V be The semantic features of maxpooling are the maximum pooling operations, F sb is the initial video segment v b The corresponding semantic representation, F se is the initial video segment v e The corresponding semantic representation.

6. The cross-modal video clip retrieval method based on dual temporal graph according to claim 3, characterized in that: The step S43 specifically includes the following steps: Traverse each candidate action segment V in the candidate action segment set be , the candidate action segment V be The visual features of a[b,e] Fill in the appearance timing diagram of If b>e, then Fill with 0; Traverse each candidate action segment V in the candidate action segment set be , the candidate action segment V be The semantic features of s[b,e] Fill in the semantic sequence diagram of If b>e, then Filled with 0.

7. The cross-modal video clip retrieval method based on dual temporal graph according to claim 1, characterized in that: In step S5, the processing method of the appearance timing diagram and the semantic timing diagram is the same, and the same method is used. Represents a timing diagram; The specific steps of step S5 include: Step S51: Average timing diagram The value of the first dimension, Q s ∈R 1×N×d , average timing diagram The value of the second dimension is Q e ∈R N×1×d ; Step S52: Q s ∈R 1×N×d and Q e ∈R N×1×d Input a convolution layer and a Sigmoid layer to get the attention score Q s ′ ∈R 1×N×1 and Q ′ e ∈R 1×N×1 ; Step S53: Q s ′ With Q ′ e As the weight of the original time series diagram The weighted calculation method is: in, It is an element-by-element multiplication based on the broadcasting mechanism in Python; Use the above steps to analyze the appearance timing diagram Semantic Sequence Diagram After processing, we can get the action perception appearance timing diagram Action-aware semantic timing diagram 8. The cross-modal video clip retrieval method based on dual temporal graph according to claim 1, characterized in that: The fusion formula of step S6 is: in, represents a tensor whose elements are all 1, represents element-by-element multiplication, ⊙ represents matrix multiplication, ||·|| F represents the Frobenius norm, F q Describe the language features of the text, is the text-aware appearance timing diagram, is a text-aware semantic temporal graph, and T is the transposition symbol.

9. The cross-modal video clip retrieval method based on dual temporal graph according to claim 1, characterized in that: The step S7 specifically includes: Text-aware appearance timing diagram The input convolution kernel size is K in L convolution layers, and after a Sigmoid activation function, the appearance activation map P is generated. a ∈R N×N×1 ; Text-aware semantic timing diagram The input convolution kernel size is K in L convolution layers, and after a Sigmoid activation function, a semantic activation map P is generated. s ∈R N×N×1 ; P a and P s Average and get the fusion activation map P, the calculation formula is:

10. The cross-modal video clip retrieval method based on dual temporal graph according to claim 1, characterized in that: The overall loss function of the cross-modal video clip retrieval model is: Among them, λ is a hyperparameter that balances the importance of different loss terms. and p i Respectively represent P a , P s and the corresponding values ​​in P, B represents the number of finite candidate actions, M is the number of valid candidate actions, It is the temperature parameter, which is set to 0.1 by default. a The score is greater than the threshold σ a , G p Activate the image P for the current appearance a The score is greater than the threshold σ a The corresponding fragment feature, G n Activate the image P for the current appearance a The score is less than the threshold σ b The corresponding fragment feature, G a G p The central feature, u i is the intersection-over-union ratio of the matched video segment and the real segment, where the real segment is the video segment corresponding to the text description, t max and t min is the preset threshold.

Images