A cross-modal video segment retrieval method based on dual time sequence diagram

Abstract

The application relates to a cross-modal video segment retrieval method based on a dual time sequence diagram, which comprises the following steps: given an arbitrary long uncut video and a text description, inputting the video and the text description into a cross-modal video segment retrieval model for processing, finally obtaining a fusion activation map of the video and the text description, and obtaining a video segment most matched with the text description through the fusion activation map, and the application innovatively proposes a semantic enhanced dual time sequence diagram framework for text video segment retrieval. The method decouples visual content and semantic information, respectively constructs an enhanced visual appearance diagram and a semantic diagram, and more accurately captures the time sequence dependence relationship between video segments. The decoupling mode enables the model to better distinguish segments with similar visual content but different semantics.

Description

Technical Field

[0001] The present invention belongs to the technical field of video understanding and multimodal learning, and in particular relates to a cross-modal video clip retrieval method based on dual temporal graphs. Background Art

[0002] At the intersection of vision and language, accurately retrieving specific clips from massive unedited videos based on text descriptions poses a core challenge. The difficulty of this task stems from the intricate temporal dependencies in video content, which not only incorporates rich visual details but also embeds multi-level semantic connotations. The high degree of interweaving of the two greatly increases the accuracy requirements of retrieval. The key difficulties that need to be overcome in the cross-modal video clip retrieval task include: 1) parsing and tracking complex and changing time series events in the video to ensure accurate recognition of continuous actions and their sequence; 2) the need to cope with the close interweaving of visual and semantic information to enhance the model's ability to distinguish between visually similar but semantically distinct video clips; 3) at the same time, given the wide variation in video clip lengths, it is necessary to develop efficient algorithms to effectively represent and capture the temporal dependencies between these clips, which further increases the complexity and depth of technical implementation. However, existing methods ignore the semantic dependencies between multiple clips and lack global understanding, resulting in limited performance in complex scenarios. Specifically, the two-stage cross-modal video clip retrieval method processes candidate clips independently, ignores the temporal dependencies between clips, lacks global understanding, and has a fixed and inflexible window size; the one-stage cross-modal video clip retrieval method usually encodes visual and semantic information in a coupled manner, making it difficult to distinguish visually similar but semantically different clips. Summary of the Invention

[0003] The purpose of the present invention is to provide a cross-modal video clip retrieval method based on dual temporal graph in order to overcome the defects of the above-mentioned prior art.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] The present invention provides a cross-modal video clip retrieval method based on a dual temporal graph, constructs a cross-modal video clip retrieval model, and performs video clip retrieval based on the model, specifically comprising the following steps:

[0006] Step S1: Given an unedited video of arbitrary length and a text description, the video and text description are input into the cross-modal video clip retrieval model;

[0007] Step S2: The retrieval model extracts the language features of the text description through the GloVe algorithm and the bidirectional long short-term memory network model;

[0008] Step S3: The unedited video is sampled to obtain multiple video clips, which are processed using an I3D model and two parallel maximum pooling-convolution-linear rectifier activation networks to obtain visual representations and semantic representations corresponding to the multiple video clips;

[0009] Step S4: generating an appearance timing diagram using the visual representation and generating a semantic timing diagram using the semantic representation;

[0010] Step S5: using the action perception mechanism to process the appearance timing graph and the semantic timing graph, thereby improving the discrimination between the appearance timing graph and the semantic timing graph, and obtaining the action perception appearance timing graph and the action perception semantic timing graph;

[0011] Step S6: fusing the text description language features obtained in step S2 with the action-aware appearance timing graph and the action-aware semantic timing graph, respectively, to obtain a text-aware appearance timing graph and a text-aware semantic timing graph;

[0012] Step S7: Input the text-aware appearance timing graph and the text-aware semantic timing graph into two L convolutional layers with a convolution kernel size of K, respectively. Then, pass them through the Sigmoid activation function to generate an appearance activation map and a semantic activation map, respectively. The appearance activation map and the semantic activation map are averaged to obtain a fused activation map.

[0013] Step S8: Select the candidate index with the maximum effective score in the fused activation map, and output the video segment corresponding to the candidate index as the video segment that best matches the text description.

[0014] Furthermore, the step S3 specifically includes the following steps: sampling the video at a fixed interval to obtain N initial video segments The N initial video clips obtained by sampling Input into the I3D model for processing, and extract the I3D features corresponding to N initial video clips The extracted I3D features Input the maximum pooling-convolution-linear rectification activation network for processing to obtain the visual representation F a ∈R N×d , the extracted I3D features Input the maximum pooling-convolution-linear rectification activation network for processing to obtain the semantic representation F s ∈R N×d ,The network parameters of the two maximum pooling-convolution-linear rectifier activation networks are different due to the parameter updates during model training.

[0015] Furthermore, the step S4 of generating an appearance timing diagram using the visual representation specifically includes the following steps:

[0016] Step S41: constructing a candidate action segment set according to the N initial video segments obtained by sampling in step S3;

[0017] Step S42: calculating the visual features and semantic features of each candidate action segment in the candidate action segment set by a max-pooling operation;

[0018] Step S43: constructing a visual time sequence diagram and a semantic time sequence diagram according to the visual feature values and semantic features of each candidate action segment calculated in step S42.

[0019] Further, the step S41 specifically includes the following steps:

[0020] determining whether the number N of the initial video segments obtained by sampling in step S3 is greater than a first preset value, if not, enumerating all possible initial video segment sequences as candidate action segments, and taking all the candidate action segments as the candidate action segment set; if N is greater than the first preset value, then enumerating all possible initial video segment sequences, the initial video sequence including a plurality of initial video segments V be (v b …v e ), and taking the initial video segment sequence as a candidate action segment when the initial video segment sequence satisfies the following conditions:

[0021] b mod g = 0 & ((e-g ′ ) mod g = 0)

[0022] wherein b and e are the index of the initial video segments, and g and g ′ are defined as:

[0023]

[0024] wherein, taking all the candidate action segments satisfying the conditions as the candidate action segment set.

[0025] Further, the step S42 of calculating the visual features and semantic features of each candidate action segment in the candidate action segment set by a max-pooling operation includes the following steps:

[0026] fusing the corresponding visual representations F be (v b …v e ) of each initial video segment v i in the candidate action segment V ai by a max-pooling operation, and the fusion formula is:

[0027] f a[b,e] = maxpooling(F ab ,…,F ae)

[0028] Among them, f a[b,e] is a candidate action segment V be Visual features, maxpooling represents the maximum pooling operation, F ab is the initial video segment v b The corresponding visual representation, F ae is the initial video segment v e corresponding visual representations;

[0029] The candidate action segments V are pooled by the maximum pooling operation be (v b …v e ) for each initial video segment v i The corresponding semantic representation F si To perform fusion, the fusion formula is:

[0030] f s[b,e] =maxpooling(F sb ,…,F se )

[0031] Among them, f s[b,e] is a candidate action segment V be The semantic features of maxpooling represent the maximum pooling operation, F sb is the initial video segment v b The corresponding semantic representation, F se is the initial video segment v e The corresponding semantic representation.

[0032] Furthermore, the step S43 specifically includes the following steps: traversing each candidate action segment V in the candidate action segment set be , the candidate action segment V be The visual features f a[b,e] Fill in the appearance timing diagram of If b>e, then Fill with 0; traverse each candidate action segment V in the candidate action segment set be , the candidate action segment V be The semantic feature f s[b,e] Fill in the semantic sequence diagram of If b>e, then Filled with 0.

[0033] Furthermore, the processing method of the appearance sequence diagram and the semantic sequence diagram in step S5 is the same, and the same method is used. The specific steps of step S5 include:

[0034] Step S51: average time sequence diagram the value of the first dimension, resulting in Q s ∈R 1×N×d , average time sequence diagram the value of the second dimension, resulting in Q e ∈R N×1×d ;

[0035] Step S52: input Q s ∈R 1×N×d and Q e ∈R N×1×d into a convolutional layer and a sigmoid layer to obtain the attention score Q s ′ ∈R 1×N×1 and Q ′ e ∈R 1×N×1 ;

[0036] Step S53: input Q s ′ and Q ′ e into the original time sequence diagram as weights, and the calculation method is as follows:

[0037]

[0038] wherein, is element-wise multiplication based on the broadcast mechanism in Python;

[0039] The above steps are used to process the appearance time sequence diagram and the semantic time sequence diagram respectively, to obtain the action-aware appearance time sequence diagram and the action-aware semantic time sequence diagram

[0040] Further, the step S6 fusion formula is as follows:

[0041]

[0042] wherein, represents a tensor with all elements being 1, represents element-wise multiplication, represents matrix multiplication, and ||·||F F represents the Frobenius norm, F q is the language feature of the text description, is the text-aware appearance time sequence diagram, is the text-aware semantic time sequence diagram, and T is the transpose symbol.

[0043] ​Furthermore, the step S7 specifically includes: converting the text perception appearance sequence diagram Input convolution kernel size K into L convolution layers, and pass through a Sigmoid activation function to generate the appearance activation map P a ∈R N×N×1 , the text-aware semantic sequence diagram Input convolution kernel size K into L convolution layers, and after a Sigmoid activation function, generate semantic activation map P s ∈R N×N×1 , P a and P s Average and get the fusion activation map P, the calculation formula is:

[0044]

[0045] Furthermore, the overall loss function of the cross-modal video clip retrieval model is:

[0046]

[0047]

[0048] Among them, λ is a hyperparameter that balances the importance of different loss terms, and p i Represents P a 、P s and the corresponding values ​​in P, B represents the number of finite candidate actions, M is the number of valid candidate actions, It is the temperature parameter, which is set to 0.1 by default. a The score is greater than the threshold σ a , G p Activate the image P for the current appearance a The score is greater than the threshold σ a The corresponding fragment features, G n Activate the image P for the current appearance a The score is less than the threshold σ b The corresponding fragment features, G a G p The central feature, u i is the intersection-over-union ratio of the matched video segment and the real segment, where the real segment is the video segment corresponding to the text description, t max and t min is the preset threshold.

[0049] Compared with the prior art, the present invention has the following advantages:

[0050] (1) This paper innovatively proposes a semantically enhanced dual temporal graph framework for text-based video segment retrieval. This method decouples visual content from semantic information, constructing enhanced visual appearance graphs and semantic graphs, respectively, to more accurately capture the temporal dependencies between video segments. This decoupling approach enables the model to better distinguish segments with similar visual content but different semantics.

[0051] (2) This paper innovatively proposes a novel moment-aware mechanism to enhance the traditional temporal proximity graph, enabling it to perceive the importance of each moment. This mechanism helps the model better capture key segments in the video and improves the accuracy of video segment retrieval;

[0052] (3) To encourage semantically similar retrieval segments corresponding to the same text query, we design a semantic-aware contrastive loss. This loss function guides the learning of semantic graphs by bringing together the normalized embeddings of identical queries while simultaneously pushing apart the embeddings of different queries. This loss function effectively utilizes the weak supervisory signal provided by the visual appearance graph and improves the learning effect of semantic graphs. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A model graph for cross-modal video clip retrieval based on the present invention. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0055] This embodiment provides a cross-modal video clip retrieval method based on dual temporal graph. The overall model framework is as follows: Figure 1 As shown, this can be achieved by following the steps below:

[0056] Step S1: Given an uncut video of any length and a text description The goal of the video segment retrieval task is to find the segment in the video that best matches the text description. i represents the i-th frame of the video, l v is the length of the video, s i represents the i-th word in the text, l s Indicates the length of the text;

[0057] Step S2: Extracting language features F of the text description S by GloVe algorithm and bidirectional Long Short-Term Memory (LSTM) model q ∈R d ;

[0058] Step S3: For the video V, first, it is segmented into non-overlapping video segments, each of which contains T consecutive frames, and then a fixed interval sampling strategy is adopted to obtain N video segments The N video segments are processed by the I3D model respectively, and the corresponding I3D features of the N video segments are extracted Then, the I3D features of the N video segments are extracted using a "Max Pooling-Convolution-Rectified Linear Unit (ReLU)" operation to obtain the corresponding visual representation F a ∈R N×d ; and the I3D features of the N video segments are extracted using a "Max Pooling-Convolution-Rectified Linear Unit (ReLU)" operation to obtain the corresponding semantic representation F s ∈R N×d .

[0059] Step S4: Using the visual representation F a to generate an appearance time chart Using the semantic representation F s to generate a semantic time chart

[0060] Preferably, the step S4 is a dual time chart proposed in this patent, which aims to decouple visual appearance features and semantic information. We construct two time charts in the same way, but the neural network parameters for constructing the time chart are updated during the model training process. Therefore, even if the same way is used for construction, the appearance time chart and the semantic time chart finally constructed contain different features. Step S4 includes the following sub-steps:

[0061] Step S41: The core idea of constructing the time chart is to densely sample the time span of the short time chart and sparsely sample the time span of the long time chart as a candidate action. Specifically, when N≤16, all possible video segment sequences are enumerated as candidate action segments, where the candidate action segment is the video segment that the text description may correspond to. When N>16, when the video segment sequence V be (v b …v e ) from v b to v e satisfies the following conditions, it is selected as a candidate action:

[0062] bmodg=0&((eg ′ )modg=0)

[0063] Among them, b and e are the subscript indexes of the initial video segment, g and g ′ is defined as:

[0064]

[0065] in, All visual candidate action segments that meet the conditions are taken as the visual candidate action segment set.

[0066] Step S42: Construct a timing diagram for the candidate action set output in step S41 and Specifically, for each candidate action, the candidate action fragment V is pooled by the maximum pooling operation. be (v b …v e ) for each initial video segment v i The corresponding visual representation F ai To perform fusion, the fusion formula is:

[0067] f a[b,e] =maxpooling(F ab ,…,F ae )

[0068] Among them, f a[b,e] is a candidate action segment V be Visual features, maxpooling represents the maximum pooling operation, F ab is the initial video segment v b The corresponding visual representation, F ae is the initial video segment v e corresponding visual representations;

[0069] The candidate action segments V are pooled by the maximum pooling operation be (v b …v e ) for each initial video segment v i The corresponding semantic representation F si To perform fusion, the fusion formula is:

[0070] f s[b,e] =maxpooling(F sb ,…,F se )

[0071] Among them, f s[b,e] is a candidate action segment V beThe semantic features of maxpooling are the maximum pooling operations, F sb is the initial video segment v b The corresponding semantic representation, F se is the initial video segment v e The corresponding semantic representation.

[0072] Step S43: Based on the features generated for all candidate actions in S42, construct an appearance sequence diagram and semantic sequence diagrams Note that the two sequence diagrams are constructed in the same way. The first two dimensions of the sequence diagram represent the start and end indexes of the candidate actions, respectively. For example, Indicates a a to v b Note that the index range here needs to be guaranteed, that is, a≤b. Otherwise, it is considered an illegal action interval and is filled with zero elements in the timing diagram.

[0073] Step S5: Utilize a novel motion perception mechanism to improve and The discriminative power of the action perception appearance sequence diagram is generated and action-aware semantic timing diagrams

[0074] Preferably, step S5 is an action perception mechanism proposed in this patent, which aims to highlight the importance of each action and includes the following sub-steps:

[0075] Step S51: For the timing diagram and The same method is used to improve the discriminability of features. express, Each element in Then it means that from the fragment v i to v j To evaluate the possibility of each segment as the start and end moment of an action, this patent proposes to average The values ​​of the first and second dimensions are obtained s ∈R 1×N×d and Q e ∈R N×1×d , as a measure of the likelihood of the start and end moments.

[0076] Step S52: The Q generated in step S51 s and Q e Input into a convolution layer and a Sigmoid layer to get the attention score Q s′ ∈R 1×N×1 and Q ′ e ∈R 1×N×1 .

[0077] Step S53: Q s ′ With Q ′ e As the weight of the original time series diagram The weighted calculation is as follows:

[0078]

[0079] in, Represents element-by-element multiplication based on the broadcasting mechanism in Python.

[0080] Step S6: Language feature F q Separate and action-aware appearance timing diagram Action-aware semantic timing diagram Fusion is performed to obtain the text perception appearance timing diagram and text-aware semantic timing diagrams The specific operations are as follows:

[0081]

[0082] in, represents a tensor whose elements are all 1, represents element-by-element multiplication, ⊙ represents matrix multiplication, ||·|| F represents the Frobenius norm, T is the transpose symbol, is the text-aware appearance timing diagram, It is a text-aware semantic timing diagram.

[0083] Step S7: Text-aware appearance sequence diagram The input is fed into L convolution layers with a convolution kernel size of K, and then passes through a Sigmoid activation function to generate an appearance activation map P. a ∈R N×N×1 , the text-aware semantic sequence diagram The input is fed into L convolution layers with a convolution kernel size of K, and then passes through a Sigmoid activation function to generate a semantic activation map P. s ∈R N×N×1 Finally, the average P a and P s The value of fused activation map P is obtained:

[0084]

[0085] Each score in P represents the degree of match between its corresponding video segment and the text description.

[0086] Step S8: Select the candidate index with the maximum valid score in P and output its corresponding video content as the video clip that best matches the search text S.

[0087] Step S9: In order to supervise the entire model training process, first calculate the Intersection over Union (IoU) u of each candidate action region and the true region i , where the true region is the video clip to which the text describes its exact correspondence. Then, two predefined thresholds t max and t min To scale u i , the specific method is as follows:

[0088]

[0089] Here is the y obtained after scaling i is used as the true value to supervise the training of the model. Specifically, for the static appearance sequence graph P generated in step S7 a and semantic sequence diagram P s , using the cross entropy loss function as the learning criterion:

[0090]

[0091] in, and p i Represents P a , P s and the corresponding value in P, B represents the number of finite candidate actions.

[0092] Step S10: In order to learn the semantic temporal graph, this patent believes that video clips retrieved from the same text should have similar semantic features. Therefore, a semantic-aware contrast loss is proposed, which encourages the features of video clips corresponding to the same text description to be pulled closer, and the features of video clips corresponding to different text descriptions to be pushed further away. To achieve this goal, the knowledge from the visual appearance temporal graph is distilled into the semantic branch. Specifically, if the knowledge from the appearance temporal graph P a The score is greater than the threshold σ a , then the corresponding segment features are taken as positive samples G p , if the score is lower than the threshold σ b , then the corresponding fragment features are taken as negative samples G n The central features of all positive samples serve as anchor points G a . Then, the following loss function is used to optimize it:

[0093]

[0094] Where M is the number of valid candidate actions, is the temperature parameter, which is set to 0.1 by default.

[0095] Step S11: The loss function of the model as a whole is:

[0096]

[0097] Among them, λ is a hyperparameter used to balance the importance of different loss terms.

[0098] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0099] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A cross-modal video clip retrieval method based on dual temporal graph, characterized in that: Construct a cross-modal video clip retrieval model and perform video clip retrieval based on this model. The specific steps include: Step S1: Given an unedited video of arbitrary length and a text description, the video and text description are input into the cross-modal video clip retrieval model; Step S2: The retrieval model extracts the language features of the text description through the GloVe algorithm and the bidirectional long short-term memory network model; Step S3: The unedited video is sampled to obtain multiple video clips, which are processed using an I3D model and two parallel maximum pooling-convolution-linear rectifier activation networks to obtain visual representations and semantic representations corresponding to the multiple video clips; Step S4: generating an appearance timing diagram using the visual representation and generating a semantic timing diagram using the semantic representation; Step S5: using the action perception mechanism to process the appearance timing graph and the semantic timing graph, thereby improving the discrimination between the appearance timing graph and the semantic timing graph, and obtaining the action perception appearance timing graph and the action perception semantic timing graph; Step S6: fusing the text description language features obtained in step S2 with the action-aware appearance timing graph and the action-aware semantic timing graph, respectively, to obtain a text-aware appearance timing graph and a text-aware semantic timing graph; Step S7: Input the text-aware appearance timing graph and the text-aware semantic timing graph into two convolution kernels of size K of L In the convolution layer, the Sigmoid activation function is then used to generate the appearance activation map and the semantic activation map respectively. The appearance activation map and the semantic activation map are averaged to obtain the fused activation map. Step S8: Select the candidate index with the maximum effective score in the fused activation map, and output the video segment corresponding to the candidate index as the video segment that best matches the text description; In step S5, the processing method of the appearance sequence diagram and the semantic sequence diagram is the same, and the same method is used. The specific steps of step S5 include: Step S51: Average timing diagram The value of the first dimension is obtained , average timing diagram The value of the second dimension is obtained ; Step S52: and Input a convolutional layer and a Sigmoid In the layer, we get the attention score and ; Step S53: and As the weight of the original time series diagram The weighted calculation method is: in, It is an element-by-element multiplication based on the broadcasting mechanism in Python; Use the above steps to analyze the appearance timing diagram Semantic Sequence Diagram Processing is performed to obtain the action perception appearance timing diagram and action-aware semantic timing diagrams .

2. The cross-modal video clip retrieval method based on dual temporal graph according to claim 1, characterized in that: The step S3 specifically includes the following steps: The video is sampled at fixed intervals to obtain N Initial video clips V= ; The sampled N Initial video clips V= Input into I3D model for processing and extraction N I3D features corresponding to the initial video clips ; The extracted I3D features Input the maximum pooling-convolution-linear rectification activation network for processing to obtain visual representation ; The extracted I3D features Input the maximum pooling-convolution-linear rectification activation network for processing to obtain semantic representation ; The two maximum pooling-convolution-linear rectification activation networks have different network parameters due to parameter updates during model training.

3. A cross-modal video clip retrieval method based on dual temporal graph according to claim 1 or 2, characterized in that: The step S4 uses the visual representation to generate the appearance timing diagram, which specifically includes the following steps: Step S41: Based on the sampling obtained in step S3 N Initial video clips are used to construct a set of candidate action clips; Step S42: Calculating the visual features and semantic features of each candidate action segment in the candidate action segment set through a maximum pooling operation; Step S43: constructing a visual timing graph and a semantic timing graph according to the visual feature values ​​and semantic features of each candidate action segment calculated in step S42.

4. The cross-modal video clip retrieval method based on dual temporal graph according to claim 3, characterized in that: The step S41 specifically includes the following steps: Determine the number of initial video clips sampled in step S3 N whether it is greater than a first preset value; if not, enumerating all possible initial video clip sequences as candidate action clips, and forming all candidate action clips as a candidate action clip set; like N If the value is greater than a first preset value, all possible initial video segment sequences are enumerated, and the initial video segment sequence includes multiple initial video segments. , when the initial video clip sequence meets the following conditions, the initial video clip sequence is used as a candidate action clip, the conditions are: in, and is the subscript index of the initial video segment, and is defined as: in, , all candidate action segments that meet the conditions are taken as candidate action segment sets.

5. The cross-modal video clip retrieval method based on dual temporal graph according to claim 3, characterized in that: Calculating the visual features and semantic features of each candidate action segment in the candidate action segment set by the maximum pooling operation in step S42 includes the following steps: The candidate action segments are pooled by the maximum pooling operation Each initial video segment Corresponding visual representation To perform fusion, the fusion formula is: in, Candidate action segments The visual characteristics of represents the maximum pooling operation, For the initial video clip The corresponding visual representation, For the initial video clip corresponding visual representations; The candidate action segments are pooled by the maximum pooling operation Each initial video segment Corresponding semantic representation To perform fusion, the fusion formula is: in, Candidate action segments The semantic features of represents the maximum pooling operation, For the initial video clip The corresponding semantic representation, For the initial video clip The corresponding semantic representation.

6. The method for cross-modal video segment retrieval based on dual temporal graph according to claim 3, characterized in that: The step S43 specifically includes the following steps: Traverse each candidate action segment in the candidate action segment set , the candidate action segments Visual features Fill in the appearance timing diagram of ,like ,but Fill with 0; Traverse each candidate action segment in the candidate action segment set , the candidate action segments Semantic features of Fill in the semantic sequence diagram of ,like ,but Filled with 0.

7. The cross-modal video clip retrieval method based on dual temporal graph according to claim 1, characterized in that: The fusion formula in step S6 is: in, represents a tensor whose elements are all 1, represents element-wise multiplication, represents matrix multiplication, represents the Frobenius norm, Describe the language characteristics of the text, is the text-aware appearance timing diagram, is a text-aware semantic sequence diagram, is the transpose symbol.

8. The method for cross-modal video segment retrieval based on dual temporal graph according to claim 1, characterized in that: The step S7 specifically includes: Text-aware appearance sequence diagram The input convolution kernel size is K of L In the convolution layer, a Sigmoid activation function is used to generate an appearance activation map. ; Text-aware semantic sequence diagram The input convolution kernel size is K of L In the convolution layer, a Sigmoid activation function is used to generate a semantic activation map. ; Will and Average to get the fusion activation map , the calculation formula is: 。 9. The method for cross-modal video segment retrieval based on dual temporal graph according to claim 1, characterized in that: The overall loss function of the cross-modal video clip retrieval model is: in, To balance the importance of different loss terms, 、 and Respectively 、 and P The corresponding value in represents the number of finite candidate actions, is the number of valid candidate actions, It is the temperature parameter, which is set to 0.1 by default. The score is greater than the threshold , Activate the image for the current appearance Score greater than threshold The corresponding fragment features, Activate the image for the current appearance Score less than threshold The corresponding fragment features, for The central feature of is the intersection-over-union ratio of the matched video segment and the real segment, where the real segment is the video segment corresponding to the text description, and is the preset threshold.

Images