A video segment positioning method

Abstract

The application discloses a video segment positioning method, and relates to the technical field of multi-modal machine learning.The method comprises the following steps: inputting a target video into a positioning model, extracting visual features, audio features and query features of the target video; constructing an audio graph for the audio features, constructing a video graph for the visual features, fusing the audio graph and the video graph to obtain fused features; preliminarily interacting the fused features and the query features to obtain initial fused features; repeatedly performing deep interaction on the initial fused features to obtain final fused features; modeling the final fused module to obtain context information; and analyzing the context information to obtain the start time and the end time of a positioning segment.The model can more effectively optimize the use of visual features in a complex environment with blurred or incomplete visual information by using the feedback of audio information.

Description

Technical Field

[0001] This invention relates to the field of multimodal machine learning technology, and in particular to a method for locating video segments. Background Technology

[0002] Video clip localization enables users to quickly and accurately find video clips of interest using natural language descriptions, thereby improving the efficiency and accuracy of video retrieval. It has wide applications in scenarios such as video content management, video editing, and video surveillance. Combined with natural language query technology and user preference information, personalized video recommendations can also be achieved, improving user satisfaction and viewing experience. At the academic level, video clip localization has promoted interdisciplinary research in natural language processing and computer vision, and facilitated the development of multimodal machine learning. Researchers need to solve problems such as semantic understanding, video content analysis, and cross-modal interaction, thus driving the theoretical deepening of these fields.

[0003] Currently, the commonly used video segment localization method without candidate segments typically involves the following steps: First, preprocessing the input video and text query to extract relevant features. Next, cross-modal alignment or fusion of video and text features is performed to establish semantic relationships between them. Then, a temporal convolutional network is used to identify potential time periods in the video to capture the temporal information of the segments. Finally, based on the fused features, boundary regression is used to predict the start and end times of the target segment.

[0004] The shortcomings of the existing technology are that it focuses on visual information in the video segment localization task, ignores the accompanying audio information, and does not consider all factors, which leads to inaccurate localization results. Summary of the Invention

[0005] Therefore, it is necessary to provide a video segment localization method to address the aforementioned technical problems.

[0006] This invention provides a video segment localization method, including:

[0007] A localization model is constructed, which includes a feature encoding module, a fusion graph module, a progressive dynamic interaction module, and a fragment localization module; the feature encoding module includes a video encoding module and a query encoding module, the fusion graph module includes an audio graph module, a video graph module, and a graph fusion module, the progressive dynamic interaction module includes a shallow interaction module and multiple deep interaction modules, and the fragment localization module includes a Transformer encoder and a multilayer perceptron (MLP).

[0008] The video encoding module is connected to the inputs of the audio graph module and the video graph module; the query encoding module is connected to the input of the shallow interaction module; the outputs of the audio graph module and the video graph module are connected to the input of the graph fusion module, and the output of the graph fusion module is also connected to the input of the shallow interaction module; the output of the shallow interaction module is connected to multiple deep interaction modules, and the output of the previous deep interaction module is connected to the input of the next deep interaction module; the output of the last deep interaction module is connected to the input of the Transformer encoder, and the output of the Transformer encoder is connected to the Multilayer Perceptron (MLP).

[0009] The localization model is trained using historical videos. The target video is then input into the trained localization model. The visual and audio features of the target video are extracted using the video encoding module, and the query features of the target video are extracted using the query encoding module.

[0010] The audio graph module constructs an audio graph for audio features, the video graph module constructs a video graph for visual features, and the graph fusion module fuses the audio graph and video graph to obtain fused features.

[0011] The initial fused features are obtained by performing preliminary interaction between the fused features and the query features through a shallow interaction module; then, the initial fused features are further interacted with through multiple deep interaction modules, with the output of the previous deep interaction module serving as the input of the next deep interaction module, to obtain the final fused features.

[0012] The final fusion module is modeled using a Transformer encoder to obtain contextual information; this contextual information is then analyzed using a Multilayer Perceptron (MLP) to determine the start and end times (τ) of the localization segments. s ,τ e ).

[0013] Optionally, visual and audio features of the target video are extracted through a video encoding module, specifically including:

[0014] The original video V is segmented into a series of fixed-length segments, and the visual features X of each segment are extracted using a pre-trained 3D CNN. v ∈R T×d The pre-trained VGGish was used to extract the audio features X of each segment. a ∈R T×d Its formula is:

[0015] X v / a =Conv1d(W seg (f v (V)));

[0016] Where f(·) represents 3DCNN, W seg Represents the learnable fragment feature embedding matrix, and Conv1d(·) represents 1D convolution;

[0017] When the input video is short and the number of segments is less than T, missing parts are filled with zeros; positional encoding is introduced into the features of each input segment and mapped to dimension d through 1D convolution.

[0018] Optionally, query features of the target video are extracted through the query encoding module, which specifically include:

[0019] For a query containing N words, word-level features are generated through word embedding and character embedding.

[0020] Word-level features are passed to a self-weighted pooling layer to obtain sentence-level features Q. s ;

[0021] Sentence-level features Q s The feature is concatenated with the (n-1)th semantic phrase feature and then projected onto a mapping space to obtain the guiding vector. The calculation formula is as follows:

[0022] g n =ReLU(W g ([W gq Q s ;e n-1 ]));

[0023] The guiding vector Q g As a query vector, semantic entity-level features are extracted through an attention mechanism. Obtain the nth semantic phrase feature e n The calculation formula is as follows:

[0024] c n =softmax(w cT (tanh(W cg g n +W cq Q T )));

[0025]

[0026] in, W g ∈R d×2d and W gq ∈R d×d All are learnable weight matrices;

[0027] semantic phrase features e n As a query feature Qe .

[0028] Optionally, an audio graph is constructed for audio features using the audio graph module, and a video graph is constructed for visual features using the video graph module, specifically including:

[0029] In an audio graph, each node represents an audio segment, and each edge represents the similarity or correlation between audio segments; in a video graph, each node represents a video segment, and each edge represents the dependency between segments.

[0030] Cosine similarity is used to measure the similarity between segments, and the cosine similarity score is used as the weight of the edge. The formula for calculation is as follows:

[0031]

[0032] Where cos(·) is the cosine similarity function.

[0033] Optionally, the audio graph and video graph are merged through a graph fusion module, which specifically includes:

[0034] Calculating audio graph features using cosine similarity and video image features The similarity between them is used to obtain the alignment matrix M∈R. T×T Each element of the alignment matrix represents the correspondence between audio graph nodes and video graph nodes, and its calculation formula is:

[0035]

[0036] Where sim(·) is the calculation and The similarity score between nodes, where softmax(·) is a column-by-column operation;

[0037] Using the alignment matrix M Convert to The features of the nodes are weighted and combined according to the alignment matrix M, and the calculation formula is as follows:

[0038]

[0039] X G a With X v G The graph is merged into a fusion graph using a gating mechanism. The calculation formula is as follows:

[0040]

[0041] Where λ is a hyperparameter.

[0042] Optionally, the initial fused features can be deeply interacted with through multiple deep interaction modules, specifically including:

[0043] The initial input of the first layer of shallow interaction and The fusion diagrams are respectively and query feature Q e The interaction process of the i-th layer is represented as:

[0044]

[0045] The feedback mechanism of the i-th layer is represented as:

[0046]

[0047] Among them, f i-1 The fusion feature of the (i-1)th layer serves as the fusion graph. and query feature Q e Additional input to update features, G va (·) and G q (·) represent the fusion graphs. and query feature Q e The feature update function.

[0048] Optionally, the start and end times (τ) of the location segment are obtained. s ,τ e Specifically, it includes:

[0049] O = attn(Conv1d(X) vaq ));

[0050] τs,τe=MLP(FFN(O));

[0051] Where attn(·) represents a multi-head self-attention layer, Conv1d(·) represents a channel-separable 1D convolution, and FFN(·) represents a feedforward network.

[0052] The video segment localization method provided in this embodiment of the invention has the following advantages compared with the prior art:

[0053] This invention accurately extracts visual, audio, and query features from a target video using a video encoding module and a query encoding module. It then constructs intuitive audio and video graphs for the audio and visual features respectively using an audio graph module and a video graph module. These are then fused using a graph fusion module to form a fused feature. This process not only tightly integrates audio and visual information but also fully considers potential noise interference between them. This allows the model to optimize the use of visual features in complex environments where visual information is blurred or incomplete, by leveraging feedback from audio information. Therefore, this invention considers all factors comprehensively, resulting in more accurate video localization. Attached Figure Description

[0054] Figure 1 This is a diagram illustrating the overall model architecture of a video segment localization method provided in one embodiment.

[0055] Figure 2 This is a visual example of a video segment localization method provided in one embodiment. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0057] In one embodiment, a video segment localization method is provided, the method comprising:

[0058] 1 Overall Model

[0059] Build a localization model, such as Figure 1 As shown, the localization model includes a feature encoding module, a fusion graph module, a progressive dynamic interaction module, and a segment localization module. The feature encoding module includes a video encoding module and a query encoding module; the fusion graph module includes an audio graph module, a video graph module, and a graph fusion module; the progressive dynamic interaction module includes a shallow interaction module and multiple deep interaction modules; and the segment localization module includes a Transformer encoder and a multilayer perceptron (MLP).

[0060] The video encoding module is connected to the inputs of the audio graph module and the video graph module, and the query encoding module is connected to the input of the shallow interaction module. The outputs of the audio graph module and the video graph module are connected to the input of the graph fusion module, and the output of the graph fusion module is also connected to the input of the shallow interaction module. The output of the shallow interaction module is connected to multiple deep interaction modules, and the output of the previous deep interaction module is connected to the input of the next deep interaction module. The output of the last deep interaction module is connected to the input of the Transformer encoder, and the output of the Transformer encoder is connected to the Multilayer Perceptron (MLP).

[0061] 2. A Progressive Dynamic Interaction Network with Audio Supplement for Video Moment Localization (PDIN) aims to address the issues of complementarity between audio and visual information and model flexibility. It mainly consists of four parts: (1) a feature encoding module extracts visual features, audio features, and query features; (2) a fusion graph module supplements and fuses the video graph and audio graph; (3) a progressive interaction module captures the semantic relationship between the query and the video; and (4) a segment localization module determines the start and end times (τ) of the localized segment in the target video. s ,τ e ).

[0062] The localization model is trained using historical videos. The target video is then input into the trained model. A video encoding module extracts visual and audio features from the target video, while a query encoding module extracts query features. An audio graph module constructs an audio graph for the audio features, and a video graph module constructs a video graph for the visual features. A graph fusion module then fuses the audio and video graphs to obtain the fused features. A shallow interaction module performs initial interaction between the fused features and the query features to obtain the initial fused features. Multiple deep interaction modules then perform deep interaction on the initial fused features, using the output of the previous deep interaction module as the input of the next, to obtain the final fused features. A Transformer encoder models the final fused features to obtain contextual information. A Multilayer Perceptron (MLP) analyzes the contextual information to obtain the start and end times (τ) of the localized segment. s ,τ e ).

[0063] The specific implementation process includes:

[0064] (1) Data preprocessing

[0065] For a given original video, it is divided into two modalities: visual and audio, V = {X}. v ,X a},in and If the number of segments in the video is less than T, the missing parts are padded with zeros. For a given query text, the words in the query text are first represented as word embedding vectors using a pre-trained GloVe algorithm. Where q n It is the word embedding vector of the nth word, and N represents the length of the query text.

[0066] (2) Feature encoding

[0067] A. Video encoding

[0068] To extract visual and audio features from the video, the original video V is segmented into a series of fixed-length segments, and a pre-trained 3D CNN is used to extract the visual features X of each segment. v ∈R T×d The pre-trained VGGish was used to extract the audio features X of each segment. a ∈R T×d Its formula is:

[0069] X v / a =Conv1d(W seg (f v (V)));

[0070] Where f(·) represents 3DCNN, W seg Let represent the learnable segment feature embedding matrix, and Conv1d(·) denote a 1D convolution. When the input video is short and the number of segments is less than T, missing parts are filled with zeros; positional encoding is introduced into the features of each input segment and mapped to dimension d through a 1D convolution.

[0071] B. Query Code

[0072] For a query containing N words, an embedding representation of the query is first generated using pre-trained GloVe. Then, the query is encoded at multiple granularities to extract information at different levels and scales, thereby improving the model's performance in complex scenarios.

[0073] Specifically, for a query containing N words, word-level features are generated through word embedding and character embedding. Then, the word-level features are passed to a self-weighted pooling layer to obtain sentence-level features Q. s To extract N semantic phrase features from the query, sentence-level features Q are... s Concatenate with the (n-1)th semantic phrase feature to obtain the sentence-level feature Q. sThe feature is concatenated with the (n-1)th semantic phrase feature and then projected onto a mapping space to obtain the guiding vector. The calculation formula is as follows:

[0074] g n =ReLU(W g ([W gq Q s ;e n-1 ]));

[0075] The guiding vector Q g As a query vector, semantic entity-level features are extracted through an attention mechanism. Obtain the nth semantic phrase feature e n The calculation formula is as follows:

[0076] c n =softmax(w cT (tanh(W cg g n +W cq Q T )));

[0077]

[0078] in, W g ∈R d×2d and W gq ∈R d×d Both are learnable weight matrices. The semantic phrase features e... n As a query feature Q e .

[0079] (3) Fusion diagram

[0080] To gain a deeper understanding and process audio and visual features, the obtained audio features X a and visual features X v Audio and video graphs are constructed separately. In the audio graph, each node represents an audio segment, and each edge represents the similarity or correlation between audio segments. In the video graph, each node represents a video segment, and each edge represents the dependency between segments. This approach allows for the effective analysis and processing of complex structures and relationships in audio and video data. The final result is the audio graph features. and video image features

[0081]

[0082] Specifically, to construct a graph from audio and visual features, cosine similarity is used to measure the similarity between segments, and the cosine similarity score is used as the weight of the edge. The formula for calculating this is as follows:

[0083]

[0084] Where cos(·) represents the cosine similarity function.

[0085] However, since the cosine similarity value ranges between -1 and 1, and the edge weights should be non-negative, edge weights less than 0 were removed to ensure the graph's validity.

[0086] Audio information contains rich details, such as background music and environmental sound effects, which can reflect the emotions, plot development, and rhythm of video content. When visual information is missing or unrecognizable, audio information provides discriminative clues as a supplement. Through the complementarity of audio and visual information, the model can better capture and understand the dynamic changes of events, characters' emotions, and the background environment, improving the model's ability to understand and identify scenes. Therefore, a graph fusion method is adopted to supplement and fuse video and audio graphs, enhancing the model's understanding ability in complex scenes.

[0087] Specifically, audio graph features are calculated using cosine similarity. and video image features The similarity between them is used to obtain the alignment matrix M∈R. T×T Each element of the alignment matrix represents the correspondence between audio graph nodes and video graph nodes, and its calculation formula is:

[0088]

[0089] Where sim(·) represents the calculation and The similarity score between nodes is calculated using softmax(·), which is a column-wise operation.

[0090] Subsequently, the alignment matrix M is used to... Convert to The node features are weighted and combined according to the alignment matrix M. Specifically, The node features in are The weighted average of all node features is calculated using the alignment matrix M, where the weights are determined by the values ​​corresponding to the alignment matrix M. The formula is as follows:

[0091]

[0092]

[0093] Then, and The graph is merged into a fusion graph using a gating mechanism. It not only integrates visual and auditory information but also optimizes the representation of this information, making it suitable for more complex analytical tasks. Its calculation formula is:

[0094]

[0095] Where λ is a hyperparameter.

[0096] (4) Progressive dynamic interaction

[0097] To improve the effectiveness of cross-modal interaction, a progressive dynamic interaction method is proposed. This method first performs preliminary interaction on the data from each modality at a shallow level, and then gradually introduces feedback information from previous interactions at deeper levels, allowing the model to optimize its multimodal representation during continuous learning. The interaction at each layer depends on the results of the previous layer, thus forming a progressive interaction process. At each step in this process, the fusion weights and methods are adaptively adjusted to capture the most relevant information between modalities.

[0098] Specifically, the purpose of shallow interaction is to capture the basic correlations between modalities, ensuring the effectiveness of the initial fusion. A dot-product attention mechanism is used to perform initial interaction between the video and query features, generating interaction weights by calculating their similarity. The output of shallow interaction serves as the initial fusion feature and is passed to the deep interaction module. In the deep interaction stage, the shallow interaction mechanism is continued, while a reverse connection and feedback mechanism are introduced, feeding back the features obtained from the previous round of interaction as context vectors to the model's input layer. Through this process, the attention weights are dynamically adjusted to ensure that each layer of interaction accurately captures the correlations between modalities. During the layer-by-layer adjustment process, the model adaptively determines the focus of each layer of interaction, thereby more accurately locating the target segment.

[0099] The initial input of the first layer of shallow interaction and These are the fusion graph and the query feature Q, respectively. e The interaction process of the i-th layer can be represented as:

[0100]

[0101] Among them, W vaq W va and W q All are learnable weight matrices.

[0102] The feedback mechanism of the i-th layer can be represented as:

[0103]

[0104] Among them, f i-1 The fusion feature of the (i-1)th layer serves as the fusion graph. and query feature Qe Additional input to update features, G va (·) and G q (·) represent the fusion graphs. and query feature Q e The feature update function.

[0105] (5) Segment location

[0106] In obtaining fusion feature X vaq Then, the features are fed into a Transformer encoder, which models the features using a self-attention mechanism to capture long-range dependencies and contextual information. Finally, the fused features are processed by an MLP to predict the start and end times (τ) of the target segment. s ,τ e Specifically, it includes:

[0107] O = attn(Conv1d(X) vaq ));

[0108] τs,τe=MLP(FFN(O));

[0109] Where attn(·) represents a multi-head self-attention layer, Conv1d(·) represents a channel-separable 1D convolution, and FFN(·) represents a feedforward network.

[0110] By way of example, comparative tests of the present invention are provided.

[0111] 1. Training

[0112] Two loss functions are used to train the network: segment localization loss and regularization loss. The segment localization loss guides the model to accurately locate and query the corresponding video segments.

[0113] L reg =L1(τ' s -τ s )+L1(τ' e -τ e );

[0114] Where L1 represents the SmoothL1 distance, (τ s ',τ e ') indicates the start and end times of a real video clip.

[0115] To extract more diverse semantic entity-level features, regularization is used as the loss function, enabling the model to better capture the relationships between semantic entities during training and further improve its generalization ability.

[0116]

[0117] in It generates semantic-level entity features Q. e Attention weights at time, ||·|| denotes the Frobenius norm, and η is a hyperparameter. Let L represent the identity matrix. The overall loss function is L. reg and L e Addition of:

[0118]

[0119] 2. Experimental Setup

[0120] To verify the video clip localization performance of PDIN, the ActivityNet Captions and Charades-STA datasets were selected for experiments and analysis. For a fair comparison with previous work, a pre-trained C3D network was used to extract visual features for the ActivityNet Captions dataset, and VGGish was used to extract audio features. For the Charades-STA dataset, a pre-trained I3D network was used to extract visual features, and PANN was used to extract audio features. The feature dimension d was set to 512. For query encoding, all query words were lowercase converted and labeled, and then GloVe with dimension 300 was used to generate the query embedding representation. The initial learning rates for Charades-STA and ActivityNet Captions were set to 0.00015 and 0.0005, respectively. All experiments were conducted on an NVIDIA GeForce RTX 4090 graphics card in a PyTorch 1.12 environment, with parameter optimization performed using the Adam optimizer.

[0121] 3. Ablation test

[0122] To evaluate the effectiveness of different modules in PDIN, in-depth ablation studies were conducted on two challenging datasets, ActivityNet Captions and Charades-STA, and the contributions of different components to the model were analyzed to validate its effectiveness. Specifically, by removing one module at a time, ablation variants of the PDIN model were generated as follows:

[0123] ①: The graph fusion module has been removed. The supplementation of audio features to visual features is not considered. Only visual features and query features are used for progressive dynamic interaction.

[0124] ②: Audio features were removed, visual features were transformed into a graph structure, and progressive dynamic interaction was performed with query features.

[0125] ③: The progressive dynamic interaction module has been removed. After passing through the graph fusion module, video features and query features are directly fused by splicing.

[0126] ④: The graph fusion module and the progressive dynamic interaction module have been removed. Audio features and visual features are directly added together and then directly fused with query features through splicing.

[0127] ⑤: A video clip localization network with complete audio supplementation and progressive dynamic interaction.

[0128] Table 1 presents the results of the ablation experiments. The results show that on the ActivityNet Captions and Charades-STA datasets, the complete model outperforms other variant models on all evaluation metrics, indicating that audio features, the graph fusion module, and the progressive dynamic interaction module play a positive role in the video segment localization task.

[0129] First, comparing Model ⑤ and Model ①, we can observe improvements of 1.43% and 2.29% in R@1 and IoU = 0.7, respectively, for the two datasets. This finding indicates that the graph fusion module plays a crucial role in cross-modal information fusion. By fusing visual and audio features, this module enhances the complementarity of cross-modal information, thereby improving the overall performance of the model.

[0130] Secondly, compared to model ①, model ② improved the R@1 and IoU=0.7 metrics by 0.47% and 1.2% respectively on both datasets. However, compared to model ⑤, it decreased by 0.96% and 1.13% respectively on both datasets. This indicates that audio features can indeed provide meaningful supplementary information to visual features, resulting in superior overall performance and helping to better capture the relevance between videos and queries.

[0131] Furthermore, compared to Model ③, Model ⑤ improved by 1.12% and 2.18% respectively on R@1 and IoU = 0.7 for the two datasets. This finding indicates that the progressive dynamic interaction module plays a crucial role in improving the accurate matching and semantic association of multimodal information. By progressively deepening multimodal interaction and optimizing feature fusion, this module gradually enhances the matching degree between videos and queries, significantly improving the model's performance.

[0132] Finally, the comparison between Model ④ and Model ⑤ shows that the Progressive Dynamic Interaction module and the Graph Fusion module are crucial for retrieval. They improved performance by 1.8% and 3.23% on the two datasets, respectively. This is because the Progressive Dynamic Interaction module better captures the deep semantic relationship between the query and the video, while the Graph Fusion module effectively combines visual and audio features, thereby enhancing the expressive power of multimodal features.

[0133] Table 1 Evaluation Results of Ablation Test Indicators

[0134]

[0135] 4. Qualitative Results Analysis

[0136] like Figure 2 As shown, PDIN demonstrates superior performance, accurately retrieving moments most relevant to the language query, even those visually very similar. Qualitative results from models ①, ③, and the complete model mentioned in the ablation experiments were also compared. It can be observed that models ① and ③ exhibit significant errors. Model ①, after removing the graph fusion module, no longer considers the supplementation of visual features by audio features, resulting in an inability to accurately distinguish differences between different objects in the video. Model ③, due to the removal of the progressive dynamic interaction module, suffers from insufficient semantic understanding and struggles to effectively capture complex cross-modal associations. Overall, the experimental results indicate that the combined application of the progressive dynamic interaction module and the graph fusion module is crucial for segment retrieval. PDIN not only performs excellently in various scenarios but also maintains high accuracy when handling visually similar but semantically different moments.

[0137] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A method of video segment positioning, characterized by, The application relates to a positioning model, and the positioning model comprises a feature coding module, a fusion graph module, a progressive dynamic interaction module and a segment positioning module; the feature coding module comprises a video coding module and a query coding module, the fusion graph module comprises an audio graph module, a video graph module and a graph fusion module, the progressive dynamic interaction module comprises a shallow interaction module and a plurality of deep interaction modules, and the segment positioning module comprises a Transformer encoder and a multilayer perceptron (MLP). The video coding module is connected with the input ends of the audio graph module and the video graph module, the query coding module is connected with the input end of the shallow interaction module, the output ends of the audio graph module and the video graph module are connected with the input end of the graph fusion module, the output end of the graph fusion module is also connected with the input end of the shallow interaction module, the output end of the shallow interaction module is connected with the plurality of deep interaction modules, the output end of a previous deep interaction module is connected with the input end of a next deep interaction module, the output end of the last deep interaction module is connected with the input end of the Transformer encoder, and the output end of the Transformer encoder is connected with the multilayer perceptron (MLP). The positioning model is trained through historical videos, the target video is input into the trained positioning model, the visual features and the audio features of the target video are extracted through the video coding module, and the query features of the target video are extracted through the query coding module. The audio graph module is used for constructing an audio graph for the audio features, the video graph module is used for constructing a video graph for the visual features, and the audio graph and the video graph are fused through the graph fusion module to obtain fusion features. The fusion features and the query features are preliminarily interacted through the shallow interaction module to obtain initial fusion features, and the initial fusion features are deeply interacted through the plurality of deep interaction modules, the output of a previous deep interaction module is taken as the input of a next deep interaction module, and final fusion features are obtained. The video coding module is used for extracting the visual features and the audio features of the target video, and the method specifically comprises the following steps. The final fusion module is modeled by a Transformer encoder to obtain context information, and the context information is analyzed by a multi-layer perceptron (MLP) to obtain the start time and end time (τ s ,τ e ) of the positioning segment.

2. A video segment positioning method as claimed in claim 1, characterized in that, When the input video is short and the number of segments is less than T, the missing part is zero-filled, position encoding is introduced into the features of each input segment, and the position encoding is mapped to a dimension d through 1D convolution. The original video V is segmented into a series of segments with fixed length, and the visual features X of each segment are extracted by using a pre-trained 3D CNN v ∈R T×d The audio features X of each segment are extracted by using a pre-trained VGGish a ∈R T×d The formula is: X v / a = Conv1d(W seg (f v (V))) ; where f(·) denotes a 3D CNN, W seg denotes a learnable segment feature embedding matrix, and Conv1d(·) denotes a 1D convolution; The query coding module is used for extracting the query features of the target video, and the method specifically comprises the following steps.

3. The video clip positioning method of claim 1, wherein, The audio graph module is used for constructing an audio graph for the audio features, and the video graph module is used for constructing a video graph for the visual features, and the method specifically comprises the following steps. For a query containing N words, word-level features are generated by word embeddings and character embeddings The word-level features are passed to a self-weighted pooling layer to obtain sentence-level features Q s ; The sentence-level feature Q s is concatenated with the (n-1)th semantic phrase feature, and then projected into a mapping space to obtain a guide vector The calculation formula is: g n = ReLU(W g ([W gq Q s ; e n-1 ]))); The guide vector Q g As a query vector, semantic entity-level features are extracted through an attention mechanism The nth semantic phrase feature e n The calculation formula is: c n = softmax(w cT (tanh(W cg g n + W cq Q T ))) ; wherein, and W gq ∈R d×d are all learnable weight matrices; The semantic phrase feature e n As query feature Q e .

4. The video clip positioning method of claim 1, wherein, In the audio graph, each node represents an audio segment, and each edge represents the similarity or correlation between the audio segments; in the video graph, each node represents a video segment, and each edge represents the dependency between the segments. The cosine similarity is used to measure the similarity between the segments, and the cosine similarity score is taken as the weight of the edge, and the calculation formula is as follows: Wherein, cos(·) is a cosine similarity function. The graph fusion module is used for fusing the audio graph and the video graph, and the method specifically comprises the following steps.

5. The video clip positioning method of claim 1, wherein, Wherein, lambda is a hyperparameter. Calculate the similarity between audio graph features X and video graph features X by cosine similarity G a and video graph features X v G Get alignment matrix M ∈ R T×T Each element of the alignment matrix represents the correspondence between the audio graph node and the video graph node, and its calculation formula is: where sim(·) is the computation of the node similarity score between and softmax(·) is the column-wise operation; Transforming to The features of the nodes are combined with weights according to the alignment matrix M, which is calculated as Will With Fusion into fusion graph through gating mechanism The calculation formula is: The plurality of deep interaction modules are used for deeply interacting with the initial fusion features, and the method specifically comprises the following steps.

6. The video clip positioning method of claim 1, wherein, The feedback mechanism of the i-th layer is represented as follows: Initial input to the first layer of shallow interactions and are respectively fusion graph and query feature Q e The interaction process of the i-th layer is represented as: where W vaq , W va , and W q are learnable weight matrices; ​ where f i-1 is the fused feature of the i-1th layer, which is updated as the fused graph and the query feature Q e . G va (·) and G q (·) are the feature update functions of the fused graph and the query feature Q e , respectively.

7. The video clip positioning method of claim 1 wherein, The start time and end time (τ s ,τ e ) of the positioning segment are obtained, which specifically includes: O = attn(Conv1d(X vaq )); τs,τe= MLP(FFN(O)); where attn(·) denotes a multi-head self-attention layer, Conv1d(·) denotes a channel separable 1D convolution, FFN(·) denotes a feed-forward network, X vaq denotes the fused features.

Images