Natural language video time retrieval method based on multi-modal fusion

Abstract

The invention discloses a natural language video moment retrieval method based on multi-modal fusion, which belongs to the technical field of data identification, and comprises the following steps: acquiring a data set for natural language video moment positioning; a channel sensing multi-scale time sequence modeling module MD1 is constructed; obtaining a text encoder; a multi-head decoupling type cross-modal feature fusion module MD2 is constructed, and a semantic modulation gating convolution module MD3 is constructed; constructing a natural language time retrieval network based on the MD1, the MD2, the text encoder and the MD3, and training the natural language time retrieval network into a natural language time retrieval model; the method is used for natural language moment detection of a to-be-detected video. According to the method, visual feature response related to query semantics can be remarkably enhanced through MD1, cross-modal fine-grained integration is achieved in multiple semantic subspaces through MD2, it is ensured that candidate fragment scores highly depend on the query semantics through MD3, and the positioning precision and robustness can be remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data recognition, and particularly relates to a natural language video time point retrieval method based on multi-modal fusion. BACKGROUND

[0002] With the explosive growth of Internet video data and the rapid development of multi-modal artificial intelligence technology, natural language-based video time localization (NLVL) has become a core task of cross-modal understanding and intelligent video analysis. This task aims to accurately locate specific segments that match the semantics through text queries from uncut videos. This task is not only the cornerstone of intelligent video summarization, interactive search, video question answering and other applications, but also an important research direction for promoting the practicality of multi-modal artificial intelligence.

[0003] The natural language video localization task aims to accurately locate the specific time from an uncut video that matches the given natural language description. Specifically, given a natural language query, such as "a woman drinks her coffee again", the goal of this task is to automatically locate and extract the exact time when the woman drinks coffee again from the video. This task requires the model not only to understand the video content, but also to understand the semantics of the natural language query and establish an accurate correspondence between the two.

[0004] Currently, deep learning-based NLVL methods are the mainstream methods for solving the natural language video time localization task. Early natural language video time localization research mainly adopts a technical route combining sliding window candidate generation and feature-level cross-modal fusion. Representative works such as CTRL generate candidate segments through sliding windows and use Hadamard product to realize video-text feature fusion. Subsequent time graph modeling candidate generation methods such as 2D-TAN construct a two-dimensional time adjacency graph to capture local time relationships, while multi-scale expansion methods MS-2D-TAN optimize candidate generation by reducing computational complexity through sparse sampling. Cross-modal fusion technology has evolved from feature concatenation, attention mechanism to decoupled fusion. VSLNet uses span-wise self-attention to model video internal dependencies, and MMN enhances cross-modal association through memory networks.

[0005] Despite the remarkable progress made by deep learning-based NLVL methods, there are still three problems with existing models: (1) Difficulty in modeling complex dynamic timing: While multi-scale two-dimensional timing graphs like MS-2D-TAN can cover candidate segments of different lengths, they do not distinguish the importance of different frames during the generation of candidate segments, leading to interference from redundant features (such as background noise) in long videos, which hinders the model's ability to capture key timing information; (2) Insufficient fine-grained cross-modal fusion: Traditional methods, such as element-wise multiplication or simple concatenation, struggle to model the complex interaction between language descriptions and video segments, especially for implicit spatiotemporal logic (such as "play again" or "open the door before sitting down") and multi-object interaction scenarios (such as "hand the red-clad person a cup"); (3) Weak query dependence: Existing methods lack sufficient correlation between candidate segment scoring and language descriptions, leading to false positives for complex queries.

[0006] Noun explanation: 2D Temporal Adjacency Graph (2D-TAG) is a new structured representation method in video localization, which models the temporal and semantic relationships between video segments as a graph structure, significantly improving the accuracy and robustness of cross-modal alignment. Unlike traditional methods that treat video as a linear sequence of frames and only model local temporal dependencies (such as RNN, Transformer), 2D-TAG constructs a two-dimensional graph to model the temporal relationships between different video segments. Its method is: candidate segments in the video are represented as nodes on a two-dimensional graph, each node corresponds to a candidate segment, defined by its start time and end time. The edges of the graph represent the temporal adjacency relationships between segments (e.g., overlap, adjacency, inclusion, etc.). By using graph convolution and other methods, the context relationships between segments are modeled to improve localization accuracy. In 2D-TAG, the vertical and horizontal coordinates represent the start and end times of the video segments, respectively. The point (i, j) in the i-th row and j-th column of the graph represents a video segment that starts at time i and ends at time j (also known as a candidate video segment). The localization principle is: after convolution operation on the 2D-TAG, a two-dimensional score graph is finally output, each cell represents the matching degree of the corresponding segment with the query sentence, and the segment with the highest score is the predicted result.

[0007] GloVe (Global Vectors) model: an unsupervised word embedding model that takes input text and outputs a d-dimensional vector for each word in the text (d = 50 / 100 / 200 / 300, etc.).

[0008] BiLSTM (Bidirectional Long Short-Term Memory Network) is an improved recurrent neural network (RNN) that combines forward and backward LSTM networks to capture bidirectional dependencies in sequence data.

[0009] GloVe+BiLSTM: GloVe is responsible for quickly transforming discrete words into high-quality, low-dimensional vectors with global statistical properties; BiLSTM is responsible for capturing bidirectional contextual dependencies in these vector sequences. Summary of the Invention

[0010] The purpose of this invention is to provide a natural language video moment retrieval method based on multimodal fusion that addresses the aforementioned difficulties in complex dynamic temporal modeling, the fine-grained semantic gap between video and text, and the insufficient dependence of candidate segment scoring on language description.

[0011] To achieve the above objectives, the technical solution adopted by the present invention is as follows: a natural language video moment retrieval method based on multimodal fusion, comprising the following steps; S1, Obtain the dataset for natural language video time localization, where each sample includes video V, several target segments with time annotations, and query text corresponding to each target segment; S2, constructs a multi-scale temporal modeling module MD1 with one-channel perception, including a preprocessing unit, an improved stacked convolution unit, and a two-dimensional temporal adjacency graph generation unit; The preprocessing unit is used to divide the video V into N non-overlapping segments, extract features from the nth segment, and adjust its dimension to d. v To obtain fragment features Then, the video features F are formed. in , , d v The preset number of output channels; The improved stacked convolutional unit is used as input F in Output M convolutional video features at different scales, where the convolutional video feature at the m-th scale is F. m,out ; The two-dimensional temporal adjacency graph generation unit is used to generate F based on... m,out Generate an initial two-dimensional temporal adjacency graph Then Adjust to Size and A two-dimensional temporal adjacency graph F is constructed by stitching the graphs along the channel dimension. m F m The point at position (u,v) represents a candidate video segment with start and end times of u and v, respectively. S3, Obtain a text encoder to input the query text and generate word-level text features. and global text features T global , Tword,l To query the word-level text features of the l-th word in the text; S4, construct a multi-head decoupled cross-modal feature fusion module MD2, used to fuse F m Reconstructed into two-dimensional temporal features, with T word and T global Perform fusion to generate fusion feature F m,fused ; S5, construct the semantic modulation gated convolutional module MD3; The semantic modulation gated convolution module is used for... Shape adjusted to N m ×N m ×d v This results in a reshaped adjacency graph. Then, the dynamic gating weight G and gating feature F are generated according to the following formula. gated The gated output feature F of the semantic modulation gated convolution module m,SG ; , , , In the formula, ReLU(∙) is the ReLU function, GAP(∙) is the global average pooling function, and W s W v These are the third and fourth weight matrices, respectively, in Conv2D. r For dilated convolution with dilation rate r, ; S6. Construct the natural language time-of-flight retrieval network and loss function, and adjust the parameters of the natural language time-of-flight retrieval network to minimize the loss, thus obtaining the natural language time-of-flight retrieval model; The natural language time-series retrieval network includes MD1, a text encoder, MD2, MD3, and a prediction head; The video V of the sample is fed into MD1, and the query text is fed into the text encoder. The outputs of the two are then processed by MD2 and MD3 to obtain M gated output features F. 1,SG ~F M,SG For each F m,SG The prediction head generates a corresponding two-dimensional fractional graph I. m,SG Collect I m,SG The set of valid fractions ,in For P m The i-th valid score, C m For P m The total number of valid scores will The corresponding candidate video segments are marked as ; The loss function is obtained from the following formula: , In the formula, for Supervisory labels, for The IoU value at real time, when hour, ,otherwise ; S7 uses a natural language time-lapse retrieval model to detect the video to be identified.

[0012] Preferably, the dataset includes the ActivityNet Captions dataset, the Charades-STA dataset, and the TACoS dataset. In S12, a pre-trained 3D convolutional network is used to extract features from the segments.

[0013] Preferably, in S2, the improved stacked convolutional unit includes a stacked convolutional unit composed of Z stacked convolutional layers. An enhancement unit is sequentially placed at the end of each convolutional layer. The enhancement unit includes a sparse sampling layer, a one-dimensional convolutional block, and a channel attention layer. When the input F... in The output F of the z-th convolutional layer conv,z Enhanced features are obtained by sequentially applying sparse sampling, one-dimensional convolution, and channel attention. Then, it is fed into the next convolutional layer; The improved stacked convolutional unit is divided into M stages, and the enhanced features are obtained at the end of each stage as enhanced video features, where the enhanced video features of the m-th stage are labeled F. m,out .

[0014] As a preferred embodiment, for the Z-layer convolutional unit of the improved stacked convolutional unit, Z = (K+1)A / 2, the kernel size of the first layer is 1 and the stride is 1, the kernel size of the (j1+1)A / 2 layer is 3 and the stride is 2, and the kernel size of the remaining layers is 2 and the stride is 1. A and K are the number of basic anchor points and the number of scales for video time-localization, respectively, and j1 is the number of convolutional layers, 1≤j1≤K-1.

[0015] Preferably, for the channel attention layer corresponding to the z-th convolutional layer, the output F of the one-dimensional convolutional block is obtained. c,z , for F c,z Perform global average pooling to generate the first channel statistical vector. Second channel statistical vector And generate enhanced features according to the following formula. ; , , where FFN(·) is a feed-forward network, σ(·) is a Sigmoid function, W c is a channel attention weight, ⊙ is an element-wise multiplication, and .

[0016] Preferably, the multi-head decoupled cross-modal feature fusion module MD2 comprises a serialization layer, a video-guided multi-head cross-attention layer, a text-guided multi-head cross-attention layer, and a fusion layer. The serialization layer is configured to serialize F m to a two-dimensional time sequence feature F rm , 、 N m is the length of F m ; H m is the sequence length of F rm . The video-guided multi-head cross-attention layer performs cross-attention on F rm as a K matrix and a V matrix, T word as a Q matrix to obtain a first attention feature F temp . The text-guided multi-head cross-attention layer performs cross-attention on F temp as a K matrix and a V matrix, F rm as a Q matrix to obtain a second attention feature F stage . The fusion layer is configured to generate a fusion feature F m,fused according to the following formula: , where σ(·) is a Sigmoid activation function, ⊙ is an element-wise multiplication, W g , W v are respectively a first weight matrix and a second weight matrix, and LayerNorm(·) is a layer normalization operation.

[0017] Preferably, in S3, the global text feature T global is calculated according to the following formula: , where L is the total number of words in the query text, T word,l is the l-th element in T word , and represents the word-level text feature of the l-th word in the query text.

[0018] Preferably, the effective score is a value other than zero in I m,SG .

[0019] As preferred, S7 is specifically: obtaining a to-be-identified video V', obtaining M two-dimensional score maps through MD1-MD3 of the natural language time point retrieval model and the prediction head, adjusting to the same size to fuse into a fused two-dimensional score map, and outputting one or more candidate video clips with the highest effective score according to the fused two-dimensional score map; The fusing is to fuse M two-dimensional score maps I 1,SG ~I M,SG Adjusting to the same size, a corresponding adjusted map I 1,SG ~I M,SG is obtained. mix A blank fused two-dimensional score map I mix is constructed. 1,SG ~I M,SG The maximum value of the position (u, v) in I mix is filled into I in .

[0020] As preferred, the text encoder is composed of a pre-trained GloVe model and a BiLSTM.

[0021] The design idea of the application is: 1. For the channel-aware multi-scale time sequence modeling module MD1: the purpose is to effectively extract key information and avoid the interference of redundant information on target event recognition.

[0022] (1) About the preprocessing unit: used for processing the video into , containing N segment features , facilitating subsequent feature extraction and related processing.

[0023] (2) About the improved stacked convolution unit: it learns from the structure of the existing stacked convolution, is also stacked by Z convolution layers, is also divided into M stages, and is used for inputting F in , respectively in the first stage to the M different scale video features, but the application is improved on this basis: (2.1) considering that the output of each layer of the convolution layer is a candidate time point feature, these time points have the same length but different starting times. In order to alleviate the problem of insufficient memory when the number of current video segments is too large, a sparse sampling strategy is adopted when generating candidate time point features, a number of basic anchor points A and a number of scales K are defined, and different convolution kernels are set in different convolution layers, so as to realize sparse sampling. When sampling, when the number of segments is less than A, all possible time points are enumerated as candidates; when the number of segments is greater than A, only the time points meeting the condition G (a, b) are selected as candidates: where a and b are segment indices, G(a, b) = 1, then select the moment as a candidate, otherwise discard. This strategy achieves dense sampling of moments with short duration, and gradually increases the sampling interval as the moment length increases. (2.2) Set a one-dimensional convolution block and a channel attention layer after sparse sampling. The one-dimensional convolution block is a combination of convolution, batch normalization and activation function, where the activation function uses the hyperbolic tangent function Tanh. The channel attention layer uses a two-way pooling channel attention to dynamically generate and adjust the channel attention weight W c To strengthen the key features, get enhanced features, through the processing of the channel attention layer, the key channel can be given high weight, so that the model pays more attention to the information highly related to the query in the video, while reducing the interference of a large amount of redundant information in the video to the model.

[0024] (3) About the two-dimensional time sequence adjacency graph generation unit: for M enhanced video features F 1,out ~F M,out , first generate an initial two-dimensional time sequence adjacency graph based on existing technology ~ Each two-dimensional time sequence adjacency graph has different scales. At this time, adjust to size and along the channel dimension to splice into a two-dimensional time sequence adjacency graph F m . This method can preserve low-scale fine-grained information (such as short-time action details) and high-scale semantic abstraction (such as long-time event context), and improve the model's ability to capture multi-granularity temporal patterns.

[0025] 2, For the multi-head decoupled cross-modal feature fusion module MD2: the purpose is to design an effective interaction mechanism to make the semantic association between video and text closer and improve the accuracy of retrieval and positioning. MD2 first reshapes F m through the serialization layer, so that it can be fused with the word-level text feature T word and the global text feature T global , and then through the specified Q matrix, K matrix and V matrix, and two layers of cross-attention operations to fuse F mr , T word and T global , to get the fused feature F m,fused . Through two-stage fusion, video and query text can fully focus on the information of the other modality, thereby realizing the delicate fusion of the two modalities, while each attention head automatically focuses on different semantic dimensions, and the model decouples the interaction mode of cross-modal features in multiple subspaces. The attention mechanism directly associates distant segments in the video, breaking through the limitations of the local receptive field of traditional convolution. Provide high discriminative cross-modal joint representation for the subsequent semantic modulation gating convolution module MD3.

[0026] 3. For the semantic modulation gated convolution module MD3; pass T global Projecting the text into the gated space yields the projected global text features g. text And then according to Generate channel-level context vectors And then according to Generate a dynamic gating weight G, whose value is in the interval [0, 1], according to... The dynamic gating weight G and Element-wise multiplication suppresses irrelevant regions and enhances relevant query features, finally using... , for F gated Introducing residual connections preserves original feature information. This alleviates the gradient vanishing problem. Gated feature F gated Driven by global textual semantics, the model's response is strictly aligned with the query intent. Fine-grained feature modulation is achieved by combining local video context (such as action continuity) with global textual constraints (such as event causal chains). This module uses a language-driven dynamic gating mechanism to deeply embed query semantics into the video feature learning process, ensuring high-precision localization.

[0027] Compared with the prior art, the advantages of the present invention are as follows: (1) A channel-aware multi-scale temporal modeling module MD1 was designed to overcome the shortcomings of existing technologies, such as the difficulty in capturing complex temporal relationships between video segments and the presence of a large amount of redundant background information in long videos. By improving the stacked convolutional unit to assign high weights to key channels, the model can focus more on the highly relevant information to the query in the video, while reducing the interference of a large amount of redundant information in the video on the model. A two-dimensional temporal adjacency graph F is generated by the two-dimensional temporal adjacency graph generation unit. m This method differs from existing technologies that generate single-scale two-dimensional temporal adjacency graphs. It can preserve both low-scale fine-grained information (such as short-term action details) and high-scale semantic abstraction (such as long-term event context), thereby improving the model's ability to capture multi-granular temporal patterns.

[0028] (2) A multi-head decoupled cross-modal feature fusion module MD2 was designed: it allows video and text to fully pay attention to each other's modal information, thereby achieving fine fusion of the two modalities, making the semantic relationship between the two closer, and effectively improving the accuracy of retrieval and positioning.

[0029] (3) A semantic modulation gated convolution module MD3 is designed, which combines local video context (such as action continuity) and global text constraints (such as event causal chain), and realizes fine-grained feature modulation. This module uses a language-driven dynamic gating mechanism to deeply embed query semantics into the video feature learning process, providing a guarantee for the final high-precision positioning.

[0030] In summary, the present application is based on the above-mentioned improved structure of a new natural language time retrieval network, which can significantly enhance the visual feature response related to the query semantics through MD1, realize cross-modal fine-grained integration in multiple semantic subspaces through MD2, and ensure that the candidate segment score is highly dependent on the query semantics through MD3, which can significantly improve the positioning accuracy and robustness. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 The flowchart of the present application; Figure 2 The natural language time retrieval network structure diagram of the present application; Figure 3 The improved stack convolution unit structure diagram; Figure 4 The multi-head decoupled cross-modal feature fusion module MD2 structure diagram; Figure 5 The feature change diagram in the video processing process of the present application; Figure 6 The visualization comparison diagram of a video after ablation experiment; Figure 7 The visualization comparison diagram of another video after ablation experiment. DETAILED DESCRIPTION

[0032] The present application will be further described below in conjunction with examples and drawings.

[0033] Example 1: Referring to Figures 1-4 A natural language video time retrieval method based on multi-modal fusion, comprising the following steps: S1, obtaining a data set for natural language video time positioning, wherein each sample includes a video V, a plurality of target segments with time annotation, and a query text corresponding to the target segment; S2, constructing a channel-aware multi-scale time sequence modeling module MD1, including a preprocessing unit, an improved stack convolution unit, and a two-dimensional time sequence adjacency graph generation unit; The preprocessing unit is used to divide the video V into N non-overlapping segments, and the nth segment is subjected to feature extraction and dimension adjustment to d v , to obtain the segment feature , and then form the video feature F in , , , d v is a preset output channel number; The improved stacked convolution unit is used for inputting F in , and outputting M different scale convolution video features, wherein the mth scale convolution video feature is F m,out ; The two-dimensional time sequence adjacency graph generation unit is used for generating an initial two-dimensional time sequence adjacency graph F m,out ; and then adjusting F to a size of N ×N and splicing F m along the channel dimension into a two-dimensional time sequence adjacency graph F m , wherein a point at a position (u, v) in F m is a candidate video segment representing start and end times of u and v respectively; S3, obtaining a text encoder used for inputting a query text to generate word-level text features and global text features T global , , T word,l is a word-level text feature of an lth word in the query text; S4, constructing a multi-head decoupled cross-modal feature fusion module MD2 used for reshaping F m into a two-dimensional time sequence feature, fusing the two-dimensional time sequence feature with T word and T global , and generating a fusion feature F m,fused ; S5, constructing a semantic modulation gating convolution module MD3; The semantic modulation gating convolution module is used for adjusting the shape of F to N m ×N m ×d v , obtaining a reshaped adjacency graph F , and generating a dynamic gating weight G, a gating feature F gated , and a gating output feature F m,SG of the semantic modulation gating convolution module according to the following formula: , , , , wherein ReLU(·) is a ReLU function, GAP(·) is a global average pooling, W s and W v are respectively a third weight matrix and a fourth weight matrix, Conv2D r is a dilated convolution with a dilution rate r, ; S6, constructing a natural language time retrieval network and a loss function loss, and adjusting the natural language time retrieval network parameters to obtain a natural language time retrieval model by minimizing the loss; The natural language time retrieval network comprises MD1, a text encoder, MD2, MD3, and a prediction head. The video V of the sample is input into MD1, and the query text is input into the text encoder. After the two outputs pass through MD2 and MD3, M gated output features F are obtained. 1,SG ~F M,SG For each F m,SG , the prediction head generates a corresponding two-dimensional score map I m,SG The effective scores in I m,SG are collected to form a set P , wherein is the i-th effective score in P m , C m is the total number of effective scores in P m , and is marked as ; The loss function loss is obtained according to the following formula: , In the formula, is the supervision label of , and is the IoU value of and the real time, when , , otherwise ; S7, using the natural language time retrieval model to detect the video to be recognized.

[0034] The data set comprises an ActivityNet Captions data set, a Charades-STA data set, and a TACoS data set. In S12, a pre-trained 3D convolutional network is used to extract features of the segments.

[0035] In S2, the improved stacked convolution unit comprises a stacked convolution unit stacked by Z convolution layers, and an enhancement unit is arranged at the end of each convolution layer. The enhancement unit comprises a sparse sampling layer, a one-dimensional convolution block, and a channel attention layer. When the input F in , the output F conv,z of the z-th convolution layer is obtained after being sequentially subjected to sparse sampling, one-dimensional convolution, and channel attention, and the enhanced feature F is input into the next convolution layer. The improved stacked convolution unit is divided into M stages, and the enhanced feature at the end of each stage is obtained as a strengthened video feature, wherein the strengthened video feature of the m-th stage is marked as Fm,out .

[0036] For the Z-layer convolutional layer of the improved stacked convolutional unit, Z=(K+1)A / 2, the first layer has a convolutional kernel size of 1 and a step size of 1, the (j1+1)A / 2 layer has a convolutional kernel size of 3 and a step size of 2, and the remaining layers have a convolutional kernel size of 2 and a step size of 1, A and K are respectively the number of basic anchor points and the number of scales of video time positioning, j1 is a convolutional layer variable, and 1≤j1≤K-1.

[0037] For the channel attention layer corresponding to the zth convolutional layer, the output F c,z of the one-dimensional convolutional block is obtained. c,z Global average pooling is performed on F c,z to generate a first channel statistical vector and a second channel statistical vector , and an enhanced feature F is generated according to the following formula. , , In the formula, FFN(∙) is a feedforward network, σ(∙) is a Sigmoid function, W c is a channel attention weight, is an element-wise multiplication, and .

[0038] The multi-head decoupling cross-modal feature fusion module MD2 includes a serialization layer, a video-guided multi-head cross-attention layer, a text-guided multi-head cross-attention layer, and a fusion layer. The serialization layer is configured to reshape F m into a two-dimensional time sequence feature F rm , , N m is the length of F m ; H m is the sequence length of F rm . The video-guided multi-head cross-attention layer takes F rm as a K matrix and a V matrix, takes T word as a Q matrix, and performs cross-attention to obtain a first attention feature F temp . The text-guided multi-head cross-attention layer takes F temp as a K matrix and a V matrix, and takes F rm as a Q matrix, and performs cross-attention to obtain a second attention feature F stage . The fusion layer is configured to generate a fusion feature F m,fused according to the following formula. , wherein, σ(·) is a Sigmoid activation function, is an element-wise multiplication, W g , W v are a first weight matrix and a second weight matrix respectively, and LayerNorm(·) is a layer normalization operation.

[0039] In S3, the global text feature T global is calculated according to the following formula: , wherein, L is the total number of words in the query text, T word,l is the l-th element in T word , and represents the word-level text feature of the l-th word in the query text.

[0040] The effective score is a non-zero value in I m,SG .

[0041] S7 specifically comprises: obtaining a to-be-recognized video V', obtaining M two-dimensional score maps through the MD1-MD3 of the natural language time point retrieval model and the prediction head, adjusting to the same size to fuse into a fused two-dimensional score map, and outputting one or more candidate video segments with the highest effective score according to the fused two-dimensional score map; The fusion comprises: adjusting M two-dimensional score maps I 1,SG -I M,SG to the same size to obtain corresponding adjusted maps I 1,SG -I M,SG , constructing a blank fused two-dimensional score map I mix , and filling the maximum value of the position (u, v) in I mix -I 1,SG -I M,SG into I mix .

[0042] The text encoder is composed of a pre-trained GloVe model and a BiLSTM.

[0043] Embodiment 2: Refer to Figures 1-5 , on the basis of Embodiment 1, the improved stacked convolution unit is improved, the number of basic anchor points A of the video time point positioning is 6, the number of scales K is 3, then Z=(K+1)A / 2=12, a total of 12 convolution layers are stacked, the 12 convolution layers are divided into M stages, assuming that M=3, the first 6 convolution layers are stage 1, the 7-9 convolution layers are stage 2, and the 9-12 convolution layers are stage 3. For the 12 convolution layers, the size of the first convolution kernel is 1 and the step is 1, the size of the (j1+1)A / 2-th convolution kernel is 3 and the step is 2, and the size of the remaining convolution kernel is 2 and the step is 1. The value of j1 is selected between 1 and K-1.

[0044] Figure 5 The changes of each feature when the video is processed by the application are shown when M=3. Figure 5 In the middle, the video is input into the channel-aware multi-scale temporal modeling module MD1, and the query text corresponding to the video is input into the text encoder. The video is segmented, feature extracted, and dimension adjusted by the preprocessing unit in MD1 to obtain segment features, which are then stacked into video features F in , Figure 5 In the dashed box of the channel-aware multi-scale temporal modeling module in the middle, the large rectangle composed of multiple small rectangles is the video feature F in , where each small rectangle is a segment feature. The video feature F in is input into the improved stacked convolution unit to extract M=3 convolutional video features, and then the two-dimensional temporal adjacency graph generation unit generates two-dimensional temporal adjacency graphs F1~F3. Based on the existing two-dimensional temporal adjacency graph, only the upper triangle is valid, and the lower triangle is invalid. When implemented, it is filled with 0, so Figure 5 The two-dimensional temporal adjacency graph shown in the middle has a transparent grid in the lower triangle, and the grid with data is filled with blue in the upper triangle, and the rest is gray. The three two-dimensional temporal adjacency graphs are input into the multi-head decoupled cross-modal feature fusion module MD2 to obtain three gating output features, such as Figure 5 The scales 1, 2, and 3 on the left side of the semantic modulation gating convolution module MD3 are constructed in the middle. The features of the three scales are respectively input into the semantic modulation gating convolution module MD3 to generate three gating output features F 1,SG ~F 3,SG , and then the prediction head obtains three two-dimensional score maps I 1,SG ~I 3,SG , and the valid scores in each two-dimensional score map are collected and represented in orange in Figure 5 . Taking the fused two-dimensional score map as an example, the lower triangle is invalid, and in the upper triangle, gray has no data, yellow has data, and orange represents valid data.

[0045] Embodiment 3: To illustrate the effect of the application, the following data set is used for experiment as follows: (1) Data set and evaluation index: (1.1) The following three benchmark data sets are used: Charades-STA data set: This data set contains 9848 indoor daily activity videos, each with a natural language description of the time annotation. This data set is widely used in video time positioning tasks.

[0046] ActivityNet Captions data set: This data set consists of 19209 videos with diverse content. The time annotation of the video corresponds to the natural language description, which is suitable for evaluating the performance of the model in complex scenarios.

[0047] TACoS dataset: This dataset contains 127 videos, and the video content mainly involves kitchen activities. The video length and target time length of this dataset vary greatly, which puts high requirements on the time localization ability of the model.

[0048] (1.2) Evaluation index: Rank@n@m is used to measure the performance of the model. This index calculates the proportion of at least one predicted IoU greater than m in the first n prediction results. In the following table, for example, R@1, R@5 represent Top-1 recall rate and Top-5 recall rate respectively, IoU is the intersection over union, and R@1@IoU0.5 represents the proportion of samples whose unique Top-1 predicted segment has an IoU ≥ 0.5 with the ground truth segment.

[0049] (2) Experimental setup: In the task of natural language time localization, in order to conduct thorough and fair comparison, in the preprocessing unit, existing standard video feature extractors such as VGG, C3D and I3D video segments are used for feature extraction. The following is a detailed description of the video feature extractor: VGG: VGG16 pre-trained on ImageNet is used. Specifically, the video is decoded at a speed of 24 frames per second (fps), and the fc7 layer output after ReLU activation is extracted, with an extraction frequency of 6 fps, and every 4 consecutive frames correspond to a segment feature, so each segment corresponds to 1 second.

[0050] C3D: C3D network pre-trained on Sport1M is used. Specifically, the video is decoded at a speed of 16 fps, and the fc6 layer output after ReLU activation of every 16 consecutive frames is extracted, and each video segment corresponds to 1 second.

[0051] I3D: I3D network pre-trained on Kinetics is used. Specifically, the video is decoded at a speed of 25 fps, and the output of the last average pooling layer of every 16 consecutive frames is extracted, so each video segment corresponds to 0.64 seconds.

[0052] There is a high overlap between some predicted times, in order to reduce redundant predictions, based on the prediction score, Non-Maximum Suppression (NMS) is used, and the IoU threshold of NMS is fixed at 0.49 in all experiments. After NMS, we use the top n times for evaluation.

[0053] During training, N consecutive segments are randomly selected using a sliding window. To make a fair comparison, we set the following hyperparameters by default for all three datasets: the number of hidden states is 512, the window size is 64, the number of scales K = 3, the number of base anchors A = 16, in the multi-head decoupled cross-modal feature fusion module MD2, the cross-attention layer is 2 layers, the number of attention heads H = 8, these anchor and scale settings can cover at least 95% of the target time in the training set, the IoU threshold is 0.7, and fine-tuning these hyperparameters on specific datasets and feature types can achieve better performance. When training the model from scratch, the learning rate is set to 0.0001, no weight decay is used, the batch size is set to 32, and Adam is used as the optimizer. All experiments are performed using a single GeForce RTX 4090D GPU.

[0054] (3) Comparative experiment and result analysis: The performance of the proposed model method on the NLVL task is evaluated on three benchmark datasets, and compared with advanced methods proposed in the past three years. The best and suboptimal results are marked in bold and underlined in the experimental table. The detailed test results of the three NLVL datasets are shown in Tables 1, 2, and 3.

[0055] In Tables 1-3, the advanced methods proposed in the past three years include: 2D-TAN: 2D Temporal Adjacency Networks, a two-dimensional temporal adjacency network for video localization; MS-2D-TAN: Multi-Scale 2D Temporal Adjacency Networks, a multi-scale two-dimensional temporal adjacency graph model; MMN: Mutual Matching Network, a bidirectional matching network; 2D-Diffusion: Multi-scale 2D Temporal Map Diffusion Models, a multi-scale two-dimensional temporal map diffusion model; MRNet: Maskable retentive network, a maskable retentive network; VSLNet: Video Span Localization Network, a span localization network; LPNet: Learnable Proposal Network, a learnable proposal network; DPHANet: Discriminative Parallel and Hierarchical Attention Network for Natural Language Video Localization M 2 DCapsN: Multimodal, Multichannel, and Dual-Step Capsule Network for Natural Language Moment Localization TVP: Text-Visual Prompting MS-DETR: Natural Language Video Localization with Sampling Moment-Moment Interaction

[0056] The invention (C3D), the invention (VGG), and the invention (I3D) respectively represent feature extraction of video clips using C3D, VGG, and C3D.

[0057] Table 1: Comparison of indicators of each model on the Charades-STA dataset , The Charades-STA dataset is mainly composed of short-time daily activity clips, the average duration of the target moment is 8.2 seconds, and the query statement is concise (average length of 6.3 words).

[0058] According to Table 1, the accuracy of the invention gradually increases from C3D to VGG to I3D features. The analysis is that the performance of VGG features is limited due to the lack of temporal information. The performance of C3D features is limited due to the short-term modeling capability, and needs to be combined with a long-term temporal module for further optimization. The dual-stream design of I3D provides complementary information of appearance and motion, and adapts to the multi-granularity temporal modeling requirement. The multi-scale temporal modeling module MD1, the multi-head decoupled cross-modal feature fusion module MD2, and the semantic modulation gate convolution module MD3 are deeply coordinated with the high-resolution spatio-temporal features of I3D, realizing fine-grained cross-modal alignment and dynamic context guidance. The multi-scale temporal modeling module MD1 matches the short window characteristics of I3D, solving the problem of action span diversification in long videos, so the model achieves the best performance under I3D features.

[0059] On the Charades-STA dataset, the application has optimal or sub-optimal performance in all evaluation indicators, especially in the high-precision strict indicators Rank@1, IoU=0.7, which reaches 39.89%. Compared with other two-dimensional graph-based models, the application has optimal performance in all indicators. This shows that the multi-head decoupling cross-modal feature fusion module MD2 proposed by us makes good use of the rich context in the two-dimensional graph, establishes accurate association between the corresponding video and text, and improves the accuracy of natural language video positioning, which also proves the superiority of the application.

[0060] Table 2: Comparison of indicators of each model on the ActivityNet Captions dataset , In Table 2, MSAT: Multi-Stage Aggregation Transformer, multi-stage aggregation Transformer network; VGCL: Video-Guided Curriculum Learning for Spoken Video Grounding, video-guided curriculum learning network.

[0061] The ActivityNet Captions dataset is characterized by its diverse open-domain videos and complex long query sentences, which puts high requirements on the cross-modal alignment ability and long-range temporal modeling of the model. As shown in Table 2, the model achieves optimal or sub-optimal performance in multiple key indicators: it reaches 49.84% and 32.37% in strict Rank@1, IoU=0.7 and Rank@1, IoU=0.5, respectively. Due to the openness of the ActivityNet Captions dataset, it is difficult for general models to focus on the key content of videos in different fields. The multi-scale temporal modeling module MD1 designed by us can make the model learn how to focus on the key content in different fields, and improve the prediction accuracy of the model by strengthening the key features. At the same time, the semantic modulation gated convolution module MD3 is designed to consider the word-level features and global text features of the query text, so that the model can fully understand the query intent. When facing complex long query sentences, the model can still have high prediction accuracy.

[0062] Table 3: Comparison of indicators of each model on the TACoS dataset , The TACoS dataset has the core challenges of long video target segment dispersion in the kitchen scene, fine-grained object interaction such as hand action, and specific object interaction time such as chopstick interaction, and complex timing logic such as "stir again" and "chop then add". As shown in Table 3, the model achieves optimal or suboptimal performance in multiple indicators: 38.62% in the most stringent Rank@1, IoU=0.7 indicator. The video content in the TACoS dataset is kitchen content in a fixed scene, such as cutting vegetables and using chopsticks. This results in highly similar video content in different time segments. The multi-head decoupling cross-modal feature fusion module MD2 proposed by us, through the design of bidirectional cross-attention, enables the video content and the query text to fully interact and establish a precise correspondence between the two, even in the face of highly similar segments, the video time corresponding to the query text can be located.

[0063] (4) Ablation experiment: Based on the three core modules of the application: multi-scale timing modeling module MD1, multi-head decoupling cross-modal feature fusion module MD2, and semantic modulation gate convolution module MD3, the effectiveness of each module is revealed through the ablation experiment results. The following analyzes the module functions and experimental data.

[0064] Table 4 Ablation experiment on the TACoS dataset , In Table 4, " " indicates that the corresponding module is selected.

[0065] As shown in Table 4, when MD1 is enabled alone, the model can effectively filter out visual channels related to the query, such as strengthening "hand action" or "kitchen tool" features in the TACoS dataset, thereby improving the basic positioning ability, and Rank1@0.3 is improved by about 2.38% compared with the baseline. However, due to the lack of cross-modal alignment and dynamic guidance of query semantics, the model lacks understanding of the timing logic in complex queries, such as insufficient understanding of "again", which limits the performance at a higher IoU threshold (0.5), and Rank1@0.5 is only 36.32%.

[0066] The introduction of MD2 significantly improves the fine-grained cross-modal alignment. For example, in the TACoS dataset, different attention heads can focus on the spatiotemporal correlation of "knife movement trajectory" and "hand stirring action", respectively, thereby accurately matching complex queries such as "chop then add". The experimental results show that Rank1@0.5 is improved by 1.0%, indicating that the model's ability to locate high-precision is enhanced, and the global association characteristics of MD2 compensate for the local limitations of MD1, such as the association of dispersed segments in long videos, which promotes Rank5@0.5 to 69.79%.

[0067] For example, in the query "stir again", MD3 suppresses the first stirring segment through the global text feature "again", while enhancing the confidence of the correct segment by combining local action continuity such as the repeated pattern of the stirring action. The experimental results show that Rank1@0.5 is increased by 1.3%, indicating a significant enhancement in the robustness of the model under high IoU thresholds. In addition, the synergistic effect of MD3 and MD1, such as MD1 screening key features and MD3 suppressing noise, makes Rank5@0.5 reach 58.23%, which is 4.11% higher than when only MD1 is enabled.

[0068] To better illustrate the effect of the ablation experiment, see Figure 6 and Figure 7 .

[0069] Figure 6 For the corresponding video, the query text is in English: A woman picked up a painting and then put it back, where "and then" represents temporal logic. The Chinese translation is "A woman picked up a painting and then put it back", and "and then" is translated as "then". The target segment corresponding to this query text is 01:19~01:31. The moment detection of the Figure 6 video is performed using three models, namely: (1) the complete natural language moment retrieval model, also known as the full model, (the 2) full model without the semantic modulation gated convolutional module MD3; (3) the full model without the multi-head decoupled cross-modal feature fusion module MD2; The detection results are as Figure 6 shown. Similarly, Figure 7 for the corresponding video, the query text is in English: A woman drink her coffee again, which is translated into Chinese as "A woman drinks her coffee again", where "again" represents temporal logic and is translated as "again". The target segment corresponding to this query text is 06:09~06:15. The moment detection of the Figure 7 video is still performed using three models, and the detection results are as Figure 7 shown.

[0070] Figure 6 , Figure 7 demonstrates the model's ability to understand and respond to complex queries containing temporal logic. In the case of removing the semantic modulation gated convolutional module MD3 and the multi-head decoupled cross-modal feature fusion module MD2 respectively, the model's localization of the query language significantly decreases, such as Figure 7 being unable to accurately locate the boundary between the two highly similar scenarios of a woman drinking coffee and not drinking coffee, which shows the effectiveness of the designed module of the present invention.

[0071] The above merely describes preferred embodiments of the present application, and is not used to limit the present application, any modification, equivalent replacement and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A natural language video time-of-view retrieval method based on multimodal fusion, characterized in that, Includes the following steps; S1, Obtain the dataset for natural language video time localization, where each sample includes video V, several target segments with time annotations, and query text corresponding to each target segment; S2, constructs a multi-scale temporal modeling module MD1 with one-channel perception, including a preprocessing unit, an improved stacked convolution unit, and a two-dimensional temporal adjacency graph generation unit; The preprocessing unit is used to divide the video V into N non-overlapping segments, extract features from the nth segment, and adjust its dimension to d. v To obtain fragment features Then, the video features F are formed. in , , d v The preset number of output channels; The improved stacked convolutional unit is used as input F in Output M convolutional video features at different scales, where the convolutional video feature at the m-th scale is F. m,out ; The two-dimensional temporal adjacency graph generation unit is used to generate F based on... m,out Generate an initial two-dimensional temporal adjacency graph Then Adjust to Size and A two-dimensional temporal adjacency graph F is constructed by stitching the graphs along the channel dimension. m F m The point at position (u,v) represents a candidate video segment with start and end times of u and v, respectively. S3, Obtain a text encoder to input the query text and generate word-level text features. and global text features T global , T word,l To query the word-level text features of the l-th word in the text; S4, construct a multi-head decoupled cross-modal feature fusion module MD2, used to fuse F m Reconstructed into two-dimensional temporal features, with T word and T global Perform fusion to generate fusion feature F m,fused ; S5, construct the semantic modulation gated convolutional module MD3; The semantic modulation gated convolution module is used for... Shape adjusted to N m ×N m ×d v This results in a reshaped adjacency graph. Then, the dynamic gating weight G and gating feature F are generated according to the following formula. gated The gated output feature F of the semantic modulation gated convolution module m,SG ; ， , ， In the formula, ReLU(∙) is the ReLU function, GAP(∙) is the global average pooling function, and W s W v These are the third and fourth weight matrices, respectively, in Conv2D. r For dilated convolution with dilation rate r, ; S6. Construct the natural language time-of-flight retrieval network and loss function, and adjust the parameters of the natural language time-of-flight retrieval network to minimize the loss, thus obtaining the natural language time-of-flight retrieval model; The natural language time-series retrieval network includes MD1, a text encoder, MD2, MD3, and a prediction head; The video V of the sample is fed into MD1, and the query text is fed into the text encoder. The outputs of the two are then processed by MD2 and MD3 to obtain M gated output features F. 1,SG ~F M,SG For each F m,SG The prediction head generates a corresponding two-dimensional fractional graph I. m,SG Collect I m,SG The set of valid fractions ,in For P m The i-th valid score, C m For P m The total number of valid scores will The corresponding candidate video segments are marked as ; The loss function is obtained from the following formula: ， In the formula, for Supervisory labels, for The IoU value at real time, when hour, ,otherwise ; S7 uses a natural language time-lapse retrieval model to detect the video to be identified.

2. The natural language video time-of-view retrieval method based on multimodal fusion according to claim 1, characterized in that, The datasets include the ActivityNet Captions dataset, the Charades-STA dataset, and the TACoS dataset. In S12, a pre-trained 3D convolutional network is used to extract features from the segments.

3. The natural language video time-of-view retrieval method based on multimodal fusion according to claim 1, characterized in that, In S2, the improved stacked convolutional unit includes a stacked convolutional unit composed of Z stacked convolutional layers. At the end of each convolutional layer, an enhancement unit is sequentially arranged. The enhancement unit includes a sparse sampling layer, a one-dimensional convolutional block, and a channel attention layer. When the input F... in The output F of the z-th convolutional layer conv,z Enhanced features are obtained by sequentially applying sparse sampling, one-dimensional convolution, and channel attention. Then, it is fed into the next convolutional layer; The improved stacked convolutional unit is divided into M stages, and the enhanced features are obtained at the end of each stage as enhanced video features, where the enhanced video features of the m-th stage are labeled F. m,out .

4. The natural language video time-of-view retrieval method based on multimodal fusion according to claim 3, characterized in that, For the Z-layer convolutional unit of the improved stacked convolutional unit, Z=(K+1)A / 2, the kernel size of the first layer is 1 and the stride is 1, the kernel size of the (j1+1)A / 2 layer is 3 and the stride is 2, and the kernel size of the remaining layers is 2 and the stride is 1. A and K are the number of basic anchor points and the number of scales for video time-localization, respectively, and j1 is the number of convolutional layers, 1≤j1≤K-1.

5. The natural language video time-of-view retrieval method based on multimodal fusion according to claim 3, characterized in that, For the channel attention layer corresponding to the z-th convolutional layer, obtain the output F of the one-dimensional convolutional block. c,z , for F c,z Perform global average pooling to generate the first channel statistical vector. Second channel statistical vector And generate enhanced features according to the following formula. ; ， ， In the formula, FFN(∙) is the feedforward network, σ(∙) is the Sigmoid function, and W c Here, represents the channel attention weight, ⊙ represents element-wise multiplication, and... .

6. The natural language video time-of-view retrieval method based on multimodal fusion according to claim 1, characterized in that, The multi-head decoupled cross-modal feature fusion module MD2 includes a serialization layer, a video-guided multi-head cross-attention layer, a text-guided multi-head cross-attention layer, and a fusion layer. The serialization layer is used to convert F m Reconstructed into two-dimensional temporal features F rm , , N m For F m Length; H m For F rm The sequence length; The video-guided multi-head cross-attention layer will F rm As the K matrix and V matrix, T word As the Q matrix, cross-attention is performed to obtain the first attention feature F. temp ; The text-guided multi-head cross-attention layer will F temp As the K matrix and V matrix, F rm As the Q matrix, cross-attention is performed to obtain the second attention feature F. stage ; The fusion layer is used to generate fusion feature F according to the following formula. m,fused ; ， In the formula, σ(∙) is the Sigmoid activation function, ⊙ represents element-wise multiplication, and W g W v The first weight matrix and the second weight matrix are respectively, and LayerNorm(∙) is the layer normalization operation.

7. The natural language video time-of-view retrieval method based on multimodal fusion according to claim 1, characterized in that, In S3, the global text feature T global Calculate according to the following formula: ， In the formula, L represents the total number of words in the query text, and T represents the total number of words in the query text. word,l For T word The l-th element represents the word-level text feature of the l-th word in the query text.

8. The natural language video time-of-view retrieval method based on multimodal fusion according to claim 1, characterized in that, The effective score is I m,SG The non-zero value in the middle.

9. The natural language video time-of-view retrieval method based on multimodal fusion according to claim 1, characterized in that, S7 specifically involves: acquiring the video V' to be identified, obtaining M two-dimensional score maps by retrieving the MD1~MD3 of the natural language time-series model and the prediction head, adjusting them to the same size and fusing them into a single fused two-dimensional score map, and outputting one or more candidate video segments with the highest effective scores based on the fused two-dimensional score map. The fusion is to combine M two-dimensional fractional graphs I 1,SG ~I M,SG Adjust to the same size to obtain the corresponding adjustment diagram I. 1,SG ~I M,SG Construct a blank fused two-dimensional fractional graph I mix , to I mix The point at position (u,v) will I 1,SG ~I M,SG Find the maximum value at position (u,v) in the middle and fill it in I. mix .

10. The natural language video time-of-view retrieval method based on multimodal fusion according to claim 1, characterized in that, The text encoder consists of a pre-trained GloVe model and a BiLSTM.

Images