A natural language video time instant retrieval method based on multi-modal fusion

Abstract

The application discloses a kind of natural language video time retrieval methods based on multimodal fusion, belong to data identification technical field, including steps: obtaining the data set for natural language video time positioning;A channel perception multi-scale time sequence modeling module MD1 is constructed;A text encoder is obtained;A multi-head decoupling cross-modal feature fusion module MD2 is constructed, a semantic modulation gate convolution module MD3 is constructed;Based on MD1, MD2, text encoder, MD3 constructs natural language time retrieval network and is trained as natural language time retrieval model;For the natural language time detection of video to be tested.The application can significantly enhance the visual feature response related to query semantics through MD1, realize cross-modal fine-grained integration in multiple semantic subspaces through MD2, and ensure that the candidate segment score is highly dependent on the query semantics through MD3, which can significantly improve the positioning accuracy and robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data recognition, and particularly relates to a natural language video time point retrieval method based on multi-modal fusion. BACKGROUND

[0002] With the explosive growth of Internet video data and the rapid development of multi-modal artificial intelligence technology, natural language-based video time localization (NLVL) has become a core task of cross-modal understanding and intelligent video analysis. This task aims to accurately locate specific segments that match the semantics through text queries from uncut videos. This task is not only the cornerstone of intelligent video summarization, interactive search, video question answering and other applications, but also an important research direction for promoting the practicality of multi-modal artificial intelligence.

[0003] The natural language video localization task aims to accurately locate the specific time from an uncut video that matches the given natural language description. Specifically, given a natural language query, such as "a woman drinks her coffee again", the goal of this task is to automatically locate and extract the exact time when the woman drinks coffee again from the video. This task requires the model not only to understand the video content, but also to understand the semantics of the natural language query and establish an accurate correspondence between the two.

[0004] Currently, deep learning-based NLVL methods are the mainstream methods for solving the natural language video time localization task. Early natural language video time localization research mainly adopts a technical route combining sliding window candidate generation and feature-level cross-modal fusion. Representative works such as CTRL generate candidate segments through sliding windows and use Hadamard product to realize video-text feature fusion. Subsequent time graph modeling candidate generation methods such as 2D-TAN construct a two-dimensional time adjacency graph to capture local time relationships, while multi-scale expansion methods MS-2D-TAN optimize candidate generation by reducing computational complexity through sparse sampling. Cross-modal fusion technology has evolved from feature concatenation, attention mechanism to decoupled fusion. VSLNet uses span-wise self-attention to model video internal dependencies, and MMN enhances cross-modal association through memory networks.

[0005] Despite the remarkable progress of deep learning-based NLVL methods, there are still three problems with existing models: (1) Difficulty in complex dynamic temporal modeling: Multi-scale two-dimensional temporal graphs like MS-2D-TAN can cover candidate segments of different lengths, but they do not distinguish the importance of different frames during the generation of candidate segments, leading to interference of redundant features (such as background noise) in long videos with the capture of key temporal information by the model; (2) Insufficient fine-grained cross-modal fusion: Traditional methods, such as element-wise multiplication or simple concatenation, have difficulty modeling the complex interaction between language descriptions and video segments, especially for implicit spatiotemporal logic (such as "play again" "open the door first and then sit down") and multi-object interaction scenarios (such as "hand the cup to the person in red"); (3) Weak query dependence: Existing methods have insufficient correlation between candidate segment scoring and language description, leading to false positives for complex queries.

[0006] Noun explanation:

[0007] Two-dimensional temporal adjacency graph (2D-TAG) is a new structured representation method in video localization, which models the temporal and semantic relationships between video segments as a graph structure, significantly improving the accuracy and robustness of cross-modal alignment. Unlike traditional methods that treat video as a linear sequence of frames and only model local temporal dependencies (such as RNN, Transformer), two-dimensional temporal adjacency graph models the temporal relationships between different video segments by constructing a two-dimensional graph. Its method is: candidate segments in the video are represented as nodes on a two-dimensional graph, each node corresponds to a candidate segment, defined by its start time and end time. The edges of the graph represent the temporal adjacency relationships between segments (e.g. overlap, adjacent, contain, etc.). By modeling the context relationship between segments through graph convolution, the localization accuracy is improved. In the two-dimensional temporal adjacency graph, the vertical and horizontal coordinates represent the start and end times of the video segments, respectively. The point (i, j) in the i-th row and j-th column of the graph represents a video segment that starts at time i and ends at time j (also known as a candidate video segment). The localization principle is: after convolution operation on the two-dimensional temporal adjacency graph, a two-dimensional score graph is finally output, each cell represents the matching degree of the corresponding segment and the query sentence, and the segment with the highest score is the prediction result.

[0008] GloVe (Global Vectors) model: an unsupervised word embedding model that takes input text and outputs a d-dimensional vector for each word in the text (d = 50 / 100 / 200 / 300, etc.).

[0009] BiLSTM (Bidirectional Long Short-Term Memory) is an improved recurrent neural network (RNN) that captures the bidirectional dependencies of sequence data by combining a forward and a backward LSTM network.

[0010] GloVe+BiLSTM: GloVe is responsible for quickly converting discrete words into high-quality, low-dimensional, and globally statistically significant vectors; BiLSTM is responsible for capturing the forward and backward context dependencies on these vector sequences. SUMMARY

[0011] The purpose of the present application is to provide a natural language video time retrieval method based on multi-modal fusion, which solves the above-mentioned difficulties in complex dynamic time series modeling, the fine-grained semantic gap between video and text, and the insufficient dependence of candidate segment scoring on language description.

[0012] In order to achieve the above-mentioned purpose, the technical scheme adopted by the present application is as follows: a natural language video time retrieval method based on multi-modal fusion, comprising the following steps:

[0013] S1, obtaining a data set for natural language video time positioning, wherein each sample includes a video V, a plurality of target segments with time annotation, and a query text corresponding to the target segment;

[0014] S2, constructing a channel-aware multi-scale time series modeling module MD1, including a preprocessing unit, an improved stacked convolution unit, and a two-dimensional time series adjacency graph generation unit;

[0015] The preprocessing unit is used to divide the video V into N non-overlapping segments, extract features from the nth segment, and adjust the dimension to d v , to obtain segment features , and then form video features F in , , , d v is a preset output channel number;

[0016] The improved stacked convolution unit is used to input F in , and output M convolutional video features of different scales, wherein the mth scale of convolutional video feature is F m,out ;

[0017] The two-dimensional time series adjacency graph generation unit is used to generate an initial two-dimensional time series adjacency graph G m,out from F ; then adjust G to the size of G , and concatenate G along the channel dimension to form a two-dimensional time series adjacency graph F m , F mThe point in the middle position (u, v) is a candidate video clip representing the start and end times as u and v respectively;

[0018] S3, obtaining a text encoder configured to input the query text and generate word-level text features and global text features T global , , T word,l is the word-level text feature of the lth word in the query text;

[0019] S4, constructing a multi-head decoupled cross-modal feature fusion module MD2 configured to fuse F m into two-dimensional time sequence features, and fuse T word and T global to generate fusion features F m,fused ;

[0020] S5, constructing a semantic modulation gate convolution module MD3;

[0021] The semantic modulation gate convolution module is configured to reshape F into N m ×N m ×d v , obtain a reshaped adjacency graph G , and generate dynamic gate weights G, gate features F gated , and gate output features F m,SG of the semantic modulation gate convolution module according to the following formula;

[0022] ,

[0023] , ,

[0024] In the formula, ReLU(·) is a ReLU function, GAP(·) is a global average pooling, W s and W v are respectively a third weight matrix and a fourth weight matrix, Conv2D r is a dilated convolution with a dilation rate r, ;

[0025] S6, constructing a natural language time instant retrieval network and a loss function loss, and adjusting the natural language time instant retrieval network parameters to obtain a natural language time instant retrieval model by minimizing the loss;

[0026] The natural language time instant retrieval network comprises MD1, the text encoder, MD2, MD3, and a prediction head;

[0027] The video V of the sample is fed into MD1, and the query text is fed into the text encoder. The outputs of the two are then processed by MD2 and MD3 to obtain M gated output features F. 1,SG ~F M,SG For each F m,SG The prediction head generates a corresponding two-dimensional fractional graph I. m,SG Collect I m,SG The set of valid fractions ,in For P m The i-th valid score, C m For P m The total number of valid scores will The corresponding candidate video segments are marked as ;

[0028] The loss function is obtained from the following formula:

[0029] ,

[0030] In the formula, for Supervisory labels, for The IoU value at real time, when hour, ,otherwise ;

[0031] S7 uses a natural language time-lapse retrieval model to detect the video to be identified.

[0032] Preferably, the dataset includes the ActivityNet Captions dataset, the Charades-STA dataset, and the TACoS dataset. In S12, a pre-trained 3D convolutional network is used to extract features from the segments.

[0033] Preferably, in S2, the improved stacked convolutional unit includes a stacked convolutional unit composed of Z stacked convolutional layers. An enhancement unit is sequentially placed at the end of each convolutional layer. The enhancement unit includes a sparse sampling layer, a one-dimensional convolutional block, and a channel attention layer. When the input F... in The output F of the z-th convolutional layer conv,z Enhanced features are obtained by sequentially applying sparse sampling, one-dimensional convolution, and channel attention. , and then feed it into the next convolutional layer;

[0034] The improved stacked convolutional unit is divided into M stages, and the enhanced features are obtained at the end of each stage as enhanced video features, where the enhanced video features of the m-th stage are labeled F. m,out .

[0035] As preferred, for the Z-layer convolutional layer of the improved stacked convolutional unit, Z=(K+1)A / 2, the first layer has a convolution kernel size of 1 and a step size of 1, the (j1+1)A / 2 layer has a convolution kernel size of 3 and a step size of 2, and the remaining layers have a convolution kernel size of 2 and a step size of 1, A and K are respectively the number of basic anchor points and the number of scales of video time positioning, j1 is a variable of the number of convolution layers, and 1≤j1≤K-1.

[0036] As preferred, for the channel attention layer corresponding to the z-th convolutional layer, the output F c,z of the one-dimensional convolution block is obtained c,z Global average pooling is performed on F to generate a first channel statistical vector and a second channel statistical vector , and an enhanced feature F is generated according to the following formula

[0037]

[0038]

[0039] In the formula, FFN(·) is a feedforward network, σ(·) is a Sigmoid function, W c is a channel attention weight, ⊙ is an element-wise multiplication, and .

[0040] As preferred, the multi-head decoupling cross-modal feature fusion module MD2 includes a serialization layer, a video-guided multi-head cross-attention layer, a text-guided multi-head cross-attention layer, and a fusion layer.

[0041] The serialization layer is configured to reshape F m into a two-dimensional time sequence feature F rm , , N m is the length of F m , H m is the sequence length of F rm .

[0042] The video-guided multi-head cross-attention layer takes F rm as a K matrix and a V matrix, takes T word as a Q matrix, and performs cross-attention to obtain a first attention feature F temp .

[0043] The text-guided multi-head cross-attention layer takes F temp as a K matrix and a V matrix, takes F rm as a Q matrix, and performs cross-attention to obtain a second attention feature F stage .

[0044] The fusion layer is used to generate fusion feature F according to the following formula. m,fused ;

[0045] ,

[0046] In the formula, σ(∙) is the Sigmoid activation function, ⊙ represents element-wise multiplication, and W g W v The first weight matrix and the second weight matrix are respectively, and LayerNorm(∙) is the layer normalization operation.

[0047] As a preferred option, in S3, the global text feature T global Calculate according to the following formula:

[0048] ,

[0049] In the formula, L represents the total number of words in the query text, and T represents the total number of words in the query text. word,l For T word The l-th element represents the word-level text feature of the l-th word in the query text.

[0050] Preferably, the effective score is I. m,SG The non-zero value in the middle.

[0051] As a preferred option, S7 specifically involves: acquiring the video V' to be identified, obtaining M two-dimensional score maps by retrieving the MD1~MD3 of the natural language time-series model and the prediction head, adjusting them to the same size and fusing them into a single fused two-dimensional score map, and outputting one or more candidate video segments with the highest effective scores based on the fused two-dimensional score map.

[0052] The fusion is to combine M two-dimensional fractional graphs I 1,SG ~I M,SG Adjust to the same size to obtain the corresponding adjustment diagram I. 1,SG ~I M,SG Construct a blank fused two-dimensional fractional graph I mix , to I mix The point at position (u,v) will I 1,SG ~I M,SG Find the maximum value at position (u,v) in the middle and fill it in I. mix .

[0053] Preferably, the text encoder consists of a pre-trained GloVe model and a BiLSTM.

[0054] The design concept of this invention is as follows:

[0055] 1. For the multi-scale temporal modeling module MD1 for channel perception: the goal is to effectively extract key information and avoid redundant information from interfering with the recognition of target events.

[0056] (1) About the preprocessing unit: used to process the video into It contains N fragment features This facilitates subsequent feature extraction and related processing.

[0057] (2) Regarding the improved stacked convolutional unit: It borrows the structure of stacked convolution in the existing technology, and is still composed of Z stacked convolutional layers. It is also divided into M stages for input F. in The video features are generated at different scales from the first stage to the Mth stage. However, this invention has made improvements on this basis: (2.1) Considering that the output of each layer of the convolutional layer is a candidate moment feature, these moments have the same length but different start times. To alleviate the problem of insufficient memory when the number of current video segments is too large. When generating candidate moment features, a sparse sampling strategy is adopted, defining the number of basic anchor points A and the number of scales K, and setting different convolutional kernels in different convolutional layers to achieve sparse sampling. When sampling, when the number of segments is less than A, all possible moments are enumerated as candidates; when the number of segments is greater than A, only moments that satisfy the condition G(a,b) are selected as candidates: , where a and b are segment indices, if G(a,b)=1, then the time moment is selected as a candidate, otherwise it is discarded. This strategy achieves dense sampling of short-duration time moments and gradually increases the sampling interval as the time length increases. (2.2) After sparse sampling, a one-dimensional convolutional block and a channel attention layer are set. The one-dimensional convolutional block is a combination of convolution + batch normalization + activation function, where the activation function adopts the hyperbolic tangent function Tanh. The channel attention layer adopts dual-channel pooling channel attention, dynamically generating and adjusting the channel attention weight W. c By strengthening key features and obtaining enhanced features, the channel attention layer can assign high weights to key channels, allowing the model to pay more attention to the highly relevant information in the video to the query, while reducing the interference of a large amount of redundant information in the video on the model.

[0058] (3) Regarding the two-dimensional temporal adjacency graph generation unit: for M enhanced video features F 1,out ~F M,out First, an initial two-dimensional temporal adjacency graph is generated based on existing technology. ~ Each two-dimensional temporal adjacency graph has a different scale. Therefore, in this case, Adjust to Size and A two-dimensional temporal adjacency graph F is constructed by stitching the graphs along the channel dimension. m This approach preserves low-scale fine-grained information (such as short-term action details) and high-scale semantic abstractions (such as long-term event context), enhancing the model's ability to capture multi-granular temporal patterns.

[0059] 2. For the multi-head decoupled cross-modal feature fusion module MD2: the goal is to design an effective interaction mechanism to strengthen the semantic association between video and text, thereby improving the accuracy of retrieval and localization. MD2 first processes the F... m Shape reshaping is performed to make it compatible with word-level text features T word and global text features T global The process involves fusing F by specifying the Q, K, and V matrices, and then performing two layers of cross-attention operations. mr T word and T global The fusion feature F is obtained. m,fused Through a two-stage fusion process, the video and query text can fully consider information from each other's modalities, achieving a nuanced fusion of the two modalities. Simultaneously, each attention head automatically focuses on different semantic dimensions, and the model decouples the interaction patterns of cross-modal features across multiple subspaces. The attention mechanism directly associates with long-distance segments in the video, overcoming the limitations of the local receptive field in traditional convolution. This provides a highly discriminative cross-modal joint representation for the subsequent semantic modulation gated convolutional module MD3.

[0060] 3. For the semantic modulation gated convolution module MD3;

[0061] pass T global Projecting the text into the gated space yields the projected global text features g. text And then according to Generate channel-level context vectors And then according to Generate a dynamic gating weight G, whose value is in the interval [0, 1], according to... The dynamic gating weight G and Element-wise multiplication suppresses irrelevant regions and enhances relevant query features, finally using... , for F gated Introducing residual connections preserves original feature information. This alleviates the gradient vanishing problem. Gated feature F gated Driven by global textual semantics, the model's response is strictly aligned with the query intent. Fine-grained feature modulation is achieved by combining local video context (such as action continuity) with global textual constraints (such as event causal chains). This module uses a language-driven dynamic gating mechanism to deeply embed query semantics into the video feature learning process, ensuring high-precision localization.

[0062] Compared with the prior art, the advantages of the present invention are as follows:

[0063] (1) A channel-aware multi-scale temporal modeling module MD1 was designed to overcome the shortcomings of existing technologies, such as the difficulty in capturing complex temporal relationships between video segments and the presence of a large amount of redundant background information in long videos. By improving the stacked convolutional unit to assign high weights to key channels, the model can focus more on the highly relevant information to the query in the video, while reducing the interference of a large amount of redundant information in the video on the model. A two-dimensional temporal adjacency graph F is generated by the two-dimensional temporal adjacency graph generation unit. m This method differs from existing technologies that generate single-scale two-dimensional temporal adjacency graphs. It can preserve both low-scale fine-grained information (such as short-term action details) and high-scale semantic abstraction (such as long-term event context), thereby improving the model's ability to capture multi-granular temporal patterns.

[0064] (2) A multi-head decoupled cross-modal feature fusion module MD2 was designed: it allows video and text to fully pay attention to each other's modal information, thereby achieving fine fusion of the two modalities, making the semantic relationship between the two closer, and effectively improving the accuracy of retrieval and positioning.

[0065] (3) A semantic modulation gated convolutional module MD3 was designed. This module combines local video context (such as action continuity) with global text constraints (such as event causal chains) to achieve fine-grained feature modulation. This module embeds the query semantic depth into the video feature learning process through a language-driven dynamic gating mechanism, which ensures the final high-precision positioning.

[0066] In summary, this invention constructs a novel natural language temporal retrieval network based on the above improvements. MD1 significantly enhances the visual feature response related to query semantics, MD2 achieves cross-modal fine-grained integration in multiple semantic subspaces, and MD3 ensures that candidate fragment scores are highly dependent on query semantics, thereby significantly improving localization accuracy and robustness. Attached Figure Description

[0067] Figure 1 This is a flowchart of the present invention;

[0068] Figure 2 This is a diagram of the natural language time-lapse retrieval network structure of the present invention;

[0069] Figure 3 A schematic diagram of the improved stacked convolutional unit structure;

[0070] Figure 4 This is a schematic diagram of the MD2 multi-head decoupled cross-modal feature fusion module;

[0071] Figure 5 This is a feature change diagram during the video processing of the present invention;

[0072] Figure 6This is a visual comparison of a video ablation experiments;

[0073] Figure 7 This is a visualization comparison of another video with ablation experiments. Detailed Implementation

[0074] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0075] Example 1: See Figures 1-4 A natural language video moment retrieval method based on multimodal fusion includes the following steps;

[0076] S1, Obtain the dataset for natural language video time localization, where each sample includes video V, several target segments with time annotations, and query text corresponding to each target segment;

[0077] S2, constructs a multi-scale temporal modeling module MD1 with one-channel perception, including a preprocessing unit, an improved stacked convolution unit, and a two-dimensional temporal adjacency graph generation unit;

[0078] The preprocessing unit is used to divide the video V into N non-overlapping segments, extract features from the nth segment, and adjust its dimension to d. v To obtain fragment features Then, the video features F are formed. in , , d v The preset number of output channels;

[0079] The improved stacked convolutional unit is used as input F in Output M convolutional video features at different scales, where the convolutional video feature at the m-th scale is F. m,out ;

[0080] The two-dimensional temporal adjacency graph generation unit is used to generate F based on... m,out Generate an initial two-dimensional temporal adjacency graph Then Adjust to Size and A two-dimensional temporal adjacency graph F is constructed by stitching the graphs along the channel dimension. m F m The point at position (u,v) represents a candidate video segment with start and end times of u and v, respectively.

[0081] S3, Obtain a text encoder to input the query text and generate word-level text features. and global text features T global , T word,lTo query the word-level text features of the l-th word in the text;

[0082] S4, construct a multi-head decoupled cross-modal feature fusion module MD2, used to fuse F m Reconstructed into two-dimensional temporal features, with T word and T global Perform fusion to generate fusion feature F m,fused ;

[0083] S5, construct the semantic modulation gated convolutional module MD3;

[0084] The semantic modulation gated convolution module is used for... Shape adjusted to N m ×N m ×d v This results in a reshaped adjacency graph. Then, the dynamic gating weight G and gating feature F are generated according to the following formula. gated The gated output feature F of the semantic modulation gated convolution module m,SG ;

[0085] ,

[0086] , ,

[0087] In the formula, ReLU(∙) is the ReLU function, GAP(∙) is the global average pooling function, and W s W v These are the third and fourth weight matrices, respectively, in Conv2D. r For dilated convolution with dilation rate r, ;

[0088] S6. Construct the natural language time-of-flight retrieval network and loss function, and adjust the parameters of the natural language time-of-flight retrieval network to minimize the loss, thus obtaining the natural language time-of-flight retrieval model;

[0089] The natural language time-series retrieval network includes MD1, a text encoder, MD2, MD3, and a prediction head;

[0090] The video V of the sample is fed into MD1, and the query text is fed into the text encoder. The outputs of the two are then processed by MD2 and MD3 to obtain M gated output features F. 1,SG ~F M,SG For each F m,SG The prediction head generates a corresponding two-dimensional fractional graph I. m,SG Collect I m,SG The set of valid fractions ,in For P mThe i-th valid score, C m For P m The total number of valid scores will The corresponding candidate video segments are marked as ;

[0091] The loss function is obtained from the following formula:

[0092] ,

[0093] In the formula, for Supervisory labels, for The IoU value at real time, when hour, ,otherwise ;

[0094] S7 uses a natural language time-lapse retrieval model to detect the video to be identified.

[0095] The datasets include the ActivityNet Captions dataset, the Charades-STA dataset, and the TACoS dataset. In S12, a pre-trained 3D convolutional network is used to extract features from the segments.

[0096] In S2, the improved stacked convolutional unit includes a stacked convolutional unit composed of Z stacked convolutional layers. At the end of each convolutional layer, an enhancement unit is sequentially arranged. The enhancement unit includes a sparse sampling layer, a one-dimensional convolutional block, and a channel attention layer. When the input F... in The output F of the z-th convolutional layer conv,z Enhanced features are obtained by sequentially applying sparse sampling, one-dimensional convolution, and channel attention. The data is fed into the next convolutional layer; the improved stacked convolutional unit is divided into M stages, and the enhanced features are obtained at the end of each stage as enhanced video features, where the enhanced video features of the m-th stage are labeled F. m,out .

[0097] For the Z-layer convolutional unit of the improved stacked convolutional unit, Z=(K+1)A / 2, the kernel size of the first layer is 1 and the stride is 1, the kernel size of the (j1+1)A / 2 layer is 3 and the stride is 2, and the kernel size of the remaining layers is 2 and the stride is 1. A and K are the number of basic anchor points and the number of scales for video time-localization, respectively, and j1 is the number of convolutional layers, 1≤j1≤K-1.

[0098] For the channel attention layer corresponding to the z-th convolutional layer, obtain the output F of the one-dimensional convolutional block. c,z , for F c,z Perform global average pooling to generate the first channel statistical vector. Second channel statistical vector And generate enhanced features according to the following formula. ;

[0099]

[0100] ,

[0101] In the formula, FFN(∙) is the feedforward network, σ(∙) is the Sigmoid function, and W c Here, represents the channel attention weight, ⊙ represents element-wise multiplication, and... .

[0102] The multi-head decoupled cross-modal feature fusion module MD2 includes a serialization layer, a video-guided multi-head cross-attention layer, a text-guided multi-head cross-attention layer, and a fusion layer.

[0103] The serialization layer is used to convert F m Reconstructed into two-dimensional temporal features F rm , , N m For F m Length; H m For F rm The sequence length;

[0104] The video-guided multi-head cross-attention layer will F rm As the K matrix and V matrix, T word As the Q matrix, cross-attention is performed to obtain the first attention feature F. temp ;

[0105] The text-guided multi-head cross-attention layer will F temp As the K matrix and V matrix, F rm As the Q matrix, cross-attention is performed to obtain the second attention feature F. stage ;

[0106] The fusion layer is used to generate fusion feature F according to the following formula. m,fused ;

[0107] ,

[0108] In the formula, σ(∙) is the Sigmoid activation function, ⊙ represents element-wise multiplication, and W g W v The first weight matrix and the second weight matrix are respectively, and LayerNorm(∙) is the layer normalization operation.

[0109] In S3, the global text feature T global Calculate according to the following formula:

[0110] ,

[0111] In the formula, L represents the total number of words in the query text, and T represents the total number of words in the query text. word,l For T word The l-th element represents the word-level text feature of the l-th word in the query text.

[0112] The effective score is I m,SG The non-zero value in the middle.

[0113] S7 specifically involves: acquiring the video V' to be identified, obtaining M two-dimensional score maps by retrieving the MD1~MD3 of the natural language time-series model and the prediction head, adjusting them to the same size and fusing them into a single fused two-dimensional score map, and outputting one or more candidate video segments with the highest effective scores based on the fused two-dimensional score map.

[0114] The fusion is to combine M two-dimensional fractional graphs I 1,SG ~I M,SG Adjust to the same size to obtain the corresponding adjustment diagram I. 1,SG ~I M,SG Construct a blank fused two-dimensional fractional graph I mix , to I mix The point at position (u,v) will I 1,SG ~I M,SG Find the maximum value at position (u,v) in the middle and fill it in I. mix .

[0115] The text encoder consists of a pre-trained GloVe model and a BiLSTM.

[0116] Example 2: See Figures 1-5 Based on Example 1, this example improves the stacked convolutional unit. The number of basic anchor points for video time-lapse localization is set to A=6, and the number of scales is K=3. Therefore, Z=(K+1)A / 2=12, resulting in a total of 12 stacked convolutional layers. These 12 layers are divided into M stages, assuming M=3. The first 6 layers are designated as stage 1, layers 7-9 as stage 2, and layers 9-12 as stage 3. For the 12 convolutional layers, the kernel size of the first layer is 1, and the stride is 1. The kernel size of the (j1+1)A / 2 layer is 3, and the stride is 2. The remaining layers have a kernel size of 2 and a stride of 1. The value of j1 can be chosen arbitrarily between 1 and K-1.

[0117] Figure 5 The changes in various features of the video after processing by this invention are shown when M=3. Figure 5In the process, the video is fed into the multi-scale temporal modeling module MD1 of channel perception, and the corresponding query text is fed into the text encoder. The video is segmented, feature extracted, and dimensionally adjusted by the preprocessing unit in MD1 to obtain segment features, which are then stacked to form the video feature F. in , Figure 5 Within the dashed box of the multi-scale temporal modeling module for mid-channel sensing, the large cuboid composed of multiple small cuboids represents the video feature F. in Each small cuboid represents a segment feature. Video feature F in The data is fed into an improved stacked convolutional unit to extract M=3 convolutional video features. Then, a 2D temporal adjacency graph generation unit generates 2D temporal adjacency graphs F1~F3. Based on existing 2D temporal adjacency graphs, it is known that only the upper triangle is effective, while the lower triangle is ineffective. Therefore, in the implementation, zeros are used for padding. Figure 5 The image shows a two-dimensional temporal adjacency graph. The lower triangle is a transparent grid, while in the upper triangle, the grids containing data are filled in blue, and the rest are gray. These three two-dimensional temporal adjacency graphs are processed by the multi-head decoupled cross-modal feature fusion module MD2 to obtain three gated output features, such as... Figure 5 The semantic modulation gated convolutional module MD3 is constructed with scales 1, 2, and 3 on the left side. Features at these three scales are respectively processed by the semantic modulation gated convolutional module MD3 to generate three gated output features F. 1,SG ~F 3,SG Then, the prediction head generates three single-scale two-dimensional fractional maps I. 1,SG ~I 3,SG Collect the valid scores for each two-dimensional fraction graph, and in Figure 5 Orange indicates the data. Taking a fused two-dimensional fraction chart as an example, the lower triangle is invalid, and within the upper triangle, gray indicates no data, yellow indicates data, and orange indicates valid scores.

[0118] Example 3: To illustrate the effectiveness of the present invention, this example uses the following dataset for experimentation:

[0119] (1) Data set and evaluation metrics:

[0120] (1.1) The following three benchmark datasets are used:

[0121] The Charades-STA dataset contains 9,848 videos of everyday indoor activities, each with a time stamp annotation in natural language. This dataset is widely used in video time-localization tasks.

[0122] The ActivityNet Captions dataset consists of 19,209 videos, offering a rich variety of content. The video time markers correspond to natural language descriptions, making it suitable for evaluating model performance in complex scenarios.

[0123] The TACoS dataset contains 127 videos, primarily depicting kitchen activities. The video lengths and target time intervals vary considerably, placing high demands on the model's temporal localization capabilities.

[0124] (1.2) Evaluation Metrics: Rank@n@m is used to measure the model's performance. This metric is calculated as the proportion of at least one prediction with an IoU greater than m among the top n predictions. In the table below, for example, R@1 and R@5 represent Top-1 recall and Top-5 recall, respectively, and IoU is the intersection-union ratio. R@1@IoU0.5 represents the proportion of samples with an IoU ≥ 0.5 with the ground truth segment among the unique Top-1 predictions given by the model.

[0125] (2) Experimental setup:

[0126] In natural language processing time localization tasks, to ensure thorough and fair comparisons, existing standard video feature extractors, such as those for VGG, C3D, and I3D video clips, are used in the preprocessing unit for feature extraction. The following is a detailed description of the video feature extractors:

[0127] VGG: VGG16 pre-trained on ImageNet is used. Specifically, the video is decoded at 24 frames per second (fps), and the output of the fc7 layer after ReLU activation is extracted at a frequency of 6fps. Every 4 consecutive frames correspond to a segment feature, so each segment corresponds to 1 second.

[0128] C3D: The C3D network is pre-trained on Sport1M. Specifically, the video is decoded at 16fps, and the output of the fc6 layer after ReLU activation is extracted for every 16 consecutive frames, with each video segment corresponding to 1 second.

[0129] I3D: Uses the I3D network pre-trained on Kinetics. Specifically, the video is decoded at 25fps, and the output of the last average pooling layer is extracted for every 16 consecutive frames, so each video segment corresponds to 0.64 seconds.

[0130] There is a high degree of overlap between some predicted times. To reduce redundant predictions, non-maximum suppression (NMS) is used based on the prediction scores. The IoU threshold of NMS is fixed at 0.49 in all experiments. After NMS, we use the top n times for evaluation.

[0131] During training, a sliding window was used to randomly select N consecutive segments. For fair comparison, we set the following hyperparameters by default for all three datasets: number of hidden states 512, window size 64, number of scales K=3, number of base anchors A=16, and in the multi-head decoupled cross-modal feature fusion module MD2, there were 2 cross-attention layers and 8 attention heads H. These anchor and scale settings can cover at least 95% of the target time points in the training set. The IoU threshold was 0.7. Fine-tuning these hyperparameters on specific datasets and feature types can achieve better performance. When training the model from scratch, the learning rate was set to 0.0001, no weight decay was used, the batch size was set to 32, and Adam was used as the optimizer. All experiments were performed using a single GeForce RTX 4090D GPU.

[0132] (3) Comparative experiments and results analysis:

[0133] The performance of the proposed model method on the NLVL task was evaluated on three benchmark datasets and compared with state-of-the-art methods proposed in the last three years. The best and second-best results are marked in bold and underlined in the experimental tables. Detailed test results for the three NLVL datasets are shown in Tables 1, 2, and 3.

[0134] The advanced methods proposed in the past three years, as shown in Tables 1-3, include:

[0135] 2D-TAN: 2D Temporal Adjacency Networks, two-dimensional temporal adjacency networks for video localization;

[0136] MS-2D-TAN: Multi-Scale 2D Temporal Adjacency Networks, a multi-scale two-dimensional temporal adjacency graph model;

[0137] MMN: Mutual Matching Network;

[0138] 2D-Diffusion: Multi-scale 2D Temporal Map Diffusion Models;

[0139] MRNet: Maskable retentive network;

[0140] VSLNet: Video Span Localization Network;

[0141] LPNet: Learnable Proposal Network;

[0142] DPHANet: Discriminative Parallel and Hierarchical Attention Network for Natural Language Video Localization;

[0143] M 2 DCapsN: Multimodal, Multichannel, and Dual-Step Capsule Network for Natural Language Moment Localization;

[0144] TVP: Text-Visual Prompting;

[0145] MS-DETR: Natural Language Video Localization with Sampling Moment-Moment Interaction, a multi-scale DETR network.

[0146] The present invention (C3D), the present invention (VGG), and the present invention (I3D) respectively represent the use of C3D, VGG, and C3D to extract features from video segments.

[0147] Table 1. Comparison of metrics for each model on the Charades-STA dataset.

[0148] ,

[0149] The Charades-STA dataset primarily consists of short snippets of daily activities, with the target moment lasting an average of 8.2 seconds, and the query statements are concise (average length 6.3 words).

[0150] As shown in Table 1, the accuracy of this invention gradually improves from C3D to VGG and then to I3D features. This is because VGG features are limited in performance due to a lack of temporal information; C3D features have limited performance improvement due to their short-term temporal modeling capabilities, requiring further optimization by combining them with long-term temporal modules. I3D's dual-stream design provides complementary information on appearance and motion, adapting to multi-granularity temporal modeling needs. The multi-scale temporal modeling module MD1, the multi-head decoupled cross-modal feature fusion module MD2, and the semantic modulation gated convolution module MD3 work in deep collaboration with I3D's high-resolution spatiotemporal features to achieve fine-grained cross-modal alignment and dynamic context guidance. The multi-scale temporal modeling module MD1 matches the short-window characteristics of I3D, solving the problem of diverse action spans in long videos. Therefore, the model achieves the best performance under I3D features.

[0151] On the Charades-STA dataset, our invention achieves best or near-best performance across all evaluation metrics, particularly excelling in the high-precision rigorous metric Rank@1 (IoU=0.7), reaching 39.89%. Compared to other 2D graph-based models, our invention demonstrates superior performance across all metrics. This indicates that our proposed multi-head decoupled cross-modal feature fusion module MD2 effectively utilizes the rich context of the 2D graph, establishing accurate associations between corresponding videos and text, thereby improving the accuracy of natural language video localization. This also proves the superiority of our invention.

[0152] Table 2. Comparison of metrics for each model on the ActivityNet Captions dataset.

[0153] ,

[0154] In Table 2, MSAT stands for Multi-Stage Aggregation Transformer, and VGCL stands for Video-Guided Curriculum Learning for Spoken Video Grounding.

[0155] The ActivityNet Captions dataset, characterized by its diverse open-domain videos and complex long queries, places high demands on the model's cross-modal alignment capabilities and long-range temporal modeling. As shown in Table 2, our model achieved optimal or near-optimal performance on several key metrics: 49.84% and 32.37% at strict Rank@1, IoU=0.7 and Rank@1, IoU=0.5, respectively. Due to the open nature of the ActivityNet Captions dataset, general models struggle to focus on key content in videos across different domains. Our multi-scale temporal modeling module MD1 enables the model to learn how to focus on key content in different domains, improving prediction accuracy by strengthening key features. Simultaneously, our semantic modulation gated convolution module MD3, by explicitly considering word-level and global text features of the query text, allows the model to fully understand the query intent. This ensures high prediction accuracy even when faced with complex long queries.

[0156] Table 3. Comparison of metrics for each model on the TACOS dataset

[0157] ,

[0158] The TACoS dataset presents core challenges such as scattered long video clips in kitchen scenes, fine-grained object interactions (e.g., hand gestures and interactions with specific objects like chopsticks), and complex temporal logic (e.g., "stir again" and "chop then add"). As shown in Table 3, our model achieves optimal or near-optimal performance across multiple metrics: achieving 38.62% for the most stringent Rank@1, IoU=0.7. The video content in the TACoS dataset consists of fixed kitchen scenes, such as chopping vegetables and using chopsticks. This leads to high similarity between video clips at different times. Our proposed multi-head decoupled cross-modal feature fusion module, MD2, uses a bidirectional cross-attention design to allow full interaction between video content and query text, establishing a precise correspondence between them. Even with highly similar clips, it can pinpoint the video moment corresponding to the query text.

[0159] (4) Ablation experiment:

[0160] Based on the three core modules of this invention—multi-scale temporal modeling module MD1, multi-head decoupled cross-modal feature fusion module MD2, and semantic modulation gated convolution module MD3—the effectiveness of each module is revealed through ablation experiments. The following analysis combines module functions with experimental data.

[0161] Table 4 Ablation experiments on the TACOS dataset

[0162] ,

[0163] In Table 4, "√" indicates that the corresponding module is selected.

[0164] As shown in Table 4, when MD1 is enabled alone, the model can effectively filter out visual channels related to the query. For example, in the TACoS dataset, it strengthens the features of "hand movement" or "kitchen tools", thereby improving the basic localization ability. Among them, Rank1@0.3 is improved by about 2.38% compared to the baseline. However, due to the lack of cross-modal alignment and dynamic guidance of query semantics, the model has insufficient understanding of temporal logic in complex queries. For example, it has insufficient understanding of "again", resulting in limited performance at a higher IoU threshold (0.5), and Rank1@0.5 is only 36.32%.

[0165] The introduction of MD2 significantly improves the fine-grained cross-modal alignment. For example, in the TACoS dataset, different attention heads can respectively focus on the spatio-temporal associations of "knife movement trajectory" and "hand stirring action", so as to accurately match complex queries such as "chop then add". The experimental results show that Rank1@0.5 is improved by 1.0%, indicating that the model's ability to perform high-precision localization is enhanced. In addition, the global association characteristic of MD2 compensates for the locality limitation of MD1, such as the association of scattered segments in long videos, and promotes Rank5@0.5 to increase to 69.79%.

[0166] For example, in the query "stir again", MD3 suppresses the first stirring segment through the global text feature "again", and at the same time enhances the confidence of the correct segment by combining local action continuity, such as the repeated pattern of the stirring action. The experimental results show that Rank1@0.5 is improved by 1.3%, indicating that the robustness of the model at a high IoU threshold is significantly enhanced. In addition, the synergistic effect of MD3 and MD1, such as MD1 screening key features and MD3 suppressing noise, makes Rank5@0.5 reach 58.23%, which is improved by 4.11% compared to when only MD1 is enabled.

[0167] To better illustrate the effect of the ablation experiment, see Figure 6 and Figure 7 .

[0168] Figure 6 The corresponding video, whose query text is in English: A woman picked up a painting and then put it back, where "and then" is temporal logic. The corresponding Chinese translation is "A woman picked up a painting and then put it back", and "and then" is translated as "then". The target segment corresponding to this query text is 01:19~01:31. Using three models forFigure 6 Video time detection is carried out, respectively: (1) a complete natural language time retrieval model, also known as the full model, (2) removing the semantic modulation gated convolution module MD3 from the full model; (3) removing the multi-head decoupled cross-modal feature fusion module MD2 from the full model; the detection results are as Figure 6 shown. Similarly, Figure 7 For the corresponding video, the query text is in English: A woman drink her coffee again, which is translated into Chinese as "A woman drinks coffee again", where "again" is a temporal logic and is translated as "again". The target segment corresponding to this query text is 06:09~06:15. Still using three models to Figure 7 carry out time detection on the video, and the detection results are as Figure 7 shown.

[0169] Figure 6 、 Figure 7 demonstrates the model's understanding and response capabilities for complex queries with temporal logic. When the semantic modulation gated convolution module MD3 and the multi-head decoupled cross-modal feature fusion module MD2 are removed respectively, the model's localization of the query language significantly decreases, such as Figure 7 it is unable to accurately locate the boundary between the two highly similar scenarios of a woman drinking coffee and not drinking coffee, which shows the effectiveness of the designed module of the present invention.

[0170] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A natural language video time-of-view retrieval method based on multimodal fusion, characterized in that, Includes the following steps; S1, Obtain the dataset for natural language video time localization, where each sample includes video V, several target segments with time annotations, and query text corresponding to each target segment; S2, constructs a multi-scale temporal modeling module MD1 with one-channel perception, including a preprocessing unit, an improved stacked convolution unit, and a two-dimensional temporal adjacency graph generation unit; The preprocessing unit is used to divide the video V into N non-overlapping segments, extract features from the nth segment, and adjust its dimension to d. v To obtain fragment features Then, the video features F are formed. in , , d v The preset number of output channels; The improved stacked convolutional unit is used as input F in Output M convolutional video features at different scales, where the convolutional video feature at the m-th scale is F. m,out ; The two-dimensional temporal adjacency graph generation unit is used to generate F based on... m,out Generate an initial two-dimensional temporal adjacency graph Then Adjust to Size and A two-dimensional temporal adjacency graph F is constructed by stitching the graphs along the channel dimension. m F m The point at position (u,v) represents a candidate video segment with start and end times of u and v, respectively. S3, Obtain a text encoder to input the query text and generate word-level text features. and global text features T global , T word,l To query the word-level text features of the l-th word in the text; S4, construct a multi-head decoupled cross-modal feature fusion module MD2, used to fuse F m Reconstructed into two-dimensional temporal features, with T word and T global Perform fusion to generate fusion feature F m,fused ; S5, construct the semantic modulation gated convolutional module MD3; The semantic modulation gated convolution module is used for... Shape adjusted to N m ×N m ×d v This results in a reshaped adjacency graph. Then, the dynamic gating weight G and gating feature F are generated according to the following formula. gated The gated output feature F of the semantic modulation gated convolution module m,SG ; ， , ， In the formula, ReLU(∙) is the ReLU function, GAP(∙) is the global average pooling function, and W s W v These are the third and fourth weight matrices, respectively, in Conv2D. r For dilated convolution with dilation rate r, ; S6. Construct the natural language time-of-flight retrieval network and loss function, and adjust the parameters of the natural language time-of-flight retrieval network to minimize the loss, thus obtaining the natural language time-of-flight retrieval model; The natural language time-series retrieval network includes MD1, a text encoder, MD2, MD3, and a prediction head; The video V of the sample is fed into MD1, and the query text is fed into the text encoder. The outputs of the two are then processed by MD2 and MD3 to obtain M gated output features F. 1,SG ~F M,SG For each F m,SG The prediction head generates a corresponding two-dimensional fractional graph I. m,SG Collect I m,SG The set of valid fractions ,in For P m The i-th valid score, C m For P m The total number of valid scores will The corresponding candidate video segments are marked as ; The loss function is obtained from the following formula: ， In the formula, for Supervisory labels, for The IoU value at real time, when hour, ,otherwise ; S7 uses a natural language time-lapse retrieval model to detect the video to be identified.

2. The natural language video time-of-view retrieval method based on multimodal fusion according to claim 1, characterized in that, The datasets include the ActivityNet Captions dataset, the Charades-STA dataset, and the TACoS dataset. In S12, a pre-trained 3D convolutional network is used to extract features from the segments.

3. The natural language video time-of-view retrieval method based on multimodal fusion according to claim 1, characterized in that, In S2, the improved stacked convolutional unit includes a stacked convolutional unit composed of Z stacked convolutional layers. At the end of each convolutional layer, an enhancement unit is sequentially arranged. The enhancement unit includes a sparse sampling layer, a one-dimensional convolutional block, and a channel attention layer. When the input F... in The output F of the z-th convolutional layer conv,z Enhanced features are obtained by sequentially applying sparse sampling, one-dimensional convolution, and channel attention. , and then feed it into the next convolutional layer; The improved stacked convolutional unit is divided into M stages, and the enhanced features are obtained at the end of each stage as enhanced video features, where the enhanced video features of the m-th stage are labeled F. m,out .

4. The natural language video time-of-view retrieval method based on multimodal fusion according to claim 3, characterized in that, For the Z-layer convolutional unit of the improved stacked convolutional unit, Z=(K+1)A / 2, the kernel size of the first layer is 1 and the stride is 1, the kernel size of the (j1+1)A / 2 layer is 3 and the stride is 2, and the kernel size of the remaining layers is 2 and the stride is 1. A and K are the number of basic anchor points and the number of scales for video time-localization, respectively, and j1 is the number of convolutional layers, 1≤j1≤K-1.

5. The natural language video time-of-view retrieval method based on multimodal fusion according to claim 3, characterized in that, For the channel attention layer corresponding to the z-th convolutional layer, obtain the output F of the one-dimensional convolutional block. c,z , for F c,z Perform global average pooling to generate the first channel statistical vector. Second channel statistical vector And generate enhanced features according to the following formula. ; ， In the formula, FFN(∙) is the feedforward network, σ(∙) is the Sigmoid function, and W c Here, represents the channel attention weight, ⊙ represents element-wise multiplication, and... .

6. The natural language video time-of-view retrieval method based on multimodal fusion according to claim 1, characterized in that, The multi-head decoupled cross-modal feature fusion module MD2 includes a serialization layer, a video-guided multi-head cross-attention layer, a text-guided multi-head cross-attention layer, and a fusion layer. The serialization layer is used to convert F m Reconstructed into two-dimensional temporal features F rm , , N m For F m Length; H m For F rm The sequence length; The video-guided multi-head cross-attention layer will F rm As the K matrix and V matrix, T word As the Q matrix, cross-attention is performed to obtain the first attention feature F. temp ; The text-guided multi-head cross-attention layer will F temp As the K matrix and V matrix, F rm As the Q matrix, cross-attention is performed to obtain the second attention feature F. stage ; The fusion layer is used to generate fusion feature F according to the following formula. m,fused ; ， In the formula, σ(∙) is the Sigmoid activation function, ⊙ represents element-wise multiplication, and W g W v The first weight matrix and the second weight matrix are respectively, and LayerNorm(∙) is the layer normalization operation.

7. The natural language video time-of-view retrieval method based on multimodal fusion according to claim 1, characterized in that, In S3, the global text feature T global Calculate according to the following formula: ， In the formula, L represents the total number of words in the query text, and T represents the total number of words in the query text. word,l For T word The l-th element represents the word-level text feature of the l-th word in the query text.

8. The natural language video time-of-view retrieval method based on multimodal fusion according to claim 1, characterized in that, The effective score is I m,SG The non-zero value in the middle.

9. The natural language video time-of-view retrieval method based on multimodal fusion according to claim 1, characterized in that, S7 specifically involves: acquiring the video V' to be identified, obtaining M two-dimensional score maps by retrieving the MD1~MD3 of the natural language time-series model and the prediction head, adjusting them to the same size and fusing them into a single fused two-dimensional score map, and outputting one or more candidate video segments with the highest effective scores based on the fused two-dimensional score map. The fusion is to combine M two-dimensional fractional graphs I 1,SG ~I M,SG Adjust to the same size to obtain the corresponding adjustment diagram I. 1,SG ~I M,SG Construct a blank fused two-dimensional fractional graph I mix , to I mix The point at position (u,v) will I 1,SG ~I M,SG Find the maximum value at position (u,v) in the middle and fill it in I. mix .

10. The natural language video time-of-view retrieval method based on multimodal fusion according to claim 1, characterized in that, The text encoder consists of a pre-trained GloVe model and a BiLSTM.

Images