A Video Moment Retrieval Method Based on Spatiotemporal Domain Information Interaction

Abstract

The present invention discloses a video moment retrieval method based on spatio-temporal domain information interaction, comprising the following steps: obtaining an original video set and a query statement set; constructing a video feature pre-extraction module; constructing a text feature pre-extraction module to obtain text features; performing multi-scale transformation on the video features to generate multi-scale video features; performing multi-scale feature extraction on the text features to generate multi-scale text features; constructing a multi-scale feature interaction module to generate multi-scale interaction features; constructing a spatio-temporal domain information fusion module to obtain spatio-temporal domain information features; constructing a moment positioning module to obtain the final start time and end time; training a video moment retrieval model based on spatio-temporal domain information interaction; and using the trained video moment retrieval model based on spatio-temporal domain information interaction to conduct experiments on video and text pairs in a test set. The present invention can achieve the improvement of the accuracy of video moment retrieval.

Description

Technical Field

[0001] The present invention relates to the technical field of video moment retrieval, and particularly relates to a video moment retrieval method based on spatio-temporal domain information interaction. Background Art

[0002] Currently, the mainstream cross-modal video moment retrieval methods can be mainly divided into early two-stage matching methods, single-stage methods, reinforcement learning-based methods, and the latest weakly supervised guidance-based methods. The early two-stage methods will first process the entire video, preprocess the entire video into multiple candidate segments through a sliding window strategy or a candidate generation network, and then rank the generated candidate segments through a cross-modal matching module to obtain the corresponding time results. Because of the excessive overlapping redundancy of candidate segments affecting the calculation efficiency, some research works have begun to try to use single-stage methods to solve the video moment retrieval task. Single-stage methods do not rely on any candidate segment generation process. In the anchor-based single-stage methods, a long short-term memory network or a convolutional neural network is used to select multi-scale candidate segments at each time step. In the anchor-free single-stage methods, each video unit, such as a video frame or a frame segment clip, is predicted to obtain the probability of its starting point and ending point as the target segment, or the coordinates of the target starting point and ending point are directly obtained based on multi-modal feature regression. In addition, some works use reinforcement learning to solve the video moment retrieval problem, and these works regard the moment localization problem as a series of continuous decision-making processes. There are also some works that propose weakly supervised methods for video-level annotation in order to reduce the heavy labor in annotating video moment datasets.

[0003] Due to the particularity and complexity of this task, the current cross-modal video moment retrieval technology still has great deficiencies, and the returned results are often not very accurate, and the accuracy still cannot satisfy users.

[0004] The patent application with the publication number CN202011575231 and the name "Cross-modal video moment retrieval method based on cross-modal dynamic convolutional network" discloses a cross-modal video moment retrieval method based on cross-modal dynamic convolutional network. This method uses a moment localization module based on cross-modal convolutional neural network to perform moment retrieval, and uses fusion features and text features to dynamically generate convolutional kernels, thereby improving the performance of the cross-modal video moment retrieval task. However, the deficiency of this method is that there is a lack of interaction at different levels between the video and the query statement, and it is impossible to accurately locate some context-dependent actions, resulting in a decrease in the retrieval accuracy.

[0005] In a video, the presentation of a lot of information depends on the context information of the video content. The content included in the video context information also varies at different spatio-temporal scales. Existing video moment retrieval methods lack the full utilization of feature information at different scales and the interaction between the video and the query statement at different levels, resulting in the inability of the moment retrieval model to accurately locate some actions that rely on the context, thus reducing the accuracy of the retrieval. Summary of the Invention

[0006] To overcome the above technical problems, the purpose of the present invention is to provide a video moment retrieval method based on spatio-temporal domain information interaction, extend the spatio-temporal domain information interaction to multiple scales, complete the video moment retrieval task, and construct a multi-scale retrieval model to improve the accuracy of video moment retrieval. It is used to solve the problems in the prior art that there is a lack of full utilization of feature information at different scales, the lack of interaction between the video and the query statement at different levels, and the inability to accurately locate some actions that rely on the video context information, resulting in a decrease in the accuracy rate of moment retrieval.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is:

[0008] A video moment retrieval system based on spatio-temporal domain information interaction, including a feature pre-extraction module, a multi-scale feature interaction module, a spatio-temporal domain information fusion module, and a moment positioning module;

[0009] The feature pre-extraction module is used to extract video features and text features;

[0010] The multi-scale feature interaction module is used for the multi-scale extraction and fusion of the video features and text features, and generates multi-scale interaction features including visual information at different scales and text information at different levels;

[0011] The visual information at different scales is video features at different scales: it can more fully explore the visual information in the video, including global action information and local detail information in the video.

[0012] The text information at different levels is text features at different scales: it can make full use of the text information of the text, and the text features at different scales contain different text information (semantic information).

[0013] Multi-scale interaction refers to the interaction between video features and text features, combines relevant video features and text features, and enhances the correspondence between the video and the text. The specific operation of multi-scale interaction is to perform fusion processing on video features and text features at different feature sizes.

[0014] The spatio-temporal domain information fusion module is used for information fusion of multi-scale interaction features and text features in the spatio-temporal domain, generating spatio-temporal domain information features and deeply encoding various information of the video in the time and space dimensions;

[0015] The moment positioning module is used to obtain the final positioning result and complete cross-modal video moment retrieval.

[0016] A video moment retrieval method based on spatio-temporal domain information interaction includes the following steps;

[0017] Step 1, obtain the original video set and query statement set, and construct a training data set and a test data set; the data set contains video and query statement (text) pairs;

[0018] Step 2, construct a video feature pre-extraction module, and use a pre-trained network to pre-extract features from the original video V of the training data set to obtain video features F;

[0019] Step 3, construct a text feature pre-extraction module, and use a pre-trained word embedding model to map the training data set S of the query statement into the embedding space to complete feature extraction and obtain text features Q;

[0020] Step 4, perform multi-scale transformation on the video features F in Step 2 to generate multi-scale video features;

[0021] Step 5, perform multi-scale feature extraction on the text features Q in Step 3 to generate multi-scale text features

[0022] Step 6, construct a multi-scale feature interaction module, and use video and text features at different scale levels to interact in the fusion stage to generate multi-scale interaction features F p ;

[0023] Step 7, construct a spatio-temporal domain information fusion module, and combine the multi-scale interaction features F i p with the multi-scale text features to obtain spatio-temporal domain information features

[0024] Step 8, construct a moment positioning module, preprocess the spatio-temporal domain information features obtained in Step 7, and then analyze and extract the features to obtain the final start time t and end time t s and end time t e ;

[0025] Step 9, use the Adam optimizer to train the video moment retrieval model based on spatio-temporal domain information interaction, and define a loss function based on KL divergence to guide model training;

[0026] Step 10: Use the trained video moment retrieval model based on spatio-temporal domain information interaction to experiment on the video and text pairs in the test set to verify the effectiveness of the video moment retrieval method based on spatio-temporal domain information interaction.

[0027] In step 2, a two-stream dilated 3D network is selected to extract video feature F. The 2D convolution in the two-stream dilated 3D network is extended to 3D convolution, and the convolution kernel and pooling layer in the two-stream dilated 3D network are added with the time dimension. The specific steps of learning time features through 3D convolution are as follows:

[0028] Step 2.1: Divide the original video V into a series of non-overlapping segments at a certain interval. The total number of frames of the original video V is T, and the video is divided into n snp segments, and each video segment contains k v frames;

[0029] Step 2.2: Use the pre-trained two-stream dilated 3D network to extract the video feature F of each video segment.

[0030] Step 3 is specifically as follows:

[0031] Select the GloVe model to convert the query statement into an embedding vector. First, construct a co-occurrence matrix X, and each element X in the matrix ij represents the number of times word i and context word j co-occur within the context window. Then, obtain the approximate relationship between the word vector and the co-occurrence matrix. Finally, construct a loss function based on the error of the word vector representation for learning; through the learning of the word vector, the GloVe model captures the semantic relationship between words and extracts the text feature Q corresponding to the query statement S.

[0032] Step 4 is specifically as follows:

[0033] Step 4.1: Compress the dimension of the video feature F in step 2 through a fully connected layer, and reduce the number of dimensions to 128;

[0034] Step 4.2: Use convolution to perform multi-scale extraction on the video feature to obtain 4 video features with different scales:

[0035] Step 4.21: The video feature F first passes through a one-dimensional convolutional layer with a convolutional kernel size of 1 and 128 layers to obtain the initially extracted detailed feature F1 o ;

[0036] Step 4.22: Pass F1 o through a one-dimensional convolutional layer with a convolutional kernel size of 1 and 64 layers to obtain the video feature of the second layer

[0037] Step 4.23: Pass The video features of the third layer are obtained through a one-dimensional convolutional layer with a kernel size of 1 and 32 layers.

[0038] Step 4.24, The video features of the fourth layer are obtained through a one-dimensional convolutional layer with a kernel size of 1 and 16 layers. The purpose of doing this is to perform multi-scale extraction of video features using convolutional operations, capture the global and local information of the video, and obtain multi-scale video features.

[0039] The specific steps of step 5 are as follows:

[0040] Step 5.1, expand the dimension of the text feature Q obtained in step 3 through a fully connected layer, and the number of dimensions is increased to 128;

[0041] Step 5.2, perform multi-scale temporal feature extraction through a bidirectional gated recurrent unit network to obtain the processed text feature The bidirectional gated recurrent unit network uses two independent gated recurrent units, one scans the input sequence from front to back, and the other scans the input sequence from back to front, and combines the hidden states obtained by them to extract text information;

[0042] Step 5.3, use one-dimensional adaptive average pooling layers with three different scales to perform pooling operations on the processed text feature :

[0043] Step 5.31, the text feature First, pass through an adaptive average pooling layer with a dimension size of 64 to obtain a pooled feature

[0044] Step 5.32, pass the pooled feature Through an adaptive average pooling layer with a dimension size of 32 to obtain a pooled feature

[0045] Step 5.33, pass the pooled feature Through an adaptive average pooling layer with a dimension size of 16 to obtain a pooled feature The purpose of doing this is to perform multi-scale extraction of text features using pooling operations, obtain multi-level text information expressions, and obtain multi-scale text features

[0046] The multi-scale video feature F o is a set: F o = {F i o, i = 1, 2, 3, 4}, which contains video features of 4 scales, namely the four-layer video features obtained by convolution in steps 4.21 - 4.24.

[0047] The specific steps of step 6 are as follows:

[0048] Step 6.1, for the multi-scale text features Use nearest neighbor interpolation to obtain Interpolate and fill the length m of the text features to be the same size as the multi-scale video feature F i o Length n snp The same size;

[0049] Step 6.2, at the same scale i, add the multi-scale video feature F i o To the corresponding multi-scale text features And add the feature of the previous layer after expanding the dimension using transposed convolution to obtain the video feature F p Of the completed multi-scale interaction, the specific fusion process is as follows:

[0050]

[0051] Among them, F i o Is the multi-scale video feature, Is the multi-scale text feature after interpolation, ConvT (·) Represents the transposed convolution with an output dimension of (·).

[0052] The purpose of doing this is to use transposed convolution to enable the network to process video features and text features in more detail at different feature sizes, and thereby capture various different information in the video and text, so as to improve the representation ability of the multi-scale feature interaction module and make the generated multi-scale interaction features contain visual information at different scales and text information at different levels;

[0053] The multi-scale text feature Q g Is a set: The text feature after temporal feature extraction in step 5.2 is After performing multi-scale transformation in step 5.3, another 3 scales of text features are obtained

[0054] The specific steps of step 7 are as follows:

[0055] Step 7.1, calculate the Hadamard product of the interpolated text features And the multi-scale interaction feature F p To obtain the spatio-temporal domain feature F i r, The calculation formula is as follows:

[0056]

[0057] Among them, F i p ={f ij}}, f ij is the element in the i-th row and j-th column of matrix F i p , q ij is the element in the i-th row and j-th column of matrix , represents the Hadamard product operation;

[0058] Step 7.2, Use convolution to perform multi-scale extraction on the spatio-temporal domain feature F r to obtain three spatio-temporal domain scale features of different scales:

[0059] Step 7.21, The spatio-temporal domain feature F r passes through a one-dimensional convolutional layer with a kernel size of 1 and 64 layers to obtain the first-layer spatio-temporal domain scale feature

[0060] Step 7.22, Pass through a one-dimensional convolutional layer with a kernel size of 1 and 64 layers to obtain the second-layer spatio-temporal domain scale feature

[0061] Step 7.23, Pass through a one-dimensional convolutional layer with a kernel size of 1 and 32 layers to obtain the third-layer spatio-temporal domain scale feature

[0062] Step 7.3, Fuse the spatio-temporal domain feature F i r with the spatio-temporal domain scale feature of the corresponding layer to obtain the required spatio-temporal domain information feature The specific fusion process is as follows:

[0063]

[0064] Among them, is the interpolated multi-scale text feature, F i p is the multi-scale interaction feature, Conv1d (·) represents the convolution with an output dimension of (·).

[0065] The purpose of doing this is to further combine and structure the spatio-temporal domain information, so as to improve the understanding and description ability of the spatio-temporal domain information fusion module for time relations and spatial relations, better capture the complex relations between features, and make the generated spatio-temporal domain information features deeply encode various information in the time and space dimensions of the video, including video content, spatial structure, dynamic features, etc. The specific steps of step 8 are as follows:

[0066] Step 8.1, use a bidirectional gated recurrent unit network to preprocess the spatio-temporal domain information features obtained in step 7 for temporal feature preprocessing, and the preprocessing is used to capture the change trend of the spatio-temporal domain information features;

[0067] Step 8.2, input the spatio-temporal domain information features preprocessed in step 8.1 into two fully connected layers with an output size of 1 respectively to obtain the predicted start time positioning vector T ps and the predicted end time positioning vector T pe ;

[0068] Step 8.3, use the Softmax function to transform the predicted vectors T ps and T pe to obtain the final predicted start time t ps and the predicted end time t pe .

[0069] The loss function formula based on KL divergence in step 9 is as follows:

[0070]

[0071] where V ps and V pe represent the predicted start time distribution and end time distribution respectively, and V gs and V ge represent the true start time and end time distributions.

[0072] Advantages of the present invention:

[0073] The present invention utilizes multi-scale spatio-temporal domain information interaction to extract video and text information at different scales, capture the global action information and local detail information of the video, and obtain multi-level text information expressions, making more comprehensive use of the information in the video and the query statement. And using spatio-temporal domain information fusion can effectively represent the content in the video, and the obtained features deeply encode various information in the time and space dimensions of the video, including video content, spatial structure, dynamic features, etc. These information features are more important in the video retrieval task, can better identify the targets, actions, etc. in the video, and achieve higher video moment retrieval accuracy. Description of the Drawings

[0074] Figure 1 This is a schematic diagram of the process of the present invention. Specific embodiments

[0075] The present invention will be further described in detail below in conjunction with embodiments.

[0076] The invention is implemented by adopting a video moment retrieval model based on spatio-temporal domain information interaction. The video moment retrieval model based on spatio-temporal domain information interaction includes a feature pre-extraction module, a multi-scale feature interaction module, a spatio-temporal domain information fusion module, and a moment positioning module. The following will be combined with the attached Figure 1 and examples to further describe the present invention in detail

[0077] Step 1, obtain the original video set and query statement set, and construct a training data set and a test data set:

[0078] In this example, the Charades-STA data set and the ActivityNet Captions data set are selected for experiments. The Charades-STA data set is constructed based on the Charades data set, containing 6672 videos and 16124 annotations. The average video length in the data set is 30.60 seconds, the average query moment length is 8.09 seconds, and the average length of the query statement is 7.22 words. The ActivityNet Captions data set is constructed based on the ActivityNet data set, containing 14962 videos and 71953 annotations. Each video contains an average of 4.82 moment positioning statements on average. The average length of the video is 117.60 seconds, the average length of the query moment is 7.14 seconds, and the average query length is 14.41 words.

[0079] Step 2, construct a video feature pre-extraction module, and use a pre-trained network to pre-extract features from the original video V to obtain video features F:

[0080] In this example, a two-stream dilated 3D network is selected to extract video features. This method expands the two-dimensional convolution in the two-stream network into three-dimensional convolution, adds the time dimension to the convolution kernels and pooling layers in the network, learns time features through three-dimensional convolution, and can effectively improve performance after adding optical flow.

[0081] Step 2.1, divide the original video V into a series of non-overlapping segments at a certain interval. The total number of frames of the original video is T, and the video is divided into n snp segments, and each video segment contains k v frames.

[0082] Step 2.2, use the pre-trained two-stream dilated 3D network to extract the video features F of each video segment;

[0083] Step 3, construct a text feature pre-extraction module, and use a pre-trained word embedding model to map the query statement S into the embedding space to complete feature extraction and obtain the text feature Q:

[0084] In this example, the GloVe model is selected to convert the query statement into an embedding vector. This method needs to first construct a co-occurrence matrix, then obtain the approximate relationship between the word vector and the co-occurrence matrix, and finally construct a loss function based on the error of the word vector representation for learning. Through the learning of the word vector, the GloVe model can capture the semantic relationship between words and extract the text feature Q corresponding to the query statement S.

[0085] Step 4, perform multi-scale transformation on the video feature F in Step 2 to generate multi-scale video features:

[0086] Step 4.1, compress the dimension of the video feature F through a fully connected layer, and reduce the number of dimensions to 128;

[0087] Step 4.2, perform multi-scale extraction on the video feature using convolution to obtain 4 video features with different scales:

[0088] Step 4.21, the video feature F first passes through a one-dimensional convolutional layer with a kernel size of 1 and 128 layers to obtain the initially extracted detailed feature F1 o ;

[0089] Step 4.22, pass F1 o through a one-dimensional convolutional layer with a kernel size of 1 and 64 layers to obtain the video feature of the second layer

[0090] Step 4.23, pass through a one-dimensional convolutional layer with a kernel size of 1 and 32 layers to obtain the video feature of the third layer

[0091] Step 4.24, pass through a one-dimensional convolutional layer with a kernel size of 1 and 16 layers to obtain the video feature of the fourth layer

[0092] Step 5, perform multi-scale feature extraction on the text feature Q in Step 3 to generate multi-scale text features Q i g :

[0093] Step 5.1, expand the dimension of the text feature Q through a fully connected layer, and increase the number of dimensions to 128;

[0094] Step 5.2, perform multi-scale temporal feature extraction through a bidirectional gated recurrent unit network to obtain The bidirectional gated recurrent unit network utilizes two independent gated recurrent units, one scanning the input sequence from front to back and the other scanning the input sequence from back to front, and combines the hidden states obtained from them to extract text information.

[0095] Step 5.3, use one-dimensional adaptive average pooling layers with three different scales to perform pooling operations on the text features as follows:

[0096] Step 5.31, the video features first pass through an adaptive average pooling layer with a dimension size of 64 to obtain the pooled features

[0097] Step 5.32, the pass through an adaptive average pooling layer with a dimension size of 32 to obtain the pooled features

[0098] Step 5.33, the pass through an adaptive average pooling layer with a dimension size of 16 to obtain the pooled features

[0099] Step 6, construct a multi-scale feature interaction module, use video and text features at different scale levels to interact in the fusion stage, and generate multi-scale interaction features F p :

[0100] Step 6.1, use nearest neighbor interpolation for the text features to obtain interpolate and fill the length m of the text features to the same size as the length n of the video features snp ;

[0101] Step 6.2, use deconvolution to expand the dimension of the video feature F o add it to the corresponding text feature, and add the feature of the previous layer to obtain the video feature F after multi-scale interaction p , and the specific fusion process is as follows:

[0102]

[0103] Step 7, construct a spatio-temporal domain information fusion module, combine the multi-scale interaction feature F i p with the multi-scale text features to obtain the spatio-temporal domain information feature

[0104] Step 7.1, calculate the Hadamard product of the interpolated text features and the multi-scale interaction feature F p to obtain the spatio-temporal domain feature Fi r , The calculation formula is as follows:

[0105]

[0106] Among them,

[0107] Step 7.2, Use convolution to perform multi-scale extraction on the spatio-temporal domain feature F r to obtain three spatio-temporal domain scale features of different scales:

[0108] Step 7.21, The spatio-temporal domain feature F r passes through a one-dimensional convolutional layer with a kernel size of 1 and 64 layers to obtain the first-layer spatio-temporal domain scale feature

[0109] Step 7.22, Pass F1 r through a one-dimensional convolutional layer with a kernel size of 1 and 64 layers to obtain the second-layer spatio-temporal domain scale feature

[0110] Step 7.23, Pass through a one-dimensional convolutional layer with a kernel size of 1 and 32 layers to obtain the third-layer spatio-temporal domain scale feature

[0111] Step 7.3, Fuse the spatio-temporal domain feature F i r with the spatio-temporal domain scale feature of the corresponding layer to obtain the required spatio-temporal domain information feature The specific fusion process is shown as follows:

[0112]

[0113] Step 8, Construct a moment positioning module to preprocess the spatio-temporal domain information feature obtained in Step 7 , and then analyze and extract the feature to obtain the final start time t s and end time t e :

[0114] Step 8.1, Use a bidirectional gated recurrent unit network to preprocess the temporal sequence feature of the spatio-temporal domain information feature F4 r to capture the change trend of the feature;

[0115] Step 8.2, Input the spatio-temporal domain information feature obtained in Step 8.1 into two fully connected layers with an output size of 1 respectively to obtain the predicted start moment positioning vector T ps and the predicted end moment positioning vector T pe ;

[0116] Step 8.3, use the Softmax function to transform the prediction vectors T ps and T pe to obtain the final predicted start time t ps and the predicted end time t pe ;

[0117] Step 9, use the Adam optimizer to train the video moment retrieval model based on spatio-temporal domain information interaction, and define a loss function based on KL divergence to guide the model training:

[0118]

[0119] where V ps and V pe represent the predicted start time distribution and end time distribution respectively, and V gs and V ge represent the distribution of the true start time and end time

[0120] Step 10, use the trained video moment retrieval model based on spatio-temporal domain information interaction to conduct experiments on the video and text pairs in the test set to verify the effectiveness of the video moment retrieval method based on spatio-temporal domain information interaction.

[0121] The present invention further extends the spatio-temporal domain information interaction to the multi-scale level, performs multi-scale transformation on the video features using convolutional and deconvolutional networks, simultaneously performs multi-scale feature extraction on the text information using gated recurrent unit networks, and finally fuses the video and text features in the spatio-temporal domain to construct a moment retrieval model with multi-scale information.

Claims

1. A video moment retrieval method based on spatio-temporal domain information interaction, characterized in that, It includes the following steps; Step 1: Obtain the original video set and query statement set, and construct a training data set and a test data set; The data set contains video and query statement pairs; Step 2: Construct a video feature pre-extraction module, and use a pre-trained network to pre-extract features from the original video V in the training data set to obtain video features F; Step 3: Construct a text feature pre-extraction module, and use a pre-trained word embedding model to map the query statement training data set S into the embedding space to complete feature extraction and obtain text features Q; Step 4: Use the video features F in Step 2 for multi-scale transformation to generate multi-scale video features; Step 5: Use the text feature Q in Step 3 to perform multi-scale feature extraction to generate multi-scale text features Step 6, construct a multi-scale feature interaction module, and use video and text features at different scale levels to interact during the fusion stage to generate multi-scale interaction feature F p ; Step 7, construct a spatio-temporal domain information fusion module to combine the multi-scale interaction feature F i p with the multi-scale text feature to obtain the spatio-temporal domain information feature Step 8, construct a time positioning module to preprocess the spatio-temporal domain information features obtained in Step 7, and then analyze and extract the features to obtain the final start time t and end time t s ; e ; Step 9: Use the Adam optimizer to train the video moment retrieval model based on spatio-temporal domain information interaction, and define a loss function based on KL divergence to guide model training; Step 10: Use the trained video moment retrieval model based on spatio-temporal domain information interaction to conduct experiments on the video and text pairs in the test set to verify the effectiveness of the video moment retrieval method based on spatio-temporal domain information interaction; The method is implemented through a video moment retrieval system based on spatio-temporal domain information interaction. The system includes a feature pre-extraction module, a multi-scale feature interaction module, a spatio-temporal domain information fusion module, and a moment localization module; The feature pre-extraction module is used to extract video features and text features; The multi-scale feature interaction module is used for multi-scale extraction and fusion of the video features and text features to generate multi-scale interaction features containing visual information at different scales and text information at different levels; The spatio-temporal domain information fusion module is used for information fusion of the multi-scale interaction features and text features in the spatio-temporal domain to generate spatio-temporal domain information features, deeply encoding various information of the video in the time and space dimensions; The moment localization module is used to obtain the final localization result and complete cross-modal video moment retrieval.

2. The video moment retrieval method based on spatio-temporal domain information interaction according to claim 1, wherein In Step 2, a two-stream dilated 3D network is selected to extract video features F. The 2D convolution in the two-stream dilated 3D network is extended to 3D convolution, and the convolution kernel and pooling layer in the two-stream dilated 3D network are added with the time dimension. The specific steps of learning time features through 3D convolution include: Step 2.1, divide the original video V into a series of non - overlapping segments at a certain interval. The total number of frames of the original video V is T, and the video is divided into n snp segments, and each video segment contains k v frames; Step 2.2: Use the pre-trained two-stream dilated 3D network to extract video features F of each video segment.

3. A video moment retrieval method based on spatio-temporal domain information interaction according to claim 1, characterized in that The specific content of Step 3 is as follows: Select the GloVe model to convert the query statement into an embedded vector. First, construct the co-occurrence matrix X, where each element X ij represents the number of times word i co-occurs with context word j within the context window. Then, obtain the approximate relationship between the word vectors and the co-occurrence matrix. Finally, construct a loss function based on the error of the word vector representation for learning. Through the learning of word vectors, the GloVe model captures the semantic relationships between words and extracts the text feature Q corresponding to the query statement S.

4. A video moment retrieval method based on spatio-temporal domain information interaction according to claim 1, characterized in that, The specific content of Step 4 is as follows: Step 4.1: Compress the dimension of the video features F in Step 2 through a fully connected layer, and reduce the number of dimensions to 128; Step 4.2: Use convolution to perform multi-scale extraction on the video features to obtain 4 video features at different scales: Step 4.21, the video feature F first passes through a one-dimensional convolutional layer with a convolutional kernel size of 1 and 128 layers to obtain the initially extracted detailed feature F1 o ; Step 4.22, take F1 o Through a one-dimensional convolutional layer with a kernel size of 1 and 64 layers, obtain the video features of the second layer Step 4.23, take through a one-dimensional convolutional layer with a convolutional kernel size of 1 and 32 layers to obtain the video features of the third layer Step 4.24, input through a one-dimensional convolutional layer with a kernel size of 1 and 16 layers to obtain the video features of the fourth layer 5. A video moment retrieval method based on spatio-temporal domain information interaction according to claim 1, characterized in that, The specific content of Step 5 is as follows: Step 5.1: Expand the dimension of the text features Q obtained in Step 3 through a fully connected layer, and increase the number of dimensions to 128; Step 5.2, perform multi-scale temporal feature extraction through a bidirectional gated recurrent unit network to obtain the processed text features The bidirectional gated recurrent unit network uses two independent gated recurrent units, one scans the input sequence from front to back, and the other scans the input sequence from back to front, and combines the hidden states obtained by them to extract text information; Step 5.3, use one-dimensional adaptive average pooling layers with three different scales to perform pooling operations on the processed text features : Step 5.31, text feature First, an adaptive average pooling layer with a dimension size of 64 is used to obtain the pooling feature Step 5.32, the pooled feature passes through an adaptive average pooling layer with a dimension size of 32 to obtain the pooled feature Step 5.33, the pooled feature passes through an adaptive average pooling layer with a dimension size of 16 to obtain the pooled feature 6. The video moment retrieval method based on spatio-temporal domain information interaction according to claim 1, wherein, The specific content of Step 6 is as follows: Step 6.1, for the multi-scale text features Use nearest neighbor interpolation to obtain Interpolate and fill the length m of the text features to the same size as the multi-scale video feature F i o Length n snp the same size; Step 6.2, for the multi-scale video feature F i o Use deconvolution to expand the dimension, and add it to the corresponding multi-scale text feature Perform addition, and add the feature of the previous layer to obtain the video feature F after multi-scale interaction p , and the specific fusion process is as follows: Among them, F i o is the multi-scale video feature, is the interpolated multi-scale text feature, and ConvT (·) represents a transposed convolution with an output dimension of (·).

7. A video moment retrieval method based on spatio-temporal domain information interaction according to claim 1, characterized in that, The specific content of Step 7 is as follows: Step 7.1, for the interpolated text features and the multi-scale interaction feature F p calculate the Hadamard product to obtain the spatio-temporal domain feature F i r , and the calculation formula is as follows: Among them, F i p = {f ij}, f ij is the element in the i-th row and j-th column of matrix F i p , q ij is the element in the i-th row and j-th column of matrix , represents the Hadamard product operation; Step 7.2, perform multi-scale extraction on the spatio-temporal domain feature F r to obtain three spatio-temporal domain scale features of different scales: Step 7.21, spatio-temporal domain feature F r Through a one-dimensional convolutional layer with a kernel size of 1 and 64 layers, the first spatio-temporal domain scale feature is obtained Step 7.22, obtain the second-layer spatio-temporal domain scale features through a one-dimensional convolutional layer with a convolutional kernel size of 1 and 64 layers Step 7.23, take through a one-dimensional convolutional layer with a kernel size of 1 and 32 layers to obtain the third-layer spatio-temporal domain scale features Step 7.3, the spatio-temporal domain feature F i r is fused with the spatio-temporal domain scale feature of the corresponding layer to obtain the required spatio-temporal domain information feature The specific fusion process is shown as follows: Among them, is the interpolated multi-scale text feature, is the multi-scale interaction feature, and Conv1d (·) represents a convolution with an output dimension of (·).

8. A video moment retrieval method based on spatio-temporal domain information interaction according to claim 1, characterized in that, The specific content of Step 8 is as follows: Step 8.1, use a bidirectional gated recurrent unit network to perform temporal feature preprocessing on the spatio-temporal domain information features obtained in the above step 7. The preprocessing is used to capture the change trend of the spatio-temporal domain information features; ​ Step 8.2, input the spatio-temporal domain information features preprocessed in Step 8.1 into two fully connected layers with an output size of 1 respectively, to obtain the predicted start time positioning vector T ps and the predicted end time positioning vector T pe ; Step 8.3, use the Softmax function to transform the prediction vectors T ps and T pe to obtain the final predicted start time t ps and the predicted end time t pe .

9. A video moment retrieval method based on spatio-temporal domain information interaction according to claim 1, characterized in that The loss function formula of step 9 based on KL divergence is as follows: Among them, V ps and V pe represent the predicted start time distribution and end time distribution respectively, and V gs and V ge represent the distributions of the true start time and end time.

Images