A video moment retrieval method based on multimodal fusion of text and video

Abstract

The present application relates to a video moment retrieval method of multimodal fusion of text and video, which globally fuses text features and multiscale video features to obtain a global offset; locally fuses text features and multiscale video features to obtain a local offset; accumulates the global offset and the local offset to obtain a total offset; determines the time segment corresponding to each candidate moment according to the total offset; processes all time segments using a non-maximum suppression method to obtain the final time segment, which is the time segment corresponding to the determined text query information in the original video. The present application solves the problem that the prior art ignores the fusion of global features and local features, and can capture the global semantic consistency between text and video, while being able to fuse at a more fine-grained level, effectively improving the accuracy of video moment retrieval.

Description

Technical Field

[0001] The present application relates to the field of video moment retrieval, and in particular, to a video moment retrieval method for multimodal fusion of text and video. Background Art

[0002] Video moment retrieval aims to locate specific moments in uncut videos that semantically correspond to a given language query. The emergence of video moment retrieval technology helps reduce the workload of video analysis and has many applications, such as in robot navigation, autonomous driving, and massive video surveillance analysis.

[0003] Advances in deep learning have significantly improved the performance of video moment retrieval. Currently, there are roughly two types of video moment retrieval methods: two-stage retrieval methods and one-stage retrieval methods. The two-stage retrieval method pre-segments the video through a multi-scale sliding window or candidate segment generation network, generates multiple candidate segments, and then determines the best matching result through repeated statement queries and candidate segment matching.

[0004] One-stage retrieval methods usually directly regress the start and end coordinates of the target time period based on the multimodal features of the provided video and sentence query. Although existing methods have made good progress, two-stage detection methods require additional candidate segment generation and candidate segment selection steps, which are computationally expensive and may affect efficiency due to the inaccuracy of the generated candidate set. For one-stage detection methods, how to effectively utilize data from multiple modalities becomes particularly important. Current methods usually simply splice or locally mix the features of the two modalities of text and video, ignoring the importance of global fusion and local fusion. Summary of the invention

[0005] In order to overcome at least one deficiency in the prior art, the present application provides a video moment retrieval method for multimodal fusion of text and video.

[0006] In a first aspect, a video moment retrieval method for multimodal fusion of text and video is provided, comprising:

[0007] A text encoder is used to extract features from text query information to obtain text features;

[0008] The video feature extraction backbone network is used to extract features from the original video to obtain video features; and the visual encoder is used to convert the video features into multi-scale video features;

[0009] The text features and the multi-scale video features are globally fused to obtain a global offset; the text features and the multi-scale video features are locally fused to obtain a local offset; the global offset and the local offset are accumulated to obtain a total offset;

[0010] Input text features and multi-scale video features into the query-dependent convolutional classifier to obtain the semantic score corresponding to each moment in the original video; select the moment corresponding to the semantic score greater than the set threshold as the candidate moment;

[0011] Determine the time segment corresponding to each candidate moment according to the total offset;

[0012] All time segments are processed using a non-maximum suppression method to obtain the final time segment, which is the time segment corresponding to the determined text query information in the original video.

[0013] In one embodiment, the text features and the multi-scale video features are globally fused to obtain a global offset, including:

[0014] After the text features are processed by the multi-layer perceptron and tensor transformation operations, the convolution kernel weights are obtained;

[0015] Concatenate text features with multi-scale video features in the channel dimension to obtain concatenated features;

[0016] Perform convolution operation on the concatenated features to obtain mixed features;

[0017] Perform Fourier convolution on the mixed features to obtain the features after Fourier convolution;

[0018] Perform dynamic mask generation on the features after Fourier convolution to obtain a mask tensor; multiply the mask tensor and the features after Fourier convolution to obtain a multiplication result; perform inverse Fourier transform on the multiplication result to obtain the features after the inverse operation;

[0019] The features after the inverse operation are convolved with the convolution kernel weights to obtain the global offset.

[0020] In one embodiment, the text features and the multi-scale video features are locally fused to obtain a local offset, including:

[0021] After the text features are processed by the multi-layer perceptron and tensor transformation operations, the convolution kernel weights are obtained;

[0022] The multi-scale video features are convolved with the convolution kernel weights to obtain the local offset.

[0023] In one embodiment, determining the time segment corresponding to each candidate moment according to the total offset includes:

[0024] Calculate the start and end time of each candidate moment:

[0025]

[0026] t

[0027] in, is the start time of the time segment to which the candidate time t belongs, is the end time of the time segment to which the candidate time t belongs, is the offset of the starting time from the candidate time in the total offset, is the offset of the end time from the candidate time in the total offset;

[0028] The time segment corresponding to each candidate moment is [ ].

[0029] In a second aspect, a video moment retrieval device for multimodal fusion of text and video is provided, comprising:

[0030] A text feature extraction module is used to extract features from text query information using a text encoder to obtain text features;

[0031] The video feature extraction module is used to extract features from the original video using a video feature extraction backbone network to obtain video features; and to convert the video features into multi-scale video features using a visual encoder;

[0032] A fusion module is used to globally fuse text features and multi-scale video features to obtain a global offset; locally fuse text features and multi-scale video features to obtain a local offset; and accumulate the global offset and the local offset to obtain a total offset;

[0033] The candidate moment determination module is used to input text features and multi-scale video features into the query-dependent convolution classifier to obtain the semantic score corresponding to each moment in the original video; the moment corresponding to the semantic score greater than the set threshold is selected as the candidate moment;

[0034] A time segment determination module, used to determine the time segment corresponding to each candidate moment according to the total offset;

[0035] The final result determination module is used to process all time segments using a non-maximum suppression method to obtain a final time segment, which is the time segment corresponding to the determined text query information in the original video.

[0036] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the video moment retrieval method for multimodal fusion of text and video is implemented.

[0037] In a fourth aspect, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned video moment retrieval method for multimodal fusion of text and video.

[0038] Compared with the prior art, the present application has the following beneficial effects: the video moment retrieval method of multimodal fusion of text and video of the present application solves the problem that the prior art ignores the fusion of global features and local features. It can not only capture the global semantic consistency between text and video, but also can fuse them at a finer-grained level, effectively improving the accuracy of video moment retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The present application may be better understood by referring to the following description given in conjunction with the accompanying drawings, which together with the following detailed description are included in this specification and form a part of this specification. In the drawings:

[0040] Figure 1 A flowchart of a video moment retrieval method for multimodal fusion of text and video is shown;

[0041] Figure 2 The schematic diagram of global fusion and local fusion is shown. DETAILED DESCRIPTION

[0042] The exemplary embodiments of the present application will be described below in conjunction with the accompanying drawings. For the sake of clarity and conciseness, not all features of the actual embodiments are described in the specification. However, it should be understood that many implementation-specific decisions can be made in the process of developing any such actual embodiments in order to achieve the specific goals of the developer, and these decisions may vary from embodiment to embodiment.

[0043] It is also necessary to explain here that, in order to avoid obscuring the present application due to unnecessary details, only the device structure closely related to the scheme according to the present application is shown in the drawings, while other details that are not closely related to the present application are omitted.

[0044] It should be understood that the present application is not limited to the described implementation forms due to the following description with reference to the accompanying drawings. In this article, where feasible, the embodiments can be combined with each other, features between different embodiments can be replaced or borrowed, and one or more features can be omitted in one embodiment.

[0045] The present application embodiment provides a video time retrieval method for multimodal fusion of text and video. Figure 1 A flowchart of a video time retrieval method for multimodal fusion of text and video is shown in FIG. Figure 1 , methods include:

[0046] Step S1, using a text encoder to perform feature extraction on text query information to obtain text features.

[0047] Here, the text encoder may adopt CLIP (Contrastive Language-Image Pre-Training), or UTF-8, UTF-16, etc., which are not specifically limited here. The text encoder here is a pre-trained model.

[0048] Step S2, extracting features from the original video using a video feature extraction backbone network to obtain video features; and converting the video features into multi-scale video features using a visual encoder.

[0049] Here, the video feature extraction backbone network can use the video recognition model I3D, or RNN (recursive neural network), etc., which is not specifically limited here. The video feature extraction backbone network here is a pre-trained model. The visual encoder can preferably use ConvNext or ContextDet. ConvNext is an efficient encoder for temporal action detection, thereby generating a series of multi-scale feature representations.

[0050] Step S3, globally fusing the text features and the multi-scale video features to obtain a global offset; locally fusing the text features and the multi-scale video features to obtain a local offset; and accumulating the global offset and the local offset to obtain a total offset.

[0051] Step S4, inputting the text features and the multi-scale video features into the query-dependent convolutional classifier to obtain the semantic score corresponding to each moment in the original video, where the semantic score is between 0 and 1; and selecting the moment corresponding to the semantic score greater than the set threshold as the candidate moment.

[0052] Here, the semantic score is used to determine whether each moment is a foreground moment or a background moment, and then the distance is predicted only on the foreground moment, that is, the moment corresponding to the semantic score greater than the set threshold is the foreground moment, that is, the candidate moment. Here, the threshold is selected according to the actual situation, for example, it can be set to 0.7.

[0053] Step S5: determine the time segment corresponding to each candidate moment according to the total offset.

[0054] Here, the total offset It is an array consisting of 2 elements. The first element represents the offset of the start time from the candidate moment, and the second element represents the offset of the end time from the candidate moment.

[0055] Specifically, calculate the start time and end time of each candidate moment:

[0056]

[0057] t

[0058] in, is the start time of the time segment to which the candidate time t belongs, is the end time of the time segment to which the candidate time t belongs, is the offset of the starting time from the candidate time in the total offset, is the offset of the end time from the candidate time in the total offset;

[0059] The time segment corresponding to each candidate moment is [ ].

[0060] Step S6, all time segments are processed by non-maximum suppression method to obtain the final time segment, which is the time segment corresponding to the determined text query information in the original video. Here, the non-maximum suppression method is used to remove highly overlapping time segments.

[0061] In this embodiment, the problem that the prior art ignores the fusion of global features and local features is solved. It can capture the global semantic consistency between text and video, and can also fuse them at a finer level, effectively improving the accuracy of video moment retrieval.

[0062] In one embodiment, Figure 2 The schematic diagram of global fusion and local fusion is shown in Figure 2 In step S3, the text features and the multi-scale video features are globally fused to obtain a global offset, including:

[0063] First, the text features are processed by the multi-layer perceptron and tensor transformation to obtain the convolution kernel weights. ;

[0064] Then, the text features G Splice with the multi-scale video feature Z in the channel dimension to obtain the spliced ​​features ; Here, we can first analyze the text features G Perform shape adjustment and filling to obtain the filled text features, and then splice them with the multi-scale video features Z in the channel dimension;

[0065] Then, the concatenated features Perform convolution operation to obtain mixed features;

[0066] Then, Fourier convolution is performed on the mixed features to obtain the features after Fourier convolution;

[0067] Then, dynamic mask generation is performed on the features after Fourier convolution to obtain a mask tensor; the mask tensor is multiplied by the features after Fourier convolution to obtain a multiplication result; the multiplication result is inverse Fourier transformed to obtain the features after the inverse operation; here, dynamic mask generation can be implemented using a 1×1 convolution (linear) layer, a ReLU activation function, and another linear layer.

[0068] Then, the inversely operated features are convolved with the convolution kernel weights to obtain the global offset.

[0069] In one embodiment, see Figure 2 In step S3, the text features and the multi-scale video features are locally fused to obtain a local offset, including:

[0070] Text Features G After multi-layer perceptron processing and tensor transformation operations, the convolution kernel weights are obtained. ;

[0071] Multi-scale video features Z and convolution kernel weights Perform a convolution operation to obtain the local offset.

[0072] Based on the same inventive concept as the video moment retrieval method of text-video multimodal fusion, this embodiment also provides a corresponding video moment retrieval device of text-video multimodal fusion, including:

[0073] A text feature extraction module is used to extract features from text query information using a text encoder to obtain text features;

[0074] The video feature extraction module is used to extract features from the original video using a video feature extraction backbone network to obtain video features; and to convert the video features into multi-scale video features using a visual encoder;

[0075] A fusion module is used to globally fuse text features and multi-scale video features to obtain a global offset; locally fuse text features and multi-scale video features to obtain a local offset; and accumulate the global offset and the local offset to obtain a total offset;

[0076] The candidate moment determination module is used to input text features and multi-scale video features into the query-dependent convolution classifier to obtain the semantic score corresponding to each moment in the original video; the moment corresponding to the semantic score greater than the set threshold is selected as the candidate moment;

[0077] A time segment determination module, used to determine the time segment corresponding to each candidate moment according to the total offset;

[0078] The final result determination module is used to process all time segments using a non-maximum suppression method to obtain a final time segment, which is the time segment corresponding to the determined text query information in the original video.

[0079] The video moment retrieval device for multimodal fusion of text and video in this embodiment has the same inventive concept as the video moment retrieval method for multimodal fusion of text and video mentioned above. Therefore, the specific implementation method of the device can be seen in the implementation example part of the video moment retrieval method for multimodal fusion of text and video mentioned above, and its technical effect corresponds to the technical effect of the above method, which will not be repeated here.

[0080] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the video moment retrieval method for multimodal fusion of text and video is implemented.

[0081] An embodiment of the present application provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the video moment retrieval method for multimodal fusion of text and video is implemented.

[0082] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A video moment retrieval method for multimodal fusion of text and video, characterized in that: include: A text encoder is used to extract features from text query information to obtain text features; The original video is subjected to feature extraction using a video feature extraction backbone network to obtain video features; and using a visual encoder to convert the video features into multi-scale video features; The text feature and the multi-scale video feature are globally fused to obtain a global offset; the text feature and the multi-scale video feature are locally fused to obtain a local offset; and the global offset and the local offset are accumulated to obtain a total offset; Inputting the text features and the multi-scale video features into a query-dependent convolutional classifier to obtain a semantic score corresponding to each moment in the original video; selecting a moment corresponding to a semantic score greater than a set threshold as a candidate moment; Determine a time segment corresponding to each candidate moment according to the total offset; All time segments are processed by a non-maximum suppression method to obtain a final time segment, which is the time segment corresponding to the determined text query information in the original video; The text feature and the multi-scale video feature are globally fused to obtain a global offset, including: The text features are processed by a multi-layer perceptron and subjected to tensor transformation operations to obtain convolution kernel weights; Splicing the text feature with the multi-scale video feature in a channel dimension to obtain a spliced ​​feature; Performing a convolution operation on the concatenated features to obtain mixed features; Performing Fourier convolution on the mixed features to obtain features after Fourier convolution; Dynamically generate a mask on the feature after the Fourier convolution to obtain a mask tensor; multiply the mask tensor and the feature after the Fourier convolution to obtain a multiplication result; perform an inverse Fourier transform on the multiplication result to obtain a feature after the inverse operation; The feature after the inverse operation is convolved with the convolution kernel weight to obtain a global offset; The text feature and the multi-scale video feature are locally fused to obtain a local offset, including: The text features are processed by a multi-layer perceptron and subjected to tensor transformation operations to obtain convolution kernel weights; The multi-scale video feature is convolved with the convolution kernel weight to obtain a local offset.

2. The method according to claim 1, characterized in that in, Determining the time segment corresponding to each candidate moment according to the total offset includes: Calculate the start and end time of each candidate moment: t in, is the start time of the time segment to which the candidate time t belongs, is the end time of the time segment to which the candidate time t belongs, is the offset of the starting time from the candidate time in the total offset, is the offset of the end time from the candidate time in the total offset; The time segment corresponding to each candidate moment is [ ].

3. A video time retrieval device for text-video multimodal fusion, characterized in that: include: A text feature extraction module is used to extract features from text query information using a text encoder to obtain text features; The video feature extraction module is used to extract features from the original video using a video feature extraction backbone network to obtain video features; and to convert the video features into multi-scale video features using a visual encoder; A fusion module is used to globally fuse the text feature and the multi-scale video feature to obtain a global offset; locally fuse the text feature and the multi-scale video feature to obtain a local offset; and accumulate the global offset and the local offset to obtain a total offset; A candidate moment determination module is used to input the text features and the multi-scale video features into a query-dependent convolution classifier to obtain a semantic score corresponding to each moment in the original video; and select a moment corresponding to a semantic score greater than a set threshold as a candidate moment; A time segment determination module, used to determine the time segment corresponding to each candidate moment according to the total offset; A final result determination module is used to process all time segments using a non-maximum suppression method to obtain a final time segment, that is, a time segment corresponding to the determined text query information in the original video; The text feature and the multi-scale video feature are globally fused to obtain a global offset, including: The text features are processed by a multi-layer perceptron and subjected to tensor transformation operations to obtain convolution kernel weights; Splicing the text feature with the multi-scale video feature in a channel dimension to obtain a spliced ​​feature; Performing a convolution operation on the concatenated features to obtain mixed features; Performing Fourier convolution on the mixed features to obtain features after Fourier convolution; Dynamically generate a mask on the feature after the Fourier convolution to obtain a mask tensor; multiply the mask tensor and the feature after the Fourier convolution to obtain a multiplication result; perform an inverse Fourier transform on the multiplication result to obtain a feature after the inverse operation; The feature after the inverse operation is convolved with the convolution kernel weight to obtain a global offset; The text feature and the multi-scale video feature are locally fused to obtain a local offset, including: The text features are processed by a multi-layer perceptron and subjected to tensor transformation operations to obtain convolution kernel weights; The multi-scale video feature is convolved with the convolution kernel weight to obtain a local offset.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the video moment retrieval method for multimodal fusion of text and video according to claim 1 or 2 is implemented.

5. A computer program product, characterized in that It includes a computer program / instruction, which, when executed by a processor, implements the video moment retrieval method for multimodal fusion of text and video as described in claim 1 or 2.

Images