Video time retrieval method based on multi-dimensional video information interaction

Abstract

The application discloses a video time retrieval method based on multi-dimensional video information interaction, and mainly solves the problems that the existing technology is difficult to extract the diversity positioning content in the video, leading to difficult cross-modal alignment and inaccurate retrieval. The implementation scheme is as follows: a training set is constructed by using a public video time retrieval data set; a video time retrieval network including a text feature extraction module, a video feature extraction module, a multi-dimensional two-dimensional time sequence feature map generation module, a scene segmentation candidate time point representation enhancement module, a cross-modal interaction and time retrieval module is constructed; the video time retrieval network is trained by using the training set through the gradient descent method; and the retrieval of the video time corresponding to the text is completed by forward calculation of the trained network. Through multi-dimensional video information interaction, the application constructs a multi-dimensional two-dimensional time sequence feature map and enhances the feature representation, promotes cross-modal semantic matching, improves the retrieval performance of the model, and can be used for intelligent education and safety monitoring.

Description

Technical Field

[0001] This invention belongs to the field of electronic digital data processing technology, and further relates to a video time-based retrieval method that can be used for smart education and security monitoring. Background Technology

[0002] With the rapid development of the internet and the widespread adoption of smart devices, the internet ecosystem, including short videos and live streaming, has led to a dramatic increase in the number of videos, making video a crucial carrier of information. In this context, video understanding tasks have become particularly important, and efficient analysis of massive amounts of video data to achieve rapid and accurate location and retrieval of video content is of great significance. Video moment retrieval aims to retrieve the most semantically relevant video moments in unedited videos based on a given natural language text query. Due to the complex semantic information contained in videos across time and space and the need for cross-modal semantic alignment between video and text, this task has significant research value. Video moment retrieval typically follows a "candidate-ranking" process. First, candidate moments are generated by constructing a two-dimensional temporal feature map. Then, video and text features are interactively fused to generate relevance scores, and the most relevant video moments are retrieved based on these scores. Since user text queries are semantically complex and diverse, with different queries corresponding to varying degrees of complexity in the video's location content, how to construct a two-dimensional temporal feature map from multiple dimensions to fully capture the location information in the video is a pressing problem to be solved in the field of video moment retrieval. The latent scene information in the video contains prior knowledge for segmenting video segments. The video time segments pre-segmented using the scene directly affect the time retrieval results. Therefore, it is necessary to use the scene information in the video to segment video segments and enhance the feature representation of video candidate times to further improve the performance of the method.

[0003] Patent application number 202011575231.7 discloses a "cross-modal video moment retrieval method based on cross-modal dynamic convolutional networks." This method first uses a hierarchical video feature extraction module and a text feature extraction module to extract features from video and text modalities respectively. Then, a cross-modal fusion mechanism is used to fuse the features from these two modalities. Finally, a moment localization module based on a cross-modal convolutional neural network is used to complete the cross-modal video moment retrieval. This method has two shortcomings: First, because it only constructs a two-dimensional temporal feature map from the extracted video segment features using convolutional networks or average pooling methods, it does not extract the diverse localization content in the video, resulting in relatively poor retrieval performance. Second, after constructing the two-dimensional temporal feature map, it directly uses Hadamard product to interactively fuse the cross-modal features, ignoring the use of scene information in the video for segmentation, making it difficult to align the cross-modal video and text features semantically.

[0004] In their paper "Negative sample matters: Arenaissance of metric learning for temporal grounding" (AAAI Conference on Artificial Intelligence, 2022), Zhenzhi Wang et al. disclosed a video moment retrieval method based on cross-modal contrastive learning. This method extracts features from the video and constructs a two-dimensional temporal feature map using max pooling to obtain candidate moment features. It extracts features from the text using a DistilBERT pre-trained model, generates a matching score by calculating the cosine similarity between the two-dimensional temporal feature maps of the text and video, ranks the matching scores to complete moment retrieval, and uses contrastive learning to increase the discriminative power of text and video features by mining negative samples from both the text and video. This method limits retrieval performance in terms of video moment feature representation because it only uses a relatively simple max pooling method to construct a two-dimensional temporal feature map when constructing candidate moments, without extracting simple concentrated or complex scattered retrieval and positioning content from the video based on diverse text queries. At the same time, since it directly calculates cross-modal similarity to generate matching scores after constructing the two-dimensional temporal feature map, without utilizing scene information in the video to enhance the feature representation of candidate moment segments, the model is not accurate enough during retrieval.

[0005] In summary, for video moment retrieval tasks, existing methods fail to extract diverse locational content from videos, whether simple and concentrated or complex and scattered, thus limiting model performance. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the prior art by proposing a video moment retrieval method based on multi-dimensional video information interaction, so as to fully extract the concentrated or dispersed diverse positioning content in the video, enhance the feature representation of video candidate moments, and improve the model retrieval performance.

[0007] The approach to achieving the objective of this invention is to generate candidate moments by constructing different two-dimensional temporal feature maps from video segments using both pooling and sparse sampling methods. This multi-dimensional two-dimensional temporal feature map is then used to fully extract diverse locational content from the video, whether simple and concentrated or complex and dispersed. This solves the problem of existing technologies struggling to accurately extract locational content from videos that match text queries due to their limited processing of video information. Furthermore, video segments are segmented by calculating video scene similarity. Two-dimensional feature maps are then constructed from the determined scene-segmented video candidate segments and weighted onto the multi-dimensional two-dimensional temporal feature map. This semantically enhances the feature representation of important video moment segments, promoting subsequent cross-modal alignment in the semantic space. This addresses the problem of existing technologies where simple semantic extraction of video candidate segments leads to difficulties in cross-modal semantic alignment and insufficient retrieval accuracy, thereby improving the model's retrieval performance.

[0008] Based on the above ideas, the technical solution of the present invention includes the following steps:

[0009] A video moment retrieval method based on multi-dimensional video information interaction, characterized by the following steps:

[0010] (1) Construct a video time-retrieval training set. Each sample in the training set contains three data items: video, text query, and real time label.

[0011] (2) Constructing a video time-of-view retrieval network:

[0012] 2a) Generate two multi-dimensional two-dimensional temporal feature maps M1 and M2 respectively through sparse sampling and pooling sampling;

[0013] 2b) Establish feature map M by segmenting video clips through scene segmentation s Furthermore, candidate time-time feature representation enhancement was performed on two multi-dimensional two-dimensional time-series feature maps, M1 and M2, to obtain the enhanced two multi-dimensional two-dimensional time-series feature maps. and

[0014] 2c) Select the existing text query feature extraction module and video feature extraction module, connect them in parallel, and then combine them with the enhanced two multi-dimensional two-dimensional temporal feature maps. The existing cross-modal interaction and time-of-flight retrieval modules are cascaded in sequence to form a video time-of-flight retrieval network;

[0015] (3) The video time-retrieval network is trained using the gradient descent method on the training set to obtain the trained video time-retrieval network;

[0016] (4) Input the text to be queried and its corresponding video into the trained video moment retrieval network, and complete the retrieval of the video moment segment corresponding to the semantics of the text to be queried through forward computation.

[0017] Compared with existing technologies, the video time-of-view retrieval network constructed by this invention has the following advantages:

[0018] First, by incorporating a multi-dimensional two-dimensional temporal feature map module into the video moment retrieval network, this invention can construct two-dimensional temporal feature maps from multiple perspectives. This facilitates the use of sparse sampling and pooling sampling to process video features and generate two different two-dimensional temporal feature maps. This enables the extraction of multi-dimensional localized content with varying complexity corresponding to different text query semantics in the video. This overcomes the shortcomings of existing technologies that rely on a single video moment processing method and struggle to extract complex localized content from videos. As a result, this invention can fully extract the content to be retrieved from the video from multiple dimensions, improving the accuracy of the retrieval.

[0019] Secondly, this invention incorporates a scene segmentation candidate moment representation enhancement module into the video moment retrieval network. This module can segment video segments using scene similarity in the video, determine each boundary based on changes in scene similarity, and then segment the video moment segments. The segmented video candidate segments are pooled and aggregated to generate a two-dimensional feature map, which enhances the candidate moment feature representation in the multi-dimensional two-dimensional temporal feature map. This overcomes the difficulty of cross-modal interactive semantic matching that existing technologies face by directly using simple temporal feature maps. As a result, this invention can promote cross-modal semantic matching and improve the retrieval performance of the model. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the implementation of the present invention.

[0021] Figure 2 This is a schematic diagram of the network structure in this invention. Detailed Implementation

[0022] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0023] Reference Figure 1 The implementation steps of the embodiments of the present invention will be further described below.

[0024] Step 1: Construct a training set for video moment retrieval.

[0025] Using data from existing publicly available video moment retrieval datasets, at least 5,000 cross-modal data samples were selected. Each sample contained three items: video, text query, and real moment label. The video content included people and actions. The query content was a natural language description of a single action or multiple consecutive actions of a person in the video corresponding to the query. The real moment label content was the start and end times of the content described in the query in the video.

[0026] These data samples are combined to form a training set for video moment retrieval.

[0027] Step 2: Construct a video moment retrieval network.

[0028] 2.1) Select existing video feature extraction modules and text feature extraction modules:

[0029] The video feature extraction module is used to divide the input raw video V into several video segments, each segment containing the same video frames, and to extract segment-level visual features using pre-trained convolutional neural network models C3D and VGG. Then, a temporal one-dimensional convolutional layer and an average pooling layer are used to map the video features to the required length N and dimension d, resulting in segment-level video features.

[0030] The text feature extraction module is used to extract features from the input query text Q using a DistilBERT pre-trained model. Specifically, for each input text, a word segmenter is used to generate word segments, and a class-embedded segmentation "[CLS]" is added at the beginning. The word segments are then input into the DistilBERT pre-trained model to obtain a text feature sequence. The embeddings of the entire text are obtained using global average pooling aggregation. Where L is the number of words in the text, and c is the feature dimension;

[0031] 2.2) Constructing a multi-dimensional two-dimensional temporal feature map generation module:

[0032] This invention considers that text queries in different videos correspond to simple, concentrated, or complex, dispersed time-based retrieval content. It requires extracting multi-dimensional localization content from the videos through sparse sampling and pooling sampling to generate multi-dimensional two-dimensional temporal feature maps. A multi-dimensional two-dimensional temporal feature map generation module is constructed using two different two-dimensional temporal feature maps.

[0033] 2.2.1) Generating a two-dimensional temporal feature map through sparse sampling:

[0034] To capture the detailed localization content corresponding to the complex and diverse continuous action scenes within a video clip, sparse sampling is required. This involves sampling the start frame, end frame, and intermediate frames of the clip. Furthermore, to prevent the loss of localization content for continuous action events, sampling is also necessary for the intermediate frames of the first and second halves of the video. This ensures sufficient capture of multiple continuous action elements and scene transition information, resulting in candidate clip features.

[0035]

[0036] Using the obtained candidate time features Generate a two-dimensional temporal feature map M1:

[0037]

[0038] in, and Let represent the start and end frames of the nth video candidate moment, respectively. This represents the nth candidate time feature in the two-dimensional temporal feature map M1, where N represents the number of video segments.

[0039] 2.2.2) Generate a two-dimensional temporal feature map through pooling sampling:

[0040] When the video location content is relatively simple and concentrated, max pooling is used to extract key location content from the video. That is, for a given video time segment, max pooling is performed on it to obtain candidate time segment features.

[0041]

[0042] Using the obtained candidate time features Generate a two-dimensional temporal feature map M2:

[0043]

[0044] in, and Let represent the start and end frames of the nth video candidate moment, respectively. This represents the feature of the nth candidate time step in the two-dimensional time series feature map M2;

[0045] This multi-dimensional two-dimensional temporal feature map can traverse and enumerate potential video moments. Its vertical and horizontal axes represent the start and end time indices of the candidate moments, respectively. Each time block (t) i ,t j () represents the candidate time points from time index i to j in the video sequence. Since the start index exceeds the corresponding end index, the lower triangular part of the two-dimensional temporal feature map is invalid.

[0046] 2.3) Construct a scene segmentation candidate moment representation enhancement module:

[0047] Video segments are segmented using latent scene information in the video. The video segments are segmented by calculating the similarity between video frames, and the candidate time representation is enhanced on the two-dimensional temporal feature map obtained by sampling in step 2.2).

[0048] For a given video, calculate the cosine similarity of each video frame. Frames with significant changes in similarity values ​​are identified as scene event transition points. Based on these transition points, the video is segmented into m scene event video segments. Average pooling is then performed on each video segment to generate relatively definite video candidate moment features. A two-dimensional feature map M for scene segmentation is generated by pooling the candidate time features of m video clips. s :

[0049]

[0050]

[0051] in, This represents the aggregated candidate moment features of the nth scene segment in the two-dimensional feature map of scene segmentation, obtained by average pooling. Only m video segments are valid in this two-dimensional feature map, and the rest are filled with 0.

[0052] Segment the scene into two-dimensional feature maps M s The two-dimensional temporal feature maps M1 and M2 obtained through sparse sampling and pooling sampling in step 2.2) are weighted respectively to enhance the feature representation of candidate time points, thereby promoting better matching of subsequent cross-modal video text, resulting in enhanced multi-dimensional two-dimensional temporal feature maps. and

[0053]

[0054]

[0055] 2.4) Select the existing cross-modal interaction and time retrieval module:

[0056] The aforementioned cross-modal interaction and time-series retrieval module utilizes the enhanced multi-dimensional two-dimensional temporal feature map. and Respectively with the text To perform cross-modal interaction, the matching score map is obtained by calculating cosine similarity to obtain the matching degree of each candidate time step in the two-dimensional temporal feature maps of text query, sparse sampling, and pooling sampling. and

[0057]

[0058] By analyzing two different matching score graphs and Different weights α and β are assigned to each other, and the final matching score map is obtained by adaptive weighting.

[0059]

[0060] Matching score graph After sorting from high to low, the candidate time with the highest score is selected as the final target time for retrieval and positioning;

[0061] 2.5) After paralleling the existing text query feature extraction module and video feature extraction module, it is then cascaded with the multi-dimensional two-dimensional temporal feature map generation module, the scene segmentation candidate time-of-day representation enhancement module, and the existing cross-modal interaction and time-of-day retrieval module to form a video time-of-day retrieval network, such as... Figure 2 As shown.

[0062] Step 3: Train the video time-of-view retrieval network using the training set.

[0063] 3.1) The training set is input into the video time-of-view retrieval network. Forward propagation is performed through the video feature extraction module and the query feature extraction module to extract video features and text query features, respectively. Forward propagation is performed through the multi-dimensional two-dimensional temporal feature map generation module to obtain two-dimensional temporal feature maps formed by different sampling methods. Forward propagation is performed through the scene segmentation candidate time-of-view representation enhancement module to segment segments using scene similarity and weight them to obtain a multi-dimensional two-dimensional temporal feature map with enhanced representation. Forward propagation is performed through the cross-modal interaction and time-of-view retrieval module to calculate the matching score for each candidate segment.

[0064] 3.2) Set the loss function value for the video time-lapse network.

[0065] 3.2.1) Calculate the prediction score of candidate segments using the binary cross-entropy loss function. The true correlation score y i Loss values ​​between

[0066]

[0067] 3.2.2) Calculate the loss value corresponding to different positive and negative samples using contrastive learning loss.

[0068]

[0069] Where C represents the total number of candidate moments, p(v|q) represents the probability that a video v matches a given text query q, and p(q|v) represents the probability that a text query q matches a given video v.

[0070] 3.2.3) The binary cross-entropy loss value Comparative learning loss value The sum is used to obtain the final loss function value.

[0071] 3.3) Based on the calculated loss function value The parameters of the video time-of-view retrieval network are iteratively updated using gradient descent until the loss function converges, resulting in a well-trained video time-of-view retrieval network.

[0072] Step 4: Use the trained video time-retrieval network to retrieve video times.

[0073] The video and text queries to be retrieved are input into the trained video temporal retrieval network, and the features of the video and text queries are extracted respectively to generate a multi-dimensional two-dimensional temporal feature map of the video features.

[0074] The multi-dimensional two-dimensional temporal feature map is interactively fused with the text query to form a matching score map. The candidate time segments are sorted according to the matching score map, and the candidate time segment with the highest relevance score is selected as the final video time retrieval result.

[0075] The above description is merely a specific example of the present invention and does not constitute any limitation on the present invention. Obviously, those skilled in the art, after understanding the content and principles of the present invention, may make various modifications and changes in form and details without departing from the principles and structure of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

[0076] It should be noted that the step numbers in the specification and claims of this invention are only for the purpose of clearly describing the embodiments of this invention and facilitating understanding, and their order is not limited.

Claims

1. A video moment retrieval method based on multi-dimensional video information interaction, characterized in that, Includes the following steps: (1) Construct a video time-of-day retrieval training set. Each sample in the training set contains three data items: video, text query, and real time-of-day label. (2) Constructing a video time-of-view retrieval network: 2a) Generate two multi-dimensional two-dimensional temporal feature maps by sparse sampling and pooling sampling, respectively. and Two-dimensional temporal feature maps are generated through sparse sampling. The implementation is as follows: 2a1) Sample the start frame, end frame, and intermediate frames of a video moment segment. Simultaneously sample the intermediate frames of the first and second halves of the video. This sparse sampling method samples multiple consecutive action contents and scene transition information in the video to generate candidate moment features. : ； 2a2) Based on the characteristics of candidate times Generate two-dimensional temporal feature maps corresponding to sparse sampling. : ； in, Indicates the number of video clips. and These represent the first two-dimensional time series feature graphs. The start and end frames of each video candidate moment. This indicates an addition operation. Represents the first in the two-dimensional time series feature graph Features of each candidate time point, two-dimensional time series feature map Iterate through all potential video moments, with the vertical and horizontal axes representing the start and end time indices of the candidate moments, respectively. Each time block... Indicates the time index in the video sequence arrive Candidate moments; Two-dimensional temporal feature maps are generated through pooling sampling. The implementation is as follows: 2a3) Perform max pooling on a video time segment to generate candidate time features. : ； 2a4) Based on the characteristics of candidate times Generate two-dimensional temporal feature maps corresponding to pooling sampling. : ； in, and These represent the first two-dimensional time series feature graphs. The start and end frames of each video candidate moment. This indicates a maximum pooling operation. Represents the first in the two-dimensional time series feature graph Features of each candidate time point; 2b) Establishing feature maps from video segments segmented by scene And for two multi-dimensional two-dimensional time series feature maps and Enhancement of candidate time-series feature representations is performed to obtain two enhanced multi-dimensional two-dimensional temporal feature maps. and ; 2c) Select the existing text query feature extraction module and video feature extraction module, connect them in parallel, and then combine them with the enhanced two multi-dimensional two-dimensional temporal feature maps. , The existing cross-modal interaction and time-of-flight retrieval modules are cascaded in sequence to form a video time-of-flight retrieval network; (3) The video time-retrieval network is trained using the gradient descent method on the training set to obtain the trained video time-retrieval network; (4) Input the text to be queried and its corresponding video into the trained video moment retrieval network, and complete the retrieval of the video moment segment corresponding to the semantics of the text to be queried through forward computation.

2. The method according to claim 1, characterized in that, In step (1), the video time-of-view retrieval training set is constructed as follows: 1a) Using data from the existing publicly available video moment retrieval dataset, select at least 5000 data samples. Each sample contains three data items that correspond one-to-one: video, text query, and real moment label. The video content contains people and shows their related actions. The text query content is a natural language description of one or more consecutive actions of the person in the video corresponding to the text query. The real moment label content is the start and end time of the content described in the text query in the video corresponding to the text query. 1b) Combine these data samples together to form a training set for video time-of-view retrieval.

3. The method according to claim 1, characterized in that, Step 2b) describes enhancing the candidate moment feature representation by segmenting video segments through scene segmentation, which is implemented as follows: 2b1) For a video, calculate the cosine similarity between each video frame, identify frames where the similarity value changes significantly as scene event transition points, and segment the video based on these scene event transition points. For each scene event video clip, perform average pooling on each clip to determine a specific video candidate moment. After pooling, the corresponding candidate time-step features of each video segment are used to generate a 2D feature map for scene segmentation. : ； in, and These represent the first two-dimensional feature maps. The start and end frames of a scene segment This indicates the average pooling operation. The 2D feature map representing scene segmentation is the first... The aggregated candidate moment features obtained by average pooling of scene segments; 2b2) Segment the scene into two-dimensional feature maps The two-dimensional temporal feature maps obtained by sparse sampling and pooling sampling in step 2a) are weighted respectively. and The enhanced candidate time-time feature representation is obtained as follows: ； in, Represents a two-dimensional feature map for scene segmentation. and This represents two multi-dimensional two-dimensional temporal feature maps after enhancement of the candidate feature representations for scene segmentation.

4. The method according to claim 1, characterized in that, In step 2c), the two existing extraction modules extract the following different features respectively: The video feature extraction module is used for the input raw video. First, the video is divided into several segments, each containing the same video frames. Then, existing pre-trained convolutional neural network models C3D and VGG are used to extract segment-level visual features. Finally, a temporal one-dimensional convolutional layer and an average pooling layer are used to map the video features to the required length. and dimensions To obtain video features at the segment level ; The text feature extraction module is used to extract the input query text. First, a word segmenter is used to generate word segments and a class-embedded segmentation "[CLS]" is added at the beginning; then, the segmented words are input into an existing DistilBERT pre-trained model to obtain text feature sequences. Then, the global average pooling aggregation method is used to obtain the embedding of the entire text. ,in It is the number of words in the text. It is the feature dimension.

5. The method according to claim 1, characterized in that, The cross-modal interaction and time retrieval module in step 2c) is used to obtain the cross-modal matching score map to achieve the retrieval of the target time. Its implementation is as follows: 2c1) Two multi-dimensional two-dimensional time series feature maps and Respectively with the text To perform cross-modal interaction, cosine similarity is calculated to obtain matching score maps between the text query and candidate time points in two two-dimensional temporal feature maps. and : ； 2c2) For two different matching score graphs and Assign different weights to the two values ​​respectively. and The final matching score graph is obtained through adaptive weighting. : ； 2c3) Matching score graph The candidate timestamps with the highest scores are sorted and selected as the final target timestamps for retrieval.

6. The method according to claim 1, characterized in that, The step (3) described above, which involves training the video time-of-view retrieval network using the gradient descent method on the training set, is implemented as follows: 3a) The training set is input into the video time-of-view retrieval network. Forward propagation is performed through the video feature extraction module and the query feature extraction module to extract video features and text query features, respectively. Forward propagation is performed through the multi-dimensional two-dimensional temporal feature map generation module to obtain two-dimensional temporal feature maps formed by different sampling methods. Forward propagation is performed through the scene segmentation candidate time-of-view representation enhancement module to segment segments using scene similarity and weight them to obtain a multi-dimensional two-dimensional temporal feature map with enhanced representation. Forward propagation is performed through the cross-modal interaction and time-of-view retrieval module to calculate the matching score of each candidate time-of-view segment. ; 3b) Use the binary cross-entropy loss and contrast loss as the loss functions of the video time-to-time retrieval network; use the binary cross-entropy loss function to calculate the prediction scores of candidate segments. Relevance score Loss values ​​between The contrastive learning loss is used to calculate the loss value corresponding to different positive and negative samples. The final loss function value is obtained by adding the binary cross-entropy loss value to the contrastive learning loss value. ; 3c) Based on the calculated loss function value The parameters of the video time-of-view retrieval network are iteratively updated using the gradient descent method until the loss function value converges, thus obtaining the trained video time-of-view retrieval network.

7. The method according to claim 6, characterized in that, The binary cross-entropy loss value in 3b) and contrastive learning loss value The calculation formulas are as follows: ； ； in, This represents the total number of candidate moments. Represents a video In a given text query The probability of matching it at that time. Represents a text query In the given video The probability of matching it at that time.

8. The method according to claim 1, characterized in that, In step (4), the retrieval of video time segments corresponding to the semantics of the text to be queried is completed through forward computation, as follows: 4a) Input the video and text queries into the video time-of-view retrieval network and extract video features and text query features respectively; 4b) Generate a multi-dimensional two-dimensional temporal feature map of video features through sparse sampling and pooling sampling; 4c) Interact and fuse the multi-dimensional two-dimensional time-series feature map with the text query features to form a matching score map; 4d) Sort the scores of each candidate moment segment in the matching score graph from high to low, and select the candidate moment segment with the highest score as the final video moment retrieval result.

Images