Video moment retrieval method based on feature representation enhancement

Abstract

The application discloses a video time retrieval method based on feature representation enhancement, and steps are as follows: generating a training set; constructing a video time retrieval network; training the video time retrieval network; and retrieving a video time. The application constructs a video time retrieval network based on feature representation enhancement, adds a query timing correction operation to a query feature extraction module, arranges all subject-predicate-object triples describing different actions in a to-be-queried sentence according to time sequences of the corresponding actions, realizes cross-modal feature sequence order matching, and reduces cross-modal semantic alignment difficulty. The application constructs a double-connection graph convolution module, takes candidate segment features as graph nodes, connects the candidate segment with global candidate segments of the front and rear parts through graph convolution, realizes fusion of the candidate segment features and the video global features, and improves candidate segment feature representation quality.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of electric digital data processing, and further relates to a video time retrieval method based on feature representation enhancement in the technical field of multi-modal content understanding and data analysis. The present application can be used for locating a semantically relevant segment in a specified video for any natural language query sentence. BACKGROUND

[0002] The purpose of the video time retrieval task is to locate a semantically relevant segment in a specified video according to a given natural language query sentence. In the information age, with the popularity of various video platforms, the number of videos on the Internet has increased rapidly, and the length of videos has also increased, making content-based video understanding tasks increasingly important. As one of the core branches, the video time retrieval task has attracted much attention due to its great application value in information retrieval and other fields. The video time retrieval method generally includes three steps: feature extraction, cross-modal interaction, and time retrieval, and the core lies in the cross-modal semantic alignment problem. Video data has a natural time sequence, and query sentences can correspond to multiple sentence representations due to the complexity of natural language syntax structure, which increases the difficulty of query sentence understanding and semantic alignment with video data. Therefore, how to deeply understand the query sentence information according to the characteristics of the video time retrieval task, i.e., query sentence feature representation enhancement, is a problem to be solved in the field of video time retrieval. The fusion feature generated in the cross-modal interaction step, i.e., the candidate segment feature, directly affects the time retrieval result, so it is necessary to effectively enhance the candidate segment feature representation to further improve the performance bottleneck of the method.

[0003] The University of Electronic Science and Technology of China discloses a video time retrieval method based on a cross-modal dynamic convolutional network in its applied patent document "Cross-modal video time retrieval method based on cross-modal dynamic convolutional network" (application number: CN 202011575231.7, application publication number: CN 112650886 A). The method first extracts video and query features through a feature extraction module, then obtains two modal fusion features using Hadamard product, then enhances the fusion feature representation using a dynamic convolutional network, and finally obtains the final time retrieval result by one-dimensional convolution feature dimension reduction. The deficiency of this method is that the query sentence directly uses the existing LSTM model to extract sentence-level features, the feature representation granularity is large, and the retrieval effect is relatively poor; when enhancing the fusion feature representation, the two-dimensional convolution method is used, which has a large calculation cost, and only the features of adjacent candidate segments can be fused, and since the adjacent candidate segments have similar semantics and features, the candidate segment feature representation enhancement effect is general.

[0004] Zhang et al. in their published paper "Multi-stage aggregated transformer network for temporal language localization in videos" (IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021) discloses a multi-modal Transformer-based video moment retrieval method. The method first extracts video and query features through a feature extraction module, then concatenates the feature sequences of the two modalities and uniformly encodes the positions before inputting them into the Transformer, then constructs candidate segment features on the video feature sequence output by the Transformer, and finally generates the final moment retrieval result through fully connected feature dimension reduction. The deficiency of this method is that the original word sequence word feature is used in the query sentence feature representation, and the query sentence is not optimized according to the task characteristics, which limits the retrieval performance from the query sentence feature representation; the constructed candidate segment features are directly reduced in dimension through full connection to obtain the moment retrieval result, without feature representation enhancement at the candidate segment level, and there is still a large space for improvement in model performance.

[0005] In summary, for the video moment retrieval task, the existing methods fail to enhance the feature representation of the query sentence and the candidate segment according to the task characteristics, thereby limiting the model performance. In the present invention, a query timing correction operation is introduced to enhance the query sentence feature representation, and a double connection graph convolution module is added to enhance the candidate segment feature representation at a low computational cost, thereby improving the performance bottleneck of the retrieval method. SUMMARY

[0006] The present invention aims to address the deficiencies of the prior art by proposing a video moment retrieval method based on feature representation enhancement. It solves the problem of performance bottleneck caused by the lack of high-quality query sentence and candidate segment feature representation enhancement according to the characteristics of the video moment retrieval task.

[0007] The idea for achieving the object of the present application is that the present application adds a query timing correction operation to the query feature extraction module in the video time retrieval network, so that all subject-predicate-object triples describing different actions in the to-be-queried sentence are arranged in the time sequence of the corresponding action occurrence. Since the multiple segments describing different actions in the video are naturally arranged in the time sequence of the action occurrence, the present application enables the extracted to-be-queried sentence feature sequence to be sequentially matched and aligned with the video feature sequence, thereby solving the problem of difficulty in cross-modal semantic alignment of the feature extraction and representation of the query sentence with the video feature in the prior art. The present application adds a double connection graph convolution module to the video time retrieval network, takes the candidate segment feature as a graph node, connects each candidate segment with the global candidate segments before and after it through graph convolution, and realizes sufficient fusion of the candidate segment feature and the video global feature. The graph convolution module proposed in the present application has a small number of graph node connections and can sufficiently fuse local and global features, thereby solving the problems of high calculation cost and weak enhancement effect of the candidate segment feature representation in the prior art.

[0008] The technical scheme adopted by the present application includes the following steps:

[0009] Step 1: generating a training set corresponding one-to-one by a video, a query sentence and a real time label;

[0010] Step 2: constructing a query feature extraction module based on query timing correction;

[0011] Step 3: constructing a double connection graph convolution module based on fusion of global candidate segment features;

[0012] Step 4: constructing and training a video time retrieval network, which is composed of the query feature extraction module and the video feature extraction module connected in parallel, and then sequentially connected with the cross-modal interaction module, the double connection graph convolution module and the time retrieval head module;

[0013] Step 5: using the trained video time retrieval network to retrieve the video time.

[0014] Compared with the prior art, the present application has the following advantages:

[0015] Firstly, the present application adds a query timing correction operation to the query feature extraction module in the video time retrieval network, so that all subject-predicate-object triples of the subject position word, the predicate position word and the object position word in the query sentence are arranged in the time sequence of the predicate position word occurrence, thereby overcoming the difficulty in cross-modal semantic alignment of the feature extraction and representation of the query sentence with the video feature in the prior art, and enabling the present application to have the advantage of being able to extract the query sentence feature representation which is easy to be aligned with the video feature in cross-modal semantics.

[0016] Secondly, the application adds a double connection graph convolution module in the video time retrieval network, takes the candidate segment features as graph nodes, connects each candidate segment feature with the global candidate segment features of the two parts before and after it through graph convolution, realizes the full fusion of the candidate segment features and the video global features, and thus overcomes the insufficient representation of the candidate segment features in the prior art, and enhances the quality of the candidate segment features, thereby improving the retrieval method accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is the implementation flowchart of the application;

[0018] Figure 2 is the query timing correction operation schematic diagram of the application;

[0019] Figure 3 is the network structure schematic diagram of the application. DETAILED DESCRIPTION

[0020] The application will be further described in detail below in combination with the drawings and embodiments.

[0021] Reference Figure 1 The implementation steps of the embodiment of the application are further described.

[0022] Step 1, generating a training set corresponding one-to-one to a video, a query sentence and a real time label.

[0023] The training set refers to selecting at least 3000 groups of samples, each group of samples containing three items corresponding one-to-one to a video, a query sentence and a real time label, wherein the video content contains at least one person and shows any action, the query sentence content is the natural language description of any action or multiple continuous actions of a person in the video corresponding to the query sentence, and the real time label content is the start and end time of the content described by the query sentence in the video corresponding to the query sentence.

[0024] In the embodiment of the application, a video with a length of 33 seconds is collected, which is the process of a boxing match between two men, a query sentence corresponding to the video, the content of which is "The man then leaves the ring after he slams his opponent.", which contains two actions "leave" and "slam" between the two men, the action "slam" occurs at the 6s of the video, the action "leave" occurs at the 17s of the video, and a real time label, the content of which is the start and end time of the sequence of the two actions described by the query sentence in the corresponding video, i.e. 6s and 17s.

[0025] Step 2, constructing a query feature extraction module based on query timing correction.

[0026] The operation of the query feature extraction module specifically includes:

[0027] Firstly, input the sentence to be queried into the trained SceneGraphParser network, and output the triplets of all subject position words, predicate position words and object position words in the sentence to be queried.

[0028] Secondly, input the sentence to be queried into the trained Uret network, and output all predicate position words in the sentence to be queried and the arrangement order of the predicate position words according to the occurrence time.

[0029] Thirdly, concatenate all the words in the triplets in the arrangement order of the predicate position words, separate the words with spaces, and then form an enhanced sentence of the sentence to be queried, thereby completing the query timing correction operation on the sentence to be queried.

[0030] Fourthly, input each word in the enhanced sentence into the trained Glove network, and the Glove network outputs a feature for each word. Concatenate the features of all the words to obtain a feature sequence of the sentence to be queried.

[0031] The enhanced sentence obtained through the query timing correction operation, wherein the plurality of triplets describing different actions are arranged in the enhanced sentence according to the time sequence of the occurrence of the actions. Since the plurality of segments describing different actions in the video are naturally arranged in the time sequence of the occurrence of the actions, the construction of the enhanced sentence reduces the difficulty of cross-modal semantic alignment of the two kinds of modal feature sequences, and further improves the effect of the retrieval model.

[0032] Reference Figure 2 Further detailed description is made to the query timing correction operation of the sentence to be queried in the embodiment of the present application. Firstly, input the sentence to be queried “The man then leaves the ring after he slams his opponent.” into the trained SceneGraphParser network, and obtain two subject-predicate-object triplets describing different actions “(man, leave, ring)” and “(he, slam, opponent)”; then, input the sentence to be queried into the trained Uret network, and obtain the time sequence of the occurrence of the actions corresponding to the two predicate verbs “slam” is earlier than “leave”; finally, according to the output results of the two networks, re-integrate the contents in the triplets to obtain the enhanced sentence “man leave ring he slam opponent”.

[0033] Step 3, constructing a double connection graph convolution module based on fusion of global candidate segment features.

[0034] The double connection graph convolution module is realized by a graph convolution operation, and the graph nodes are the features of all candidate segments in the video corresponding to the query sentence, and the feature of each candidate segment is a graph node; each graph node is connected to other two graph nodes through an incoming edge, and the two graph nodes correspond to the predecessor global candidate segment and the successor global candidate segment of the candidate segment corresponding to the current graph node; the predecessor global candidate segment of the candidate segment with start and end time (s, e) in the video is the segment with start and end time (0, e), and the successor global candidate segment is the segment with start and end time (s, l), and l is the duration of the video in which the candidate segment is located; all graph nodes are updated by graph convolution operation according to the incoming edge; the updated graph node content is used as the feature representation enhanced candidate segment feature.

[0035] The graph convolution operation is obtained by the following formula:

[0036]

[0037] wherein, denotes the feature representation enhanced candidate segment feature corresponding to the video with start and end time i and j of the tth updated graph node, the candidate segment feature corresponding to the video with start and end time i and j of the t-1th updated graph node, and l represents the video duration, LN represents the layer normalization operation, and FC represents the full connection operation.

[0038] The graph convolution operation realizes the feature representation enhancement of the current candidate segment by fusing the feature of each candidate segment with the feature of its predecessor global candidate segment and the feature of its successor global candidate segment. The number of incoming edges connected to each graph node is only 2, so the operation has low calculation cost; the feature of the predecessor global candidate segment and the feature of the successor global candidate segment respectively contain all the preceding information and the following information of the current candidate segment, so the operation has sufficient information to enhance the feature representation of the current candidate segment to the greatest extent, thereby improving the performance of the retrieval method.

[0039] Step 4, constructing and training a video time retrieval network, which is composed of a query feature extraction module and a video feature extraction module connected in parallel, and then connected in series with a cross-modal interaction module, a double connection graph convolution module, and a time retrieval head module. The connection mode of the video time retrieval network is referred to Figure 3 .

[0040] The video feature extraction module is to uniformly divide the video corresponding to the query statement into a plurality of frame number 16 segments; input each frame number 16 segment into the trained C3D network respectively, and output the features of each segment; arrange the features in time sequence, divide them into T groups averagely, and pool each group of features into one feature in a mean manner to obtain a feature sequence of the video.

[0041] The cross-modal interaction module is to map the feature sequence of the query statement and the feature sequence of the corresponding video to the same dimension through different fully connected layers respectively; connect the feature sequence of the query statement and the feature sequence of the corresponding video, and perform position coding; input the position coded feature sequence of the query statement and the feature sequence of the corresponding video into a single layer cross-modal Transformer, wherein the parameter Q is the feature sequence of the video, K and V are the feature sequence of the query statement, and output a fusion feature sequence; connect any two features in the fusion feature sequence to obtain the features of all candidate segments.

[0042] The time retrieval head module is to input all the feature representation enhanced candidate segment features into a fully connected layer, and output the relevance scores of each candidate segment and the query statement.

[0043] The video time retrieval network training includes the following steps: input the training set into the video time retrieval network, forward propagate the video feature extraction module and the query feature extraction module, and extract the video features and the features of the query statement respectively; forward propagate the cross-modal interaction module to obtain the candidate segment features; forward propagate the double connection graph convolution module to obtain the feature representation enhanced candidate segment features; forward propagate the time retrieval head module to obtain the relevance scores of each candidate segment and the query statement; use a cross-entropy loss function to calculate the loss value between the predicted relevance scores of the candidate segments and the true relevance scores, wherein the true relevance scores are obtained from the intersection over union of the start time of each candidate segment in the corresponding video and the start and end times in the corresponding true time label; train the randomly generated initial network weights by using the gradient descent method, iteratively update the network parameters, and stop until the network loss function converges, thereby obtaining the trained video time retrieval network.

[0044] The cross-entropy loss function is as follows:

[0045]

[0046] Wherein, L represents the loss value between the predicted relevance scores of the candidate segments and the true relevance scores, D represents the sample number of the training set, G represents the number of candidate segments constructed in the prediction process for each sample, p represents the probability distribution, y represents the true relevance score, represents the predicted relevance score.

[0047] Step 5, retrieving the video time point by using the trained video time point retrieval network.

[0048] The retrieving video time point refers to inputting the video and the query sentence to be queried into the video time point retrieval network, outputting a plurality of candidate segments and a relevance score of the query sentence, sorting the candidate segments according to the relevance score, and selecting the candidate segment with the highest relevance score as the final retrieval result.

[0049] The query sentence to be queried and the corresponding video in the embodiment of the application are input into the trained video time point retrieval network, and 8256 candidate segments are generated in the retrieval process, wherein the candidate segment corresponding to the start time of the 6th second and the end time of the 17th second has the highest relevance score, and the start time of the 6th second and the end time of the 17th second corresponding to the candidate segment are taken as the time point retrieval result of the embodiment of the application.

Claims

1. A video moment retrieval method based on feature representation enhancement, characterized in that, The query feature extraction module and the biconnected graph convolution module are constructed separately; the steps of this retrieval method are as follows: Step 1: Generate a training set that corresponds one-to-one with video, query statement, and real-time label; Step 2: Construct a query feature extraction module based on query time-series correction. Specific operations include: The first step is to input the query statement into the trained SceneGraphParser network, which outputs triples of all words at the subject, predicate, and object positions in the query statement. The second step is to input the query statement into the trained Uret network, and output all the predicate position words in the query statement and the order of the predicate position words by occurrence time. The third step is to string together all the words in the triplet according to the order of the words in the predicate position, and then separate the words with spaces to form an enhanced statement of the query statement, thus completing the query timing correction operation of the query statement. The fourth step is to input each word in the enhanced sentence into the trained GloVe network. The GloVe network outputs a feature for each word, and the features of all words are concatenated to obtain the feature sequence of the query sentence. Step 3, a double connection graph convolution module based on fusion of global candidate segment features is constructed, the double connection graph convolution module is realized through a graph convolution operation, graph nodes are features of all candidate segments in a video corresponding to a query sentence, and features of each candidate segment are taken as a graph node; each graph node is connected to other two graph nodes through an incoming edge, the two graph nodes correspond to a predecessor global candidate segment and a successor global candidate segment of a candidate segment corresponding to the current graph node; in a candidate segment with start and end times of , a predecessor global candidate segment is a segment with start and end times of , and a successor global candidate segment is a segment with start and end times of , is the duration of a video where the candidate segment is located; all graph nodes are updated through a graph convolution operation according to an incoming edge; and updated graph node content is taken as enhanced candidate segment features. Step 4: Construct and train the video time-of-view retrieval network. This network consists of a query feature extraction module and a video feature extraction module connected in parallel, which are then cascaded with a cross-modal interaction module, a bi-connected graph convolution module, and a time-of-view retrieval head module. Step 5: Use the trained video time-retrieval network to retrieve video times.

2. The video time-of-view retrieval method based on feature representation enhancement according to claim 1, characterized in that, The training set mentioned in step 1 refers to selecting at least 3,000 sets of samples. Each set of samples contains three items that correspond one-to-one: video, query statement, and real-time label. The video content contains at least one person and displays any action. The query statement is a natural language description of one or more consecutive actions of any person in the video corresponding to the query statement. The real-time label is the start and end time of the content described by the query statement in the video corresponding to the query statement.

3. The video time-of-view retrieval method based on feature representation enhancement according to claim 1, characterized in that, The graph convolution operation is obtained through the following formula: ； in, Indicates the first The video start and end times corresponding to each updated graph node are: and The features represent enhanced candidate fragment features. No. The start and end times of the video corresponding to each graph node before the update are: and Candidate fragment features, Indicates video duration. Presentation layer normalization operation, This indicates a fully connected operation.

4. The video time-of-view retrieval method based on feature representation enhancement according to claim 1, characterized in that, The video feature extraction module described in step 4 involves uniformly dividing the video corresponding to the query statement into several segments of 16 frames each; inputting each of the 16-frame segments into a trained C3D network, and outputting the features of each segment; arranging the features in chronological order, and dividing them into average values. Each feature group is pooled into a single feature using the mean pooling method to obtain the feature sequence of the video.

5. The video time-of-view retrieval method based on feature representation enhancement according to claim 1, characterized in that, The cross-modal interaction module described in step 4 maps the features of the query statement and the corresponding video to the same dimension through different fully connected layers; it then concatenates the feature sequences of the query statement and the corresponding video to perform positional encoding; it inputs the positionally encoded feature sequences of the query statement and the corresponding video into a single-layer cross-modal Transformer, where the parameters Q are the video feature sequences and K and V are the query statement feature sequences, and outputs a fused feature sequence; finally, it concatenates any two features in the fused feature sequence to obtain the features of all candidate segments.

6. The video time-of-view retrieval method based on feature representation enhancement according to claim 1, characterized in that, The time retrieval head module described in step 4 inputs all feature representation-enhanced candidate segment features into a fully connected layer and outputs a relevance score between each candidate segment and the query statement.

7. The video time-of-view retrieval method based on feature representation enhancement according to claim 1, characterized in that, Step 4 of the video time-of-view retrieval network training includes the following steps: The training set is input into the video time-of-view retrieval network; the video feature extraction module and the query feature extraction module are forward-propagated to extract video features and query statement features, respectively; the cross-modal interaction module is forward-propagated to obtain candidate segment features; the bi-connected graph convolution module is forward-propagated to obtain candidate segment features with enhanced feature representations; the time-of-view retrieval head module is forward-propagated to obtain the relevance score between each candidate segment and the query statement; the cross-entropy loss function is used to calculate the loss value between the predicted relevance score and the true relevance score of the candidate segment, where the true relevance score comes from the intersection-union ratio of the start time of each candidate segment in the corresponding video to the start and end times in the corresponding true time label; the gradient descent method is used to train the randomly generated initial weights of the network, iteratively updating the network parameters until the network loss function converges, resulting in the trained video time-of-view retrieval network. The cross-entropy loss function is as follows: ； in, This represents the loss value between the predicted relevance score and the true relevance score of the candidate fragment. This indicates the number of samples in the training set. This represents the number of candidate fragments constructed for each sample during the prediction process. Represents a probability distribution. Indicates the true relevance score. This represents the predicted relevance score.

8. The video time-of-view retrieval method based on feature representation enhancement according to claim 1, characterized in that, The video time retrieval mentioned in step 5 refers to inputting the video and the query statement into the video time retrieval network, outputting the relevance scores of multiple candidate segments with the query statement, sorting the candidate segments according to the relevance scores, and selecting the candidate segment with the highest relevance score as the final retrieval result.

Images