Sentence localization in video clips based on progressive interaction and multimodal alignment

Abstract

The present invention relates to the technical field of computer vision and natural language processing, and in particular to a video segment sentence positioning algorithm based on progressive interaction and multimodal alignment. The steps are as follows: first, all query sentences related to the video are sorted according to the order of the video segments corresponding to the query sentences in the video, and then spliced ​​with the video features and in the feature dimension, and then the multimodal alignment module is used to extract the intra-modal information of the video and query sentence features and the information between the two modalities, and then multiple candidate segments are generated according to the video features after interaction with the query sentence, and the relationship between the candidate segments is learned through the grouping candidate segment interaction module, and then the difference between the corresponding candidate segment features and the query sentence feature pairs is reduced through metric learning, and finally all the candidate segment features and the single query sentence features are calculated The matching score, the high score is used as the prediction result. The present invention can accurately locate the video segment.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and natural language processing, and in particular to a method for localizing sentences in video clips based on progressive interaction and multimodal alignment. Background Art

[0002] Video segment sentence localization was first proposed at the 2017 ICCV conference and immediately attracted widespread attention. In 2017, Gao et al. added natural language query sentence annotations to the Charades dataset to create the Charades-STA dataset. Since then, a large amount of research has emerged in the field of video segment sentence localization. As datasets continue to expand, the performance of video segment sentence localization methods has gradually improved across various datasets.

[0003] The current mainstream methods are mainly divided into two parts, proposal-based methods and proposal-free methods. The proposal-based methods basically follow this process: first, a certain sampling method is performed on the video. The sampling method should cover the possibility of the entire video segment as much as possible. Then, the sampled sample segments are used as candidate segments, and these candidate segments are scored with the given natural language query sentences, and one or several with higher scores are selected as the final result. Early candidate segment generation methods initially relied on manually generated sampling segments. The MCN method proposed by Hendricks et al. divides the video into multiple 5-second segments and obtains the final candidate segment by combining continuous segments. Although this sampling method can completely cover the video, this method has certain limitations and cannot cover segments within 5 seconds; subsequently, the CTRL method proposed by Gao et al. generates candidate segments by adopting a multi-scale sliding window. This sampling method can obtain more comprehensive candidate segments. However, for the multimodal fusion part, this method only uses simple addition, multiplication and feature splicing, which makes it difficult to fully fuse multimodal features, and simple operations cannot completely align the features between the two modalities; Yuan Yitian et al. proposed a video clip sentence localization method based on refined multimodal association, which fuses the two modalities of video and natural language query sentences through a multimodal co-attention mechanism; relying solely on the information of candidate clips and natural language query sentences, the method can achieve limited results, so many researchers began to explore other information; for example, the SLTA method proposed by Jiang et al. captures object-level interaction information as supplementary information to more accurately align the two modalities. The ACL method proposed by Ge et al. recognizes activity concepts as additional information, further enhancing the feature interaction between video and text; the QSPN method proposed by Xu et al. provides a different solution to better align the two modalities. It first interacts video information with text information, and then generates candidate segments, and obtains better fusion effects through advance information fusion; since the proposal-based method needs to generate a large number of candidate segments and there is a lot of redundancy between these candidate segments, in order to reduce the waste of computing resources, Zhang et al. proposed the 2D-TAN method to solve the redundancy problem of candidate segments; this method uses sparse sampling to ensure that candidate segments are sampled as comprehensively as possible while reducing redundant segments.

[0004] Although the proposal-based method has good results, the computational complexity of the model is large. Therefore, some researchers have proposed to directly predict the corresponding start and end times in the video without relying on candidate segments. For example, the ABLR method proposed by Yuan et al. directly predicts the start and end times of related video segments through regression based on video attention weights and regression based on video features and natural language query sentence features. A video contains a large number of video frames, but the positive samples are only the start and end frames, which causes data imbalance. Therefore, some researchers have studied this problem. For example, the DEBUG method proposed by Lu et al. uses the frames between the start and end frames as positive samples and densely predicts the distance from each frame to the boundary to alleviate the data imbalance problem. Subsequently, He et al. found that the process of video segment sentence localization is similar to reinforcement learning, and designed the RWM-RL method to obtain the start and end times of related segments in the video through a continuously learning sliding window. These problems can be summarized as follows: (1) Video semantic information is not fully mined; (2) Visual information and text information are not fully aligned.

[0005] Therefore, the present invention provides a video clip sentence localization method based on progressive interaction and multimodal alignment to solve the above problems. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the present invention proposes a video clip sentence localization method based on progressive interaction and multimodal alignment. The main purpose is to fully mine the video semantic information and study how to more accurately and robustly align the video semantic information with the text semantic information.

[0007] The technical solution to the technical problem solved by the present invention is a method for localizing sentences in video clips based on progressive interaction and multimodal alignment, which is as follows:

[0008] S1. Determine the video to be sentence-localized and the sentence to be queried, extract video features from the video, and extract sentence features from the sentence to be queried;

[0009] S2. Preprocessing the extracted video features and sentence features to transform the video features and sentence features into specified shapes to obtain processed video features and sentence features;

[0010] S3. Align the features of the video modality and the sentence modality, splice the processed video features and sentence features in the channel dimension, and then perform an inverse operation on the spliced ​​features in the length dimension to obtain the reverse order features. Then, the forward order features that have not been inverted and the reverse order features that have been inverted are input into the dual-branch network to respectively mine the information between the video modality and the sentence modality to obtain the multimodal sequence fusion features. ;

[0011] S4. Fusion features based on multimodal sequences Generate candidate segment group features ;

[0012] S5. Group the candidate segments into features The information between video clips is learned in each channel through depth-wise separable convolution, and then the information between each group is mined from coarse to fine through group convolution to obtain new candidate clip features. ;

[0013] S6. By contrasting the learning loss function, the features of the two modalities, video and sentence, are placed in a common embedding space, completing the learning of the video and sentence modal features.

[0014] S7. Based on the characteristics of candidate segments Get the prediction scores of the candidate video clips and constrain the prediction scores.

[0015] S1 is as follows:

[0016] The video to be sentence located contains multiple sentences. A sentence to be queried is given, and the position of the sentence to be queried in the video to be sentence located is determined based on the extracted sentence features and video features.

[0017] Among them, the sentence to be queried is a natural language query sentence, through Interactive 3D models or The 3D convolutional model extracts video features from the video by The model extracts sentence features from the query sentence. The extracted video features are 512-dimensional for each segment and 768-dimensional for each sentence.

[0018] S2 is as follows:

[0019] Convolution, pooling, and linear operations are performed on the extracted video features and sentence features to transform them into specified shapes, thereby obtaining the processed video features and sentence features. The calculation process is as follows:

[0020] ,

[0021] ,

[0022] in, represents the extracted video features, represents the extracted sentence features, and Represent the processed video features and sentence features respectively, features and Aggregates global timing information, 、 and Represent the average pooling operation, activation function and convolution operation respectively, represents the learnable projection matrix, represents the learnable offset parameter.

[0023] S3 is as follows:

[0024] The processed video features and sentence features are spliced ​​in the feature dimension to obtain features , the calculation process is as follows:

[0025] ,

[0026] ,

[0027] in, represents the feature concatenation operation, Indicates the number of sentences corresponding to the video, Indicates the Features of the processed sentences;

[0028] Then the features after splicing Perform inverse operation on the length dimension to obtain reverse order features , features that have not been inverted is a positive sequence feature, and then the positive sequence feature and reverse order features Input to the dual-branch network, the dual branches include forward branch and reverse branch, specifically the positive sequence feature Input to the forward branch, reverse the feature Input to the reverse branch;

[0029] In the forward branch, a dual branch is first constructed again through the gating mechanism, one branch is used to calculate the positive sequence feature The weight matrix of the information in the other branch is used to capture the positive sequence features Information in, positive sequence characteristics After a linear layer, the positive sequence features The dimension is projected onto Dimension, represents the dimension of the extracted video features, , the positive sequence feature The feature dimension is divided into two equal feature channels. Features and features , the features and features They are respectively used as the input of the dual branches of the gating mechanism; the operations in the forward branch and the reverse branch are consistent, and similarly, the dual branch input features of the gating mechanism in the reverse branch are obtained and features , the calculation process is as follows:

[0030] ,

[0031] ),

[0032] in, and denote the learnable projection matrices in the forward branch and the reverse branch, respectively, and denote the learnable offset parameters in the forward branch and the reverse branch, respectively, Represents the operation of dividing the feature into two features according to the number of channels in the feature dimension.

[0033] The specific process within the branch in the gating mechanism is as follows:

[0034] For the two branches of the gating mechanism in the forward branch, the features Enter a branch, through The activation function obtains positive sequence features The weight matrix of the information , the features Input to another branch, feature First, a one-dimensional convolution with a convolution kernel of 4 is performed, and then Module alignment characteristics Mining the intra-modal and inter-modal information of the two modalities of video and sentence. The module is the module of the spatial state model; similarly, the reverse feature is obtained The weight matrix of the information and reverse multimodal sequence features ;

[0035] The multimodal sequence features in two directions are added together and passed through a linear layer to obtain the multimodal sequence fusion feature , The feature dimension is , the calculation process is as follows:

[0036] ,

[0037] ,

[0038] ,

[0039] in, Represents the activation function Operation, represents one-dimensional convolution, represents a linear layer, Represents the operation of a sequence module.

[0040] S4 is as follows:

[0041] Fusion of multimodal sequence features Split in the time dimension to obtain video features All sentence features related to this video , , Indicates the first The video features are then split using the 2D-TAN method. After splitting, input into the maximum pooling operation to obtain candidate segment features , by the candidate segment features Composition of candidate segment group features ,2D-TAN stands for two-dimensional temporal connection network, which is a method for video positioning. The calculation process is as follows:

[0042] ,

[0043] ,

[0044] ,

[0045] in, Indicates the The features of candidate segments, Represents the video features at different positions after the video features are split. Video features representing the starting position, Video features indicating the end position, represents the maximum pooling operation, represents the total number of candidate segments, represents the index of the candidate segment, Indicates the number of sentences corresponding to the video, The operation represents a split along the channel dimension.

[0046] S5 is as follows:

[0047] Candidate fragment features The candidate segment features are transformed into the channel dimension through grouped convolution The information between each group of fragments is learned by dividing them into 8 groups, 4 groups, and 2 groups respectively. Then, the information of all channels is aggregated by depthwise separable convolution with a convolution kernel of 1 to obtain new candidate fragment features. , the calculation process is as follows:

[0048] ,

[0049] in, represents depthwise separable convolution, Represents 3 grouped convolutions with groups of 2, 4, and 8 respectively.

[0050] S6 is as follows:

[0051] The contrastive learning loss is used to make the features of the two modalities, video and sentence, in a common embedding space. The calculation process is as follows:

[0052] ,

[0053] in, represents the total number of video-natural language query sentence pairs used for training, and Respectively represent The contrastive learning scores obtained by taking the natural language query sentence as the standard and the The contrastive learning score obtained by taking candidate segments as the standard, Indicates the Negative samples of natural language query sentences corresponding to candidate segments, Indicates the Negative samples of candidate segments corresponding to natural language query sentences, Indicates the candidate fragments, Indicates the A natural language query sentence.

[0054] S7 is as follows:

[0055] The new candidate segment features After convolution and The function obtains the prediction score of each candidate matrix , the process is as follows:

[0056] ,

[0057] For the prediction score To constrain, by The prediction score of the candidate matrix The intersection over union (IoU) of the video segments corresponding to the candidate moments and the text in the video is approximated using the cross entropy loss as follows:

[0058] ,

[0059] in, represents the number of valid candidate matrices, represents the index of the valid candidate matrix, Indicates the The intersection over union (IoU) of the candidate moments and the video time segments corresponding to the text in the video, Indicates the The prediction scores of candidate matrices.

[0060] The effects provided in the summary of the invention are only the effects of the embodiments, rather than all the effects of the invention. The above technical solution has the following advantages or beneficial effects:

[0061] The present invention fine-tunes and learns the information between the video modality and the sentence modality by aligning the features of the two modalities, extracting the complex relationships within each modality and the semantic relationships between different modalities. This can effectively align the information between the two modalities and help the model locate more accurate video segments. The present invention uses a progressive candidate segment interaction module to learn the information between the candidate segments, enabling the model to learn the ability to distinguish the correct answer. Compared with existing methods, the progressive candidate segment interaction module can greatly reduce the number of model parameters for processing candidate segment information and enable the model to fully learn the information of the candidate segments, which can achieve better results while being more lightweight.

[0062] In summary, the present invention has achieved excellent performance through comprehensive learning of candidate segment information and can more accurately locate sentences in videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0064] Figure 1 Schematic diagram of the method of the present invention.

[0065] Figure 2 The comparison between the results of adding the technical solution of the present invention to the baseline model and the baseline model is shown in FIG. Figure 1 .

[0066] Figure 3 The comparison between the results of adding the technical solution of the present invention to the baseline model and the baseline model is shown in FIG. Figure 2 . DETAILED DESCRIPTION

[0067] In order to clearly illustrate the technical features of this solution, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0068] Example 1

[0069] like Figure 1 As shown in the figure, a video clip sentence localization method based on progressive interaction and multimodal alignment is as follows:

[0070] S1. Determine the video to be sentence-localized and the sentence to be queried, extract video features from the video, and extract sentence features from the sentence to be queried;

[0071] S2. Preprocessing the extracted video features and sentence features to transform the video features and sentence features into specified shapes to obtain processed video features and sentence features;

[0072] S3. Align the features of the video modality and the sentence modality, splice the processed video features and sentence features in the channel dimension, and then perform an inverse operation on the spliced ​​features in the length dimension to obtain the reverse order features. Then, the forward order features that have not been inverted and the reverse order features that have been inverted are input into the dual-branch network to respectively mine the information between the video modality and the sentence modality to obtain the multimodal sequence fusion features. ;

[0073] S4. Fusion features based on multimodal sequences Generate candidate segment group features ;

[0074] S5. Group the candidate segments into features The information between video clips is learned in each channel through depth-wise separable convolution, and then the information between each group is mined from coarse to fine through group convolution to obtain new candidate clip features. ;

[0075] S6. By contrasting the learning loss function, the features of the two modalities, video and sentence, are placed in a common embedding space, completing the learning of the video and sentence modal features.

[0076] S7. Based on the characteristics of candidate segments Get the prediction scores of the candidate video clips and constrain the prediction scores.

[0077] S1 is as follows:

[0078] The video to be sentence located contains multiple sentences. A sentence to be queried is given, and the position of the sentence to be queried in the video to be sentence located is determined based on the extracted sentence features and video features.

[0079] Among them, the sentence to be queried is a natural language query sentence, through Interactive 3D models or The 3D convolutional model extracts video features from the video by The model extracts sentence features from the query sentence. The extracted video features are 512-dimensional for each segment and 768-dimensional for each sentence.

[0080] S2 is as follows:

[0081] Convolution, pooling, and linear operations are performed on the extracted video features and sentence features to transform them into specified shapes, thereby obtaining the processed video features and sentence features. The calculation process is as follows:

[0082] ,

[0083] ,

[0084] in, represents the extracted video features, represents the extracted sentence features, and Represent the processed video features and sentence features respectively, features and Aggregates global timing information, 、 and Represent the average pooling operation, activation function and convolution operation respectively, represents the learnable projection matrix, represents the learnable offset parameter.

[0085] S3 is as follows:

[0086] The processed video features and sentence features are spliced ​​in the feature dimension to obtain features , the calculation process is as follows:

[0087] ,

[0088] ,

[0089] in, represents the feature concatenation operation, Indicates the number of sentences corresponding to the video, Indicates the Features of the processed sentences;

[0090] Then the features after splicing Perform inverse operation on the length dimension to obtain reverse order features , features that have not been inverted is a positive sequence feature, and then the positive sequence feature and reverse order features Input to the dual-branch network, the dual branches include forward branch and reverse branch, specifically the positive sequence feature Input to the forward branch, reverse the feature Input to the reverse branch;

[0091] In the forward branch, a dual branch is first constructed again through the gating mechanism, one branch is used to calculate the positive sequence feature The weight matrix of the information in the other branch is used to capture the positive sequence features Information in, positive sequence characteristics After a linear layer, the positive sequence features The dimension is projected onto Dimension, represents the dimension of the extracted video features, , the positive sequence feature The feature dimension is divided into two equal feature channels. Features and features , the features and features They are respectively used as the input of the dual branches of the gating mechanism; the operations in the forward branch and the reverse branch are consistent, and similarly, the dual branch input features of the gating mechanism in the reverse branch are obtained and features , the calculation process is as follows:

[0092] ,

[0093] ),

[0094] in, and denote the learnable projection matrices in the forward branch and the reverse branch, respectively, and denote the learnable offset parameters in the forward branch and the reverse branch, respectively, Represents the operation of dividing the feature into two features according to the number of channels in the feature dimension.

[0095] The specific process within the branch in the gating mechanism is as follows:

[0096] For the two branches of the gating mechanism in the forward branch, the features Enter a branch, through The activation function obtains positive sequence features The weight matrix of the information , the features Input to another branch, feature First, a one-dimensional convolution with a convolution kernel of 4 is performed, and then Module alignment characteristics Mining the intra-modal and inter-modal information of the two modalities of video and sentence. Module by 、 and The three frameworks are integrated, and finally the two branch results are multiplied pixel by pixel to obtain the positive multimodal sequence features. , The framework is an open source platform, The framework is a model view controller framework, A framework is a Persistence layer framework; similarly, reverse order features are obtained The weight matrix of the information and reverse multimodal sequence features ;

[0097] The multimodal sequence features in two directions are added together and passed through a linear layer to obtain the multimodal sequence fusion feature , The feature dimension is , the calculation process is as follows:

[0098] ,

[0099] ,

[0100] ,

[0101] in, Represents the activation function Operation, represents one-dimensional convolution, represents a linear layer, Represents the operation of a sequence module.

[0102] S4 is as follows:

[0103] Fusion of multimodal sequence features Split in the time dimension to obtain video features All sentence features related to this video , , Indicates the first The video features are then split using the 2D-TAN method. After splitting, input into the maximum pooling operation to obtain candidate segment features , by the candidate segment features Composition of candidate segment group features ,2D-TAN stands for two-dimensional temporal connection network, which is a method for video positioning. The calculation process is as follows:

[0104] ,

[0105] ,

[0106] ,

[0107] in, Indicates the The features of candidate segments, Represents the video features at different positions after the video features are split. Video features representing the starting position, Video features indicating the end position, represents the maximum pooling operation, represents the total number of candidate segments, represents the index of the candidate segment, Indicates the number of sentences corresponding to the video, The operation represents a split along the channel dimension.

[0108] S5 is as follows:

[0109] Candidate fragment features The candidate segment features are transformed into the channel dimension through grouped convolution The information between each group of fragments is learned by dividing them into 8 groups, 4 groups, and 2 groups respectively. Then, the information of all channels is aggregated by depthwise separable convolution with a convolution kernel of 1 to obtain new candidate fragment features. , the calculation process is as follows:

[0110] ,

[0111] in, represents depthwise separable convolution, Represents 3 grouped convolutions with groups of 2, 4, and 8 respectively.

[0112] S6 is as follows:

[0113] The contrastive learning loss is used to make the features of the two modalities, video and sentence, in a common embedding space. The calculation process is as follows:

[0114] ,

[0115] in, represents the total number of video-natural language query sentence pairs used for training, and Respectively represent The contrastive learning scores obtained by taking the natural language query sentence as the standard and the The contrastive learning score obtained by taking candidate segments as the standard, Indicates the Negative samples of natural language query sentences corresponding to candidate segments, Indicates the Negative samples of candidate segments corresponding to natural language query sentences, Indicates the candidate fragments, Indicates the A natural language query sentence.

[0116] S7 is as follows:

[0117] The new candidate segment features After convolution and The function obtains the prediction score of each candidate matrix , the process is as follows:

[0118] ,

[0119] For the prediction score To constrain, by The prediction score of the candidate matrix The intersection over union (IoU) of the video segments corresponding to the candidate moments and the text in the video is approximated using the cross entropy loss as follows:

[0120] ,

[0121] in, represents the number of valid candidate matrices, represents the index of the valid candidate matrix, Indicates the The intersection over union (IoU) of the candidate moments and the video time segments corresponding to the text in the video, Indicates the The prediction scores of candidate matrices.

[0122] Example 2

[0123] This paper uses a pre-trained I3D model for the Charades-STA dataset (a word puzzle dataset), and a pre-trained C3D model to extract features from the TACoS dataset (a kitchen dataset) and the ActivityNet-Captions dataset (a video dataset). All experiments in this chapter are run on a Titan RTX graphics card with 24GB of video memory. The model is optimized using the AdamW optimizer. For the ActivityNet Captions, TACoS, and Charades-STA datasets, learning rates of 8×10⁻¹⁴, 15×10⁻¹⁴, and 1×10⁻³ are used, and batch sizes of 12, 2, and 12 are used, respectively.

[0124] During the model training process, when the model reaches a bottleneck using contrast loss and cross entropy loss, the model will significantly reduce the weight of contrast loss in subsequent training, reducing it to 1 / 100 of the original. The parameter settings of contrast learning used in the experiment are consistent with the MMN method; the number of sample clips for the Charades-STA and ActivityNet Captions datasets is set to 64, while the number of samples in the TACoS dataset is set to 128; in order to reduce the computational complexity of the model, the size of all hidden states in the model is set to 256; in the cross entropy loss process of model training, the thresholds of the Charades-STA and ActivityNet Captions datasets are set to 0.5 and 1.0, while those for the TACoS dataset are set to 0.3 and 0.7, which are consistent with the 2D-TAN method.

[0125] The evaluation metric mIoU is derived from target detection and is used to measure the similarity between two bounding boxes in the target detection field. It is obtained by calculating the IoU score between the label time boundary and the time boundary predicted by the model, and finally averaging the IoU of all sample pairs.

[0126] The evaluation indicator "R@n, IoU=m" means that among all the prediction results of the current video and the natural language query sentence, if there is a prediction score greater than the set threshold m among the top n candidate segments with higher prediction scores, it is considered a correct prediction. Otherwise, the prediction result of the video and the natural language query sentence is considered an incorrect prediction. The proportion of correct predictions in all the prediction results is then calculated.

[0127] During the test, the video is first subjected to feature extraction by an I3D or C3D model, and the natural language query sentence is encoded by a Bert model. The video and all queries related to the video are first spliced ​​in the feature dimension to form a multimodal feature, which is then sent to a sequence model for learning intra-modal and inter-modal information. The video features after interacting with the query are sampled according to the 2D-TAN method proposed by Zhang et al. to obtain a set of video clip features; using contrastive learning, the video clip corresponding to the query and the query are used as positive samples, and other video clips of the same video that do not correspond to the query and video clips of different videos are used as negative samples, so that video clips with the same semantics are closer to the query features, and video clips with different semantics are farther away from the query features. Then, the potential relationship between the video clips is learned, and there is a certain correlation between different video clips. Then, the matching score of the video clip features and the query features is calculated;

[0128] The method in this invention is compared with other methods on the Charades-STA, ActivityNet-Captions and TACoS datasets, and the experimental results are shown in Tables 1 and 2.

[0129] The comparison of the experimental results of the present invention with other methods on the Charades-STA dataset is shown in Table 1, the comparison of the experimental results of the present invention with other methods on the TACoS dataset is shown in Table 2, and the comparison of the experimental results of the present invention with other methods on the ActivityNet-Captions dataset is shown in Table 3.

[0130] Table 1 Comparison of the results of the method of the present invention and other methods on the Charades-STA dataset

[0131]

[0132] For the Charades-STA dataset, we can achieve 57.80 and 35.73 results on R@1,IoU0.5 and R@1,IoU0.7, 88.71 and 70.08 results on R@1,IoU=0.5 and R@5,IoU=0.7, and 51.50 results on mIoU.

[0133] Table 2 Comparison of the results of the method of the present invention and other methods on the TACoS dataset

[0134]

[0135] For the TACoS dataset, the results can be 62.38 and 47.69 on R@1,IoU0.5 and R@1,IoU0.7, the results can be 82.03 and 70.83 on R@5,IoU0.5 and R@5,IoU=0.7, and the result can be 43.89 on mIoU.

[0136] Table 3 Comparison of the results of the method of the present invention and other methods on the ActivityNet-Captions dataset

[0137]

[0138] For the ActivityNet-Captions dataset, the results can be 64.55 and 44.10 on R@1,IoU0.5 and R@1,IoU0.7, the results can be 86.88 and 76.42 on R@5,IoU0.5 and R@5,IoU=0.7, and the result can be 58.44 on mIoU.

[0139] Example 3

[0140] In order to highlight the advantages of the method of the present invention, the technical solution of the present invention is added to the benchmark model and the results are compared with those of the benchmark model. Figure 2 and Figure 3 As shown in the figure, given the search sentences "Query: The person gets a plate out of the cupboard." and "Query: The woman grabs a cuttingboard.", the video interval where the sentence is located is located according to the given sentence. ① represents the baseline model, ② represents the baseline model with the multimodal alignment module added, and ③ represents the method of the present invention. Figure 2 and Figure 3 It can be seen that compared with the true label, which is the actual position of the retrieved sentence, the baseline model with the technical solution of the present invention can achieve a result that is more inclined to the correct answer.

[0141] Although the above describes the specific implementation methods of the invention in conjunction with the accompanying drawings, it does not limit the scope of protection of the invention. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A video clip sentence localization method based on progressive interaction and multimodal alignment, characterized in that: The following steps are involved: S1. Determine the video to be sentence located and the sentence to be queried, extract video features from the video, and extract sentence features from the sentence to be queried; S2, preprocessing the extracted video features and sentence features, transforming the video features and sentence features into specified shapes, and obtaining processed video features and sentence features; S3. Align the features of the video modality and the sentence modality, concatenate the processed video features and sentence features in the channel dimension, and then perform an inverse operation on the concatenated features in the length dimension to obtain the reverse order features. Then, the forward order features that have not been inverted and the reverse order features that have been inverted are input into the dual-branch network to respectively mine the information between the video modality and the sentence modality to obtain the multimodal sequence fusion features. ; S4. Fusion features based on multimodal sequences Generate candidate segment group features ; Fusion of multimodal sequence features Split in the time dimension to obtain video features All sentence features related to this video , , Indicates the first The video features are then split using the 2D-TAN method. After splitting, input to the maximum pooling operation to obtain candidate segment features , by the candidate fragment features Composition of candidate fragment group features , 2D-TAN stands for two-dimensional time domain connection network, which is a method used for video positioning. The calculation process is as follows: ， ， ， in, Indicates The features of the candidate segments, Represents the video features at different positions after the video features are split. The video feature representing the starting position, The video feature indicating the end position, represents the maximum pooling operation, represents the total number of candidate segments, represents the index of the candidate segment, Indicates the number of sentences corresponding to the video, The operation represents segmentation in the channel dimension; S5. Group the candidate fragment features The information between video clips is learned in each channel through depthwise separable convolution, and then the information between each group is mined from coarse to fine through grouped convolution to obtain new candidate clip features. ; S6. By contrasting the learning loss function, the features of the two modalities, video and sentence, are placed in a common embedding space, thus completing the learning of the features of the video and sentence modalities. S7. Based on the characteristics of candidate fragments Get the prediction scores of the candidate video clips and constrain the prediction scores.

2. The method for locating sentences in video clips based on progressive interaction and multimodal alignment according to claim 1, characterized in that: S1 is as follows: The video to be sentence located contains multiple sentences, a sentence to be queried is given, and the position of the sentence to be queried in the video to be sentence located is determined according to the extracted sentence features and video features; Among them, the sentence to be queried is a natural language query sentence, through Interactive 3D models or The 3D convolutional model extracts video features from the video by The model extracts sentence features from the query sentence. The extracted video features are 512-dimensional for each segment and 768-dimensional for each sentence feature.

3. The method for locating sentences in video clips based on progressive interaction and multimodal alignment according to claim 2, characterized in that: S2 is as follows: The extracted video features and sentence features are subjected to convolution, pooling, and linear operations to transform the extracted video features and sentence features into specified shapes, thereby obtaining the processed video features and sentence features. The calculation process is as follows: ， ， in, represents the extracted video features, represents the extracted sentence features, and Represent the processed video features and sentence features respectively. and Aggregates global timing information, , and Represent the average pooling operation, activation function and convolution operation respectively. represents the learnable projection matrix, represents the learnable offset parameter.

4. The method for locating sentences in video clips based on progressive interaction and multimodal alignment according to claim 3 is characterized in that: S3 is as follows: The processed video features and sentence features are concatenated in the feature dimension to obtain features , the calculation process is as follows: ， ， in, represents the feature concatenation operation, Indicates the number of sentences corresponding to the video, Indicates Features of the processed sentences; Then the features after splicing Perform inverse operation on the length dimension to obtain the reverse feature , features that have not been inverted is the positive sequence feature, and then the positive sequence feature and reverse order features Input to the dual-branch network, the dual branches include forward branch and reverse branch, specifically the positive sequence feature Input to the forward branch, and reverse the features Input to the reverse branch; In the forward branch, a dual branch is first constructed again through the gating mechanism, one branch is used to calculate the positive sequence feature The weight matrix of the information in the middle, and the other branch is used to capture the positive sequence features Information in, positive sequence characteristics After a linear layer, the positive sequence features The dimension is projected onto Dimensions, represents the dimension of the extracted video features, , the positive sequence feature The feature dimension is divided into two equal feature channels. Features and Features , the feature and Features They are used as the input of the dual branches of the gating mechanism respectively; the operations in the forward branch and the reverse branch are the same, and the dual branch input features of the gating mechanism in the reverse branch are obtained in the same way and Features , the calculation process is as follows: ， )， in, and denote the learnable projection matrices in the forward branch and the reverse branch, respectively. and denote the learnable offset parameters in the forward branch and the reverse branch, respectively. Represents the operation of dividing the feature into two features according to the number of channels in the feature dimension.

5. The method for locating sentences in video clips based on progressive interaction and multimodal alignment according to claim 4, characterized in that: The process within the branch in the gating mechanism is as follows: For the two branches of the gating mechanism in the forward branch, the features Enter a branch, through The activation function obtains the positive sequence features The weight matrix of the information , the feature Input to another branch, feature First, it passes through a one-dimensional convolution with a convolution kernel of 4, and then passes through Module alignment characteristics The intra-modal and inter-modal information of the two modalities of video and sentence are mined. Modules by , and The three frameworks are integrated, and finally the two branch results are multiplied pixel by pixel to obtain the positive multimodal sequence features. , The framework is an open source platform, The framework is a model view controller framework. A framework is a Persistence layer framework; similarly, the reverse order feature is obtained The weight matrix of the information and reverse multimodal sequence features ; The multimodal sequence features in two directions are added and passed through a linear layer to obtain the multimodal sequence fusion feature , The feature dimension is , the calculation process is as follows: ， ， ， in, Represents the activation function Operation, represents one-dimensional convolution, represents a linear layer, Represents the operation of a sequence module.

6. The method for locating sentences in video clips based on progressive interaction and multimodal alignment according to claim 5, characterized in that: S5 is as follows: Candidate fragment features The candidate segment features are transformed in the channel dimension through grouped convolution The information between each group of fragments is learned by dividing them into 8 groups, 4 groups, and 2 groups respectively. Then, the information of all channels is aggregated by depthwise separable convolution with a convolution kernel of 1 to obtain new candidate fragment features. , the calculation process is as follows: ， in, represents depthwise separable convolution, Represents 3 grouped convolutions with groups of 2, 4, and 8 respectively.

7. The video segment sentence localization method based on progressive interaction and multimodal alignment according to claim 6, characterized in that S6 The details are as follows: The contrastive learning loss enables the features of the two modalities, video and sentence, to be in a common embedding space. The calculation process is as follows: ， in, represents the total number of video-natural language query sentence pairs used for training, and Respectively represent The contrastive learning scores obtained by taking the natural language query sentence as the standard and the The contrastive learning score obtained by taking candidate segments as the standard is Indicates Negative samples of natural language query sentences corresponding to candidate segments, Indicates Negative samples of candidate segments corresponding to natural language query sentences, Indicates candidate fragments, Indicates A natural language query sentence.

8. The method for locating sentences in video clips based on progressive interaction and multimodal alignment according to claim 7, characterized in that: S7 is as follows: The new candidate segment features After convolution and The function obtains the prediction score of each candidate matrix , the process is as follows: ， For the prediction score To constrain, by The prediction scores of candidate matrices The intersection over union (IoU) calculated for the video time segment corresponding to the candidate moment and the text in the video is approximated using the cross entropy loss as follows: ， in, represents the number of valid candidate matrices, represents the index of the valid candidate matrix, Indicates The IoU of the video time segment corresponding to the candidate moments and the text in the video, Indicates The prediction scores of the candidate matrices.

Images