Video clip sentence positioning algorithm based on progressive interaction and multi-modal alignment

Abstract

The invention relates to the technical field of computer vision and natural language processing, in particular to a video clip sentence positioning algorithm based on progressive interaction and multi-modal alignment. The method comprises the following steps of: firstly, sorting all query sentences related to a video according to the sequence of video clips corresponding to the query sentences in the video, and then splicing the query sentences with video features and feature dimensions; extracting intra-modal information of video and query sentence features and information between the two modals through a multi-modal alignment module, generating a plurality of candidate segments according to the video features interacted with the query sentence, and learning the relationship between the candidate segments through a grouping candidate segment interaction module; then reducing the difference between corresponding candidate segment features and query sentence feature pairs through metric learning, finally calculating matching scores of all candidate segment features and a single query sentence feature, and taking the feature with the high score as a prediction result. According to the invention, the video clip can be accurately positioned.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and natural language processing, and in particular to a video clip sentence localization algorithm based on progressive interaction and multimodal alignment. Background Art

[0002] Video segment sentence localization was first proposed at the ICCV conference in 2017, and immediately attracted the attention of a large number of researchers. In 2017, Gao et al. added natural language query sentence annotations to the Charades dataset to obtain the Charades-STA dataset. Since then, a large number of studies in the field of video segment sentence localization have begun to emerge. With the continuous expansion of datasets, the performance of video segment sentence localization algorithms on various datasets has also gradually improved.

[0003] The current mainstream methods are mainly divided into two parts, proposal-based methods and proposal-free methods. Proposal-based methods basically follow this process: first, a sampling algorithm is performed on the video. The sampling algorithm should cover the possibility of the entire video segment as much as possible. Then, the sampled sample segments are used as candidate segments, and these candidate segments are scored with the given natural language query sentences, and one or several with higher scores are selected as the final result. Early candidate segment generation methods initially relied on manually generated sampling segments. The MCN method proposed by Hendricks et al. divides the video into multiple 5-second segments and obtains the final candidate segment by combining continuous segments. Although this sampling method can cover the video completely, this method has certain limitations and cannot cover segments within 5s. Subsequently, the CTRL method proposed by Gao et al. generates candidate segments by adopting a multi-scale sliding window. This sampling method can obtain more comprehensive candidate segments. However, for the multimodal fusion part, this method only uses simple addition, multiplication and feature concatenation, which makes it difficult to fully integrate the multimodal features, and simple operations cannot completely align the features between the two modalities; Yuan Yitian et al. proposed a video clip sentence localization algorithm based on refined multimodal associations, which integrates the two modalities of video and natural language query sentences through a multimodal co-attention mechanism; the algorithm can only achieve limited results by relying on the information of candidate clips and natural language query sentences, so many researchers have begun to explore other information; for example, the SLTA method proposed by Jiang et al. captures object-level interaction information as supplementary information to more accurately align the two modalities. The ACL method proposed by Ge et al. recognizes activity concepts as additional information, further enhancing the feature interaction of video and text; the QSPN method proposed by Xu et al. provides a different solution to better align the two modalities. It first interacts video information with text information, and then generates candidate segments, and obtains better fusion effect through advance information fusion; since the proposal-based method needs to generate a large number of candidate segments and there is a lot of redundancy between these candidate segments, in order to reduce the waste of computing resources, Zhang et al. proposed the 2D-TAN method to solve the redundancy problem of candidate segments; this method uses sparse sampling to ensure that the candidate segments are sampled as comprehensively as possible while reducing redundant segments.

[0004] Although the proposal-based method has good results, the computational complexity of the model is large. Therefore, some researchers have proposed to directly predict the corresponding start and end times in the video without relying on candidate segments. For example, the ABLR method proposed by Yuan et al. directly predicts the start and end times of related video segments through regression based on video attention weights and regression based on video features and natural language query sentence features. A video contains a large number of video frames, but the positive samples are only the start and end frames, which causes data imbalance. Therefore, some researchers have studied this issue. For example, the DEBUG algorithm proposed by Lu et al. uses the frames between the start frame and the end frame as positive samples and densely predicts the distance from each frame to the boundary to alleviate the data imbalance problem. Subsequently, He et al. found that the process of video segment sentence localization is similar to reinforcement learning, and designed the RWM-RL algorithm to obtain the start and end times of related segments in the video through a continuously learning sliding window. These problems can be summarized as follows: (1) Video semantic information is not fully mined; (2) Visual information and text information are not fully aligned.

[0005] Therefore, the present invention provides a video clip sentence localization algorithm based on progressive interaction and multimodal alignment to solve the above problems. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a video clip sentence localization algorithm based on progressive interaction and multimodal alignment. The main purpose is to fully mine the video semantic information and study how to align the video semantic information with the text semantic information more accurately and robustly.

[0007] The technical solution of the present invention to solve the technical problem is a video clip sentence localization algorithm based on progressive interaction and multimodal alignment, which is as follows: S1. Determine the video to be sentence located and the sentence to be queried, extract video features from the video, and extract sentence features from the sentence to be queried; S2, preprocessing the extracted video features and sentence features, transforming the video features and sentence features into specified shapes, and obtaining processed video features and sentence features; S3. Align the features of the video modality and the sentence modality, concatenate the processed video features and sentence features in the channel dimension, and then perform an inverse operation on the concatenated features in the length dimension to obtain the reverse order features. Then, the forward order features that have not been inverted and the reverse order features that have been inverted are input into the dual-branch network to respectively mine the information between the video modality and the sentence modality to obtain the multimodal sequence fusion features. ; S4. Fusion features based on multimodal sequences Generate candidate segment group features ; S5. Group the candidate fragments into features The information between video clips is learned in each channel through depthwise separable convolution, and then the information between each group is mined from coarse to fine through grouped convolution to obtain new candidate clip features. ; S6. By contrasting the learning loss function, the features of the two modalities, video and sentence, are placed in a common embedding space, thus completing the learning of the features of the video and sentence modalities. S7. Based on the characteristics of candidate fragments The prediction scores of the candidate video clips are obtained and constraints are imposed on the prediction scores.

[0008] S1 is as follows: The video to be sentence located contains multiple sentences, a sentence to be queried is given, and the position of the sentence to be queried in the video to be sentence located is determined according to the extracted sentence features and video features; Among them, the sentence to be queried is a natural language query sentence, through Interactive 3D models or The 3D convolutional model extracts video features from the video by The model extracts sentence features from the query sentence. The extracted video features are 512-dimensional for each segment and 768-dimensional for each sentence feature.

[0009] S2 is as follows: The extracted video features and sentence features are subjected to convolution, pooling, and linear operations to transform the extracted video features and sentence features into specified shapes, thereby obtaining the processed video features and sentence features. The calculation process is as follows: , , in, represents the extracted video features, represents the extracted sentence features, and Represent the processed video features and sentence features respectively. and Aggregates global timing information, , and Represent the average pooling operation, activation function and convolution operation respectively. represents the learnable projection matrix, represents the learnable offset parameter.

[0010] S3 is as follows: The processed video features and sentence features are concatenated in the feature dimension to obtain features , the calculation process is as follows: , , in, represents the feature concatenation operation, Indicates the number of sentences corresponding to the video, Indicates Features of the processed sentences; Then the features after splicing Perform inverse operation on the length dimension to obtain the reverse feature , features that have not been inverted is the positive sequence feature, and then the positive sequence feature and reverse order features Input to the dual-branch network, the dual branches include forward branch and reverse branch, specifically the positive sequence feature Input to the forward branch, and reverse the features Input to the reverse branch; In the forward branch, a dual branch is first constructed again through the gating mechanism, one branch is used to calculate the positive sequence feature The weight matrix of the information in the middle, and the other branch is used to capture the positive sequence features Information in, positive sequence characteristics After a linear layer, the positive sequence features The dimension is projected onto Dimensions, represents the dimension of the extracted video features, , the positive sequence feature The feature dimension is divided into two equal feature channels. Features and Features , the feature and Features They are used as the input of the dual branches of the gating mechanism respectively; the operations in the forward branch and the reverse branch are the same, and the dual branch input features of the gating mechanism in the reverse branch are obtained in the same way and Features , the calculation process is as follows: , ), in, and denote the learnable projection matrices in the forward branch and the reverse branch, respectively. and denote the learnable offset parameters in the forward branch and the reverse branch, respectively. Represents the operation of dividing the feature into two features according to the number of channels in the feature dimension.

[0011] The process within the branch in the gating mechanism is as follows: For the two branches of the gating mechanism in the forward branch, the features Enter a branch, through The activation function obtains the positive sequence features The weight matrix of the information in , the feature Input to another branch, feature First, it passes through a one-dimensional convolution with a convolution kernel of 4, and then passes through Module alignment characteristics The intra-modal and inter-modal information of the two modalities of video and sentence are mined. The module is the module of the spatial state model; similarly, the reverse feature is obtained The weight matrix of the information in and reverse multimodal sequence features ; The multimodal sequence features in two directions are added and passed through a linear layer to obtain the multimodal sequence fusion feature , The feature dimension is , the calculation process is as follows: , , , in, Represents the activation function Operation, represents one-dimensional convolution, represents a linear layer, Represents the operation of a sequence module. S4 is as follows: Fusion of multimodal sequence features Split in the time dimension to obtain video features All sentence features related to this video , , Indicates the first The video features are then split using the 2D-TAN method. After splitting, input to the maximum pooling operation to obtain candidate segment features , by the candidate fragment features Composition of candidate fragment group features , 2D-TAN stands for two-dimensional time domain connection network, which is a method used for video positioning. The calculation process is as follows: , , , in, Indicates The features of the candidate segments, Represents the video features at different positions after the video features are split. The video feature representing the starting position, The video feature indicating the end position, represents the maximum pooling operation, represents the total number of candidate segments, represents the index of the candidate segment, Indicates the number of sentences corresponding to the video, The operation represents a split on the channel dimension.

[0012] S5 is as follows: Candidate fragment features The candidate segment features are transformed in the channel dimension through grouped convolution The information between each group of fragments is learned by dividing them into 8 groups, 4 groups, and 2 groups respectively. Then, the information of all channels is aggregated by depthwise separable convolution with a convolution kernel of 1 to obtain new candidate fragment features. , the calculation process is as follows: , in, represents depthwise separable convolution, Represents 3 grouped convolutions with groups of 2, 4, and 8 respectively.

[0013] S6 is as follows: The contrastive learning loss enables the features of the two modalities, video and sentence, to be in a common embedding space. The calculation process is as follows: , in, represents the total number of video-natural language query sentence pairs used for training, and Respectively represent The contrastive learning scores obtained by taking the natural language query sentence as the standard and the The contrastive learning score obtained by taking candidate segments as the standard is Indicates Negative samples of natural language query sentences corresponding to candidate segments, Indicates Negative samples of candidate segments corresponding to natural language query sentences, Indicates candidate fragments, Indicates A natural language query sentence.

[0014] S7 is as follows: The new candidate segment features After convolution and The function obtains the prediction score of each candidate matrix , the process is as follows: , For the prediction score To constrain, by The prediction scores of candidate matrices The intersection over union (IoU) calculated for the video time segment corresponding to the candidate moment and the text in the video is approximated using the cross entropy loss as follows: , in, represents the number of valid candidate matrices, represents the index of the valid candidate matrix, Indicates The IoU of the video time segment corresponding to the candidate moments and the text in the video, Indicates The prediction scores of the candidate matrices.

[0015] The effects provided in the content of the invention are only the effects of the embodiments, rather than all the effects of the invention. The above technical solution has the following advantages or beneficial effects: The present invention fine-tunes and learns the information between the two modalities by aligning the features of the video modality and the sentence modality, extracts the complex relationships within each modality and the semantic relationships between different modalities, and can well align the information between the two modalities, helping the model to locate more accurate video segments; the present invention uses the progressive candidate segment interaction module to learn the information between the candidate segments so that the model can learn the ability to distinguish the correct answer. Compared with the existing method, the progressive candidate segment interaction module can greatly reduce the number of model parameters for processing candidate segment information, and enables the model to fully learn the information of the candidate segments, so that the model can achieve better results under the premise of being more lightweight; In summary, the present invention has achieved excellent performance through comprehensive learning of candidate segment information and can locate sentences in videos more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0017] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0018] Figure 2 The comparison between the results of adding the technical solution of the present invention to the benchmark model and the benchmark model is shown in FIG. Figure 1 .

[0019] Figure 3 The comparison between the results of adding the technical solution of the present invention to the benchmark model and the benchmark model is shown in FIG. Figure 2 . DETAILED DESCRIPTION

[0020] In order to clearly illustrate the technical features of the present invention, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0021] Example 1 like Figure 1 As shown in the figure, a video clip sentence localization algorithm based on progressive interaction and multimodal alignment is as follows: S1. Determine the video to be sentence located and the sentence to be queried, extract video features from the video, and extract sentence features from the sentence to be queried; S2, preprocessing the extracted video features and sentence features, transforming the video features and sentence features into specified shapes, and obtaining processed video features and sentence features; S3. Align the features of the video modality and the sentence modality, concatenate the processed video features and sentence features in the channel dimension, and then perform an inverse operation on the concatenated features in the length dimension to obtain the reverse order features. Then, the forward order features that have not been inverted and the reverse order features that have been inverted are input into the dual-branch network to respectively mine the information between the video modality and the sentence modality to obtain the multimodal sequence fusion features. ; S4. Fusion features based on multimodal sequences Generate candidate segment group features ; S5. Group the candidate fragments into features The information between video clips is learned in each channel through depthwise separable convolution, and then the information between each group is mined from coarse to fine through grouped convolution to obtain new candidate clip features. ; S6. By contrasting the learning loss function, the features of the two modalities, video and sentence, are placed in a common embedding space, thus completing the learning of the features of the video and sentence modalities. S7. Based on the characteristics of candidate fragments The prediction scores of the candidate video clips are obtained and constraints are imposed on the prediction scores.

[0022] S1 is as follows: The video to be sentence located contains multiple sentences, a sentence to be queried is given, and the position of the sentence to be queried in the video to be sentence located is determined according to the extracted sentence features and video features; Among them, the sentence to be queried is a natural language query sentence, through Interactive 3D models or The 3D convolutional model extracts video features from the video by The model extracts sentence features from the query sentence. The extracted video features are 512-dimensional for each segment and 768-dimensional for each sentence feature.

[0023] S2 is as follows: The extracted video features and sentence features are subjected to convolution, pooling, and linear operations to transform the extracted video features and sentence features into specified shapes, thereby obtaining the processed video features and sentence features. The calculation process is as follows: , , in, represents the extracted video features, represents the extracted sentence features, and Represent the processed video features and sentence features respectively. and Aggregates global timing information, , and Represent the average pooling operation, activation function and convolution operation respectively. represents the learnable projection matrix, represents the learnable offset parameter.

[0024] S3 is as follows: The processed video features and sentence features are concatenated in the feature dimension to obtain features , the calculation process is as follows: , , in, represents the feature concatenation operation, Indicates the number of sentences corresponding to the video, Indicates Features of the processed sentences; Then the features after splicing Perform inverse operation on the length dimension to obtain the reverse feature , features that have not been inverted is the positive sequence feature, and then the positive sequence feature and reverse order features Input to the dual-branch network, the dual branches include forward branch and reverse branch, specifically the positive sequence feature Input to the forward branch and reverse the features Input to the reverse branch; In the forward branch, a dual branch is first constructed again through the gating mechanism, one branch is used to calculate the positive sequence feature The weight matrix of the information in the middle, and the other branch is used to capture the positive sequence features Information in, positive sequence characteristics After a linear layer, the positive sequence features The dimension is projected onto Dimensions, represents the dimension of the extracted video features, , the positive sequence feature The feature dimension is divided into two equal feature channels. Features and Features , the feature and Features They are used as the input of the dual branches of the gating mechanism respectively; the operations in the forward branch and the reverse branch are the same, and the dual branch input features of the gating mechanism in the reverse branch are obtained in the same way and Features , the calculation process is as follows: , ), in, and denote the learnable projection matrices in the forward branch and the reverse branch, respectively. and denote the learnable offset parameters in the forward branch and the reverse branch, respectively. Represents the operation of dividing the feature into two features according to the number of channels in the feature dimension.

[0025] The process within the branch in the gating mechanism is as follows: For the two branches of the gating mechanism in the forward branch, the features Enter a branch, through The activation function obtains the positive sequence features The weight matrix of the information in , the feature Input to another branch, feature First, it passes through a one-dimensional convolution with a convolution kernel of 4, and then passes through Module alignment characteristics The intra-modal and inter-modal information of the two modalities of video and sentence are mined. Modules by , and The three frameworks are integrated, and finally the two branch results are multiplied pixel by pixel to obtain the positive multimodal sequence features. , The framework is an open source platform, The framework is a model view controller framework. A framework is a Persistence layer framework; similarly, the reverse order feature is obtained The weight matrix of the information in and reverse multimodal sequence features ; The multimodal sequence features in two directions are added and passed through a linear layer to obtain the multimodal sequence fusion feature , The feature dimension is , the calculation process is as follows: , , , in, Represents the activation function Operation, represents one-dimensional convolution, represents a linear layer, Represents the operation of a sequence module. S4 is as follows: Fusion of multimodal sequence features Split in the time dimension to obtain video features All sentence features related to this video , , Indicates the first The video features are then split using the 2D-TAN method. After splitting, input to the maximum pooling operation to obtain candidate segment features , by the candidate fragment features Composition of candidate fragment group features , 2D-TAN stands for two-dimensional time domain connection network, which is a method used for video positioning. The calculation process is as follows: , , , in, Indicates The features of the candidate segments, Represents the video features at different positions after the video features are split. The video feature representing the starting position, The video feature indicating the end position, represents the maximum pooling operation, represents the total number of candidate segments, represents the index of the candidate segment, Indicates the number of sentences corresponding to the video, The operation represents a split on the channel dimension.

[0026] S5 is as follows: Candidate fragment features The candidate segment features are transformed in the channel dimension through grouped convolution The information between each group of fragments is learned by dividing them into 8 groups, 4 groups, and 2 groups respectively. Then, the information of all channels is aggregated by depthwise separable convolution with a convolution kernel of 1 to obtain new candidate fragment features. , the calculation process is as follows: , in, represents depthwise separable convolution, Represents 3 grouped convolutions with groups of 2, 4, and 8 respectively.

[0027] S6 is as follows: The contrastive learning loss enables the features of the two modalities, video and sentence, to be in a common embedding space. The calculation process is as follows: , in, represents the total number of video-natural language query sentence pairs used for training, and Respectively represent The contrastive learning scores obtained by taking the natural language query sentence as the standard and the The contrastive learning score obtained by taking candidate segments as the standard is Indicates Negative samples of natural language query sentences corresponding to candidate segments, Indicates Negative samples of candidate segments corresponding to natural language query sentences, Indicates candidate fragments, Indicates A natural language query sentence.

[0028] S7 is as follows: The new candidate segment features After convolution and The function obtains the prediction score of each candidate matrix , the process is as follows: , For the prediction score To constrain, by The prediction scores of candidate matrices The intersection over union (IoU) calculated for the video time segment corresponding to the candidate moment and the text in the video is approximated using the cross entropy loss as follows: , in, represents the number of valid candidate matrices, represents the index of the valid candidate matrix, Indicates The IoU of the video time segment corresponding to the candidate moments and the text in the video, Indicates The prediction scores of the candidate matrices.

[0029] Example 2 The present invention uses the pre-trained I3D model for the Charades-STA dataset (guessing word puzzle dataset), and uses the pre-trained C3D model to extract features of the TACoS dataset (kitchen dataset) and ActivityNet-Captions (video dataset). The experiments in this chapter are all run on a Titan RTX graphics card with 24G video memory, and the model is optimized using the AdamW optimizer. For the ActivityNet Captions, TACoS, and Charades-STA datasets, the learning rates are 8×10-4, 15×10-4, and 1×10-3, and the batch sizes are 12, 2, and 12, respectively. During the model training process, when the model reaches a bottleneck using contrast loss and cross entropy loss, the model will greatly reduce the weight of contrast loss in subsequent training, reducing it to 1 / 100 of the original. The contrastive learning parameter settings used in the experiment are consistent with the MMN algorithm; the number of sampled clips for the Charades-STA and ActivityNet Captions datasets is set to 64, while the number of samples in the TACoS dataset is set to 128; in order to reduce the amount of model calculation, the size of all hidden states in the model is set to 256; in the cross entropy loss process of model training, the thresholds of the Charades-STA and ActivityNet Captions datasets are set to 0.5 and 1.0, while those for the TACoS dataset are set to 0.3 and 0.7, which are consistent with the 2D-TAN algorithm.

[0030] The evaluation index mIoU is derived from target detection and is used to measure the similarity between two bounding boxes in the target detection field. It is obtained by calculating the IoU score of the label time boundary and the time boundary predicted by the model, and finally averaging the IoU of all sample pairs. The evaluation index "R@n, IoU=m" means that among all the prediction results in the current video and the natural language query sentence, if there is a prediction score greater than the set threshold m in the top n candidate segments with higher prediction scores, it is considered a correct prediction. Otherwise, the prediction result of the video and the natural language query sentence is considered an incorrect prediction. Then, the proportion of correct prediction results in all the prediction results is calculated.

[0031] During the test, the video is first subjected to an I3D or C3D model to extract features, and the natural language query sentence is encoded by a Bert model. The video and all queries related to the video are first spliced ​​in the feature dimension to form a multimodal feature, and then sent to the sequence model for learning intra-modal and inter-modal information. The video features that interact with the query are sampled according to the 2D-TAN algorithm proposed by Zhang et al. to obtain a set of video clip features; using contrastive learning, the video clip corresponding to the query and the query are used as positive samples, and other video clips of the same video that do not correspond and video clips of different videos are used as negative samples, so that video clips with the same semantics are closer to the query features, and video clips with different semantics are farther away from the query features. Then, the potential relationship between the video clips is learned, and there is a certain correlation between different video clips. Then, the matching score of the video clip features and the query features is calculated; The method in the present invention is compared with other methods on the Charades-STA, ActivityNet-Captions and TACoS datasets, and the experimental results are shown in Tables 1 and 2.

[0032] The comparison of the experimental results of the present invention with other methods on the Charades-STA dataset is shown in Table 1, the comparison of the experimental results of the present invention with other methods on the TACoS dataset is shown in Table 2, and the comparison of the experimental results of the present invention with other methods on the ActivityNet-Captions dataset is shown in Table 3.

[0033] Table 1 Comparison of the results of the method of the present invention and other methods on the Charades-STA dataset For the Charades-STA dataset, the results can be 57.80 and 35.73 on R@1,IoU0.5 and R@1,IoU0.7, 88.71 and 70.08 on R@1,IoU=0.5 and R@5,IoU=0.7, and 51.50 on mIoU.

[0034] Table 2 Comparison of the results of the method of the present invention and other methods on the TACoS dataset For the TACoS dataset, the results can be 62.38% and 47.69% on R@1,IoU0.5 and R@1,IoU0.7, the results can be 82.03% and 70.83% on R@5,IoU0.5 and R@5,IoU=0.7, and the result can be 43.89% on mIoU.

[0035] Table 3 Comparison of the results of the method of the present invention and other methods on the ActivityNet-Captions dataset For the ActivityNet-Captions dataset, the results can be 64.55% and 44.10% on R@1,IoU0.5 and R@1,IoU0.7, 86.88% and 76.42% on R@5,IoU0.5 and R@5,IoU=0.7, and 58.44% on mIoU.

[0036] Example 3 In order to highlight the advantages of the method in the present invention, the technical solution of the present invention is added to the benchmark model and compared with the results of the benchmark model, such as Figure 2 and Figure 3 As shown in FIG. 1 , given the search sentences, “Query: The person gets a plate out of the cupboard.” and “Query: The woman grabs a cuttingboard.”, the video interval where the sentences are located is located according to the given sentences, where ① represents the baseline model, ② represents the baseline model with the multimodal alignment module added, and ③ represents the method of the present invention. Figure 2 and Figure 3 It can be seen that compared with the real label, which is the actual position of the retrieved sentence, the baseline model adding the technical solution of the present invention can achieve a result that is more inclined to the correct answer.

[0037] Although the above describes the specific implementation mode of the invention in conjunction with the drawings, it is not intended to limit the scope of protection of the invention. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A video clip sentence localization algorithm based on progressive interaction and multimodal alignment, characterized by: The following steps are involved: S1. Determine the video to be sentence located and the sentence to be queried, extract video features from the video, and extract sentence features from the sentence to be queried; S2, preprocessing the extracted video features and sentence features, transforming the video features and sentence features into specified shapes, and obtaining processed video features and sentence features; S3. Align the features of the video modality and the sentence modality, concatenate the processed video features and sentence features in the channel dimension, and then perform an inverse operation on the concatenated features in the length dimension to obtain the reverse order features. Then, the forward order features that have not been inverted and the reverse order features that have been inverted are input into the dual-branch network to respectively mine the information between the video modality and the sentence modality to obtain the multimodal sequence fusion features. ; S4. Fusion features based on multimodal sequences Generate candidate segment group features ; S5. Group the candidate fragment features The information between video clips is learned in each channel through depthwise separable convolution, and then the information between each group is mined from coarse to fine through grouped convolution to obtain new candidate clip features. ; S6. By contrasting the learning loss function, the features of the two modalities, video and sentence, are placed in a common embedding space, thus completing the learning of the features of the video and sentence modalities. S7. Based on the characteristics of candidate fragments Get the prediction scores of the candidate video clips and constrain the prediction scores.

2. The video clip sentence localization algorithm based on progressive interaction and multimodal alignment according to claim 1 is characterized in that: S1 is as follows: The video to be sentence located contains multiple sentences, a sentence to be queried is given, and the position of the sentence to be queried in the video to be sentence located is determined according to the extracted sentence features and video features; Among them, the sentence to be queried is a natural language query sentence, through Interactive 3D models or The 3D convolutional model extracts video features from the video by The model extracts sentence features from the query sentence. The extracted video features are 512-dimensional for each segment and 768-dimensional for each sentence feature.

3. The video clip sentence localization algorithm based on progressive interaction and multimodal alignment according to claim 2 is characterized in that: S2 is as follows: The extracted video features and sentence features are subjected to convolution, pooling, and linear operations to transform the extracted video features and sentence features into specified shapes, thereby obtaining the processed video features and sentence features. The calculation process is as follows: ， ， in, represents the extracted video features, represents the extracted sentence features, and Represent the processed video features and sentence features respectively. and Aggregates global timing information, , and Represent the average pooling operation, activation function and convolution operation respectively. represents the learnable projection matrix, represents the learnable offset parameter.

4. The video clip sentence localization algorithm based on progressive interaction and multimodal alignment according to claim 3 is characterized in that: S3 is as follows: The processed video features and sentence features are concatenated in the feature dimension to obtain features , the calculation process is as follows: ， ， in, represents the feature concatenation operation, Indicates the number of sentences corresponding to the video, Indicates Features of the processed sentences; Then the features after splicing Perform inverse operation on the length dimension to obtain the reverse feature , features that have not been inverted is the positive sequence feature, and then the positive sequence feature and reverse order features Input to the dual-branch network, the dual branches include forward branch and reverse branch, specifically the positive sequence feature Input to the forward branch, and reverse the features Input to the reverse branch; In the forward branch, a dual branch is first constructed again through the gating mechanism, one branch is used to calculate the positive sequence feature The weight matrix of the information in the middle, and the other branch is used to capture the positive sequence features Information in, positive sequence characteristics After a linear layer, the positive sequence features The dimension is projected onto Dimensions, represents the dimension of the extracted video features, , the positive sequence feature The feature dimension is divided into two equal feature channels. Features and Features , the feature and Features They are used as the input of the dual branches of the gating mechanism respectively; the operations in the forward branch and the reverse branch are the same, and the dual branch input features of the gating mechanism in the reverse branch are obtained in the same way and Features , the calculation process is as follows: ， )， in, and denote the learnable projection matrices in the forward branch and the reverse branch, respectively. and denote the learnable offset parameters in the forward branch and the reverse branch, respectively. Represents the operation of dividing the feature into two features according to the number of channels in the feature dimension.

5. The video clip sentence localization algorithm based on progressive interaction and multimodal alignment according to claim 4 is characterized in that: The process within the branch in the gating mechanism is as follows: For the two branches of the gating mechanism in the forward branch, the features Enter a branch, through The activation function obtains the positive sequence features The weight matrix of the information , the feature Input to another branch, feature First, it passes through a one-dimensional convolution with a convolution kernel of 4, and then passes through Module alignment characteristics The intra-modal and inter-modal information of the two modalities of video and sentence are mined. Modules by , and The three frameworks are integrated, and finally the two branch results are multiplied pixel by pixel to obtain the positive multimodal sequence features. , The framework is an open source platform, The framework is a model view controller framework. A framework is a Persistence layer framework; similarly, the reverse order feature is obtained The weight matrix of the information and reverse multimodal sequence features ; The multimodal sequence features in two directions are added and passed through a linear layer to obtain the multimodal sequence fusion feature , The feature dimension is , the calculation process is as follows: ， ， ， in, Represents the activation function Operation, represents one-dimensional convolution, represents a linear layer, Represents the operation of a sequence module.

6. The video segment sentence localization algorithm based on progressive interaction and multimodal alignment according to claim 5 is characterized in that: S4 is as follows: Fusion of multimodal sequence features Split in the time dimension to obtain video features All sentence features related to this video , , Indicates the first The video features are then split using the 2D-TAN method. After splitting, input to the maximum pooling operation to obtain candidate segment features , by the candidate fragment features Composition of candidate fragment group features , 2D-TAN stands for two-dimensional time domain connection network, which is a method used for video positioning. The calculation process is as follows: ， ， ， in, Indicates The features of the candidate segments, Represents the video features at different positions after the video features are split. The video feature representing the starting position, The video feature indicating the end position, represents the maximum pooling operation, represents the total number of candidate segments, represents the index of the candidate segment, Indicates the number of sentences corresponding to the video, The operation represents a split on the channel dimension.

7. The video clip sentence localization algorithm based on progressive interaction and multimodal alignment according to claim 6 is characterized in that: S5 is as follows: Candidate fragment features The candidate segment features are transformed in the channel dimension through grouped convolution The information between each group of fragments is learned by dividing them into 8 groups, 4 groups, and 2 groups respectively. Then, the information of all channels is aggregated by depthwise separable convolution with a convolution kernel of 1 to obtain new candidate fragment features. , the calculation process is as follows: ， in, represents depthwise separable convolution, Represents 3 grouped convolutions with groups of 2, 4, and 8 respectively.

8. The video segment sentence localization algorithm based on progressive interaction and multimodal alignment according to claim 7, characterized in that S6 The details are as follows: The contrastive learning loss enables the features of the two modalities, video and sentence, to be in a common embedding space. The calculation process is as follows: ， in, represents the total number of video-natural language query sentence pairs used for training, and Respectively represent The contrastive learning scores obtained by taking the natural language query sentence as the standard and the The contrastive learning score obtained by taking candidate segments as the standard is Indicates Negative samples of natural language query sentences corresponding to candidate segments, Indicates Negative samples of candidate segments corresponding to natural language query sentences, Indicates candidate fragments, Indicates A natural language query sentence.

9. The video clip sentence localization algorithm based on progressive interaction and multimodal alignment according to claim 8, characterized in that: S7 is as follows: The new candidate segment features After convolution and The function obtains the prediction score of each candidate matrix , the process is as follows: ， For the prediction score To constrain, by The prediction scores of candidate matrices The intersection over union (IoU) calculated for the video time segment corresponding to the candidate moment and the text in the video is approximated using the cross entropy loss as follows: ， in, represents the number of valid candidate matrices, represents the index of the valid candidate matrix, Indicates The IoU of the video time segment corresponding to the candidate moments and the text in the video, Indicates The prediction scores of the candidate matrices.

Images