A natural language temporal localization method based on sparse candidate boxes

Abstract

The application provides a natural language timing positioning method based on sparse candidate frames, comprising the following steps: S1, obtaining original images of each sampling frame and obtaining discrete text token representation; S2, using a video encoder and a text encoder to obtain feature expression of video and text modalities; S3, after obtaining the feature expression of the video and text modalities, cross-modality feature fusion is performed to build the correlation between the video and language content, and cross-modality features are generated; S4, sparse candidate frame generation; S5, appearance feature modeling; S6, position feature modeling: introducing position features as explicit prior information to adjust the update of learnable timing candidate frames; S7, for each candidate frame, cascade the original candidate frame features with the appearance and position features to obtain the final candidate frame features; S8, classification and positioning: using a set loss function to train the network. The method further improves the timing behavior positioning performance.

Description

Technical Field

[0001] This invention relates to the fields of visual perception and artificial intelligence, and in particular, to a natural language temporal localization method based on sparse candidate boxes. Background Technology

[0002] The Natural Language Processing (NLP) temporal behavior localization task aims to locate the temporal position of a target behavior in undressed video based on query text. This task has significant application value in areas such as streaming media retrieval, intelligent video creation, and intelligent analysis of surveillance videos, thus becoming a research hotspot in computer vision and multimedia analysis.

[0003] In recent years, with the continuous development of deep learning technology and the emergence of numerous large-scale video datasets, the task of natural language temporal behavior localization has received increasing attention from industry and academia. Currently, mainstream deep learning-based natural language temporal behavior localization tasks are mainly divided into two categories: top-down methods and bottom-up methods.

[0004] Top-down methods first extract temporal candidate boxes from the input video, then perform cross-modal matching between these candidate boxes and the query text to generate localization results. Bottom-up methods directly predict the start and end boundaries of the target time. These methods can be divided into two types: methods based on temporal boundary regression and methods based on temporal probability distributions. Methods based on temporal boundary regression calculate the regression value of the predicted temporal interval relative to the ground truth interval to supervise network training. Methods based on temporal probability distributions aim to predict the probability distribution of each video frame as the start and end position of the target time.

[0005] Currently, most bottom-up methods still lag behind top-down methods in temporal behavior localization, especially on long video datasets (such as TACoS). Furthermore, top-down methods are intuitively more similar to human reasoning. To perform temporal behavior localization, people typically first perceive salient time segments in a video and then match them with a query. Therefore, this invention focuses on improving the performance and efficiency of top-down temporal behavior localization methods.

[0006] Current top-down temporal behavior localization methods use either sliding window-based or anchor-box-based strategies to generate temporal candidate boxes. The former explicitly segments the video using a predefined temporal sliding window, while the latter assigns multi-scale temporal anchor boxes to each video frame.

[0007] The main problems with existing technologies are as follows: 1) They rely on densely distributed temporal candidate boxes. To achieve high recall and maximum coverage, current methods require a large number of temporal candidate boxes to cover different temporal locations and scales, leading to significant computational redundancy; 2) They lack semantic information. Specifically, a temporal candidate box contains two aspects of semantic information: appearance information and location information. Appearance information describes the action content within the candidate box, while location information can be used to distinguish candidate boxes that are similar in appearance but different in temporal location. However, current methods neglect to utilize these two types of information.

[0008] The difficulties in solving the above problems are: 1) Modeling the temporal relationships between video frames. Different types of actions in a video change at different speeds and have varying durations. Furthermore, the camera's shooting angle and its own motion also pose challenges to recognition. In addition, redundant information in the video presents a significant challenge to video understanding; 2) The lack of large-scale, universally applicable video benchmark datasets. Compared to the ImageNet dataset in image classification, which contains 1.28 million data points, the Kinetics and UCF-101 datasets in video understanding are relatively small, which restricts the development of this field.

[0009] The challenges in solving these problems are: 1) promoting multimodal understanding and interaction, improving semantic alignment and information fusion capabilities between video and language, and providing a foundation and support for other video understanding tasks; 2) expanding the application scenarios of video analysis, such as video summarization, video editing, video question answering, and video recommendation, to enhance the value and functionality of video understanding. Summary of the Invention

[0010] This invention provides a natural language temporal localization method based on sparse candidate boxes, which solves the problems of high redundancy in candidate box generation and unclear semantic information in current natural language temporal localization methods.

[0011] The technical solution of the present invention is as follows:

[0012] The present invention relates to a natural language temporal behavior localization method based on sparse candidate boxes. 1. A natural language temporal behavior localization method based on sparse candidate boxes, comprising the following steps: S1. Sparsely sampling a video sequence according to sampling coefficient N to obtain the original image of each sampled frame; serializing the text to obtain a discretized text token representation; S2. Obtaining feature representations of the video and text modalities using a video encoder and a text encoder; S3. After obtaining the feature representations of the video and text modalities, performing cross-modal feature fusion to construct the correlation between the video and language content, and generating cross-modal features; S4. Sparse candidate box generation; S5. Appearance feature modeling; S6. Position feature modeling: introducing position features as explicit prior information to regulate the update of learnable temporal candidate boxes; S7. For each candidate box, concatenating the original candidate box features with appearance and position features to obtain the final candidate box features; S8. Classification and localization: training the network using an ensemble loss function.

[0013] Optionally, in the above natural language temporal behavior localization method based on sparse candidate boxes, in step S2, the original images of each sampled frame are processed using a convolutional neural network to obtain frame-level feature vectors representing each sampled frame, i.e., video features; features are extracted from the discretized text tokens using a Transformer to obtain text features, wherein the video features are represented as follows: Text features are represented as

[0014] Optionally, in the above natural language temporal behavior localization method based on sparse candidate boxes, in step S3, for each frame in the video, its feature similarity with all words is calculated, and normalized using softmax:

[0015] e t,s =softmax(W e tanh(W v v t +W s q s )),

[0016] Among them W e W v W s For learnable parameters,

[0017] The fused cross-modal feature vector is then expressed as:

[0018] The correlation is calculated between each frame and each word, and the correlation e in the formula is... t,s This is the cross-modal relevance score between the t-th frame of video and the s-th word.

[0019] Optionally, in the above natural language temporal behavior localization method based on sparse candidate boxes, in step S4, N temporal candidate boxes are set based on the cross-modal representation generated in S3. in Let f represent the center position and length of the normalized temporal candidate box, respectively. Then, the corresponding sparse candidate box features f are obtained through region-of-interest pooling. n :

[0020] f n =maxpool(u,φ) n ).

[0021] Optionally, in the above natural language temporal behavior localization method based on sparse candidate boxes, in step S5, the cosine similarity between the sparse candidate box features and the features of each concept is calculated, and then the K concepts with the highest responses are selected as matching terms, i.e., appearance concepts.

[0022]

[0023] In obtaining the relevant appearance concept Then, calculate its features f with the sparse candidate boxes. n The correlation between these factors forms the final appearance characteristics:

[0024]

[0025] Optionally, in the above natural language temporal behavior localization method based on sparse candidate boxes, in step S6, the center coordinates of the sparse candidate boxes are used as the basis for determining the behavior. and length As an input suggestion,

[0026]

[0027] Where sinu represents the sinusoidal position coding function.

[0028] Optionally, in the above natural language temporal behavior localization method based on sparse candidate boxes, the set loss function in step S8 is as follows:

[0029]

[0030] in Let λ be the crossover ratio loss function. giou For the corresponding coefficients; The cross-entropy loss function is:

[0031]

[0032] λ loc Using the smoothed L1 loss function:

[0033]

[0034] The beneficial effects of the technical solution of the present invention are as follows:

[0035] This invention generates sparse temporal candidate boxes in a learnable manner and combines them with corresponding semantic information to refine candidate box features. Without relying on predefined dense candidate boxes, the method of this invention achieves state-of-the-art performance using fewer candidate boxes. Furthermore, by introducing semantic information into the candidate box generation process and updating the candidate box representation using prior information, the temporal behavior localization performance is further improved.

[0036] To better understand and illustrate the concept, working principle, and effects of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments: Attached Figure Description

[0037] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below.

[0038] Figure 1 This is a flowchart of the natural language temporal localization method based on sparse candidate boxes according to the present invention;

[0039] Figure 2 This is a flowchart illustrating the specific operation of a preferred embodiment of the natural language temporal localization method based on sparse candidate boxes of the present invention.

[0040] Figure 3 This is a comparison between the method of this invention and current state-of-the-art methods on the TACOS dataset. Detailed Implementation

[0041] To make the objectives, technical methods, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific examples. These examples are merely illustrative and not intended to limit the scope of the invention.

[0042] like Figure 1 and Figure 2 As shown, the natural language temporal behavior localization method based on sparse candidate boxes of the present invention includes the following steps:

[0043] S1. Perform sparse sampling on the video sequence according to the sampling coefficient N to obtain the original image of each sampled frame, and perform serialization processing on the text to obtain a discretized text token representation.

[0044] S2. Obtain feature representations of video and text modalities using a video encoder and a text encoder. Specifically, the original images of each sampled frame are processed using convolutional neural networks of the video encoder and text encoder to obtain frame-level feature vectors representing each sampled frame, i.e., video features; features are extracted from the discretized text tokens using a Transformer to obtain text features, where the video features are represented as follows: Text features are represented as

[0045] S3. After obtaining the feature representations of video and text modalities, perform cross-modal feature fusion to construct the correlation between video and language content and generate cross-modal features.

[0046] Specifically, for each frame in the video, its feature similarity to all words is calculated, and then normalized using softmax:

[0047] e t,s =softmax(W e tanh(W v v t +W s q s )),

[0048] Among them W e W v W s These are learnable parameters.

[0049] The fused cross-modal feature vector is represented as follows:

[0050]

[0051] The correlation is calculated between each frame and each word, and the correlation e in the formula is... t,s This is the cross-modal relevance score between the t-th frame of video and the s-th word.

[0052] S4. Sparse Candidate Box Generation: Based on the cross-modal features generated in S3, N temporal candidate boxes are set. in These represent the center position and length of the normalized temporal candidate boxes, respectively. This is achieved through Region of Interest Pooling (RoIPooling)... Figure 2 The sparse candidate box features f are obtained by using max pooling in the algorithm. n :

[0053] f n =maxpool(u,φ) n ).

[0054] S5. Appearance feature modeling.

[0055] The method of this invention defines the key concepts in a video as objects, actions, and relationships, which are characterized as follows: To eliminate some redundant concepts, the method of this invention performs concept filtering, such as appearance concepts. Figure 2 The keywords are "woman, horse, shirt, wearing, tree, brushing". Specifically, the cosine similarity between the candidate box features and each concept feature is calculated, and then the K concepts with the highest responses (K is an integer of 3 in the experiment) are selected as the matching terms, i.e., appearance concepts.

[0056]

[0057] In obtaining the relevant appearance concept Then, calculate its features f with the sparse candidate boxes. n The correlation between these factors forms the final appearance characteristics:

[0058]

[0059]

[0060] S6. Location Feature Modeling. Location features are introduced as explicit prior information to regulate the update of learnable temporal candidate boxes. Specifically, the center coordinates of sparse candidate boxes are used... and length (in, Using the concept of location as input cues, the location feature of the nth learnable candidate box is represented as:

[0061]

[0062] Where sinu represents the sinusoidal position coding function.

[0063] S7. For each candidate box, concatenate the original candidate box features with the appearance and location features to obtain the final sparse candidate box features.

[0064]

[0065] S8. Classification and Localization. The integrated network constructed from the above steps is trained using an ensemble loss function, as follows:

[0066]

[0067] in Let cross-entropy be the loss function. The generalized intersection-over-union (CUI) loss function is λ. giou For the corresponding coefficients:

[0068]

[0069] λ loc Using the smoothed L1 loss function:

[0070]

[0071] like Figure 3 As shown, the method of this invention achieves better results on the TACoS dataset compared with current state-of-the-art methods (2DTAN, SCDM, ACRN and CTRL). Specifically, it reduces the number of candidate boxes required to 1 / 250 while improving the R@1, IoU=0.5 metric by 4.11%.

[0072] Therefore, the method of the present invention adaptively generates sparse temporal candidate boxes with rich semantic information, thereby improving the speed and efficiency of temporal localization while ensuring the accuracy of natural language temporal behavior localization.

[0073] The above description represents the preferred embodiment based on the inventive concept and working principle. The above embodiments should not be construed as limiting the scope of protection of these claims; other embodiments and combinations of implementations based on the inventive concept are all within the scope of protection of this invention.

Claims

1. A natural language temporal behavior localization method based on sparse candidate boxes, characterized in that, Includes the following steps: S1. Perform sparse sampling on the video sequence according to the sampling coefficient N to obtain the original image of each sampled frame, and perform serialization processing on the text to obtain a discretized text token representation. S2. Obtain feature representations of video and text modalities using video encoders and text encoders; In step S2, the original images of each sampled frame are processed using a convolutional neural network to obtain frame-level feature vectors, i.e., video features, representing each sampled frame. The Transformer is used to extract features from the discretized text tokens to obtain text features, where video features are represented as follows. The text features are represented as ; S3. After obtaining the feature representations of the video and text modalities, cross-modal feature fusion is performed to construct the correlation between the video and language content and generate cross-modal features; In step S3, for each frame in the video, its feature similarity with all words is calculated, and then normalized using softmax: in , For learnable parameters, The fused cross-modal feature vector is then expressed as: , The correlation is calculated between each frame and each word, and the correlation in the formula is... This is the cross-modal relevance score between the t-th video frame and the s-th word; S4. Sparse candidate box generation; In step S4, based on the cross-modal representation generated in S3, the following settings are made: N One time series candidate box ,in Let represent the center position and length of the normalized temporal candidate boxes, respectively. Then, the corresponding sparse candidate box features are obtained through region-of-interest pooling. : ； S5. Appearance feature modeling; In step S5, the cosine similarity between the sparse candidate box features and each concept feature is calculated, and then the highest response is selected. K One concept is used as a matching item, namely, the appearance concept: In obtaining relevant appearance concepts Then, its features are calculated in relation to the sparse candidate boxes. The correlation between these factors forms the final appearance characteristics: ； S6. Location feature modeling: Introduce location features as explicit prior information to regulate the update of learnable temporal candidate boxes; In step S6, the coordinates of the center of the sparse candidate box are used. and length As an input suggestion, in Represents a sinusoidal position coding function; S7. For each candidate box, concatenate the original candidate box features with the appearance and position features to obtain the final candidate box features; S8. Classification and Localization: Train the network using the ensemble loss function; In step S8, the ensemble loss function is as follows: in, , The cross-entropy loss function is: Use the smoothed L1 loss function.

Images