Open vocabulary space-time action detection method and device based on visual language pre-training

Abstract

The invention provides an open vocabulary space-time action detection method and device based on visual language pre-training, and relates to the technical field of space-time action detection. The method comprises the following steps: acquiring video data to be detected, and inputting the video data to a local branch to obtain human-related region features; the local branch comprises a human-centered and category-independent pipe detector and a position sensing region analyzer; inputting the video data into the global branch to obtain global video features; the global branch comprises a visual language pre-training model of a video level; and fusing the regional features related to the person and the global video features, and calculating the similarity between the fused video regional features and the text features to obtain a classification result of behavior categories. A wide range of experiments on a UCF-JHMDB data set show that the performance of the method is superior to that of an existing method. A feasible solution is provided for solving the problem of insufficient open recognition capability, and a new direction and thought are provided for further research on space-time action detection.

Description

Technical Field

[0001] The present invention relates to the technical field of spatiotemporal motion detection, and in particular to an open vocabulary spatiotemporal motion detection method and device based on visual language pre-training. Background Art

[0002] STAD (Spatio-temporal Action Detection) is a key task in the field of video understanding. Its main purpose is to accurately identify and locate specific actions in the input video clips. Spatio-temporal action detection not only locates the spatial position of the action subject, but also predicts the start and end time of the specific action and the category of the action completed by the action subject. When applied to the real world, the types of behavior are often dynamically changing. It is not feasible to predefine all categories of things in the world. The model needs to have good generalization ability to make it possible to recognize and understand behaviors in new classes or new scenes that have not been seen. OV-STAD (Open-vocabulary Spatio-temporal Action Detection) is a new setting for spatio-temporal action detection, which is trained in the base class and tested in the new class that has not been seen. Simulating the application of spatio-temporal action detection algorithm in the real world tests whether the model can learn generalized behavior characteristics from limited annotated data and can effectively adapt to new behavior categories caused by environmental changes. The successful application of this task covers multiple fields, including smart security, smart rehabilitation, smart driving and smart sports, and has promoted the widespread application of video understanding technology in practical applications.

[0003] At present, spatiotemporal action detection algorithms can be divided into two categories: a two-stage detection method based on frame level and a single-stage detection method based on tube level.

[0004] (1) Two-stage spatiotemporal action detection method based on frame level: First, the target recognition algorithm is applied to identify the action subject in the video frame by frame, and then the detected human bounding boxes are connected into a series of action sequences through the association connection algorithm. Finally, these action sequences are further processed and classified to determine the specific action. This method decomposes the complex spatiotemporal action detection problem into two simpler subtasks: one is human object detection in the spatial domain, and the other is cross-temporal information extraction and action recognition. This reduces the processing complexity, but it should also be recognized that a single static image is difficult to fully convey the semantics of the action, because the video is the result of combining continuous images in time order, and the time dimension is crucial for showing scene changes and individual activities. Therefore, how to efficiently capture time series information has become the core challenge of this type of method research. According to the different strategies for processing time series information, frame-level spatiotemporal action detection methods can be divided into three categories: methods combined with optical flow, methods using three-dimensional convolutional neural networks, and methods based on recurrent neural networks. These methods have achieved remarkable results in the field of spatiotemporal action detection.

[0005] (2) Single-stage detection method based on tube level: locate the spatial position of the action subject and predict the start and end time of the action and its behavior category from the entire video sequence, without the need for an explicit action proposal generation stage. Compared with the two-stage method, the single-stage method integrates candidate generation and classification into a single deep learning model, and directly predicts the spatial position of the action subject and the temporal position of the action and the corresponding behavior category by directly modeling the entire video sequence. It is generally processed in a top-down manner, and its general process is often to first obtain a preliminary candidate frame sequence (i.e., action tube), then perform precise regression on each frame, and finally perform regression classification to obtain the final result.

[0006] However, both of the above methods can only detect and classify predefined behavior categories, and cannot classify unseen behavior categories. That is, it is necessary to determine and train the model to recognize specific action types in advance. Because the actions in real life are numerous and changeable, far beyond the scope of any predefined categories, they cannot be truly applied to the real world.

[0007] Spatiotemporal action detection is a difficult and challenging video understanding task, which aims to locate the spatial position of the action subject in the untrimmed video, define the temporal boundaries of the action category, and predict the action category completed by the classified action subject. The current problem of STAD is that it lacks open capabilities and almost all relies on pre-set behavior categories for training. Although the existing methods have good performance on closed categories through optimization of model architecture and improvement of training equipment capabilities, most methods have poor recognition capabilities for new categories that have not been seen before and cannot be applied to the real physical world.

[0008] 1) Insufficient open recognition capability: Open recognition capability tests whether the model can learn generalized behavioral features from limited annotated data and can effectively adapt to new behavioral categories or environmental changes. The current problem with STAD is that it lacks open recognition capability and almost all of it relies on pre-set behavioral categories for training. This means that they can only recognize action types that have appeared in the training dataset, and cannot cope with unseen or new behaviors that are widely present in real scenarios. This limitation greatly limits the flexibility and adaptability of existing methods in practical applications.

[0009] 2) Insufficient regional recognition capability: The visual language pre-training model uses contrastive learning to map videos and texts into a shared feature space under the training of a large number of video-text pairs. It has demonstrated strong zero-sample capabilities in many downstream visual tasks, making it possible to achieve open spatiotemporal action detection. With the help of the conceptual knowledge learned from massive image-text pair data by the video-level visual language pre-training model, an open concept space is constructed, and the model is fine-tuned and migrated to downstream perception tasks through a small amount of downstream task training data. The model can locate and identify related entities of different granularities in the video according to arbitrary descriptions. However, the visual language pre-training model is trained on complete video and text pairs, while the open vocabulary spatiotemporal action detection task is a regional-level recognition task within a certain spatial range of the action subject. There are differences between the two. Directly applying the visual language pre-training model to the open vocabulary recognition task can only achieve limited accuracy. Summary of the invention

[0010] In order to solve the problem that existing spatiotemporal action detection mostly focuses on optimizing the recognition and positioning of closed action categories (i.e., predefined, limited number of behavior types). These methods can provide relatively accurate results in specific scenarios, such as sports event analysis or common behavior monitoring in surveillance environments. However, their original design and training methods determine the limitations of their application scope. They do not fully consider the complexity and diversity of the real world and cannot cope with the technical problems of the complex and diverse behavior categories in the real world. The embodiments of the present invention provide an open vocabulary spatiotemporal action detection method and device based on visual language pre-training. The technical solution is as follows:

[0011] In one aspect, a method for open vocabulary spatiotemporal action detection based on visual language pre-training is provided, the method being implemented by an open vocabulary spatiotemporal action detection device, the method comprising:

[0012] S1. Obtain video data to be detected.

[0013] S2: Input the video data into the local branch and extract the regional features related to people.

[0014] Among them, the local branch includes a human-centric category-independent tube detector (HCTD) and a position-aware region parser (PRP).

[0015] S3: Input the video data into the global branch to extract the global video features.

[0016] Among them, the global branch includes the video-level visual language pre-training model ViCLIP.

[0017] S4. Fusing the regional features related to people and the global video features to obtain fused video regional features.

[0018] S5. Calculate the similarity between the fused video region features and the text features, and obtain the classification result of the behavior category according to the similarity.

[0019] Optionally, in S2, the video data is input into a local branch to extract regional features related to people, including:

[0020] S21. Input the video data into a video feature extractor to extract video features.

[0021] S22, input the video features into the human-centric category-independent tube detector HCTD, and output the action tube.

[0022] S23: Input the action pipe into the position-aware region parser PRP to extract the region features related to the person.

[0023] Optionally, in S22, the video features are input into a human-centric category-independent tube detector HCTD, and the action tube is output, including:

[0024] The video features are input into the spatiotemporal self-attention layer to obtain the spatiotemporal self-attention layer output.

[0025] The output of the spatiotemporal self-attention layer is input into the spatiotemporal cross-attention layer, and the spatial position of the action subject in the video is located through a learnable query in the temporal dimension, and an action tube is output; the action tube consists of bounding box links of the action subject between video frames.

[0026] Optionally, in S23, the action pipe is input to a position-aware region parser PRP to extract region features related to people, including:

[0027] S231, select a key frame of the video, and use layers 1-3 of the CLIP image encoder to extract the entire image features of the key frame.

[0028] S232: for the whole image features of the key frame, the region features corresponding to the position of the action subject in the key frame are obtained by using the region of interest alignment and the action tube.

[0029] S233. Input the regional features corresponding to the position of the action subject into the 4th layer of the CLIP image encoder to obtain the encoder output.

[0030] S234: Input the encoder output to the position-aware region parser PRP to obtain region features related to people.

[0031] Among them, learnable position embedding is introduced in the CLIP image encoder.

[0032] Optionally, a position-aware region parser PRP comprises: a position-aware feature alignment layer and a local-global feature adapter.

[0033] The local-global feature adapter includes a first linear layer, an activation function and a second linear layer.

[0034] Optionally, in S4, the regional features related to people and the global video features are fused to obtain fused video regional features, including:

[0035] Through the multi-layer perceptron MLP, the regional features related to people are mapped to the feature dimensions of the global video features and fused with the global video features to obtain the fused video regional features.

[0036] Optionally, the loss of the human-centric category-independent tube detector HCTD includes: a first binary classification loss function, a second binary classification loss function, a GIoU loss, and an L1 loss.

[0037] The first two-class loss function is used to supervise the matching between the query and the target, ensuring that the model can accurately assign the detection query to the correct target instance.

[0038] The second binary classification loss function is used to supervise the existence of frame-level actions and help the model identify whether there is a specific action in each frame, that is, to determine whether the current frame is within the time boundary.

[0039] GIoU loss and L1 loss work together to optimize the accuracy of bounding box prediction.

[0040] On the other hand, a device for detecting an open vocabulary spatiotemporal action based on visual language pre-training is provided, and the device is applied to a method for detecting an open vocabulary spatiotemporal action based on visual language pre-training, and the device comprises:

[0041] The acquisition module is used to acquire the video data to be detected.

[0042] The local feature extraction module is used to input video data into the local branch and extract regional features related to people.

[0043] Among them, the local branch includes a human-centric category-independent tube detector (HCTD) and a position-aware region parser (PRP).

[0044] The global feature extraction module is used to input video data into the global branch and extract global video features.

[0045] Among them, the global branch includes the video-level visual language pre-training model ViCLIP.

[0046] The fusion module is used to fuse the regional features related to people and the global video features to obtain fused video regional features.

[0047] The output module is used to calculate the similarity between the fused video region features and the text features, and obtain the classification results of the behavior category based on the similarity.

[0048] Optionally, the local feature extraction module is further used to:

[0049] S21. Input the video data into a video feature extractor to extract video features.

[0050] S22, input the video features into the human-centric category-independent tube detector HCTD, and output the action tube.

[0051] S23: Input the action pipe into the position-aware region parser PRP to extract the region features related to the person.

[0052] Optionally, the local feature extraction module is further used to:

[0053] The video features are input into the spatiotemporal self-attention layer to obtain the spatiotemporal self-attention layer output.

[0054] The output of the spatiotemporal self-attention layer is input into the spatiotemporal cross-attention layer, and the spatial position of the action subject in the video is located through a learnable query in the temporal dimension, and an action tube is output; the action tube consists of bounding box links of the action subject between video frames.

[0055] Optionally, the local feature extraction module is further used to:

[0056] S231, select a key frame of the video, and use layers 1-3 of the CLIP image encoder to extract the entire image features of the key frame.

[0057] S232: for the whole image features of the key frame, the region features corresponding to the position of the action subject in the key frame are obtained by using the region of interest alignment and the action tube.

[0058] S233. Input the regional features corresponding to the position of the action subject into the 4th layer of the CLIP image encoder to obtain the encoder output.

[0059] S234: Input the encoder output to the position-aware region parser PRP to obtain region features related to people.

[0060] Among them, learnable position embedding is introduced in the CLIP image encoder.

[0061] Optionally, a position-aware region parser PRP comprises: a position-aware feature alignment layer and a local-global feature adapter.

[0062] The local-global feature adapter includes a first linear layer, an activation function and a second linear layer.

[0063] Optionally, the fusion module is further configured to:

[0064] Through the multi-layer perceptron MLP, the regional features related to people are mapped to the feature dimensions of the global video features and fused with the global video features to obtain the fused video regional features.

[0065] Optionally, the loss of the human-centric category-independent tube detector HCTD includes: a first binary classification loss function, a second binary classification loss function, a GIoU loss, and an L1 loss.

[0066] The first two-class loss function is used to supervise the matching between the query and the target, ensuring that the model can accurately assign the detection query to the correct target instance.

[0067] The second binary classification loss function is used to supervise the existence of frame-level actions and help the model identify whether there is a specific action in each frame, that is, to determine whether the current frame is within the time boundary.

[0068] GIoU loss and L1 loss work together to optimize the accuracy of bounding box prediction.

[0069] On the other hand, an open vocabulary spatiotemporal action detection device is provided, the open vocabulary spatiotemporal action detection device comprising: a processor; a memory, the memory having computer-readable instructions stored thereon, the computer-readable instructions, when executed by the processor, implementing any one of the above-mentioned open vocabulary spatiotemporal action detection methods based on visual language pre-training.

[0070] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned open vocabulary spatiotemporal action detection methods based on visual language pre-training.

[0071] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0072] In the present invention, a video-level visual language pre-training model ViCLIP is used to solve the problem of insufficient open recognition ability of existing spatiotemporal action detection models. ViCLIP is trained on a limited number of base classes to adapt it to the open vocabulary spatiotemporal action detection task.

[0073] We design a position-aware two-stage detector that decomposes the problem into localization followed by classification. This approach does not require additional large-scale pre-training of region-text pairs and bridges the gap between ViCLIP trained on the entire video and region-based recognition tasks.

[0074] We design a human-centric category-independent tube detector (HCTD) that extracts spatiotemporal action representations and predicts category-independent action detection tubes. We also design a position-aware region parser (PRP) that improves region classification results by optimizing position-aware embeddings and local-global feature adapters. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0076] Figure 1 is a flow chart of an open vocabulary spatiotemporal action detection method based on visual language pre-training provided by an embodiment of the present invention;

[0077] Figure 2 It is a local branch architecture diagram provided by an embodiment of the present invention;

[0078] Figure 3 is a global branch architecture diagram provided by an embodiment of the present invention;

[0079] Figure 4 It is a general model architecture diagram provided by an embodiment of the present invention;

[0080] Figure 5 It is the result of the model provided by the embodiment of the present invention on the UCF-JHMDB benchmark test set;

[0081] Figure 6 It is a visualization comparison between the present invention and ViCLIP on the JHMDB data set provided by an embodiment of the present invention;

[0082] Figure 7 It is a block diagram of an open vocabulary spatiotemporal action detection device based on visual language pre-training provided by an embodiment of the present invention;

[0083] Figure 8 It is a schematic diagram of the structure of an open vocabulary spatiotemporal action detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0084] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0085] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0086] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0087] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0088] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0089] The embodiment of the present invention provides an open vocabulary spatiotemporal action detection method based on visual language pre-training, which can be implemented by an open vocabulary spatiotemporal action detection device, which can be a terminal or a server. Figure 1 The flowchart of the open vocabulary spatiotemporal action detection method based on visual language pre-training is shown in the figure. The processing flow of the method may include the following steps:

[0090] S1. Obtain video data to be detected.

[0091] The video data may be a video clip of a human body action.

[0092] Furthermore, the present invention designs a simple and effective method for regional modeling of video features, accurately captures the spatial position information corresponding to the action subject, and accurately analyzes the regional information of the captured spatial position, thereby improving the regional feature extraction ability of the model. It is divided into two modules: the local branch includes HCTD (Human-centric Class-agnostic Tube Detector) and PRP (Position-aware Region Parser), such as Figure 2 shown.

[0093] S2: Input the video data into the local branch and extract the regional features related to people.

[0094] In a feasible implementation, the local branch is used to capture regional features related to people from video frames, and more accurately reflect the human-centered action features.

[0095] Optionally, the above step S2 may include the following steps S21-S23:

[0096] S21. Input the video data into a video feature extractor to extract video features.

[0097] Specifically, given an input video sequence, video features are extracted using the video feature extractor CSN-152.

[0098] S22, input the video features into the human-centric category-independent tube detector HCTD, and output the action tube.

[0099] Optionally, the above step S22 may include the following steps S221-S222:

[0100] S221. Input the video features into the spatiotemporal self-attention layer to obtain the spatiotemporal self-attention layer output.

[0101] S222. Input the output of the spatiotemporal self-attention layer into the spatiotemporal cross-attention layer, and locate the spatial position of the action subject in the video through a learnable query in the time dimension, and output an action tube; the action tube is composed of bounding box links of the action subject between video frames.

[0102] In one possible implementation, a human-centric class-independent tube detector learns video features extracted by the video feature extractor CSN-152 through spatiotemporal self-attention encoding, then locates human instances in the video through learnable queries in the temporal dimension, and outputs action tubes consisting of action subject bounding box links between video frames. These detection results serve as spatial-temporal anchors for the subsequent classification stage.

[0103] Specifically, the human-centric category-independent tube detector (HCTD) will frame the person in each frame of the video, that is, output the bounding boxes of the person in each frame of the image. The bounding boxes of each frame of the video can be connected into an action tube.

[0104] S23: Input the action pipe into the position-aware region parser PRP to extract the region features related to the person.

[0105] Optionally, the above step S23 may include the following steps S231-S234:

[0106] S231, select a key frame of the video, and use layers 1-3 of the CLIP image encoder to extract the entire image features of the key frame.

[0107] S232: for the whole image features of the key frame, the region features corresponding to the position of the action subject in the key frame are obtained by using the region of interest alignment and the action tube.

[0108] S233. Input the regional features corresponding to the position of the action subject into the 4th layer of the CLIP image encoder to obtain the encoder output.

[0109] S234: Input the encoder output to the position-aware region parser PRP to obtain region features related to people.

[0110] Among them, learnable position embedding is introduced in the CLIP image encoder.

[0111] In one possible implementation, a position-aware region parser (PRP) uses the detection results of the intermediate frames to lock on the target person for accurate action classification.

[0112] Specifically, the local features enhanced by position-aware feature alignment and local-global feature adapter enhance regional feature extraction. The present invention takes the video key frame (intermediate frame), uses the 1-3 layers of the CLIP image encoder to extract the key frame image features (the feature map of the entire image), and then uses the region of interest alignment (RoIAlign, the human body bounding box in the key frame output by HCTD defines the position of the human body in the feature map) to obtain the regional features corresponding to the human body position in the entire feature map, and then passes through the 4th layer of the CLIP image encoder, and then inputs it into the position-aware regional parser.

[0113] Since CLIP is trained on whole image and text pairs, it is not particularly sensitive to region-specific images. In order to better align regional features, learnable position embeddings are introduced in the CLIP image encoder. By training position embeddings on real bounding box annotations, the RoI features are enhanced so that they can focus more on region-specific features in the whole image, thereby facilitating alignment with global features. The enhanced regional features are then obtained through a local-global feature adapter consisting of a linear layer, an activation function, and a linear layer. Finally, the regional features are mapped to the feature dimensions of the global video branch extracted by the global branch through a multi-layer perceptron (MLP) and fused with them.

[0114] In short, the local branch consisting of a human-centric category-independent tube detector and a location-aware region parser captures human-related regional features from video frames and reflects human-centric action features more accurately.

[0115] S3: Input the video data into the global branch to extract the global video features.

[0116] In a feasible implementation, the global branch captures global information from the video through the video-level visual language pre-training model ViCLIP, integrates the background context, and assists the model classification. It is then integrated with the regional features extracted by the local branch, such as Figure 3 shown.

[0117] S4. Fusing the regional features related to people and the global video features to obtain fused video regional features.

[0118] Specifically, through the multi-layer perceptron MLP, the regional features related to people are mapped to the feature dimensions of the global video features and fused with the global video features to obtain the fused video regional features.

[0119] The features obtained by the global branch capture temporal and dynamic changes while providing rich background and contextual information. Meanwhile, the local features captured by the human-centric local branch contain both static visual information and key action-related details. By combining these two aspects of information, the global features (which encode motion information across the entire video sequence) enhance the discriminability of region-specific features. This enables the model to capture subtle differences in actions, leading to more accurate action classification.

[0120] S5. Calculate the similarity between the fused video region features and the text features, and obtain the classification result of the behavior category according to the similarity.

[0121] In a feasible implementation, the similarity between the fused spatiotemporal features and the text embedding is calculated to generate the final action detection result.

[0122] The overall architecture of the model is as follows Figure 4 As shown. The workflow of the model can be described as follows: First, input a video clip of human action. For the global branch, the video encoder of the video-level visual language pre-training model ViCLIP encodes the video to obtain global video features. For the local branch, the video extractor CSN-152 extracts video features, and only detects the spatial position of the action subject in the positioning video through a human-centered category-independent tube detector, and outputs an action tube composed of the action subject boundary box link between video frames. Subsequently, the present invention uses the detection results output by the image-level visual language pre-training model CLIP and HCTD to process the key intermediate frame image through the region of interest alignment operation (RoIAlign) to obtain regional features, and enhances the regional features through a position-aware regional parser. The regional features are then mapped to the feature dimensions of the global video branch extracted by the global branch through a multi-layer perceptron (MLP) and fused with them. Finally, the fused video regional features are calculated similarity with the text features to obtain the classification result of the final behavior category.

[0123] Optionally, the loss of the human-centric category-independent tube detector HCTD includes: a first binary classification loss function, a second binary classification loss function, a GIoU loss, and an L1 loss.

[0124] The first two-class loss function is used to supervise the matching between the query and the target, ensuring that the model can accurately assign the detection query to the correct target instance.

[0125] The second binary classification loss function is used to supervise the existence of frame-level actions and help the model identify whether there is a specific action in each frame, that is, to determine whether the current frame is within the time boundary.

[0126] GIoU loss and L1 loss work together to optimize the accuracy of bounding box prediction.

[0127] Model training: The training process is divided into two stages, focusing on the optimization of the human-centric and category-independent tube detector module and the location-aware region parser module respectively.

[0128] 1) Human-centric category-independent detector: HCTD only focuses on optimizing the positioning of the action subject and does not predict the behavior category corresponding to the classification. A total of four losses are used to jointly optimize the model: a binary classification loss function to supervise the matching between the query and the target, ensuring that the model can accurately assign the detection query to the correct target instance; a binary classification loss function to supervise the existence of frame-level actions, helping the model to identify whether there is a specific action in each frame, that is, to determine whether the current frame is within the time boundary; GIoU loss and L1 loss work together to optimize the accuracy of bounding box prediction. GIoU loss not only considers the overlapping area of ​​the bounding boxes, but also their relative positions, so as to better guide the regression of the bounding boxes. L1 loss directly measures the distance between the predicted bounding box and the true bounding box to ensure the accuracy of the regression result.

[0129] 2) Position-aware Region Parser: Use the ground-truth bounding boxes in the base data to train position-aware embeddings and local-global feature adapters. Standard cross-entropy loss function is used for supervision.

[0130] Model Reasoning:

[0131] 1) Detection phase:

[0132] The HCTD module first generates a detection box. The present invention uses a confidence threshold to filter the prediction results and selects those detection boxes whose confidence is higher than the threshold.

[0133] 2) Classification stage:

[0134] The detection boxes filtered by the confidence threshold are input into the PRP module to extract regional features. The global branch extracts global video features, fuses them with regional features, and then performs similarity analysis with text features to obtain the final classification results.

[0135] The research purpose of this invention is to establish a simple and effective open vocabulary spatiotemporal action detection technology for the real world. It is expected to build an open concept space with the help of the concept knowledge learned from the massive text-image data by the video-level visual language pre-training model, and fine-tune the migration to the downstream perception task through a small amount of downstream task training data. The model can locate and identify related entities of different granularity in the video according to any description.

[0136] Figure 5Comparison results between the performance of the present invention and the state-of-the-art methods on the UCF-JHMDB dataset are provided. Although the present method is primarily designed for open vocabulary action detection, it demonstrates comparable performance to fully supervised methods in a traditional fully supervised setting. Notably, relying solely on similarity-based classification supervision, the present method improves ACT by 5.6 percentage points on the Frame-mAP@0.5 metric and surpasses T-CNN by 30.7 percentage points. These results demonstrate the high efficiency of the present framework even in a traditional, fully supervised setting.

[0137] In addition to the fully supervised evaluation, the method of the present invention is also compared with the recent Vision-Language Models (VLMs) to study the ability of the method in the open vocabulary scenario. The results show that the method of the present invention shows consistent superiority in both basic and novel categories. In particular, compared with the ViCLIP model as the backbone network of the present invention, the method of the present invention is significantly better than other models. Specifically, in the basic category, the method of the present invention is 67.5, 65.8 and 59.1 percentage points higher than Open-VCLIP, VideoCLIP and ViCLIP respectively; and in the novel category, it is 17.2, 28.9 and 10.2 percentage points higher, respectively. The experimental results verify the effectiveness of the model design of the present invention and demonstrate the potential of open recognition related problems.

[0138] In order to evaluate the classification performance of the PRP module of the present invention, the present invention conducted experiments using the real bounding box annotations of human bodies in the dataset. This setting enables the present invention to directly compare the method of the present invention with ViCLIP and PRP. Although ViCLIP performs well in the video-text matching task, its accuracy is significantly lower in the region-level action classification task. This performance difference is mainly attributed to the domain gap between the ViCLIP pre-training objectives and the fine-grained requirements required for localized action recognition. This observation prompted the present invention to adopt a specific action-text alignment training strategy to bridge the gap between global video understanding and localized action recognition. Figure 5 As shown in the results of VOPD*(Ours), our method demonstrates highly accurate classification capabilities with true bounding box annotations.

[0139] Figure 6The visualization results of ViCLIP and the method of the present invention on the new class test set JHMDB in the UCF-JHMDB benchmark are shown. The light yellow bounding box represents the true annotation box in the data set, the light blue bounding box represents the detection result of the present invention, and the light blue action tube represents the result of spatiotemporal action detection. The text labels and scores in the light blue box correspond to the predicted category labels and their classification scores, respectively. Among them, the results of ViCLIP represent the output of zero-sample classification. The present invention observes that, compared with ViCLIP, the VOPD of the present invention achieves highly accurate results in bounding box positioning, while showing significant improvement in classification performance.

[0140] Conclusion: Extensive experiments on the UCF-JHMDB dataset show that the proposed method outperforms the state-of-the-art methods. The proposed algorithm provides a feasible solution to the problem of insufficient open recognition capability and provides new directions and ideas for further research on spatiotemporal action detection.

[0141] This paper proposes a position-aware two-stage detector, which is trained from a limited number of base classes under the supervision of human bounding boxes, action time boundaries and corresponding behavior categories, in the hope of adapting the video-level visual language pre-training model ViCLIP to the open vocabulary spatiotemporal action detection task. The idea of ​​the solution is to first locate the human body in the video frame and then classify the actions performed by the human body. The overall framework is mainly composed of two branches, the local branch and the global branch. The local branch is used to capture human-related regional features from the video frame to more accurately reflect the human-centered action features; the global branch is used to capture global information from the video. By integrating the background context, the model can assist in classification. For example, the action "climbing stairs" is usually associated with stairs, which helps to distinguish it from the action "combing hair" that usually occurs in indoor environments.

[0142] In an embodiment of the present invention, a video-level visual language pre-training model ViCLIP is used to solve the problem of insufficient open recognition ability of existing spatiotemporal action detection models, and training is performed on a limited number of base classes to adapt ViCLIP to the open vocabulary spatiotemporal action detection task.

[0143] We design a position-aware two-stage detector that decomposes the problem into localization followed by classification. This approach does not require additional large-scale pre-training of region-text pairs and bridges the gap between ViCLIP trained on the entire video and region-based recognition tasks.

[0144] We design a human-centric category-independent tube detector (HCTD) that extracts spatiotemporal action representations and predicts category-independent action detection tubes. We also design a position-aware region parser (PRP) that improves region classification results by optimizing position-aware embeddings and local-global feature adapters.

[0145] Figure 7 1 is a block diagram of an open vocabulary spatiotemporal action detection device based on visual language pre-training according to an exemplary embodiment, wherein the device is used in an open vocabulary spatiotemporal action detection method based on visual language pre-training. Figure 7 The device includes an acquisition module 310, a local feature extraction module 320, a global feature extraction module 330, a fusion module 340 and an output module 350. Among them:

[0146] The acquisition module 310 is used to acquire the video data to be detected.

[0147] The local feature extraction module 320 is used to input the video data into the local branch and extract the regional features related to people.

[0148] Among them, the local branch includes a human-centric category-independent tube detector (HCTD) and a position-aware region parser (PRP).

[0149] The global feature extraction module 330 is used to input video data into the global branch to extract global video features.

[0150] Among them, the global branch includes the video-level visual language pre-training model ViCLIP.

[0151] The fusion module 340 is used to fuse the region features related to people and the global video features to obtain fused video region features.

[0152] The output module 350 is used to calculate the similarity between the fused video region feature and the text feature, and obtain the classification result of the behavior category according to the similarity.

[0153] In an embodiment of the present invention, a video-level visual language pre-training model ViCLIP is used to solve the problem of insufficient open recognition ability of existing spatiotemporal action detection models, and training is performed on a limited number of base classes to adapt ViCLIP to the open vocabulary spatiotemporal action detection task.

[0154] We design a position-aware two-stage detector that decomposes the problem into localization followed by classification. This approach does not require additional large-scale pre-training of region-text pairs and bridges the gap between ViCLIP trained on the entire video and region-based recognition tasks.

[0155] We design a human-centric category-independent tube detector (HCTD) that extracts spatiotemporal action representations and predicts category-independent action detection tubes. We also design a position-aware region parser (PRP) that improves region classification results by optimizing position-aware embeddings and local-global feature adapters.

[0156] Figure 8 is a schematic diagram of the structure of an open vocabulary spatiotemporal action detection device provided by an embodiment of the present invention, such as Figure 8 As shown, the open vocabulary spatiotemporal action detection device may include the above Figure 7 The open vocabulary spatiotemporal action detection apparatus based on visual language pre-training is shown. Optionally, the open vocabulary spatiotemporal action detection apparatus 410 may include a first processor 2001 .

[0157] Optionally, the open vocabulary spatiotemporal action detection device 410 may further include a memory 2002 and a transceiver 2003 .

[0158] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0159] Combine the following Figure 8 The components of the open vocabulary spatiotemporal motion detection device 410 are specifically introduced:

[0160] The first processor 2001 is the control center of the open vocabulary spatiotemporal motion detection device 410, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0161] Optionally, the first processor 2001 may perform various functions of the open vocabulary spatiotemporal motion detection device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0162] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 8CPU0 and CPU1 are shown in FIG.

[0163] In a specific implementation, as an embodiment, the open vocabulary spatiotemporal action detection device 410 may also include multiple processors, such as Figure 8 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0164] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0165] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0166] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0167] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 8 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0168] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and communicate with the first processor 2001 through the interface circuit ( Figure 8 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0169] It should be noted that Figure 8 The structure of the open vocabulary spatiotemporal action detection device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0170] In addition, the technical effects of the open vocabulary spatiotemporal action detection device 410 can refer to the technical effects of the open vocabulary spatiotemporal action detection method based on visual language pre-training described in the above method embodiment, and will not be repeated here.

[0171] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0172] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0173] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0174] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0175] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0176] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0177] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0178] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0179] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0180] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0181] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0182] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0183] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An open vocabulary spatiotemporal action detection method based on visual language pre-training, characterized in that: The method comprises: S1. Obtain the video data to be detected; S2, inputting the video data into a local branch to extract regional features related to people; Wherein, the local branch includes a human-centric category-independent tube detector HCTD and a position-aware regional parser PRP; S3, inputting the video data into the global branch to extract global video features; The global branch includes a video-level visual language pre-training model ViCLIP; S4, fusing the human-related regional features and the global video features to obtain fused video regional features; S5. Calculate the similarity between the fused video region feature and the text feature, and obtain a classification result of the behavior category according to the similarity.

2. The open vocabulary spatiotemporal action detection method based on visual language pre-training according to claim 1, characterized in that: The step of inputting the video data into a local branch and extracting regional features related to people in S2 includes: S21, inputting the video data into a video feature extractor to extract video features; S22, inputting the video features into the human-centric category-independent tube detector HCTD, and outputting an action tube; S23: Input the action pipe into the position-aware region parser PRP to extract region features related to people.

3. The open vocabulary spatiotemporal action detection method based on visual language pre-training according to claim 1, characterized in that: The step of inputting the video features into the human-centric category-independent tube detector HCTD and outputting an action tube in S22 includes: Inputting the video features into the spatiotemporal self-attention layer to obtain the spatiotemporal self-attention layer output; The output of the spatiotemporal self-attention layer is input into the spatiotemporal cross-attention layer, and the spatial position of the action subject in the video is located through a learnable query in the time dimension, and an action tube is output; the action tube is composed of bounding box links of the action subject between video frames.

4. The open vocabulary spatiotemporal action detection method based on visual language pre-training according to claim 1, characterized in that: The step S23 of inputting the action pipe into the position-aware region resolver PRP to extract the region features related to the person includes: S231, selecting a key frame of the video, and using layers 1-3 of the CLIP image encoder to extract the whole image features of the key frame; S232, for the whole image feature of the key frame, using the region of interest alignment and the action tube, obtain the regional feature corresponding to the position of the action subject in the key frame; S233, inputting the regional features corresponding to the position of the action subject into the 4th layer of the CLIP image encoder to obtain the encoder output; S234, inputting the encoder output into the position-aware region resolver PRP to obtain region features related to people; Among them, learnable position embedding is introduced in the CLIP image encoder.

5. The open vocabulary spatiotemporal action detection method based on visual language pre-training according to claim 4, characterized in that: The position-aware region parser PRP comprises: a position-aware feature alignment layer and a local-global feature adapter; The local-global feature adapter includes a first linear layer, an activation function and a second linear layer.

6. The open vocabulary spatiotemporal action detection method based on visual language pre-training according to claim 1, characterized in that: The step of fusing the human-related regional features and the global video features in S4 to obtain fused video regional features includes: The human-related regional features are mapped to feature dimensions of global video features through a multi-layer perceptron MLP, and fused with the global video features to obtain fused video regional features.

7. The open vocabulary spatiotemporal action detection method based on visual language pre-training according to claim 1, characterized in that: The loss of the human-centric category-independent tube detector HCTD includes: a first two-class loss function, a second two-class loss function, a GIoU loss and an L1 loss; The first binary classification loss function is used to supervise the matching between the query and the target, ensuring that the model can accurately assign the detection query to the correct target instance; The second binary classification loss function is used to supervise the existence of frame-level actions and help the model identify whether a specific action exists in each frame, that is, to determine whether the current frame is within the time boundary; The GIoU loss and the L1 loss work together to optimize the accuracy of bounding box prediction.

8. An open vocabulary spatiotemporal motion detection device based on visual language pre-training, the open vocabulary spatiotemporal motion detection device based on visual language pre-training is used to implement the open vocabulary spatiotemporal motion detection method based on visual language pre-training as claimed in any one of claims 1 to 7, characterized in that: The device comprises: An acquisition module, used for acquiring video data to be detected; A local feature extraction module, used for inputting the video data into a local branch to extract regional features related to people; Wherein, the local branch includes a human-centric category-independent tube detector HCTD and a position-aware regional parser PRP; A global feature extraction module, used for inputting the video data into a global branch to extract global video features; The global branch includes a video-level visual language pre-training model ViCLIP; A fusion module, used for fusing the human-related regional features and the global video features to obtain fused video regional features; The output module is used to calculate the similarity between the fused video region feature and the text feature, and obtain the classification result of the behavior category according to the similarity.

9. An open vocabulary spatiotemporal action detection device, characterized in that The open vocabulary spatiotemporal action detection device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images