Weakly supervised spatiotemporal action detection method and device based on reconstructed dual-stream label matching

Abstract

The present invention relates to a weakly supervised spatiotemporal action detection method and device based on reconstructed dual-stream label matching. First, a set of video clips is generated, and the background is fused through an action foreground semantic enhancement module to obtain a set of video clips with action foreground semantic enhancement; then, a two-dimensional convolution and a three-dimensional convolution neural network are used to obtain a set of target bounding box sequences, and a target-level spatiotemporal feature set is extracted; then, a mask occlusion feature reconstruction module is used to perform mask reconstruction on the target-level spatiotemporal feature set, and a target-level reconstructed feature set is output; finally, a dual-stream label matching module is used to assign action category labels to the prediction results, and a predicted action category probability set is output. The present invention is not only robust for video action detection in target occlusion scenarios, but also can complete model training by relying only on coarse-grained video-level labels, thereby reducing the time overhead of data labeling and improving the accuracy of spatiotemporal action detection.

Description

Technical Field

[0001] The present invention belongs to the field of spatiotemporal action detection in video processing, and relates to a weakly supervised spatiotemporal action detection method and device based on reconstructed dual-stream label matching. Background Art

[0002] In an era of increasingly convenient smart network connectivity, video data is growing at an unprecedented rate. The rise of short video platforms, live streaming, and self-media has made video a major form of internet content. This video data plays a vital role not only in entertainment and social interaction, but also in other areas. However, compared to traditional image, text, or audio data, video not only contains rich spatial information but also exhibits dynamic temporal variations, making its structure more complex and challenging to process. In this context, enabling machines to accurately understand video content has become a key research topic. Traditional action recognition methods struggle to meet this requirement, especially in scenarios requiring precise identification of specific human actions and their spatiotemporal locations. To address this challenge, spatiotemporal action detection technology has emerged. This technology not only requires identifying the action category in a video but also accurately locating the time period and spatial location of the action. Currently, the computing power provided by hardware devices is increasing rapidly, making complex spatiotemporal action detection feasible for applications such as public security, sports analysis, and human-computer interaction. However, accurately labeling the time period and spatial location of actions is prohibitively expensive, limiting the further expansion of spatiotemporal action detection's application scenarios. Consequently, weakly supervised spatiotemporal action detection has become a key research area. This task relies only on video-level annotations, striving to achieve efficient and low-cost action localization and recognition.

[0003] Currently, there are two problems in weakly supervised spatiotemporal action detection: 1) target occlusion and 2) action label misassignment. The target occlusion problem refers to the situation where multiple objects move or interact simultaneously in a video and one object may partially or completely occlude another object, making it difficult for the detection model to accurately identify the occluded action. The action label misassignment problem refers to the situation where, due to the lack of precise spatiotemporal labels, the model may incorrectly assign action labels to irrelevant people in the video, reducing detection accuracy. To address the target occlusion problem, existing methods use pose estimation models to identify occluded parts and predict the motion characteristics of the occluded parts. There are also methods that use diffusion models to reconstruct the occluded parts of the video. To address the other problem, existing methods generate packet-level predictions (action categories contained in the video clip) by maximum pooling the action prediction probabilities of all targets to achieve matching with video-level labels.

[0004] The shortcomings of the aforementioned spatiotemporal action detection methods are primarily manifested in the following two aspects: 1) Utilizing pose estimation models or diffusion models to reconstruct occluded parts at the feature or image level increases model complexity and requires a large number of both occluded and unoccluded video samples for training, increasing training costs. 2) By max-pooling the action prediction probabilities of all targets to generate packet-level prediction probabilities, only a small number of targets are focused on, affecting overall recall. Therefore, there is an urgent need to design a weakly supervised spatiotemporal action detection method that can both reduce the interference of occlusion on target feature semantics and achieve frame-level object and video-level tag matching, thereby improving action detection accuracy and reducing data annotation overhead. Summary of the Invention

[0005] The purpose of the present invention is to address the shortcomings of existing methods and provide a dual-stream label matching weakly supervised spatiotemporal action detection method based on mask feature reconstruction; this method can not only alleviate the target occlusion problem, reduce the interference of occlusion on the target feature semantics, and achieve accurate capture of action semantic features, but also achieve video frame-level target and video-level label matching, and generate accurate action time interval, action category and target bounding box.

[0006] The method specifically comprises the following steps:

[0007] Step (1) obtain a set of video clips based on the original video, extract features using two-dimensional convolutional and three-dimensional convolutional neural networks, and output a set of target bounding box sequences and a target-level spatiotemporal feature tensor;

[0008] Step (2) constructing a mask occlusion feature reconstruction module consisting of a multi-mask occlusion submodule and a feature reconstruction submodule, with the input being the target-level spatiotemporal feature tensor and the output being the target-level reconstructed feature set;

[0009] Step (3) construct a dual-stream tag matching module consisting of a target feature spatiotemporal processing submodule, a target-level prediction branch, and a packet-level prediction branch, with the input being a target-level reconstructed feature set and the output being a target action category probability set;

[0010] Step (4) Calculate the feature reconstruction loss based on the target-level reconstructed features, calculate the action classification loss based on the target action category probability and the packet-level action prediction, and train the spatiotemporal action detection model composed of the mask occlusion feature reconstruction module and the dual-stream tag matching module until the model converges;

[0011] Step (5) inputs the new video sequence into the trained model to obtain the predicted target bounding box sequence set and action category probability set, and thus obtain the action start time, end time, action category and target bounding box.

[0012] Preferably, step (1) specifically includes the following steps:

[0013] (1-1) Sampling the original video to obtain a set of video frame sequences;

[0014] (1-2) Segment the video frame sequence set to obtain a video clip set;

[0015] (1-3) Input the video clip into a 3D convolutional neural network to obtain a spatiotemporal feature map of the video clip;

[0016] (1-4) The middle frame of the video clip is used as the key frame of the video clip and input into the target detection model based on the two-dimensional convolutional neural network to detect the human target and obtain the corresponding target bounding box sequence set;

[0017] (1-5) Obtain the corresponding target bounding box on the spatiotemporal feature map by scaling;

[0018] (1-6) Based on the target bounding box of the corresponding target on the spatiotemporal feature map, a Region of Interest Align (RoIAlign) operation is performed on the spatiotemporal feature map to obtain the target-level spatiotemporal feature set of the video clip.

[0019] In the steps (1-5) to (1-6), the target bounding box is mapped to the spatiotemporal feature map of the video clip in a proportional manner according to the position of the target bounding box in the video clip, thereby obtaining the bounding box of the target on the spatiotemporal feature map; then, within the mapped bounding box area, a RoIAlign operation is performed to uniformly sample several feature points through a bilinear interpolation method, thereby achieving precise alignment of the features.

[0020] Preferably, in step (1-3), the three-dimensional convolutional neural network is the action recognition model SlowFast; in step (1-4), the target detection model is the target detection model Faster-RCNN, and the target detection model is used to detect all characters in the key frames of the video clip.

[0021] Preferably, step (2) specifically includes the following steps:

[0022] Construct a multi-mask occlusion submodule. The input is the target-level spatiotemporal feature tensor. According to the random division of the target-level spatiotemporal feature tensor in the spatial dimension, a number of sub-mask sets with no intersection and a union covering all spatial positions of the feature are obtained. The sub-mask sets are used to assign zeros to the corresponding spatial positions of the target-level spatiotemporal feature set to obtain the target-level spatiotemporal feature set with occlusion.

[0023] The above steps are randomly divided into multiple sub-masks with no intersection and whose union covers all spatial positions of the feature to ensure comprehensive occlusion without overlap, enhance feature diversity, avoid model dependence on local areas, and improve robustness and generalization ability.

[0024] A feature reconstruction submodule is constructed using a self-attention layer, which performs a dimension transformation (Reshape) operation on the target-level spatiotemporal feature set with occlusion and then inputs it into the self-attention layer;

[0025] In the self-attention layer, each row of feature vectors after the dimension transformation operation is first input into three fully connected layers to calculate the query vector, key vector and value vector respectively, and then use Calculate the feature relationship weights of the query vector and key vector obtained based on the row feature vector, and obtain the reconstructed action feature set based on the feature relationship weights and the value vectors of each layer; sum and average the reconstructed action feature sets of each target to obtain the target-level reconstructed feature set.

[0026] Preferably, the calculation of the feature reconstruction loss in step (4) specifically includes: performing a dimensionality transformation operation on the reconstructed action feature set obtained in step (2), and then calculating the mean square error loss with the unoccluded target-level spatiotemporal feature set as the feature reconstruction loss.

[0027] The present invention models the global dependency between features through the self-attention mechanism, which can effectively capture the relationship between occluded areas and visible areas in the spatial dimension, thereby achieving accurate completion in the case of missing features; in addition, through multi-occlusion reconstruction and loss supervision, the model's robustness to occlusion is enhanced, and the semantic integrity of target-level spatiotemporal features is improved.

[0028] Preferably, step (3) specifically includes:

[0029] Input the target-level reconstructed feature set into the target feature spatiotemporal processing submodule, and use spatial maximum pooling and temporal averaging operations to obtain the final target-level action feature set;

[0030] Inputting the target-level action feature set into the target-level prediction branch, and converting it into a target action category probability set using a classifier composed of a multi-layer perceptron;

[0031] In the packet-level prediction branch, a learnable aggregated feature query vector is randomly initialized as a query for cross-attention, and a target-level action feature set is used as the key and value for cross-attention. A packet-level feature vector is generated through the cross-attention mechanism, which is then input into a classifier composed of a multi-layer perceptron for action classification, thereby obtaining a packet-level action prediction vector.

[0032] Preferably, in step (4), the calculation of the action classification loss specifically includes:

[0033] Calculate the cross entropy loss between the true action category and the target action category probability in step (3), i.e., the target-level action classification loss;

[0034] The step (3) uses video-level labels to guide target-level feature learning, and achieves alignment between categories and targets by selecting the target with the highest probability in each category from the classification probabilities as positive samples, thus avoiding dependence on target-level labels.

[0035] Calculate the cross entropy loss between the true action category and the bag-level action prediction probability of step (3), i.e., the bag-level action classification loss.

[0036] The step (3) also introduces a learnable query vector and adaptively aggregates target-level action features through a cross-attention mechanism, which can effectively capture the overall action semantics between multiple targets and avoid information omission. At the same time, compared with simple averaging or maximum pooling, this method has stronger feature selection capabilities, thereby improving action classification performance.

[0037] The present invention also provides a weakly supervised spatiotemporal action detection device based on reconstructed dual-stream label matching, the device comprising:

[0038] An acquisition unit is configured to obtain a set of video clips using the original video, extract features using two-dimensional convolutional and three-dimensional convolutional neural networks, and output a set of target bounding box sequences and a target-level spatiotemporal feature tensor.

[0039] A processing unit is configured to: construct a mask occlusion feature reconstruction module consisting of a multi-mask occlusion submodule and a feature reconstruction submodule, with the target-level spatiotemporal feature tensor as input and the target-level reconstructed feature set as output; construct the dual-stream label matching module consisting of a target feature spatiotemporal processing submodule, a target-level prediction branch, and a packet-level prediction branch, with the target-level reconstructed feature set as input and the target action category probability set as output; calculate a feature reconstruction loss based on the target-level reconstructed features, calculate an action classification loss based on the target action category probability and the packet-level action prediction probability, and train the spatiotemporal action detection model consisting of the mask occlusion feature reconstruction module and the dual-stream label matching module until the model converges;

[0040] The detection unit is used to input the new video sequence into the trained model to obtain a set of predicted target bounding box sequences and a set of action category probabilities, thereby obtaining the action start time, end time, action category and target bounding box.

[0041] The present invention also provides a weakly supervised spatiotemporal action detection device based on reconstructed dual-stream label matching, the device comprising: a memory, a processor, and a computer program stored on the memory and runnable on the processor, the computer program being configured to implement the steps of the weakly supervised spatiotemporal action detection based on reconstructed dual-stream label matching.

[0042] The present invention also provides a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the weakly supervised spatiotemporal action detection method based on reconstructed dual-stream label matching are implemented.

[0043] The present invention proposes a dual-stream label matching weakly supervised spatiotemporal action detection method based on mask feature reconstruction. The method has the following features: 1) The mask occlusion feature reconstruction module is used to reconstruct the occluded features. By generating multiple occlusion masks at the feature level and learning occlusion reconstruction, the model can better adapt to different occlusion patterns by utilizing the randomness of multiple masks; 2) The dual-stream label matching module is used to aggregate target features through a cross-attention mechanism to generate a packet-level feature vector, thereby capturing global information. By selecting the feature with the highest prediction probability for each action category as the positive sample, it is ensured that representative features of each action category are paid attention to.

[0044] This method addresses the issues of target occlusion and misassignment of action labels in the context of weakly supervised spatiotemporal action detection. It offers several advantages: 1) Through a masked occlusion feature reconstruction module, features are occluded and reconstructed, enabling the model to learn how to handle occlusion without requiring extensive training with both occluded and unoccluded video samples, thus reducing training costs. 2) Through a dual-stream label matching module, action category labels are assigned to targets, ensuring that most targets are focused and improving overall recall. This provides an effective solution for weakly supervised spatiotemporal action detection, addressing the high labeling costs in practical intelligent surveillance applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of the method of the present invention;

[0046] Figure 2 This is a structural diagram of the model involved in the present invention;

[0047] Figure 3 A schematic diagram of the structure of a weakly supervised spatiotemporal action detection device 7000 based on mask-reconstructed dual-stream label matching provided in an embodiment of the present application;

[0048] Figure 4 This is a structural example that can be used to implement the device described in the embodiments of the present application. DETAILED DESCRIPTION

[0049] The method of the present invention performs the following operations on a given video data set:

[0050] Step (1) preprocess the original video to obtain a set of video clips, extract features through two-dimensional convolution and three-dimensional convolutional neural networks, and output them as a set of target bounding box sequences and target-level spatiotemporal feature tensors;

[0051] Step (2) construct a mask occlusion feature reconstruction module, the input is the target-level spatiotemporal feature tensor, and the output is the target-level reconstructed feature set;

[0052] Step (3) construct a dual-stream tag matching module, with the input being the target-level reconstructed feature set and the output being the target action category probability set;

[0053] Step (4) Calculate the feature reconstruction and action classification losses, and then use the stochastic gradient descent algorithm to optimize the spatiotemporal action detection model composed of the mask occlusion feature reconstruction module and the dual-stream label matching module. Perform steps (1) to (3) on the new video sequence to obtain the spatial position and action category of all targets at different times until the model converges.

[0054] Step (5) inputs the new video sequence into the trained model to obtain the predicted target bounding box sequence set and action category probability set, and thus obtain the action start time, end time, action category and target bounding box.

[0055] Furthermore, step (1) is specifically:

[0056] (1-1) per second Frame sampling rate ( The value is usually 15) Sample the input video and get the number of frames A collection of video frame sequences ,in, Represents the first Frame video frame, represents the field of real numbers, and Respectively represent the height and width of the video frame, 3 represents the RGB three channels;

[0057] (1-2) Set of video frame sequences The sliding window method is used for segmentation, and the window length is Frame, intercept The length is Frame video clip , forming a collection of video clips ,in Indicates the index of the video segment;

[0058] (1-3) Video clips Input into the 3D convolutional neural network to obtain video clips Spatiotemporal feature map of ,in, 、 、 、 are the temporal length, height, width and number of channels of the spatiotemporal feature map respectively, , , , thus obtaining the spatiotemporal feature maps of all video clips;

[0059] The three-dimensional convolutional neural network used in the steps (1-3) is the SlowFast action recognition model, which is the mainstream in this field. The spatiotemporal feature map of the video clip is obtained by the three-dimensional convolutional neural network for subsequent spatiotemporal action detection.

[0060] (1-4) Video clips Keyframes (intermediate frames of a video clip) Input into the target detection model based on two-dimensional convolutional neural network to detect human targets and obtain the target bounding box sequence set ,in, Represents a keyframe The number of targets, Represents a keyframe No. The target bounding box of each target, and Represents key frames No. The horizontal and vertical coordinates of the center point of the target bounding box of the target, and Represents key frames No. The width and height of the object bounding box of each object;

[0061] The two-dimensional convolutional neural network used in the steps (1-4) is the mainstream target detection model Faster-RCNN in this field. The target detector can detect all people in the key frames of the video clip.

[0062] (1-5) According to key frame No. The object bounding box of the object , the spatiotemporal feature map is obtained by scaling The corresponding The object bounding box of the object ,in, , , , ;

[0063] (1-6) According to the spatiotemporal feature map The corresponding The object bounding box of the object , in the spatiotemporal feature map Perform the Region of Interest Align (RoIAlign) operation on the target bounding box The corresponding feature map area is uniformly sampled using the bilinear interpolation algorithm to obtain feature points, among which Take the value as 7 to get the video clip No. The target-level spatiotemporal feature tensor of each target .

[0064] In the steps (1-5) to (1-6), the target bounding box is mapped to the spatiotemporal feature map of the video clip in a proportional manner according to the position of the target bounding box in the video clip, thereby obtaining the bounding box of the target on the spatiotemporal feature map; then, within the mapped bounding box area, a RoIAlign operation is performed to uniformly sample several feature points through a bilinear interpolation method, thereby achieving precise alignment of the features.

[0065] Furthermore, step (2) is specifically:

[0066] (2-1) The masked occlusion feature reconstruction module consists of a multi-mask occlusion submodule and a feature reconstruction submodule. It generates multiple occlusion masks of target-level spatiotemporal features at the feature level. By utilizing the randomness of multiple masks, the model can better adapt to different occlusion patterns. The occlusion reconstruction is learned through the Transformer decoder, reducing the interference of occlusion on the semantics of target features and achieving accurate capture of action semantic features.

[0067] (2-2) Construct a multi-mask occlusion submodule for the target-level spatiotemporal feature tensor , according to this feature, the spatial dimension is randomly divided into A set of sub-masks that have no intersection and whose union covers all spatial locations of the feature , using the sub-mask set to target-level spatiotemporal feature tensor The corresponding spatial positions are assigned zeros to obtain the target-level spatiotemporal feature set with occlusion ;

[0068] The step (2-2) is randomly divided into multiple sub-mask sets that have no intersection and whose union covers all spatial positions of the feature to ensure comprehensive occlusion without overlap, enhance feature diversity, avoid model dependence on local areas, and improve robustness and generalization ability.

[0069] (2-3) Construct a feature reconstruction submodule, which consists of a self-attention layer to collect the target-level spatiotemporal features with occlusion Perform the dimension transformation (Reshape) operation to obtain ; The input of the self-attention layer is ,Will No. Row eigenvector Input into three fully connected layers to calculate the query vector , key vector Sum vector ,in, , , is the learnable matrix of the self-attention layer, Indicates the number of channels for query features and key features, , obtained in the same way No. Row eigenvector The corresponding key vector Sum vector , calculate the feature relationship weight ,in, represents the exponential normalization (Softmax) function, represents the inner product, and Represents two vector indices of target-level spatiotemporal features with occlusion, and the reconstructed feature vector output is , get the reconstructed action feature matrix , perform Reshape operation on the reconstructed action feature matrix and get ,Will and the unoccluded object-level spatiotemporal feature tensor The reconstruction loss is calculated as ,in, is the mean square error loss function, and the total reconstruction loss is , the reconstructed action feature set of each target Sum and average to obtain the target-level reconstructed feature tensor , and finally obtain the target-level reconstruction feature set .

[0070] The steps (2-3) described above model the global dependency between features through the self-attention mechanism, which can effectively capture the relationship between occluded areas and visible areas in the spatial dimension, thereby achieving accurate completion in the case of missing features; in addition, through multi-occlusion reconstruction and loss supervision, the model's robustness to occlusion is enhanced, and the semantic integrity of the target-level spatiotemporal feature tensor is improved.

[0071] Furthermore, step (3) is specifically:

[0072] (3-1) The dual-stream tag matching module consists of a target feature spatiotemporal processing submodule, a target-level prediction branch, and a packet-level prediction branch. The input is a target-level reconstructed feature set. , the output is the video-level action prediction result and the target action prediction result;

[0073] (3-2) Construct the target feature spatiotemporal processing submodule, with the input being the target-level reconstructed feature set , and use spatial maximum pooling and temporal averaging operations to obtain the final target-level action feature set ;

[0074] (3-3) Construct the target-level prediction branch, and input the target-level action feature set obtained in step (3-2) , input it into the classifier composed of multi-layer perceptron, and output is the target action category probability set ,in, is the number of action categories, given the real video-level action labels ,in, Represents real video-level action labels The action categories, Indicates a video clip Existence action categories, Indicates a video clip There is no action categories, for the categories, from the action category probability set Find the The target with the highest probability in the category is taken as the The positive sample of each category is defined as , calculate the target-level action classification loss ,in, for No. The probability value of each category;

[0075] The step (3-3) uses video-level labels to guide target-level feature learning, and achieves alignment between categories and targets by selecting the target with the highest probability in each category from the classification probabilities as positive samples, thus avoiding dependence on target-level labels.

[0076] (3-4) Construct the package-level prediction branch. First, randomly initialize a learnable aggregate feature query vector As a cross-attention query, the target-level action feature set obtained in step (3-2) As the key and value of cross attention, the bag-level feature vector is generated by cross attention , which is then input into a classifier composed of a multi-layer perceptron for action classification, and the packet-level action prediction vector is obtained. , calculate the bag-level action classification loss ,in, for No. The probability value of each category.

[0077] The above steps (3-4) introduce a learnable query vector and adaptively aggregate target-level action features using a cross-attention mechanism, which can effectively capture the overall action semantics between multiple targets and avoid information omission. At the same time, compared with simple averaging or maximum pooling, this method has stronger feature selection capabilities, thereby improving action classification performance.

[0078] Going further, step (4) is specifically:

[0079] (4-1) Construct a spatiotemporal action detection model consisting of a mask occlusion feature reconstruction module and a dual-stream label matching module;

[0080] (4-2) Extract frames from the training video at a fixed frame rate, sample video clips, and input them into the spatiotemporal action detection model to obtain the spatial position of all targets at each moment (represented by the target bounding box) and the corresponding action category;

[0081] (4-3) Video clip The total loss of the reconstruction obtained by steps (2-3) is , target-level action classification loss obtained in step (3-3) and the bag-level action classification loss obtained in steps (3-4) The specific calculation formula is as follows:

[0082] ,

[0083] in, 、 and are the weights of the total reconstruction loss, target-level action classification loss, and package-level action classification loss, respectively, and their value ranges are .

[0084] (4-4) The total loss obtained according to step (4-3) ,The stochastic gradient descent algorithm is used to optimize the spatiotemporal action detection model consisting of the mask occlusion feature reconstruction module and the dual-stream tag matching module. Steps (1) to (3) are sequentially executed on the new video sequence to obtain the spatial positions and action categories of all targets at different moments until the model converges;

[0085] In the step (4), the performance evaluation index of the trained model is the frame-level mean average precision (f-mAP). The f-mAP index comprehensively measures the average precision (AP) of each category at the frame level, and reflects the overall detection performance by averaging the AP of all categories.

[0086] Going further, step (5) is specifically:

[0087] (5-1) Preprocess the new video sequence according to step (1) to obtain a set of video clips, which are input into the trained spatiotemporal action detection model. During the inference process, the multi-mask occlusion submodule is removed, and the target-level spatiotemporal feature tensor is directly input into the feature reconstruction submodule for feature reconstruction. The packet-level prediction branch is removed, and the target-level prediction branch predicts the action category to obtain a set of predicted target bounding box sequences. and action category probability set , output the action category set corresponding to the dimension greater than 0.5 in the action category probability ,in, represents the number of predicted targets, Indicates the number of categories;

[0088] (5-2) The target bounding box sequence set obtained according to step (5-1) and action categories , determine the start and end time of the action; the specific method is: from the video clip Start processing and collect the target bounding box sequence of the fragment As the initial action tracking object, and record the corresponding action category , with video clips For example, the video clip A collection of target bounding box sequences Each target bounding box, and the next video segment middle The IoU comparison is performed for each target bounding box, and the Hungarian algorithm is used to match the target bounding box. The input is and , the output is the optimal target matching set ,in, Indicates a video clip No. object bounding boxes and video clips No. pair of object bounding boxes, Indicates a video clip With video clips The number of object bounding box pairs;

[0089] The core idea of ​​the Hungarian algorithm is to construct a cost matrix, where each element represents the cost of matching the detection boxes in the current frame with the next frame (e.g., 1-IoU). The algorithm optimally associates the detection boxes in the current frame with the objects in the next frame in a one-to-one correspondence by finding the matching solution with the lowest total cost.

[0090] Then, according to the optimal target matching set Connect the same target bounding box forward and backward to obtain a set of spatiotemporal action pipelines ,in, Indicates the number of spatiotemporal action pipelines, which is ultimately composed of the spatiotemporal action pipeline set and action categories Determine the start and end times of all actions in the new video sequence and the target bounding box sequence set at the corresponding moments ,in, Indicates that the action pipeline is first classified as The moment of the action, that is, the starting time, Indicates the last detection of The moment of the action, that is, the end time, Indicates the Action from start to end time To the end time The target bounding box, Indicates the The action category of the action, Indicates the number of actions in the new video sequence.

[0091] The present invention will be further described below with reference to the accompanying drawings.

[0092] like Figure 1This method, based on mask feature reconstruction and dual-stream label matching, weakly supervised spatiotemporal action detection, first samples a given video to obtain a set of video clips. It then uses two-dimensional convolutional and three-dimensional convolutional neural networks to obtain a set of target bounding box sequences and extract target-level spatiotemporal feature tensors. The masked occlusion feature reconstruction module then performs masked reconstruction on the target-level spatiotemporal feature tensors, outputting a set of target-level reconstructed features. The dual-stream label matching module then assigns action category labels to the predicted results and outputs a set of predicted action category probabilities. Finally, a new video is input and the previous steps are repeated until the model converges. The new video sequence is fed into the trained model to obtain a set of predicted target bounding box sequences and a set of action category probabilities, from which the action start time, end time, action category, and target bounding box are derived. This method utilizes the masked occlusion feature reconstruction module to mask features and reconstruct them, enabling the model to learn how to handle occluded scenes. The dual-stream label matching module then assigns action category labels to targets, ensuring that most targets are focused and improving overall recall.

[0093] The structural diagram of the model involved in the present invention is as follows Figure 2 The present invention not only has the ability to reconstruct spatiotemporal features in target occlusion scenarios, enhancing the model's robustness to occlusion interference, but can also complete training by relying solely on coarse-grained video-level labels, reducing the manpower and time costs of data annotation.

[0094] Figure 3 This is a schematic diagram of the structure of a weakly supervised spatiotemporal action detection device 7000 based on mask reconstruction dual-stream label matching provided in an embodiment of the present application. The device may include: an acquisition unit 7001, a processing unit 7002, and a detection unit 7003. The main functions of each component module are as follows:

[0095] The acquisition unit 7001 is configured to obtain a set of video clips using the original video, extract features using a two-dimensional convolutional neural network and a three-dimensional convolutional neural network, and output a set of target bounding box sequences and a target-level spatiotemporal feature tensor.

[0096] Processing unit 7002 is configured to: construct a mask occlusion feature reconstruction module consisting of a multi-mask occlusion submodule and a feature reconstruction submodule, with the target-level spatiotemporal feature tensor as input and the target-level reconstructed feature set as output; construct the dual-stream label matching module consisting of a target feature spatiotemporal processing submodule, a target-level prediction branch, and a packet-level prediction branch, with the target-level reconstructed feature set as input and the target action category probability set as output; calculate feature reconstruction loss based on the target-level reconstructed features, calculate action classification loss based on the target action category probability and the packet-level action prediction probability, and train the spatiotemporal action detection model consisting of the mask occlusion feature reconstruction module and the dual-stream label matching module until the model converges;

[0097] The detection unit 7003 is used to: input the new video sequence into the trained model to obtain a predicted target bounding box sequence set and an action category probability set, and thereby obtain the action start time, end time, action category and target bounding box.

[0098] This application provides a device for weakly supervised spatiotemporal action detection based on reconstructed dual-stream label matching. The device includes one or more computing cores (such as a central processing unit or graphics processing unit) and a storage medium in communication with the computing cores. The storage medium stores instructions executable by the computing cores. When executed, the instructions drive the computing cores to implement the weakly supervised spatiotemporal action detection method based on reconstructed dual-stream label matching described in the preceding embodiments.

[0099] Figure 4 The following shows a structural example of a device that can be used to implement the embodiments of the present application. The device can be embodied as a mobile terminal such as a mobile phone, a tablet computer or a portable computer. Figure 4 This is only an illustrative description and should not be understood as limiting the function and application scope of this application.

[0100] Reference Figure 4 The hardware architecture of the weakly supervised spatiotemporal action detection device based on reconstructed dual-stream label matching realizes the interconnection of core components through the system bus 8001. The architecture includes a program storage device 8003 (such as a read-only memory) for storing basic program code, and a data storage device 8006 (such as a solid-state drive or a mechanical hard disk) for storing program code and data and loading them into the running memory 8004 (such as a random access memory); the running memory 8004 receives the loaded program code and stores the instructions and temporary data required for the operation of the device. The main controller 8005 (such as a central processing unit or a graphics processing unit) is responsible for reading and executing instructions from the running memory 8004, specifically implementing the core computing tasks of the weakly supervised spatiotemporal action detection method based on reconstructed dual-stream label matching. The I / O interface 8002 connects to various external devices, including information acquisition devices (such as cameras, microphones, scanners), user interaction devices (such as touch screens, keyboards, mice), information output devices (such as displays, speakers, indicator lights) and data storage devices 8006, and supports the device to communicate data with other systems using wired or wireless networks through the network adapter 8007. It should be emphasized that Figure 4 The structure shown is only a feasible implementation example of the device. During actual deployment, the system architecture and its functional components can be expanded, streamlined or replaced according to specific application requirements.

[0101] In particular, according to the embodiments disclosed herein, the processes described in the flowcharts described herein can be implemented in the form of a computer software program. Specifically, the present application covers a computer program product comprising a computer program stored in a computer-readable medium, wherein the program comprises program code for executing the method illustrated in the flowchart. In these embodiments, the computer program can be downloaded from the data storage device 8006 via a network and installed. When the computer program is executed by the main controller 8005, the corresponding functions defined in the embodiments disclosed herein will be executed.

[0102] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0103] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, and the computer-readable program instructions are used to execute the weakly supervised spatiotemporal action detection method based on reconstructed dual-stream label matching in the above-mentioned embodiment.

[0104] The computer-readable storage media involved in this application include USB flash drives, portable disks, hard disk drives, semiconductor carrier modules, magnetic storage devices, optical fiber media, random access memory (RAM), read-only memory (ROM), and any combination thereof, and also encompass physical implementations such as electrical connection devices with wires. Such media can be deployed independently or in combination, and the computer-executable instructions stored therein can be directly executed or collaboratively processed by associated systems, or transmitted across systems via physical transmission channels such as wires / optical cables. It should be noted that the aforementioned media forms and transmission methods are exemplary technical implementations and do not limit the scope of protection of this application.

[0105] The above-mentioned computer-readable storage medium can be included in the weakly supervised spatiotemporal motion detection device based on reconstructed dual-stream label matching; or it can exist independently without being assembled into the weakly supervised spatiotemporal motion detection device based on reconstructed dual-stream label matching.

[0106] The flowcharts and block diagrams shown in the accompanying drawings illustrate possible architectures, functional modules, and modes of operation of systems, methods, and computer program products according to various embodiments of the present application. In these diagrams, each box corresponds to a module, program fragment, or code portion, which contains one or more executable instructions that implement a specific logical function. In certain alternative implementations, the order in which the functions shown in the figure are executed may be different from the order shown in the diagram. For example, two consecutive boxes may be executed simultaneously, or in reverse order in certain circumstances, depending on actual needs. In addition, a single box or a combination thereof in a block diagram or flowchart may be implemented by a dedicated hardware system or by a combination of hardware and computer instructions to implement a specified function.

[0107] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A weakly supervised spatiotemporal action detection method based on reconstructed dual-stream label matching, characterized by: The following steps are involved: Step (1) obtain a set of video clips based on the original video, extract features using two-dimensional convolutional and three-dimensional convolutional neural networks, and output a set of target bounding box sequences and a target-level spatiotemporal feature tensor; Step (2) constructing a mask occlusion feature reconstruction module consisting of a multi-mask occlusion submodule and a feature reconstruction submodule, with the input being the target-level spatiotemporal feature tensor and the output being the target-level reconstructed feature set; Step (3) constructs a dual-stream tag matching module consisting of a target feature spatiotemporal processing submodule, a target-level prediction branch, and a packet-level prediction branch. The input is a target-level reconstructed feature set, and the output is a target action category probability set. Step (3) specifically includes: (3-1) Inputting the target-level reconstructed feature set into the target feature spatiotemporal processing submodule, and applying spatial maximum pooling and temporal averaging operations to obtain a target-level action feature set; (3-2) Inputting the target-level action feature set into the target-level prediction branch, and converting it into a target action category probability set using a classifier composed of a multi-layer perceptron; (3-3) In the packet-level prediction branch, a learnable aggregated feature query vector is randomly initialized as a query for cross-attention, and a target-level action feature set is used as the key and value of the cross-attention. A packet-level feature vector is generated through the cross-attention mechanism, and the feature vector is input into a classifier composed of a multi-layer perceptron for action classification, thereby obtaining a packet-level action prediction vector. Step (4) Calculate the feature reconstruction loss based on the target-level reconstructed features, calculate the action classification loss based on the target action category probability and the packet-level action prediction, and train the spatiotemporal action detection model composed of the mask occlusion feature reconstruction module and the dual-stream tag matching module until the model converges; Step (5) inputs the new video sequence into the trained model to obtain the predicted target bounding box sequence set and action category probability set, and thus obtain the action start time, end time, action category and target bounding box.

2. The weakly supervised spatiotemporal action detection method based on reconstructed dual-stream label matching according to claim 1 is characterized in that Step (1) specifically includes the following steps: (1-1) Sampling the original video to obtain a set of video frame sequences; (1-2) Segment the video frame sequence set to obtain a video clip set; (1-3) Input the video clip into a 3D convolutional neural network to obtain a spatiotemporal feature map of the video clip; (1-4) The middle frame of the video clip is used as the key frame and input into the object detection model based on the two-dimensional convolutional neural network to detect the human object and obtain the corresponding target bounding box sequence set; (1-5) Obtain the corresponding target bounding box on the spatiotemporal feature map by scaling; (1-6) According to the corresponding target bounding box on the spatiotemporal feature map, the region of interest alignment operation is performed on the spatiotemporal feature map to obtain the target-level spatiotemporal feature set of the video clip.

3. The weakly supervised spatiotemporal action detection method based on reconstructed dual-stream label matching according to claim 2 is characterized in that In step (1-3), the three-dimensional convolutional neural network is the action recognition model SlowFast; in step (1-4), the target detection model is the target detection model Faster-RCNN, and the target detection model is used to detect all the characters in the key frames of the video clip.

4. The weakly supervised spatiotemporal action detection method based on reconstructed dual-stream label matching according to claim 3 is characterized in that Step (2) specifically includes the following steps: (2-1) Construct a multi-mask occlusion submodule, with the target-level spatiotemporal feature tensor as input. Randomly divide the target-level spatiotemporal feature tensor in the spatial dimension to obtain a number of sub-mask sets that have no intersection and whose union covers all spatial positions of the feature. Use the sub-mask sets to assign zeros to the corresponding spatial positions of the target-level spatiotemporal feature set to obtain the target-level spatiotemporal feature set with occlusion. (2-2) The feature reconstruction submodule is constructed using the self-attention layer. After the occluded target-level spatiotemporal feature set is dimensionally transformed, it is input into the self-attention layer. In the self-attention layer, each row of feature vectors after the dimension transformation operation is first input into three fully connected layers to calculate the query vector, key vector and value vector respectively. Calculate the feature relationship weights of the query vector and key vector obtained based on the row feature vector, and obtain the reconstructed action feature set based on the feature relationship weights and the value vectors of each layer; sum and average the reconstructed action feature sets of each target to obtain the target-level reconstructed feature set.

5. The weakly supervised spatiotemporal action detection method based on reconstructed dual-stream label matching according to claim 4 is characterized in that The feature reconstruction loss is calculated in step (4), specifically including: After performing dimension transformation on the reconstructed action feature set of step (2-2), the mean square error loss is calculated with the target-level spatiotemporal feature set of step (1-6) and used as the feature reconstruction loss.

6. The weakly supervised spatiotemporal action detection method based on reconstructed dual-stream label matching according to claim 1, characterized in that The calculation of action classification loss in step (4) specifically includes: Calculate the cross entropy loss between the true action category and the target action category probability in step (3), i.e., the target-level action classification loss; Calculate the cross entropy loss between the true action category and the bag-level action prediction probability of step (3), i.e., the bag-level action classification loss.

7. A weakly supervised spatiotemporal action detection device based on reconstructed dual-stream label matching, characterized in that: The device comprises: An acquisition unit is configured to obtain a set of video clips using the original video, extract features using two-dimensional convolutional and three-dimensional convolutional neural networks, and output a set of target bounding box sequences and a target-level spatiotemporal feature tensor. The processing unit is used to: construct a mask occlusion feature reconstruction module composed of a multi-mask occlusion submodule and a feature reconstruction submodule, with the input being a target-level spatiotemporal feature tensor and the output being a target-level reconstructed feature set; construct the dual-stream label matching module composed of a target feature spatiotemporal processing submodule, a target-level prediction branch and a packet-level prediction branch, with the input being a target-level reconstructed feature set and the output being a target action category probability set, specifically comprising: inputting the target-level reconstructed feature set into the target feature spatiotemporal processing submodule, adopting spatial maximum pooling and temporal averaging operations to obtain a target-level action feature set; inputting the target-level action feature set into the target-level prediction branch, and using a classifier composed of a multi-layer perceptron to convert it into a target action category probability set; in the packet-level prediction branch, randomly initializing a learnable aggregated feature query vector as a cross-attention query, using the target-level action feature set as the key and value of the cross-attention, generating a packet-level feature vector through the cross-attention mechanism, and inputting it into a classifier composed of a multi-layer perceptron for action classification to obtain a packet-level action prediction vector; The feature reconstruction loss is calculated based on the object-level reconstructed features, and the action classification loss is calculated based on the target action category probability and the packet-level action prediction probability. The spatiotemporal action detection model composed of the mask occlusion feature reconstruction module and the dual-stream label matching module is trained until the model converges. The detection unit is used to input the new video sequence into the trained model to obtain a set of predicted target bounding box sequences and a set of action category probabilities, thereby obtaining the action start time, end time, action category and target bounding box.

Images