Weakly supervised spatio-temporal action detection method based on temporal causal graph interactive decoupling

Abstract

The application relates to the technical field of computer vision, and discloses a weakly supervised spatiotemporal action detection method based on time sequence causal graph interaction decoupling. First, original video is sampled and segmented to obtain a video segment set, and a two-dimensional convolution and a three-dimensional convolution neural network are used to extract video segment features and candidate frames of characters; then, a dynamic time sequence causal perception module is used for causal structure learning and time sequence enhancement on the video segment features, and a causal enhanced segment level feature sequence is output; a spatiotemporal interaction decoupling graph reasoning module is used to output the character action category probability and the spatial position of each node; finally, a spatiotemporal action tube is formed according to the prediction result, and the start and end time of the action, the action category and the corresponding character bounding box sequence are obtained. The application effectively alleviates the time positioning fragmentation and spatial correlation ambiguity problems in the field, reduces the labor and time cost of data labeling under the condition of using only video level action labels, and improves the detection precision.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically the field of weakly supervised spatiotemporal action detection in video processing, and relates to a weakly supervised spatiotemporal action detection method based on temporal causal graph interaction decoupling. Background Technology

[0002] With the rapid popularization of mobile internet, IoT, and surveillance equipment, video data is growing at an explosive rate. Short video platforms, live streaming, smart security, and autonomous driving generate massive amounts of video content daily. How to enable machines to efficiently and accurately understand these videos has become a research hotspot in the fields of artificial intelligence and computer vision. Spatiotemporal action detection, as one of the core tasks of video understanding, aims to simultaneously locate the spatiotemporal position of actions (i.e., the start and end times of the action, and the bounding boxes of the people performing the action at each moment) and identify the action category from uncropped long videos. This technology has broad application prospects and significant social value in scenarios such as public safety, smart elderly care, sports analysis, and human-computer interaction. However, fully supervised spatiotemporal action detection methods heavily rely on large-scale, high-quality labeled data. Each action instance not only needs to be labeled with the video-level category but also needs to accurately label its time interval and the bounding boxes of the people performing the action in each frame. This fine-grained annotation of time and space is extremely costly, requiring a large number of professional annotators to draw and repeatedly check frame by frame, which is not only time-consuming and laborious but also prone to introducing subjective bias. Statistics show that annotating the spatiotemporal action tags of a one-minute multi-person interactive video requires an average of several hours of manual work. Therefore, weakly supervised spatiotemporal action detection using only video-level category labeling has become a research focus of common interest in academia and industry, aiming to achieve feasible action localization and recognition performance with extremely low annotation costs.

[0003] Currently, weakly supervised spatiotemporal action detection mainly faces two core problems: temporal localization ambiguity and spatial association ambiguity. Temporal localization ambiguity refers to the fact that with only video-level category labeling, the model lacks direct supervision of the start and end times of the action, resulting in fragmented action segments: either only a small climax of the action is captured, leading to severe temporal boundary contraction; or unwarranted breaks occur during the action execution, forming multiple discontinuous short segments that fail to cover the complete time interval of the entire action instance. Spatial association ambiguity refers to the fact that in multi-person interaction scenarios, video-level labeling cannot indicate which specific person performed the action. The model is easily influenced by background or bystanders, incorrectly associating the action with regions other than the performer's, or failing to generate accurate bounding boxes for the person.

[0004] The shortcomings of the above-mentioned spatiotemporal motion detection methods are mainly reflected in the following two aspects: 1) Traditional weakly supervised methods usually rely on class activation sequences or attention mechanisms to infer the contribution of each segment to action classification from the correlation between video segment features and global classification loss, and then determine the segment with high contribution as the time region where the action occurred. Since classification tasks naturally tend to find the most discriminative pattern, they ignore the complete temporal evolution process of the action. 2) Existing graph methods typically couple a person’s appearance features and interaction influence into the same message passing process, failing to effectively decouple these two different types of information. Appearance features are mainly used for cross-frame identity tracking and trajectory smoothing, while interaction influence determines who is actually performing the action and transmitting the influence to the surrounding participants.

[0005] Based on these considerations, there is an urgent need to design a model that can automatically mine and learn the temporal causal chain of actions from coarse video-level tagging, while decoupling the visual appearance of the character from the interactive influence. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing methods by providing a weakly supervised spatiotemporal action detection method based on temporal causal graph interaction decoupling. This method can learn the temporal causal structure of actions from video-level tags, generate complete and coherent temporal localization, and accurately associate actions with performers in multi-person scenarios through decoupled graph reasoning, thereby achieving efficient and robust weakly supervised spatiotemporal action detection.

[0007] The weakly supervised spatiotemporal action detection method based on temporal causal graph interaction decoupling of the present invention specifically includes the following steps: Step 1: Preprocess the original video to obtain a set of video segments. Use two-dimensional convolutional and three-dimensional convolutional neural networks to extract features, extract video segment features and candidate boxes for people, and obtain target-level spatiotemporal feature tensors and spatiotemporal feature maps. Step 2: Construct a dynamic temporal causal perception module, input the target-level spatiotemporal feature tensor and spatiotemporal feature map, and output the causal-enhanced fragment-level feature sequence; Step 3: Construct a spatiotemporal interaction decoupled graph reasoning module, input causal enhanced fragment-level feature sequences, and output the probability of each target's character action category and spatial location; Step 4: Calculate the temporal smoothing loss based on the causal enhanced fragment-level feature sequence obtained in Step 2, calculate the graph inference loss based on the target's action category probability and spatial location obtained in Step 3, and optimize the spatiotemporal action detection model composed of the dynamic temporal causal perception module and the spatiotemporal interaction decoupled graph inference module using the stochastic gradient descent algorithm. For the new video sequence, execute Steps 1 to 3 in sequence to obtain the spatial location and action category probability of all targets at different times until the spatiotemporal action detection model converges. Step 5: Input the new video sequence into the trained spatiotemporal action detection model to obtain the predicted character bounding box sequence and action category probability. Based on this, obtain the action start time, end time, action category, and corresponding character bounding box sequence.

[0008] Furthermore, step 1 specifically includes the following steps: Step 11: at a rate of per second Frame sampling rate ( Typically, a value of 15 is used to sample the input raw video, resulting in a frame count of... video frame sequence collection ,in, Represents the set of video frame sequences. Frame video frame, Represents the real number field. and These represent the height and width of the video frame, respectively, and 3 represents the RGB three channels; Step 12: Process the video frame sequence set The sliding window method is used for segmentation, and the window length is... Frames (can be 16, 32, etc.), extract A length of Frame video clip To form a collection of video clips ,in Indicates the index of a video segment; Step 13: Extract the video clip The video clip is input into a 3D convolutional neural network to obtain the video segment. Spatiotemporal feature map ,in, , , , These represent the temporal length, height, width, and number of channels of the spatiotemporal feature map. , , This yields spatiotemporal feature maps of all video segments. Step 13 uses the SlowFast 3D convolutional neural network, a mainstream action recognition model in the field, to obtain spatiotemporal feature maps of video segments for subsequent spatiotemporal action detection.

[0009] Step 14: Extract the video clip Keyframes (middle frames of a video clip) The input is fed into a target detection model based on a two-dimensional convolutional neural network to perform human target detection and obtain a set of target bounding box sequences. ,in, Keyframe The target quantity Keyframe The The bounding box of the target object. and These represent keyframes. The The x and y coordinates of the center point of the target bounding box of each target and These represent keyframes. The The width and height of the target bounding box of each target; Step 14 uses the two-dimensional convolutional neural network Faster-RCNN, a mainstream target detection model in this field, which can detect all people in the key frames of the video segment.

[0010] Step 15: Based on keyframes The The bounding box of the target. Spatiotemporal feature maps are obtained by scaling them proportionally. The corresponding number above The bounding box of the target. ,in, , , , ; Step 16: Based on the spatiotemporal feature map The corresponding number above The bounding box of the target. In the spatiotemporal feature map Perform Region of Interest Align (RoIAlign) operation on the target bounding box. Uniform sampling is performed on the corresponding feature map region using a bilinear interpolation algorithm to obtain... There are feature points, among which... A value of 7 is used to obtain a video clip. The Target-level spatiotemporal feature tensor of each target Steps 15-16 map the target bounding box onto the spatiotemporal feature map of the video clip based on the target bounding box position in the video clip and scale it proportionally to obtain the target bounding box on the spatiotemporal feature map. Then, within the mapped bounding box region, the RoIAlign operation is used to uniformly sample several feature points using bilinear interpolation to achieve accurate feature alignment.

[0011] Furthermore, step 2 specifically includes the following steps: Step 21: Construct a dynamic temporal causal sensing module. First, design a learnable temporal causal graph generator composed of multi-layer sensing mechanisms. The target-level spatiotemporal feature tensor obtained in step 16 As input, the conditional independence test is performed based on a neural network, depending on any two objectives. and Given the potential causal relationships and spatial locations between them, output a causal adjacency matrix. , of which elements Representing fragments For the fragment The probability of a causal relationship existing; Step 22: Convert the causal adjacency matrix Spatiotemporal feature map A causally guided Mamba encoder is input together; this encoder is based on a state-space model, and the state update formula is modified to incorporate causality guidance: First, ... reshape obtained ,in It is the sequence length. It is the feature dimension, based on the causal adjacency matrix. Propagation characteristics of the upper triangular part (forward dependency): GCN stands for Graph Convolutional Network; based on the causal adjacency matrix Propagation characteristics of the lower triangular part (backward dependency): ;right Perform forward propagation to capture dynamic causality over time: ;right Perform backward time propagation (process time steps in reverse order): , of which In terms of output dimension, SSM stands for State Space Model. Step 23: Fuse forward and backward GCN and SSM features using the SiLU activation function and a linear layer, and introduce a gating mechanism: , ,in, , For learnable parameters, SiLU is the activation function, and Linear is a linear layer; Step 24: For each time step Repeating the above operations will eventually yield a fragment-level feature sequence enhanced with causal relationships. And calculate the time series smoothing loss here. .

[0012] Furthermore, step 3 specifically includes the following steps: Step 31: Analyze the fragment-level feature sequences obtained in Step 24. After reconstruction, the data is input to the detection head. The spatiotemporally decoupled graph inference module performs the following operations to generate the detection results for each frame: predicting the class probability of the target through the classification branch and predicting the precise coordinates of the bounding box through the regression branch; and outputting the feature vector of the detected target instance in each frame. and action category probability (i.e., the node features required for subsequent construction of the spatiotemporal interaction graph); Step 32: Construct the Spatiotemporal Interaction Graph (STIG): Using the detection results of each frame in all video segments obtained in Step 31 as nodes, construct the graph. Edge set It includes two types: temporal edges: connecting nodes of the same person in adjacent frames, obtained through IoU matching and tracking algorithms (such as ByteTrack), used to model the trajectory association of the same person across time; spatial edges: connecting nodes within the same frame whose spatial distance is less than a threshold. Node pairs (each with a value of 50 pixels) are used to model the interaction relationships between characters within the same frame; each node Initialize two states: appearance feature vector Action Influence Score (Initial value set to 0); Step 33: Design decoupled message propagation rules: In each graph convolution layer, node state updates are divided into two independent paths: Appearance feature propagation path: mainly propagating along temporal edges, using an attention mechanism to aggregate features of the same person from the previous frame for trajectory smoothing; simultaneously, slightly propagating along spatial edges to distinguish different people. The appearance feature vector update formula is: ,in It is a weight matrix. For temporal attention weights, This is the spatial decay weight (inversely proportional to distance). , These are temporal neighbors and spatial neighbors, respectively; the action influence propagation path primarily involves decaying propagation along spatial edges. High-influence nodes (predicted as action initiators) pass their influence to their interacting neighbors, with the propagation weight determined by the interaction strength between nodes. The action influence score update formula is as follows: ,in It is a weight matrix. , Inter-node IOU This refers to the spatial distance, specifically the pixel displacement distance. Step 34: Node Classification and Action Generation: After... L After decoupling propagation through layers (3 or 4), each node The final action category probability is calculated from the fused features. , ,in These are the appearance feature vectors and action influence scores after L-layer propagation, with the spatial location information of the node itself used as the bounding box; and supervision is performed using a loss function that matches the video-level labels: for each action category Select the node with the highest probability in that category as the positive sample, and calculate the target-level action classification loss. ,in It is the actual category corresponding to the node. For category The maximum probability among all nodes represents the node Determined as an action category The executor This is an adjustment factor, with a value range of [value range missing]. ; For regular loss, where It is a space regularization term. It is a scaling factor, and its value range is... Constraining the spatial distance between adjacent nodes within the same action tube ensures a compact action tube. It is a time regularization term that constrains the positional changes of the same person detection box between adjacent frames to ensure smooth trajectory. It is a hyperparameter, and its value range is... .

[0013] Furthermore, step 4 specifically includes the following steps: Step 41: Construct a spatiotemporal action detection model consisting of a dynamic temporal causal perception module and a spatiotemporal interaction decoupled graph reasoning module; Step 42: Extract frames from the training video at a fixed frame rate and sample them into video segments. Input the video segments into the dynamic temporal causal perception module of Step 2 to obtain the causal-enhanced segment-level feature sequence. Input the feature segments and the candidate bounding boxes of the characters from Step 16 into the spatiotemporal interaction decoupling graph inference module of Step 3 to obtain the bounding box of each character and its action category probability in each frame. Step 43: The total loss of the video is calculated from the temporal smoothing loss in Step 24. And the action classification loss and regularization loss in step 34 The graph reasoning loss consists of: ,in These are hyperparameter weights, and their values ​​range from [value range missing]. ; Step 44: According to The entire spatiotemporal action detection model is optimized using the stochastic gradient descent algorithm until the spatiotemporal action detection model converges.

[0014] Furthermore, step 5 specifically includes the following steps: Step 51: Preprocess the new video sequence according to Step 1 to obtain a set of video segments, and input them into the trained spatiotemporal action detection model. During inference, the spatiotemporal interaction decoupling graph inference module predicts the action category and obtains the predicted target bounding box sequence. and action category probability set Output the set of action categories corresponding to the dimensions with a probability greater than 0.5 in the action category output. ,in, Indicates the predicted number of targets. Indicates the number of categories; Step 52: Based on the target bounding box sequence obtained in Step 51 With action category set Determine the start and end times of the actions; specifically, for each action category, retain those with a probability greater than a threshold. The algorithm identifies nodes and connects nodes of the same character across frames using the Hungarian matching algorithm, forming a spatiotemporal action tube. The core idea of ​​the Hungarian algorithm is to construct a cost matrix, where each element represents the matching cost between the detection boxes in the current frame and the next frame (using IoU and node appearance feature vector similarity), prioritizing the matching of the target detection boxes with the highest action influence scores. The algorithm finds the matching scheme with the minimum total cost, achieving optimal association between the detection boxes in the current frame and the targets in the next frame, thus obtaining the optimal target matching set. .

[0015] Then, based on the optimal target matching set Connecting identical target bounding boxes together yields a spatiotemporal action pipeline set. ,in, This represents the number of spatiotemporal action pipelines, which is ultimately determined by the set of spatiotemporal action pipelines. and action category set Determine the start and end times of all actions in the new video sequence and the corresponding set of target bounding box sequences at those times. ,in, This indicates that the action pipeline is initially classified as the first... The moment of the action, i.e., the start time. Indicates the last time the number was detected. The moment of the action, i.e., the termination time. Indicates the first Each action from start and end time Until the end time The target bounding box, Indicates the first The action category of each action. This indicates the number of actions in the new video sequence.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. The weakly supervised spatiotemporal action detection method based on temporal causal graph interaction decoupling of the present invention, through a dynamic temporal causal perception module, uses a learnable temporal causal graph generator to infer the potential causal adjacency matrix, forcing the model to learn the causal evolution logic of action development, rather than simply relying on statistical correlation. In the feature extraction stage, forward causal modeling of video sequences is explicitly performed. This mechanism enhances the model's ability to model action temporal dependencies and causal evolution logic, providing temporally consistent high-quality feature representations for subsequent accurate localization, and effectively avoiding the fragmentation of localization segments.

[0017] 2. The weakly supervised spatiotemporal action detection method based on temporal causal graph interaction decoupling of the present invention can effectively distinguish between the target that actually performs the action and the background / bystanders through explicit decoupling modeling of the graph reasoning module through spatiotemporal interaction decoupling. This solves the problem of erroneous association caused by relying solely on appearance features and provides an effective weakly supervised spatiotemporal action detection solution.

[0018] 3. The weakly supervised spatiotemporal action detection method based on temporal causal graph interaction decoupling of the present invention utilizes a multi-dimensional feature collaboration mechanism: when constructing the spatiotemporal action tube, in addition to referring to the basic category prediction probability, appearance feature vector similarity and action influence score are also introduced as auxiliary constraints. The robustness of node matching is improved by weighted fusion of multi-dimensional information. Attached Figure Description

[0019] Figure 1 This is a flowchart of the weakly supervised spatiotemporal action detection method based on temporal causal graph interaction decoupling of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of this invention.

[0021] Example 1 The present invention provides a weakly supervised spatiotemporal action detection method based on temporal causal graph interaction decoupling, such as... Figure 1 As shown, it includes the following steps: Step 1: Sample and preprocess the given initial video to obtain a set of video segments. For each video segment, use a 3D convolutional neural network to extract its spatiotemporal feature map, which contains the segment's motion and appearance information. Simultaneously, select the middle frame of each segment as a keyframe and use a 2D convolutional neural network object detection model to detect all human targets in that frame, obtaining a series of target bounding boxes. Finally, obtain a set of target bounding box sequences and extract the target-level spatiotemporal feature tensors. Specifically, step 1 includes the following steps: Step 11: at a rate of per second The frame sampling rate is used to sample the input raw video. In this embodiment... A value of 15 results in a number of frames. video frame sequence collection ,in, Represents the set of video frame sequences. Frame video frame, Represents the real number field. and These represent the height and width of the video frame, respectively, and 3 represents the RGB three channels; Step 12: Process the video frame sequence set The sliding window method is used for segmentation, and the window length is... Frames (can be 16, 32, etc.), extract A length of Frame video clip To form a collection of video clips ,in Indicates the index of a video segment; Step 13: Extract the video clip The video clip is input into a 3D convolutional neural network to obtain the video segment. Spatiotemporal feature map ,in, , , , These represent the temporal length, height, width, and number of channels of the spatiotemporal feature map. , , This yields spatiotemporal feature maps of all video segments. In this embodiment, the three-dimensional convolutional neural network used in step 13 is the mainstream action recognition model SlowFast in the field. The spatiotemporal feature maps of the video segments are obtained through the three-dimensional convolutional neural network and used for subsequent spatiotemporal action detection.

[0022] Step 14: Extract the video clip Keyframes, i.e., the middle frames of a video segment. The input is fed into a target detection model based on a two-dimensional convolutional neural network to perform human target detection and obtain a set of target bounding box sequences. ,in, Keyframe The target quantity Keyframe The The bounding box of the target object. and These represent keyframes. The The x and y coordinates of the center point of the target bounding box of each target and These represent keyframes. The The width and height of the target bounding box of each target; in this embodiment, the two-dimensional convolutional neural network used in step 14 is the mainstream target detection model in the field, Faster-RCNN, which can detect all people in the key frames of the video segment.

[0023] Step 15: Based on keyframes The The bounding box of the target. Spatiotemporal feature maps are obtained by scaling them proportionally. The corresponding number above The bounding box of the target. ,in, , , , ; Step 16: Based on the spatiotemporal feature map The corresponding number above The bounding box of the target. In the spatiotemporal feature map Perform Region of Interest Align (RoIAlign) operation on the target bounding box. Uniform sampling is performed on the corresponding feature map region using a bilinear interpolation algorithm to obtain... There are feature points, among which... A value of 7 is used to obtain a video clip. The Target-level spatiotemporal feature tensor of each target .

[0024] Steps 15-16 map the target bounding box onto the spatiotemporal feature map of the video clip according to the target bounding box position in the video clip and scale it proportionally to obtain the target bounding box on the spatiotemporal feature map. Then, within the mapped bounding box region, the RoIAlign operation is used to uniformly sample several feature points through bilinear interpolation to achieve accurate feature alignment.

[0025] Step 2: Construct a dynamic temporal causal perception module. Input the target-level spatiotemporal feature tensor and spatiotemporal feature map. Analyze the dependencies between targets through a neural causal discovery network, infer an adjacency matrix representing the potential causal influence between segments, and use the dynamic temporal causal perception module to recursively calculate the sequence features to capture long-distance temporal dynamics and causal evolution, generating segment-level feature sequences with causal enhancement. Specifically, step 2 includes the following steps: Step 21: Construct a dynamic temporal causal sensing module. First, design a learnable temporal causal graph generator composed of multi-layer sensing mechanisms. The target-level spatiotemporal feature tensor obtained in step 16 As input, the conditional independence test is performed based on a neural network, depending on any two objectives. and Given the potential causal relationships and spatial locations between them, output a causal adjacency matrix. , of which elements Representing fragments For the fragment The probability of a causal relationship existing; Step 22: Convert the causal adjacency matrix Spatiotemporal feature map A causally guided Mamba encoder is input together; this encoder is based on a state-space model, and the state update formula is modified to incorporate causality guidance: First, ... reshape obtained ,in It is the sequence length. It is the feature dimension, based on the causal adjacency matrix. Propagation characteristics of the upper triangular part (forward dependency): GCN stands for Graph Convolutional Network; based on the causal adjacency matrix Propagation characteristics of the lower triangular part (backward dependency): ;right Perform forward propagation to capture dynamic causality over time: ;right Perform backward time propagation (process time steps in reverse order): , of which In terms of output dimension, SSM stands for State Space Model. Step 23: Fuse forward and backward GCN and SSM features using the SiLU activation function and a linear layer, and introduce a gating mechanism: , ,in, , For learnable parameters, SiLU is the activation function, and Linear is a linear layer; Step 24: For each time step Repeating the above operations will eventually yield a fragment-level feature sequence enhanced with causal relationships. And calculate the time series smoothing loss here. .

[0026] Step 3: Construct a spatiotemporal interaction decoupled graph reasoning module, use the detector to obtain the target box of each frame as a node, extract appearance features, and initialize the appearance feature vector and action influence score, pass messages along the temporal edge and spatial edge, dynamically update the action influence score of the node in this process, input the causal enhanced fragment-level feature sequence, and output the character action category probability and spatial position of each target. Specifically, step 3 includes the following steps: Step 31: Analyze the fragment-level feature sequences obtained in Step 24. After reconstruction, the data is input to the detection head. The spatiotemporally decoupled graph inference module performs the following operations to generate the detection results for each frame: predicting the class probability of the target through the classification branch and predicting the precise coordinates of the bounding box through the regression branch; and outputting the feature vector of the detected target instance in each frame. and action category probability (i.e., the node features required for subsequent construction of the spatiotemporal interaction graph); Step 32: Construct the Spatiotemporal Interaction Graph (STIG): Using the detection results of each frame in all video segments obtained in Step 31 as nodes, construct the graph. Edge set It includes two types: temporal edges: connecting nodes of the same person in adjacent frames, obtained through IoU matching and the ByteTrack tracking algorithm, used to model the trajectory association of the same person across time; spatial edges: connecting nodes within the same frame whose spatial distance is less than a threshold. The node pairs, in this embodiment, the threshold The value is 50 pixels, used to model the interaction relationships between characters within the same frame; each node Initialize two states: appearance feature vector Action Influence Score (Initial value set to 0); Step 33: Design decoupled message propagation rules: In each graph convolution layer, node state updates are divided into two independent paths: Appearance feature propagation path: Primarily propagates along temporal edges, employing an attention mechanism to aggregate features of the same person from the previous frame for trajectory smoothing; simultaneously, it slightly propagates along spatial edges to distinguish different people. The appearance feature vector update formula is: ,in It is a weight matrix. For temporal attention weights, This is the spatial decay weight (inversely proportional to distance). , These are temporal neighbors and spatial neighbors, respectively. Action influence propagation path: Primarily propagates along spatial edges in a decaying manner. High-influence nodes (predicted as action initiators) pass their influence to their interacting neighbors, with the propagation weight determined by the interaction strength between nodes. The action influence score update formula is as follows: ,in It is a weight matrix. , Inter-node IOU This refers to the spatial distance, specifically the pixel displacement distance. Step 34: Node Classification and Action Generation: After... L After decoupling propagation through layers (3 or 4), each node The final action category probability is calculated from the fused features. , ,in These are the appearance feature vectors and action influence scores after L-layer propagation, with the spatial location information of the node itself used as the bounding box; and supervision is performed using a loss function that matches the video-level labels: for each action category Select the node with the highest probability in that category as the positive sample, and calculate the target-level action classification loss. ,in It is the actual category corresponding to the node. For category The maximum probability among all nodes represents the node Determined as an action category The executor This is an adjustment factor, with a value range of [value range missing]. ; For regular loss, where It is a space regularization term. It is a scaling factor, and its value range is... Constraining the spatial distance between adjacent nodes within the same action tube ensures a compact action tube. It is a time regularization term that constrains the positional changes of the same person detection box between adjacent frames to ensure smooth trajectory. It is a hyperparameter, and its value range is... .

[0027] Step 4: Calculate the temporal smoothing loss based on the causal-enhanced fragment-level feature sequence obtained in Step 2, calculate the graph inference loss based on the target's action category probability and spatial location obtained in Step 3, optimize the spatiotemporal action detection model composed of the dynamic temporal causal perception module and the spatiotemporal interaction decoupled graph inference module using the stochastic gradient descent algorithm, optimize all parameters until the model converges on the training set. Input a new video, and execute steps 1 to 3 sequentially on the new video sequence to obtain the spatial location and action category probability of all targets at different times until the spatiotemporal action detection model converges. Specifically, step 4 includes the following steps: Step 41: Construct a spatiotemporal action detection model consisting of a dynamic temporal causal perception module and a spatiotemporal interaction decoupled graph reasoning module; Step 42: Extract frames from the training video at a fixed frame rate and sample them into video segments. Input the video segments into the dynamic temporal causal perception module of Step 2 to obtain the causal-enhanced segment-level feature sequence. Input the feature segments and the candidate bounding boxes of the characters from Step 16 into the spatiotemporal interaction decoupling graph inference module of Step 3 to obtain the bounding box of each character and its action category probability in each frame. Step 43: The total loss of the video is calculated from the temporal smoothing loss in Step 24. And the action classification loss and regularization loss in step 34 The graph reasoning loss consists of: ,in These are hyperparameter weights, and their values ​​range from [value range missing]. ; Step 44: According to The entire spatiotemporal action detection model is optimized using the stochastic gradient descent algorithm until the spatiotemporal action detection model converges.

[0028] Step 5: Input the new video sequence into the trained spatiotemporal action detection model to obtain the predicted character bounding box sequence and action category probability set, thereby obtaining the action start time, end time, action category and corresponding character bounding box sequence.

[0029] Specifically, step 5 includes the following steps: Step 51: Preprocess the new video sequence according to Step 1 to obtain a set of video segments, and input them into the trained spatiotemporal action detection model. During inference, the spatiotemporal interaction decoupling graph inference module predicts the action category and obtains the predicted target bounding box sequence. and action category probability set Output the set of action categories corresponding to the dimensions with a probability greater than 0.5 in the action category output. ,in, Indicates the predicted number of targets. Indicates the number of categories; Step 52: Based on the target bounding box sequence obtained in Step 51 With action category set Determine the start and end times of the actions; specifically, for each action category, retain those with a probability greater than a threshold. The node, in this embodiment, the threshold The value ranges from 0.5 to 0.7, and the Hungarian matching algorithm connects nodes of the same character across frames to form a spatiotemporal action tube. The core idea of ​​the Hungarian algorithm is to construct a cost matrix, where each element represents the matching cost between the detection boxes in the current frame and the next frame (using IoU and node appearance feature vector similarity), and prioritizes matching the target detection boxes with the highest action influence scores. The algorithm finds the matching scheme with the minimum total cost to achieve optimal association between the detection boxes of the current frame and the targets in the next frame, obtaining the optimal target matching set. .

[0030] Then, based on the optimal target matching set Connecting identical target bounding boxes together yields a spatiotemporal action pipeline set. ,in, This represents the number of spatiotemporal action pipelines, which is ultimately determined by the set of spatiotemporal action pipelines. and action categories Determine the start and end times of all actions in the new video sequence and the corresponding set of target bounding box sequences at those times. ,in, This indicates that the action pipeline is initially classified as the first... The moment of the action, i.e., the start time. Indicates the last time the number was detected. The moment of the action, i.e., the termination time. Indicates the first Each action from start and end time Until the end time The target bounding box, Indicates the first The action category of each action. This indicates the number of actions in the new video sequence.

[0031] The present invention provides a weakly supervised spatiotemporal action detection method based on temporal causal graph interaction decoupling. It utilizes a temporal causal perception module to mine the causal evolution logic of actions, enabling the model to learn robust and generalizable essential features of actions. Then, it uses a spatiotemporally interactive decoupling graph reasoning module to assign action category labels to targets, so that most targets are noticed, effectively solving the problem of misassignment of action labels and improving the overall recall rate.

[0032] The embodiments and implementation process of the present invention have been described in detail above with reference to the accompanying drawings and tables, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A weakly supervised spatiotemporal action detection method based on temporal causal graph interaction decoupling, characterized in that, Includes the following steps, Step 1: Preprocess the original video to obtain a set of video segments. Use two-dimensional convolutional and three-dimensional convolutional neural networks to extract features, extract video segment features and candidate boxes for people, and obtain target-level spatiotemporal feature tensors and spatiotemporal feature maps. Step 2: Construct a dynamic temporal causal perception module, input the target-level spatiotemporal feature tensor and spatiotemporal feature map, and output a causal-enhanced fragment-level feature sequence; Step 3: Construct a spatiotemporal interaction decoupled graph reasoning module, input causal enhanced fragment-level feature sequences, and output the probability of each target's character action category and spatial location; Step 4: Calculate the temporal smoothing loss based on the causal enhanced fragment-level feature sequence obtained in Step 2, calculate the graph inference loss based on the target's action category probability and spatial location obtained in Step 3, and optimize the spatiotemporal action detection model composed of the dynamic temporal causal perception module and the spatiotemporal interaction decoupled graph inference module using the stochastic gradient descent algorithm. For the new video sequence, execute Steps 1 to 3 in sequence to obtain the spatial location and action category probability of all targets at different times until the spatiotemporal action detection model converges. Step 5: Input the new video sequence into the trained spatiotemporal action detection model to obtain the predicted character bounding box sequence and action category probability. Based on this, obtain the action start time, end time, action category, and corresponding character bounding box sequence.

2. The weakly supervised spatiotemporal action detection method based on temporal causal graph interaction decoupling as described in claim 1, characterized in that: Step 1 specifically includes the following steps: Step 11: at a rate of per second The frame sampling rate samples the input raw video, resulting in a number of frames. video frame sequence collection ,in, Represents the set of video frame sequences. Frame video frame, Represents the real number field. and These represent the height and width of the video frame, respectively, and 3 represents the RGB three channels; Step 12: Process the video frame sequence set The sliding window method is used for segmentation, and the window length is... Frame, capture A length of Frame video clip To form a collection of video clips ,in Indicates the index of a video segment; Step 13: Extract the video clip The video clip is input into a 3D convolutional neural network to obtain the video segment. Spatiotemporal feature map ,in, , , , These represent the temporal length, height, width, and number of channels of the spatiotemporal feature map. , , This allows us to obtain spatiotemporal feature maps of all video segments; Step 14: Extract the video clip keyframes The input is fed into a target detection model based on a two-dimensional convolutional neural network to perform human target detection and obtain a set of target bounding box sequences. ,in, Keyframe The target quantity Keyframe The The bounding box of the target object. and These represent keyframes. The The x and y coordinates of the center point of the target bounding box of each target and These represent keyframes. The The width and height of the target bounding box of each target; Step 15: Based on keyframes The The bounding box of the target. Spatiotemporal feature maps are obtained by scaling them proportionally. The corresponding number above The bounding box of the target. ,in, , , , ; Step 16: Based on the spatiotemporal feature map The corresponding number above The bounding box of the target. In the spatiotemporal feature map Perform region of interest alignment operation on the target bounding box. Uniform sampling is performed on the corresponding feature map region using a bilinear interpolation algorithm to obtain... There are feature points, among which... A value of 7 is used to obtain a video clip. The Target-level spatiotemporal feature tensor of each target .

3. The weakly supervised spatiotemporal action detection method based on temporal causal graph interaction decoupling as described in claim 2, characterized in that, Step 2 includes the following steps: Step 21: Construct a dynamic temporal causal sensing module. First, design a learnable temporal causal graph generator composed of multi-layer sensing mechanisms. The target-level spatiotemporal feature tensor obtained in step 16 As input, the conditional independence test is performed based on a neural network, depending on any two objectives. and Given the potential causal relationships and spatial locations between them, output a causal adjacency matrix. , of which elements Representing fragments For fragments The probability of a causal relationship existing; Step 22: Convert the causal adjacency matrix Spatiotemporal feature map A causally guided Mamba encoder is input together; this encoder is based on a state-space model, and the state update formula is modified to incorporate causality guidance: First, ... reshape obtained ,in It is the sequence length. It is the feature dimension, based on the causal adjacency matrix. Propagation characteristics of the upper triangular part: GCN stands for Graph Convolutional Network; based on the causal adjacency matrix Propagation characteristics of the lower triangular part: ;right Perform forward propagation to capture dynamic causality over time: ;right Perform backward propagation: , of which Output dimension, SSM stands for State-Space Model.

4. The weakly supervised spatiotemporal action detection method based on temporal causal graph interaction decoupling as described in claim 3, characterized in that: Step 2 also includes the following steps: Step 23: Fuse forward and backward GCN and SSM features using the SiLU activation function and a linear layer, and introduce a gating mechanism: , ,in, , For learnable parameters, SiLU is the activation function, and Linear is a linear layer; Step 24: For each time step Repeating the above operations will eventually yield a fragment-level feature sequence enhanced with causal relationships. And calculate the time-series smoothing loss here. .

5. The weakly supervised spatiotemporal action detection method based on temporal causal graph interaction decoupling according to claim 4, characterized in that, Step 3 includes the following steps: Step 31: Analyze the fragment-level feature sequences obtained in Step 24. After reconstruction, the data is input to the detection head; the spatiotemporally decoupled graph inference module performs the following operations to generate the detection results for each frame: predicting the class probability of the target through the classification branch and predicting the precise coordinates of the bounding box through the regression branch; and outputting the feature vector of the detected target instance in each frame. and action category probability ; Step 32: Construct the Spatiotemporal Interaction Graph (STIG): Using the detection results of each frame in all video segments obtained in Step 31 as nodes, construct the graph. edge set It includes two types: temporal edges: connecting nodes of the same person in adjacent frames, obtained through IoU matching and tracking algorithms, used to model the trajectory association of the same person across time; spatial edges: connecting nodes within the same frame whose spatial distance is less than a threshold. Node pairs are used to model the interaction relationships between characters within the same frame; each node Initialize two states: appearance feature vector Action Influence Score .

6. The weakly supervised spatiotemporal action detection method based on temporal causal graph interaction decoupling according to claim 5, characterized in that, Step 4 also includes the following steps: Step 33: Design decoupled message propagation rules: In each graph convolution layer, node state updates are divided into two independent paths: Appearance feature propagation path: mainly propagating along temporal edges, using an attention mechanism to aggregate features of the same person from the previous frame for trajectory smoothing; simultaneously, it slightly propagates along spatial edges to distinguish different people; the appearance feature vector update formula is: ,in It is a weight matrix. For temporal attention weights, For spatial decay weights, , These are temporal neighbors and spatial neighbors, respectively. Action influence propagation path: Primarily propagates along spatial edges in a decaying manner; high-influence nodes pass their influence to their interacting neighbors, with the propagation weight determined by the interaction strength between nodes; the action influence score update formula is... ,in It is a weight matrix. , For inter-node IOU, This represents the spatial distance, specifically the pixel displacement distance.

7. The weakly supervised spatiotemporal action detection method based on temporal causal graph interaction decoupling according to claim 6, characterized in that, Step 4 also includes the following steps: Step 34: Node Classification and Action Generation: After... L After layer decoupling propagation, each node The final action category probability is calculated from the fused features. , ,in These are the appearance feature vectors and action influence scores after L-layer propagation, with the spatial location information of the node itself used as the detection box; and supervision is performed using a loss function that matches the video-level labels: for each action category Select the node with the highest probability in that category as the positive sample and calculate the target-level action classification loss. ,in It is the actual category corresponding to the node. For category The maximum probability among all nodes represents the node Determined as an action category The executor This is an adjustment factor, with a value range of [value range missing]. ; For regular loss, where It is a space regularization term. It is a scaling factor, and its value range is... Constraining the spatial distance between adjacent nodes within the same action tube ensures a compact action tube. It is a time regularization term that constrains the positional changes of the same person detection box between adjacent frames to ensure smooth trajectory. It is a hyperparameter, and its value range is... .

8. The weakly supervised spatiotemporal action detection method based on temporal causal graph interaction decoupling according to claim 7, characterized in that, Step 4 specifically includes the following steps: Step 41: Construct a spatiotemporal action detection model consisting of a dynamic temporal causal perception module and a spatiotemporal interaction decoupled graph reasoning module; Step 42: Extract frames from the training video at a fixed frame rate and sample them into video segments. Input the video segments into the dynamic temporal causal perception module of Step 2 to obtain the causal-enhanced segment-level feature sequence. Input the feature segments and the candidate bounding boxes of the characters from Step 16 into the spatiotemporal interaction decoupling graph inference module of Step 3 to obtain the bounding box of each character and its action category probability in each frame. Step 43: The total loss of the video is calculated from the temporal smoothing loss in Step 24. And the action classification loss and regularization loss in step 34 The graph reasoning loss consists of: ,in These are hyperparameter weights, and their values ​​range from [value range missing]. ; Step 44: According to The entire spatiotemporal action detection model is optimized using the stochastic gradient descent algorithm until the spatiotemporal action detection model converges.

9. The weakly supervised spatiotemporal action detection method based on temporal causal graph interaction decoupling according to claim 8, characterized in that, Step 5 includes the following steps: Step 51: Preprocess the new video sequence according to Step 1 to obtain a set of video segments, and input them into the trained spatiotemporal action detection model. During inference, the spatiotemporal interaction decoupling graph inference module predicts the action category and obtains the predicted target bounding box sequence. and action category probability set Output the set of action categories corresponding to the dimensions with a probability greater than 0.5 in the action category output. ,in, Indicates the predicted number of targets. Indicates the number of categories.

10. The weakly supervised spatiotemporal action detection method based on temporal causal graph interaction decoupling according to claim 9, characterized in that, Step 5 also includes the following steps: Step 52: Based on the target bounding box sequence obtained in Step 51 With action category set Determine the start and end times of actions: For each action category, retain those with a probability greater than a threshold. The nodes are identified, and the optimal target matching set is obtained through the Hungarian matching algorithm. ; Then, based on the optimal target matching set Connecting identical target bounding boxes together yields a spatiotemporal action pipeline set. ,in, This represents the number of spatiotemporal action pipelines, which is ultimately determined by the set of spatiotemporal action pipelines. and action category set Determine the start and end times of all actions in the new video sequence and the corresponding set of target bounding box sequences at those times. ,in, This indicates that the action pipeline is initially classified as the first... The moment of the action, i.e., the start time. Indicates the last time the number was detected. The moment of the action, i.e., the termination time. Indicates the first Each action from start and end time Until the end time The target bounding box, Indicates the first The action category of each action. This indicates the number of actions in the new video sequence.

Images