Semi-supervised space-time action detection method, device and equipment based on semantic causal pseudo-mark uncertainty constraint and storage medium

Abstract

The invention discloses a semi-supervised space-time action detection method, device and equipment based on semantic causal pseudo-mark uncertainty constraint and a storage medium. The method comprises the following steps: firstly, sampling a video to generate a fragment set, and extracting a target bounding box, a target-level spatial-temporal feature and a neighborhood background region feature through a two-dimensional and three-dimensional convolutional neural network; the target-level spatial-temporal features are enhanced through a semantic causal context enhancement module, and an enhanced target-level spatial-temporal feature set is output; unmarked samples are screened through an unmarked sample selection module with uncertainty constraint, and a reliable sample set is output; a pseudo mark calibration module calibrates a pseudo mark and outputs a high-quality pseudo mark set; and finally, outputting an action category probability set by the student model. According to the method, background features can be adaptively weakened, dynamic interference backgrounds can be inhibited, unmarked data can be fully utilized when marked data is limited, the action semantic discrimination capability is enhanced, the data marking cost is reduced, and the problem of balance between the detection efficiency and the cost under the limited marked data is effectively solved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, especially the field of spatiotemporal action detection in video understanding, and relates to a semi-supervised spatiotemporal action detection method, device, equipment and storage medium based on semantic causal pseudo-label uncertainty constraints. Background Technology

[0002] With the rapid development of embodied intelligence and human-machine collaboration technologies, robots have gradually transformed from isolated executors in structured environments into intelligent agents capable of working efficiently and safely with humans. In scenarios such as smart factories and medical surgery assistance, robots must possess real-time and accurate understanding of the intentions of their human collaborators to achieve core functions such as motion prediction, task relay, and collaborative operation. This means that robots need to continuously perceive and understand the dynamic actions of humans in physical space, not only identifying the type of action but also accurately detecting the time period and location of the action to generate safe and predictable response behaviors. However, real-world human-machine collaboration environments are highly open and uncertain. Factors such as changes in lighting, multi-person interactions, and frequent adjustments to workstation layouts result in video data exhibiting extremely strong unstructured characteristics. Compared to static images, video introduces complex temporal dynamic information; compared to pure sequence signals (such as audio and skeletal point sequences), video also possesses rich spatial semantics and multi-target concurrent phenomena. The aforementioned spatiotemporal complexity makes traditional action recognition methods insufficient to meet the stringent requirements of long-term, fine-grained, and accurate spatiotemporal localization in human-robot collaboration. Spatiotemporal action detection technology has emerged and is gradually becoming a core perceptual capability for embodied intelligent robots to achieve human-robot collaboration. This requires simultaneously locating the start and end times of actions performed by the subject in a video, as well as their accurate spatial position, and identifying the corresponding action category. However, fully supervised spatiotemporal action detection, due to the need for precise frame-by-frame annotation of actions, incurs extremely high annotation costs, severely limiting the training scale and generalization ability of the model. Semi-supervised spatiotemporal action detection has become a key research direction. This task only requires annotation of a small amount of video data, aiming to more efficiently utilize a small amount of labeled data and a large amount of unlabeled data to achieve low-cost, high-accuracy, and robust spatiotemporal action localization and recognition, thereby providing technical support for human-robot collaboration and safe interaction in intelligent robots.

[0003] Currently, semi-supervised spatiotemporal action detection faces two main challenges: 1) the dynamic weakening and enhancement of local background regions, and 2) difficulty in semantic discrimination of actions with limited labeled data. The dynamic weakening and enhancement of local background regions refers to the model's overemphasis on dynamic features in the video during learning, leading to misidentification of dynamic elements in the background as actions, while neglecting static features related to the action subject. The difficulty in semantic discrimination of actions with limited labeled data stems from the model's reliance on the most salient discriminative features of local regions (such as a raised arm) during learning, neglecting deeper action semantics. This can result in weak discriminative ability for complex actions, further impacting generalization and learning performance on unlabeled data. To address the dynamic weakening and enhancement of local background regions, existing methods employ random weakening of background pixels in video frames to enhance robustness. For the other challenge, existing methods primarily utilize unlabeled data to construct complex consistency constraint mechanisms or introduce multiple auxiliary modules to enhance feature representation and learn complex action semantics.

[0004] The shortcomings of the aforementioned semi-supervised spatiotemporal action detection methods are mainly reflected in the following two aspects: 1) Randomly weakening background pixels may weaken background information useful to the action subject. During the learning process, the model overemphasizes dynamic features in the video, making local dynamic elements in the background easily misdetected as foreground actions, while static context features highly related to the action subject are easily ignored, thus reducing the model's prediction accuracy to some extent; 2) Constructing complex consistency constraint mechanisms to enhance feature representation, while improving the model's understanding of action semantics to some extent, also brings additional computational burden and storage overhead, leading to a longer training time and higher requirements for hardware resources, thus increasing the actual deployment cost of the method. Therefore, it is urgent to design a semi-supervised spatiotemporal action detection method that can adaptively weaken and enhance background features based on the action subject and enhance action semantics in a low-cost manner, making full use of unlabeled action videos to improve the accuracy and robustness of action detection while reducing data annotation costs. Summary of the Invention

[0005] The purpose of this invention is to provide a semi-supervised spatiotemporal action detection method, apparatus, device, and storage medium based on semantic causal pseudo-label uncertainty constraints, so as to solve the above-mentioned technical problems.

[0006] To address the aforementioned technical problems, the specific technical solutions of the semi-supervised spatiotemporal action detection method, apparatus, device, and storage medium based on semantic causal pseudo-marker uncertainty constraints of the present invention are as follows: A semi-supervised spatiotemporal action detection method based on semantic causal pseudo-label uncertainty constraint includes the following steps: Step 1: The original video of the labeled sample set and the unlabeled sample set collected by the robot is sampled at a fixed frame rate and segmented into a set of video segments. The target bounding box and the target-level spatiotemporal feature tensor are extracted using two-dimensional convolutional and three-dimensional convolutional networks. The feature tensor of the background region in the neighborhood of the target bounding box is obtained using average pooling operation. Step 2: Construct a semantic causal context enhancement module. The input is the target-level spatiotemporal feature tensor and the target bounding box neighborhood background region feature tensor. The output is the target-level context enhancement feature tensor. Step 3: Construct an uncertainty-constrained unlabeled sample selection module, with the target-level context-enhanced feature tensor as input and the selected unlabeled samples as output; Step 4: Construct a pseudo-label calibration module. The inputs are the context enhancement features of the labeled sample set and the reliable unlabeled sample set, as well as the real labels corresponding to the labeled samples. The module constructs an initial category prototype on the labeled samples, calibrates the pseudo-labels of the reliable unlabeled samples, and outputs the calibrated pseudo-labels corresponding to each target in the reliable unlabeled samples. Step 5: Construct a semi-supervised spatiotemporal action detection model, calculate the loss by combining target-level context enhancement features and high-quality pseudo-labels of unlabeled samples, and optimize the spatiotemporal action detection model using the stochastic gradient descent algorithm; Step 6: Extract the target bounding box, target-level spatiotemporal feature tensor, and target bounding box neighborhood background region feature tensor from the new video. Input them into the trained spatiotemporal action detection model and output the action category probability and the corresponding target bounding box set to obtain the complete spatiotemporal action detection result.

[0007] Further, step 1 specifically includes: Step 1.1: Sampling the video from the labeled sample set and the unlabeled sample set at a fixed frame rate to obtain a video frame sequence; Step 1.2: Segmenting the video frame sequence using a sliding window to obtain a video segment set; Step 1.3: Extracting the spatiotemporal feature map of each video segment using a three-dimensional convolutional neural network; Step 1.4: Extracting the target bounding box set from the keyframes of the video segment using a two-dimensional convolutional object detector; Step 1.5: Mapping the target bounding boxes in the keyframes onto the spatiotemporal feature map; Step 1.6: Performing a region of interest alignment operation within the mapped target bounding box region to extract the target-level spatiotemporal feature tensor; Step 1.7: Dividing the neighborhood of the target bounding box into multiple patches, performing a region of interest alignment operation on each patch, and extracting the background patch feature set and its geometric description.

[0008] Furthermore, the semantic causal context enhancement module in step 2 includes a semantic geometry graph submodule, a causal intervention submodule, and a context feature fusion submodule, specifically including: Step 2.1: The semantic geometry graph submodule constructs node embeddings based on target features and background patch features, calculates initial edge weights by combining geometric relationships and dynamic indicators, and constructs a semantic geometry graph; Step 2.2: The causal intervention submodule dynamically adjusts edge weights through a gating network to enhance semantically relevant static region features and suppress dynamically interfering region features; Step 2.3: The context feature fusion submodule fuses target features and background features after causal intervention through graph message aggregation and outputs context-enhanced features.

[0009] Furthermore, the labelless sample selection module with uncertainty constraints in step 3 includes: step 3.1: an uncertainty estimation module based on the energy function, which calculates the uncertainty of the labelless sample using the energy function; step 3.2: a labelless sample selection module, which filters high-confidence labelless samples according to the uncertainty threshold to form a reliable labelless sample set.

[0010] Furthermore, the pseudo-label calibration module in step 4 includes: Step 4.1: Category prototype construction submodule, which calculates prototype vectors for each category based on the context enhancement features of labeled samples; Step 4.2: Prototype consistency pseudo-label calibration module, which calculates the similarity between the features of unlabeled samples and the prototype vectors, performs weighted calibration on the pseudo-labels using a calibration function, and outputs the calibrated pseudo-labels.

[0011] Furthermore, the process of constructing and training the semi-supervised spatiotemporal action detection model in step 5 specifically includes: Step 5.1: Construct a spatiotemporal action detection model based on a teacher-student network architecture, where the student network and teacher network have the same structure, both including a 3D convolutional feature extractor, a 2D object detector, a semantic causal context enhancement module, and a classification and regression head; Step 5.2: Initialize category prototypes using labeled samples and generate pseudo-labels for high-confidence unlabeled samples using a teacher network, including target bounding boxes and action category score distributions; Step 5.3: In each training batch, the labeled samples and the filtered unlabeled samples are input into the student network, and their prediction results for object detection and action classification are calculated respectively; Step 5.4: Based on the matching relationship between the prediction results and the true or false labels, calculate the following loss function: Action localization loss: Based on generalized intersection-union ratio (GUCR), it measures the spatial consistency between the predicted bounding box and the true or pseudo-labeled bounding box; Action classification loss: Based on cross-entropy loss, it measures the consistency between the predicted class distribution and the true or pseudo-labeled class distribution; Causal consistency loss: used to constrain the adaptive enhancement and suppression process of background features in the semantic causal context enhancement module; Prototype consistency loss: used to constrain the semantic alignment between features of labeled and unlabeled samples and class prototypes; Step 5.5: The total loss is the weighted sum of the losses mentioned above. The student network parameters are then backpropagated and updated using the stochastic gradient descent algorithm. Step 5.6: The teacher network parameters are smoothly updated from the student network using an exponential moving average method to maintain training stability; Step 5.7: Repeat the above training process until the model converges on the validation set, and obtain the final spatiotemporal action detection model.

[0012] Furthermore, step 6 specifically includes the following process: Step 6.1: Sample the video to be detected at a fixed frame rate and divide it into a set of video segments. Input each video segment into the trained student model and output the target bounding box sequence and its corresponding action category probability distribution in each segment. Step 6.2: Based on the set action category probability threshold, filter out action category tags with high confidence; Step 6.3: Perform cross-frame matching of target bounding boxes in adjacent video segments, construct a cost matrix and use the Hungarian algorithm to solve for the optimal match, forming a continuous target action pipeline; Step 6.4: Based on the action pipeline and its corresponding category label, determine the start and end times of each action in the video and its spatial bounding box at each time point, and output the complete spatiotemporal action detection results.

[0013] The present invention also discloses a semi-supervised spatiotemporal action detection device based on semantic causal pseudo-label uncertainty constraint, used to execute the method described above. The device includes: an acquisition unit, used to: sample and segment the original video with labeled samples and unlabeled samples into a set of video segments, and extract a set of target bounding box sequences, a target-level spatiotemporal feature tensor, and a target bounding box neighborhood background region feature tensor through two-dimensional convolutional and three-dimensional convolutional neural networks. The processing unit is used to: construct a semantic causal context enhancement module, which takes the target-level spatiotemporal feature tensor and the target bounding box neighborhood background region feature tensor as inputs and outputs a target-level context enhancement feature tensor; construct an uncertainty-constrained unlabeled sample selection module, which outputs a reliable unlabeled sample set; construct a pseudo-label calibration module, which takes the context enhancement features of labeled samples, the context enhancement features of reliable unlabeled samples, and the real labels of labeled samples as inputs and outputs a calibrated pseudo-label for each target in the reliable unlabeled samples; and calculate the total loss, including action localization loss, action classification loss, causal consistency loss, and prototype consistency loss, based on the target-level context enhancement features, the real labels of labeled samples, and the calibrated pseudo-labels, and train the semi-supervised spatiotemporal action detection model using the teacher-student framework until the model converges. The detection unit is used to: perform the same preprocessing as the acquisition unit on the new video sequence and input it into the trained student model to obtain a set of predicted target bounding box sequences and a set of action category probabilities, thereby obtaining the action start time, end time, action category, and target bounding box.

[0014] The present invention also discloses a semi-supervised spatiotemporal action detection device based on semantic causal pseudo-label uncertainty constraints. The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement the steps of the semi-supervised spatiotemporal action detection method based on semantic causal pseudo-label uncertainty constraints.

[0015] The present invention also discloses a storage medium, which is a computer-readable storage medium, and stores a computer program thereon. When the computer program is executed by a processor, it implements the steps of the semi-supervised spatiotemporal action detection method based on semantic causal pseudo-label uncertainty constraint as described in any one of claims 1 to 7.

[0016] This invention proposes a semi-supervised spatiotemporal action detection method based on semantic causal pseudo-label uncertainty constraints. This method has the following characteristics: 1) It utilizes a semantic causal context enhancement module to construct a semantic geometric causal relationship graph between the target and its neighboring background regions. A causal intervention mechanism is introduced, adaptively adjusting feature weights based on the semantic relevance and dynamic features of the background region and the action subject. This enhances semantically relevant static background features while suppressing the response of dynamic interference regions, thereby achieving effective modeling of context features and improving the discriminativeness and robustness of action feature representation; 2) It utilizes an uncertainty-constrained unlabeled sample selection module. The energy value of the unlabeled sample is calculated using an energy function, representing the uncertainty of the unlabeled sample prediction. The unlabeled sample selection module filters high-confidence samples based on the corresponding uncertainty, thereby achieving accurate selection of reliable unlabeled samples and improving the model's efficient use of unlabeled data; 3) It utilizes a pseudo-label calibration module, introducing a learnable prototype vector. Based on the consistency between the embedded features of reliable unlabeled samples and the prototype vector, the pseudo-labels of reliable unlabeled samples are calibrated using a calibration function. This helps improve the quality of pseudo-label generation for reliable unlabeled samples and enhances the model's ability to discriminate action semantics.

[0017] This invention addresses the challenges of background dynamic interference and semantic discrimination difficulties in semi-supervised spatiotemporal action detection tasks. The method offers several advantages: 1) Through a semantic causal context enhancement module, the model can adaptively weaken or enhance background region features, improving the discriminative power of features in complex scenes; 2) Through an uncertainty-constrained unlabeled sample selection module, an energy function is used to measure the uncertainty of unlabeled sample predictions, accurately selecting high-confidence reliable unlabeled samples and improving the model's efficiency in utilizing unlabeled data; 3) Through a pseudo-label calibration module, pseudo-labels for reliable unlabeled samples are calibrated, reducing dependence on labeled data while improving the quality and semantic consistency of pseudo-labels, thus enhancing overall generalization ability. This provides an effective semi-supervised spatiotemporal action detection solution to address the high data annotation costs in practical embodied intelligence and human-machine collaboration applications. Attached Figure Description

[0018] Figure 1 This is a simplified flowchart of the method of the present invention.

[0019] Figure 2 This is a flowchart illustrating the specific process of the method of this invention.

[0020] Figure 3 This is a schematic diagram of the device structure of the present invention.

[0021] Figure 4 This is a schematic diagram of the device structure of the present invention. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings.

[0023] like Figure 1 As shown, a semi-supervised spatiotemporal action detection method based on semantic causal pseudo-label uncertainty constraints first samples a given video to obtain a set of video segments. Then, it uses 2D and 3D convolutional neural networks to obtain a set of target bounding box sequences and extracts the target-level spatiotemporal feature tensor and the feature set of the target bounding box's neighborhood background region. Next, a semantic causal context enhancement module adaptively weakens and enhances the feature tensor of the target bounding box's neighborhood background region, outputting an enhanced target-level spatiotemporal feature set. Then, an uncertainty-constrained unlabeled sample selection module selects unlabeled samples, outputting a set of reliable unlabeled samples. Next, a pseudo-label calibration module calibrates the pseudo-label distribution of the reliable unlabeled samples, outputting a high-quality pseudo-label distribution of the reliable unlabeled samples. Finally, a new video is input, and the previous steps are repeated until the model converges. New video sequences are input into the trained model to obtain a predicted set of target bounding box sequences and a corresponding set of action category probabilities, from which the start and end times of the actions are obtained. This method utilizes a semantic causal context enhancement module to adaptively weaken and enhance the features of the background region in the neighborhood of the target bounding box, enabling the model to better model the dependency relationship between the action subject and the semantically relevant background and suppress the activation of dynamic interference background. It uses an uncertainty-constrained unlabeled sample selection module to select reliable unlabeled samples, thereby improving the model's efficiency in utilizing unlabeled data and training stability in a low-cost manner. Furthermore, a pseudo-label calibration module is used to further calibrate the pseudo-labels of the reliable unlabeled samples, improving the quality of pseudo-labels and the overall accuracy of action detection, while reducing data annotation costs.

[0024] like Figure 2 As shown, the semi-supervised spatiotemporal action detection method based on semantic causal pseudo-label uncertainty constraint of the present invention includes the following steps: Step 1: Sample the original video of the labeled sample set and the unlabeled sample set at a fixed frame rate and segment it into a set of video segments. Use two-dimensional convolutional and three-dimensional convolutional networks to extract the target bounding box and the target-level spatiotemporal feature tensor. Use average pooling operation to obtain the feature tensor of the background region in the neighborhood of the target bounding box. Step 1.1: For the labeled sample set and unlabeled sample set With a sampling rate of per second frame( Typically, a value of 24 is used to sample the input video, resulting in a frame count of [number missing]. video frame sequence collection ,in, Represents the set of video frame sequences. Frame video frame, Represents the real number field. and These represent the height and width of the video frame, respectively, and 3 represents the RGB three channels; Step 1.2: Segment the video frame sequence set using a sliding window, with a sliding window length of [length missing]. The frame, the step size of the sliding operation is Frames (usually smaller) Frames (indicating overlap between different segments), truncation A length of Frame video clip To form a collection of video clips ,in, Indicates the index of a video segment; Step 1.3: Extract each video clip Input a 3D convolutional neural network and output a video clip. Spatiotemporal feature map ,in, , , , These represent the temporal length, height, width, and number of channels of the spatiotemporal feature map, respectively. , , ; Step 1.3 uses a three-dimensional convolutional neural network, which is a commonly used feature extraction model in this field (such as SlowFast, I3D). The three-dimensional convolutional neural network is used to extract the spatiotemporal feature map corresponding to the video segment for subsequent semi-supervised spatiotemporal action detection.

[0025] Step 1.4: Extract the video clip The corresponding keyframes (usually the middle frame of a video clip is taken as the keyframe). The input is fed into a target detector based on a two-dimensional convolutional neural network for target detection, resulting in a set of target bounding boxes. ,in, Keyframe The number of target bounding boxes detected by the target detector Keyframe The The bounding box of each target. , These represent keyframes. The The x and y coordinates of the center point of each target , These represent keyframes. The The width and height of the target bounding box of each target; The object detector based on a two-dimensional convolutional neural network used in step 1.4 is a commonly used object detection model in this field (such as Faster-RCNN). The object detector based on the two-dimensional convolutional neural network extracts all people in the key frames of the video segment for spatial localization in subsequent tasks.

[0026] Step 1.5: Set keyframes The The bounding box of the target. In the video clip Corresponding spatiotemporal feature map Perform a proportional mapping on top to obtain the first... The target bounding box corresponding to each target on the spatiotemporal feature map. ,in, , , , ; Step 1.6: In the spatiotemporal feature map Based on the corresponding target bounding box Performing Region of Interest Align (RoIAlign) operation involves the following steps: First, the spatiotemporal feature map region corresponding to the target bounding box is divided into... A uniform grid, wherein... The value is set to 7, and then selected within each grid cell. Sampling points (commonly used) (the sampling point), based on the feature map according to the first sampling point). The target bounding box is located at the coordinates of the target feature map on the spatiotemporal feature map. Its feature value is calculated using bilinear interpolation. Then, the feature values ​​of all sampling points within the grid cell are averaged to obtain the feature representation of the corresponding grid cell. Finally, the features of all grid cells are stitched together to obtain the result in the video clip. Upper Target-level spatiotemporal feature tensor of each target ; In steps 1.5 to 1.6, the target bounding box position in the keyframe of the video segment is mapped to the spatiotemporal feature map corresponding to the video segment by scaling proportionally, thereby obtaining the target bounding box corresponding to the target on the spatiotemporal feature map. Subsequently, within the mapped target bounding box region, the RoIAlign operation is used to obtain uniformly sampled points of the grid cells, and the feature value is calculated using the bilinear interpolation method, thereby achieving accurate feature alignment.

[0027] Step 1.7: In the first... The neighborhood of each target bounding box is divided using a grid. patch Typically, 8 is taken, in the spatiotemporal feature map. Perform the same Region of Interest Align (RoIAlign) operation as steps (1-6) on each patch to obtain the nth patch. The target bounding box of the target corresponds to the first... Features of a patch ,Will The feature summary of the nth patch yields the nth patch. Background patch feature set corresponding to each target Simultaneously calculate the geometric descriptions of the patch and the target, the first... The first goal The geometric description corresponding to each patch is: ,in, , For the first The first goal The patch is relative to the first The horizontal and vertical offsets of the target , Indicates the first The first goal The patch and the The ratios of the target bounding boxes in the width and height scales, respectively. Indicates the first The first goal The patch and the The ratio of the area of ​​each target bounding box.

[0028] In step 1.7, several patches are obtained as background regions according to the grid division method. Then, the RoIAlign operation described in steps (1-6) is performed on each patch to obtain the sampling features of each patch, thereby realizing the extraction and alignment of background patches.

[0029] Step 2: Construct a semantic causal context enhancement module. The input is the target-level spatiotemporal feature tensor and the target bounding box neighborhood background region feature tensor. The output is the target-level context enhancement feature tensor. Step 2.1: The semantic causal context enhancement module consists of a semantic geometry graph submodule, a causal intervention submodule, and a context feature fusion submodule. By establishing a semantic-geometric causal relationship graph structure between the target candidate box region and its surrounding patches, and utilizing the characteristics of causal modeling, the model can better model the dependency relationship between the action subject and the semantically related background, suppress the activation of dynamic interference background, and enhance the spatiotemporal context features in the semi-supervised spatiotemporal action detection task. Step 2.2: Construct the semantic geometry graph submodule. First, the first... Target features With the Background patch feature set corresponding to each target The corresponding feature tensors are obtained by global average pooling operations. Node embedding corresponding to each target , No. The first goal The node embedding of each patch is as follows Construct the first node based on node embedding A set of nodes for each target , No. Initial edge weights between target nodes and patch nodes ,in, This represents the weight of the three (usually taken as the weight of the three). ), This indicates the cosine similarity calculation operation. express Norm, Indicates the first The first goal The geometric description corresponding to each patch is mapped to... That is, the first The first goal The geometric relationships between the patches are used to constrain the model's feature learning. This represents a dynamic indicator between adjacent time-series dimensions within the same patch, where... This refers to the Sigmoid activation function, used to normalize the dynamic metrics between adjacent temporal dimensions of the same patch to a normal value. It is used to measure the magnitude of patch change over time. Indicates the first The first goal The nth patch feature One time-series dimension To represent the numerically stable term, it is usually taken as To prevent the denominator from being zero and ensure numerical stability, a semantic geometric graph is constructed. ; Step 2.3: Construct a causal intervention submodule to dynamically adjust the initial edge weights of the semantic geometry graph; first, the first... The first goal Geometric relationship of each patch , No. Dynamic metrics of a patch and the The first goal and the first The first goal Cosine similarity of features in each patch Perform splicing, and then combine the spliced ​​results. Input-learnable gating networks The output result is , Among them, The intervention coefficient, used as the initial edge weight, represents the degree to which the features of the patch are enhanced or weakened. For the Sigmoid function, , For trainable parameters, if This indicates that semantically relevant static region features are enhanced, if This indicates the characteristics of the region where dynamic interference is suppressed, based on the intervention coefficient. For the first The first goal Each patch feature is weakened or enhanced to obtain the processed patch features. The updated edge weights after causal intervention are The causal consistency loss function corresponding to the gated network is: ,in For a hyperparameter (the static region is usually taken as...), The dynamic region is usually taken as This indicates that the features of semantically relevant static regions are enhanced, while the features of dynamically interfering regions are weakened. This is an entropy regularization term, which limits the distribution of gated outputs from being too concentrated. Step 2.4: Construct a contextual feature fusion submodule, which will integrate the causal features of the first... The background patch features corresponding to the target and the first target The features of each target are aggregated using graph message features to obtain the fused context-enhanced features. , Ultimately, the output is As context-enhanced features, they are input into subsequent modules.

[0030] Steps 2.2 to 2.4, by constructing a semantic geometry graph between the target and its corresponding patch set and using causal intervention to weaken or enhance patch features, can fully utilize the effective relationship between the target and the patches, enhance the feature interaction between the target and the background patches in subsequent steps, and improve the robustness of the features. The normalization operation described in step 2.2 is to balance the influence of different indicators on the initial edge weights, ensuring that the influence of different indicators on the initial edge weights is fair.

[0031] Step 3: Construct an uncertainty-constrained unlabeled sample selection module, with the target-level context-enhanced feature tensor as input and the selected unlabeled samples as output; Step 3.1: The unlabeled sample selection module with uncertainty constraints consists of an uncertainty estimation module based on the energy function and an unlabeled sample selection module. The input is the context enhancement feature, and the output is the selected unlabeled sample. Step 3.2: Uncertainty estimation module based on energy function. First, for the bounding box... Define the energy function As shown below:

[0032] in, Indicates sample The number of bounding boxes, Indicates the number of categories. Indicates sample The The bounding box belongs to the first The predicted probabilities of each category, Temperature parameter (usually taken as...) ); In the unlabeled sample set Above, using the output of the initial classifier, the first... The first unlabeled sample The predicted action category distribution of the predicted target bounding boxes (considered as pseudo-labels for the categories of unlabeled samples). ,in, Indicates the first The first unlabeled sample The predicted target bounding box belongs to the first... The predicted probability of each action category; through the energy function described above. Calculate the first Energy value of an unlabeled sample This energy value is considered as the first Output the uncertainty of each unlabeled sample; Step 3.3: The unlabeled sample selection module, based on the output of Step 3.2... Uncertainty of an unlabeled sample For the first Select from unlabeled samples, if Then the first Unlabeled samples are considered reliable and retained; otherwise, they are considered unreliable and do not participate in the current training iteration. Indicates the threshold (usually taken as...). This module ensures that only high-confidence samples are used in subsequent model parameter updates and pseudo-label optimization, and aggregates all reliable unlabeled samples into a reliable unlabeled sample set. Output; Steps 3.2 to 3.3 use the energy function to calculate the energy value of the unlabeled sample to represent the uncertainty of the unlabeled sample. Samples with low uncertainty represent high confidence predictions. The reliability of the prediction confidence of the unlabeled sample is evaluated by the energy value, which realizes the adaptive screening of unlabeled samples in the training stage and avoids the use of unlabeled sample predictions with low reliability for model training.

[0033] Step 4: Construct a pseudo-label calibration module. The inputs are the context enhancement features of the labeled sample set and the reliable unlabeled sample set, as well as the real labels corresponding to the labeled samples. The module constructs an initial category prototype on the labeled samples, calibrates the pseudo-labels of the reliable unlabeled samples, and outputs the calibrated pseudo-labels corresponding to each target in the reliable unlabeled samples. Step 4.1: The pseudo-label calibration module consists of a category prototype construction sub-module and a prototype consistency pseudo-label calibration module. The input is the context enhancement features of the labeled sample set and the reliable unlabeled sample set, as well as the real labels corresponding to the labeled samples. The output is the pseudo-label corresponding to each target in the reliable unlabeled samples. Step 4.2: Category Prototype Construction Submodule, in the labeled sample set First, the context-enhanced features corresponding to the labeled samples output in step 2.4 are... Perform global average pooling to obtain one-dimensional feature vectors corresponding to all labeled target samples. The labeled sample's i-th... One-dimensional feature vector corresponding to each target Calculate the corresponding category prototype vector based on the actual action category corresponding to the target. ,in, Indicates action category prototype vector, This indicates that it belongs to the action category. The number of labeled samples, Indicates belonging to the first The first action category The one-dimensional feature vector corresponding to each target Number of action categories; Step 4.3: Prototype Consistency Pseudo-label Calibration Module. In each training batch, there will be a set of labeled samples. With reliable unlabeled sample sets Merged into training sets within the same batch Contextual augmentation features for each target bounding box in the set Perform global average pooling to obtain the first... The first reliable label-free sample Embedded features corresponding to each target bounding box and the The first labeled sample Embedded features corresponding to each target bounding box The similarity is calculated between this vector and the prototype vector of each action category. The similarity for reliable unlabeled samples is... The initial pseudo-labels are weighted and calibrated based on similarity to obtain the first pseudo-label. The first unlabeled sample The predicted target bounding box belongs to the first... Pseudo-label scores after class calibration ,in, Controlling the influence of similarity on strength, Typically, a value of 5 is used; labeled samples and true class labels. corresponding prototype vector Calculate similarity The prototype loss function for labeled samples is: ,in, Indicates the first The first labeled sample One target bounding box, This represents the total number of target bounding boxes in the same batch of labeled sample sets. This controls the degree of similarity amplification (usually set to 5), causing the embeddings of labeled samples to cluster in the semantic space to the vicinity of their true class prototypes; the prototype loss function for unlabeled samples is... , This represents the total number of target bounding boxes in the same batch of reliable unlabeled sample sets. This term encourages the embedding features of unlabeled samples to align with the prototype of the most likely class, thereby stabilizing the semantic direction of pseudo-labels; the total prototype loss is... ,in, , To balance the weights, it is usually taken that... , The prototype loss is used to update the class prototype vectors, ensuring that the class prototype vectors can be continuously updated as the model learns.

[0034] Step 4.4: By introducing learnable class prototype vectors, prototype similarity is calculated in labeled samples and reliable unlabeled samples. The pseudo-labels are weighted and calibrated using a calibration function, so that the prediction results are more in line with the prototype distribution, thereby improving the pseudo-label generation quality of the teacher model.

[0035] Step 5: Construct a semi-supervised spatiotemporal action detection model, calculate the loss by combining target-level context enhancement features and high-quality pseudo-labels of unlabeled samples, and optimize the spatiotemporal action detection model using the stochastic gradient descent algorithm; Step 5.1: Construct a spatiotemporal action detection model consisting of a semantic causal context enhancement module, an uncertainty-constrained label-free sample selection module, and a pseudo-label calibration module. A teacher-student network architecture is adopted, with the teacher network... With student network The model architectures are the same, consisting of three parts: a three-dimensional convolutional neural network, an object detector, a context enhancement module, and a classification and regression head; Step 5.2: At the start of training, the student network... Parameters are initialized randomly; the teacher model parameters are initialized exactly the same as the student model parameters. (Student network...) The prediction results are used to calculate the training loss and perform backpropagation, thereby updating the student network parameters and the teacher network parameters. During the entire training process, it does not participate in backpropagation, but instead uses the Exponential Moving Average (EMA) algorithm to learn from the student network. The latest parameters are obtained and updated smoothly to ensure that the teacher network maintains a stable feature representation during training. Its forward prediction results are only used to generate pseudo-labels for unlabeled samples. Step 5.3: Based on labeled samples in the teacher network The context-enhanced features are used to construct the initial category prototype in step 4.2; Step 5.4: Input the unlabeled samples into the teacher model and execute step 3.3 to obtain the set of high-confidence unlabeled samples, and then obtain the predicted set of target bounding boxes corresponding to the high-confidence unlabeled samples in the teacher model. Action category score distribution corresponding to each predicted target bounding box Step 4.3 is performed on the action category score distribution corresponding to each predicted target bounding box to obtain the weighted and calibrated action category score distribution corresponding to each predicted target bounding box. The predicted set of target bounding boxes corresponding to high-confidence unlabeled samples is compared with the action category score corresponding to each predicted target bounding box after weighted calibration. As pseudo-labels to guide the learning of student models (equivalent to the real target bounding boxes and corresponding real action category labels corresponding to labeled samples); Step 5.5: In each training batch, sample both labeled and unlabeled samples at a fixed frame rate, and input the resulting video segments into the student network. In this process, the set of predicted target bounding boxes for all targets at each time step of the labeled samples is obtained. and the corresponding action category distribution ,in, This indicates the first labeled sample. The target belongs to the first The probability scores of each class, and the set of predicted target bounding boxes for all targets in the unlabeled samples at each time step. and the corresponding action category distribution ,in, This indicates the first labeled sample. The target belongs to the first The probability scores of each class; Step 5.6: For each time step of the labeled samples, generate the set of predicted target bounding boxes for all targets. For each target bounding box in the dataset, compute the set of ground truth target bounding boxes corresponding to it and the labeled samples. The intersection-over-union (IoU) ratio between each ground truth object bounding box is used to obtain a two-dimensional IoU matrix. The intersection-union ratio (IUGR) is the ratio of the area of ​​the intersection of the predicted target bounding box and the ground truth target bounding box to the area of ​​their union. This indicates that there are labeled samples in the keyframe. The first A true target bounding box, These represent labeled samples in the keyframes. The first The x-coordinate, y-coordinate, width, and height of the center point of each real target bounding box; the set of predicted target bounding boxes for all targets at each time step for the unlabeled samples. In calculating the two-dimensional intersection-union ratio, it is only necessary to replace the true target bounding boxes of the labeled samples with the predicted target bounding boxes of the high-confidence unlabeled samples from the teacher network. That's all; Step 5.7: Since the determination method for labeled and unlabeled samples is the same, this step will be described using the procedure for labeled samples. Set the IoU threshold to 0.5, and in the two-dimensional IoU matrix... The labeled samples can be filtered out. The set of true target bounding boxes greater than the IoU threshold corresponding to the predicted target bounding boxes is used as the target bounding box with the maximum value in the set. Each predicted target bounding box corresponds to a matched ground truth target bounding box, if the matrix No. If the set of true target bounding boxes satisfying the conditions is empty, then the first row will be... The predicted target bounding box corresponds to the background; for the labeled sample, the first... Predicted target bounding box The corresponding matching true target bounding box is The true category label corresponding to the true target bounding box is (If it includes the first) Each category, then (otherwise it is 0); Step 5.8: The action localization loss for both labeled and unlabeled samples is the GIoU loss. The GIoU loss for labeled samples is... ,in Indicates the labeled sample number The smallest rectangular bounding box formed by the target bounding box and its corresponding ground truth bounding box. The area enclosed by the bounding box; GIoU loss for unlabeled samples. Simply replace the ground truth labels of the labeled samples with the predicted bounding boxes of the target objects corresponding to the high-confidence unlabeled samples in the teacher network. That is, the total motion positioning loss ; Step 5.9: Action Classification Loss The classification loss is used to measure the difference between the probability distribution of the action class predicted by the model and the true action class. On labeled samples, this is the classification loss. Calculated using the cross-entropy loss function. On unlabeled samples, the classification loss for unlabeled samples Form and Consistent, only the real labels corresponding to the labeled samples are replaced with the teacher network. The generated high-confidence pseudo-labels are used to calculate the total action classification loss. ; Step 5.10: The total loss in each batch is obtained from the action localization loss in step 5.8. Step 5.9 Obtaining the action classification loss Step 2.3 causal consistency loss and the prototype loss obtained in step 4.3 The composition, and the specific calculation formula are as follows: , in, These represent the weights of the action classification loss, action localization loss, causal context loss, and prototype loss, respectively (usually taken as...). ); Step 5.11: After the student model completes the total loss calculation for labeled and unlabeled samples, backpropagation is performed on the 3D convolutional neural network, object detector, context enhancement module, and classification and regression heads of the student model to calculate gradients. Stochastic gradient descent is used to update the parameters, thereby minimizing the loss of the student model on the current batch of data. After the student model parameters are updated, EMA is used to dynamically update the teacher model parameters to ensure that the teacher model maintains a relatively smooth and stable feature representation during training. The teacher model parameters are updated in the following way: ,in, Indicates the first The learnable parameters of the teacher model in the next iteration Indicates the first Learnable parameters of the student model after the next iteration This represents the EMA attenuation factor (usually taken as...). After each round of training, repeat steps 5.8 to 5.10 until the student model converges on the validation set. The trained student model is then regarded as the final trained semi-supervised spatiotemporal action detection model.

[0036] The performance evaluation metric for the model trained in step 5 is frame-level mean average precision (f-mAP). f-mAP measures the accuracy of the model in recognizing actions frame by frame in a video and is the core metric for evaluating the frame-by-frame detection effect of spatiotemporal action monitoring.

[0037] Step 6: Extract the target bounding box, target-level spatiotemporal feature tensor, and target bounding box neighborhood background region feature tensor from the new video. Input them into the trained spatiotemporal action detection model and output the action category probability and the corresponding target bounding box set to obtain the complete spatiotemporal action detection result.

[0038] Step 6.1: Perform frame sampling and segmentation on the video to be detected according to Steps 1.1 and 1.2 to obtain keyframes and their corresponding video segment sets. During inference, the unlabeled sample selection module and pseudo-label calibration module based on uncertainty constraints are removed. Each video segment is input into the pre-trained student model for target detection and corresponding action classification, outputting the corresponding target bounding box sequence set. and the corresponding action category probability score distribution set When the first The target bounding box in the th If the predicted probability for a given category is greater than a threshold (usually 0.5), then the target is considered to be performing the first category. For each action, output a set of action category labels that satisfy the threshold condition. ,in, This indicates the number of target bounding boxes predicted for the current video segment. Indicates the total number of action categories; Step 6.2: Based on the target bounding box sequence set obtained in Step 6.1 With action category tag set This involves determining the start and end times of each action in the video; specifically, matching the target detection results in adjacent video segments: to achieve cross-frame target association, for adjacent video segments... and The corresponding target bounding box set and Construct the cost matrix , ,in, Indicates video clip The Targets and video clips The The cost of matching between targets This indicates the calculation of intersection-union ratio. For video clips The The center point coordinates of the target Indicates video clip The Targets and video clips The The difference in area ratio between the target bounding boxes This is a balancing coefficient used to adjust the influence weights of different cost terms (usually taken as...). , , Subsequently, the Hungarian algorithm is used to solve for optimal matching on the cost matrix to obtain the video segment. and The optimal target matching set between ,in, Indicates video clip The Targets and video clips The The pairing formed between target bounding boxes Indicates video clip and video clips The number of target bounding box pairs; Then, based on the optimal target matching set The paired target bounding boxes are then concatenated front to back to obtain the action pipeline set for the corresponding target. ,in, This represents the total number of action pipelines, ultimately determined by the action pipeline set. and action category tag set Determine the start and end times of all actions in the new video sequence and the corresponding set of target bounding boxes at those times. ,in, These represent the action pipelines classified as numbered... The start and end times of each action. Indicates the first The action within a time period The target bounding box at the corresponding time. Indicates the first The action category of each action. This indicates the total number of actions in the new video sequence.

[0039] Figure 3 A schematic diagram of the structure of a semi-supervised spatiotemporal action detection device 9100 based on semantic causal pseudo-marker uncertainty constraints provided in this application embodiment is shown below. Figure 3 As shown, the device may include: an acquisition unit 9101, a processing unit 9102, and a detection unit 9103. The main functions of each component module are as follows: The acquisition unit 9101 is used to acquire the semi-supervised spatiotemporal action detection model to be trained and the original spatiotemporal action detection video dataset (including training set and test set), wherein the training set includes labeled samples and unlabeled samples, and the test set only includes labeled samples. The processing unit 9102 is used to input the original video dataset of spatiotemporal motion detection into the semi-supervised spatiotemporal motion detection model to be trained, and to process the semi-supervised spatiotemporal motion detection model to be trained to obtain the semi-supervised spatiotemporal motion detection model to be detected. The detection unit 9103 is used to input the test set video into the semi-supervised spatiotemporal motion detection model to be detected, output the category, spatial location and start and end time of each action in the video, and obtain the complete spatiotemporal motion detection results of the test set video.

[0040] This application provides a semi-supervised spatiotemporal action detection device 5000 based on semantic causal pseudo-label uncertainty constraints. The device includes: at least one processor and a storage medium communicatively connected to the at least one processor. The storage medium stores instructions that can be executed by the at least one processor. When the instructions are executed, they can drive the at least one processor to implement the semi-supervised spatiotemporal action detection method based on semantic causal pseudo-label uncertainty constraints described in the above embodiments.

[0041] Figure 4 An example structure for implementing the device described in the embodiments of this application is illustrated. This device can be embodied as an intelligent sensing mobile terminal such as an intelligent monitoring host, a robot vision terminal, a portable computer, or an edge computing node. It should be noted that... Figure 4 This is for illustrative purposes only and should not be construed as a limitation on the functionality and scope of this application.

[0042] Reference Figure 4 The hardware architecture of the semi-supervised spatiotemporal motion detection device 5000 based on semantic causal pseudo-label uncertainty constraints achieves interconnection of core components through the system bus 5001. This architecture can perform various appropriate operations (e.g., storing intermediate features of the model inference stage and various instructions required for device operation) based on programs stored in read-only memory (ROM) 5006 or loaded from storage device 5003 (e.g., hard disk drive or solid-state drive) into running memory 5007 (e.g., random access memory). The processor 5002 (e.g., central processing unit or graphics processing unit) is responsible for controlling the operation of the entire device, reading and executing program instructions from running memory 5007, specifically implementing the core computational tasks of the semi-supervised spatiotemporal action detection method based on semantic causal pseudo-label uncertainty constraints. The input / output device 5005 can connect to various external devices, including input devices (e.g., camera, control panel, image sensor, touchscreen, keyboard), output devices (e.g., speaker, monitor), storage device 5003, and network device 5004. The network device 5004 is used to receive video data, transmit detection results, or perform remote model updates via wired or wireless communication networks, further realizing cloud data synchronization and remote inference deployment. It should be emphasized that... Figure 4 The diagram shown is merely a feasible structural example of the device. In actual applications, the system architecture and its functional components can be replaced or more or fewer system devices can be added according to specific application requirements.

[0043] Specifically, according to the embodiments disclosed in this application, the processes described in the flowcharts herein can be implemented in the form of computer software programs. Specifically, this application includes a computer program product comprising a computer program stored in a computer-readable medium, which includes program code for executing the semi-supervised spatiotemporal action detection method based on semantic causal pseudo-label uncertainty constraints shown in the flowcharts. In these embodiments, the computer program can be downloaded and installed from storage device 5003 via a network. When the program is executed by processor 5002, it will perform the functions defined in the embodiments disclosed in this application.

[0044] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, modules, or functions can be recombinated or combined in a suitable manner in any one or more embodiments or examples to achieve the overall technical solution described.

[0045] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the semi-supervised spatiotemporal action detection method based on semantic causal pseudo-marker uncertainty constraints in the above embodiments.

[0046] The computer-readable storage media involved in this application include, but are not limited to, USB flash drives, portable hard disks, hard disk drives, semiconductor storage modules, magnetic or optical storage devices, fiber optic media, random access memory (RAM), read-only memory (ROM), and any combination thereof, as well as physical implementations such as electrical connection devices with wires and wireless communication carriers. Such media can be deployed independently or in combination, and the computer-executable instructions stored therein can be directly executed or collaboratively processed by the device's processor to implement the methods defined in the foregoing embodiments. It should be noted that the aforementioned media forms and data transmission methods (including wired or wireless signal transmission channels) are illustrative and should not be considered as limiting the scope of protection of this application.

[0047] The aforementioned computer-readable storage medium may be included in a semi-supervised spatiotemporal motion detection device based on semantic causal pseudo-label uncertainty constraints; or it may exist independently and not be assembled into a semi-supervised spatiotemporal motion detection device based on semantic causal pseudo-label uncertainty constraints.

[0048] The flowcharts and block diagrams in the accompanying drawings illustrate the potential architecture, functional components, and execution modes of systems, methods, and computer program products according to various embodiments of this application. In these diagrams, each block corresponds to a module, program fragment, or code section, containing one or more executable instructions for implementing a specific logical function. In some alternative embodiments, the order of function execution shown in the figures may differ from the order presented in the diagrams. For example, two consecutive blocks may execute synchronously, or in a specific scenario, in reverse order; the specific execution method needs to be determined based on actual requirements. Furthermore, the single blocks or combinations of blocks in the block diagrams or flowcharts can achieve the specified function either through a dedicated hardware system or by integrating dedicated hardware with computer instructions.

[0049] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A semi-supervised spatio-temporal action detection method based on semantic causal pseudo-label uncertainty constraint, characterized in that, The method comprises the following steps: step 1: sampling and segmenting the original video of the labeled sample set and the unlabeled sample set collected by the robot into a video segment set at a fixed frame rate, extracting a target bounding box and a target level spatio-temporal feature tensor by using a two-dimensional convolution and a three-dimensional convolution network, and obtaining a target bounding box neighborhood background region feature tensor by using an average pooling operation; Step 2: constructing a semantic causal context enhancement module, inputting the target level spatio-temporal feature tensor and the target bounding box neighborhood background region feature tensor, and outputting a target level context enhanced feature tensor; Step 3: constructing an uncertainty constrained unlabeled sample selection module, inputting the target level context enhanced feature tensor, and outputting selected unlabeled samples; Step 4: constructing a pseudo-label calibration module, inputting the context enhanced features of the labeled sample set and the reliable unlabeled sample set and the real labels corresponding to the labeled samples, constructing an initial class prototype on the labeled samples, calibrating the pseudo labels of the reliable unlabeled samples, and outputting the calibrated pseudo labels of each target of the reliable unlabeled samples; Step 5: constructing a semi-supervised spatio-temporal action detection model, combining the target level context enhanced features and the high-quality pseudo labels of the unlabeled samples to calculate a loss, and optimizing the spatio-temporal action detection model by using a stochastic gradient descent algorithm; Step 6: extracting a target bounding box, a target level spatio-temporal feature tensor and a target bounding box neighborhood background region feature tensor from a new video, inputting the trained spatio-temporal action detection model, outputting an action class probability and a corresponding target bounding box set, and obtaining a complete spatio-temporal action detection result.

2. The semi-supervised spatio-temporal action detection method of claim 1, wherein, The step 1 specifically comprises: step 1.1: sampling the video of the labeled sample set and the unlabeled sample set at a fixed frame rate to obtain a video frame sequence; step 1.2: segmenting the video frame sequence by using a sliding window to obtain a video segment set; step 1.3: extracting a spatio-temporal feature map of each video segment by using a three-dimensional convolutional neural network; step 1.4: extracting a target bounding box set by using a two-dimensional convolution target detector on the key frame of the video segment; step 1.5: mapping the target bounding box in the key frame to the spatio-temporal feature map; step 1.6: performing a region of interest alignment operation in the mapped target bounding box region to extract a target level spatio-temporal feature tensor; and step 1.7: dividing a plurality of patches in the neighborhood of the target bounding box, performing a region of interest alignment operation on each patch, and extracting a background patch feature set and a geometric description thereof.

3. The semi-supervised spatio-temporal action detection method of claim 1, wherein, The semantic causal context enhancement module in step 2 includes a semantic geometric graph sub-module, a causal intervention sub-module, and a context feature fusion sub-module, specifically comprising: step 2.1: the semantic geometric graph sub-module constructs node embedding according to the target feature and the background patch feature, calculates the initial edge weight combining the geometric relationship and the dynamic index, and constructs the semantic geometric graph; step 2.2: the causal intervention sub-module dynamically adjusts the edge weight through the gating network, enhances the semantic related static region feature, and suppresses the dynamic interference region feature; step 2.3: the context feature fusion sub-module fuses the target feature and the background feature after causal intervention through graph message aggregation, and outputs the context enhanced feature.

4. The semi-supervised spatio-temporal action detection method of claim 1, wherein, The uncertainty constrained unlabeled sample selection module in step 3 includes: step 3.1: an uncertainty estimation module based on an energy function, which calculates the uncertainty of the unlabeled sample using the energy function; step 3.2: an unlabeled sample selection module, which selects high-confidence unlabeled samples according to the uncertainty threshold to form a reliable unlabeled sample set.

5. The semi-supervised spatio-temporal action detection method of claim 1, wherein, The pseudo-label calibration module in step 4 includes: step 4.1: a class prototype construction sub-module, which calculates the prototype vector of each class based on the context enhanced feature of the labeled sample; step 4.2: a prototype consistency pseudo-label calibration module, which calculates the similarity between the unlabeled sample feature and the prototype vector, and uses a calibration function to weight and calibrate the pseudo-label, outputting the calibrated pseudo-label.

6. The semi-supervised spatio-temporal action detection method of claim 1, wherein, The process of constructing and training the semi-supervised spatio-temporal action detection model in step 5 specifically includes: Step 5.1: constructing a spatio-temporal action detection model based on a teacher-student network architecture, wherein the student network and the teacher network have the same structure, both including a three-dimensional convolution feature extractor, a two-dimensional target detector, a semantic causal context enhancement module, and a classification regression head; Step 5.2: initializing the class prototype using the labeled sample, and generating pseudo-labels for high-confidence unlabeled samples through the teacher network, including target bounding boxes and action class score distribution; Step 5.3: in each training batch, input the labeled sample and the filtered unlabeled sample into the student network to calculate their prediction results in target detection and action classification, respectively; Step 5.4: according to the matching relationship between the prediction results and the true label or pseudo-label, the following loss functions are calculated: Action localization loss: based on generalized intersection over union to measure the spatial consistency of the predicted bounding box and the true or pseudo-label bounding box; Action classification loss: based on cross-entropy loss to measure the consistency of the predicted class distribution and the true or pseudo-label class distribution; Causal consistency loss: used to constrain the adaptive enhancement and suppression process of the background feature in the semantic causal context enhancement module; Prototype consistency loss: used to constrain the semantic alignment between the unlabeled sample feature and the class prototype; Step 5.5: the total loss is the weighted sum of the above loss functions, and the student network parameters are updated through the stochastic gradient descent algorithm; Step 5.6: the teacher network parameters are updated smoothly from the student network through exponential moving average to maintain training stability; Step 5.7: Repeat the above training process until the model converges on the validation set to obtain the final trained spatiotemporal action detection model.

7. The semi-supervised spatio-temporal action detection method of claim 1, wherein, The specific process of the step 6 includes: Step 6.1: Sample and divide the video to be detected into a video segment set at a fixed frame rate, and input each video segment into the trained student model to output the target bounding box sequence in each segment and the corresponding action class probability distribution; Step 6.2: According to the set action class probability threshold, filter out the action class labels with high confidence; Step 6.3: Cross-frame matching of target bounding boxes in adjacent video segments is performed by constructing a cost matrix and using the Hungarian algorithm to solve the optimal matching to form a continuous target action pipeline; Step 6.4: According to the action pipeline and the corresponding class label, determine the start and end time of each action in the video and its spatial bounding box at each time, and output the complete spatiotemporal action detection result.

8. A device for semi-supervised spatio-temporal action detection based on semantic causal pseudo-label uncertainty constraints, configured to perform the method according to any one of claims 1 to 7, characterized in that, The device includes an acquisition unit for sampling and dividing the original video of the labeled sample and the unlabeled sample into a video segment set, extracting a target bounding box sequence set, a target level spatiotemporal feature tensor, and a target bounding box neighborhood background region feature tensor through a two-dimensional convolution and a three-dimensional convolution neural network; A processing unit is configured to construct a semantic causal context enhancement module, input the target level spatiotemporal feature tensor and the target bounding box neighborhood background region feature tensor into the module, and output a target level context enhanced feature tensor; construct an uncertainty constrained unlabeled sample selection module to output a reliable unlabeled sample set; construct a pseudo-label calibration module, input the context enhanced features of the labeled sample, the context enhanced features of the reliable unlabeled sample, and the real label of the labeled sample, and output the calibrated pseudo-label of each target of the reliable unlabeled sample; calculate the total loss including action localization loss, action classification loss, causal consistency loss, and prototype consistency loss based on the target level context enhanced feature, the real label of the labeled sample, and the calibrated pseudo-label, and train the semi-supervised spatiotemporal action detection model using the teacher-student framework until the model converges; A detection unit is configured to input a new video sequence into the trained student model after the same preprocessing as the acquisition unit to obtain a predicted target bounding box sequence set and an action class probability set, and thereby obtain the action start time, end time, action class, and target bounding box.

9. A device for semi-supervised spatio-temporal action detection based on semantic causal pseudo-label uncertainty constraints, characterized in that, The device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the semi-supervised spatiotemporal action detection method based on semantic causal pseudo-label uncertainty constraint according to any one of claims 1 to 7.

10. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the semi-supervised spatiotemporal action detection method based on semantic causal pseudo-label uncertainty constraint according to any one of claims 1 to 7.

Images