Spatiotemporal phase alignment contrastive learning method for zero-shot compositional action recognition

Abstract

The application belongs to the technical field of computer vision and pattern recognition, and particularly relates to a spatiotemporal phase alignment contrast learning method for zero-shot combined action recognition, comprising data preprocessing and enhancement, construction of a timing-semantic collaborative network architecture, model training and optimization, model evaluation and tuning, and model deployment and application; the application has obvious recognition accuracy improvement for unobserved verb-object combined actions through the fine-grained timing modeling of the TCS module and the stage-level semantic alignment of the PASA module, especially in the action timing difference and semantic detail distinction, and the framework has stable improvement in top-1 and top-5 accuracy rates on mainstream benchmark datasets.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and pattern recognition technology, specifically relating to a spatiotemporal phase alignment contrastive learning method for zero-shot combined action recognition. Background Technology

[0002] For the Zero-Shot Combined Action Recognition (ZS-CAR) task, existing technologies mainly revolve around temporal feature modeling, cross-modal semantic alignment, and stage segmentation: temporal augmentation-based methods (such as SimCLR-Video) optimize video-level global features through fixed data augmentation and contrastive learning; cross-modal alignment methods (such as ActionCLIP) utilize the comparison of action label text features with video features; and stage modeling methods (such as Two-Stream I3D) rely on manually fixing and aggregating stage features in segments. The core goal of all these methods is to improve the accuracy and generalization ability of action recognition.

[0003] However, existing technologies have significant limitations: First, temporal modeling is coarse, with fixed enhancement strategies and global feature optimization ignoring frame-level fine-grained dynamics, making it unable to distinguish between temporally opposite actions such as "push" and "pull." This stems from the lack of dynamic perturbations and frame-level modeling specifically designed for the temporal characteristics of actions. Second, cross-modal alignment is coarse-grained, using only single-label text to align with the overall video, resulting in loose semantic associations and poor generalization to novel verb-object combinations due to the lack of stage-level text descriptions and dynamic bindings. Third, stage division lacks adaptability, with fixed rules unable to match the temporal distribution of different actions such as "place" and "transport," stemming from the lack of data-driven learning for stage weights. Fourth, module collaboration is insufficient, with temporal enhancement and semantic alignment trained independently without closed-loop feedback, leading to a disconnect between temporal optimization and semantic requirements, ultimately limiting recognition accuracy in zero-shot scenarios.

[0004] To address the above issues, this application solves the problem of accurately recognizing "unseen verb-object combination actions" through visual-text cross-modal collaboration and refined process feature optimization. Core application areas include technical scenarios requiring dynamic action semantic analysis, such as intelligent monitoring, human-computer interaction, behavior understanding in autonomous driving scenarios, and robot action imitation learning. Summary of the Invention

[0005] The purpose of this invention is to provide a spatiotemporal phase alignment contrastive learning method for zero-shot combined action recognition that is accurate and has strong generalization ability.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A spatiotemporal phase-aligned contrastive learning method for zero-shot combined action recognition includes the following steps: S1. Data Preprocessing and Enhancement: First, the input video dataset is standardized and uniformly adjusted to a standard size of 256×256. Dynamic temporal enhancement strategies are introduced, including frame-level random flipping, temporal local rearrangement, and multi-scale scaling. Distributed preprocessing is implemented using a PyTorch Lightning data pipeline. S2. Construct a temporal-semantic collaborative network architecture; introduce a dual-module collaborative structure of TCS module and PASA module. The two modules form a collaboration by sharing backbone network features. The refined frame features of TCS module provide high-quality temporal input for PASA module, and the semantic loss of PASA module guides the feature optimization direction of TCS module in reverse. S3. Model Training and Optimization: Input the preprocessed video frame features into the network and calculate the training loss through the combined loss function; The total loss is composed of the combined contrast loss of the TCS module and the stage-level InfoNCE loss of the PASA module; The AdamW optimizer is used, combined with the cosine annealing learning rate scheduling strategy, and the network parameters are updated through backpropagation. S4. Model Evaluation and Tuning: During training, the model's accuracy on the zero-shot combined action recognition task is monitored in real time using the validation set. Overfitting is suppressed by adjusting hyperparameters such as loss weights and stage partitioning granularity. After the model training is completed, the generalization performance is evaluated on the test set, with a focus on verifying the ability to recognize unseen verb-object combined actions. During the inference stage, the frame-level features of the TCS module and the stage features of the PASA module are weighted and fused to output the final result. S5. Model Deployment and Application: The trained model parameters are serialized and saved. During deployment, the model is loaded and adapted to the input format of the actual application scenario. During inference, the fused single-branch feature output structure is adopted.

[0007] Furthermore, in step S2, the TCS module constructs contrast samples through frame-level local perturbation, global temporal reversal, cross-sample frame mixing, and dynamic masking to refine and optimize the frame-level features output by the backbone network, thereby enhancing the ability to capture short-term temporal patterns of actions. The PASA module is based on a dynamic stage projection mechanism, which decomposes the optimized frame-level features into three stages: start, execution, and end. The weight tensor of each stage is dynamically allocated through gradient learning. At the same time, LLM is used to generate stage text descriptions, which are then converted into text features by the CLIP encoder to achieve visual-text stage-level semantic alignment.

[0008] The beneficial effects of this invention are as follows: This invention significantly improves the accuracy of recognizing unseen verb-object combination actions by combining the fine-grained temporal modeling of the TCS module with the stage-level semantic alignment of the PASA module. It is particularly effective in distinguishing action temporal differences and semantic details. Furthermore, the framework has achieved stable improvements in top-1 and top-5 accuracy on mainstream benchmark datasets.

[0009] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0010] Figure 1 This is a schematic diagram illustrating the steps of a spatiotemporal phase alignment contrastive learning method for zero-sample combined action recognition, as shown in an embodiment of the present invention.

[0011] Figure 2 The diagram illustrates the thermodynamics of different comparison methods.

[0012] Figure 3 This is a schematic diagram of t-SNE for different comparison methods. Detailed Implementation

[0013] Please see Figure 1 The preferred embodiment of this application presents a spatiotemporal phase alignment contrastive learning method for zero-shot combined action recognition that is accurate and has strong generalization ability, comprising the following steps: S1. Data Preprocessing and Augmentation: First, the input video dataset is standardized to a standard size of 256×256 to meet the feature extraction requirements of subsequent modules. Dynamic temporal augmentation strategies are introduced, including frame-level random flipping, temporal local rearrangement (e.g., shuffling the order of 3 consecutive frames), and multi-scale scaling (random sampling from 224×224 to 288×288) to enhance the model's robustness to temporal variations in actions. Distributed preprocessing is implemented using a PyTorch Lightning data pipeline, supporting parallel loading across multiple nodes and improving the processing efficiency of large-scale video data. S2. Construct a temporal-semantic collaborative network architecture; introduce a dual-module collaborative structure of TCS module and PASA module. The two modules cooperate by sharing backbone network features. The refined frame features of TCS module provide high-quality temporal input for PASA module, and the semantic loss of PASA module guides the feature optimization direction of TCS module. This architecture is compatible with basic backbone networks such as ResNet-50 and Swin Transformer. Taking Motion Transformer as an example, initialize its window attention layer, temporal convolution module and cross-stage connection layer. S3. Model Training and Optimization: The preprocessed video frame features are input into the network, and the training loss is calculated using a combined loss function. The total loss consists of the combined contrast loss of the TCS module (frame-level loss weight 0.7, global loss weight 0.3, temperature parameter τ=0.3) and the stage-level InfoNCE loss of the PASA module (weight parameters λ_TCS and λ_stage are determined through ablation experiments). The AdamW optimizer is used, combined with a cosine annealing learning rate scheduling strategy, and the network parameters (including the positive and negative sample contrast parameters of the TCS module, the stage weight tensor of the PASA module, and the backbone network parameters) are updated through backpropagation, which minimizes the total loss. S4. Model Evaluation and Tuning: During training, the model's accuracy (e.g., top-1 accuracy) on zero-shot combined action recognition tasks is monitored in real time using a validation set. Overfitting is suppressed by adjusting hyperparameters such as loss weights and stage partitioning granularity. After model training is completed, generalization performance is evaluated on a test set, with a focus on verifying the ability to recognize unseen verb-object combined actions. During the inference phase, the frame-level features of the TCS module and the stage features of the PASA module are weighted and fused to output the final result, ensuring improved inference efficiency without increasing additional computation. S5. Model Deployment and Application: The trained model parameters (including the weights of the backbone network, TCS module, and PASA module) are serialized and saved. During deployment, the model is loaded and adapted to the input format of the actual application scenario (such as intelligent monitoring and human-computer interaction). During inference, a fused single-branch feature output structure is adopted to ensure recognition accuracy while meeting real-time requirements, thereby achieving rapid semantic understanding and classification of dynamic actions.

[0014] Preferably, the TCS module in step S2 constructs contrast sample pairs through four negative sample generation strategies (frame-level local perturbation, global temporal reversal, cross-sample frame mixing, and dynamic masking), which refines the frame-level features output by the backbone network and enhances the ability to capture short-term temporal patterns of actions (such as abrupt changes in direction and amplitude changes). The PASA module is based on a dynamic stage projection mechanism, which decomposes the optimized frame-level features into three stages: start, execution, and end. The weight tensor of each stage is dynamically allocated through gradient learning (initially preset according to early / mid / late frames, and then adaptively adjusted in combination with temporal cues). At the same time, LLM is used to generate stage text descriptions, which are then converted into text features by the CLIP encoder to achieve visual-text stage-level semantic alignment.

[0015] like Figure 2The video frames are processed using both the baseline method and the method described above to output heatmaps of spatial attention. Attention intensity is encoded using colors ranging from blue (low) to red (high). The baseline method has lower accuracy in diverting attention, often missing key areas related to actions; while the method described above accurately focuses attention on relevant objects and actions, demonstrating an advantage in highlighting prominent spatial features.

[0016] like Figure 3 The features of eight error-prone actions were embedded into the t-SNE plots obtained by the baseline method and the method described above. The left plot shows the t-SNE plot obtained by the baseline method, and the right plot shows the t-SNE plot obtained by the method described above. In the left plot, the clusters show obvious overlap and dispersion, indicating that semantically similar operations cannot be aggregated. In contrast, the right plot shows more compact clusters with clearer boundaries between them. Even though minor overlaps still exist, the overall clustering structure is significantly more organized, highlighting the model's ability to learn discriminative features that tightly group similar behaviors and clearly separate different behaviors. This supports the method's ability to capture semantic nuances to improve generalization.

[0017] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0018] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A spatiotemporal phase-aligned contrastive learning method for zero-shot combined action recognition, characterized in that, Includes the following steps: S1. Data Preprocessing and Enhancement: First, the input video dataset is standardized and uniformly adjusted to a standard size of 256×256. Dynamic temporal enhancement strategies are introduced, including frame-level random flipping, temporal local rearrangement, and multi-scale scaling. Distributed preprocessing is implemented using a PyTorch Lightning data pipeline. S2. Construct a temporal-semantic collaborative network architecture; introduce a dual-module collaborative structure of TCS module and PASA module. The two modules form a collaboration by sharing backbone network features. The refined frame features of TCS module provide high-quality temporal input for PASA module, and the semantic loss of PASA module guides the feature optimization direction of TCS module in reverse. S3. Model Training and Optimization: Input the preprocessed video frame features into the network and calculate the training loss through the combined loss function; The total loss is composed of the combined contrast loss of the TCS module and the stage-level InfoNCE loss of the PASA module; The AdamW optimizer is used, combined with the cosine annealing learning rate scheduling strategy, and the network parameters are updated through backpropagation. S4. Model Evaluation and Tuning: During training, the model's accuracy on the zero-shot combined action recognition task is monitored in real time using the validation set. Overfitting is suppressed by adjusting hyperparameters such as loss weights and stage partitioning granularity. After the model training is completed, the generalization performance is evaluated on the test set, with a focus on verifying the ability to recognize unseen verb-object combined actions. During the inference stage, the frame-level features of the TCS module and the stage features of the PASA module are weighted and fused to output the final result. S5. Model Deployment and Application: The trained model parameters are serialized and saved. During deployment, the model is loaded and adapted to the input format of the actual application scenario. During inference, the fused single-branch feature output structure is adopted.

2. The spatiotemporal phase alignment contrastive learning method for zero-sample combined action recognition as described in claim 1, characterized in that, The TCS module in step S2 constructs comparison samples through frame-level local perturbation, global temporal reversal, cross-sample frame mixing, and dynamic masking, and performs fine optimization on the frame-level features output by the backbone network to enhance the ability to capture short-term temporal patterns of actions. The PASA module is based on a dynamic stage projection mechanism. It decomposes the optimized frame-level features into three stages: start, execution, and end. The weight tensor of each stage is dynamically allocated through gradient learning. At the same time, LLM is used to generate stage text descriptions, which are then converted into text features by the CLIP encoder to achieve visual-text stage-level semantic alignment.

Images