A long-term action prediction method based on spatiotemporal prior guidance and selective state space denoising

Abstract

The application discloses a long-term action prediction method based on space-time prior guidance and selective state space denoising, and belongs to the field of video understanding and action sequence prediction. The application constructs a whole-process space-time prior constraint framework: extracting video frame features and dividing observation and prediction intervals to construct a basic condition vector; relying on an action transition matrix, an action semantic prototype and an action duration prior to realize future segment structure perception initialization; adopting a selective state space denoising network composed of multiple MANTA denoising blocks; injecting semantic prototypes in the middle layer according to a preset starting layer and an injection interval to realize progressive semantic enhancement; and completing training by combining a frame-level L2 loss and a bidirectional KL transfer loss. The application distinguishes observation frames and mask frames through a gating self-adaption, adapts to long sequences with linear complexity, effectively suppresses action disorder jumping, can improve prediction accuracy through multi-dataset verification, and is suitable for first-view driving and assembly action prediction scenes.

Description

Technical Field

[0001] This invention belongs to the fields of video understanding, temporal modeling, diffusion generation model and action prediction technology. Specifically, it relates to a long-term action prediction method based on spatiotemporal prior guidance and selective state space denoising. It can predict the future long-term frame-by-frame action categories based on observed video segments, and obtain the action duration and action evolution and transition process from the predicted sequence. Background Technology

[0002] Long-term motion prediction tasks predict complete frame-by-frame motion sequences over a long period of time based on preceding video observations. These tasks are widely used in scenarios such as first-person human-computer interaction, autonomous driving risk warning, and robot assembly assistance analysis. Existing technologies fall into two categories: deterministic prediction methods and stochastic diffusion prediction methods. Deterministic methods output only a single prediction sequence, making it difficult to fully represent multiple plausible future behaviors corresponding to the same observation segment. Stochastic diffusion models can generate multiple candidate motion sequences, but they suffer from four major drawbacks: The future prediction fragment is initialized with only zeros, without any action structure or temporal constraints to guide it, and the initial generated benchmark deviates from the actual behavior pattern. The long-span diffusion denoising iterative process lacks continuous semantic constraints, and action semantic drift and logical action jumps occur as the prediction time extends; The training supervision only uses single-frame classification loss and does not constrain the global action transfer logic. The single-frame prediction is accurate, but the overall action flow is unreasonable. Traditional temporal modeling networks (dilated convolution, Transformer) struggle to balance global receptive field and computational efficiency, incur significant overhead for long sequence inference, and cannot differentiate between observed real frames and masked frames to be predicted.

[0003] Existing diffusion models based on Mamba (Selective State Space SSM) mainly focus on replacing the temporal backbone network, without designing a bidirectional temporal modeling structure for long-term action prediction, and without matching action-specific spatiotemporal prior constraints throughout the entire process; traditional action temporal constraints only add loss at the model output layer, making it difficult to correct prediction biases in advance during initialization and denoising, and there is still room for further improvement. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a long-term action prediction method based on spatiotemporal prior guidance and selective state-space denoising. By modeling long sequences using a selective state-space layer based on Mamba, a global receptive field can be formed while maintaining linear complexity. Furthermore, data dependency gating is used to adaptively process observed and future segments. Combining this type of denoising block with spatiotemporal prior guidance helps to further improve the efficiency and accuracy of long-term action prediction.

[0005] To achieve the above objectives, the technical solution of the present invention is implemented as follows: A long-term action prediction method based on spatiotemporal prior guidance and selective state-space denoising includes the following steps: Step S1: Obtain the input video sequence, extract frame-by-frame visual features using a feature extractor, and base the extraction on the observation ratio. Predicted proportion Divide the observed segment into the future prediction segment and construct the basic condition vector for diffusion prediction. ;in For the visual features of the i-th frame, For the number of observation frames, The total number of video frames, the number of predicted frames. The positions of future predicted segments in the basic condition vector are initially set to zero; Step S2: Obtain the forward action transition matrix based on training set statistics. Backward action transition matrix Action semantic prototype dictionary Average action duration table Three types of spatiotemporal priors are used. Based on the visual features of the last few frames of the observed segment, the current action category is estimated through action prototype matching. Multiple pseudo-future action sequences are then generated based on this action category. These sequences are expanded into frame-level pseudo-label sequences by combining the average action duration. All candidate pseudo-features are fused using a weight that decays with prediction distance to generate a future-aware conditional vector. ; Step S3: Construct a selective state-space denoising generator consisting of stacked MANTA blocks. Each MANTA block includes layer normalization, a bidirectional selective state-space layer (BSSL), a channel mixing feedforward layer, and residual connections. During the diffusion iterative denoising process, the noise action label variables and the future perception condition vector are... Fusion input denoising generator; setting semantic injection start layer With injection interval In the intermediate layer of the network that meets the conditions, the semantic prototype of the action is retrieved based on the pseudo-label of the current diffusion step and fused with the intermediate features to achieve progressive semantic enhancement; Step S4: Construct a joint loss function for the selective state-space denoising generator described in Step S3. Complete model training; among which For frame-level reconstruction loss, Forward transfer consistency loss, For backward transfer consistency loss, The weights are used to transfer the loss. After training, multiple candidate future action sequences are generated through random diffusion sampling. Based on the average class confidence of the candidate sequences, the prediction distribution, and the consistency score between the candidate sequence transition matrix and the training set transition matrix, the long-term action prediction results are selected and output.

[0006] Furthermore, the frame-by-frame visual feature extraction formula in step S1 is as follows: ; For feature extractors, The input is the original image of the i-th frame of the video. The feature extractor can be any one of I3D, TSM, video Transformer or temporal convolutional network.

[0007] Furthermore, in step S2, the action semantic prototype dictionary Internal Single Category Action Semantic Prototype Generated from the average visual features of all frames corresponding to the action category in the training set; the weights of the multi-candidate pseudo-future feature fusion satisfy... j is the current frame number, and the weight decreases linearly with the distance from the observed segment.

[0008] Furthermore, in step S3, the bidirectional selective state space layer (BSSL) includes a parallel forward temporal path and a backward temporal path; the forward path processes the sequence along the positive time direction, while the backward path first reverses the sequence temporal order to complete the calculation and then restores the temporal order; the output features of the two paths are aggregated to obtain the overall temporal representation of BSSL.

[0009] Furthermore, both the forward and backward temporal paths include a one-dimensional convolutional layer, a nonlinear activation function, a selective state space layer, and a residual gating module. The selective state space layer employs an input-related selection gating mechanism to adaptively adjust the state update parameters based on input features, thereby distinguishing between observed real frames and future masked prediction frames.

[0010] Furthermore, the discrete state update logic of the selective state space layer is as follows:

[0011] In the formula: Input features at time t, Let be the state vector at time t. Output the time series features at time t, A t B t C t These are selective state parameters that change dynamically with the input.

[0012] Furthermore, in step S3, progressive semantic enhancement, the single-frame predicted pseudo-label calculation method is as follows:

[0013] in, Let C be the noise action label variable for the k-th diffusion step, and C be the total number of action categories. This represents the current network layer number.

[0014] Furthermore, in step S4, the frame-level reconstruction loss adopts L2 loss, expressed as follows: ; in, A matrix of real action labels. The action label matrix is ​​predicted for the k-th step of the diffusion model.

[0015] Furthermore, in step S4, both the forward and backward transition consistency loss adopt the KL divergence loss. Forward transfer loss

[0016] Backward transfer loss ; in, These are the forward and backward transition matrices of the predicted sequence, respectively. To obtain the true forward transition matrix and the true backward transition matrix from the training set, The summation index is used to indicate the summation of all subscripts ending in 1. The summation of the KL divergence between the predicted and labeled values ​​of the action sequence at the start of the action, where KL is the relative entropy; the true forward transition matrix and the true backward transition matrix are given a smoothing coefficient before normalization to avoid KL divergence instability caused by zero probability transitions.

[0017] Furthermore, the method is applicable to scenarios including first-person perspective behavior video prediction, vehicle driving scene action prediction, industrial assembly process action prediction, and all video analysis scenarios that require multi-step intensive long-term action sequence prediction.

[0018] Beneficial effects: Three-stage full pipeline spatiotemporal prior constraints: injecting action statistics priors in the initialization stage, continuous semantic prototype guidance in the diffusion middle layer, and global transfer loss supervision in the output layer, suppressing action jumps and temporal drift throughout the entire generation process; Innovative BSSL bidirectional selective state space structure: Compared with unidirectional Mamba, it simultaneously captures forward action evolution and backward sequence context, with linear computational complexity adapted to ultra-long prediction sequences, and built-in gating adaptively distinguishes between real observation frames and generated mask frames. Dual temporal consistency constraints: Simultaneous use of forward and backward action transition matrices KL loss to compensate for the lack of single-direction temporal constraints, greatly improving the logical rationality of long-process action sequences; Multi-candidate random generation capability: During the reasoning stage, multiple reasonable future action paths can be output, which can be adapted to scenarios with multiple possible behavior predictions such as autonomous driving and human-machine integration; Significant performance improvement in real-world testing: On the three major standard public datasets Breakfast, 50Salads, and Assembly101, the Mean MoC and Top-1 MoC metrics for both short-term and long-term predictions are comprehensively superior to baseline models such as Tri-gram, UAAA, DiffAnt, GTDA, and the basic MANTA. Attached Figure Description

[0019] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is an overall flowchart of the long-term action prediction method based on spatiotemporal prior guidance and selective state space denoising as described in an embodiment of the present invention. Figure 2 This diagram illustrates a comparison of the processing flow of the method of this invention with existing diffusion-based methods in long-term motion prediction tasks.

[0020] Figure 3 This is a schematic diagram of the structure of the structure-aware future initialization, MANTA / BSSL denoising module, and progressive semantic enhancement module of the present invention. Detailed Implementation

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0022] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] Glossary MANTA stands for Mamba for Action Prediction, which means Mamba network for action prediction.

[0024] Mamba stands for Selective State Space Model, which means a selective state space model used for efficient sequence modeling.

[0025] BSSL stands for Bidirectional Selective State-Space Layer.

[0026] Example 1 See appendix Figure 2 , attached Figure 2This diagram illustrates a comparison between the method of this invention and existing diffusion-based long-term action prediction processing flows. Traditional diffusion models simply feed the blank mask conditional vector directly into the denoising network, lacking two crucial constraint stages: pre-structural awareness initialization and progressive semantic enhancement in the diffusion layer. This makes long-term prediction prone to problems such as action semantic drift, disordered action jumps, and unreasonable action durations. This invention adds an appendix... Figure 1 The two core units shown are the structure-aware initialization module and the selective state-space denoising network. Spatiotemporal prior constraints are introduced throughout the entire process of initialization, iterative denoising, and model training. The following section will discuss these in conjunction with the appendix. Figure 1 Overall process, appendix Figure 3 The module's internal details are implemented step by step in a complete manner.

[0027] S1, Construction of Video Observation Information This step is used to obtain observational and conditional information from the input video that can be used by subsequent diffusion prediction models. The system first receives a length of... video frame sequence The video can come from a first-person view camera, a vehicle-mounted camera, an assembly station camera, or other visual acquisition devices capable of recording continuous motion processes.

[0028] Before performing this step, the video can undergo preprocessing such as frame sampling, size normalization, frame rate unification, feature caching, and label alignment. The above preprocessing does not change the core technical solution of this invention; its purpose is to enable video samples from different sources to be uniformly input into the subsequent feature extractor and diffusion generator.

[0029] S11: Observed Video Feature Acquisition Input video frame sequence Input Feature Extractor The frame-by-frame visual feature representation is obtained:

[0030] in, Indicates the first The visual features corresponding to each frame of video, with the feature dimension denoted as . The feature extractor can be I3D, TSM, video Transformer, temporal convolutional network, or other visual models capable of outputting frame-level or fragment-level representations.

[0031] S12: Generation of Prediction Interval Condition Vector Set observation scale and predicted proportion ,in ,make Indicates the length of the observed segment. Indicates the predicted length of the future segment. This represents the total sequence length that the model needs to process. The basic condition vector can be represented as:

[0032] S2, Constructing Appendix Figure 1 The structure-aware initialization module shown is used to fill in the spatiotemporal prior of blank future segments. This module executes the following logic in its entirety: This step addresses the issue of blank future segments in the basic conditional vector. Based on action transition patterns, action semantic prototypes, and action duration information obtained from training data, the system constructs a future-aware conditional vector for the future segment that has a reasonable trend but is not overly certain.

[0033] S21: Action Transfer and Duration Prior Construction The system calculates the forward action transition matrix based on frame-by-frame action annotations in the training set. ,in Indicates action Transition to action in the next frame The system also calculates the average number of frames per action in the training data for each action category, resulting in a table of average action duration. .

[0034] Specifically, let the total number of action categories be... Training Set No. Frame action tag is Then the action To action The forward transition probability can be obtained from the following normalized statistics:

[0035]

[0036] in, This indicates that for a period of time T, the action... Transfer to action The number of times, This indicates an indicator function that takes the value 1 if the condition within the parentheses is true, and 0 otherwise. This is a smoothing coefficient used to avoid zero-probability transitions when there are few samples of certain action categories. If a backward transition matrix needs to be constructed, it can be calculated in the same way. arrive The reverse transfer relationship.

[0037] For the duration of an action, define the action category. The set of consecutive segments appearing in the training set is , of which The length of each segment is The average duration of the action can then be expressed as:

[0038] In practical implementation, the action transition matrix can be obtained by counting and normalizing adjacent frame labels, or it can be extended to a higher-order transition matrix to utilize the action context within a longer history window. For scenarios with many action categories and uneven sample distribution, a smoothing term can also be added during the statistical analysis.

[0039] S22: Action Semantic Prototypes and Future-Aware Conditional Generation The system constructs an action semantic prototype dictionary. Each semantic prototype Indicates action category The representative vector in the feature space can be obtained by averaging the features of all training frames of that category, or by embedding visual language models, embedding category text descriptions, or fusing multiple features.

[0040] When the semantic prototype is obtained from the statistical features of the training video, let it belong to the action category. The training frame index set is Then the action semantic prototype can be represented as:

[0041] When generating future sensing conditions, the system takes the length of the end of the observation segment as... The window features are averaged and matched against the current action category in the action semantic prototype dictionary. Subsequently, the system generates a rolling sequence based on the forward action transition matrix. A pseudo-future action-level sequence is generated, and the action-level sequence is expanded into a frame-level pseudo-label sequence based on the average action duration table.

[0042] The expanded pseudo-labels are then mapped to corresponding semantic prototype features, forming candidate pseudo-future features. For multiple candidate pseudo-future features, the system fuses them using weights that decay linearly with the prediction time range. Future frames closer to the observed segment rely more on statistical priors, while future frames farther from the observed segment retain higher uncertainty. The final result is the future-aware conditional vector:

[0043] For the Let the nth future frame be... The semantic prototype features obtained by mapping pseudo-future candidate sequences are The number of candidate sequences is Then the future pseudo-features after fusion can be represented as:

[0044] The generation of pseudo-future action-level sequences can employ maximum transition probability selection, probability-based sampling, temperature-adjusted sampling, or Top-k sampling. Maximum probability selection results in more stable initialization results; probability-based sampling preserves more future possibilities; and Top-k sampling avoids interference from low-probability noise actions.

[0045] Average action duration table Its function is to transform action-level prediction into frame-level prediction. For action categories with large duration variations, the duration variance can also be calculated simultaneously, and the unfolding length can be adjusted based on the variance, making action duration modeling more flexible.

[0046] S3, Using Appendix Figure 1 The selective state-space denoising network shown performs diffusion iterative denoising. This network is composed of multiple stacked MANTA blocks, and the network execution flow is as follows: This step is used to continuously inject action semantic priors during the iterative denoising process of the diffusion model. The diffusion model receives noisy action label variables in each denoising step. diffusion step and future-aware conditional vectors and through a denoising generator Gradually restore the action label sequence.

[0047] During the forward diffusion noise addition process, the real action label matrix Gradually perturbed into noise action label variables In one implementation, the first... The noise-added form of the step can be expressed as:

[0048] Specifically, setting the semantic injection start layer and injection interval When the denoising network reaches an intermediate layer that satisfies the injection conditions, the system calculates the pseudo-label for each future frame based on the current intermediate prediction results:

[0049] Subsequently, the system retrieves the corresponding semantic prototype from the action semantic prototype dictionary based on the pseudo-label and overlays it onto the conditional features of the corresponding layer, enabling the subsequent denoising layer to reconstruct features under the guidance of category semantics.

[0050] Semantic Injection Start Layer The injection interval can be set according to network depth and noise intensity. Used to control the frequency of semantic enhancement. When the model is smaller, it receives semantic prototypes in more layers; when When the size is large, the computational load is small, and the model has more room for autonomous modeling.

[0051] Specific implementation of the MANTA denoising generator: In a preferred embodiment, the denoising generator of the present invention uses MANTA blocks as the basic temporal processing unit. These blocks are suitable for random long-term dense action prediction tasks. The core idea is to jointly model the action label states in the diffuse latent variables with the conditional features of the observed video, and to utilize a Mamba-based selective state space layer to process long sequence dependencies.

[0052] Given observed video features and a future-aware conditional vector, the system first... Noise action label variable of the current diffusion step The input features are obtained by concatenating them along the channel dimension. The input features contain both action label spatial information and video conditional spatial information, and their sequence length covers both observed segments and future predicted segments.

[0053] The MANTA denoising generator is formed by sequentially stacking B MANTA blocks. For the b-th MANTA block, its input features are... First, the input undergoes layer normalization, then is fed into a bidirectional selective state-space layer (BSSL) for temporal modeling. Subsequently, feature transformation is performed through a channel mixing feedforward layer, and the result is added to the original input via a residual connection. This block's processing can be represented as:

[0054] The final prediction head will output the features of the last MANTA block. The mapping is to an action category probability distribution, which can be expressed in the form of:

[0055] Here, LN represents layer normalization, BSSL represents bidirectional selective state space layer, and FF represents channel hybrid feedforward network. Through the stacking of multiple MANTA blocks, the model can gradually fuse observation fragment information, future prior information, and diffused noise state, and finally output the frame-by-frame action category distribution by the prediction head.

[0056] Bidirectional Selective State Space Layer (BSSL): BSSL is a key temporal modeling component in the MANTA denoising generator. This layer contains forward and backward temporal processing paths. The forward path processes the input sequence in the forward temporal direction, while the backward path first reverses the sequence in time, then processes it in the reverse direction, and finally reverses it back to the original temporal order. Through this bidirectional processing, the model can utilize both the evolutionary information of future actions observed from the past and the backward context within the future sequence.

[0057] In each path, the input features first undergo channel transformation and one-dimensional convolution. The one-dimensional convolution kernel can be used to supplement the local temporal context, enabling the model to retain sensitivity to local changes in adjacent frames while possessing global sequence modeling capabilities. Subsequently, the convolution output is processed by activation functions and selective state-space layers to obtain long-range temporal dependency representations.

[0058] Taking the b-th MANTA block as an example, the processing of forward and backward paths in BSSL can be abstractly represented as follows: ,

[0059] in, Indicates length is One-dimensional convolution kernel, This represents the convolution operation. Represents a non-linear activation function. This indicates a sequence that is reversed along the time dimension. This represents a selective state-space sequence transformation.

[0060] The selective state space layer can employ a Mamba or S6-like mechanism. This mechanism dynamically adjusts the state update process through input-related selection gating, enabling the model to adaptively decide which temporal information to retain, forget, or propagate based on the features of the current frame. Since the input sequence for long-term action prediction naturally contains two parts—observed frames and future masked frames—data-dependent gating helps the model process these two types of segments in different ways.

[0061] Furthermore, the discrete state update of the selective state space layer can be abstractly represented as:

[0062] in, This indicates the input at the current time. Represents the state vector. This indicates the output representation. , and These are selective state parameters that change dynamically with the input.

[0063] In a specific implementation, the forward path output... Back path output The BSSL first aggregates the forward and backward path outputs to obtain a bidirectional temporal representation; this bidirectional temporal representation is then processed by the channel mixing feedforward layer in the MANTA block. This structure can reduce the information bias caused by unidirectional modeling and improve the problem of insufficient consistency within future segments in long-term predictions.

[0064] Technical effects of MANTA blocks: The advantage of MANTA blocks over traditional dilated convolutional denoising networks lies in their global receptive field along the sequence length dimension, while their computational complexity increases approximately linearly with the sequence length. For long-term action prediction tasks, the prediction sequence often contains a large number of frames. Using attention structures or multi-stage dilated convolutional structures may lead to high computational costs or insufficient local receptive fields.

[0065] BSSL does not rely on sparse dilated convolutions to expand the receptive field layer by layer, so early network layers can propagate observation fragment information to future mask regions more quickly. This is especially important for diffusion denoising, because if the future regions of early denoising layers lack observation information for a long time, they are prone to producing incorrect action fragments and oversegmentation results.

[0066] The selective state-space structure can also distinguish between observed frames and future masked frames through data dependency gating. Observed frames have real visual features, while future frames mainly depend on prior knowledge and the generation process. The gating mechanism can adjust the information propagation method according to the input content, enabling the model to adaptively process these two types of variables in the same sequence.

[0067] In this invention, the MANTA block and the spatiotemporal prior are not two independent parts. The spatiotemporal prior provides future initialization and semantic anchors, while the MANTA block provides efficient long sequence denoising capabilities. The combination of the two can improve the local accuracy, global consistency, and generation efficiency of the predicted sequence.

[0068] S4. Training objective constraints from local to global This step is used to supervise the training of the model output, ensuring that the predicted sequence simultaneously satisfies frame-level action accuracy and sequence-level transition consistency. At a local level, the system processes the predicted label matrix... With the real label matrix calculate Reconstruction losses:

[0069] Wherein, the true action label matrix Y0 is the label matrix formed by one-hot encoding of each frame's action category, and the model predicts the action label matrix. Since it shares the same dimensionality, L2 loss can be used for reconstruction constraints. This loss constrains the action category prediction for each frame, giving the model basic frame-by-frame classification capabilities. However, using only this loss may result in locally correct sequences with illogical overall transitions; therefore, this invention further introduces sequence-level transition consistency constraints.

[0070] S41: Forward Transfer Consistency Constraint The system is based on the predicted category distribution and adjacent frame predicted category distribution Perform soft counting to obtain the predicted forward transition matrix. Subsequently, the system calculates the KL divergence between the predicted forward transition matrix and the true forward transition matrix, thus obtaining the forward transition consistency loss. .

[0071] The predicted forward transition matrix can be obtained by summing and normalizing the outer accumulations of the class distributions of adjacent frames: ,

[0072] S42: Back-transfer Consistency and Overall Optimization The system further constructs the prediction backward transition matrix in the same manner. The KL divergence is calculated between the KL divergence and the true backward transition matrix to obtain the backward transition consistency loss. Forward constraints emphasize the direction of action evolution from the observed segment to future segments, while backward constraints supplement the inverse dependencies within future sequences. The final training objective is expressed as: The consistency loss for forward and backward transfers can be expressed as follows: ,

[0073]

[0074] in, The consistency loss weights are used to balance frame-level reconstruction accuracy and sequence-level spatiotemporal consistency. Through the above objective function, the model can not only improve the prediction accuracy of a single frame, but also reduce problems such as unreasonable action jumps, abnormal action duration, and unnatural overall transition paths.

[0075] Backward transition consistency is not simply a repetition of forward constraints, but rather an observation of the predicted sequence from the opposite temporal direction. For certain action sequences, future actions have a reverse explanatory effect on preceding actions. Backward constraints help the model supplement dependencies that might be missed by forward constraints alone, starting from the overall structure of the prediction results.

[0076] During training, the system can proceed from multiple diffusion steps. The training target can be randomly sampled or trained according to a fixed diffusion step sequence. By learning to reconstruct the target and the transfer consistency target under different noise intensities, the model can be subject to spatiotemporal prior constraints in both the early coarse-grained generation stage and the later fine-grained correction stage.

[0077] Data verification: This embodiment uses three long-term action prediction public datasets—Breakfast, 50Salads, and Assembly101—as examples to illustrate the actual prediction performance of the method of this invention. The system first calculates the forward action transition matrix, backward action transition matrix, and average action duration table based on the action annotation sequences in the training sets of each dataset, and constructs an action semantic prototype dictionary based on the video features of the training sets. During the inference phase, the model only observes the preceding input video... Proportional frames, and predict subsequent frames. The proportion of future action sequences, in which Indicates the proportion of observations. This indicates the predicted proportion. In the experiment, Take 20% and 30%, Take 10%, 20%, 30% and 50%.

[0078] For the same observed video segment, the system can perform multiple diffusion samplings starting from different random noise levels to generate multiple candidate future action sequences. To evaluate the prediction results, two metrics are used: Mean MoC and Top-1 MoC. Mean MoC represents the average class prediction accuracy of the multiple sampling results, used to measure the overall generation quality of the model; Top-1 MoC represents the class accuracy of the candidate sequence that best matches the true future action sequence, used to measure the model's ability to generate high-quality future sequences in multi-candidate prediction scenarios.

[0079] (a) Breakfast Dataset Example The Breakfast dataset contains long-duration videos of the breakfast preparation process, typically involving actions such as taking utensils, adding ingredients, stirring, heating, and cleaning. These videos exhibit a relatively clear sequence of actions, and multiple plausible future events may occur after the same observation segment. Therefore, this dataset can validate the predictive capabilities of this invention in scenarios with process structure and future uncertainty.

[0080] Table 1. Comparison of the method of this invention with other major conventional baseline methods on the Breakfast dataset.

[0081] On the Breakfast dataset, the method of this invention demonstrates higher MeanMoC or Top-1 MoC compared to the main baseline methods. =30% Taking a long prediction setting of 50% as an example, the Mean MoC of the MANTA method is 27.70%, and the Top-1 MoC is 51.90%; the Mean MoC of the method of this invention is 29.24%, and the Top-1 MoC is 53.15%, representing improvements of 1.54 percentage points and 1.25 percentage points, respectively. This result demonstrates that even with a long prediction span and increased uncertainty in future actions, the present invention can still maintain high prediction stability.

[0082] The present invention also demonstrates significant advantages at shorter prediction settings. For example, when =20% When the content is 10%, the Mean MoC of the method of this invention is 28.12%, and the Top-1 MoC is 55.96%, both higher than MANTA's 27.70% and 55.50%; when =30% When the target value is 10%, the Mean MoC of the method of this invention is 35.42%, and the Top-1 MoC is 61.13%, which are also higher than MANTA's 34.20% and 59.60%. The above results show that the present invention can not only improve long-span prediction, but also improve the accuracy of action category judgment when the prediction range is short.

[0083] (II) 50Salads Dataset Example The 50Salads dataset contains long-duration motion videos of salad preparation, with strong process dependencies between actions. For example, steps such as preparing ingredients, cutting, mixing, and seasoning usually follow a certain order. This dataset can validate the predictive ability of this invention in cooking process scenarios with long action durations and clearly defined action stages.

[0084] Table 2. Comparison of the method of this invention with other major conventional baseline methods on the 50Salads dataset.

[0085] On the 50Salads dataset, under the settings listed in the table, the method of this invention achieves the highest results. =30% Taking 50% as an example, MANTA's Mean MoC is 13.00% and Top-1 MoC is 31.10%; the Mean MoC of the method of this invention is 14.48% and Top-1 MoC is 33.31%, representing improvements of 1.48 percentage points and 2.21 percentage points, respectively. This result indicates that when the future prediction range is long, this invention can utilize action transfer relationships and duration priors to alleviate the problems of action phase breaks and action sequence drift in long-form videos.

[0086] exist =20% With a 50% setting, the Mean MoC of the method of this invention is 15.43%, significantly higher than MANTA's 13.60%; the Top-1 MoC is 35.74%, higher than MANTA's 31.30%. =30% With a 10% threshold, the Mean MoC of the method in this invention is 31.72%, and the Top-1 MoC is 71.95%, both higher than MANTA's 31.30% and 71.70%, respectively. These results demonstrate that this invention improves both short-term and long-term predictions on the 50Salads dataset, with a more significant improvement when the future timeframe is large.

[0087] (III) Assembly101 Dataset Example The Assembly101 dataset contains fine-grained hand movements and object interactions in assembly tasks. It features a large number of movement categories with complex sequential dependencies and high visual similarity between different movements. Therefore, the overall prediction difficulty of this dataset is high, validating the applicability of this invention in fine-grained movement prediction scenarios.

[0088] Table 3. Comparison of the method of this invention with other major conventional baseline methods on the Assembly101 dataset.

[0089] On the Assembly101 dataset, the method of this invention also achieves better results than existing comparative methods under different observation and prediction ratios. Taking Mean MoC as an example, when =20% When the concentration is 10%, the result of MANTA is 6.70%, while the method of this invention achieves 7.25%; when =20% When the concentration is 20%, the result of MANTA is 5.30%, while the method of this invention achieves 5.79%; when =30% When the accuracy is 50%, MANTA's result is 3.50%, while the method of this invention achieves 4.14%. These results demonstrate that the present invention can improve the average prediction quality in assembly videos with more detailed action categories and more complex action combinations.

[0090] The method of this invention also demonstrates good candidate sequence generation capability in the Top-1 MoC metric. For example, when =20% When the content is 10%, the method of this invention achieves 18.97%, which is higher than MANTA's 16.90%; when =20% When the content is 20%, the method of this invention achieves 13.89%, which is higher than MANTA's 13.30%; when =30% When the accuracy is 50%, the method of this invention achieves 8.50%, which is higher than MANTA's 8.40%. This result shows that even in a fine-grained, high-uncertainty assembly scenario like Assembly101, this invention can still generate higher-quality candidate future action sequences.

[0091] In summary, this embodiment demonstrates that the method of the present invention achieves superior prediction results compared to existing methods on three long-term action prediction datasets: Breakfast, 50Salads, and Assembly101. The performance improvement mainly stems from the following aspects: First, the structure-aware future initialization module provides conditional representations that conform to action transition rules for unknown future segments at the beginning of prediction; second, the progressive semantic enhancement module continuously injects action semantic prototypes during diffusion denoising, reducing action category drift; and third, the MANTA / BSSL denoising network effectively models long-sequence dependencies. Therefore, the present invention does not simply rely on the random sampling capability of the basic diffusion model, but rather improves the average generation quality and the quality of the best candidate predictions for long-term action predictions by introducing spatiotemporal priors and semantic constraints throughout the entire prediction pipeline.

[0092] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A long-term action prediction method based on spatiotemporal prior guidance and selective state-space denoising, characterized in that, Includes the following steps: Step S1: Obtain the input video sequence, extract frame-by-frame visual features using a feature extractor, and base the extraction on the observation ratio. Predicted proportion Divide the observed segment into the future prediction segment and construct the basic condition vector for diffusion prediction. ;in For the visual features of the i-th frame, For the number of observation frames, The total number of video frames, the number of predicted frames. The positions of future predicted segments in the basic condition vector are initially set to zero; Step S2: Obtain the forward action transition matrix based on training set statistics. Backward action transition matrix Action semantic prototype dictionary Average action duration table Three types of spacetime priors; Based on the visual features of the last few frames of the observed segment, the current action category is estimated through action prototype matching. Multiple pseudo-future action sequences are then generated by combining this action category with the forward action transition matrix. These sequences are expanded into frame-level pseudo-label sequences by incorporating the average action duration. All candidate pseudo-features are fused using a weighting method that decays with prediction distance to generate a future-aware conditional vector. ; Step S3: Construct a selective state-space denoising generator consisting of stacked MANTA blocks. Each MANTA block includes layer normalization, a bidirectional selective state-space layer (BSSL), a channel mixing feedforward layer, and residual connections. During the diffusion iterative denoising process, the noise action label variables and the future-aware conditional vector are... Fusion input denoising generator; setting semantic injection start layer With injection interval In the intermediate layer of the network that meets the conditions, the semantic prototype of the action is retrieved based on the pseudo-label of the current diffusion step and fused with the intermediate features to achieve progressive semantic enhancement; Step S4: Construct a joint loss function for the selective state-space denoising generator described in step S3. Complete model training; among which For frame-level reconstruction loss, Forward transfer consistency loss, For backward transfer consistency loss, To transfer loss weights; After training, multiple candidate future action sequences are generated through random diffusion sampling. Based on the average class confidence of the candidate sequences, the prediction distribution, and the consistency score between the candidate sequence transition matrix and the training set transition matrix, the long-term action prediction results are selected and output.

2. The long-term action prediction method based on spatiotemporal prior guidance and selective state-space denoising as described in claim 1, characterized in that, The frame-by-frame visual feature extraction formula in step S1 is as follows: ; For feature extractors, The input is the original image of the i-th frame of the video. The feature extractor can be any one of I3D, TSM, video Transformer or temporal convolutional network.

3. The long-term action prediction method based on spatiotemporal prior guidance and selective state-space denoising as described in claim 1, characterized in that, Action semantic prototype dictionary in step S2 Internal Single Category Action Semantic Prototype Generated from the average visual features of all frames corresponding to the action category in the training set; the weights of the multi-candidate pseudo-future feature fusion satisfy... j is the current frame number, and the weight decreases linearly with the distance from the observed segment.

4. The long-term action prediction method based on spatiotemporal prior guidance and selective state-space denoising according to claim 1, characterized in that, In step S3, the bidirectional selective state space layer (BSSL) includes a parallel forward temporal path and a backward temporal path. The forward path processes the sequence along the positive time direction, while the backward path first reverses the sequence temporal order to complete the calculation and then restores the temporal order. The output features of the two paths are aggregated to obtain the overall temporal representation of BSSL.

5. The long-term action prediction method based on spatiotemporal prior guidance and selective state-space denoising according to claim 4, characterized in that, Both the forward and backward temporal paths include a one-dimensional convolutional layer, a nonlinear activation function, a selective state space layer, and a residual gating module. The selective state space layer adopts an input-related selection gating mechanism to adaptively adjust the state update parameters according to the input features, so as to distinguish between observed real frames and future masked prediction frames.

6. The long-term action prediction method based on spatiotemporal prior guidance and selective state-space denoising according to claim 5, characterized in that, The discrete state update logic for the selective state space layer is as follows: In the formula: Input features at time t, Let be the state vector at time t. To output the time series features at time t, A t B t C t These are selective state parameters that change dynamically with the input.

7. The long-term action prediction method based on spatiotemporal prior guidance and selective state-space denoising according to claim 1, characterized in that, In step S3, progressive semantic enhancement, the single-frame predicted pseudo-label calculation method is as follows: ； in, Let C be the noise action label variable for the k-th diffusion step, and C be the total number of action categories. This represents the current network layer number.

8. The long-term action prediction method based on spatiotemporal prior guidance and selective state-space denoising according to claim 1, characterized in that, In step S4, the frame-level reconstruction loss uses L2 loss, expressed as follows: ; in, A matrix of real action labels. The action label matrix is ​​predicted for the k-th step of the diffusion model.

9. The long-term action prediction method based on spatiotemporal prior guidance and selective state-space denoising according to claim 1, characterized in that, In step S4, both the forward and backward transition consistency loss adopt the KL divergence loss. Forward transfer loss Backward transfer loss ; in, These are the forward and backward transition matrices of the predicted sequence, respectively. To obtain the true forward transition matrix and the true backward transition matrix from the training set, The summation index is used to indicate the summation of all subscripts ending in 1. The summation of the KL divergence between the predicted and labeled values ​​of the action sequence at the start of the action, where KL is the relative entropy; the true forward transition matrix and the true backward transition matrix are given a smoothing coefficient before normalization to avoid KL divergence instability caused by zero probability transitions.

10. The long-term action prediction method based on spatiotemporal prior guidance and selective state-space denoising according to any one of claims 1-9, characterized in that, The method is applicable to scenarios including first-person perspective behavior video prediction, vehicle driving scene action prediction, industrial assembly process action prediction, and video analysis scenarios that require multi-step intensive long-term action sequence prediction.

Images