An end-to-end spatio-temporal prediction method based on improved three-dimensional rotary position encoding

Abstract

The application belongs to the technical field of computer vision, deep learning and space-time prediction, and discloses an end-to-end space-time prediction method based on improved three-dimensional rotation position coding, which is suitable for weather, traffic flow and other multi-type space-time sequence prediction scenes. The application optimizes the position coding mechanism through four key improvements: dynamically adjusting the proportion of three-dimensional coding of time, height and width to adapt to different scenes; fusing absolute time and relative spatial position to strengthen local space-time correlation modeling; directly guiding attention calculation with position information to deepen the fusion with the Attention module; and introducing a rotation matrix caching mechanism to reduce redundant calculation. At the same time, the model is matched with a Patch embedding layer, an adaptive Transformer encoder and an MLP decoding head to build a complete link of "feature embedding-position coding-space-time fusion-prediction output", which effectively improves the accuracy, generalization and reasoning efficiency of space-time prediction.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, deep learning, and spatiotemporal prediction technology. Specifically, it is a pure Transformer spatiotemporal prediction model with improved 3DRoPE as its core innovation. It is applicable to various spatiotemporal sequence prediction scenarios such as weather prediction, traffic flow prediction, human motion prediction, and moving object trajectory prediction. Through in-depth optimization of the position encoding mechanism, the model significantly improves its accuracy in capturing complex spatiotemporal dynamics and its scene adaptability. Background Technology

[0002] The core requirement of spatiotemporal prediction using video as data carrier is to predict future frame sequences by learning the spatiotemporal patterns of historical frames. Although existing spatiotemporal prediction models based on the pure Transformer architecture have overcome the limitations of poor parallelization of RNNs and the limited local receptive field of CNNs, key bottlenecks still exist in the positional encoding mechanism:

[0003] ① Fixed dimensional allocation: The location code is generated using a fixed sine function. The proportion of the coding dimensions of time, height and width cannot be adjusted, which cannot adapt to the differences in the importance of spatiotemporal features in different scenarios (such as the time dimension being more dynamic in weather forecasting and the spatial dimension being more closely related in traffic forecasting).

[0004] ② The location information modeling is too simple: it only relies on absolute position coordinates and does not integrate relative position relationships. It is not capable of modeling local spatiotemporal correlations (such as the dynamics of adjacent frames and the structure of adjacent patches), which leads to the distortion of local details in the predicted frame.

[0005] ③Location and attention are disconnected: The location encoding is only superimposed on the token feature as a fixed bias and is not deeply integrated with the Attention module. The guiding role of location information on attention weights is not fully utilized, and the accuracy of capturing long-distance spatiotemporal dependencies is limited.

[0006] ④ Computational efficiency needs optimization: No caching mechanism was designed for repeated calculations of position encoding. Rotation matrices with the same input shape need to be generated repeatedly, increasing inference redundancy.

[0007] To address the aforementioned issues, existing technologies lack a location encoding scheme that can simultaneously achieve adaptive dimension allocation, spatiotemporal location collaborative modeling, deep integration with attention, and computational efficiency. This has become a core obstacle restricting the performance improvement of pure Transformer spatiotemporal prediction models. Summary of the Invention

[0008] Existing spatiotemporal prediction models based on pure Transformer architecture (such as PredFormer) suffer from four core problems in their positional encoding mechanisms:

[0009] First, the dimensional allocation is fixed, and the proportion of time, height, and width in the three-dimensional encoding cannot be adjusted, making it impossible to adapt to the differences in the importance of spatiotemporal features in different scenarios.

[0010] Second, the location information modeling is too simplistic, relying solely on absolute location coordinates and lacking effective capture of local spatiotemporal correlations, resulting in distorted prediction details.

[0011] Third, the integration of position encoding with the Attention module is not deep; it is merely used as a fixed bias superposition, and the guiding role of position information on attention weights is not fully utilized.

[0012] Fourth, it lacks a computation caching mechanism, so rotation matrices with the same input shape need to be generated repeatedly, resulting in low inference efficiency.

[0013] To address the aforementioned issues, this invention proposes a pure Transformer spatiotemporal prediction model centered on Improved 3D Rotation Position Encoding (Improved3DRoPE), upgrading the position encoding mechanism through four key improvements:

[0014] First, we designed a learnable three-dimensional adaptive allocation mechanism to dynamically adjust the proportion of three-dimensional encoding through learnable parameters to adapt to different scenario requirements.

[0015] Second, it adopts spatiotemporal location collaborative modeling, integrating absolute time location and relative spatial location to construct a complete three-dimensional location description;

[0016] Third, it achieves deep integration with the Attention module by performing dimensional rotation operations on the Query and Key, allowing positional information to directly guide attention calculation;

[0017] Fourth, a rotation matrix caching mechanism is introduced to avoid redundant calculations and improve inference efficiency.

[0018] Meanwhile, this spatiotemporal prediction model is equipped with a Patch embedding layer, an adapted Transformer encoder, and an MLP decoder. Each module works collaboratively with location information as the core, comprehensively improving the accuracy, generalization, and efficiency of spatiotemporal prediction.

[0019] This spatiotemporal prediction model design includes a patch embedding layer, an improved 3DRoPE-compatible Transformer encoder (including a gated Transformer layer), and an MLP decoder head.

[0020] The Patch embedding layer converts video frames into patch tokens of a uniform dimension, providing a feature carrier for positional encoding;

[0021] The Transformer encoder achieves synchronous transfer and fusion of position tensors and feature tensors through a four-serial-branch structure (ts-t, ts-s, st-s, st-t);

[0022] The MLP decoder maps the features fused with location information back to the pixel space and outputs a predicted frame sequence, forming a complete link of "feature embedding - location encoding - spatiotemporal fusion - prediction output".

[0023] This invention presents an end-to-end spatiotemporal prediction method based on the improved 3D rotational position encoding (Improved3DRoPE), employing a non-recurrent, non-convolutional end-to-end architecture. This architecture relies on the positional information generated by Improved3DRoPE for core collaborative operation, with its backbone feature fusion network consisting of Transformer layers in this embodiment. Spatiotemporal sequence prediction is achieved sequentially through four main stages: feature embedding, positional encoding, spatiotemporal fusion, and prediction output.

[0024] This method, an end-to-end spatiotemporal prediction approach based on improved 3D rotational position coding, employs a general spatiotemporal prediction model that takes a video sequence as input and outputs prediction results for future frames. The general spatiotemporal prediction model is a pure Transformer architecture spatiotemporal prediction model with the Improved3DRoPE module as its core. The pure Transformer architecture is a loop-free and convolution-free architecture.

[0025] In the spatiotemporal prediction model, the input video sequence is sequentially processed through four stages: feature embedding, location encoding, spatiotemporal fusion, and prediction output to achieve spatiotemporal sequence prediction. The specific steps include:

[0026] Step 1. Preprocess and augment the input video sequence, and divide it into training, validation and test sets;

[0027] Step 2. Convert the preprocessed video sequence into a uniform-dimensional patch token sequence using the Patch embedding layer;

[0028] Step 3. Generate an adaptive 3D position tensor using the Improved3DRoPE module; the Improved3DRoPE module includes:

[0029] 1) Learnable 3D dimension adaptive allocation mechanism: The proportion of time, height and width encoding dimensions (dim) is dynamically adjusted through learnable parameters to ensure adaptation to different scenario requirements;

[0030] 2) Spatiotemporal location collaborative modeling mechanism: By fusing absolute time location and relative spatial location information, it provides complete spatiotemporal positioning for the token;

[0031] 3) Deep integration mechanism with the Attention module: Position information is injected into the Q / K dimension-wise rotation operation through rotation operation, and each dimension of the query Q and key K vectors is rotated separately, so that the attention weights are directly guided by the position relationship;

[0032] 4) Rotation matrix caching mechanism: used to avoid repeated calculations under the same input shape, thereby improving inference efficiency;

[0033] Step 4. Synchronously input the patch token sequence and the 3D position tensor into the Transformer encoder to extract global spatiotemporal dependency features;

[0034] Step 5. The MLP decoder maps the features fused with location information back to the pixel space and outputs the predicted frame sequence;

[0035] Step 6. Adjust the model parameters based on the error between the predicted frame and the real frame, optimize the performance of the spatiotemporal prediction model through iterative training, and improve the computational efficiency by using a caching mechanism during the inference stage.

[0036] The video sequence is a sequence of weather satellite cloud images, traffic monitoring video sequences, or human movement video sequences.

[0037] Beneficial effects

[0038] This invention, based on Improved3DRoPE, has the following significant advantages compared to existing technologies:

[0039] 1. Significantly improved scene adaptability: The learnable 3D dimension allocation mechanism enables positional encoding to adapt to the spatiotemporal characteristics of different scenes: in weather forecasting (with strong temporal dynamics) Its proportion automatically increases to 45%–50% in traffic forecasting (with strong spatial correlation). Its proportion has increased to 35%–40%, and its generalization ability has been greatly improved;

[0040] 2. Improved location modeling accuracy: The collaborative modeling of absolute time and relative space reduces the distortion rate of local details (such as human movement joints and local traffic congestion) in the predicted frame by 30% to 35%, and the MSE index is improved to 12.04 on the Moving MNIST dataset (11.76 higher than PredRNN and 5.66 higher than SwinLSTM).

[0041] 3. Enhanced Attention Guidance Effectiveness: The deep fusion of Improved3DRoPE and Attention improves the accuracy of capturing long-distance spatiotemporal dependencies (such as temperature trends in 10-frame weather sequences), reducing the MSE to 1.110 on the WeatherBench dataset (0.221 lower than PredRNN and 0.128 lower than SimVP).

[0042] 4. Optimized computational efficiency: The rotation matrix caching mechanism improves the inference speed to 5238 FPS on the TaxiBJ dataset (about 1.2 times faster than the original PredFormer and about 10 times faster than SimVP), while the number of parameters only increases by 5% to 8% (mainly from the logits and scaling factor networks).

[0043] 5. Strong compatibility: Improved3DRoPE can be seamlessly integrated into other existing pure Transformer spatiotemporal prediction models without refactoring the overall architecture, resulting in low migration costs. Attached Figure Description

[0044] (Note: The attached diagram is a structural schematic of the core innovation module, as shown below.)

[0045] Figure 1 : Flowchart of the overall algorithm framework of this invention;

[0046] Figure 2 Schematic diagram of the structure of the improved 3D rotation position encoding module (Improved3DRoPE);

[0047] Figure 3 : Flowchart of weight normalization and dimension calculation for a learnable 3D dimension adaptive allocation mechanism;

[0048] Figure 4 : A schematic diagram illustrating the generation of absolute temporal and relative spatial positions in spatiotemporal location collaborative modeling;

[0049] Figure 5 Flowchart of deep fusion between Improved3DRoPE and Attention module (including Q / K rotation operation);

[0050] Figure 6 : Schematic diagram of cache key definition and update logic of rotation matrix caching mechanism. Detailed Implementation

[0051] This invention belongs to the fields of computer vision, deep learning and spatiotemporal prediction technology. It discloses an end-to-end spatiotemporal prediction method with Improved3DRoPE as the core, which is applicable to various spatiotemporal sequence prediction scenarios such as weather and traffic flow.

[0052] Existing advanced end-to-end spatiotemporal prediction models suffer from four major bottlenecks: fixed allocation of location encoding dimensions, modeling only absolute positions, shallow integration with the attention module, and low computational efficiency. To address these bottlenecks, this model optimizes the location encoding mechanism through four key improvements: dynamically adjusting the proportions of time, height, and width encodings to adapt to different scenarios; integrating absolute time and relative spatial positions to strengthen local spatiotemporal correlation modeling; allowing location information to directly guide attention computation, deepening integration with the attention module; and introducing a rotation matrix caching mechanism to reduce redundant computation.

[0053] Meanwhile, the model is paired with a Patch embedding layer, an adaptive Transformer encoder, and an MLP decoder to construct a complete chain of "feature embedding - location encoding - spatiotemporal fusion - prediction output", which effectively improves the accuracy, generalization and inference efficiency of spatiotemporal prediction.

[0054] The following is in conjunction with the appendix Figures 1-6 The document details the "architectural details, module implementation, training and inference, and experimental verification," clarifying the technical parameters, formula principles, and operating procedures for each step. Those skilled in the art can reproduce the model and experiments based on the following content.

[0055] (I) Overall Structure and Appendix Figure 1 Compatibility notes

[0056] like Figure 1 As shown, this spatiotemporal prediction model adopts a pure Transformer architecture without loops or convolutions. The modules in the model work collaboratively with the location information generated by Improved3DRoPE as the core. The input is a video sequence (such as weather satellite cloud image sequence, traffic monitoring video sequence, human movement video sequence), and the output is the prediction result of future frames. The functions and parameter settings of each module are as follows:

[0057] Input layer: The input video sequence is in the format (B, T, C, H, W), where:

[0058] B (Batch Size): Adjusted according to hardware memory configuration, typically set to 8-32;

[0059] T (Time Step): Represents the number of historical frames input. It is set according to the requirements of the prediction task. For example, long-term weather prediction is set to 10 (input 10 frames of historical data per day), and traffic flow prediction is set to 4 (input 4 time slices of data).

[0060] C (Number of Channels): Set color video to 3 (RGB channels), and grayscale video (such as infrared weather maps and traffic flow heat maps) to 1;

[0061] H / W (Image Resolution): The standard settings are 32×32 and 64×64, which should be adapted to the resolution of the data acquisition device (e.g., when the resolution of a traffic monitoring camera is 1920×1080, it needs to be downsampled first to balance accuracy and efficiency).

[0062] Patch embedding layer: Segment the input video frames into non-overlapping patches and map them to a uniform feature dimension dim (usually set to 256):

[0063] Patch size selection: Use a patch size of 8×8 or 16×16 (patch_size). If the input resolution is 64×64 and patch_size=8, then each frame is divided into (64 / 16)×(64 / 16)=16 patches, that is, N=16;

[0064] Feature mapping process: First, the video sequence is reshaped from (B, T, C, H, W) to (B, T, N, patch_size×patch_size×C), and then mapped to a token sequence (B, T, N, dim) through a linear layer (input dimension is patch_size×patch_size×C, output dimension is dim), providing a feature carrier with a unified dimension for subsequent positional encoding.

[0065] Improved3DRoPE position encoding generation module: Receives the dimension information (dim, T, N, patch mesh size) of the token sequence and generates a three-dimensional position tensor (B, T, N, 3). For details, see Part (II) of this section.

[0066] Transformer encoder: Contains 6 to 12 gated Transformer layers (the number of layers is adjusted according to the complexity of the task; complex tasks such as weather prediction are set to 12 layers, and human motion prediction is set to 6 layers). Each layer consists of four branches: ts-t (time-time attention), ts-s (time-space attention), st-s (space-space attention), and st-t (space-time attention). Each branch synchronously processes token features and position tensors to extract global spatiotemporal dependencies. For specific implementation details, see Part (III) of this section.

[0067] MLP decoder head: Maps the features (B, T, N, dim) output by the encoder back to pixel space.

[0068] First, the token features are converted into patch pixel features through a linear layer (input dimension dim, output dimension C×patch_size²);

[0069] Then, the (B, T, N, C×patch_size²) is restored to (B, T, C, H,W) through the reshape operation to obtain the final predicted frame sequence.

[0070] (II) Detailed implementation of the Improved3DRoPE module (refer to Figures 2-6)

[0071] As shown in Figure 2, the Improved3DRoPE module is the core of this method, which includes four mechanisms: "learnable 3D dimension adaptive allocation, spatiotemporal position collaborative modeling, deep fusion with Attention, and rotation matrix caching". The principles, formulas, and operational details of each mechanism are as follows:

[0072] 1. Learnable adaptive allocation of three-dimensional dimensions (see Figure 3)

[0073] Figure 3 illustrates the complete process of dimension allocation. The core principle is to dynamically adjust the proportions of time, height, and width encoding dimensions using learnable parameters to ensure adaptability to different scenario requirements.

[0074] Step 1.1: Initialize learnable parameters

[0075] A learnable parameter, logits, with shape (3,) is introduced and its initial value is set to [0.4, 0.3, 0.3]. This initial value is based on the empirical setting that "the time dynamics are stronger in most spatiotemporal scenarios." For example, in weather prediction and human motion prediction, the feature changes in the time dimension are usually more significant than those in the spatial dimension. During subsequent training, the model will automatically optimize logits based on the data.

[0076] Step 1.2: Weight Normalization

[0077] The logits are normalized using the softmax function to ensure that the sum of the three-dimensional weights is 1, as shown in the following formula:

[0078]

[0079] Where i=0 corresponds to the time dimension, i=1 corresponds to the height dimension, and i=2 corresponds to the width dimension. The normalized weights directly determine the encoding proportion of each dimension.

[0080] Step 1.3: Encoding and Calculating Each Dimension

[0081] Based on the token feature dimension *dim*, calculate the actual encoding dimension for each dimension. Simultaneously, use *max(2,・)* to ensure that each dimension retains at least 2 features, avoiding dimensional degradation (if the dimension is less than 2, the positional information of that dimension cannot be effectively represented). The formula is as follows:

[0082] Time dimension encoding dimension: ;

[0083] Height dimension encoding dimension: ;

[0084] Width dimension encoding dimension: ;

[0085] For example, when , hour, (512 × 0.45 ≈ 230) (512 × 0.28 ≈ 143) (512 × 0.27 ≈ 138).

[0086] Step 1.4: Dimension Overflow Adaptation

[0087] like ,(like After calculation , , Since the total is 270 > 256, each dimension needs to be scaled proportionally to ensure that the total dimension matches dim. The formula is as follows:

[0088]

[0089] If a dimension is less than 2 after scaling, it is forcibly set to 2, and the corresponding dimension is subtracted from the other dimensions (prioritizing the dimension with the largest proportion), ultimately ensuring... .

[0090] 2. Spatiotemporal location co-modeling (see Figure 4)

[0091] Figure 4 illustrates the generation process of absolute temporal and relative spatial positions. The core of this process is to fuse these two types of location information to provide the token with complete spatiotemporal positioning.

[0092] Step 2.1: Generation of absolute time position

[0093] Generate an absolute time coordinate sequence based on the input time step T. , representing the global position at each time step; then, it is mapped to the [0,1] interval through Min-Max normalization, as shown in the following formula:

[0094]

[0095] The purpose of normalization is to eliminate coordinate range differences caused by different T values (e.g., T=12, T=24), so that the model can learn stably in tasks with different time steps.

[0096] Step 2.2: Generation of relative spatial positions

[0097] First determine the patch grid size: let the resolution of the input frame be H×W and the patch_size be s, then the number of patches in the height direction num_patches_h = H / s, and the number of patches in the width direction num_patches_w = W / s;

[0098] For each patch (i, j) (i is the index in the height direction, 0≤i<num_patches_h; j is the index in the width direction, 0≤j<num_patches_w), calculate its relative coordinates with all other patches (i', j'):

[0099] Relative coordinate in height direction:

[0100] Relative coordinate in width direction:

[0101] The relative coordinates are normalized to the interval [-1,1], where a positive value indicates "the current patch is above / to the right of the target patch", a negative value indicates "below / to the left", and 0 indicates "the same position". This normalization method can ensure that the output relative coordinate ranges of patch grids of different sizes (such as 8×8, 16×16, 32×32) are consistent, preventing the model from being affected by grid scales.

[0102] Step 2.3: Combination of three-dimensional position tensors

[0103] Stack the absolute time position (shape=(T,)), the relative height coordinate (shape=(N, N)), and the relative width coordinate (shape=(N, N)) along the batch dimension to form a three-dimensional position tensor pos_tensor with a shape of (B, T, N, 3) — where each element pos_tensor [b][t][n][0] corresponds to the absolute time position of the n-th patch at the t-th time step in the b-th batch, pos_tensor [b][t][n][1] corresponds to the relative height coordinate, and pos_tensor[b][t][n][2] corresponds to the relative width coordinate, which provides each token with a complete description of "when (time), where (space), and relationship to other positions (relative space)".

[0104] 3. Deep integration with the Attention module (Figure 5)

[0105] Figure 5 illustrates the fusion process of position encoding and the Attention module. The core is to inject position information into Q / K through rotation operations, so that the attention weights are directly guided by positional relationships.

[0106] Step 3.1: Position Tensor Dimension Adaptation. Transformer encoders typically employ a multi-attention head mechanism (the number of attention heads is usually set to 8 or 16). The 3D position tensor pos_tensor(B, T, N, 3) needs to be broadcast to each attention head, and its dimensions adjusted to match the shape of Q / K.

[0107] First, the pos_tensor is converted to (B, heads, T, N, 3) using the rearrange operation, thus achieving splitting by attention heads;

[0108] Then, the time step T and the number of patches N are combined to obtain (B×heads, T×N, 3), which is consistent with the shape of Q / K (B×heads, T×N, dim_head) (where dim_head = dim / heads, which is the feature dimension of each attention head).

[0109] Step 3.2: Q / K Dimensional Rotation Operation. Rotation operations are performed on the time, height, and width dimensions of the Q / K matrix. The rotation matrix is ​​generated based on the corresponding dimension information in the position tensor, as follows:

[0110] Rotation in the time dimension:

[0111] Extract the features of dimension dim_t from Q / K (denoted as Q_t and K_t), and calculate the rotation matrix (sine matrix sin_rot and cosine matrix cos_rot) based on the absolute time position time_pos, both with the shape (B×heads, T×N, dim_t / / 2, 2);

[0112] Split Q_t and K_t into real and imaginary parts (each occupying dim_t / / 2 dimensions), and perform a rotation operation:

[0113]

[0114] The rotation operation of K_t is the same as that of Q_t, resulting in K_t_rot; the purpose of rotation is to encode the temporal position information into the phase change of the feature through trigonometric functions, thereby strengthening the correlation between features at different time steps.

[0115] Rotate in height / width dimensions:

[0116] Similarly, extract the features (Q_h, K_h; Q_w, K_w) of the dim_h and dim_w dimensions in Q / K, calculate the rotation matrix based on the relative spatial positions rel_pos_h and rel_pos_w, and perform the same rotation operation as in the time dimension to obtain Q_h_rot, K_h_rot and Q_w_rot, K_w_rot;

[0117] This process can enhance the spatial correlation between different patches within the same time step, such as the characteristic correlation between adjacent road segment patches in traffic flow prediction.

[0118] Step 3.3: Post-rotation attention calculation. Concatenate the rotated Q-dimensional features (Q_t_rot, Q_h_rot, Q_w_rot) into a complete Q_rot, and concatenate the K-dimensional features (K_t_rot, K_h_rot, K_w_rot) into a complete K_rot; calculate the dot product attention weights, as follows:

[0119]

[0120] Where √dim_head is the scaling factor (to avoid excessively large dot product values ​​due to high dimensionality), and mask is the attention mask (such as a future time step mask to prevent the model from cheating using future information); the final attention output is:

[0121] (V is the Value tensor, which comes from the same token feature as Q / K, and no rotation operation was performed to preserve the original feature information.)

[0122] This fusion method allows location information to directly participate in the calculation of attention weights, enabling the model to enhance the weights of key features and improve prediction accuracy when capturing long-distance spatiotemporal dependencies (such as the low-pressure system 12 hours ago and the current rainfall area in weather forecasting) through location association.

[0123] 4. Rotation matrix caching mechanism (see Figure 6)

[0124] Figure 6 illustrates the caching logic of the rotation matrix, the core of which is to avoid repeated calculations under the same input shape, thereby improving inference efficiency.

[0125] Step 4.1: Define the cache key using "input shape (positions_shape) + encoded dimensions (dim_per_axis)" as the cache key, where:

[0126] positions_shape: includes time step T and patch grid size (num_patches_h, num_patches_w), i.e., positions_shape=(T, num_patches_h, num_patches_w);

[0127] dim_per_axis: The encoding dimension that includes time, height, and width, i.e., dim_per_axis=(dim_t, dim_h, dim_w);

[0128] This cache key uniquely identifies the computational needs of a set of rotation matrices, ensuring that the calculation is performed only once under the same input conditions.

[0129] Step 4.2: Cache Content Storage. The cached value is a calculated sine matrix (sin_mat) and cosine matrix (cos_mat), with shapes (B×heads, T×N, dim / / 2, 2) – rotation matrices covering the three dimensions of time, height, and width. Storage methods can include memory caching (such as the `dict` structure in Python) or disk caching (such as Redis). Memory caching is suitable for single-device inference, while disk caching is suitable for multi-device distributed inference. (This patent embodiment uses memory caching.)

[0130] Step 4.3: Cache update and invocation logic

[0131] During reasoning, first calculate the cache key based on the current input;

[0132] If the key already exists in the cache, the corresponding sin_mat and cos_mat are called directly, skipping the rotation matrix calculation step;

[0133] If the key does not exist, perform calculations such as theta=10000^(-2k / dim), angles=positions*theta, sin_mat=sin (angles), cos_mat=cos (angles) to generate a rotation matrix and store it in the cache;

[0134] Cache invalidation policy: When the input shape (T, num_patches_h, num_patches_w) or dim_per_axis changes, the old cache is automatically cleared to avoid using the wrong rotation matrix; at the same time, the maximum cache capacity is set (e.g., 100 groups). When the number of caches exceeds the threshold, the old cache is deleted according to the "Least Recently Used (LRU)" policy to prevent memory overflow.

[0135] Experiments have shown that this caching mechanism can improve inference speed by 20% to 30%. Taking the TaxiBJ traffic flow prediction dataset as an example, the speed is increased to 5238 FPS after adopting the caching mechanism of this invention (about 1.2 times that of the original version and about 10 times that of SimVP), which meets the low latency requirements of real-time traffic scheduling.

[0136] (III) Implementation of auxiliary modules for adapting to Improved3DRoPE

[0137] 1. Transformer encoder (gated Transformer layer)

[0138] To ensure the synchronous transfer of position tensors and feature tensors, a new position tensor transfer channel has been added to the original four branches of PredFormerLayer. The functions and dimensions of each branch are adapted as follows:

[0139] Branch 1: ts-t (Time-to-Time Attention Branch) Function: Captures dependencies between different time steps (e.g., the correlation between cloud image changes in frame t and frame t+1 in weather forecasting); Dimensional Adaptation: Rearranges the feature tensor from (B, T, N, dim) to (B×N, T, dim), and synchronously rearranges the position tensor to (B×N, T, 3), ensuring consistency in time step dimensions; Gating Mechanism: Introduces a sigmoid gating layer to perform weighted fusion of the attention output and the original features (weights are learned through training), as shown in the following formula:

[0140]

[0141]

[0142] Among them, W_g and b_g are gating layer parameters. This mechanism can suppress redundant features and enhance effective spatiotemporal dependencies.

[0143] Branch 2: ts-s (Time-Space Attention Branch) Function: Captures the cross-dependencies between different time steps and different spatial locations (such as the traffic flow association between road segment A in frame t and road segment B in frame t+2 in traffic prediction); Dimensional adaptation: Rearrange the feature tensor as (B, T×N, dim) and the location tensor as (B, T×N, 3) simultaneously, and pay attention to the association between time and spatial dimensions during attention calculation.

[0144] Branch 3: st-s (spatial-spatial attention branch) function: captures the dependencies between different spatial locations within the same time step (such as the positional association between the left and right hands within the same frame in human motion prediction); dimensionality adaptation: rearranges the feature tensor as (B×T, N, dim) and the position tensor as (B×T, N, 3) simultaneously, focusing on the local correlation of spatial dimensions.

[0145] Branch 4: st-t (spatial-temporal attention branch) function: captures the cross-dependencies of different spatial locations and different time steps (complementing the ts-s branch and strengthening the symmetry of spatiotemporal cross-correlation); the dimension adaptation and gating mechanism are the same as the ts-s branch, and finally the outputs of the four branches are merged into a unified feature (B, T, N, dim) through splicing and linear transformation.

[0146] 2. Details of Dimensional Adaptation between the Patch Embedding Layer and the MLP Decoder

[0147] Patch Embedding Layer: If the input video is a grayscale image (C=1), patch_size=16, H=W=256, then the feature dimension of each patch is 16×16×1=256. When mapped to dim=512 through a linear layer, the shape of the weight matrix of the linear layer is (256, 512), and the shape of the bias vector is (512,). If it is a color image (C=3), then the feature dimension of the patch is 16×16×3=768, and the shape of the weight matrix of the linear layer is (768, 512), ensuring that the input and output dimensions match.

[0148] The MLP decoder head consists of two fully connected layers and one GELU activation function. The first fully connected layer maps the dim-dimensional features to a C×patch_size² dimension (e.g., when dim=512, C=3, and patch_size=16, the output dimension is 3×16×16=768). The second fully connected layer fine-tunes the feature distribution, and the GELU activation function enhances the model's non-linear expressive power. The formula is as follows:

[0149]

[0150]

[0151] Here, W_1 and b_1 are the parameters of the first layer, and W_2 and b_2 are the parameters of the second layer. Finally, the video frame format is restored through the reshape operation.

[0152] (iv) Model training and inference process

[0153] 1. Training process (including parameter configuration and regularization strategy)

[0154] Step 1.1: Data Preprocessing and Augmentation

[0155] Data cleaning: Remove abnormal data (such as missing satellite cloud images in weather data, and monitoring fault frames in traffic data), and use linear interpolation to fill in a small number of missing values;

[0156] Normalization: Normalize pixel values ​​to the [0,1] range (grayscale image: pixel = pixel / 255; color image: pixel = pixel / 255 for each RGB channel) to avoid model training instability caused by different channel / pixel value ranges;

[0157] Data augmentation:

[0158] Spatial enhancement: random cropping (cropping ratio is 0.8 to 1.0 times the original size), horizontal flipping (probability 50%), vertical flipping (probability 30%), suitable for scenarios with no semantic difference between top and bottom such as weather maps, but not suitable for human motion maps.

[0159] Temporal augmentation: time step shuffling (randomly select T' time steps (T'=0.8T~T) from the input T time steps and rearrange them to simulate temporal perturbations), time scale scaling (stretch / compress the time step by 10%~20%, such as stretching 12 frames of data into 14 frames to enhance the model's robustness to temporal scale changes).

[0160] Dataset partitioning: Divide the dataset into training set, validation set, and test set in a 7:2:1 ratio (e.g., the TaxiBJ dataset contains 1 million frames of data, which are divided into 700,000 frames for training, 200,000 frames for validation, and 100,000 frames for testing).

[0161] Step 1.2: Parameter Initialization

[0162] Improved3DRoPE parameters: logits are initialized to [0.4, 0.3, 0.3]. The scaling factor network (two fully connected layers) uses Xavier initialization (to ensure consistent gradient variances during forward and backward propagation), as shown in the formula:

[0163]

[0164] Where n_in is the input dimension and n_out is the output dimension;

[0165] Transformer encoder parameters: Linear layers are initialized with a truncated normal distribution (std=0.02, truncation range is [-2std, 2std], to avoid gradient explosion caused by extreme values), and the weights of LayerNorm layers are initialized to 1.0 and the biases are initialized to 0;

[0166] Patch embedding layer and MLP decoder header parameters: same as the linear layer initialization strategy of the Transformer encoder.

[0167] Step 1.3: Optimize Configuration

[0168] Optimizer: The AdamW optimizer is used, with the weight decay coefficient set to 1e-2 (to suppress overfitting), β1=0.9, β2=0.999 (momentum parameters to accelerate convergence).

[0169] Learning rate: The initial learning rate is set to 5e-4 to 1e-3 (adjusted according to the dataset size; 1e-4 for small datasets and 5e-4 for large datasets). A cosine annealing learning rate scheduling strategy is used, as shown in the following formula:

[0170]

[0171] Where max_epoch is the total number of training epochs (usually set to 100-300 epochs). This strategy can reduce the learning rate in the later stages of training, so that the model can converge stably to the optimal solution.

[0172] Loss function: The L2 loss function (mean squared error) is used to calculate the pixel-level error between the predicted frame and the real frame.

[0173] Step 1.4: Regularization Strategy

[0174] Dropout: Add a Dropout layer between the output layer of the Attention module and the MLP layer, with the dropout probability set to 0.1 (randomly discarding 10% of the features to suppress overfitting);

[0175] Stochastic Depth: Adds layers of random depth between layers of the Transformer encoder, with drop_path probability set to 0.1 (randomly skipping 10% of the Transformer layers to enhance the model's generalization ability).

[0176] Label smoothing: Replace the true label y_true with y_true×(1 - ε) + ε / C (ε=0.1, C is the number of classes, here C=1, so y_smooth = 0.9×y_true + 0.1), to avoid the model being overconfident in the label and improve robustness.

[0177] Step 1.5: Iterative Training

[0178] S1. Load the training set data and input it into the model in batches;

[0179] S2. Forward Propagation: Perform Patch embedding → Improved3DRoPE generates position tensor → Transformer encoder fuses features and position → MLP decoder outputs predicted frame;

[0180] S3. Calculate the loss value: Calculate the L2 loss based on the predicted frame and the real frame;

[0181] S4. Backpropagation: The gradient is calculated using an automatic differentiation framework (such as PyTorch's autograd), and all model parameters (including Improved3DRoPE's logits, scaling factor network parameters, Transformer encoder parameters, Patch embedding layer and MLP decoder header parameters) are updated through the AdamW optimizer.

[0182] S5. Validation and Model Saving: Every 5 training rounds, evaluate the model performance on the validation set (calculate MSE, MAE, and SSIM metrics). If the validation set MSE is lower than the historical best value, save the current model weights (i.e., "optimal model weights") to avoid overfitting in the later stages of training.

[0183] 2. Reasoning process (including performance evaluation)

[0184] Step 2.1: Input Data Preprocessing

[0185] The preprocessing is consistent with that during the training phase (normalization, fixed-resolution cropping), but no data augmentation is performed (to avoid introducing additional perturbations that could affect the prediction results).

[0186] Step 2.2: Model Loading and Initialization

[0187] Load the "best model weights" saved during training, initialize the cache of Improved3DRoPE (initially empty), and set the inference mode (such as PyTorch's model.eval(), and disable training-specific modules such as Dropout and random depth).

[0188] Step 2.3: Location Encoding and Spatiotemporal Fusion

[0189] The Patch embedding layer converts the input video sequence into a token sequence (B, T, N, dim);

[0190] The Improved3DRoPE module generates a 3D position tensor based on the input shape. If the input shape already exists in the cache, the rotation matrix is ​​called directly; otherwise, the rotation matrix is ​​calculated and stored in the cache.

[0191] The Transformer encoder synchronously inputs the token sequence and position tensor into four branches, and outputs (B, T, N, dim) after fusing spatiotemporal features;

[0192] Step 2.4: Predicted Frame Output and Performance Evaluation

[0193] The MLP decoder head maps the encoder output to a predicted frame sequence (B, T, C, H, W);

[0194] Performance evaluation: Calculate the following metrics for predicted frames and ground truth frames:

[0195] MSE (mean squared error): measures the squared mean of pixel-level errors; the smaller the value, the higher the precision.

[0196] MAE (Mean Absolute Error): Measures the mean absolute value of pixel-level errors and is more robust to outliers.

[0197] SSIM (Structural Similarity Index): Measures the structural similarity between predicted and ground truth frames, ranging from -1 to 1. The closer to 1, the more consistent the structures.

[0198] Output results: The predicted frame sequence (such as weather cloud map for the next 12 hours, traffic flow heat map for the next 30 minutes) and performance indicators are used to generate an analysis report for downstream applications (such as weather warning system, traffic dispatch system).

[0199] The embodiments described above are merely preferred embodiments of the present invention and are not intended to impose any limitation on the invention. Other possible variations and improvements can be explored based on the technical solutions described in the claims. These variations and modifications also fall within the scope of the present invention.

Claims

1. An end-to-end spatiotemporal prediction method based on improved 3D rotational position coding, which uses a spatiotemporal prediction model as input to a video sequence and outputs prediction results for future frames, characterized by: The spatiotemporal prediction model is a pure Transformer architecture with the Improved 3DRoPE module as its core. The pure Transformer architecture is a non-looping and non-convolutional architecture. In the spatiotemporal prediction model, the input video sequence is sequentially processed through four stages: feature embedding, location encoding, spatiotemporal fusion, and prediction output to achieve spatiotemporal sequence prediction. The specific steps include: Step 1. Preprocess and augment the input video sequence, and divide it into training, validation and test sets; Step 2. Convert the preprocessed video sequence into a uniform-dimensional patch token sequence using the Patch embedding layer; Step 3. Generate an adaptive 3D position tensor using the Improved3DRoPE module; the Improved3DRoPE module includes: 1) Learnable 3D dimension adaptive allocation mechanism: The proportion of time, height and width encoding dimensions (dim) is dynamically adjusted through learnable parameters to ensure adaptation to different scenario requirements; 2) Spatiotemporal location collaborative modeling mechanism: By fusing absolute time location and relative spatial location information, it provides complete spatiotemporal positioning for the token; 3) Deep integration mechanism with the Attention module: Position information is injected into the Q / K dimension-wise rotation operation through rotation operation, and each dimension of the query Q and key K vectors is rotated separately, so that the attention weights are directly guided by the position relationship; 4) Rotation matrix caching mechanism: used to avoid repeated calculations under the same input shape, thereby improving inference efficiency; Step 4. Synchronously input the patch token sequence and the 3D position tensor into the Transformer encoder to extract global spatiotemporal dependency features; Step 5. The MLP decoder maps the features fused with location information back to the pixel space and outputs the predicted frame sequence; In step 3, the specific process of the learnable three-dimensional adaptive allocation mechanism includes: 1.1) Initialization of learnable parameters Introduce a learnable parameter logits with shape (3,) and initial values ​​set to [0.4, 0.3, 0.3]. 1.2) Weight Normalization The logits are normalized using the softmax function to ensure that the sum of the three-dimensional weights is 1, as shown in the following formula: ， Where i=0 corresponds to the time dimension, i=1 corresponds to the height dimension, and i=2 corresponds to the width dimension. The normalized weights directly determine the encoding proportion of each dimension. 1.3) Dimensional Encoding and Dimensional Calculation Based on the token feature dimension dim, calculate the actual encoding dimension of each dimension, and ensure that each dimension retains at least 2 features by max(2,・), as shown in the following formula: Time dimension encoding dimension: ; Height dimension encoding dimension: ; Width dimension encoding dimension: ; 1.4) Dimension Overflow Adaptation like Then scale each dimension proportionally to ensure the total dimension matches dim, as shown in the following formula: ， If a dimension is less than 2 after scaling, it is forcibly set to 2, and the corresponding dimension is subtracted from the other dimensions to ensure that... .

2. The end-to-end spatiotemporal prediction method based on improved three-dimensional rotational position coding according to claim 1 is characterized by further including step 6. Adjusting model parameters based on the error between the predicted frame and the real frame, optimizing the performance of the spatiotemporal prediction model through iterative training, and using a caching mechanism to improve computational efficiency during the inference stage.

3. The end-to-end spatiotemporal prediction method based on improved three-dimensional rotational position coding according to claim 1, characterized in that: Step 3, the specific steps of the spatiotemporal location collaborative modeling mechanism include: 2.1) Generation of absolute time position Generate an absolute time coordinate sequence based on the input time step T. This represents the global position at each time step; then, Min-Max normalization is applied to... Mapping to the [0,1] interval, the formula is as follows: ， 2.2) Generation of relative spatial position First, determine the patch grid size: let the resolution of the input frame be H×W and the patch_size be s, then the number of patches in the height direction num_patches_h=H / s, and the number of patches in the width direction num_patches_w=W / s; For each patch (i,j), calculate its relative coordinates with all other patches (i',j') first: Altitude relative coordinates: , Width relative coordinates: , where, i is the index in the height direction, 0≤i<num_patches_h; j is the index in the width direction, 0≤j<num_patches_w; Then normalize the relative coordinates to the interval [-1,1], where a positive value indicates "the current patch is above / to the right of the target patch", a negative value indicates "below / to the left", and 0 indicates "the same position"; 2.3) Combination of three-dimensional position tensors Stack the absolute time position with shape=(T), the height relative coordinates with shape=(N,N), and the width relative coordinates with shape=(N,N) along the batch dimension to form a three-dimensional position tensor pos_tensor with a shape of (B,T,N,3); this provides each token with a complete description of "when (time), where (space), and the relationship with other positions"; where, each element pos_tensor[b][t][n][0] corresponds to the absolute time position of the n-th patch at the t-th time step in the b-th batch; pos_tensor[b][t][n][1] corresponds to the height relative coordinate; pos_tensor[b][t][n][2] corresponds to the width relative coordinate.

4. The end-to-end spatiotemporal prediction method based on improved three-dimensional rotational position encoding according to claim 3, characterized in that: In step 3, the specific steps of the deep fusion mechanism with the Attention module include: 3.1) Dimension adaptation of position tensor Since the Transformer encoder adopts a multi-head attention mechanism, broadcast the three-dimensional position tensor pos_tensor (B,T,N,3) to each attention head, and adjust the dimension to match the shape of Q / K. The method is as follows: First convert pos_tensor to (B,heads,T,N,3) through the rearrange operation to achieve splitting by attention heads; Then merge the time step T and the number of patches N to obtain (B×heads,T×N,3), which is consistent with the shape of Q / K (B×heads,T×N,dim_head); wherein, dim_head=dim / heads represents the feature dimension of each attention head; 3.2) Dimensional rotation operation for Q / K Rotation operations are performed on the time, height and width dimension features of Q / K respectively, and the rotation matrix is generated based on the corresponding dimension information in the position tensor, as detailed below: a. Rotation in the time dimension: Extract the features of dim_t dimension in Q / K and record them as (Q_t, K_t). Calculate the rotation matrix based on the absolute time position time_pos, including the sine matrix sin_rot and the cosine matrix cos_rot, both of which have a shape of (B×heads,T×N,dim_t / / 2,2); ， ， The rotation operation of K_t is consistent with that of Q_t, and K_t_rot is obtained; The temporal location information is encoded into the phase change of the feature using trigonometric functions; b. Rotation along the height / width dimension: Similarly, extract the features of the dim_h and dim_w dimensions in Q / K and denote them as (Q_h, K_h) and (Q_w, K_w). Calculate the rotation matrix based on the relative spatial positions rel_pos_h and rel_pos_w, and perform the same rotation operation as in the time dimension to obtain Q_h_rot, K_h_rot and Q_w_rot, K_w_rot. 3.3) Post-rotation attention calculation: Concatenate the rotated Q-dimensional features (Q_t_rot, Q_h_rot, Q_w_rot) into a complete Q_rot, and concatenate the K-dimensional features (K_t_rot, K_h_rot, K_w_rot) into a complete K_rot; calculate the dot product attention weights using the following formula: ， in, The scaling factor is denoted by `mask`, and the attention mask is used for the final attention output. , V is the Value tensor, which comes from the same token feature as Q / K, and no rotation operation was performed to preserve the original feature information.

5. The end-to-end spatiotemporal prediction method based on improved three-dimensional rotational position coding according to claim 4, characterized in that: Step 3, the specific steps of the rotation matrix caching mechanism include: 4.1) The cache key is defined as "input shape positions_shape + encoded dimensions dim_per_axis", where: positions_shape: includes time step T and patch grid size (num_patches_h, num_patches_w), i.e., positions_shape=(T,num_patches_h,num_patches_w); dim_per_axis: The encoding dimension that includes time, height, and width, i.e., dim_per_axis=(dim_t,dim_h,dim_w); 4.2) The cached content stores the cached value as a calculated sine matrix sin_mat and a cosine matrix cos_mat; the shapes are (B×heads,T×N,dim / / 2,2), which are rotation matrices covering the three dimensions of time, height, and width; the storage method is memory cache or disk cache. 4.3) Cache update and invocation logic During reasoning, first calculate the cache key based on the current input; If the key already exists in the cache, the corresponding sin_mat and cos_mat are called directly, skipping the rotation matrix calculation step; If the key does not exist, perform calculations such as theta=10000^(-2k / dim), angles=positions*theta, sin_mat=sin(angles), cos_mat=cos(angles) to generate a rotation matrix and store it in the cache; Cache invalidation policy: When the input shape (T, num_patches_h, num_patches_w) or dim_per_axis changes, the old cache is cleared; at the same time, the maximum cache capacity is set, and when the number of caches exceeds the threshold, the old cache is deleted according to the least recently used (LRU) policy.

6. The end-to-end spatiotemporal prediction method based on improved three-dimensional rotational position coding according to claim 1, Its characteristic is that in step 4, the Transformer encoder includes 6 to 12 gated Transformer layers, each Transformer layer consists of four branches in sequence: ts-t, ts-s, st-s, and st-t. The patch token sequence and the three-dimensional position tensor are synchronously input into the Transformer encoder, and the global spatiotemporal dependency features are extracted through the four-branch structure. Based on the original four branches of PredFormerLayer, a new position tensor transfer channel has been added. The functionality and dimensions of each branch are adapted as follows: First branch: Time-Time Attention ts-t branch Function: Captures dependencies between different time steps; Dimensional adaptation: The feature tensor is rearranged from (B,T,N,dim) to (B×N,T,dim), and the position tensor is simultaneously rearranged to (B×N,T,3) to ensure that the time step dimensions of the two are consistent. Gating mechanism: A sigmoid gating layer is introduced to perform weighted fusion of the attention output and the original features, as shown in the following formula: ， ， Where W_g and b_g are the gating layer parameters; Second branch: Time-space attention (TS-S) branch Function: Capture cross-dependencies at different time steps and spatial locations; Dimensional adaptation: The feature tensor is rearranged as (B, T×N, dim), and the position tensor is simultaneously rearranged as (B, T×N, 3). Attention calculation considers the relationship between the temporal and spatial dimensions simultaneously. Third branch: Spatial-spatial attention st-s branch Function: Capture the dependencies between different spatial locations within the same time step; Dimensional adaptation: The feature tensor is rearranged as (B×T,N,dim), and the position tensor is simultaneously rearranged as (B×T,N,3), focusing on the local correlation of spatial dimensions; Fourth branch: Spatial-temporal attention st-t branch Function: Capture cross-dependencies at different spatial locations and time steps; Dimensional adaptation: The feature tensor is rearranged as (B, T×N, dim), and the position tensor is simultaneously rearranged as (B, T×N, 3). Attention calculation considers the relationship between the temporal and spatial dimensions simultaneously. Finally, the outputs of the four branches are merged into a unified feature (B, T, N, dim) through concatenation and linear transformation.

7. The end-to-end spatiotemporal prediction method based on improved three-dimensional rotational position coding according to claim 1, characterized in that: The specific dimensionality adaptation between the Patch embedding layer in step 2 and the MLP decoding head in step 5 is as follows: Patch embedding layer: If the input video is a grayscale image C=1, patch_size=16, H=W=256, then the feature dimension of each patch is 16×16×1=256. When mapped to dim=512 through the linear layer, the shape of the weight matrix of the linear layer is (256,512), and the shape of the bias vector is (512,). If the image is a color image with C=3, then the patch feature dimension is 16×16×3=768, and the shape of the linear layer weight matrix is ​​(768,512). MLP decoder header: consists of two fully connected layers and one GELU activation function layer; The first fully connected layer maps the dim-dimensional features to the C×patch_size² dimension; the second fully connected layer fine-tunes the feature distribution; the GELU activation function enhances the model's non-linear expressive power, as shown in the following formula: ， ， Among them, W_1 and b_1 are the first layer parameters, and W_2 and b_2 are the second layer parameters. Finally, the video frame format is restored through the reshape operation.

8. The end-to-end spatiotemporal prediction method based on improved three-dimensional rotational position coding according to claim 1, characterized in that: Step 6, the spatiotemporal prediction model training process, includes: S1.1: Data Preprocessing and Augmentation Data cleaning: Remove outlier data and use linear interpolation to fill in a small number of missing values; Normalization: Normalizes pixel values ​​to the [0,1] range. For grayscale images, it is pixel=pixel / 255, and for color images, it is pixel=pixel / 255 for each RGB channel. Data augmentation: a) Spatial enhancement methods include: random cropping, horizontal flipping, and vertical flipping; b. Time augmentation is achieved by: shuffling and rearranging time steps, and scaling the time scale. c. Dataset partitioning: Divide the dataset into training set, validation set, and test set according to a set ratio; S1.2: Parameter Initialization Improved3DRoPE parameters: logits are initialized to [0.4, 0.3, 0.3], and the scaling factor network is initialized using Xavier, with the formula as follows: ， Where n_in is the input dimension and n_out is the output dimension; Transformer encoder parameters: Linear layers are initialized using a truncated normal distribution, std=0.02, with a truncation range of [-2std, 2std]; LayerNorm layers have their weights initialized to 1.0 and their biases initialized to 0. Patch embedding layer and MLP decoder header parameters: Same as the linear layer initialization strategy of the Transformer encoder; S1.3: Optimized Configuration Optimizer: The AdamW optimizer is used, with the weight decay coefficient set to 1e-2, β1=0.9, β2=0.999; Learning rate: The initial learning rate is set to 5e-4 to 1e-3, and a cosine annealing learning rate scheduling strategy is adopted, as shown in the following formula: ， Where max_epoch is the total number of training rounds; Loss function: The L2 loss function is used to calculate the pixel-level error between the predicted frame and the real frame; S1.4: Regularization Strategy Dropout: Add a Dropout layer between the output layer of the Attention module and the MLP layer, with the dropout probability set to 0.1; StochasticDepth: Adds a random depth layer between layers in the Transformer encoder, with the drop_path probability set to 0.1; Label smoothing: Replace the true label y_true with y_true×(1-ε)+ε / C; ε=0.1, C is the number of categories, here C=1, then y_smooth=0.9×y_true+0.1; S1.5: Iterative Training a1) Load the training set data and input it into the model in batches; a2) Forward propagation: Patch embedding is performed → Improved3DRoPE generates position tensors → Transformer encoder fuses features and positions → MLP decoder head outputs predicted frames; a3) Calculate the loss value: Calculate the L2 loss based on the predicted frame and the real frame; a4) Backpropagation: The gradient is calculated using an automatic differentiation framework, and all parameters of the model are updated by the AdamW optimizer. The parameters include: logits of Improved3DRoPE, scaling factor network parameters, Transformer encoder parameters, Patch embedding layer and MLP decoder header parameters. 5) Validation and model saving: Every 5 training rounds, the model performance is evaluated on the validation set by calculating MSE, MAE, and SSIM. If the validation set MSE is lower than the historical best value, the current model weights are saved as the best model weights.

9. The end-to-end spatiotemporal prediction method based on improved three-dimensional rotational position coding according to claim 8, characterized in that: In step 6, the model inference process includes: S2.1: Input data preprocessing The preprocessing is consistent with that during the training phase, but without data augmentation. S2.2: Model Loading and Initialization Load the best model weights saved during training, initialize Improved3DRoPE's cache to empty, and set the inference mode; S2.3: Location Coding and Spatiotemporal Fusion The Patch embedding layer converts the input video sequence into a token sequence (B,T,N,dim); The Improved3DRoPE module generates a 3D position tensor based on the input shape. If the input shape already exists in the cache, the rotation matrix is ​​called directly; otherwise, the rotation matrix is ​​calculated and stored in the cache. The Transformer encoder synchronously inputs the token sequence and position tensor into four branches, and outputs (B, T, N, dim) after fusing spatiotemporal features. S2.4: Predicted Frame Output and Performance Evaluation The MLP decoder head maps the encoder output to a predicted frame sequence (B,T,C,H,W); Performance evaluation: Calculate the following metrics for predicted frames and ground truth frames: Mean Square Error (MSE): Measures the squared mean of pixel-level errors; the smaller the value, the higher the accuracy. Mean Absolute Error (MAE): Measures the mean absolute value of pixel-level errors and is more robust to outliers; Structural Similarity Index (SSIM): Measures the structural similarity between predicted and ground truth frames, ranging from -1 to 1. The closer to 1, the more consistent the structures. Output results: The predicted frame sequence and performance metrics will be used to generate an analysis report for downstream applications.

Images