A traffic flow prediction method and system based on multi-view spatio-temporal modeling

Abstract

This invention provides a traffic flow prediction method and system based on multi-perspective spatiotemporal modeling, belonging to the field of traffic information management technology. This invention decouples the spatiotemporal modeling process into two complementary perspectives: implicit spatiotemporal modeling and globally explicit spatiotemporal relationships. Based on a selective state-space model architecture, it performs in-depth analysis of input features from both channel and spatial location perspectives, thereby constructing a multi-perspective temporal understanding path. Furthermore, since the implicit and globally explicit temporal modeling capture corresponding temporal information from the channel and spatial location perspectives respectively, a channel space attention module is introduced to effectively filter out unimportant temporal information interference, further improving the overall model's traffic flow prediction capability. The method described in this invention not only effectively improves the accuracy of traffic flow prediction, but its design concept can also provide valuable insights for other spatiotemporal data-driven downstream tasks (such as weather forecasting).

Description

Technical Field

[0001] This invention relates to the field of traffic information management technology, and in particular to a traffic flow prediction method and system based on multi-view spatiotemporal modeling. Background Technology

[0002] Traffic flow prediction is one of the core tasks of Intelligent Transportation Systems (ITS). Its goal is to predict short-term or long-term traffic conditions by analyzing historical traffic surveillance video data and modeling the movement patterns of traffic participants such as vehicles and pedestrians. Accurate prediction is crucial for applications such as traffic management, congestion warning, and route planning. With the development of deep learning technology, spatiotemporal modeling methods based on video sequences have become the mainstream in this field.

[0003] Currently, mainstream traffic flow prediction methods mainly focus on how to effectively capture the spatiotemporal dependencies between video frames, and can be roughly divided into the following categories: 1. Methods based on Recurrent Neural Networks (RNNs) and their variants: Early methods often used recurrent neural network units such as LSTM and GRU to model temporal dynamics. These methods input the features of each frame sequentially into the RNN, implicitly passing historical information. However, the inherent sequential processing nature of RNNs leads to low training efficiency and the problem of vanishing / exploding gradients, making it difficult to capture long-distance spatiotemporal dependencies. Although subsequent models such as ConvLSTM were proposed, introducing convolutional operations to capture spatial features, their core temporal modeling mechanism is still limited by the sequential bottleneck of RNNs, making it difficult to fully model complex global spatiotemporal interactions.

[0004] 2. Methods based on 3D Convolutional Neural Networks (3D-CNN): These methods treat the temporal dimension as a third dimension parallel to the spatial dimension, using 3D convolutional kernels to extract spatiotemporal features simultaneously. While 3D-CNNs can directly process video blocks, their computational cost is enormous, and the fixed receptive field of the convolutional kernels limits their ability to model long-range, dynamically changing spatiotemporal relationships. The models often focus more on capturing local, short-term patterns, and their modeling effectiveness is limited for long-range dependencies commonly found in traffic flow, caused by global traffic conditions (such as the transmission of congestion from distant locations).

[0005] 3. Transformer-based approaches: In recent years, Vision Transformer and its spatiotemporal variants have shown potential in capturing long-range dependencies by explicitly modeling the relationships between all spatiotemporal locations (tokens) through a self-attention mechanism. However, when directly applied to high-resolution, long-sequence video data, the computational complexity of the self-attention mechanism is proportional to the square of the number of spatiotemporal tokens, leading to a sharp increase in memory consumption and computational costs, severely limiting its feasibility in real-time or large-scale traffic prediction scenarios. Furthermore, a single global attention mechanism may treat all spatiotemporal relationships equally, lacking the ability to differentiate the spatiotemporal interactions at multiple levels (such as local vehicle following and global road network traffic) in traffic scenarios.

[0006] In summary, the main challenges facing existing technologies can be summarized as follows: (1) Single modeling perspective: Existing methods mostly rely on a single temporal modeling paradigm, failing to systematically deconstruct and integrate spatiotemporal dependencies from multiple complementary perspectives, resulting in an insufficient representation of complex traffic dynamics.

[0007] (2) Long-range dependency and computational efficiency are difficult to balance: RNN and 3D-CNN are difficult to capture long-range dependency efficiently, while Transformer has potential but is constrained by high computational complexity.

[0008] (3) Insufficient utilization of features: In the modeling process, the original spatiotemporal features are often not screened and enhanced in a targeted manner. Non-critical information may interfere with the extraction of core spatiotemporal relationships and affect the accuracy of the model.

[0009] Therefore, there is an urgent need for a traffic flow prediction method and system based on multi-perspective spatiotemporal modeling to solve the technical problem that current methods simply merge the time dimension (T) and the channel dimension (C) for unified processing, or rely on a single global attention mechanism in the Transformer architecture to model temporal dependencies, without capturing the spatiotemporal feature relationships from multiple perspectives in the spatiotemporal process. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of the prior art and provide a traffic flow prediction method and system based on multi-view spatiotemporal modeling. It aims to integrate multi-view temporal understanding to efficiently and accurately capture the complex spatiotemporal dependencies of local and global, explicit and implicit data in traffic flow data, thereby significantly improving the accuracy and robustness of traffic flow prediction under controllable computational cost.

[0011] To achieve the above objectives, this application proposes a traffic flow prediction method based on multi-view spatiotemporal modeling, comprising the following steps: Step S1: Encode the input continuous traffic video frames frame by frame using an encoder to obtain the semantic features of each frame; Step S2: Input the semantic features into the time series module to perform spatiotemporal relationship modeling; Step S3: In the temporal module, the input semantic features are filtered using the channel space attention module to filter out unimportant temporal information; Step S4: Perform dimensional transformation on the filtered semantic features. Perform dimensional transformation on the semantic features of consecutive frames to obtain the first temporal dimension shape corresponding to the first temporal angle and the second temporal dimension shape corresponding to the second temporal angle. Step S5: Perform two-dimensional cross-scanning and three-dimensional cross-scanning on the first time-series angle and the second time-series angle respectively to obtain the semantic features of the one-dimensional sequence in different directions under each time-series angle; Step S6: Input the semantic features of one-dimensional sequences in different directions under each temporal angle into different spatiotemporal relation extraction base models; wherein, the spatiotemporal relation extraction base model is constructed based on a selective state space module and a linear gating structure, and is used to extract temporal relations; Step S7: The semantic features of the one-dimensional sequences in different directions after being processed by the base model of spatiotemporal relation extraction under each temporal angle are fused and mapped to the shape of the image feature dimension; then, based on the first temporal angle and the second temporal angle, different angle temporal fusion is performed so that each semantic feature obtains spatiotemporal information of different directions and different temporal angle modeling methods, and thus obtains temporal semantic features; Step S8: Input the fused temporal semantic features into the next layer temporal module and repeat steps S3 to S7; wherein the number of repetitions is consistent with the number of model layers; Step S9: After the temporal processing is completed, the temporal semantic features obtained are decoded into the pixel space by the decoder, and then decoded into the predicted traffic flow video frames.

[0012] As a further solution, the encoder consists of multiple identical layer structures, with each layer comprising two sub-layers; wherein, The first sub-layer consists of 2D convolution, GroupNorm normalization, and Silu activation function, which are used for feature extraction through convolution operations. The second sub-layer consists of 2D convolution, GroupNorm normalization, and Silu activation function, which are used to adjust the convolution stride to achieve downsampling operation; The encoder outputs a jump connection feature to the decoder.

[0013] As a further solution, the number of layers in the encoder is determined based on the input data resolution, and the features processed by the first layer in the encoder need to be returned at the end to implement the skip key structure.

[0014] As a further solution, the channel spatial attention module is used to simultaneously support the modeling of implicit spatiotemporal relationships and global explicit spatiotemporal relationships, and introduces channel spatial attention to filter important spatiotemporal relationship features; wherein, the first temporal perspective extracts implicit spatiotemporal relationships from the channel level, and the second temporal perspective extracts global explicit spatiotemporal relationships from spatial location attributes.

[0015] As a further solution, in step S4, the dimensional transformation is as follows: First time sequence angle = [B, T*C1, H, W] Second time series angle = [B,C2,T,H,W] Where B is the batch size, T is the number of frames, C1 and C2 are the number of channels for the two viewpoints, and H and W are the height and width of the feature map.

[0016] As a further solution, in step S5, the two-dimensional cross scan and the three-dimensional cross scan respectively include four directions: horizontal scan, vertical scan, reverse horizontal scan, and reverse vertical scan. The specific dimensional transformation is represented as follows: Dimensional transformation from the first time-series perspective: Horizontal scan = [B, T*C1, H*W] Vertical scan = B, T*C1, W*H] Reverse transverse scan = [B, T*C1, -H*W] Reverse vertical scan = [B, T*C1, - W*H] Dimensional transformation from the second time-series perspective: Horizontal scan = [B, C2, T*H*W] Vertical scan = [B, C2, H*T*W] Reverse transverse scan = [B, C2, -T*H*W] Reverse vertical scan = [B, C2, -T*W*H] Where B represents the batch size of the training model, T represents the number of consecutive input frames, C1 represents the channel dimension used in the first time sequence angle, C2 represents the channel dimension used in the second time sequence angle, and - represents the reverse order direction.

[0017] As a further solution, the structure of the spatiotemporal relation extraction base model is as follows: z = SSM(σ(Linear(x)) s = σ(Linear(x)) o = Linear(z ⊙ s) Where z represents the latent temporal features after processing by the selective state-space model SSM, s represents the gated signal generated by the input through a linear layer and an activation function, o represents the final output feature of the spatiotemporal relation extraction base model, x represents the input feature, SSM represents the Mamba selective state-space model, Linear represents the linear layer, σ represents the activation function Silu, and ⊙ represents the Hadamard product.

[0018] As a further solution, in step S7, the time series fusion from different angles is used to fuse the spatiotemporal relationships of one-dimensional time series from different directions and angles after processing by the spatiotemporal relationship extraction base model in step S6. The specific execution method is as follows: The one-dimensional sequences of horizontal scanning, vertical scanning, anti-horizontal scanning, and anti-vertical scanning corresponding to the first temporal angle are converted into shapes [B, T*C1, H*W], then directly added together, and the linear layer mapping in the linear gated structure of the base model is extracted through the spatiotemporal relationship as [B, T*C, H*W]; where C is the channel dimension; The one-dimensional sequence of the horizontal scan, vertical scan, anti-horizontal scan, and anti-vertical scan corresponding to the second temporal angle is converted into the shape [B, C2, T*H*W]. Then, the four directions are directly added according to the specified dimensions, and the linear layer in the linear gating structure of the base model is extracted and mapped to [B, C, T*H*W] through the spatiotemporal relationship.

[0019] As a further solution, the decoder has a structure symmetrical to the encoder, with each layer consisting of two sub-layers: the first sub-layer consists of 2D convolution, GroupNorm normalization, and Silu activation function for upsampling; the second sub-layer consists of 2D convolution, PixelShuffle upsampling, GroupNorm normalization, and Silu activation function for feature decoding; the decoder receives skip-connected features from the encoder.

[0020] On the other hand, the present invention also provides a traffic flow prediction system based on multi-view spatiotemporal modeling, for implementing a traffic flow prediction method based on multi-view spatiotemporal modeling as described in any of the preceding claims, comprising: The data processing module is used to normalize traffic flow video data; An encoder is used to encode consecutive video frames and extract semantic features; The temporal module is used for multi-perspective spatiotemporal relationship modeling and fusion of semantic features; A decoder is used to decode temporal features into predicted video frames; The temporal module includes a channel spatial attention submodule, a dimension transformation submodule, a cross-scan submodule, a spatiotemporal relationship extraction base model submodule, and a feature fusion submodule.

[0021] Compared with related technologies, the traffic flow prediction method and system based on multi-view spatiotemporal modeling provided by this invention has the following advantages: 1. This invention decouples spatiotemporal relationships into two complementary perspectives: "implicit temporal relationships in the channel dimension" and "explicit temporal relationships in the spatial dimension," and designs targeted modeling paths (two-dimensional and three-dimensional cross-scanning) for each, overcoming the limitations of existing single modeling paradigms (such as using only convolution, RNN, or attention). This method can collaboratively capture local fine-grained evolution and global structural changes in traffic flow data, achieving a more comprehensive and deeper representation of complex spatiotemporal dynamics, thereby significantly improving prediction accuracy. As shown in Table 1, on the same dataset (taxibj), the model of this invention (ours) outperforms the listed mainstream models in key indicators such as mean squared error (MSE), absolute error (MAE), and structural similarity index (SSIM). 2. The core spatiotemporal relationship extraction base model of this invention is built upon an advanced selective state-space model (such as Mamba's S6 algorithm). This structure, through a selective mechanism of input dependencies, can dynamically retain important information and filter redundancy, exhibiting excellent long-sequence modeling capabilities while theoretically possessing linear computational complexity. This avoids the single-perspective temporal modeling approach of using only Transformer, convolutional, or recursive models to capture spatiotemporal dependencies. This allows the model to maintain predictive performance while keeping the number of parameters and computational cost low (e.g., only 2.88M parameters in Table 1) when processing long-term traffic flow sequences, making it more suitable for practical deployment. 3. This invention innovatively introduces a Channel Spatial Attention (CBAM) module before temporal modeling to adaptively recalibrate the encoded spatial features. This module can automatically focus on channels and spatial regions that are crucial for predicting spatiotemporal evolution, effectively filtering out interference from background noise and other unimportant information. This provides cleaner and more discriminative input features for subsequent dual-view temporal modeling, optimizing the information flow from the source and further enhancing the robustness and accuracy of the model.

[0022] 4. The multi-perspective spatiotemporal modeling framework proposed in this invention has universality. Its core idea—to fuse temporal information from different dimensions through decoupling perspectives, selective state-space modeling, and cross-scanning strategies—is not limited to traffic flow prediction. This framework can be transferred to other sequence prediction tasks with spatiotemporal dependencies, such as weather forecasting, video frame prediction, and human motion recognition. It provides a novel and effective technical approach for solving a wide range of spatiotemporal data modeling problems and has good application prospects and promotional value. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0025] Figure 1 A schematic diagram illustrating the steps of a traffic flow prediction method based on multi-view spatiotemporal modeling provided by the present invention; Figure 2 This is a schematic diagram of the module for extracting the spatiotemporal relationship of traffic flow in the embodiment. Figure 3 This is a schematic diagram of the time series module structure of the prediction model in the embodiment; Figure 4 This is a schematic diagram of the two-dimensional cross-scan strategy in the embodiment; Figure 5 This is a schematic diagram of the three-dimensional cross-scan strategy in the embodiment; Figure 6 This is a schematic diagram illustrating the fusion of two-dimensional and three-dimensional cross-scan strategies in the embodiment; Figure 7 This is a schematic diagram of the overall prediction model architecture of the traffic flow prediction method provided by the present invention; The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0027] Example 1 Please see Figure 1 This embodiment provides a traffic flow prediction method based on multi-view spatiotemporal modeling, including the following steps: Step S1: Encode the input continuous traffic video frames frame by frame using an encoder to obtain the semantic features of each frame; Step S2: Input the semantic features into the time series module to perform spatiotemporal relationship modeling; Step S3: In the temporal module, the input semantic features are filtered using the channel space attention module to filter out unimportant temporal information; Step S4: Perform dimensional transformation on the filtered semantic features. Perform dimensional transformation on the semantic features of consecutive frames to obtain the first temporal dimension shape corresponding to the first temporal angle and the second temporal dimension shape corresponding to the second temporal angle. Step S5: Perform two-dimensional cross-scanning and three-dimensional cross-scanning on the first time-series angle and the second time-series angle respectively to obtain the semantic features of the one-dimensional sequence in different directions under each time-series angle; Step S6: Input the semantic features of the one-dimensional sequence in different directions under each time series angle into different spatiotemporal relationship extraction base models; wherein, the spatiotemporal relationship extraction base model is constructed based on the selective state space module (S6 algorithm) and the linear gating structure, and is used to extract time series relationships; Step S7: The semantic features of the one-dimensional sequences in different directions after being processed by the base model of spatiotemporal relation extraction under each temporal angle are fused and mapped to the shape of the image feature dimension; then, based on the first temporal angle and the second temporal angle, different angle temporal fusion is performed so that each semantic feature obtains spatiotemporal information of different directions and different temporal angle modeling methods, and thus obtains temporal semantic features; Step S8: Input the fused temporal semantic features into the next layer temporal module and repeat steps S3 to S7; wherein the number of repetitions is consistent with the number of model layers; Step S9: After the temporal processing is completed, the temporal semantic features obtained are decoded into the pixel space by the decoder, and then decoded into the predicted traffic flow video frames.

[0028] It should be noted that existing spatiotemporal modeling methods still have room for further exploration in capturing long-range dependencies. To further improve the predictive ability of traffic flow data, this invention decouples the spatiotemporal modeling process into two complementary perspectives: implicit spatiotemporal modeling and global explicit spatiotemporal relationships. Based on a selective state-space model architecture, it performs in-depth analysis of input features from both channel dimension and spatial location perspectives, thereby constructing a multi-faceted temporal understanding path. This design enables the model to simultaneously capture implicit dependencies in local dynamic evolution and explicit patterns in global structural changes, collaboratively characterizing the complex and heterogeneous temporal dependencies in spatiotemporal data from multiple modeling perspectives. Building upon this foundation, since implicit temporal modeling and global explicit temporal modeling capture corresponding temporal information from the perspectives of channel and spatial location, respectively, a channel spatial attention module is introduced to effectively filter out unimportant temporal information interference from the features, further enhancing the overall model's ability to predict traffic flow. The method described in this invention not only effectively improves the accuracy of traffic flow prediction, but its design concept can also provide valuable insights for other spatiotemporal data-driven downstream tasks (such as weather forecasting).

[0029] Specifically, this embodiment decouples the spatiotemporal relationships between consecutive frames into two complementary dimensions: implicit temporal information (reflected at the channel level) and global explicit temporal information (reflected at the spatial location level). By performing in-depth analysis of the features between input frames from both channel and spatial perspectives, and then effectively fusing them, this strategy significantly enhances the ability to model complex spatiotemporal dynamics, avoiding the incomplete spatiotemporal dependency information caused by relying solely on a single perspective for temporal modeling.

[0030] Furthermore, to support the aforementioned dual-perspective modeling, this invention designs a novel basic temporal series model as the core component of the temporal series module. This model is constructed based on a linear gating structure and Mamba's selective state-space model. Given that object-oriented data can essentially be viewed as a one-dimensional sequence, and Mamba exhibits superior efficiency and expressive power in long sequence modeling tasks, this invention successfully transfers its sequence modeling advantages to spatiotemporal tasks. The proposed base model not only possesses powerful long-range dependency capture capabilities but can also be directly embedded into the dual-perspective framework to achieve joint modeling of implicit and explicit temporal series information.

[0031] Compared with existing methods, this invention can more accurately characterize the spatiotemporal evolution of traffic flow, providing a more discriminative temporal representation for downstream spatiotemporal tasks such as prediction and reconstruction. It also demonstrates superior performance and generalization ability in other related tasks, such as weather forecasting and video analysis.

[0032] Specifically, to further enhance the ability of the entire prediction model to extract spatiotemporal representations, contributing to both implicit and explicit spatiotemporal modeling, a lightweight Channel Spatial Attention (CBAM) module is introduced to effectively filter out unimportant temporal information interference from features. The specific algorithm of the CBAM module is as follows: Co = σ(MLP(AP(X)) + MLP(MP(X))) Xc =Co⊙X So=σ(Conv(cat[AvgPool(Xc),MaxPool(Xc)])) Xout =So⊙Xc X represents the input feature, MLP represents the linear layer, AP and MP represent global average pooling and global max pooling, respectively, Co is the result after channel attention processing, So is the result after spatial attention processing, Conv represents convolution processing, Xc represents the result after interaction with channel weights, Xout is the output result after processing the entire module, ⊙ represents the Hadamard product, and finally the CBAM compression rate is 8.

[0033] More specifically, the selective state-space model is configured using the S6 algorithm, an efficient sequence modeling architecture. Its core idea is a key improvement upon the classic S4 algorithm. Essentially, it is a continuous state-space model, a time-invariant system, because its state transition parameters remain constant throughout the inference process. The continuous state-space model (SSM) algorithm is represented as follows: h′(t) = Ah(t) + B(t)x(t), y(t) = C(t)h(t) +Dx(t) Where, A∈C N×N B∈R N×1 C∈R 1xN and D∈R 1 Let be the system matrix, x(t)∈R be the input, and y(t)∈R be the output. h(t) represents the hidden state vector at time t, h′(t) represents the derivative of the hidden state h(t) with respect to time t, and R refers to the set of real numbers.

[0034] The key extension of the S6 algorithm to the classic continuous state-space model (SSM) lies in the introduction of input-driven dynamic parameters. Specifically, B(t), C(t), and the time step Δt are all dynamically generated from the current input x(t), thereby enabling content-aware selection of information "memory" or "forgetting." Furthermore, to adapt this continuous system to practical discrete sequence tasks, the zero-order hold algorithm (ZOH) is used for discretization. Therefore, the S6 algorithm is expressed as follows: Aˉ t =exp(Δ t A), Bˉ t =A 1 (exp(Δ t A) I)B t , h t = Aˉ t h t 1+Bˉ t x t y t = C t h t +D x t , Where, Δ t The step size for time sampling is also determined by the input parameters: exp represents the matrix exponent; I represents the identity matrix; Aˉt and Bˉt are the equivalent parameters after discretization; the remaining parameters are represented in the same way as those in the continuous state-space model algorithm.

[0035] Furthermore, the encoder consists of multiple identical layer structures, and each layer includes two sub-layers; wherein, The first sub-layer consists of 2D convolution, GroupNorm normalization, and Silu activation function, which are used for feature extraction through convolution operations. The second sub-layer consists of 2D convolution, GroupNorm normalization, and Silu activation function, which are used to adjust the convolution stride to achieve downsampling operation; The encoder outputs a jump connection feature to the decoder.

[0036] Furthermore, the number of layers in the encoder is determined based on the input data resolution. For example, if a 64x64 resolution is used in this method, a two-layer structure is used. In addition, the features processed by the first layer of the encoder need to be returned at the end to implement the skip key structure.

[0037] Furthermore, the decoder has a structure symmetrical to the encoder, with each layer comprising two sub-layers: the first sub-layer consists of 2D convolution, GroupNorm normalization, and Silu activation function for upsampling; the second sub-layer consists of 2D convolution, PixelShuffle upsampling, GroupNorm normalization, and Silu activation function for feature decoding; the decoder receives skip-connected features from the encoder.

[0038] Specifically, because the temporal module in the overall prediction model processes features at the semantic level, it can compress the input resolution, reduce GPU memory requirements, and keep the sequence length processed in the temporal module relatively short. The encoder and decoder of the overall prediction model adopt a U-Net architecture design, as shown in the schematic diagram. Figure 1 Encoder and decoder.

[0039] (1) Encoding module design: The convolutional network module in the encoder uses 2D convolution, GroupNorm, and Silu activation function as one layer. This two-layer substructure is then used as the main encoder layer. The first layer determines whether downsampling is needed, and the second layer performs convolutional network processing. The 2D convolution uses a 3×3 kernel with 1 padding size; the stride is 1 if downsampling is not needed, and 2 otherwise. GroupNorm uses a group normalization parameter of 2, which makes the model more stable during training. The Silu activation function provides the model with non-linear fitting capabilities, allowing it to learn more complex data. Finally, the features processed by the first layer of the encoder are returned to implement the skip key structure. (2) Decoder module design: The decoder and encoder are largely similar overall, but their functions are reversed. The decoder uses 2D convolution, GroupNorm, and Silu activation functions as a sub-layer primarily for upsampling. Then, a second sub-layer uses 2D convolution, PixelShuffle, GroupNorm, and Silu activation functions for progressive feature decoding. These two sub-layers are combined to form the main layer structure of the decoder, and the number of layers must match that of the encoder. The 2D convolution uses a 3×3 kernel with a padding size of 1, and the PixelShuffle parameter is 2 during upsampling, indicating a doubling of the scale. Finally, the last sub-layer of the last layer in the decoder needs to incorporate skip key features from the encoder to implement the skip key structure.

[0040] Furthermore, to directly utilize Mamba's sequence modeling capabilities in spatiotemporal sequence tasks and obtain post-features directly from the encoder, a spatiotemporal relation extraction base model is proposed here based on the Mamba Selective State-Space Model (SSM) and a linear gating structure, such as... Figure 2 As shown, the structure of the spatiotemporal relation extraction base model is as follows: z = SSM(σ(Linear(x)) s = σ(Linear(x)) o = Linear(z ⊙ s) Where z represents the latent temporal features after processing by the selective state-space model SSM, s represents the gated signal generated by the input through a linear layer and an activation function, o represents the final output feature of the spatiotemporal relation extraction base model, x represents the input feature, SSM represents the Mamba selective state-space model, Linear represents the linear layer, σ represents the activation function Silu, and ⊙ represents the Hadamard product.

[0041] The temporal module structure of this prediction model is based on extracting base models through spatiotemporal relationship concatenation to achieve implicit and globally explicit spatiotemporal relationships between features corresponding to consecutive frames. To further enhance the ability of this structure to extract spatiotemporal representation relationships, providing benefits both at the implicit and global explicit levels, a lightweight channel spatial attention module (CBAM) is introduced to effectively filter out unimportant temporal information interference from features. Figure 3 As shown. The specific model flow is as follows: H = CBAM(H) IF = S6(SS2D(H)) EF = S6(SS3D(H)) H¯ = IF + EF Where H∈R B×T×C×H×W This represents the input from the previous layer. S6 refers to the Mamba selective state-space model, IF∈R. B×T×C1×H×W Represents implicit temporal characteristics, EF∈R B×T×C2×H×W Represents a globally explicit feature. Hˉ∈R B×T×C×H×W This represents the fusion of global explicit features and implicit spatiotemporal features. SS2D and SS3D use different selective scanning strategies to extract base models for different spatiotemporal relationships. SS2D represents 2D selective scanning, and SS3D represents 3D selective scanning.

[0042] Furthermore, the complete timing module is constructed using multiple modules forming a single layer of the complete timing module structure, with each layer being identical. This module needs to capture a first timing perspective (implicit spatiotemporal relationship) and a second timing perspective (global explicit spatiotemporal relationship). The first timing perspective extracts spatiotemporal relationships from the channel level, while the second timing perspective primarily extracts spatiotemporal relationships from spatial location attributes. Therefore, to simultaneously meet the lightweight requirements and attention requirements of the entire timing module, channel spatial attention is introduced to filter important spatiotemporal relationship features.

[0043] In step S4, the semantic features of the consecutive frames undergo dimensional transformation to extract the spatiotemporal relationships from a first temporal perspective (implicit spatiotemporal relationship) and a second temporal perspective (global explicit spatiotemporal relationship). The specific dimensional transformation is as follows: First time sequence angle = [B, T*C1, H, W] Second time series angle = [B,C2,T,H,W] Where B represents the batch size of the training model, T represents the number of consecutive input frames, C1 represents the channel dimension used in the first temporal angle, C2 represents the channel dimension used in the second temporal angle, and C1 and C2 can be equal, and H and W represent the length and width of the feature map after encoder processing.

[0044] In step S5, the two-dimensional cross-scanning strategy and the three-dimensional cross-scanning strategy are used to convert the obtained three-dimensional data into a one-dimensional sequence. At the same time, when the temporal fusion is performed at different angles, each feature point can obtain temporal modeling information in different directions and at different angles, so as to obtain richer spatiotemporal information.

[0045] Regarding the two-dimensional cross-scanning strategy: Previous Mamba models performed temporal scanning on sequence data, which lacks spatial directional characteristics. Directly applying this strategy to image tasks would yield poor results. Therefore, we combine our proposed spatiotemporal relationship extraction base model with the cross-scanning strategy from the Vmamba model. By concatenating the time and channel dimensions, we capture implicit spatiotemporal relationships from a channel-level perspective. The cross-scanning strategy then involves four different directions, such as... Figure 4 As shown, the four directions of the specific 2D selective scanning are represented as follows: Horizontal scan = [B, T*C1, H*W] Vertical scan = B, T*C1, W*H] Reverse transverse scan = [B, T*C1, -H*W] Reverse vertical scan = [B, T*C1, - W*H] Where B represents the batch size of the training model, T represents the number of consecutive input frames, C1 represents the channel dimension used in the implicit timing, and - represents the reverse order direction.

[0046] For the 3D cross-scan strategy: This approach is similar to the 2D cross-scan strategy, but from a different temporal perspective. Similarly, the Mamba model is designed for temporal scanning of sequence data. Therefore, based on our proposed spatiotemporal relationship extraction base model and drawing inspiration from the cross-scan strategy in the Vmamba model, we combine the above methods to capture the required spatiotemporal relationships from a global explicit perspective, transforming it into a 3D cross-scan strategy. This cross-scan strategy also involves four different directions, but the difference is that it scans at a 3D level, such as... Figure 5 As shown. The four directions of the specific 3D selective scanning are represented as follows: Horizontal scan = [B, C2, T*H*W] Vertical scan = [B, C2, H*T*W] Reverse transverse scan = [B, C2, -T*H*W] Reverse vertical scan = [B, C2, -T*W*H] Where B represents the batch size of the training model, T represents the number of consecutive input frames, C2 represents the channel dimension used in the global explicit temporal angle, and - represents the reverse order direction.

[0047] Furthermore, in step S7, the different angle time series fusion is used to fuse the spatiotemporal relationships of one-dimensional time series from different directions and angles processed by the spatiotemporal relationship extraction base model in step S6. The specific execution method is as follows: The one-dimensional sequences of horizontal scanning, vertical scanning, anti-horizontal scanning, and anti-vertical scanning corresponding to the first temporal angle are converted into shapes [B, T*C1, H*W], then directly added together, and the linear layer mapping in the linear gated structure of the base model is extracted through the spatiotemporal relationship as [B, T*C, H*W]; where C is the channel dimension; The one-dimensional sequence of the horizontal scan, vertical scan, anti-horizontal scan, and anti-vertical scan corresponding to the second temporal angle is converted into the shape [B, C2, T*H*W]. Then, the four directions are directly added according to the specified dimensions, and the linear layer in the linear gating structure of the base model is extracted and mapped to [B, C, T*H*W] through the spatiotemporal relationship.

[0048] Specifically, regarding the fusion of two-dimensional and three-dimensional cross-scanning strategies: we propose a spatiotemporal relationship extraction base model based on the Mamba selective state-space model and a linear gating structure. This base model is obtained by concatenating these models. Each base model extracts either implicit spatiotemporal relationships or global explicit spatiotemporal relationships between features corresponding to consecutive frames. Then, the spatiotemporal relationships obtained from these two spatiotemporal modeling methods are fused. This allows for the extraction of temporal information from different directions and from different angles for semantic space feature points corresponding to consecutive frames, further enriching the extraction of temporal information. See detailed example diagrams below. Figure 6 As shown.

[0049] Example 2 Please see Figure 7 This embodiment also provides a traffic flow prediction system based on multi-view spatiotemporal modeling, used to implement a traffic flow prediction method based on multi-view spatiotemporal modeling as described in any one of Embodiment 1, including: The data processing module is used to normalize traffic flow video data; An encoder is used to encode consecutive video frames and extract semantic features; The temporal module is used for multi-perspective spatiotemporal relationship modeling and fusion of semantic features; A decoder is used to decode temporal features into predicted video frames; The temporal module includes a channel spatial attention submodule, a dimension transformation submodule, a cross-scan submodule, a spatiotemporal relationship extraction base model submodule, and a feature fusion submodule.

[0050] In a specific test embodiment, this embodiment uses the publicly available traffic flow dataset taxibj as an example to implement the traffic flow prediction method described in this invention. This dataset contains continuous video frames of vehicle movement in an urban road network. The goal of this embodiment is to predict future continuous frame sequences given a continuous sequence of historical frames. The overall implementation process follows... Figure 1 The architecture shown is based on building and training an end-to-end prediction model that includes a spatial encoder, a temporal module, and a spatial decoder, and its effectiveness is verified through comparative experiments.

[0051] Data processing module: Selects continuous traffic monitoring video clips from the taxibj dataset. The input sequence length (historical frames) is set to T=4 frames, and the predicted sequence length (future frames) is also set to T=4 frames. The data is 2-channel data with a resolution of 32x32. Before inputting the data into the model, the pixel values ​​need to be normalized to the [0, 1] interval. Subsequently, the data is randomly divided into training, validation, and test sets according to the dataset's partitioning criteria.

[0052] The encoder employs a two-layer structure. The first sub-layer of each layer uses a 3x3 convolution with a stride of 1 for feature extraction. The channel dimension remains consistent at 48 during the encoder stage. The second sub-layer of each layer uses a 3x3 convolution with a stride of 2 for downsampling the data. The encoder ultimately outputs a feature tensor with dimensions [B, 4, 48, 16, 16] (B being the batch size), retaining the low-level semantic features from the first layer as skip-connected features.

[0053] Timing modules: stacked in 8 layers.

[0054] Channel spatial attention module: such as Figure 3 As shown at the entrance, the input features are first subjected to channel attention operation with a compression ratio of 8; then spatial attention operation is performed with a 7x7 convolution kernel.

[0055] Viewpoint splitting: The features output by the attention module are reorganized in terms of dimensions. Let C1=C2=32, then we get the first viewpoint feature F1 (dimension [B, 48*T, 16, 16]) and the second viewpoint feature F2 (dimension [B, 48,T, 16, 16]), where T represents the prediction frame length, and this system uses T=4.

[0056] Cross-scan and base model processing: Perform a two-dimensional cross scan on F1 ( Figure 4), generating a one-dimensional sequence in four directions, with dimensions [B, 48*T, 16*16].

[0057] Perform a three-dimensional cross scan on F2 ( Figure 5 This generates four one-dimensional sequences in four directions, each with dimensions [B, 48, 16*16*T]. These eight sequences are then input into a spatiotemporal relation extraction base model (structure as follows). Figure 2 (As shown). The selective state-space model (SSM) of each base model has a state dimension N of 16. The hidden layer dimension in the linear gating structure needs to be increased by a factor of 2 to the channel dimension. After the channel linear gating structure is processed, the output has the same channel dimension as the input channel.

[0058] Decoder: Employs a 2-layer structure symmetrical to the encoder. Each layer first processes features using 3x3 convolutions and the SiLU activation function, then performs a 2x upsampling using PixelShuffle. The decoder receives skip-connected features from the corresponding encoder layer and fuses them through skip connections. The final output layer uses 1x1 convolutions to map the number of channels to 1 (grayscale), outputting a sequence of predicted frames with dimensions [B, 4, 1, 32, 32].

[0059] Model training: The AdamW optimizer was used with an initial learning rate of 1e-3, decayed using cosine annealing. The batch size (B) was set to 16. The mean squared error (MSE) loss function was used. The model was trained on the training set and its performance was monitored on the validation set for a total of 200 epochs.

[0060] Performance Evaluation and Result Analysis: The trained model was evaluated on an independent test set, using mean squared error (MSE), mean absolute error (MAE), and structural similarity index (SSIM) as evaluation metrics. To verify the superiority of this invention, current mainstream spatiotemporal prediction models such as ConvLSTM, PredRNN, SimVP, TAU, SwinLSTM, and VMRNN were selected as baselines, and comparative experiments were conducted on the same taxibj dataset and under the same training-test split.

[0061] Experimental results: As shown in Table 1, the model proposed in this invention (labeled "ours") achieved the best prediction performance.

[0062] Table 1 Comparison of Model Indicators (ours represents the model of this invention, and taxibj is a public traffic flow dataset) In terms of prediction accuracy: This model achieved the lowest MSE (0.2827) and MAE (14.62), and the highest SSIM (0.9859), indicating that its predicted frames are closest to the real frames in terms of pixel accuracy and structural similarity.

[0063] Regarding model efficiency: the model has only 2.88M parameters, far lower than most baseline models (such as PredRNN++'s 38.4M and E3D-LSTM's 51.0M), and even lower than some lightweight models (such as SwinLSTM's 2.9M and VMRNN's 2.6M). This confirms that the temporal module based on the selective state-space model maintains strong modeling capabilities while possessing extremely high parameter efficiency.

[0064] Results Analysis: Experimental results show that this invention effectively integrates implicit dynamic and explicit structural information through "channel-space" dual-perspective modeling. Simultaneously, the spatiotemporal relationship extraction base model based on the selective state-space model successfully achieves efficient modeling of long sequences with low parameter cost. The introduction of the channel-space attention module further purifies the input features, contributing to the model's comprehensive advantages in both accuracy and efficiency.

[0065] In summary, this specific implementation fully demonstrates the entire process from data preparation, model building, training to evaluation. Combined with sufficient comparative experiments, it proves that the method provided by this invention can achieve more accurate and efficient predictions in traffic flow prediction tasks, and has clear practicality and significant progress.

[0066] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A traffic flow prediction method based on multi-view spatiotemporal modeling, characterized in that, Includes the following steps: Step S1: Encode the input continuous traffic video frames frame by frame using an encoder to obtain the semantic features of each frame; Step S2: Input the semantic features into the time series module to perform spatiotemporal relationship modeling; Step S3: In the temporal module, the input semantic features are filtered using the channel space attention module to filter out unimportant temporal information; Step S4: Perform dimensional transformation on the filtered semantic features. Perform dimensional transformation on the semantic features of consecutive frames to obtain the first temporal dimension shape corresponding to the first temporal angle and the second temporal dimension shape corresponding to the second temporal angle. Step S5: Perform two-dimensional cross-scanning and three-dimensional cross-scanning on the first time-series angle and the second time-series angle respectively to obtain the semantic features of the one-dimensional sequence in different directions under each time-series angle; Step S6: Input the semantic features of the one-dimensional sequence in different directions under each time series angle into the spatiotemporal relationship extraction base model; wherein, the spatiotemporal relationship extraction base model is constructed based on a selective state space module and a linear gating structure, and is used to extract time series relationships; Step S7: The semantic features of the one-dimensional sequences in different directions after being processed by the base model of spatiotemporal relation extraction under each temporal angle are fused and mapped to the shape of the image feature dimension; then, based on the first temporal angle and the second temporal angle, different angle temporal fusion is performed so that each semantic feature obtains spatiotemporal information of different directions and different temporal angle modeling methods, and thus obtains temporal semantic features; Step S8: Input the fused temporal semantic features into the next layer temporal module and repeat steps S3 to S7; wherein the number of repetitions is consistent with the number of model layers; Step S9: After the temporal processing is completed, the temporal semantic features obtained are decoded into the pixel space by the decoder, and then decoded into the predicted traffic flow video frames.

2. The traffic flow prediction method based on multi-view spatiotemporal modeling according to claim 1, characterized in that, The encoder consists of multiple identical layer structures, and each layer includes two sub-layers; wherein, The first sub-layer consists of 2D convolution, GroupNorm normalization, and Silu activation function, which are used for feature extraction through convolution operations. The second sub-layer consists of 2D convolution, GroupNorm normalization, and Silu activation function, which are used to adjust the convolution stride to achieve downsampling operation; The encoder outputs a jump connection feature to the decoder.

3. The traffic flow prediction method based on multi-view spatiotemporal modeling according to claim 2, characterized in that, The number of layers in the encoder is determined based on the input data resolution, and the features processed by the first layer in the encoder need to be returned at the end to implement the skip key structure.

4. The traffic flow prediction method based on multi-view spatiotemporal modeling according to claim 1, characterized in that, The channel spatial attention module is used to simultaneously support the modeling of implicit spatiotemporal relationships and global explicit spatiotemporal relationships, and introduces channel spatial attention to filter important spatiotemporal relationship features; wherein, the first temporal perspective extracts implicit spatiotemporal relationships from the channel level, and the second temporal perspective extracts global explicit spatiotemporal relationships from spatial location attributes.

5. The traffic flow prediction method based on multi-view spatiotemporal modeling according to claim 1, characterized in that, In step S4, the dimensional transformation is as follows: First time sequence angle = [B, T*C1, H, W] Second time series angle = [B,C2,T,H,W] Where B is the batch size, T is the number of frames, C1 and C2 are the number of channels for the two viewpoints, and H and W are the height and width of the feature map.

6. The traffic flow prediction method based on multi-view spatiotemporal modeling according to claim 1, characterized in that, In step S5, the two-dimensional cross scan and the three-dimensional cross scan each include four directions: horizontal scan, vertical scan, reverse horizontal scan, and reverse vertical scan. The specific dimensional transformations are represented as follows: Dimensional transformation from the first time-series perspective: Horizontal scan = [B, T*C1, H*W] Vertical scan = B, T*C1, W*H] Reverse transverse scan = [B, T*C1, -H*W] Reverse vertical scan = [B, T*C1, - W*H] Dimensional transformation from the second time-series perspective: Horizontal scan = [B, C2, T*H*W] Vertical scan = [B, C2, H*T*W] Reverse transverse scan = [B, C2, -T*H*W] Reverse vertical scan = [B, C2, -T*W*H] Where B represents the batch size of the training model, T represents the number of consecutive input frames, C1 represents the channel dimension used in the first time sequence angle, C2 represents the channel dimension used in the second time sequence angle, and - represents the reverse order direction.

7. The traffic flow prediction method based on multi-view spatiotemporal modeling according to claim 1, characterized in that, The structure of the spatiotemporal relation extraction basis model is as follows: z = SSM(σ(Linear(x)) s = σ(Linear(x)) o = Linear(z ⊙ s) Where z represents the latent temporal features after processing by the selective state-space model SSM, s represents the gated signal generated by the input through a linear layer and an activation function, o represents the final output feature of the spatiotemporal relation extraction base model, x represents the input feature, SSM represents the Mamba selective state-space model, Linear represents the linear layer, σ represents the activation function Silu, and ⊙ represents the Hadamard product.

8. A traffic flow prediction method based on multi-view spatiotemporal modeling according to claim 6, characterized in that, In step S7, the time series fusion from different angles is used to fuse the spatiotemporal relationships of one-dimensional time series from different directions and angles after processing by the spatiotemporal relationship extraction base model in step S6. The specific execution method is as follows: The one-dimensional sequences of horizontal scanning, vertical scanning, anti-horizontal scanning, and anti-vertical scanning corresponding to the first temporal angle are converted into shapes [B, T*C1, H*W], then directly added together, and the linear layer mapping in the linear gated structure of the base model is extracted through the spatiotemporal relationship as [B, T*C, H*W]; where C is the channel dimension; The one-dimensional sequence of the horizontal scan, vertical scan, anti-horizontal scan, and anti-vertical scan corresponding to the second temporal angle is converted into the shape [B, C2, T*H*W]. Then, the four directions are directly added according to the specified dimensions, and the linear layer in the linear gating structure of the base model is extracted and mapped to [B, C, T*H*W] through the spatiotemporal relationship.

9. A traffic flow prediction method based on multi-view spatiotemporal modeling according to claim 2, characterized in that, The decoder has a structure symmetrical to the encoder, with each layer consisting of two sub-layers: the first sub-layer consists of 2D convolution, GroupNorm normalization, and Silu activation function for upsampling; the second sub-layer consists of 2D convolution, PixelShuffle upsampling, GroupNorm normalization, and Silu activation function for feature decoding; the decoder receives skip-connected features from the encoder.

10. A traffic flow prediction system based on multi-view spatiotemporal modeling, used to implement the traffic flow prediction method based on multi-view spatiotemporal modeling as described in any one of claims 1 to 9, characterized in that, include: The data processing module is used to normalize traffic flow video data; An encoder is used to encode consecutive video frames and extract semantic features; The temporal module is used for multi-perspective spatiotemporal relationship modeling and fusion of semantic features; A decoder is used to decode temporal features into predicted video frames; The temporal module includes a channel spatial attention submodule, a dimension transformation submodule, a cross-scan submodule, a spatiotemporal relationship extraction base model submodule, and a feature fusion submodule.

Images