Multi-modal traffic flow prediction method based on multi-source data feature fusion

Abstract

The invention discloses a multi-modal traffic flow prediction method based on multi-source data feature fusion. The method comprises the following steps: acquiring traffic data; dividing a training set and a test set according to the traffic data; constructing a traffic prediction model, and training the traffic prediction model by adopting the training set; verifying the traffic prediction model by using the test set; the traffic prediction model comprises a cloud image encoder used for carrying out feature extraction on cloud image data to obtain cloud image features; the space-time encoder is used for performing feature extraction on the space-time data to obtain space-time features; the fusion module is used for fusing the cloud image features and the spatial-temporal features to obtain fusion features; and carrying out addition operation on the cloud picture features, the spatial-temporal features and the fusion features, and inputting into a spatial-temporal decoder for prediction to obtain a final prediction result. According to the method, multi-azimuth and multi-dimensional information is fused, redundant information in the spatial-temporal feature extraction process is simplified, efficient and high-precision prediction is achieved, and the method can be widely applied to the technical field of traffic flow prediction.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic flow prediction, and in particular to a multimodal traffic flow prediction method based on multi-source data feature fusion. Background Art

[0002] Traffic flow forecasting refers to the use of historical traffic data to estimate vehicle flow conditions in a certain area within a certain time period in the future. This task plays a key role in areas such as traffic planning and vehicle scheduling, and has important application significance for promoting the development of smart transportation. Traditional forecasting methods typically use statistical methods to construct prediction models to predict future traffic flows. However, traffic flows have complex characteristics such as nonlinearity and spatiotemporal correlation, and traditional methods have certain limitations when dealing with such problems. With the development of big data technology, artificial intelligence technologies applied to the field of traffic flow forecasting are gaining increasing attention. These technologies have higher accuracy and real-time performance, and have become a hot research topic.

[0003] In recent years, convolutional neural networks have become widely used due to their ability to effectively handle nonlinear data. Graph convolutional networks (GCNNs) have excelled in topological structures and have been fully exploited. GCNNs can comprehensively consider non-Euclidean distances between nodes, dynamically learning correlations between nodes in a region, and effectively extracting spatially heterogeneous features. Research on temporal correlations has gradually shifted from recurrent neural networks and their variants to temporal convolutional networks, aiming to better handle long-term temporal correlations and further improve learning efficiency through parallelization.

[0004] Within a traffic area, changes at each node are influenced by both spatial and temporal factors, resulting in significant differences at different spatial locations or at different points in time. Previous studies have often modeled spatiotemporal data independently, often treating road conditions with varying road conditions (e.g., one-way versus two-way, two-lane versus four-lane) as the same node type, or treating traffic conditions at different times of the day (e.g., morning and evening rush hour versus light traffic in the afternoon) as equivalent. This approach fails to fully account for the diversity and imbalance of space and time, which is a manifestation of spatiotemporal heterogeneity. To achieve more accurate predictions, it is imperative to account for this spatiotemporal heterogeneity to more accurately understand and extract the changing patterns of spatiotemporal features. However, highly accurate predictions often come with increased complexity. Simultaneously considering spatial and temporal factors can lead to larger, more complex, and more difficult-to-train deep learning models, increasing training time and reducing optimization efficiency. Although the spatiotemporal data used for training is derived from real-world conditions and accurately reflects weather-influenced traffic conditions, actual weather changes are only implicitly reflected in the spatiotemporal data. Therefore, even traffic conditions at the same time and location on different dates can present vastly different results due to varying weather conditions. However, changes in traffic conditions are often influenced by multiple factors, and these implicit variations can be difficult for models to fully learn. Furthermore, current traffic training data relies entirely on data acquisition devices such as sensors. Therefore, the spatiotemporal traffic flow training data obtained is prone to missing and isolated nodes, further complicating model learning. Summary of the Invention

[0005] In order to at least solve one of the technical problems existing in the prior art to a certain extent, the purpose of the present invention is to provide a multimodal traffic flow prediction method, device and medium based on multi-source data feature fusion.

[0006] The first technical solution adopted by the present invention is:

[0007] A multimodal traffic flow prediction method based on multi-source data feature fusion includes the following steps:

[0008] Acquiring traffic data, including vehicle speed information, density information, flow information, and weather information;

[0009] Divide the training set and test set according to the obtained traffic data;

[0010] Construct a traffic prediction model, use the training set to train the traffic prediction model; use the test set to verify the traffic prediction model;

[0011] The traffic prediction model includes a cloud map encoder, a spatiotemporal encoder, a fusion module, and a spatiotemporal decoder;

[0012] The cloud image encoder is used to extract features from the input cloud image data to obtain cloud image features;

[0013] The spatiotemporal encoder is used to extract features from the input spatiotemporal data to obtain spatiotemporal features;

[0014] The fusion module is used to fuse cloud image features and spatiotemporal features to obtain fusion features;

[0015] The cloud map features, spatiotemporal features and fusion features are added together and input into the spatiotemporal decoder for prediction to obtain the final prediction result.

[0016] Furthermore, the obtaining of traffic data includes:

[0017] Integrate and summarize traffic data of different time periods and different traffic areas and cloud map image data of corresponding traffic areas to obtain spatiotemporal data and cloud map data;

[0018] Among them, when collecting cloud map data, ensure that the time stamp is aligned with the time part of the spatiotemporal data to ensure that each piece of information input by the cloud map encoder and the spatiotemporal encoder corresponds to the same timestamp.

[0019] Furthermore, the cloud image encoder includes:

[0020] The image embedding layer is used to divide the cloud image into regions and obtain a multi-dimensional sequence to ensure that the input dimension is suitable for the subsequent multi-head self-attention layer and the first feedforward network layer;

[0021] Multi-head self-attention layer, which is used to enable each position in the sequence to focus on other positions in the sequence through the attention mechanism to capture dependencies within the sequence;

[0022] The first feedforward network layer is used to introduce nonlinear expressions to integrate different information and ensure information diversity.

[0023] Furthermore, the processing of the cloud image by region division includes:

[0024] Divide the cloud image into N fixed-size image blocks through embedding;

[0025] The obtained image blocks are flattened and mapped into a vector sequence of fixed length through linear transformation;

[0026] Add a time stamp before the vector sequence to indicate the time corresponding to the data;

[0027] A positional encoding is added before the time tag, ultimately forming an embedding input consisting of N+2 sequences.

[0028] Furthermore, the cloud image encoder also adopts residual connection, the output of the picture embedding layer is used as the input of the multi-head self-attention layer, the output of the picture embedding layer and the output of the multi-head self-attention layer are added as the input of the first feedforward network layer, and the output of the self-attention layer and the output of the first feedforward network layer are added as the output of the cloud image encoder.

[0029] Furthermore, the fusion module includes:

[0030] Cross-attention layer: The output of the cloud encoder and the output of the spatiotemporal encoder are both used as inputs to the cross-attention layer to process the association between multiple modal sequences through the attention mechanism;

[0031] The second feedforward network layer is used to integrate long-term dependencies and enrich feature information by introducing diverse nonlinear expressions.

[0032] Furthermore, the structure of the spatiotemporal encoder is the same as that of the spatiotemporal decoder; the input of the spatiotemporal encoder is spatiotemporal data; the input of the spatiotemporal decoder is the addition of the cloud features and spatiotemporal features after alignment and fusion features;

[0033] The spatiotemporal decoder comprises:

[0034] The graph convolutional neural network layer is used to extract spatial features from the input data and generate new features by aggregating information from adjacent nodes;

[0035] The dimension conversion layer is used to convert the generated features into dimension-matching ones to obtain one-dimensional time series to adapt to time feature extraction;

[0036] The TimesNet network layer includes a timing change module and a convolution extraction module; wherein the timing change module is used to take the period of the input one-dimensional timing features and convert them into a two-dimensional tensor to represent the two-dimensional timing changes; the convolution extraction module is used to capture the time features according to the two-dimensional time series and make predictions based on the captured time features.

[0037] Furthermore, the calculation formula of the graph convolutional neural network layer is as follows:

[0038]

[0039] Where, X S-T Represents the features of this layer, X S-T ' is the next layer feature after update, W is the weight matrix that needs to be updated, and σ() represents the nonlinear activation function. Indicates that a symmetric normalization operation is performed on the adjacency matrix A.

[0040] Furthermore, the timing change module operates as follows:

[0041] The periodicity in one-dimensional time series data is obtained by using the Fast Fourier Transform (FFT) function. The expression is as follows:

[0042] A=Avg(Amp(FFT(X Sp-Te )))

[0043] Where A represents the intensity of each frequency component in the time input data, Avg represents the average, and AMP represents the amplitude value;

[0044] The k frequencies with the largest intensity are obtained through the argTopk() function, and the k most significant period lengths are obtained based on the relationship between frequency and period. The expression is as follows:

[0045] f1,...,f k =argTopk(A)

[0046]

[0047] Where, f k represents the kth frequency, p k represents the length of the kth period, and T represents the time length;

[0048] According to the above calculation, the frequencies f1,...,f k and the corresponding period lengths p1,...,p k , the one-dimensional time series is reshaped into a two-dimensional time series by the following equation, that is, the Padding() function is used to fill zeros at the end of the sequence and the Reshape() function is used to fold based on periodicity. The expression is as follows:

[0049] X Sp-Te 2D =Reshape(Padding(X Sp-Te ))

[0050] Where, X Sp-Te 2D Represents a two-dimensional time series.

[0051] Furthermore, the convolution extraction module works as follows:

[0052] The Inception-like convolutional network used for temporal feature extraction in TimesNet is replaced with a depth-wise separable convolution. The expression is as follows:

[0053] X Sp-Te 2D '=DWC(X Sp-Te 2D )

[0054] Where, X Sp-Te2D′ Represents the captured temporal features.

[0055] The second technical solution adopted by the present invention is:

[0056] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a multimodal traffic flow prediction method based on multi-source data feature fusion as described above.

[0057] The third technical solution adopted by the present invention is:

[0058] A computer-readable storage medium stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a multimodal traffic flow prediction method based on multi-source data feature fusion as described above.

[0059] The fourth technical solution adopted by the present invention is:

[0060] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, so that the computer device performs the above method.

[0061] The beneficial effects of the present invention are as follows: the present invention designs a lightweight traffic prediction model, which integrates multi-directional and multi-dimensional information; through the method of serial extraction of spatiotemporal features, it avoids the uncertainty and instability problems that may be caused by spatiotemporal heterogeneity, and at the same time simplifies the redundant information in the spatiotemporal feature extraction process, thereby achieving efficient and high-precision prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0063] Figure 1 It is a technical roadmap for traffic flow prediction in an embodiment of the present invention;

[0064] Figure 2is a schematic diagram of a multimodal model architecture for traffic flow prediction in an embodiment of the present invention;

[0065] Figure 3 Schematic diagram of a lightweight depth-wise separable convolution in an embodiment of the present invention;

[0066] Figure 4 This is a flowchart of the steps of a multimodal traffic flow prediction method based on multi-source data feature fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided for ease of explanation only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0068] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.

[0069] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0070] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0071] To address existing technical challenges, including insufficient consideration of spatiotemporal heterogeneity, the difficulty in balancing accuracy and efficiency, and the lack of comprehensiveness of single data sources, innovative research solutions are needed. By comprehensively considering spatiotemporal correlations and integrating weather data to learn weather characteristics, this approach provides more precise model calibration, guides model learning, and reduces learning difficulty. Through multimodal alignment and fusion, more accurate traffic flow forecasts can be achieved through complementary approaches. Furthermore, to ensure optimal model efficiency and timely predictions, the model needs to be lightweight and redundant. This allows for more efficient predictions while maintaining high accuracy.

[0072] This paper proposes a multimodal architecture design for traffic flow prediction tasks, which can extract spatiotemporal features and integrate weather features. For this task, we systematically prepare data, design models, and continuously optimize them, gradually building each subtask to ultimately form a high-precision and high-efficiency prediction process, such as Figure 1 As shown in the figure, the first stage is data collection. Sensors and other devices enable real-time acquisition of required traffic data, such as vehicle speed, density, and traffic volume. Data such as vehicle speed can easily determine whether traffic conditions are smooth or congested. Data from different time periods and traffic areas, along with cloud map images of the corresponding areas, are aggregated and summarized to prepare for subsequent processing. In the second stage, because the data collected by the equipment is somewhat rough, data preprocessing is required to enable the model to more effectively analyze the data and extract features. Historical data is cleaned to remove outliers and meaningless data. Furthermore, the data is formatted for use in the traffic prediction model and split into training and test sets to further validate the model's effectiveness. In the third stage, the processed data is fed into our designed model, which is trained to automatically extract features from the data. This step involves designing a neural network-based prediction model. The specific model architecture design will be detailed later. The fourth stage involves continuous optimization and testing, adjusting model parameters based on feedback to improve prediction accuracy. After optimization, the data is applied to the test set for validation, using historical data to predict traffic conditions in a specific area over a period of time. In practice, the prediction results obtained by the present invention can be used to implement operations such as traffic vehicle scheduling and traffic condition relief, thereby promoting the further development of smart transportation.

[0073] In real life, changes in traffic conditions are often influenced by multiple factors. Compared to current prediction models that only mine single-modal data, this invention fully leverages the advantages of diverse data characteristics. First, it utilizes the Transformer class and corresponding convolutional networks to efficiently extract features from image data, spatial data, and time series data. This personalized processing method can extract and express key information in a targeted manner based on the characteristics of the data type, thereby achieving more accurate and meaningful results in model training and application.

[0074] The present invention has also made important progress in integrating multi-source information. By cleverly designing the fusion module and feature fusion mechanism, the diverse information from time series, spatial data and weather data is successfully integrated into one. This comprehensive processing method gives the model a more comprehensive information perspective, solves the learning difficulty problem of implicit expression of weather data features, and explicitly enhances the association between data. On the one hand, the model can learn the characteristics of weather data more accurately, and on the other hand, it can more easily grasp the changes in spatiotemporal data affected by weather, thereby showing higher accuracy and robustness when dealing with complex problems. This integration of comprehensive information not only enhances the expressiveness of the model, but also improves its robustness in dealing with changing situations.

[0075] Furthermore, to address issues of model size and training efficiency, this invention reduces redundant design, making the prediction model more versatile and adaptable to a wider range of application scenarios. This comprehensive consideration not only improves the model's performance but also enhances its flexibility and reliability in practical applications, making it more capable of handling diverse tasks and further enhancing the method's practical value and effectiveness.

[0076] Example 1

[0077] like Figure 4 As shown, this embodiment provides a multimodal traffic flow prediction method based on multi-source data feature fusion, including the following steps:

[0078] S1. Acquire traffic data, wherein the traffic data includes vehicle speed information, density information, flow information, and weather information;

[0079] S2. Divide the training set and test set according to the obtained traffic data;

[0080] S3. Build a traffic prediction model and train the traffic prediction model using a training set;

[0081] S4. Optimize the trained traffic prediction model and use the test set to verify the traffic prediction model.

[0082] The traffic prediction model includes a cloud map encoder, a spatiotemporal encoder, a fusion module, and a spatiotemporal decoder;

[0083] The cloud image encoder is used to extract features from the input cloud image data to obtain cloud image features;

[0084] The spatiotemporal encoder is used to extract features from the input spatiotemporal data to obtain spatiotemporal features;

[0085] The fusion module is used to fuse cloud image features and spatiotemporal features to obtain fusion features;

[0086] The cloud map features, spatiotemporal features and fusion features are added together and input into the spatiotemporal decoder for prediction to obtain the final prediction result.

[0087] In some embodiments, the cloud map data specifically includes the following information: the height, thickness, shape, etc. of the cloud layer; the spatiotemporal data specifically includes the following information: vehicle speed, traffic volume, etc. Cloud map data can be obtained with the help of historical data provided by various remote sensing related official websites, such as NASA, China Meteorological Administration and other websites. These data record the satellite's capture of cloud changes in the area, which can be used to explore weather changes. At the same time, spatiotemporal data mainly relies on various sensors installed on the road. These sensors are deployed in the road network to capture and record spatiotemporal information in the area, providing important data support for research. The traffic prediction model consists of four core modules: cloud map encoder, spatiotemporal encoder, spatiotemporal decoder and fusion module. See Figure 2 Cloud data (in pink) and spatiotemporal data (in green) are fed into the cloud encoder and spatiotemporal encoder, respectively, for feature extraction. The extracted high-level feature outputs are then compared to obtain an alignment loss. This loss aims to align the feature spaces of the cloud encoder and spatiotemporal encoder by promoting similar representations of weather and traffic information for the same timestamp and region. This multimodal contrastive loss has been proven to be an effective optimization technique in image-text multimodal tasks. Furthermore, to better integrate weather features with spatiotemporal features, these two features are simultaneously fed into the fusion module to obtain fused features and a matching loss. Compared to the alignment loss, the matching loss focuses on learning a multimodal representation of the fusion of the two feature information to capture the correlation between them. These features, rich in cloud information, spatiotemporal information, and fused information, are then simply but effectively added together to form the input to the spatiotemporal decoder, which generates the final prediction results. This design aims to fully leverage information from different data sources to improve the overall performance and effectiveness of the prediction model.

[0088] The following is a detailed explanation of the four core models of the traffic prediction model with reference to the accompanying drawings.

[0089] (1) Cloud image encoder

[0090] For the cloud image encoder, in the data preprocessing stage, the cloud image encoder processes the cloud image by region division to ensure that it corresponds to the spatial region input by the spatiotemporal encoder; at the same time, it ensures that the time stamp is aligned with the time part of the spatiotemporal data during collection to ensure that each piece of information input by the spatiotemporal encoder corresponds to the data with the same timestamp. This module needs to process the image, so the ViT (Vision Transformer) class architecture widely used in the field of computer vision is adopted. The ViT structure only contains an encoder, which is mainly used to extract image features and build a simple and effective model. The cloud image encoder mainly consists of three parts: an image embedding layer, a multi-head self-attention layer, and a feedforward network layer. When processing images, the Transformer class architecture needs to be embedded to ensure that the input dimension is suitable for the processing of the temporal architecture. We first divide the input image into fixed-size image blocks, which are further flattened and mapped to fixed-length vectors through linear transformation. Assume that the dimension of the input image is As for the T dimension, as mentioned above, during the data preprocessing phase, we have already mapped the cloud data to the areas covered by the spatiotemporal data, and we will also ensure temporal alignment based on the input timestamps. Therefore, we will then process the image at each timestamp specifically to adapt it to the cloud encoder, focusing on the H×W×C dimensions. By embedding, it is divided into N image blocks, each of size P×P, and then flattened to map the image block dimensions to the specified embedding dimensions. See Figure 2 , in order to align with the spatiotemporal features, unlike the original ViT architecture, we add a time marker (represented by a light gray block) in front of the vector sequence to indicate the time corresponding to the data. Subsequently, a learnable position code (represented by a dark gray block) is added and input into the encoder to finally form an embedding input consisting of N+2 sequences. After dealing with the image embedding dimension problem, it will be processed by the multi-head self-attention layer. The attention mechanism enables each position in the sequence to focus on other positions in the sequence to capture dependencies within the sequence, especially long-term dependencies. This module will generate the query matrix Q, key matrix K, and value matrix V by multiplying the input matrix with the corresponding parameter matrix. In the attention mechanism, the query operation is performed first, that is, the query matrix Q is multiplied by the key matrix K to obtain the attention score, which is then normalized by the Softmax function. In order to avoid the gradient being close to zero due to the large input value of the Softmax function, we divide the multiplication matrix by Then, the normalized attention score is multiplied by the value matrix V to obtain a value matrix V with different weights, which helps to distinguish the key features worthy of attention. The specific expression of the entire attention mechanism is as follows:

[0091]

[0092] The multi-head attention mechanism is based on the self-attention mechanism, which divides each query matrix Q, key matrix K, and value matrix V into multiple branches to focus on different aspects, and finally splices these multiple branches to restore the dimensions before splitting. This method enables the model to capture various types of key features in the input information from multiple angles and dimensions, significantly enhancing the model's expressive power. Subsequently, a nonlinear expression is introduced through a fully connected layer with an activation function, that is, a feedforward network layer, to integrate different information and ensure information diversity. In addition, the encoder also uses a residual connection, allowing the input to be directly added to the output of these layers. This design allows the module network to learn a residual function instead of a complete function that maps the input to the output. For example, the residual connection operation before and after the multi-head self-attention layer can be expressed by formula (2), where the input image X Cloud , after the operation of the multi-head self-attention layer (referred to as the F() function), the temporary output F(X Cloud The final result is based on this, and the original input X is added through the residual connection Cloud Therefore, what the multi-head self-attention layer learns is actually the residual information between the input and the expected output, which promotes the direct transmission of information, calibrates the optimization direction, avoids the gradient vanishing problem, and further improves learning efficiency.

[0093] Output Cloud =F(X Cloud )+X Cloud (2)

[0094] (2) Fusion module

[0095] See also Figure 2The fusion module includes a cross-attention layer and a feedforward network layer. These two components play a key role in processing the fused feature information extracted by the cloud encoder and the spatiotemporal encoder. The cross-attention layer is designed to handle the correlation between sequences of different modalities, which is particularly common in text-image multimodal scenarios. In this layer, we use spatiotemporal sequence data as the basic condition to control the generation of new features. The attention mechanism calculates the correlation between each sequence and different regions of the image, and obtains a normalized similarity score based on this. Finally, a weighted summation is performed to obtain a new conditioned representation. This process promotes the interaction and fusion of the two types of feature information. When applying the attention mechanism, the formulation used is the same as the self-attention mechanism mentioned above, involving the query matrix Q, the key matrix K, and the value matrix V. The difference is that the attention mechanism here uses two different matrix inputs. The query matrix Q is generated from image feature information, while the key matrix K and the value matrix V are derived from the spatiotemporal sequence features. The feedforward network layer, on the other hand, functions similarly to the previous one, with its main purpose being to introduce diverse nonlinear representations to incorporate long-term dependencies and enrich feature information. In this way, we can further improve the model's ability to understand and express complex data relationships, thereby enhancing overall predictive performance.

[0096] (3) Space-time decoder

[0097] See also Figure 2 , the spatiotemporal encoder first uses the graph convolutional neural network to train the input data X S-T Extract spatial features. There are significant spatial dependencies between adjacent or close nodes in graph data. For example, when a node encounters traffic congestion, the traffic conditions of its adjacent nodes are usually affected, showing similar congestion conditions. Traditional convolutional neural networks or recurrent networks are difficult to capture this specific spatial dependency. Therefore, when processing similar graph data, graph neural networks are usually chosen. Before entering the graph convolutional neural network, the input dimension is first converted from [10,15] to

[15] through a simple embedding layer. Convert to To match the input of the graph convolutional neural network. The core task of the graph convolutional network is to extract the spatial features of the graph structure and generate new features for subsequent processing by aggregating the information of adjacent nodes. The specific convolution operation method can be shown by formula (3). S-T Represents the features of this layer, X S-T ' is the updated next layer feature, W is the weight matrix to be updated, and σ() represents the nonlinear activation function. Finally, after graph convolution, we will obtain a result rich in spatial information. in, It can be regarded as a whole. This step only performs a symmetrical normalization operation on the adjacency matrix A to prevent the scale of the eigenvector from being too large when accumulated layer by layer. The combination of can be understood as integrating the information of adjacent nodes. This makes graph convolutional networks different from traditional convolutional networks in handling spatial dependencies. This processing method is of great significance in capturing complex spatial relationships and improving model performance.

[0098]

[0099] After obtaining the spatial features, we perform dimension matching transformation through a simple linear projection layer, which is composed of Convert to A module adapted for temporal feature extraction. After dimension matching, the data is serially fed into the temporal feature extraction layer, resulting in the lightweight TimesNet network. This innovative work in the field of time series forecasting has demonstrated remarkable results. This network utilizes a novel multi-period processing approach to analyze temporal changes, achieving remarkable results in both long-term and short-term forecasting. The TimesNet network consists of two main components: a temporal change module and a convolutional extraction module.

[0100] Among them, the time series change module takes the period of the input one-dimensional time series features and converts them into a two-dimensional tensor to represent the two-dimensional time series changes. First, the periodicity in the time series data is obtained through the fast Fourier transform FFT() function, as shown in formula (4), where A represents the intensity of each frequency component in the time input data. Through the argTopk() function, we obtain the k frequencies with the largest intensity, and according to the relationship between frequency and period, we obtain the most significant k period lengths, corresponding to formula (5) and formula (6) respectively. Based on this, we can perform a conversion operation to fold the original one-dimensional time series into a two-dimensional time series, that is, through Padding() to fill zeros at the end of the sequence and Reshape() function, folding based on periodicity, as shown in formula (7).

[0101] A=Avg(Amp(FFT(X Sp-Te ))) (4)

[0102] f1,...,f k =argTopk(A) (5)

[0103]

[0104] X Sp-Te 2D =Reshape(Padding(X Sp-Te )) (7)

[0105] After the temporal changes, the TimesNet network uses the classic Inception-like convolutional network to capture temporal features. However, this convolutional network uses convolution kernels of various scales, with the largest scale reaching 11×11, which brings certain difficulties to the training process. To address the challenges of excessive parameter count and long training time, this embodiment replaces the Inception-like convolutional network originally used for temporal feature extraction in TimesNet with a depthwise separable convolution, as shown in formula (8).

[0106] X Sp-Te 2D '=DWC(X Sp-Te 2D ) (8)

[0107] Figure 3 The convolution kernels and detailed convolution processes of Inception-type convolution and depthwise separable convolution are demonstrated. Inception-type networks use concatenation to combine convolution results of different scales, from 1×1 to 11×11, to extract and fuse multi-scale features. Large convolution kernels increase the receptive field of view to a certain extent, but for temporal information processing, only intra-cycle and inter-cycle information is typically considered. Excessively large convolution kernels not only fail to improve accuracy but also lead to significant parameter redundancy. Depthwise separable convolution extracts spatial information and channel features by separating depthwise convolution from pointwise convolution. In depthwise convolution, a single convolution kernel is applied to each input channel, generating a feature map with the same number of channels. However, depthwise convolution alone cannot effectively utilize information from different channels at the same spatial location. To address this, pointwise convolution is required to integrate this channel information and generate a new feature map by applying a 1×1 convolution. The input and output dimensions of depthwise separable convolutions are the same as those of standard convolutions, but this decomposition significantly reduces computational cost and model size, reducing the number of parameters by 8-9 times compared to standard 3×3 convolutions. However, due to the redundancy inherent in information across different periods, this convolution separation process has minimal impact on accuracy. Furthermore, to enhance expressive richness, we introduce nonlinear activations, using ReLU activation layers after depthwise and pointwise convolutions.

[0108] As an optional implementation, see Figure 2The spatiotemporal decoder and spatiotemporal encoder are similar in basic structure, but their input methods differ slightly. The spatiotemporal encoder only receives spatiotemporal information data as input, while the spatiotemporal decoder's input includes extracted cloud image information, extracted spatiotemporal information, and fused cloud image-spatiotemporal information, forming a comprehensive embedded input. The dimensional alignment of the three types of information is achieved through a simple and effective mapping function. The combination of spatiotemporal information and weather information can provide a more comprehensive and multidimensional input for the model, helping to improve the model's understanding of environmental changes and impacts. Therefore, the final output result mainly depends on the spatiotemporal decoder, resulting in decoding loss as one of the goals of optimizing network parameters.

[0109] In summary, existing methods primarily rely on extracting a single type of temporal or spatial feature for prediction. However, spatial nodes are time-varying, meaning that changes in traffic conditions are not only influenced by complex spatial regions but also significantly influenced by temporal factors. Considering only temporal or spatial factors alone clearly cannot meet the demand for more accurate predictions. Although research has recognized the importance of extracting both temporal and spatial features, relevant research remains relatively limited. Furthermore, networks capable of extracting both temporal and spatial features often face challenges such as large parameter counts and lengthy training times, which hinder model efficiency. This performance is insufficient to meet the demands for efficient processing and rapid delivery, limiting its applicability to a wider range of application scenarios. Furthermore, changes in traffic conditions are often influenced by multiple factors. For example, under the same conditions, severe weather is often more likely to cause traffic congestion, yet the impact of weather factors is often only implicitly reflected in spatiotemporal data. Complex traffic conditions can easily make models less sensitive to data changes, making it difficult to accurately interpret the weather characteristics embedded in spatiotemporal data, increasing system complexity. The present method employs a lightweight traffic prediction model that incorporates multifaceted and multidimensional information. By extracting spatiotemporal features serially, we avoid the uncertainty and instability that can arise from spatiotemporal heterogeneity, while streamlining redundant information during the spatiotemporal feature extraction process, achieving efficient and highly accurate forecasting. Furthermore, by incorporating weather information as auxiliary guidance, weather features can be explicitly learned, reducing the difficulty of model learning and improving model performance and training efficiency, thereby enhancing system stability and reliability.

[0110] Example 2

[0111] An embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the following Figure 1 and / or Figure 4A multimodal traffic flow prediction method based on multi-source data feature fusion is shown.

[0112] It is understood that the memory may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the data storage area may store data created based on the use of the server, etc.

[0113] The processor may include one or more processing cores. The processor utilizes various interfaces and circuits to connect various components within the server. It executes various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory, as well as accessing data stored in memory. Optionally, the processor may be implemented using at least one of the following hardware forms: digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor may integrate one or a combination of a central processing unit (CPU) and a modem. The CPU primarily processes the operating system and application programs, while the modem handles wireless communications. It is understood that the modem may not be integrated into the processor and may be implemented separately via a single chip.

[0114] Since the electronic device is an electronic device corresponding to a multimodal traffic flow prediction method based on multi-source data feature fusion in an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0115] Example 4

[0116] An embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 1 and / or Figure 4 A multimodal traffic flow prediction method based on multi-source data feature fusion is shown.

[0117] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0118] Since the storage medium is a storage medium corresponding to a multimodal traffic flow prediction method based on multi-source data feature fusion in an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0119] Example 5

[0120] In some possible implementations, various aspects of the methods of the embodiments of the present invention may also be implemented in the form of a program product, which includes program code. When the program product is executed on a computer device, the program code is used to cause the computer device to execute the steps of a multimodal traffic flow prediction method based on multi-source data feature fusion according to various exemplary embodiments of the present application described above in this specification. The executable computer program code or "code" used to execute each embodiment may be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0121] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0122] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0123] The above embodiments are intended only to illustrate the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent changes or modifications made based on the essence of the present invention are intended to be covered by the scope of protection of the present invention.

Claims

1. A multimodal traffic flow prediction method based on multi-source data feature fusion, characterized in that: The following steps are involved: Get traffic data; Divide the training set and test set according to the obtained traffic data; Construct a traffic prediction model, use the training set to train the traffic prediction model, and use the test set to verify the traffic prediction model; The traffic prediction model includes a cloud map encoder, a spatiotemporal encoder, a fusion module and a spatiotemporal decoder; The cloud map encoder is used to extract features from the input cloud map data to obtain cloud map features; The spatiotemporal encoder is used to extract features from the input spatiotemporal data to obtain spatiotemporal features; The fusion module is used to fuse the cloud image features and spatiotemporal features to obtain fusion features; The cloud map features, spatiotemporal features and fusion features are added together and input into the spatiotemporal decoder for prediction to obtain the final prediction result.

2. According to claim 1, a multi-modal traffic flow prediction method based on multi-source data feature fusion is characterized in that: The obtaining of traffic data includes: Integrate and summarize the traffic data of different time periods and different traffic areas and the cloud map image data of the corresponding traffic areas to obtain spatiotemporal data and cloud map data; Among them, when collecting cloud map data, ensure that the time stamp is aligned with the time part of the space-time data to ensure that each piece of information input by the cloud map encoder and the space-time encoder corresponds to the same timestamp.

3. The multimodal traffic flow prediction method based on multi-source data feature fusion according to claim 1 is characterized in that: The cloud image encoder comprises: The image embedding layer is used to divide the cloud image into regions and obtain a multi-dimensional sequence to ensure that the input dimension is suitable for the subsequent multi-head self-attention layer and the first feed-forward network layer. Multi-head self-attention layer, which is used to enable each position in the sequence to focus on other positions in the sequence through the attention mechanism to capture dependencies within the sequence; The first feedforward network layer is used to introduce nonlinear expressions to integrate different information and ensure the diversity of information.

4. The multimodal traffic flow prediction method based on multi-source data feature fusion according to claim 3 is characterized in that: The processing of the cloud image by region division includes: Divide the cloud image into N fixed-size image blocks by embedding; The obtained image blocks are flattened and mapped into a vector sequence of fixed length through linear transformation; Add a time stamp before the vector sequence to indicate the time corresponding to the data; Add a positional encoding before the time tag, and finally form an embedding input consisting of N+2 sequences.

5. The multimodal traffic flow prediction method based on multi-source data feature fusion according to claim 3 is characterized in that: The cloud map encoder also adopts residual connection. The output of the picture embedding layer is used as the input of the multi-head self-attention layer. The output of the picture embedding layer and the output of the multi-head self-attention layer are added as the input of the first feedforward network layer. The output of the self-attention layer and the output of the first feedforward network layer are added as the output of the cloud map encoder.

6. The multimodal traffic flow prediction method based on multi-source data feature fusion according to claim 1 is characterized in that: The fusion module includes: Cross-attention layer: The output of the cloud encoder and the output of the spatiotemporal encoder are both used as the input of the cross-attention layer to process the association between multiple modal sequences through the attention mechanism; The second feed-forward network layer is used to integrate long-term dependencies and enrich feature information by introducing diverse nonlinear expressions.

7. The multimodal traffic flow prediction method based on multi-source data feature fusion according to claim 1 is characterized in that: The structure of the spatiotemporal encoder is the same as that of the spatiotemporal decoder; the input of the spatiotemporal encoder is spatiotemporal data; the input of the spatiotemporal decoder is the addition of the cloud map features and the spatiotemporal features after alignment with the fusion features; The spatiotemporal decoder comprises: The graph convolutional neural network layer is used to extract spatial features from the input data and generate new features by aggregating information from adjacent nodes; The dimension conversion layer is used to convert the generated features into dimension matching and obtain a one-dimensional time series to adapt to time feature extraction; The TimesNet network layer includes a timing change module and a convolution extraction module; wherein the timing change module is used to take the period of the input one-dimensional timing features and convert it into a two-dimensional tensor to represent the two-dimensional timing changes; the convolution extraction module is used to capture the time features according to the two-dimensional time series and make predictions based on the captured time features.

8. The multimodal traffic flow prediction method based on multi-source data feature fusion according to claim 7 is characterized in that: The calculation formula of the graph convolutional neural network layer is as follows: Where, X S-T represents the layer feature, X S-T ' is the updated next layer feature, W is the weight matrix to be updated, and σ() represents the nonlinear activation function. It means that a symmetric normalization operation is performed on the adjacency matrix A.

9. The multimodal traffic flow prediction method based on multi-source data feature fusion according to claim 7 is characterized in that: The working mode of the timing change module is specifically as follows: The periodicity in one-dimensional time series data is obtained by the Fast Fourier Transform FFT() function. The expression is as follows: A=Avg(Amp(FFT(X Sp-Te ))) In the formula, A represents the intensity of each frequency component in the time input data, Avg represents the average, and AMP represents the amplitude value; The k frequencies with the largest intensity are obtained through the argTopk() function, and the k most significant period lengths are obtained based on the relationship between frequency and period. The expression is as follows: f1,...,f k =argTopk(A) In the formula, f k represents the kth frequency, p k represents the length of the kth cycle, and T represents the time length; According to the calculated frequencies f1,...,f k and the corresponding period lengths p1,...,p k , reshape the one-dimensional time series into a two-dimensional time series through the following equation, that is, padding the end of the sequence with zeros and Reshape() function, folding based on periodicity, the expression is as follows: X Sp-Te 2D =Reshape(Padding(X Sp-Te )) Where, X Sp-Te 2D Represents a two-dimensional time series.

10. The multimodal traffic flow prediction method based on multi-source data feature fusion according to claim 7 is characterized in that: The convolution extraction module works as follows: Replace the Inception-like convolutional network used for temporal feature extraction in TimesNet with a depth-separable convolution. The expression is as follows: X Sp-Te 2D '=DWC(X Sp-Te 2D ) Where, X Sp-Te 2D′ Represents the temporal features of the capture.

Images