A multi-modal traffic flow prediction method based on multi-source data feature fusion

Abstract

This invention discloses a multimodal traffic flow prediction method based on multi-source data feature fusion, comprising: acquiring traffic data; dividing the traffic data into training and testing sets; constructing a traffic prediction model and training the traffic prediction model using the training set; and validating the traffic prediction model using the testing set. The traffic prediction model includes: a cloud map encoder for extracting features from cloud map data to obtain cloud map features; a spatiotemporal encoder for extracting features from spatiotemporal data to obtain spatiotemporal features; a fusion module for fusing cloud map features and spatiotemporal features to obtain fused features; and adding the cloud map features, spatiotemporal features, and fused features, inputting the sum to the spatiotemporal decoder for prediction to obtain the final prediction result. This invention integrates multi-dimensional and multi-faceted information, simplifies redundant information in the spatiotemporal feature extraction process, and achieves efficient and high-precision prediction, which can be widely applied in the field of traffic flow prediction technology.

Description

Technical Field

[0001] This invention relates to the field of traffic flow prediction technology, and in particular to a multimodal traffic flow prediction method based on multi-source data feature fusion. Background Technology

[0002] Traffic flow prediction refers to estimating vehicle flow in a specific area over a future time period using historical traffic data. This task plays a crucial role in traffic planning and vehicle scheduling, and has significant application implications for promoting the development of intelligent transportation. Traditional prediction methods typically utilize statistical methods to build predictive models to forecast future traffic flow. However, traffic flow exhibits complex characteristics such as nonlinearity and spatiotemporal correlation, limiting the effectiveness of traditional methods in handling such problems. With the development of big data technology, artificial intelligence technologies applied to traffic flow prediction have gained increasing attention. These technologies offer higher accuracy and real-time performance, and have become a popular research direction.

[0003] In recent years, convolutional neural networks (CNNs) have been widely used due to their ability to effectively handle nonlinear data. Among them, graph convolutional networks (GCNNs) have demonstrated outstanding performance and have been extensively explored in topological structures. GCNNs can comprehensively consider the non-Euclidean distances between nodes and dynamically learn the correlations between different nodes in a region, effectively extracting features of spatial heterogeneity. Research on temporal correlations has gradually shifted from initial studies of recurrent neural networks and their variants to temporal convolutional networks, aiming to better handle long-term temporal correlations and further improve learning efficiency through parallelization.

[0004] Within a traffic area, changes at each node are influenced by both spatial and temporal factors, exhibiting significant differences across different spatial locations and time points. Previous studies often modeled spatiotemporal data independently, treating road conditions with different characteristics (e.g., one-way vs. two-way, two-lane vs. four-lane) as the same type of node, or treating traffic conditions at different times of day (e.g., morning / evening rush hours vs. the quieter afternoon hours) in the same way. This approach fails to adequately consider the diversity and unevenness of space and time, which is precisely the manifestation of spatiotemporal heterogeneity. To achieve more accurate predictions, it is imperative to incorporate this spatiotemporal heterogeneity into the calculations to more accurately understand and extract the changing patterns of spatiotemporal features. However, highly accurate predictions often come with increased complexity. Simultaneously considering spatial and temporal factors can lead to larger, more complex, and more difficult-to-train deep learning models, increasing training time costs and reducing optimization efficiency. Although the spatiotemporal data used for training originates from real-world conditions and can realistically reflect weather-affected traffic conditions, actual weather changes are only implicitly reflected in the spatiotemporal data. Therefore, even traffic conditions at the same time and place on different dates can vary drastically due to differences in weather conditions. However, changes in traffic conditions are often influenced by multiple factors, and these implicit changes may be difficult for models to fully learn. Furthermore, current traffic training data relies entirely on data acquisition devices such as sensors, which can easily lead to missing data and isolated nodes in the acquired spatiotemporal traffic flow training data, further increasing the complexity of model learning. Summary of the Invention

[0005] In order to at least partially solve one of the technical problems existing in the prior art, the purpose of this invention is to provide a multimodal traffic flow prediction method, device and medium based on multi-source data feature fusion.

[0006] The first technical solution adopted in this invention is:

[0007] A multimodal traffic flow prediction method based on multi-source data feature fusion includes the following steps:

[0008] Acquire traffic data, including vehicle speed information, density information, traffic flow information, and weather information;

[0009] The training and test sets are divided based on the obtained traffic data;

[0010] A traffic prediction model is constructed, trained using a training set, and validated using a test set.

[0011] The traffic prediction model includes a cloud map encoder, a spatiotemporal encoder, a fusion module, and a spatiotemporal decoder.

[0012] The cloud map encoder is used to extract features from the input cloud map data to obtain cloud map features;

[0013] The spatiotemporal encoder is used to extract features from the input spatiotemporal data to obtain spatiotemporal features;

[0014] The fusion module is used to fuse cloud map features and spatiotemporal features to obtain fused features;

[0015] The cloud map features, spatiotemporal features, and fused features are added together and then input into the spatiotemporal decoder for prediction to obtain the final prediction result.

[0016] Furthermore, the acquisition of traffic data includes:

[0017] Traffic data from different time periods and traffic areas, along with cloud image data of the corresponding traffic areas, are integrated and summarized to obtain spatiotemporal data and cloud image data.

[0018] During the acquisition of cloud map data, the time portion of the spatiotemporal data is ensured to be timestamped to guarantee that each piece of information input by the cloud map encoder and the spatiotemporal encoder corresponds to the same timestamp.

[0019] Furthermore, the cloud map encoder includes:

[0020] The image embedding layer is used to perform region segmentation on the cloud image to obtain a multi-dimensional sequence, ensuring that the input dimension is suitable for the subsequent multi-head self-attention layer and the first feedforward network layer.

[0021] Multi-head self-attention layer is used to enable each position in a sequence to focus on other positions in the sequence through an attention mechanism, in order to capture the dependencies within the sequence;

[0022] The first feedforward network layer is used to introduce non-linear representations to integrate different information and ensure information diversity.

[0023] Furthermore, the processing of the cloud map through region division includes:

[0024] The cloud map is divided into N fixed-size image blocks by embedding;

[0025] The obtained image patches are flattened and mapped to a fixed-length vector sequence through a linear transformation;

[0026] Add a time stamp before the vector sequence to indicate the time corresponding to the data;

[0027] Position codes are added before the time stamps, resulting in an embedded input consisting of N+2 sequences.

[0028] Furthermore, the cloud map encoder also employs residual connections, where the output of the image embedding layer is used as the input of the multi-head self-attention layer, the output of the image embedding layer and the output of the multi-head self-attention layer are added together as the input of the first feedforward network layer, and the output of the self-attention layer and the output of the first feedforward network layer are added together as the output of the cloud map encoder.

[0029] Furthermore, the fusion module includes:

[0030] The cross-attention layer uses the outputs of both the cloud map encoder and the spatiotemporal encoder as inputs to process the correlation between multiple modal sequences through an attention mechanism.

[0031] The second feedforward network layer is used to integrate long-term dependencies and enrich feature information by introducing diverse non-linear representations.

[0032] Furthermore, the spatiotemporal encoder and the spatiotemporal decoder have the same structure; the input of the spatiotemporal encoder is spatiotemporal data; the input of the spatiotemporal decoder is the sum of the aligned cloud map features and spatiotemporal features and the fused features.

[0033] The spatiotemporal decoder includes:

[0034] Graph convolutional neural network layers are used to extract spatial features from input data and generate new features by aggregating information from neighboring nodes;

[0035] The dimension transformation layer is used to transform the generated features by dimension matching to obtain a one-dimensional time series, so as to adapt to the extraction of time features;

[0036] The TimesNet network layer includes a temporal variation module and a convolution extraction module. The temporal variation module is used to extract the period of the input one-dimensional temporal features and convert them into a two-dimensional tensor to represent the two-dimensional temporal variation. The convolution extraction module is used to capture the temporal features based on the two-dimensional temporal features and make predictions based on the captured temporal features.

[0037] Furthermore, the calculation formula for the graph convolutional neural network layer is as follows:

[0038]

[0039] In the formula, X S-T X represents the feature of this layer. S-T ' represents the updated features of the next layer, W is the weight matrix to be updated, and σ() represents the non-linear activation function. This indicates that a symmetric normalization operation has been performed on the adjacency matrix A.

[0040] Furthermore, the specific operation mode of the timing change module is as follows:

[0041] The periodicity in one-dimensional time series data can be obtained using the Fast Fourier Transform (FFT) function, as shown in the following expression:

[0042] A = Avg(Amp(FFT(X)) Sp-Te )))

[0043] In the formula, A represents the intensity of each frequency component in the time input data, Avg means to calculate the average, and AMP means to calculate the amplitude value;

[0044] The k frequencies with the highest intensity are obtained using the argTopk() function, and the lengths of the k most significant periods are obtained based on the relationship between frequency and period, as shown in the following expression:

[0045] f1,...,f k =argTopk(A)

[0046]

[0047] In the formula, f k p represents the k-th frequency. k T represents the length of the k-th cycle and T represents the time length.

[0048] The frequencies f1,...,f obtained from the above calculations are... k and the corresponding period lengths p1,...,p k The one-dimensional time series can be reshaped into a two-dimensional time series using the following equation: padding the end of the sequence with zeros using the Padding() function and folding it based on periodicity using the Reshape() function. The expression is as follows:

[0049] X Sp-Te 2D =Reshape(Padding(X) Sp-Te ))

[0050] In the formula, X Sp-Te 2D It represents a two-dimensional timing sequence.

[0051] Furthermore, the convolution extraction module operates as follows:

[0052] Replace the Inception-type convolutional network used for temporal feature extraction in TimesNet with a depthwise separable convolution, as shown in the following expression:

[0053] X Sp-Te 2D =DWC(X) Sp-Te 2D )

[0054] In the formula, X Sp-Te2D′ This indicates the time characteristics of the capture.

[0055] The second technical solution adopted in this invention is:

[0056] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a multimodal traffic flow prediction method based on multi-source data feature fusion as described above.

[0057] The third technical solution adopted in this invention is:

[0058] A computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a multimodal traffic flow prediction method based on multi-source data feature fusion as described above.

[0059] The fourth technical solution adopted in this invention is:

[0060] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions to cause the computer device to perform the method described above.

[0061] The beneficial effects of this invention are as follows: This invention designs a lightweight traffic prediction model that integrates multi-directional and multi-dimensional information; by using a serial extraction method of spatiotemporal features, it avoids the uncertainty and instability problems that may be caused by spatiotemporal heterogeneity, while simplifying redundant information in the spatiotemporal feature extraction process, thus achieving efficient and high-precision prediction. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a roadmap of traffic flow prediction technology in an embodiment of the present invention;

[0064] Figure 2This is a schematic diagram of a multimodal model architecture for traffic flow prediction in an embodiment of the present invention;

[0065] Figure 3 This is a schematic diagram of a lightweight design for depth-separable convolution in an embodiment of the present invention;

[0066] Figure 4 This is a flowchart illustrating the steps of a multimodal traffic flow prediction method based on multi-source data feature fusion in an embodiment of the present invention. Detailed Implementation

[0067] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0068] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0069] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0070] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0071] To address existing technical challenges, including insufficient consideration of spatiotemporal heterogeneity, difficulty in balancing accuracy and efficiency, and a lack of comprehensive and singular data sources, innovative research solutions are needed. This approach combines spatiotemporal correlation with weather data for weather feature learning, providing explicit calibration to the model, guiding its learning direction, and reducing learning complexity. Multimodal alignment and fusion further enhance traffic flow prediction accuracy. Simultaneously, to ensure model optimization efficiency and timely prediction, lightweight model design is necessary, eliminating redundant components. This achieves more efficient prediction while maintaining high accuracy.

[0072] This invention proposes a multimodal architecture design for traffic flow prediction, capable of extracting spatiotemporal features and fusing weather features. For this task, we systematically prepare data, design models, and continuously optimize them, progressively building each sub-task to ultimately form a high-precision, high-efficiency prediction process, such as... Figure 1 As shown. The first stage is data acquisition. Sensors and other devices acquire real-time traffic data, such as vehicle speed, density, and traffic flow. Data like vehicle speed easily indicates whether traffic is smooth or congested. Data from different time periods and traffic areas, along with corresponding cloud image data, are integrated and summarized to prepare for subsequent processing. The second stage addresses the inherent coarseness of the acquired data. To enable the model to analyze the data and extract features more effectively, preprocessing is necessary. Historical data is cleaned, removing outliers and meaningless data. Simultaneously, the data needs to be formatted for traffic prediction models and divided into training and test sets to further validate the model's effectiveness. The third stage inputs the processed data into the designed model, training it to automatically extract features. This step involves designing a neural network-based prediction model; the specific model architecture will be detailed later. The fourth stage includes continuous optimization and testing, adjusting model parameters based on feedback to improve prediction accuracy. After optimization, the data is applied to the test set for validation, using historical data to predict traffic conditions in a specific area over a future period. In practice, the prediction results obtained by this invention can be used to perform operations such as traffic vehicle scheduling and traffic congestion relief, thereby promoting the further development of intelligent transportation.

[0073] In real life, traffic conditions are often influenced by multiple factors. Compared with current prediction models that only target single-modal data mining, this invention fully leverages the advantages of different data characteristics. First, it utilizes the Transformer class and corresponding convolutional networks to efficiently extract features from image data, spatial data, and time-series data, adapting the approach to local conditions. This personalized processing method can selectively extract and express key information based on the characteristics of the data type, thereby obtaining more accurate and meaningful results in model training and application.

[0074] This invention also achieves significant progress in integrating multi-source information. Through ingenious design of the fusion module and feature fusion mechanism, it successfully integrates diverse information from time series, spatial data, and weather data. This comprehensive processing approach provides the model with a more comprehensive information perspective, solving the learning difficulty of implicit representations of weather data features and explicitly enhancing the correlation between data. This enables the model to learn the features of weather data more accurately and to more easily grasp the changes in spatiotemporal data affected by weather, thus exhibiting higher accuracy and robustness when dealing with complex problems. This integration of comprehensive information not only enhances the model's expressiveness but also improves its robustness in dealing with changing conditions.

[0075] Furthermore, addressing the issues of model size and training efficiency, this invention reduces redundant design, making the prediction model more versatile and adaptable to a wider range of application scenarios. This comprehensive approach not only improves model performance but also enhances its flexibility and reliability in practical applications, making it more capable of handling diverse task requirements, thereby further improving the practical value and effectiveness of the method.

[0076] Example 1

[0077] like Figure 4 As shown in the figure, this embodiment provides a multimodal traffic flow prediction method based on multi-source data feature fusion, including the following steps:

[0078] S1. Acquire traffic data, including vehicle speed information, density information, traffic flow information, and weather information;

[0079] S2. Divide the obtained traffic data into training and test sets;

[0080] S3. Construct a traffic prediction model and train the traffic prediction model using a training set;

[0081] S4. Optimize the trained traffic prediction model and validate it using a test set.

[0082] The traffic prediction model includes a cloud map encoder, a spatiotemporal encoder, a fusion module, and a spatiotemporal decoder.

[0083] The cloud map encoder is used to extract features from the input cloud map data to obtain cloud map features;

[0084] The spatiotemporal encoder is used to extract features from the input spatiotemporal data to obtain spatiotemporal features;

[0085] The fusion module is used to fuse cloud map features and spatiotemporal features to obtain fused features;

[0086] The cloud map features, spatiotemporal features, and fused features are added together and then input into the spatiotemporal decoder for prediction to obtain the final prediction result.

[0087] In some embodiments, the cloud image data specifically includes the following information: cloud height, thickness, shape, etc.; the spatiotemporal data specifically includes the following information: vehicle speed, traffic flow, etc. Cloud image data can be obtained using historical data provided by various remote sensing-related official websites, such as NASA and the China Meteorological Administration. This data records satellite capture of cloud changes in the region, which can be used to explore weather changes. Meanwhile, spatiotemporal data mainly relies on various sensors installed on roads. These sensors are deployed in the road network to capture and record spatiotemporal information within the region, providing important data support for research. The traffic prediction model consists of four core modules: a cloud image encoder, a spatiotemporal encoder, a spatiotemporal decoder, and a fusion module. See also Figure 2 Cloud imagery data (represented in pink) and spatiotemporal data (represented in green) are input into the cloud imagery encoder and spatiotemporal encoder, respectively, for feature extraction. The extracted high-level feature outputs are then compared to obtain an alignment loss. This loss aims to align the feature spaces of the cloud imagery encoder and the spatiotemporal encoder by promoting similar representations of weather and traffic information for the same timestamp and region. This multimodal contrastive loss has proven to be an effective optimization technique in image-text multimodal tasks. Simultaneously, to better integrate weather features into spatiotemporal features, both types of features are input into the fusion module to obtain fused features and a matching loss. Compared to the alignment loss, the matching loss focuses more on learning the multimodal representation after the fusion of the two feature information to capture the correlation between them. These features, rich in cloud imagery, spatiotemporal, and fused information, undergo a simple but effective addition operation to form the input to the spatiotemporal decoder, which generates the final prediction result. This design aims to fully utilize information from different data sources to improve the overall performance and effectiveness of the prediction model.

[0088] The following section, in conjunction with the accompanying diagram, provides a detailed explanation of the four core models of the traffic prediction model.

[0089] (1) Cloud map encoder

[0090] For the cloud image encoder, during the data preprocessing stage, it processes the cloud image through region partitioning to ensure correspondence with the spatial region input by the spatiotemporal encoder. Simultaneously, during acquisition, it ensures timestamp alignment with the temporal portion of the spatiotemporal data to guarantee that each piece of information input by the spatiotemporal encoder corresponds to the same timestamp. This module requires image processing, therefore employing the ViT (VisionTransformer) architecture, widely used in computer vision. The ViT structure contains only the encoder, primarily used to extract image features, constructing a concise and effective model. The cloud image encoder mainly consists of three parts: an image embedding layer, a multi-head self-attention layer, and a feedforward network layer. When processing images, the Transformer architecture requires embedding to ensure the input dimension is suitable for the temporal architecture's processing. We first segment the input image into fixed-size image patches, which are then further flattened and mapped to fixed-length vectors through a linear transformation. Assume the dimension of the input image is... For the T dimension, as mentioned above, during the data preprocessing stage, we have already mapped the cloud map data to the regions involved in the spatiotemporal data, and ensured temporal alignment based on the input timestamps. Therefore, the next step is to process the image at each timestamp specifically to adapt it to the cloud map encoder, focusing on the H×W×C dimension. This is achieved by embedding the image into N image patches, each P×P in size, followed by a flattening operation to map the dimensions of the image patches to the specified embedding dimension. See [link to documentation]. Figure 2 To align with spatiotemporal features, unlike the original ViT architecture, we added a time stamp (represented by a light gray square) before the vector sequence to indicate the time corresponding to the data. Then, a learnable positional encoding (represented by a dark gray square) was added and input into the encoder, ultimately forming an embedding input consisting of N+2 sequences. After addressing the image embedding dimension issue, the next step is processing through a multi-head self-attention layer. The attention mechanism allows each position in the sequence to focus on other positions in the sequence, capturing dependencies within the sequence, especially long-term dependencies. This module generates the query matrix Q, key matrix K, and value matrix V by multiplying the input matrix with the corresponding parameter matrix. In the attention mechanism, the first operation is the query operation, where the query matrix Q is multiplied by the key matrix K to obtain an attention score, followed by normalization using the Softmax function. To avoid the Softmax function input value being too large, causing the gradient to approach zero, we divide the multiplication matrix by... The normalized attention score is then multiplied by the value matrix V to obtain value matrices V with different weights, which helps to distinguish key features worth noting. The specific expression of the entire attention mechanism is as follows: (1)

[0091]

[0092] The multi-head attention mechanism, based on the self-attention mechanism, divides each query matrix Q, key matrix K, and value matrix V into multiple branches to focus on different aspects, and finally concatenates these multiple branches to restore the dimensions before splitting. This method enables the model to capture various key features in the input information from multiple angles and dimensions, significantly enhancing the model's expressive power. Subsequently, a non-linear expression is introduced through a fully connected layer with an activation function, i.e., a feedforward network layer, to integrate different information and ensure information diversity. In addition, the encoder also employs residual connections, allowing the input to be directly added to the outputs of these layers. This design allows the modular network to learn a residual function instead of a complete function that maps the input to the output. For example, the residual connection operation before and after the multi-head self-attention layer can be represented by formula (2), where the input image X Cloud After the operation of the multi-head self-attention layer (referred to as the F() function), a temporary output F(X) is obtained. Cloud The final result is then based on this, with the original input X added through the residual concatenation. Cloud Therefore, what the multi-head self-attention layer learns is actually the residual information between the input and the expected output, which promotes the direct transmission of information, calibrates the optimization direction, avoids the gradient vanishing problem, and further improves the learning efficiency.

[0093] Output Cloud =F(X) Cloud )+X Cloud (2)

[0094] (2) Fusion Module

[0095] See Figure 2The fusion module comprises a cross-attention layer and a feedforward network layer, both of which play crucial roles in processing the fused feature information extracted by the cloud image encoder and the spatiotemporal encoder. The cross-attention layer is designed to handle the correlations between different modal sequences, which is particularly common in text-image multimodal scenarios. In this layer, we use spatiotemporal sequence data as a basic condition to control the generation process of new features. The correlation between each sequence and different regions of the image is calculated through an attention mechanism, and a normalized similarity score is obtained accordingly. Finally, a new conditionally controlled representation is obtained through weighted summation. This process promotes the full interaction and fusion of the two types of feature information. When applying the attention mechanism, the expression formula used is the same as that of the self-attention mechanism mentioned above, involving a query matrix Q, a key matrix K, and a value matrix V. The difference lies in that the attention mechanism here involves two matrix inputs from different sources. The query matrix Q is generated from image feature information, while the key matrix K and value matrix V originate from spatiotemporal sequence features. On the other hand, the feedforward network layer plays a similar role as described above, its main purpose being to introduce diverse nonlinear expressions to fuse long-term dependencies and enrich feature information. In this way, we can further improve the model's ability to understand and express complex data relationships, thereby enhancing overall prediction performance.

[0096] (3) Spacetime Decoder

[0097] See Figure 2 The spatiotemporal encoder first utilizes a graph convolutional neural network to process the input data X. S-T Spatial feature extraction is performed. Significant spatial dependencies exist between adjacent or nearby nodes in graph data. For example, when a node experiences traffic congestion, the traffic situation of its neighboring nodes is usually affected, exhibiting similar congestion conditions. Traditional convolutional neural networks or recurrent networks struggle to capture these specific spatial dependencies; therefore, graph neural networks are typically chosen for processing graph-like data. Before entering the graph convolutional neural network, a simple embedding layer is first used to reduce the input dimension from... Convert to To match the input of the graph convolutional neural network. The core task of the graph convolutional network is to extract the spatial features of the graph structure and generate new features by aggregating the information of adjacent nodes for subsequent processing. The specific convolution operation can be shown by formula (3). Where X S-T X represents the feature of this layer. S-T ' represents the updated features of the next layer, W is the weight matrix to be updated, and σ() represents the non-linear activation function. Finally, after graph convolution, we obtain a result rich in spatial information. in, It can be viewed as a whole. This step simply performs a symmetric normalization operation on the adjacency matrix A to prevent the scale of the eigenvectors from becoming too large as they accumulate layer by layer. The combination of these elements can be understood as integrating the information of adjacent nodes. This makes graph convolutional networks different from traditional convolutional networks in handling spatial dependencies, and this approach is significant in capturing complex spatial relationships and improving model performance.

[0098]

[0099] After acquiring the spatial features, we perform a dimension matching transformation using a simple linear projection layer. Convert to A module adapted for temporal feature extraction is included. After dimension matching, the features are sequentially imported into the temporal feature extraction layer, namely the lightweight TimesNet network. As an innovative work in the field of time series prediction, it has demonstrated remarkable effectiveness. This network employs a novel multi-period processing method to analyze temporal variations, achieving outstanding results in both long-term and short-term predictions. The TimesNet network mainly consists of two parts: a temporal variation module and a convolutional extraction module.

[0100] The time series transformation module extracts the periodicity of the input one-dimensional time series features and transforms them into a two-dimensional tensor to represent the two-dimensional time series transformation. First, the periodicity in the time series data is obtained through the Fast Fourier Transform (FFT) function, as shown in formula (4), where A represents the intensity of each frequency component in the time input data. Through the argTopk() function, we obtain the k frequencies with the strongest intensity, and based on the relationship between frequency and period, we obtain the lengths of the k most significant periods, corresponding to formulas (5) and (6), respectively. Based on this, we can perform a transformation operation to fold the original one-dimensional time series into a two-dimensional time series, that is, by padding the end of the sequence with zeros using Padding() and folding based on periodicity using the Reshape() function, as shown in formula (7).

[0101] A = Avg(Amp(FFT(X)) Sp-Te (4)

[0102] f1,...,f k =argTopk(A) (5)

[0103]

[0104] X Sp-Te 2D =Reshape(Padding(X) Sp-Te (7)

[0105] After temporal variation, the TimesNet network uses a classic Inception-type convolutional network to capture temporal features. However, this convolutional network uses convolutional kernels of various scales, with the largest scale even reaching 11×11, which brings certain difficulties to the training process. In order to cope with the challenges of excessive number of parameters and excessive training time, this embodiment replaces the Inception-type convolutional network originally used for temporal feature extraction in TimesNet with a depthwise separable convolution, as shown in Equation (8).

[0106] X Sp-Te 2D =DWC(X) Sp-Te 2D (8)

[0107] Figure 3 This section showcases the convolutional kernels and specific processes of Inception-type convolutions and depthwise separable convolutions. Inception-type networks integrate convolutional results of different scales, from 1×1 to 11×11, through concatenation operations, achieving multi-scale feature extraction and fusion. While large convolutional kernels increase the receptive field to some extent, in temporal information processing, only intra-period and inter-period information is typically considered. Excessively large kernels not only fail to improve accuracy but also lead to significant parameter redundancy. Depthwise separable convolutions extract spatial information and channel features by separating depthwise convolution and pointwise convolution. In depthwise convolution, a single kernel is applied to each input channel, generating a feature map with the same number of channels. However, depthwise convolution alone cannot effectively utilize information from different channels at the same spatial location. Therefore, pointwise convolution is needed to integrate this channel information and generate new feature maps by applying 1×1 convolutions. Depthwise separable convolutions have the same input and output dimensions as standard convolutions, but this decomposition method significantly reduces computational cost and model size, reducing parameters by 8-9 times compared to a standard 3×3 convolution. However, since there is some redundancy in information across different periods, the impact of this convolutional separation on accuracy is minimal. Furthermore, to enhance expressive richness, we introduce non-linear activation, using ReLU activation layers after depthwise and pointwise convolutions.

[0108] As an optional implementation method, see [link to implementation details]. Figure 2While the spatiotemporal decoder and spatiotemporal encoder are similar in basic structure, their input methods differ slightly. The spatiotemporal encoder only receives spatiotemporal information data as input, while the spatiotemporal decoder's input includes extracted cloud image information, extracted spatiotemporal information, and fused cloud image-spatiotemporal information, forming a comprehensive embedded input. The dimensional alignment of these three types of information is achieved through a simple and effective mapping function. The combination of spatiotemporal and weather information provides the model with a more comprehensive and multidimensional input, helping to improve the model's understanding of environmental changes and their impacts. Therefore, the final output mainly depends on the spatiotemporal decoder, resulting in decoding loss, which serves as one of the objectives for optimizing network parameters.

[0109] In summary, existing methods primarily rely on extracting single-type temporal or spatial features for prediction. However, spatial nodes are time-varying, meaning that traffic conditions are influenced not only by complex spatial regions but also by significant temporal factors. Simply considering a single temporal or spatial factor is clearly insufficient to meet the demand for higher-precision predictions. Although research has recognized the importance of simultaneously extracting spatiotemporal features, related studies are still relatively limited. Furthermore, networks capable of simultaneously extracting spatiotemporal features often face problems such as large parameter counts and excessively long training times, impacting model efficiency. This performance is insufficient to meet the demands for efficient processing and rapid delivery, limiting its applicability to a wider range of scenarios. Moreover, traffic conditions are typically influenced by multiple factors; for example, under the same conditions, severe weather often leads to traffic congestion, and the impact of weather factors is often only implicitly reflected in spatiotemporal data. Complex traffic conditions can make models less sensitive to data changes, failing to correctly interpret the weather features contained in the spatiotemporal data, increasing system complexity. This invention employs a lightweight traffic prediction model that integrates multi-faceted and multi-dimensional information. By employing a serial extraction method for spatiotemporal features, the uncertainties and instabilities that may arise from spatiotemporal heterogeneity are avoided. Furthermore, redundant information in the spatiotemporal feature extraction process is simplified, achieving efficient and high-precision prediction. In addition, incorporating weather information as auxiliary guidance allows weather features to be explicitly learned, reducing the model's learning difficulty, improving model performance and training efficiency, and ultimately enhancing the system's stability and reliability.

[0110] Example 2

[0111] This invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to achieve the following: Figure 1 and / or Figure 4This paper presents a multimodal traffic flow prediction method based on the fusion of features from multiple data sources.

[0112] It is understood that the memory may include random access memory (RAM) or read-only memory. Optionally, the memory may include non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the stored data area may store data created according to the use of the server, etc.

[0113] A processor may include one or more processing cores. The processor connects to various parts of the server via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various server functions and process data. Optionally, the processor may be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor may integrate one or more of the following: Central Processing Unit (CPU) and Modem. The CPU primarily handles the operating system and applications; the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0114] Since this electronic device is the electronic device corresponding to the multimodal traffic flow prediction method based on multi-source data feature fusion in the embodiments of the present invention, and the principle of solving the problem by this electronic device is similar to that of the method, the implementation of this electronic device can refer to the implementation process of the above method embodiments, and the repeated parts will not be described again.

[0115] Example 4

[0116] This invention also provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to achieve the following: Figure 1 and / or Figure 4 This paper presents a multimodal traffic flow prediction method based on the fusion of features from multiple data sources.

[0117] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0118] Since this storage medium is the storage medium corresponding to a multimodal traffic flow prediction method based on multi-source data feature fusion in an embodiment of the present invention, and the principle of this storage medium in solving the problem is similar to that of this method, the implementation of this storage medium can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0119] Example 5

[0120] In some possible implementations, various aspects of the methods of the embodiments of the present invention can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of a multimodal traffic flow prediction method based on multi-source data feature fusion according to various exemplary embodiments of the present application described above. The executable computer program code or "code" used to perform the various embodiments can be written in high-level programming languages ​​such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0121] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0122] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0123] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A multimodal traffic flow prediction method based on multi-source data feature fusion, characterized in that, Includes the following steps: Obtain traffic data; The training and test sets are divided based on the obtained traffic data; A traffic prediction model is constructed, trained using a training set, and validated using a test set. The traffic prediction model includes a cloud map encoder, a spatiotemporal encoder, a fusion module, and a spatiotemporal decoder. The cloud map encoder is used to extract features from the input cloud map data to obtain cloud map features; The spatiotemporal encoder is used to extract features from the input spatiotemporal data to obtain spatiotemporal features; The fusion module is used to fuse cloud map features and spatiotemporal features to obtain fused features; The cloud map features, spatiotemporal features, and fused features are added together and then input into the spatiotemporal decoder for prediction to obtain the final prediction result. The cloud map encoder includes: The image embedding layer is used to perform region segmentation processing on the cloud image to obtain a multi-dimensional sequence, so as to ensure that the input dimension is suitable for the subsequent multi-head self-attention layer and the first feedforward network layer. Multi-head self-attention layer is used to enable each position in a sequence to focus on other positions in the sequence through an attention mechanism, in order to capture the dependencies within the sequence; The first feedforward network layer is used to introduce non-linear representations to integrate different information and ensure information diversity; The spatiotemporal encoder and the spatiotemporal decoder have the same structure; the input of the spatiotemporal encoder is spatiotemporal data; the input of the spatiotemporal decoder is the sum of the aligned cloud map features and spatiotemporal features and the fused features. The spatiotemporal decoder includes: Graph convolutional neural network layers are used to extract spatial features from input data and generate new features by aggregating information from neighboring nodes; The dimension transformation layer is used to transform the generated features by dimension matching to obtain a one-dimensional time series, so as to adapt to the extraction of time features; The TimesNet network layer includes a temporal variation module and a convolution extraction module. The temporal variation module is used to extract the period of the input one-dimensional temporal features and convert them into a two-dimensional tensor to represent the two-dimensional temporal variation. The convolution extraction module is used to capture the temporal features based on the two-dimensional temporal features and make predictions based on the captured temporal features.

2. The multimodal traffic flow prediction method based on multi-source data feature fusion according to claim 1, characterized in that, The acquisition of traffic data includes: Traffic data from different time periods and traffic areas, along with cloud image data of the corresponding traffic areas, are integrated and summarized to obtain spatiotemporal data and cloud image data. During the acquisition of cloud map data, the time portion of the spatiotemporal data is ensured to be timestamped to guarantee that each piece of information input by the cloud map encoder and the spatiotemporal encoder corresponds to the same timestamp.

3. The multimodal traffic flow prediction method based on multi-source data feature fusion according to claim 1, characterized in that, Cloud maps are processed through region division, including: The cloud map is divided into segments by embedding. N A fixed-size image block; The obtained image patches are flattened and mapped to a fixed-length vector sequence through a linear transformation; Add a time stamp before the vector sequence to indicate the time corresponding to the data; Adding a position code before the time stamp ultimately forms a sequence of elements. N +2 sequences as the embedded input.

4. The multimodal traffic flow prediction method based on multi-source data feature fusion according to claim 1, characterized in that, The cloud map encoder also employs residual connections. The output of the image embedding layer is used as the input of the multi-head self-attention layer. The output of the image embedding layer and the output of the multi-head self-attention layer are added together as the input of the first feedforward network layer. The output of the self-attention layer and the output of the first feedforward network layer are added together as the output of the cloud map encoder.

5. The multimodal traffic flow prediction method based on multi-source data feature fusion according to claim 1, characterized in that, The fusion module includes: The cross-attention layer uses the outputs of both the cloud map encoder and the spatiotemporal encoder as inputs to process the correlation between multiple modal sequences through an attention mechanism. The second feedforward network layer is used to integrate long-term dependencies and enrich feature information by introducing diverse non-linear representations.

6. The multimodal traffic flow prediction method based on multi-source data feature fusion according to claim 1, characterized in that, The calculation formula for the graph convolutional neural network layer is as follows: In the formula, This indicates the features of this layer. For the updated next layer features, For the weight matrix that needs to be updated, This indicates a nonlinear activation function. Represents the adjacency matrix A symmetrical normalization operation was performed.

7. The multimodal traffic flow prediction method based on multi-source data feature fusion according to claim 1, characterized in that, The specific working method of the timing change module is as follows: The periodicity in one-dimensional time series data can be obtained using the Fast Fourier Transform (FFT) function, as shown in the following expression: In the formula, This represents the intensity of each frequency component in the time-input data. Avg This indicates calculating the average. Amp This indicates the calculation of amplitude. The strongest value is obtained through the argTopk() function. Several frequencies were identified, and the most significant frequency was obtained based on the relationship between frequency and period. The length of each period is expressed as follows: In the formula, Indicates the first k One frequency, Indicates the first k The length of each cycle, Indicates the length of time; Based on the calculated frequency and the corresponding period length The one-dimensional time series can be reshaped into a two-dimensional time series using the following equation: padding the end of the sequence with zeros using the Padding() function and folding it based on periodicity using the Reshape() function. The expression is as follows: In the formula, It represents a two-dimensional timing sequence.

8. The multimodal traffic flow prediction method based on multi-source data feature fusion according to claim 1, characterized in that, The convolution extraction module works as follows: Replace the Inception-type convolutional network used for temporal feature extraction in TimesNet with a depthwise separable convolution, as shown in the following expression: In the formula, This indicates the time characteristics of the capture.

Images