A Multi-Level Embedded Spatiotemporal Prediction Method for Traffic Flow Based on a Pre-trained Large Language Model

Abstract

This invention discloses a multi-layered embedding method for traffic flow spatiotemporal prediction based on a pre-trained large language model. The traffic flow spatiotemporal prediction model includes a spatiotemporal embedding layer, a multi-layer temporal coding network, a word segmentation alignment layer, a pre-trained large language model, and a traffic flow decoding layer. Through adaptive traffic flow spatiotemporal embedding and the multi-layer temporal coding network, the complex dynamic spatiotemporal relationships of traffic flow are constructed and captured. The word segmentation alignment layer aligns and combines traffic flow node tokens (described by node-level word segmentation of temporal semantic information) with prompt text tokens. Finally, the multi-layered embedding, which integrates historical features, dynamic spatiotemporal features, and prompt text features, is input into the pre-trained large language model, and the downstream prediction task is completed by training the large language model. This invention's method can improve prediction accuracy and reduce the number of trainable parameters.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation, specifically to a multi-layered embedding method for spatiotemporal prediction of traffic flow based on a pre-trained large language model. Background Technology

[0002] With urbanization, the increasing number of vehicles on the road has led to more and more road congestion and traffic accidents. Traffic flow prediction, as an important research direction in the field of intelligent transportation, is of great significance for alleviating road congestion and improving road utilization. With the continuous development of big data, artificial intelligence, and intelligent transportation technologies, traffic flow prediction methods are also constantly evolving and improving. By analyzing multi-dimensional information such as historical traffic data, real-time traffic events, and road network data, traffic flow prediction can accurately predict future traffic conditions, providing guidance and support for urban transportation systems.

[0003] Traffic flow data is spatiotemporal data, encompassing three dimensions: time, space, and traffic flow characteristics. Therefore, traffic flow prediction tasks require uncovering the spatiotemporal dependencies between traffic flow data. Existing prediction methods often use static, predefined graphs to model the spatial structure of traffic flow, making it difficult to capture the potential dynamic spatiotemporal relationships within the traffic flow, ultimately resulting in large prediction biases. Furthermore, existing prediction models often only support spatiotemporal data input, failing to effectively integrate natural language text related to traffic prediction, such as dataset descriptions, weather, holidays, and traffic accidents. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a multi-layered embedding method for traffic flow spatiotemporal prediction based on a pre-trained large language model. This method adaptively constructs the dynamic spatiotemporal relationships of traffic flow and integrates textual features of traffic flow. By utilizing multi-layered embedding and a pre-trained large language model, it achieves better prediction results. The specific technical solution is as follows:

[0005] A multi-layered embedding method for spatiotemporal prediction of traffic flow based on a pre-trained large language model, comprising the following steps:

[0006] S1: Obtain historical traffic flow data within a predetermined range and preprocess it;

[0007] S2: Construct and train a traffic flow spatiotemporal prediction model; the traffic flow spatiotemporal prediction model includes a spatiotemporal embedding layer, a multi-layer temporal coding network, a word segmentation and alignment layer, a pre-trained large language model, and a traffic flow decoding layer; the spatiotemporal embedding layer is used to generate a traffic flow spatiotemporal embedding input from preprocessed historical traffic flow data based on the historical characteristics, temporal dependencies, and spatial dependencies of the traffic flow; each layer of the multi-layer temporal coding network is a temporal coding network based on a self-attention mechanism, and the input of the multi-layer temporal coding network is the traffic flow spatiotemporal embedding input. The multi-layer temporal coding network uses a self-attention mechanism to capture the temporal dependencies between all time slices of different traffic nodes; the word segmentation and alignment layer converts the output of the multi-layer temporal coding network into traffic flow node tokens through a fully connected layer, and concatenates the traffic flow node tokens with prompt text tokens related to traffic flow prediction to obtain the input of the pre-trained large language model; the traffic flow decoding layer is used to convert the output of the pre-trained large language model into the prediction dimension of traffic flow for each traffic node, and output the predicted traffic flow;

[0008] S3: Use the trained traffic flow spatiotemporal prediction model to predict traffic flow in spatiotemporal terms.

[0009] Furthermore, step S1 specifically includes the following sub-steps:

[0010] Acquire real-world traffic flow data, including traffic flow data collected by multiple sensors in a predetermined road segment over a certain period of time, along with the date and time of collection;

[0011] Traffic flow data were normalized using the Z-Score normalization method.

[0012] Traffic flow data is segmented according to the traffic flow prediction task to obtain historical traffic flow data with the required step size.

[0013] Further, in step S2, the spatiotemporal embedding layer is used to generate a spatiotemporal embedding input for traffic flow from the preprocessed historical traffic flow data based on the historical characteristics, temporal dependencies, and spatial dependencies of the traffic flow. This is specifically achieved through the following sub-steps:

[0014] (1) The historical traffic flow data of all traffic nodes after preprocessing After being converted to a fully connected component in, This represents the traffic flow data of N traffic nodes in the road network at time t, where C represents the dimension of the traffic flow feature, T represents the time step of the historical traffic flow data, and d represents the time step of the historical traffic flow data. f Embedding dimensions into historical traffic flows;

[0015] Define a trainable spatial adaptive embedding for each node in the traffic flow and extend it in the time dimension to obtain the spatial adaptive embedding. Where d s Representation space adaptive embedding dimension;

[0016] Define two trainable time-period embedding dictionaries for each time step of the traffic flow: one is a timestamp embedding dictionary for the entire day. Where N d d represents the average number of time steps in a day. t The first dimension represents the embedding dimension of the time period; the second is the weekday embedding dictionary. Where N w This represents the week number within a week; based on the time slices of historical traffic flow data, the corresponding timestamp embedding and weekday embedding are retrieved, and then expanded in the spatial dimension to obtain two time period embeddings. and

[0017] (2) E data E s E day E week By splicing the data, we obtain the spatiotemporal embedding input X of traffic flow. ST =E data ∥E s ∥E day ∥E week Where, ∥ represents the concatenation operation, and the embedding dimension d = d f +d s +2d t .

[0018] Furthermore, in S2, the multi-layer temporal coding network utilizes a self-attention mechanism to capture the temporal dependencies between all time slices of different traffic nodes. Each layer of the self-attention-based temporal coding network specifically includes:

[0019] (1) Calculate the query Q matrix, key K matrix, and value V matrix in sequence, and then calculate the self-attention score representing the temporal dependencies of different nodes. Finally, the output X of the self-attention encoding is calculated. t =A t V t The specific calculation formula is as follows:

[0020]

[0021] in, All of these are learnable parameters;

[0022]

[0023] (2) Apply self-attention coding to a multi-head network, and after layer normalization and residual connection, pass it through a fully connected feedforward network, and then apply layer normalization and residual connection again to obtain the output of a single-layer time coding network based on self-attention mechanism.

[0024] Further, in S2, the word segmentation alignment layer converts the output of the multi-layer time coding network into traffic flow node tokens through a fully connected layer, and concatenates the traffic flow node tokens with prompt text tokens related to traffic flow prediction to obtain the input of the pre-trained large language model, specifically including:

[0025] The output of the multilayer time-coding network is connected via a fully connected layer. Converted into tokens input of the pre-trained large language model T q =MLP(X encoder ), where D represents the dimension of the large language model token, and each token represents the historical traffic flow information of a node;

[0026] The traffic flow prompt text, which includes traffic flow datasets, weather, and traffic accidents, is processed by a frozen pre-trained large language model's text segmenter to obtain prompt text tokens input T. p ;

[0027] T p With T q Align and concatenate to obtain the input T of the pre-trained large language model. LLM =T p ∥T q , where ∥ represents the concatenation operation.

[0028] Furthermore, when training the traffic flow spatiotemporal prediction model, the feedforward network and multi-head self-attention module of the pre-trained large language model are frozen, and only the position encoding module, residual and layer normalization module of the pre-trained large language model are trained.

[0029] Furthermore, the traffic flow decoding layer is used to convert the output of the pre-trained large language model into the prediction dimension of traffic flow at each traffic node, and output the predicted traffic flow, specifically including:

[0030] The output of the traffic flow node token positions of the pre-trained large language model is decoded through a fully connected layer to obtain the traffic flow results predicted for all nodes: in This represents the last N tokens output by the large model, which represent the corresponding traffic flow node tokens.

[0031] By inversely normalizing Y using the Z-Score, the predicted traffic flow output of the traffic flow spatiotemporal prediction model is obtained. Where T′ and C′ represent the time step and dimension of the predicted traffic flow.

[0032] Furthermore, the loss function used when training the traffic flow spatiotemporal prediction model is the Huber loss function.

[0033] A multi-level embedded traffic flow spatiotemporal prediction device based on a pre-trained large language model includes one or more processors for implementing a multi-level embedded traffic flow spatiotemporal prediction method based on a pre-trained large language model.

[0034] An electronic device, comprising:

[0035] One or more processors;

[0036] A storage device for storing one or more programs that, when executed by the electronic device, enable the electronic device to implement a multi-layered embedded spatiotemporal prediction method for traffic flow based on a pre-trained large language model.

[0037] The beneficial effects of this invention are as follows:

[0038] This invention utilizes an adaptive spatiotemporal embedding layer and a multi-layer temporal coding network to construct and capture the complex dynamic spatiotemporal relationships of traffic flow, effectively improving the ability to capture these temporal relationships. A word segmentation alignment layer combines spatiotemporal word segmentation of traffic flow (described by node-level word segmentation of temporal semantic information) with word segmentation of prompt text. Finally, a multi-layered embedding input, incorporating historical features, dynamic spatiotemporal features, and prompt text features, is used to input a pre-trained large language model. The downstream prediction task is completed by training this pre-trained large language model. During model training, the natural language understanding ability and domain knowledge of the large language model are preserved while enhancing the ability to capture spatiotemporal dependencies, improving prediction accuracy, and reducing trainable parameters. The method of this invention can effectively integrate natural language text related to traffic prediction, achieving high prediction accuracy with low computational cost. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of a multi-layered embedding method for spatiotemporal prediction of traffic flow based on a pre-trained large language model, according to an embodiment of the present invention.

[0040] Figure 2 This is a schematic diagram of the overall structure of the traffic flow spatiotemporal prediction model according to an embodiment of the present invention.

[0041] Figure 3 This is a model structure diagram of a multilayer time coding network according to an embodiment of the present invention. Detailed Implementation

[0042] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0043] like Figure 1 and 2 As shown, the present invention provides a multi-layered embedding method for spatiotemporal prediction of traffic flow based on a pre-trained large language model, comprising the following steps:

[0044] Step 1: Obtain historical traffic flow data within the predetermined range and preprocess it.

[0045] Acquire real-world traffic flow data, including traffic flow data collected by multiple sensors within a certain time period in a predetermined road segment, along with the date and time of collection; normalize the traffic flow data using the Z-Score normalization method; and obtain historical traffic flow data and predicted traffic flow data with the required step size based on the traffic flow prediction task data segmentation.

[0046] Step 2: Build and train a traffic flow spatiotemporal prediction model.

[0047] The traffic flow spatiotemporal prediction model includes a spatiotemporal embedding layer, a multi-layer temporal coding network, a word segmentation alignment layer, a pre-trained large language model, and a traffic flow decoding layer.

[0048] 1. Spatiotemporal Embedding Layer

[0049] The spatiotemporal embedding layer is used to generate spatiotemporal embedding inputs for traffic flow from preprocessed historical traffic flow data, based on the historical characteristics, temporal dependencies, and spatial dependencies of the traffic flow. For example... Figure 2 As shown, it specifically includes:

[0050] use Let C represent the traffic flow data of N nodes in the road network at time t, where C represents the dimension of the traffic flow features. This represents the historical traffic flow data for all nodes, where T represents the time step of the historical traffic flow data; [The last part, "historical traffic flow data," appears to be a fragment and doesn't translate directly. It's left as is.] After a fully connected transformation Where d f Embedding dimensions into historical traffic flows;

[0051] Define a trainable spatial adaptive embedding for each node in the traffic flow and extend it in the time dimension to obtain the spatial adaptive embedding. Where d s The representation space adapts to the embedding dimension.

[0052] Define two trainable time-period embedding dictionaries for each time step of the traffic flow: one is a timestamp embedding dictionary for the entire day. Where N d d represents the average number of time steps in a day. t The first dimension represents the embedding dimension of the time period; the second is the weekday embedding dictionary. Where N w This represents the week number within a week; based on the time slices of historical traffic flow data, the corresponding timestamp embedding and weekday embedding are retrieved, and then expanded in the spatial dimension to obtain two time period embeddings. and

[0053] E data E s E day E week By splicing the data, we obtain the spatiotemporal embedding input X of traffic flow. ST =E data ∥E s ∥E day ∥E week Where, ∥ represents the concatenation operation, and the embedding dimension d = d f +d s +2d t .

[0054] 2. Multilayer Time Coding Network

[0055] Each layer of the multi-layer temporal coding network is a self-attention-based temporal coding network. The input to the multi-layer temporal coding network is the traffic flow spatiotemporal embedding. The multi-layer temporal coding network utilizes the self-attention mechanism to capture the temporal dependencies between all time slices of different traffic nodes. For each layer of the self-attention-based temporal coding network, such as... Figure 3 As shown, it specifically includes:

[0056] (1) Embedding X via input ST The corresponding query Q matrix, key K matrix, and value V matrix are calculated. Then, the self-attention score representing the temporal dependencies of different nodes is calculated. Finally, the output X of the self-attention encoding is calculated. t =A t V t ;

[0057] The specific calculation formula is as follows:

[0058]

[0059] in, All of these are learnable parameters;

[0060]

[0061] (2) Apply self-attention coding to a multi-head network, and after layer normalization and residual connection, pass it through a fully connected feedforward network, and then apply layer normalization and residual connection again to obtain the output of a single-layer time coding network based on self-attention mechanism.

[0062] After passing through multiple layers of the aforementioned self-attention-based temporal coding network, the final output of the multi-layer temporal coding network is obtained.

[0063] 3. Word segmentation alignment layer

[0064] The word segmentation alignment layer converts the output of the multi-layer time-coding network into traffic flow node tokens through a fully connected layer. It then concatenates these traffic flow node tokens with prompt text tokens related to traffic flow prediction, using the concatenated tokens as input to a pre-trained large language model. For example... Figure 2 As shown, it specifically includes:

[0065] To transform spatiotemporal data into a representation that large language models can understand and integrate with text prompts, each node in the output of a multi-layer temporal coding network is treated as a token, i.e., X is processed through a fully connected layer. encoder Convert to tokens input for a large language model T q =MLP(X encoder ), where D represents the dimension of the large language model token, and each token represents the historical traffic flow information of a node.

[0066] To enable large language models to better perform traffic flow prediction tasks and integrate traffic flow information from text modalities, a prompt text related to traffic flow datasets, weather, and traffic accidents is designed. This prompt text is then processed by a frozen, pre-trained large language model's text segmenter to obtain prompt text tokens, which are input to T. p .

[0067] T p With T q Align and concatenate to obtain the input T of the pre-trained large language model. LLM =T p ∥T q , where ∥ represents the concatenation operation.

[0068] 4. Pre-trained large language model

[0069] like Figure 2 As shown, the token input T of the pre-trained large language model LLM After multi-layer self-attention encoding using a large language model, the corresponding traffic flow prediction output T can be obtained. output =LLM(T LLMThe training method for the pre-trained large language model adopts a partial fine-tuning strategy: in order to preserve the natural language understanding ability and domain knowledge of the large language model, its feedforward network and multi-head self-attention module are frozen; in order to enhance the ability of the large language model to capture the spatiotemporal dependencies of traffic flow and adapt to downstream prediction tasks, its position encoding module and layer normalization module are trained.

[0070] 5. Traffic Flow Decoding Layer

[0071] The traffic flow decoding layer is used to convert the output of the pre-trained large language model into the prediction dimension of traffic flow at each traffic node, and outputs the predicted traffic flow, specifically including:

[0072] like Figure 2 As shown, to obtain the traffic flow prediction results for the required dimensions, the output of the traffic flow node token positions in the large language model is decoded through a fully connected layer to obtain the traffic flow prediction results for all nodes: in The last N tokens of the large model output represent the corresponding traffic flow node tokens. Then, Y is denormalized using Z-Score to obtain the predicted traffic flow output of the traffic flow spatiotemporal prediction model. Where T′ and C′ represent the time step and dimension of the predicted traffic flow.

[0073] The traffic flow spatiotemporal prediction model uses the Huber loss function during training, and the calculation formula is as follows:

[0074]

[0075] Among them, Y ′ The traffic flow output predicted by the model. The model output represents the actual traffic flow, and δ is a hyperparameter representing the error loss. The model parameters are continuously optimized through backpropagation to obtain an optimal model.

[0076] Finally, the trained traffic flow spatiotemporal prediction model is used to predict traffic flow in the spatiotemporal direction.

[0077] On the other hand, the present invention also provides a multi-layered embedded traffic flow spatiotemporal prediction device based on a pre-trained large language model, comprising one or more processors for implementing the multi-layered embedded traffic flow spatiotemporal prediction method based on a pre-trained large language model described in the above embodiments. Embodiments of the multi-layered embedded traffic flow spatiotemporal prediction device based on a pre-trained large language model of the present invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data-processing device reading the corresponding computer program instructions from non-volatile memory into memory and running them. From a hardware perspective, the multi-layered embedded traffic flow spatiotemporal prediction device based on a pre-trained large language model, in addition to the processor, memory, network interface, and non-volatile memory, may also include other hardware depending on the actual function of the data-processing device; these will not be elaborated further.

[0078] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the multi-layered embedding traffic flow spatiotemporal prediction method based on a pre-trained large language model as described in the above embodiments.

[0079] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computing program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0080] The effectiveness of this invention is verified using a real-world traffic flow spatiotemporal dataset, PeMS08. The PeMS08 dataset contains traffic data collected from 170 sensors on major roads in California at a 5-minute frequency from July to August 2016. Comparison methods include SVR (Support Vector Regression), STGCN (Spatiotemporal Graph Convolutional Network), GMAN (Graph Multi-Head Attention Network), and the method of this invention (a multi-layered embedding method for spatiotemporal prediction of traffic flow based on a pre-trained large language model).

[0081] To ensure a fair comparison with contrasting methods, as with existing research, we divided the training, validation, and test sets in a 6:2:2 ratio and used historical traffic flow over 12 time steps to predict future 12 time steps.

[0082] For the prediction task on the PeMS08 traffic flow dataset, a basic prompt text example can be designed, including two parts: traffic domain knowledge and task description: "Domain description: This dataset contains traffic flow values ​​on California highways, with daytime traffic flow typically higher. Task description: Your task is to predict the traffic flow value for the next <1 hour> at <Sunday> <18:00:00>, based on the following..." <170> Traffic flow values ​​for the previous hour at each node: . The content within the quotation marks (<>) varies depending on the traffic flow dataset and the specific prediction task.

[0083] The spatiotemporal prediction performance results are shown in Table 1.

[0084] Table 1 shows the prediction performance of each comparative method on the PEMS08 dataset.

[0085]

[0086] As can be seen from Table 1, the multi-level embedding traffic flow spatiotemporal prediction method based on a pre-trained large language model proposed in this embodiment of the invention is significantly better than the comparative methods in terms of prediction performance.

[0087] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A multi-layered embedding method for spatiotemporal prediction of traffic flow based on a pre-trained large language model, characterized in that, The method includes the following steps: S1: Obtain historical traffic flow data within a predetermined range and preprocess it; S2: Construct and train a traffic flow spatiotemporal prediction model; the traffic flow spatiotemporal prediction model includes a spatiotemporal embedding layer, a multi-layer temporal coding network, a word segmentation and alignment layer, a pre-trained large language model, and a traffic flow decoding layer; the spatiotemporal embedding layer is used to generate a traffic flow spatiotemporal embedding input from preprocessed historical traffic flow data based on the historical characteristics, temporal dependencies, and spatial dependencies of the traffic flow; each layer of the multi-layer temporal coding network is a temporal coding network based on a self-attention mechanism, and the input of the multi-layer temporal coding network is the traffic flow spatiotemporal embedding input. The multi-layer temporal coding network uses a self-attention mechanism to capture the temporal dependencies between all time slices of different traffic nodes; the word segmentation and alignment layer converts the output of the multi-layer temporal coding network into traffic flow node tokens through a fully connected layer, and concatenates the traffic flow node tokens with prompt text tokens related to the traffic flow dataset to obtain the input of the pre-trained large language model; the traffic flow decoding layer is used to convert the output of the pre-trained large language model into the prediction dimension of traffic flow for each traffic node, and output the predicted traffic flow; In step S2, the spatiotemporal embedding layer is used to generate a spatiotemporal embedding input for traffic flow from the preprocessed historical traffic flow data based on the historical characteristics, temporal dependencies, and spatial dependencies of the traffic flow. This is specifically achieved through the following sub-steps: (1) The historical traffic flow data of all traffic nodes after preprocessing After being converted to a fully connected component ;in, , Indicates in In the time-based road network Traffic flow data for each traffic node, Dimensions representing traffic flow characteristics This indicates the time step of historical traffic flow data. Embedding dimensions into historical traffic flows; Define a trainable spatial adaptive embedding for each node in the traffic flow and extend it in the time dimension to obtain the spatial adaptive embedding. ;in Representation space adaptive embedding dimension; Define two trainable time-period embedding dictionaries for each time step of the traffic flow: one is a timestamp embedding dictionary for the entire day. ,in This represents the average number of time steps in a day. The first dimension represents the embedding dimension of the time period; the second is the weekday embedding dictionary. ,in This represents the week number within a week; based on the time slices of historical traffic flow data, the corresponding timestamp embedding and weekday embedding are retrieved, and then expanded in the spatial dimension to obtain two time period embeddings. and ; (2) , , , By splicing the data, we obtain the spatiotemporal embedding input of traffic flow. ,in, This indicates a concatenation operation, embedding dimension. ; S3: Use the trained traffic flow spatiotemporal prediction model to predict traffic flow in spatiotemporal terms.

2. The multi-layered embedding method for spatiotemporal prediction of traffic flow based on a pre-trained large language model according to claim 1, characterized in that, S1 specifically includes the following sub-steps: Acquire real-world traffic flow data, including traffic flow data collected by multiple sensors in a predetermined road segment over a certain period of time, along with the date and time of collection; Traffic flow data were normalized using the Z-Score normalization method. Traffic flow data is segmented according to the traffic flow prediction task to obtain historical traffic flow data with the required step size.

3. The multi-layered embedding method for spatiotemporal prediction of traffic flow based on a pre-trained large language model according to claim 1, characterized in that, In step S2, the multi-layer time coding network utilizes a self-attention mechanism to capture the temporal dependencies between all time slices of different traffic nodes. Each layer of the time coding network based on the self-attention mechanism specifically includes: (1) Calculate the query Q matrix, key K matrix, and value V matrix in sequence, and then calculate the self-attention score representing the time dependency of different nodes. Finally, the output of the self-attention encoding is calculated. The specific calculation formula is as follows: ， ， ； in, , , All of these are learnable parameters; ； (2) Apply self-attention coding to a multi-head network, and after layer normalization and residual connection, pass through a fully connected feedforward network, and then apply layer normalization and residual connection again to obtain the output of a single-layer time coding network based on self-attention mechanism.

4. The multi-layered embedding method for spatiotemporal prediction of traffic flow based on a pre-trained large language model according to claim 1, characterized in that, In step S2, the word segmentation alignment layer converts the output of the multi-layer time coding network into traffic flow node tokens through a fully connected layer, and concatenates the traffic flow node tokens with prompt text tokens related to traffic flow prediction to obtain the input of the pre-trained large language model, specifically including: The output of the multilayer time-coding network is connected via a fully connected layer. Converted into tokens input of the pre-trained large language model , ,in, The dimension of the large language model token is represented by each token, which represents the historical traffic flow information of a node. The traffic flow prompt text, which includes traffic flow datasets, weather, and traffic accidents, is processed by a frozen pre-trained large language model's text segmenter to obtain prompt text tokens. ; Will and Align and concatenate to obtain the input of the pre-trained large language model. ,in, This indicates a concatenation operation.

5. The multi-layered embedding method for spatiotemporal prediction of traffic flow based on a pre-trained large language model according to claim 4, characterized in that, When training the traffic flow spatiotemporal prediction model, the feedforward network and multi-head self-attention module of the pre-trained large language model are frozen, and only the position encoding module, residual and layer normalization module of the pre-trained large language model are trained.

6. The multi-layered embedding method for spatiotemporal prediction of traffic flow based on a pre-trained large language model according to claim 5, characterized in that, The traffic flow decoding layer is used to convert the output of the pre-trained large language model into the prediction dimension of traffic flow at each traffic node, and output the predicted traffic flow, specifically including: The output of the traffic flow node token positions of the pre-trained large language model is decoded through a fully connected layer to obtain the traffic flow results predicted for all nodes: ;in Represents the output of the large model Each token represents the corresponding output traffic flow node token; Will After Z-Score inverse normalization, the predicted traffic flow output of the traffic flow spatiotemporal prediction model is obtained. ,in and This indicates the time step and dimension of the predicted traffic flow.

7. The multi-layered embedding method for spatiotemporal prediction of traffic flow based on a pre-trained large language model according to claim 6, characterized in that, The loss function used when training the traffic flow spatiotemporal prediction model is the Huber loss function.

8. A multi-layered embedded traffic flow spatiotemporal prediction device based on a pre-trained large language model, characterized in that, It includes one or more processors for implementing the multi-level embedding traffic flow spatiotemporal prediction method based on a pre-trained large language model, as described in any one of claims 1 to 7.

9. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by the electronic device, cause the electronic device to implement the multi-layered embedding method for spatiotemporal prediction of traffic flow based on a pre-trained large language model as described in any one of claims 1 to 7.

Images