Multi-level embedded traffic flow space-time prediction method based on pre-training large language model

Abstract

The invention discloses a multi-level embedded traffic flow space-time prediction method based on a pre-training large language model. A traffic flow space-time prediction model comprises a space-time embedding layer, a multi-layer time coding network, a word segmentation alignment layer, the pre-training large language model and a traffic flow decoding layer. The method comprises the following steps: constructing and capturing a complex dynamic space-time relationship of traffic flow through adaptive traffic flow space-time embedding and a multi-layer time coding network; a traffic flow node token and a prompt text token which describe time semantic information through node dimension word segmentation are aligned and combined through a word segmentation alignment layer; and finally, inputting the multi-level embedding integrated with the historical features, the dynamic spatio-temporal features and the prompt text features into a pre-training large language model, and finishing a downstream prediction task by training the large language model. According to the method, the prediction precision can be improved, and trainable parameters can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation, and in particular to a multi-level embedded spatiotemporal prediction method for traffic flow based on a pre-trained large language model. Background Art

[0002] With the progress of urbanization, the continuous increase of road vehicles has also caused more and more road congestion and traffic accidents. As an important research direction in the field of intelligent transportation, traffic flow prediction is of great significance in alleviating road congestion and improving road utilization. With the continuous development of big data, artificial intelligence and intelligent transportation technology, traffic flow prediction methods are also constantly evolving and improving. By analyzing multi-dimensional information such as historical traffic data, real-time traffic events, road network data, etc., traffic flow prediction can accurately predict future traffic conditions and provide guidance and support for urban transportation systems.

[0003] Traffic flow data is spatiotemporal data, which includes three dimensions: time, space, and traffic flow characteristics. Therefore, the traffic flow prediction task needs to explore the spatiotemporal dependencies between traffic flow data. Existing prediction methods often use static predefined graphs to model the spatial structure of traffic flow, which makes it difficult to capture the potential dynamic spatiotemporal relationships of traffic flow, and ultimately produces results with large prediction deviations. In addition, existing prediction models often only support the input of spatiotemporal data, but cannot integrate natural language texts related to traffic prediction, such as data set descriptions, weather, holidays, traffic accidents, etc. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes a multi-level embedded spatiotemporal prediction method for traffic flow based on a pre-trained large language model, which can adaptively construct the dynamic spatiotemporal relationship of traffic flow, integrate the text features of traffic flow, and use multi-level embedding and pre-trained large language model to obtain better prediction results. The specific technical solution is as follows:

[0005] A multi-level embedded spatiotemporal traffic flow prediction method based on a pre-trained large language model, the method comprising the following steps:

[0006] S1: Obtain historical traffic flow data within a predetermined range and pre-process it;

[0007] S2: Construct and train a spatiotemporal prediction model for traffic flow; the spatiotemporal prediction model for traffic flow includes a spatiotemporal embedding layer, a multi-layer time coding network, a word segmentation alignment layer, a pre-trained large language model and a traffic flow decoding layer; the spatiotemporal embedding layer is used to generate a spatiotemporal embedding input for traffic flow based on the historical characteristics, time dependencies and spatial dependencies of the pre-processed historical traffic flow data; each layer of the multi-layer time coding network is a time coding network based on a self-attention mechanism, the input of the multi-layer time coding network is the spatiotemporal embedding input for traffic flow, and the multi-layer time coding network uses a self-attention mechanism to capture the time dependencies between all time slices of different traffic nodes; the word segmentation alignment layer converts the output of the multi-layer time coding network into traffic flow node tokens through full connection, and concatenates the traffic flow node tokens with the prompt text tokens related to traffic flow prediction to obtain the input of the pre-trained large language model; the traffic flow decoding layer is used to convert the output of the pre-trained large language model into the prediction dimension of the traffic flow of each traffic node, and output the predicted traffic flow;

[0008] S3: Use the trained traffic flow spatiotemporal prediction model to perform traffic flow spatiotemporal prediction.

[0009] Furthermore, the S1 specifically includes the following sub-steps:

[0010] Obtaining traffic flow data collected in the real world, including traffic flow data collected by multiple sensors in a predetermined road section within a certain period of time, the date and specific time of the collection day;

[0011] The Z-Score normalization method is used to normalize the traffic flow data;

[0012] The traffic flow data is segmented according to the traffic flow prediction task to obtain the historical traffic flow data of the required step length.

[0013] Furthermore, in S2, the spatiotemporal embedding layer is used to generate a spatiotemporal embedding input of traffic flow based on the preprocessed historical traffic flow data according to the historical characteristics, time dependency and spatial dependency of the traffic flow, which is specifically achieved through the following sub-steps:

[0014] (1) The historical traffic flow data of all traffic nodes after preprocessing After full connection, it is converted to in, represents the traffic flow data of N traffic nodes in the road network at time t, C represents the dimension of traffic flow characteristics, T represents the time step of historical traffic flow data, d f Embed dimensions for historical traffic flows;

[0015] Define a trainable spatially adaptive embedding for each node in the traffic flow and expand it in the time dimension to obtain a spatially adaptive embedding where d s Represents spatially adaptive embedding dimensions;

[0016] Define two trainable time period embedding dictionaries for each time step of the traffic flow, one for the timestamp embedding dictionary within a day Where N d Indicates the average number of time steps in a day, d t represents the time period embedding dimension; the other is the embedding dictionary of the week within a week Where N w Represents the day of the week; according to the time slice of historical traffic flow data, the corresponding timestamp embedding and week embedding are retrieved and expanded in the spatial dimension to obtain two time period embeddings and

[0017] (2) E data 、E s 、E day 、E week Splice and get the traffic flow spatiotemporal embedding input X ST =E data ∥E s ∥E day ∥E week , where ∥ represents concatenation operation and the embedding dimension d = d f +d s +2d t .

[0018] Furthermore, in S2, the multi-layer time coding network uses a self-attention mechanism to capture the time dependency between all time slices of different traffic nodes. For each layer of the time coding network based on the self-attention mechanism, it specifically includes:

[0019] (1) Calculate the query Q matrix, key K matrix, and value V matrix in sequence, and then calculate the self-attention score representing the temporal dependency of different nodes Finally, calculate the output X of the self-attention code t =A t V t ; The specific calculation formula is as follows:

[0020]

[0021] in, All are learnable parameters;

[0022]

[0023] (2) Self-attention encoding is applied to multiple heads, and after layer normalization and residual connection, it passes through a fully connected feedforward network, and then uses layer normalization and residual connection again to obtain the output of a single-layer temporal encoding network based on the self-attention mechanism.

[0024] Furthermore, in S2, the word segmentation alignment layer converts the output of the multi-layer time coding network into traffic flow node tokens through full connection, and concatenates the traffic flow node tokens and the prompt text tokens related to traffic flow prediction to obtain the input of the pre-trained large language model, which specifically includes:

[0025] The output of the multi-layer time encoding network is fully connected Convert to the tokens input of the pre-trained large language model T q =MLP(X encoder ), where D represents the dimension of the large language model token, and each token represents the historical traffic flow information of a node;

[0026] The traffic flow prompt text including traffic flow dataset, weather, and traffic accidents is passed through the text segmenter of the frozen pre-trained large language model to obtain the prompt text tokens input T p ;

[0027] T p With T q Align and concatenate to obtain the input T of the pre-trained large language model LLM =T p ∥T q , where ∥ represents the concatenation operation.

[0028] Furthermore, when training the traffic flow spatiotemporal prediction model, the feedforward network and multi-head self-attention module of the pre-trained large language model are frozen, and only the position encoding module, residual and layer normalization module of the pre-trained large language model are trained.

[0029] Furthermore, the traffic flow decoding layer is used to convert the output of the pre-trained large language model into the predicted dimension of the traffic flow of each traffic node and output the predicted traffic flow, specifically including:

[0030] The output of the traffic flow node token position of the pre-trained large language model is decoded through a fully connected layer to obtain the traffic flow results predicted by all nodes: in Indicates the last N tokens output by the large model, representing the corresponding output traffic flow node tokens;

[0031] Denormalize Y by Z-Score to obtain the predicted traffic flow output of the traffic flow spatiotemporal prediction model Where T′ and C′ represent the predicted traffic flow time step and dimension.

[0032] Furthermore, the loss function when training the traffic flow spatiotemporal prediction model is the Huber loss function.

[0033] A multi-level embedded traffic flow spatiotemporal prediction device based on a pre-trained large language model includes one or more processors for implementing a multi-level embedded traffic flow spatiotemporal prediction method based on a pre-trained large language model.

[0034] An electronic device, comprising:

[0035] one or more processors;

[0036] A storage device is used to store one or more programs. When the one or more programs are executed by the electronic device, the electronic device implements a multi-level embedded spatiotemporal traffic flow prediction method based on a pre-trained large language model.

[0037] The beneficial effects of the present invention are as follows:

[0038] The present invention utilizes an adaptive spatiotemporal embedding layer and a multi-layer time coding network to construct and capture the complex dynamic spatiotemporal relationship of traffic flow, effectively improving the ability to capture the temporal relationship of traffic flow; utilizes a segmentation alignment layer to combine the spatiotemporal segmentation of traffic flow that describes temporal semantic information with node-dimensional segmentation with prompt text segmentation alignment; finally, the multi-level embedding input that integrates historical features, dynamic spatiotemporal features, and prompt text features is input into a pre-trained large language model, and the downstream prediction task is completed by training the pre-trained large language model. During model training, the natural language understanding ability and domain knowledge of the large language model are retained, while the ability to capture spatiotemporal dependencies is enhanced, prediction accuracy is improved, and trainable parameters are reduced. The method of the present invention can better integrate natural language text related to traffic prediction, and has high prediction accuracy and low computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of a multi-level embedded spatiotemporal prediction method for traffic flow based on a pre-trained large language model according to an embodiment of the present invention.

[0040] Figure 2 Schematic diagram of the overall structure of the traffic flow spatiotemporal prediction model according to an embodiment of the present invention.

[0041] Figure 3 This is a model structure diagram of a multi-layer time coding network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be described in detail below based on the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more clear. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] like Figure 1 and 2 As shown, the present invention provides a multi-level embedded traffic flow spatiotemporal prediction method based on a pre-trained large language model, comprising the following steps:

[0044] Step 1: Obtain historical traffic flow data within a predetermined range and preprocess it.

[0045] Obtain traffic flow data collected in the real world, including traffic flow data collected by multiple sensors in a predetermined road section within a certain period of time, the date and specific time of the collection day; use the Z-Score normalization method to normalize the traffic flow data; split the data according to the traffic flow prediction task to obtain the historical traffic flow data and predicted traffic flow data with the required step size.

[0046] Step 2: Build and train a spatiotemporal traffic flow prediction model.

[0047] The traffic flow spatiotemporal prediction model includes a spatiotemporal embedding layer, a multi-layer time encoding network, a word segmentation alignment layer, a pre-trained large language model and a traffic flow decoding layer.

[0048] 1. Spatiotemporal Embedding Layer

[0049] The spatiotemporal embedding layer is used to generate spatiotemporal embedding input of traffic flow based on the historical characteristics, temporal dependencies and spatial dependencies of the preprocessed historical traffic flow data. Figure 2 As shown, specifically including:

[0050] use represents the traffic flow data of N nodes in the road network at time t, where C represents the dimension of traffic flow characteristics, using Represents the historical traffic flow data of all nodes, where T represents the time step of the historical traffic flow data; After a full connection conversion where d f Embed dimensions for historical traffic flows;

[0051] Define a trainable spatially adaptive embedding for each node in the traffic flow and expand it in the time dimension to obtain a spatially adaptive embedding where d s Represents the spatially adaptive embedding dimension.

[0052] Define two trainable time period embedding dictionaries for each time step of the traffic flow, one for the timestamp embedding dictionary within a day Where N d Indicates the average number of time steps in a day, d t represents the time period embedding dimension; the other is the embedding dictionary of the week within a week Where N w Represents the day of the week; according to the time slice of historical traffic flow data, the corresponding timestamp embedding and week embedding are retrieved and expanded in the spatial dimension to obtain two time period embeddings and

[0053] E data 、E s 、E day 、E week Splice and get the traffic flow spatiotemporal embedding input X ST =E data ∥E s ∥E day ∥E week , where ∥ represents concatenation operation and the embedding dimension d = d f +d s +2d t .

[0054] 2. Multi-layer time coding network

[0055] Each layer of the multi-layer time coding network is a time coding network based on the self-attention mechanism. The input of the multi-layer time coding network is the spatiotemporal embedding input of the traffic flow. The multi-layer time coding network uses the self-attention mechanism to capture the time dependency between all time slices of different traffic nodes. For each layer of the time coding network based on the self-attention mechanism, such as Figure 3 As shown, specifically including:

[0056] (1) By embedding X ST Calculate the corresponding query Q matrix, key K matrix, and value V matrix: Then calculate the self-attention scores representing the time dependencies of different nodes Finally, calculate the output X of the self-attention code t =A t V t ;

[0057] The specific calculation formula is as follows:

[0058]

[0059] in, All are learnable parameters;

[0060]

[0061] (2) Self-attention encoding is applied to multiple heads, and after layer normalization and residual connection, it passes through a fully connected feedforward network, and then uses layer normalization and residual connection again to obtain the output of a single-layer temporal encoding network based on the self-attention mechanism.

[0062] After multiple layers of the above-mentioned time coding network based on the self-attention mechanism, the output of the final multi-layer time coding network is obtained

[0063] 3. Word segmentation alignment layer

[0064] The word segmentation alignment layer converts the output of the multi-layer time encoding network into traffic flow node tokens through full connection, and concatenates the traffic flow node tokens and the prompt text tokens related to traffic flow prediction, and then uses the concatenation as the input of the pre-trained large language model. Figure 2 As shown, specifically including:

[0065] In order to convert spatiotemporal data into a representation that can be understood by a large language model and merged with text prompts, each node in the output of the multi-layer temporal encoding network is regarded as a token, that is, X is converted into encoder Convert to tokens input for large language model T q =MLP(X encoder ), where D represents the dimension of the large language model token, and each token represents the historical traffic flow information of a node.

[0066] In order to enable the large language model to better complete the traffic flow prediction task and integrate the traffic flow information of the text modality, a prompt text related to the traffic flow dataset, weather, traffic accidents, etc. is designed; the prompt text is frozen through the text segmentation of the pre-trained large language model to obtain the prompt text tokens input T p .

[0067] T p With T q Align and concatenate to obtain the input T of the pre-trained large language model LLM =T p ∥T q , where ∥ represents the concatenation operation.

[0068] 4. Pre-training large language models

[0069] like Figure 2 As shown, the token input T of the pre-trained large language model LLM After multi-layer self-attention encoding of the large language model, the corresponding traffic flow prediction output T can be obtained. output =LLM(T LLM); Among them, a partial fine-tuning strategy is adopted for the training method of the pre-trained large language model: in order to retain the natural language understanding ability and domain knowledge of the large language model, its feedforward network and multi-head self-attention module are frozen; in order to enhance the ability of the large language model to capture the spatiotemporal dependencies of traffic flow and adapt to downstream prediction tasks, its position encoding module and layer normalization module are trained.

[0070] 5. Traffic flow decoding layer

[0071] The traffic flow decoding layer is used to convert the output of the pre-trained large language model into the predicted dimensions of the traffic flow at each traffic node and output the predicted traffic flow, including:

[0072] like Figure 2 As shown in the figure, in order to obtain the traffic flow prediction results of the required dimension, the output of the traffic flow node token position of the large language model is decoded through a fully connected layer to obtain the traffic flow results predicted by all nodes: in Represents the last N tokens output by the large model, representing the corresponding output traffic flow node tokens; then, Y is denormalized by Z-Score to obtain the predicted traffic flow output of the traffic flow spatiotemporal prediction model Where T′ and C′ represent the predicted traffic flow time step and dimension.

[0073] When training the traffic flow spatiotemporal prediction model, the loss function used is the Huber loss function, and the calculation formula is as follows:

[0074]

[0075] Among them, Y ′ is the traffic flow output predicted by the model, is the real traffic flow output, δ is a hyperparameter of error loss; the model parameters are continuously optimized through back propagation to obtain an optimal model.

[0076] Finally, the trained traffic flow spatiotemporal prediction model is used to perform traffic flow spatiotemporal prediction.

[0077] On the other hand, the present invention also provides a multi-level embedded traffic flow spatiotemporal prediction device based on a pre-trained large language model, including one or more processors, for implementing the multi-level embedded traffic flow spatiotemporal prediction method based on a pre-trained large language model in the above-mentioned embodiment. The embodiment of the multi-level embedded traffic flow spatiotemporal prediction device based on a pre-trained large language model of the present invention can be applied to any device with data processing capabilities, and the any device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located to read the corresponding computer program instructions in the non-volatile memory into the memory and run it. From the hardware level, in addition to the processor, memory, network interface, and non-volatile memory, any device with data processing capabilities in which the multi-level embedded traffic flow spatiotemporal prediction device based on a pre-trained large language model is located, usually according to the actual function of the device with data processing capabilities, It can also include other hardware, which will not be repeated here.

[0078] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the multi-level embedded spatiotemporal traffic flow prediction method based on a pre-trained large language model in the above embodiment is implemented.

[0079] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0080] The effectiveness of the present invention is verified in combination with a real-world spatiotemporal traffic flow dataset PeMS08. The PeMS08 dataset contains traffic data collected from July to August 2016 by 170 sensors on major roads in California at a frequency of 5 minutes. The selected comparison methods include SVR (Support Vector Regression), STGCN (Spatiotemporal Graph Convolutional Network), GMAN (Graph Multi-Head Attention Network) and the method of the embodiment of the present invention (a multi-level embedded spatiotemporal traffic flow prediction method based on a pre-trained large language model).

[0081] To make a fair comparison with the contrasting methods, we divide the training set, validation set, and test set into a 6:2:2 ratio as in the existing studies, and use the historical traffic flow of 12 time steps to predict the future 12 time steps.

[0082] For the prediction task of the traffic flow dataset PeMS08, a basic prompt text case can be designed, including two parts: traffic domain knowledge and task description: "Domain description: This dataset contains traffic flow values ​​on California highways, and the traffic flow values ​​are usually higher during the day. Task description: Your task is to predict the traffic flow value in the future <1 hour> at <18:00:00> on <Sunday>, according to the following <170> The traffic value of the node <1 hour> before:". The content in <> changes according to the traffic flow dataset and the specific prediction task.

[0083] The obtained spatiotemporal prediction performance results are shown in Table 1.

[0084] Table 1 Prediction performance results of various comparison methods on the PEMS08 dataset

[0085]

[0086] It can be seen from Table 1 that the multi-level embedded traffic flow spatiotemporal prediction method based on the pre-trained large language model proposed in the embodiment of the present invention is significantly better than the comparison method in terms of prediction performance.

[0087] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.

Claims

1. A multi-level embedded spatiotemporal traffic flow prediction method based on a pre-trained large language model, characterized in that: The method comprises the following steps: S1: Obtain historical traffic flow data within a predetermined range and pre-process it; S2: Construct and train a spatiotemporal prediction model for traffic flow; the spatiotemporal prediction model for traffic flow includes a spatiotemporal embedding layer, a multi-layer time coding network, a word segmentation alignment layer, a pre-trained large language model and a traffic flow decoding layer; the spatiotemporal embedding layer is used to generate a spatiotemporal embedding input for traffic flow based on the historical characteristics, time dependencies and spatial dependencies of the pre-processed historical traffic flow data; each layer of the multi-layer time coding network is a time coding network based on a self-attention mechanism, and the input of the multi-layer time coding network is the spatiotemporal embedding input for traffic flow, and the multi-layer time coding network uses a self-attention mechanism to capture the time dependencies between all time slices of different traffic nodes; the word segmentation alignment layer converts the output of the multi-layer time coding network into traffic flow node tokens through full connection, and concatenates the traffic flow node tokens with the prompt text tokens related to the traffic flow data set to obtain the input of the pre-trained large language model; the traffic flow decoding layer is used to convert the output of the pre-trained large language model into the prediction dimension of the traffic flow of each traffic node, and output the predicted traffic flow; S3: Use the trained traffic flow spatiotemporal prediction model to perform traffic flow spatiotemporal prediction.

2. The multi-level embedded spatiotemporal traffic flow prediction method based on a pre-trained large language model according to claim 1 is characterized in that: The S1 specifically includes the following sub-steps: Obtaining traffic flow data collected in the real world, including traffic flow data collected by multiple sensors in a predetermined road section within a certain period of time, the date and specific time of the collection day; The Z-Score normalization method is used to normalize the traffic flow data; The traffic flow data is segmented according to the traffic flow prediction task to obtain the historical traffic flow data of the required step length.

3. The multi-level embedded traffic flow spatiotemporal prediction method based on a pre-trained large language model according to claim 1 is characterized in that: In S2, the spatiotemporal embedding layer is used to generate a spatiotemporal embedding input of traffic flow based on the historical characteristics, temporal dependency and spatial dependency of the preprocessed historical traffic flow data, which is specifically achieved through the following sub-steps: (1) The historical traffic flow data of all traffic nodes after preprocessing After full connection, it is converted to in, represents the traffic flow data of N traffic nodes in the road network at time t, C represents the dimension of traffic flow characteristics, T represents the time step of historical traffic flow data, d f Embed dimensions for historical traffic flows; Define a trainable spatially adaptive embedding for each node in the traffic flow and expand it in the time dimension to obtain a spatially adaptive embedding where d s Represents spatially adaptive embedding dimensions; Define two trainable time period embedding dictionaries for each time step of the traffic flow, one for the timestamp embedding dictionary within a day Where N d Indicates the average number of time steps in a day, d t represents the time period embedding dimension; the other is the embedding dictionary of the week within a week Where N w Represents the day of the week; according to the time slice of historical traffic flow data, the corresponding timestamp embedding and week embedding are retrieved and expanded in the spatial dimension to obtain two time period embeddings and (2) E data 、E s 、E day 、E week Splice and get the traffic flow spatiotemporal embedding input X ST =E data ∥E s ∥E day ∥E week , where ∥ represents concatenation operation and the embedding dimension d = d f +d s +2d t .

4. The multi-level embedded spatiotemporal traffic flow prediction method based on a pre-trained large language model according to claim 3 is characterized in that: In S2, the multi-layer time coding network uses a self-attention mechanism to capture the time dependency between all time slices of different traffic nodes. For each layer of the time coding network based on the self-attention mechanism, it specifically includes: (1) Calculate the query Q matrix, key K matrix, and value V matrix in sequence, and then calculate the self-attention score representing the temporal dependency of different nodes Finally, calculate the output X of the self-attention code t =A t V t ; The specific calculation formula is as follows: in, All are learnable parameters; (2) Self-attention encoding is applied to multiple heads, and after layer normalization and residual connection, it passes through a fully connected feedforward network, and then uses layer normalization and residual connection again to obtain the output of a single-layer temporal encoding network based on the self-attention mechanism.

5. The multi-level embedded spatiotemporal traffic flow prediction method based on a pre-trained large language model according to claim 4 is characterized in that: In S2, the word segmentation alignment layer converts the output of the multi-layer time coding network into traffic flow node tokens through full connection, and concatenates the traffic flow node tokens and the prompt text tokens related to traffic flow prediction to obtain the input of the pre-trained large language model, which specifically includes: The output of the multi-layer time encoding network is fully connected Convert to the tokens input of the pre-trained large language model T q =MLP(X encoder ), where D represents the dimension of the large language model token, and each token represents the historical traffic flow information of a node; The traffic flow prompt text including traffic flow dataset, weather, and traffic accidents is passed through the text segmenter of the frozen pre-trained large language model to obtain the prompt text tokens input T p ; T p With T q Align and concatenate to obtain the input T of the pre-trained large language model LLM =T p ∥T q , where ∥ represents the concatenation operation.

6. The multi-level embedded spatiotemporal traffic flow prediction method based on a pre-trained large language model according to claim 5 is characterized in that: When training the traffic flow spatiotemporal prediction model, the feedforward network and the multi-head self-attention module of the pre-trained large language model are frozen, and only the position encoding module, the residual and layer normalization module of the pre-trained large language model are trained.

7. The multi-level embedded spatiotemporal traffic flow prediction method based on a pre-trained large language model according to claim 6 is characterized in that: The traffic flow decoding layer is used to convert the output of the pre-trained large language model into the predicted dimension of the traffic flow of each traffic node and output the predicted traffic flow, specifically including: The output of the traffic flow node token position of the pre-trained large language model is decoded through a fully connected layer to obtain the traffic flow results predicted by all nodes: in Indicates the last N tokens output by the large model, representing the corresponding output traffic flow node tokens; Denormalize Y by Z-Score to obtain the predicted traffic flow output of the traffic flow spatiotemporal prediction model Where T′ and C′ represent the predicted traffic flow time step and dimension.

8. The multi-level embedded spatiotemporal traffic flow prediction method based on a pre-trained large language model according to claim 7 is characterized in that: The loss function when training the traffic flow spatiotemporal prediction model is the Huber loss function.

9. A multi-level embedded traffic flow spatiotemporal prediction device based on a pre-trained large language model, characterized in that: The method comprises one or more processors for implementing the multi-level embedded spatiotemporal traffic flow prediction method based on a pre-trained large language model as described in any one of claims 1 to 8.

10. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the electronic device, enables the electronic device to implement the multi-level embedded traffic flow spatiotemporal prediction method based on a pre-trained large language model as described in any one of claims 1 to 8.

Images