Traffic flow prediction-oriented vertical domain multi-modal large model training method and prediction system

Abstract

The invention provides a traffic flow prediction-oriented vertical domain multi-modal large model training method and prediction system, and the method comprises the steps: obtaining historical flow time sequence data, historical meteorological data, and historical POI data, performing preprocessing and time alignment on the historical traffic time sequence data, the historical meteorological data and the historical POI data; the method comprises the following steps: acquiring historical news text data, screening out key news event text data which has significant influence on traffic flow based on a language large model, marking the key news event text data based on the language large model, and performing time alignment with historical flow time sequence data, historical meteorological data and historical POI data to form a training data set; and training the multi-modal large model based on the training data set. According to the vertical domain multi-modal large model training method, the modal data are integrated, so that the real-time influence of emergencies or extreme environments on the traffic flow can be effectively captured.

Description

Technical Field

[0001] The present invention relates to the technical field of traffic flow prediction, and in particular to a vertical multi-modal large model training method and a prediction system for traffic flow prediction. Background Art

[0002] Traffic flow forecasting is a key task in intelligent transportation systems (ITS). Its purpose is to predict traffic flow changes in a certain period of time in the future by analyzing historical and real-time data. This work is of great significance for alleviating traffic congestion, improving road safety, optimizing traffic management and improving urban traffic efficiency. With the acceleration of urbanization and the surge in the number of motor vehicles, traffic flow forecasting can not only help traffic managers formulate effective traffic control strategies, but also provide real-time route planning suggestions for navigation systems and travelers, thereby improving the intelligence level and service quality of the overall transportation system.

[0003] The main methods of current traffic forecasting include statistical models, traditional machine learning methods, and deep learning techniques. Statistical models such as ARIMA and VAR have good interpretability, but are limited in dealing with complex spatial-temporal dependencies. Although traditional machine learning methods have made some progress in nonlinear modeling, their reliance on feature engineering and insufficient processing capabilities for spatial dependencies make them challenging to use in practice. With the rise of deep learning technology, models based on recurrent neural networks (RNNs), convolutional neural networks (CNNs), and Transformers have performed well in capturing complex time series features.

[0004] In order to manage urban traffic flow more accurately and efficiently, researchers and engineers seek to adopt more advanced technical methods. Big data and artificial intelligence technologies provide new possibilities for traffic flow prediction and management. By collecting multimodal traffic data, such as traffic data and weather conditions, more comprehensive information can be provided for traffic management.

[0005] However, existing big data and artificial intelligence methods lack accuracy and robustness in dealing with dynamic traffic environment changes. In particular, when faced with the integration and processing of multimodal data (such as real-time weather, POI, news events, and traffic thermal data), traditional methods are particularly weak and difficult to effectively integrate these diverse data sources, resulting in a significant decrease in prediction accuracy when responding to emergencies and weather changes. In addition, when processing text information (such as news and social media content), existing methods usually rely on simple keywords, which cannot capture the deep meaning and contextual associations of the text content, and it is difficult to accurately reflect the impact of events on time series.

[0006] Therefore, there is an urgent need to develop new methods to better integrate and process these multimodal data to improve the accuracy and practicality of traffic flow prediction.

[0007] In order to solve the above problems, people have been seeking an ideal technical solution. Summary of the invention

[0008] Based on this, it is necessary to provide a vertical multimodal large model training method and prediction system for traffic flow prediction in response to the above technical problems.

[0009] In order to achieve the above-mentioned object, the first aspect of the present invention provides a vertical multimodal large model training method for traffic flow prediction, comprising the following steps: Obtain historical traffic time series data, historical meteorological data, and historical POI data, and perform preprocessing and time alignment on the historical traffic time series data, historical meteorological data, and historical POI data; obtain historical news text data, and screen out key news event text data that have a significant impact on traffic flow based on the language big model, mark the key news event text data based on the language big model, and perform time alignment with the historical traffic time series data, historical meteorological data, and historical POI data to form a training data set; Perform vector embedding on the historical traffic time series data in the training data set to obtain the traffic flow time series embedding vector; Perform text vector embedding on the historical key news event text data, historical meteorological data, and historical POI data in the training data set to obtain unstructured text embedding vectors; Align and concatenate the traffic flow time series embedding vector and the unstructured text embedding vector as the input vector; Train a large multimodal model based on the input vector.

[0010] In order to achieve the above object, the second aspect of the present invention provides a traffic flow prediction method of a multi-modal large model, comprising the following steps: A multimodal large model is obtained by training based on the vertical domain multimodal large model training method described in the first aspect; Acquire real-time traffic time series data, real-time meteorological data, and real-time POI data, and perform preprocessing and time alignment on the real-time traffic time series data, real-time meteorological data, and real-time POI data; acquire real-time news text data, and screen out key news event text data that have a significant impact on traffic flow based on the language big model, and mark the key news event text data based on the language big model, and perform time alignment with the real-time traffic time series data, real-time meteorological data, and real-time POI data to form the data to be predicted; Perform vector embedding on the real-time traffic time series data in the predicted data to obtain the traffic flow time series embedding vector; Perform text vector embedding on the real-time key news event text data, real-time meteorological data, and real-time POI data in the forecasted data to obtain unstructured text embedding vectors; The traffic flow time series embedding vector and the unstructured text embedding vector are aligned and spliced, and sent as input vectors to the multimodal large model obtained by the above training to obtain the traffic flow prediction results.

[0011] In order to achieve the above object, the present invention provides a traffic flow prediction system of a multi-modal large model in a third aspect, comprising: The historical data acquisition module is used to obtain historical traffic time series data, historical meteorological data, historical POI data and historical news text data; Traffic data acquisition module, used to obtain traffic time series data in real time; Weather API interface, used to obtain meteorological data based on time and sensor latitude and longitude; News API interface, used to obtain news text data based on location information; Point of interest API interface, used to obtain POI data based on sensor latitude and longitude; Traffic time series data preprocessing module, used to preprocess real-time or historical traffic time series data; A news processing module is used to filter out the text data of key news events that have a significant impact on traffic flow based on the language model, and to mark the text data of key news events based on the language model; Time alignment module, used to time align real-time or historical meteorological data, POI data, key news event text data with traffic time series data; The time vector embedding module is used to embed the traffic time series data into vectors to obtain the traffic flow time series embedding vectors; The text vector embedding module is used to embed the aligned key news event text data, meteorological data, and POI data into text vectors to obtain unstructured text embedding vectors; A vector concatenation module is used to align and concatenate the aligned traffic flow time series embedding vectors and the unstructured text embedding vectors as input vectors; A training module is used to train a multimodal large model based on input vectors obtained from historical traffic time series data, historical meteorological data, historical POI data, and historical news text data; The module is used to input the input vectors obtained from real-time traffic time series data, real-time meteorological data, real-time POI data and real-time news text data into the trained multimodal large model to obtain traffic flow prediction results.

[0012] In order to achieve the above-mentioned purpose, the fourth aspect of the present invention provides a computer device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; the processor is used to execute the program stored in the memory to implement the vertical multimodal large model training method described in the first aspect above, or the multimodal large model traffic flow prediction method described in the second aspect.

[0013] In order to achieve the above-mentioned objectives, the fifth aspect of the present invention provides a computer-readable storage medium having instructions stored thereon, and when the computer program is executed by a processor, it implements the vertical multimodal large model training method as described in the first aspect above, or the multimodal large model traffic flow prediction method as described in the second aspect.

[0014] In order to achieve the above-mentioned objectives, the sixth aspect of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the vertical multimodal large model training method as described in the first aspect above, or the traffic flow prediction method of the multimodal large model as described in the second aspect.

[0015] The beneficial effects of the present invention are: (1) The present invention embeds different types of data such as weather, real-time news, and POI into the prediction model, fully considering the nonlinear impact of external events on time series data. Compared with the prior art, the present invention can effectively capture the real-time impact of emergencies or extreme environments on traffic flow by integrating these modal data.

[0016] (2) By leveraging the big model's deep understanding and intelligent screening capabilities of news text data, the potential impact of emergencies, policy changes, social behaviors, etc. on traffic flow is analyzed, effectively avoiding the limitations of traditional technologies that rely on keyword matching or simple statistics.

[0017] (3) This paper proposes an efficient fine-tuning strategy for multimodal large models, which achieves the optimal balance between model performance and computational efficiency through multi-level innovative design: First, MLP is used to convert the multimodal weighted vector to the base model dimension, and the pre-trained knowledge is retained by freezing the first N attention layers; second, a targeted fine-tuning optimization scheme including normalization layer fine-tuning and residual connection weight adjustment is designed to effectively deal with emergencies and distribution changes in traffic flow data; finally, the progressive thawing strategy or LoRA method is innovatively combined to significantly reduce the consumption of computing resources while ensuring the model's rapid adaptability to multimodal inputs. This multi-level optimization scheme not only improves the performance of the model in traffic flow prediction tasks, but also achieves efficient use of computing resources, providing a feasible solution for the deployment of large models in practical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of the vertical domain multimodal large model training method of the present invention; Figure 2 It is a flow chart of the traffic flow prediction method of the multi-modal large model of the present invention; Figure 3 It is a structural block diagram of the traffic flow prediction system of the multi-modal large model of the present invention; Figure 4 is a schematic diagram of the structure of the computer device of the present invention; DETAILED DESCRIPTION

[0019] The technical solution of the present invention is further described in detail below through specific implementation methods.

[0020] Example 1 The first aspect of the present invention provides a vertical multi-modal large model training method for traffic flow prediction, such as Figure 1 As shown, the following steps are included: Step 1: Obtain historical traffic time series data, historical meteorological data, and historical POI data, and preprocess and time-align the historical traffic time series data, historical meteorological data, and historical POI data.

[0021] The meteorological data includes environmental information such as temperature, humidity, wind speed, visibility, rainfall, etc., and the real-time meteorological data is obtained through the meteorological API interface.

[0022] POI (Point of Interest) data refers to a dataset containing information about specific locations, such as shopping malls, restaurants, attractions, etc.

[0023] Specifically, data with different update frequencies (such as half an hour, hour, and daily) are collected from different types of data interfaces (such as traffic flow, weather data, and POI data). These data are standardized to maintain numerical accuracy and are aligned with the same time step to ensure the consistency of the input time series data.

[0024] Obtain historical news text data, filter out key news event text data that have a significant impact on traffic flow based on the language big model, and mark the key news event text data based on the language big model.

[0025] Specifically, news data related to traffic are obtained through news crawling tools. These news are filtered through screening logic to select news events that have direct or potential impact on traffic forecasts, namely key news events. Then the text data of the selected key news events are marked, including the time, location and impact type (short-term or long-term impact) of the event, to ensure alignment with the time series data in the time dimension.

[0026] In specific implementation, an online open source or closed source large model can be used to filter all news events and extract events that are critical to traffic prediction. This method not only improves the accuracy of event screening, but also effectively avoids the interference of irrelevant information, thereby reducing prediction errors. Compared with the traditional simple method of processing text information by relying on keywords, this embodiment uses the deep understanding and intelligent screening capabilities of the large model for text data (such as news and social media content), and analyzes the potential impact of emergencies, policy changes, social behaviors, etc. on traffic flow, effectively avoiding the limitations of traditional technologies that rely on keyword matching or simple statistics.

[0027] Specifically, we use the thinking chain of the big model to filter all news events and extract events that are crucial to traffic prediction. The specific steps are as follows: Role: You are a professional traffic data analyst with expertise in both news analysis and traffic impact assessment.

[0028] Task: Analyze given news and identify events or information that may affect traffic flow.

[0029] Analysis ideas: 1. First determine whether the news involves the following factors: - Large events (concerts, sports events, exhibitions, etc.) - Infrastructure (road construction, bridge maintenance, subway construction, etc.) - Policy changes (traffic restriction policies, toll adjustments, etc.) - Abnormal weather (heavy rain, heavy snow, typhoon, etc.) - Emergencies (traffic accidents, natural disasters, etc.) 2. Then analyze its impact step by step: - Geographical scope of impact → roads and areas affected - Time span of impact → duration and peak hours - Degree of impact → Mild / Moderate / Severe - Size of the population affected → Estimated number of people affected 3. Finally, evaluate the overall impact: - Direct impact: Changes in road capacity - Indirect impact: chain reaction on surrounding roads - Long-term impact: Sustained changes in traffic patterns Output format: { "News Title": "", "Impact Type": ["Large Events" / "Infrastructure" / "Policy Changes" / "Abnormal Weather" / "Emergency"], "Impact Analysis": { "Geographical Scope": "", "Time span": "" "Impact level": "", "Crowd size": "" }, "Traffic Impact Assessment": { "Direct Impact": "", "Indirect impact": "", "Long-term impact": "" }, "Suggested Action": [] } Example analysis: Input news: "XX Stadium will host a famous singer's concert this Saturday, with an estimated audience of 50,000 people" Analysis process: 1. Type judgment: belongs to the large-scale event category 2. Influence chain: Concert (50,000 people) → Central arrival / departure→ Congested surrounding roads → Driving traffic pressure in surrounding areas → Public transport capacity surges 3. Specific impact: Geographical scope: Within 3 km of the stadium Time range: 14:00-23:00 on the day of the concert Peak forecast: 17:00-19:00 for entry and 22:00-23:00 for exit Impact: Severe It should be noted that when using online open source or closed source large models to filter all news events and extract events that are critical to traffic forecasting, it is necessary to give priority to news with clear temporal and spatial information, consider the scale and duration of events, evaluate chain reactions and cumulative effects, and pay attention to seasonal and cyclical effects.

[0030] It can be understood that after a news event that is closely related to traffic flow (such as a major event or road closure) is identified, the news event will be involved in subsequent predictions, thereby improving the rationality and adaptability of the prediction.

[0031] Finally, the text data of key news events are stored together with the historical traffic time series data, historical meteorological data, and historical POI data in the same time range and granularity. Each type of data is stored as an independent file to form a training data set.

[0032] Step 2: Perform vector embedding on the historical traffic time series data in the training data set to obtain the traffic flow time series embedding vector, and perform text vector embedding on the historical key news event text data, historical meteorological data, and historical POI data in the training data set to obtain the unstructured text embedding vector.

[0033] Specifically, when embedding traffic flow time series, the numerical values ​​of the traffic time series are first encoded into discrete digital symbols or continuous numerical vectors, and then the time series is sent to a projector based on a multi-layer perceptron (MLP) to convert the time series into an embedding vector in the text embedding space.

[0034] When performing text vector embedding, you can directly use open source or closed source text embedding models to complete vectorization, and the vector dimension remains consistent with time series embedding.

[0035] Step 3: Align and concatenate the traffic flow time series embedding vector and the unstructured text embedding vector as the input vector.

[0036] Specifically, the traffic flow time series embedding vector and the unstructured text embedding vector are aligned and spliced ​​using timestamps and sensor IDs.

[0037] Specifically, when performing multimodal feature splicing and fusion, an attention weighted fusion method can be used. Specifically, different weights are assigned to different embedding vectors, and the weights are also used as training parameters.

[0038] It should be noted that in practical applications, the attention mechanism may not be used, but the feature concatenation and fully connected layer fusion method may be used.

[0039] Specifically, the steps of the feature splicing and fully connected layer fusion method are as follows: Splicing step: directly splice the feature vectors of different modes (such as time, space, weather, events, etc.) bit by bit to form a high-dimensional vector. After splicing, all modal features are expressed in the same dimensional space, maintaining the independence of each modal feature.

[0040] Fully connected layer fusion: The concatenated high-dimensional vector is input to the fully connected layer, which performs dimensional compression to fuse different modal information into a unified feature representation. In this process, the fully connected layer can adjust the weights of different modalities through linear transformation to achieve the purpose of fusion.

[0041] Although this solution does not have an adaptive weighting mechanism, it can achieve consistency and unified expression of multimodal data through splicing and fully connected layer compression, and is suitable for scenarios with small data volume or limited model complexity.

[0042] Step 4: Train the multimodal large model based on the input vector.

[0043] During the training process of the multimodal large model, in order to reduce the consumption of computing resources and training costs while ensuring the integrity of pre-trained knowledge, this embodiment adopts the strategy of "selective freezing" and "task-specific fine-tuning", which effectively balances the computational efficiency and prediction accuracy of the model.

[0044] It should be noted that the training of the multimodal large model includes two stages: multimodal pre-training and multimodal instruction fine-tuning. This embodiment uses open source autoregressive large models such as Llama and Qwen at home and abroad as the base large model, where the model parameters are selected according to the GPU hardware resources.

[0045] The specific training steps include: An input projector is also provided in front of the large base model. The input projector adopts an MLP projector to convert the multimodal weighted vector obtained in step 3 into the hidden layer dimension of the large base model.

[0046] After multimodal pre-training, the first N self-attention layers of the base large model are further set to a completely frozen state, the remaining layers of the base large model are set to an unfrozen state, the normalization layer is fine-tuned, and the weight ratio of the residual connection is adjusted.

[0047] In the pre-training stage of the multimodal large model, the self-attention layer has captured a lot of extensive knowledge about language and time series data. By setting these layers to a frozen state, we avoid relearning this knowledge during the fine-tuning stage, thus saving computing resources.

[0048] Specifically in this embodiment, the first N self-attention layers of the base model are completely frozen, and only fine-tuned in the unfrozen last layer. This setting ensures that the model can still rely on the learned global feature capture capabilities during the time series and multimodal fusion process, enhancing the model's wide adaptability in prediction tasks. For example, long-term dependencies and event-driven short-term fluctuations in traffic flow data can be retained in the frozen attention layer.

[0049] In the unfrozen layers, the normalization layer is fine-tuned to adapt to the dynamic characteristics of traffic flow data. For example, unexpected events in traffic flow (such as traffic accidents and weather changes) often lead to sudden changes in data distribution. By fine-tuning the normalization layer, the model can be more resilient to the distribution of input data, thereby improving stability in a variable traffic environment.

[0050] In particular, during the fine-tuning process, the weight ratio of the residual connection is also adjusted to adapt to the specific task requirements, ensuring that the information of the previous frozen layer can be better transferred to the unfrozen layer. This adjustment strategy enables the model to be more accurately applied to specific traffic flow prediction while maintaining the original knowledge.

[0051] During the training process, a gradual unfreezing strategy is adopted to dynamically unfreeze the frozen layers according to the training progress of the model, and adjust the learning rate of the unfrozen layers.

[0052] Specifically, in the initial stage of training, the model only unfreezes the last layer to ensure that the model's reliance on previous knowledge is not destroyed when learning new tasks. As training progresses, some intermediate layers can be selectively unfrozen to further refine the learning of specific task features.

[0053] It should be noted that in each stage of unfreezing the layers, the system sets a lower learning rate for the unfrozen layers to control the update amplitude of the parameters. The process of dynamically adjusting the learning rate helps the model to robustly integrate new knowledge while avoiding the destruction of the original model knowledge due to large parameter adjustments. This strategy is particularly suitable for multimodal large model training in traffic flow prediction, which can effectively improve training efficiency, enhance the generalization performance of the model, and avoid the high computational burden of fine-tuning the full model, so that the model can play a more efficient role in specific traffic flow prediction tasks.

[0054] In particular, in addition to the method of freezing some layers, the LoRA (Low-Rank Adaptation) method can also be used. LoRA decomposes certain layer parameters of the model by adding a small amount of low-rank matrices, and only updates the low-rank matrix part, thereby achieving efficient parameter updates while maintaining the large model structure. This method can not only effectively reduce the amount of calculation during the training process, but also allow rapid adaptation to the differences in multimodal inputs while ensuring model performance. The main advantage of LoRA is its scalability, which can be easily fine-tuned on multiple different tasks, and is suitable for the application scenarios of multimodal large models in the present invention.

[0055] After training, the model output is evaluated based on the mean square error and mean absolute error metrics to identify changes in prediction errors and model accuracy.

[0056] It should be noted that during the training, the generalization ability of the model is further ensured by cross-validation. K-fold cross-validation is tested on different data groups to ensure the stability of the model in multiple scenarios. After training, the model is deployed in a real traffic environment for testing, and the prediction effect of the model is verified using actual traffic data. The system compares real-time data with the prediction results to ensure the practicality and accuracy of the model in different cities and scenarios.

[0057] Based on the basic multimodal large model, this embodiment expands the time series input modality, enabling the system to understand and integrate dynamic information from different sources, ensuring the consistency of various modal data in space, time and events, thereby providing more accurate input for complex traffic flow prediction tasks.

[0058] Example 2 The difference between this embodiment and embodiment 1 is that: The present application also provides a multi-modal large model traffic flow prediction method, such as Figure 2 As shown, the specific steps include: A multimodal large model is obtained by training based on the vertical domain multimodal large model training method described in Example 1; Acquire real-time traffic time series data, real-time meteorological data, and real-time POI data, and perform preprocessing and time alignment on the real-time traffic time series data, real-time meteorological data, and real-time POI data; acquire real-time news text data, and screen out key news event text data that have a significant impact on traffic flow based on the language big model, and mark the key news event text data based on the language big model, and perform time alignment with the real-time traffic time series data, real-time meteorological data, and real-time POI data to form the data to be predicted; Perform vector embedding on the real-time traffic time series data in the predicted data to obtain the traffic flow time series embedding vector; Perform text vector embedding on the real-time key news event text data, real-time meteorological data, and real-time POI data in the forecasted data to obtain unstructured text embedding vectors; The traffic flow time series embedding vector and the unstructured text embedding vector are aligned and spliced, and sent as input vectors to the multimodal large model obtained by the above training to obtain the traffic flow prediction results.

[0059] This embodiment innovatively embeds unstructured news text, POI, weather and numerical time series data into a unified way and inputs them into a large model for time series prediction. This method not only retains the world knowledge and context understanding ability of the language model, but also enables the model to combine the dynamic changes of external weather, POI, and events with numerical trends, effectively responding to nonlinear changes driven by external events.

[0060] Specifically, the specific limitations on each step in this embodiment can be found in the limitations on the vertical multimodal large model training method in Example 1, and will not be repeated here.

[0061] Example 3 Based on the same inventive concept, the embodiment of the present application also provides a multimodal large model traffic flow prediction system for implementing the multimodal large model traffic flow prediction method involved above. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the traffic flow prediction system of one or more multimodal large models provided below can be referred to the limitations of the traffic flow prediction method of the multimodal large model above, and will not be repeated here.

[0062] Specifically, Figure 3 As shown, the traffic flow prediction system of the multimodal large model includes: The historical data acquisition module is used to obtain historical traffic time series data, historical meteorological data, historical POI data and historical news text data; Traffic data acquisition module, used to obtain traffic time series data in real time; Weather API interface, used to obtain meteorological data based on time and sensor latitude and longitude; News API interface, used to obtain news text data based on location information; Point of interest API interface, used to obtain POI data based on sensor latitude and longitude; Traffic time series data preprocessing module, used to preprocess real-time or historical traffic time series data; A news processing module is used to filter out the text data of key news events that have a significant impact on traffic flow based on the language model, and to mark the text data of key news events based on the language model; Time alignment module, used to time align real-time or historical meteorological data, POI data, key news event text data with traffic time series data; The time vector embedding module is used to embed the traffic time series data into vectors to obtain the traffic flow time series embedding vectors; The text vector embedding module is used to embed the aligned key news event text data, meteorological data, and POI data into text vectors to obtain unstructured text embedding vectors; A vector concatenation module is used to align and concatenate the aligned traffic flow time series embedding vectors and the unstructured text embedding vectors as input vectors; A training module is used to train a multimodal large model based on input vectors obtained from historical traffic time series data, historical meteorological data, historical POI data, and historical news text data; The module is used to input the input vectors obtained from real-time traffic time series data, real-time meteorological data, real-time POI data and real-time news text data into the trained multimodal large model to obtain traffic flow prediction results.

[0063] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0064] Example 4 This embodiment provides a computer device, which may be a terminal, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, the method described in Example 1 or the method described in Example 2 is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0065] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0066] Example 5 Based on the above embodiments, this embodiment provides a computer-readable storage medium on which instructions are stored. When the computer program is executed by the processor, it implements the vertical multimodal large model training method described in the above-mentioned embodiment 1, or the multimodal large model traffic flow prediction method described in embodiment 2.

[0067] Example 6 Based on the above embodiments, this embodiment provides a computer program product, including a computer program, which, when executed by a processor, implements the vertical multimodal large model training method described in the above-mentioned embodiment 1, or the multimodal large model traffic flow prediction method described in embodiment 2.

[0068] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0069] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0070] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or some technical features can be replaced by equivalents without departing from the spirit of the technical solution of the present invention, which should be included in the scope of the technical solution for protection of the present invention.

Claims

1. A vertical multi-modal large model training method for traffic flow prediction, characterized in that: The following steps are involved: Obtain historical traffic time series data, historical meteorological data, and historical POI data, and perform preprocessing and time alignment on the historical traffic time series data, historical meteorological data, and historical POI data; obtain historical news text data, and screen out key news event text data that have a significant impact on traffic flow based on the language big model, mark the key news event text data based on the language big model, and perform time alignment with the historical traffic time series data, historical meteorological data, and historical POI data to form a training data set; Perform vector embedding on the historical traffic time series data in the training data set to obtain the traffic flow time series embedding vector; Perform text vector embedding on the historical key news event text data, historical meteorological data, and historical POI data in the training data set to obtain unstructured text embedding vectors; Align and concatenate the traffic flow time series embedding vector and the unstructured text embedding vector as the input vector; Train a large multimodal model based on the input vector.

2. According to claim 1, a vertical multi-modal large model training method for traffic flow prediction is characterized in that: Training of multimodal large models based on input vectors, including: The autoregressive model is selected as the base model; An input projector is set in front of the base large model, wherein the input projector adopts an MLP projector to convert the input vector into a hidden layer dimension of the base large model; After multimodal pre-training, the first N self-attention layers of the base model are set to a completely frozen state, the remaining layers of the base model are set to an unfrozen state, the normalization layer is fine-tuned, and the weight ratio of the residual connection is adjusted; During the training process, the frozen layers are dynamically unfrozen according to the training progress of the model, and the learning rate of the unfrozen layers is adjusted; After training, the model output is evaluated based on the mean square error and mean absolute error indicators.

3. A vertical multi-modal large model training method for traffic flow prediction according to claim 1 or 2, characterized in that: Based on the language model, we screen out the text data of key news events that have a significant impact on traffic flow, including: The large model thinking chain is used to filter the news event text data layer by layer to extract the news events that are crucial to the communication volume prediction task.

4. A traffic flow prediction method based on a multi-modal large model, characterized in that: The following steps are involved: A multimodal large model is obtained by training based on the vertical domain multimodal large model training method according to any one of claims 1 to 3; Obtain real-time traffic time series data, real-time meteorological data, and real-time POI data, and pre-process and time-align the real-time traffic time series data, real-time meteorological data, and real-time POI data; Acquire real-time news text data, filter out key news event text data that have a significant impact on traffic flow based on the language big model, mark the key news event text data based on the language big model, and align it with real-time traffic time series data, real-time meteorological data, and real-time POI data to form the data to be predicted; Perform vector embedding on the real-time traffic time series data in the predicted data to obtain the traffic flow time series embedding vector; Perform text vector embedding on the real-time key news event text data, real-time meteorological data, and real-time POI data in the forecasted data to obtain unstructured text embedding vectors; The traffic flow time series embedding vector and the unstructured text embedding vector are aligned and spliced, and sent as input vectors to the multimodal large model obtained by the above training to obtain the traffic flow prediction results.

5. A multi-modal large-scale traffic flow prediction system, characterized in that: include: The historical data acquisition module is used to obtain historical traffic time series data, historical meteorological data, historical POI data and historical news text data; Traffic data acquisition module, used to obtain traffic time series data in real time; Weather API interface, used to obtain meteorological data based on time and sensor latitude and longitude; News API interface, used to obtain news text data based on location information; Point of interest API interface, used to obtain POI data based on sensor latitude and longitude; Traffic time series data preprocessing module, used to preprocess real-time or historical traffic time series data; A news processing module is used to filter out the text data of key news events that have a significant impact on traffic flow based on the language model, and to mark the text data of key news events based on the language model; Time alignment module, used to time align real-time or historical meteorological data, POI data, key news event text data with traffic time series data; The time vector embedding module is used to embed the traffic time series data into vectors to obtain the traffic flow time series embedding vectors; The text vector embedding module is used to embed the aligned key news event text data, meteorological data, and POI data into text vectors to obtain unstructured text embedding vectors; A vector concatenation module is used to align and concatenate the aligned traffic flow time series embedding vectors and the unstructured text embedding vectors as input vectors; A training module is used to train a multimodal large model based on input vectors obtained from historical traffic time series data, historical meteorological data, historical POI data, and historical news text data; The module is used to input the input vectors obtained from real-time traffic time series data, real-time meteorological data, real-time POI data and real-time news text data into the trained multimodal large model to obtain traffic flow prediction results.

6. A computer device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor is used to implement a vertical multimodal large model training method for traffic flow prediction as described in claims 1 to 3, or implement a traffic flow prediction method of a multimodal large model as described in claim 4, when executing a program stored in a memory.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the vertical multimodal large model training method for traffic flow prediction as described in claims 1 to 3, or implements the traffic flow prediction method of the multimodal large model as described in claim 4.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements a vertical multimodal large model training method for traffic flow prediction as described in claims 1 to 3, or implements a traffic flow prediction method of a multimodal large model as described in claim 4.

Images