A training method, system, and device for a large language model

Abstract

This invention belongs to the field of artificial intelligence technology and discloses a training method, system, and device for a large language model. The invention includes receiving multivariate traffic flow time series and converting them into fixed-dimensional traffic feature tensors; concatenating a learnable soft cue tensor with the traffic feature tensor along the sequence dimension to form an LLM input sequence and inputting it into the LLM to generate a high-dimensional output; the LLM introduces only a small number of trainable parameters; calculating the main task loss based on the high-dimensional output and the corresponding target future traffic flow time series; calculating the cross-modal auxiliary loss based on the soft cue tensor and the traffic feature tensor; and then constructing a composite loss function and training the LLM based on this function to optimize the trainable parameters. This invention can dynamically and adaptively generate or adjust guidance signals according to the unique characteristics of each input traffic flow sequence to the large oracle model, improving the accuracy and robustness of traffic flow prediction using large language models.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a training method, system, and device for a large language model. Background Technology

[0002] Traffic flow prediction is a core task in analyzing and mining the time dependencies in urban traffic data, and it has wide applications in urban traffic management, intelligent route planning, and traffic light timing optimization. The technological development in this field has gone through several stages. Traditional statistical models, such as ARIMA (Autoregressive Integrated Moving Average), while theoretically sound, suffer from limitations in handling the highly nonlinear and non-stationary traffic flow data in the real world due to their linearity and stationarity assumptions. Subsequently, deep learning models, represented by Recurrent Neural Networks (RNNs) and Long Short-Term Memory Networks (LSTMs), have made significant progress in sequence modeling, but they still face efficiency and gradient issues when capturing complex long-distance spatiotemporal dependencies in traffic networks.

[0003] With the rise of Large Language Models (LLMs), represented by the GPT series of models, a promising frontier direction is to apply these powerful inference engines to the field of traffic flow prediction. The core motivation lies in leveraging the powerful contextual understanding, logical reasoning, pattern extrapolation, and even common-sense judgment capabilities learned by LLMs from massive text corpora, and transferring these advanced cognitive abilities to the numerical traffic flow domain, aiming to discover deeper patterns and causal relationships in congestion evolution that traditional numerical models cannot capture.

[0004] However, applying LLM to the specific cross-modal task of traffic flow prediction faces more severe challenges than with other modalities (such as images). The internal knowledge structure of LLM is built around natural language symbols; while traffic flow is a continuous, semantically undefined numerical stream, lacking the stable spatial structure of images, and often accompanied by non-stationary switching between peak and off-peak hours, multi-periodic tidal characteristics, and complex dynamic characteristics such as high noise caused by sudden events like traffic accidents. This makes the "modal gap" problem of establishing an effective bridge between numerical data and linguistic symbols particularly prominent. How to effectively "translate" purely numerical traffic flow data into inputs that LLM can deeply understand and reason about is the key bottleneck in unleashing the cross-modal prediction potential of LLM.

[0005] To address this challenge, Parameter Efficient Fine-Tuning (PEFT) techniques, particularly Prompt Tuning, have been introduced as a promising solution for LLM training. This technique guides model behavior by adding learnable "soft cue" vectors before the LLM input. However, existing prompt tuning methods designed for natural language tasks reveal fundamental flaws when directly applied to the unique cross-modal scenario of traffic flow:

[0006] 1. Staticity and Lack of Adaptability of Messages: Traditional soft cue vectors are fixed after training a large language model. This means that for all different traffic flow samples in a dataset—whether it's congested data during morning rush hour, sparse data at night, or abnormal data during holidays—the trained large language model uses the same soft cue for guidance. This "one-size-fits-all" static cue cannot adaptively adjust according to the unique dynamic characteristics of each traffic scenario instance, thus limiting the prediction accuracy and generalization ability of the trained large language model.

[0007] 2. Lack of Correlation Between Cues and Temporal Features: Traditional cue fine-tuning is supervised only by the final prediction error. There is no direct, structured correlation between the cue vector and the inherent dynamic characteristics of the input traffic flow—such as congestion spread trends, periodic tidal intensity, and road flow fluctuation levels. This makes soft cues a "black box" disconnected from the input, with a vague and inefficient guiding effect, failing to accurately focus LLM's inference capabilities on the most critical dynamic patterns of the current traffic flow.

[0008] Therefore, there is an urgent need in this field for a new and more powerful training method for large language models, employing cross-modal fine-tuning of the model parameters. This method should not only connect the numerical and linguistic modalities, but more importantly, it should truly unlock the inference potential of LLMs, enabling their guidance mechanisms to possess instance-awareness. This means dynamically and adaptively generating or adjusting guidance signals based on the unique characteristics of each input traffic flow sequence, thereby achieving more accurate and robust traffic flow prediction. Summary of the Invention

[0009] The purpose of this invention is to address the shortcomings of existing technologies by providing a training method, system, and device for large language models. This invention solves the technical problem that when existing cue fine-tuning methods are applied to the training of large language models for traffic flow prediction, soft cues, as static guidance signals, cannot adaptively adjust according to specific traffic conditions (such as peak congestion versus smooth nighttime traffic). It enables guidance signals to dynamically perceive rapidly changing traffic conditions, thereby truly unleashing the powerful potential of LLM in handling complex and variable traffic flow sequence prediction tasks.

[0010] Specifically, the present invention is implemented using the following technical solutions.

[0011] On the one hand, the present invention provides a training method for a large language model, comprising:

[0012] Receive a multivariate traffic flow time series X, which includes at least one of historical traffic volume, vehicle speed, or road occupancy, and convert the multivariate traffic flow time series X into a fixed-dimensional traffic feature tensor X. tensor ;

[0013] The learnable soft cue tensor P is coupled to the traffic feature tensor X in the sequence dimension. tensor The sequences are concatenated to form the input sequence X for the large language model. in The large language model is then input into sequence X. in Input a large language model to generate a high-dimensional output containing predicted traffic state information; the main parameters of the large language model are frozen, and only a few trainable parameters are introduced through efficient parameter fine-tuning.

[0014] Based on the high-dimensional output and the corresponding target future traffic flow time series, the main task loss is calculated; based on the soft cue tensor P and the traffic feature tensor X... tensor Calculate the similarity or dissimilarity measure between the two as a cross-modal auxiliary loss; construct a composite loss function based on the main task loss and the cross-modal auxiliary loss.

[0015] Based on the composite loss function, the large language model is trained using training samples containing the multivariate traffic flow time series X and the target future traffic flow time series, thereby optimizing the trainable parameters.

[0016] Furthermore, the calculation of the main task loss based on the high-dimensional output and the corresponding target future traffic flow time series includes:

[0017] From the high-dimensional output containing predicted traffic state information, discard the portion corresponding to the soft cue tensor, and retain only the portion corresponding to the traffic feature tensor X. tensor The corresponding part: the retained high-dimensional output is decoded through the prediction head network to obtain the predicted value. .

[0018] Furthermore, the soft cue tensor P and the traffic feature tensor X are based on... tensor The similarity or dissimilarity measure between the two is calculated as a cross-modal auxiliary loss, including:

[0019] Traffic feature tensor X is generated through a learnable projection network. tensorMapped to the same space as the soft cue tensor P.

[0020]

[0021] in, This indicates a global pooling operation on the sequence length dimension, X. tensor It is the traffic feature tensor, It is a learnable projection network. Indicates the length of the prompt. Indicates the hidden dimension of the target;

[0022] Calculate the soft cue tensor P and the traffic feature tensor X. tensor Similarity or dissimilarity measures between them.

[0023] Furthermore, the dissimilarity measure between the two is the cosine distance, and the function of the cross-modal auxiliary loss is... The calculation method is as follows:

[0024]

[0025] Where P is a dimension of The soft cue tensor; It is a vectorization operation. It is a vector dot product operation; This represents the L2 norm of a vector.

[0026] Furthermore, the composite loss function for:

[0027]

[0028] in, It is the main task loss function, used to measure prediction accuracy; It is a cross-modal auxiliary loss function used to establish the association between cue and feature; It is determined by the gating network based on the input traffic feature tensor X. tensor The dynamically generated gating weights are calculated as follows:

[0029]

[0030] Among them, X tensor It is the traffic feature tensor of the input gating network; GlobalPool It's a global pooling operation; GateNet It is a gated network; It is an activation function.

[0031] Furthermore, the multivariate traffic flow time series X is converted into a fixed-dimensional traffic feature tensor X. tensor Specifically, it includes:

[0032] Data segmentation: Dividing a multi-source traffic flow time series X of length S into segments of a set length along the time axis. Each data block reshapes continuous traffic flow data into a shape of... tensor, It is the number of data blocks; C is the data batch size; C is the channel dimension of the input multivariate traffic flow time series.

[0033] Data block embedding: All data blocks are flattened and projected onto the target hidden dimension D of the large language model through a block embedding network; the block embedding network is configured to capture local temporal information within each data block.

[0034] Furthermore, the training method for the large language model also includes merging the channel dimension C with the data batch size B to obtain the traffic feature tensor that is finally input to the subsequent modules, and its shape is... .

[0035] Furthermore, the training method for the large language model also includes converting the multivariate traffic flow time series X into a fixed-dimensional traffic feature tensor X. tensor Previously, a reversible instance normalization operation was performed on the multivariate traffic flow time series X; correspondingly, after generating the final prediction result using the trained large language model, the final prediction result was restored to the original numerical scale through the corresponding inverse normalization operation.

[0036] On the other hand, the present invention also provides a training system for a large language model, which implements the above-mentioned training method for the large language model. The training system for the large language model includes a traffic data input representation module, a large language model inference module, a composite loss calculation module, and a model optimizer.

[0037] The traffic data input representation module is configured to: receive a multivariate traffic flow time series X, and convert the multivariate traffic flow time series X into a fixed-dimensional traffic feature tensor X. tensor ;

[0038] The large language model inference module is configured to: intersect the soft cue tensor P with the traffic feature tensor X in the sequence dimension. tensor The sequences are concatenated to form a large language model input sequence, and the large language model input sequence is then input into the large language model to generate a high-dimensional output containing predicted traffic state information.

[0039] The composite loss calculation module is configured to: calculate the main task loss based on the high-dimensional output and the corresponding target future traffic flow time series; calculate the cross-modal auxiliary loss; and construct a composite loss function based on the main task loss and the cross-modal auxiliary loss; the composite loss calculation module includes a cross-modal loss calculation unit and a gating network; the cross-modal loss calculation unit is configured to be based on the soft cue tensor P and the traffic feature tensor X. tensor The similarity or dissimilarity metric between the two is calculated as the cross-modal auxiliary loss; the gating network is configured to calculate the similarity or dissimilarity metric between the two based on the traffic feature tensor X. tensor Dynamically generate gating weights for weighting the cross-modal auxiliary loss. ;

[0040] The model optimizer is configured to optimize and update the trainable parameters in the large language model based on the composite loss function.

[0041] In another aspect, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-mentioned training method for a large language model.

[0042] The beneficial effects of the training method, system, and device for the large language model of the present invention are as follows:

[0043] The training method, system, and device for a large language model of the present invention guide the training of the large language model by constructing a composite loss function. This loss function not only includes the main task loss for measuring prediction accuracy, but also creatively introduces two key mechanisms:

[0044] 1) Cross-modal auxiliary loss term: By calculating the similarity or dissimilarity measure between the learnable soft cue tensor and the traffic feature tensor, a direct and learnable association is established between different modalities, thereby forcing the soft cue to have the ability to perceive instances in traffic scenarios.

[0045] 2) Weights dynamically generated by the gating network based on the traffic flow characteristics of the input large language model: used to adaptively adjust the influence of the aforementioned cross-modal auxiliary loss terms in the total loss.

[0046] Through the above design, this invention innovates the static prompting fine-tuning paradigm into a dynamic, instance-aware adaptive fine-tuning process, thereby more effectively applying the powerful reasoning ability of large language models to complex and ever-changing traffic flow prediction tasks, significantly improving the prediction accuracy and robustness of the trained large language model in response to traffic emergencies.

[0047] Compared with existing technologies, the training method, system, and device for large language models of the present invention have the following advantages:

[0048] 1. Achieves true traffic scene perception and prompting: By introducing an innovative cross-modal auxiliary loss, this invention fundamentally solves the static nature of soft prompts in the training process of traditional large language models. The soft prompts learned by the trained large language model are no longer "one-size-fits-all" general instructions, but rather continuous vector representations "tailor-made" for each input traffic flow sequence, containing its unique dynamic characteristics. For example, the trained large language model can generate distinctly different guidance signals for congested flows during peak hours and smooth flows at night, making the guidance of the large language model more accurate and effective than ever before, thus enabling it to better handle complex and ever-changing real-world traffic conditions.

[0049] 2. An adaptive regularization mechanism is introduced to improve the performance and robustness of the trained large language model: The unique gating network can dynamically and intelligently adjust the strength of regularization (i.e., auxiliary loss) based on the inherent characteristics of the input traffic data (such as congestion level and volatility). For smooth traffic flows with clear patterns, it can reduce constraints, allowing the trained large language model to focus more on the prediction task itself; for traffic flows with complex patterns and including sudden events, it can strengthen constraints and provide stronger structured guidance. This adaptive learning mechanism makes the training process more stable, and ultimately significantly improves the prediction accuracy and robustness of the trained large language model in the face of sudden traffic conditions.

[0050] 3. Unlocking the potential of large language models more deeply: This invention provides a more intelligent and context-aware guidance paradigm for training large language models, thereby more effectively bridging the gap between traffic flow numerical modalities and language modalities. This enables the trained large language model to truly transfer and apply its powerful pattern recognition and logical reasoning capabilities to complex traffic flow prediction tasks, discovering deep abstract patterns such as congestion propagation that are difficult for traditional models to grasp.

[0051] 4. Maintains end-to-end efficiency and practicality: The entire framework is fully differentiable, allowing for unified end-to-end optimization without the need for cumbersome multi-stage training or manual feature engineering. Furthermore, this invention is entirely built upon Parameter Efficient Fine-Tuning (PEFT) technology, requiring only the training of a very small number of additional parameters (soft hints, LoRA parameters, and parameters related to small gated networks). While maintaining low computational and storage costs, it achieves outstanding performance, making it highly suitable for real-time deployment in practical application scenarios such as urban traffic control centers. Attached Figure Description

[0052] Figure 1 This is a system structure block diagram of an embodiment of the present invention.

[0053] Figure 2 This is a schematic diagram of the method flow according to an embodiment of the present invention. Detailed Implementation

[0054] The present invention will now be described in further detail with reference to the embodiments and the accompanying drawings.

[0055] One embodiment of the present invention is a training system for a large language model, such as... Figure 1 As shown, it includes a traffic data input representation module, a large language model inference module, a composite loss calculation module, and a model optimizer. Among them,

[0056] The traffic data input representation module is configured to receive multivariate traffic flow time series as input and convert them into a fixed-dimensional traffic feature tensor suitable for input into a large language model.

[0057] The large language model inference module is configured to: concatenate the learnable soft cue tensor with the traffic feature tensor in the sequence dimension to form the large language model input sequence, and input the large language model input sequence into the large language model to generate a high-dimensional output containing predicted traffic state information.

[0058] The composite loss calculation module is configured to: calculate the main task loss based on the high-dimensional output and the corresponding target future traffic flow time series; calculate the cross-modal auxiliary loss based on the soft cue tensor and the traffic feature tensor; and construct a composite loss function based on the main task loss and the cross-modal auxiliary loss. The composite loss calculation module includes a cross-modal loss calculation unit and a gating network. The cross-modal loss calculation unit is configured to calculate a similarity or dissimilarity metric between the soft cue tensor and the traffic feature tensor, which serves as the cross-modal auxiliary loss. The gating network is configured to dynamically generate gating weights for weighting the cross-modal auxiliary loss based on the input traffic feature tensor.

[0059] The model optimizer is configured to optimize and update the trainable parameters in the large language model based on the composite loss function.

[0060] Given a multivariate traffic flow time series with a time step of S, collected from historical observations of multiple sensors or road segments. Where C represents the number of monitoring points (e.g., the number of sensors on a road segment), and the data from each monitoring point constitutes one or more channels (i.e., variables), such as traffic flow, average speed, road occupancy, etc. The data from each monitoring point may contain one or more physical quantities, such as traffic flow, average speed, road occupancy, etc.

[0061] This invention aims to train a prediction function. It has learnable parameters. To predict future traffic flow sequence values ​​with a time step of H. In actual training, data is usually organized in mini-batches, with the input being... Where B is the amount of data processed each time during model training, i.e., the data batch size.

[0062] The training method for the large language model of the present invention, such as Figure 2 As shown, it includes the following steps:

[0063] I. Receiving Time Series of Multiple Traffic Flows The multi-dimensional traffic flow time series Includes at least one of historical traffic flow, vehicle speed, or road occupancy, and the multivariate traffic flow time series Transform into a fixed-dimensional traffic feature tensor .

[0064] Preferably, step one is implemented through a block embedding mechanism, specifically including:

[0065] 1-1) Data partitioning:

[0066] Multivariate traffic flow time series with time step S Divide along the time axis into a set length One data block.

[0067]

[0068] in, The length of each data block is s, where s is the preset sliding step size.

[0069] This operation segments continuous traffic flow data into a series of fragments representing short-term traffic patterns, reshaping the continuous traffic flow data into a shape of... tensor, It is the number of data blocks; C is the data batch size; C is the channel dimension of the input multivariate traffic flow time series.

[0070] 1-2) Data block embedding:

[0071] All data blocks are flattened and projected onto the target hidden dimension D of the large language model using a block embedding network. This block embedding network is configured to capture local temporal information within each traffic data block.

[0072] Block embedding networks are not required to be limited to a specific structure. In another embodiment, a block embedding network can be a simple linear projection layer or a multilayer perceptron (MLP).

[0073] Time series of multiple traffic flows Transform into a fixed-dimensional traffic feature tensor The process can be represented as:

[0074]

[0075] in, It is a segmented embedded network; It is a data partitioning operation; It is the input traffic flow time series; The traffic feature tensor, which is the final output of step one, is used as the input for step two. It is the number of data blocks; C is the data batch size; D is the channel dimension of the input multivariate traffic flow time series; and D is the target hidden dimension.

[0076] Preferably, in another embodiment, the channel dimension C and the data batch size B can be combined to obtain the traffic feature tensor that is finally input to the subsequent modules, and its shape is as follows: The purpose of this operation is to adapt to the standard three-dimensional input format (batch size, sequence length, hidden dimension) of large language models, where the original data batch B and channel C are flattened to the new batch dimension, so that each channel (e.g., each road segment sensor) is treated as an independent sequence instance, thereby allowing large language models to process all channel data in parallel.

[0077] Preferably, in another embodiment, to address the potential distribution shifts between different road segments and at different times (such as peak and off-peak hours), the method further includes processing the multi-dimensional traffic flow time series... Transform into a fixed-dimensional traffic feature tensor Previously, the time series of multiple traffic flows Perform a Reversible Instance Normalization (RevIN) operation. This step is particularly effective for traffic data because it independently calibrates the temporal characteristics of each road segment or region, eliminating dimensional differences and allowing the model to focus more on learning universal traffic evolution patterns. Accordingly, after generating the final prediction results using the trained large language model, the final prediction results are restored to the original numerical scale through the corresponding inverse normalization operation.

[0078] II. The learnable soft cue tensor The traffic feature tensor in the sequence dimension The sequences are concatenated to form the input sequence for the large language model, and then input into the large language model to generate a high-dimensional output containing predicted traffic state information. .

[0079] The main parameters of the large language model are frozen, and only a few trainable parameters are introduced through efficient parameter fine-tuning.

[0080] Step two specifically includes:

[0081] 2-1) Sequence Combinations:

[0082] Learnable soft cue tensors (in To indicate the length, (Hidden dimension for target) and traffic feature tensor in sequence dimension The sequences are concatenated to form the input sequence for the large language model. The dimension index of this tensor starts from 0:

[0083]

[0084] in, This indicates concatenation along the sequence dimension (i.e., the dimension with index 1); Indicates batch size. Indicates the number of channels. Indicates the number of data blocks. Indicates the length of the prompt. This indicates the hidden dimension of the target.

[0085] 2-2) LLM reasoning:

[0086] Input the large language model into the sequence The input is fed into a large language model with a Transformer architecture containing only the decoder portion. Preferably, the main parameters of this large language model are frozen, and only a few trainable parameters are introduced through efficient fine-tuning techniques such as Low-Rank Adaptation (LoRA). The backbone network of the large language model consists of L stacked Transformer layers, each containing a multi-head self-attention (MSA) and a feedforward network (FFN). Further, normalization methods such as RMSnorm can be selectively employed to stabilize activation values ​​during neural network training, prevent gradient vanishing or exploding, thereby accelerating model convergence and improving performance. After processing through L layers, a high-dimensional output is obtained. .

[0087] III. Loss Calculation

[0088] Based on the high-dimensional output and the corresponding target future traffic flow time series, the main task loss is calculated; based on the soft cue tensor... With the traffic feature tensor Calculate the similarity or dissimilarity measure between the two as a cross-modal auxiliary loss; based on the main task loss and the cross-modal auxiliary loss, construct a composite loss function. This guides the end-to-end training of the entire model.

[0089] Composite loss function It consists of a main task loss function and a dynamically weighted auxiliary loss function:

[0090]

[0091] in, It is the main task loss function, used to measure prediction accuracy; It is a cross-modal auxiliary loss function used to establish the association between cue and feature; The gating weights are dynamically generated by the gating network based on the input features.

[0092] Among them, the main task loss function Used to measure predicted values Compared with the target value of real future traffic flow The error between them. Main task loss function. The calculation method is as follows:

[0093] From the high-dimensional output containing predicted traffic state information, the portion corresponding to soft cues is discarded, retaining only the portion corresponding to the traffic feature tensor; the retained high-dimensional output is then decoded by a prediction head network to obtain the predicted value. The loss function can be, but is not limited to, mean squared error (MSE) or mean absolute error (MAE):

[0094]

[0095] Cross-modal auxiliary loss function The aim is to establish a soft cue tensor With traffic feature tensor The direct correlation between them enables soft cues to possess instance-aware capabilities. Since the two may reside in different dimensional spaces, the traffic feature tensor first needs to be processed through a learnable projection network (such as an MLP). Mapped to soft cue tensor The same space.

[0096]

[0097] in, This indicates a global pooling operation on the sequence length dimension; It is the traffic feature tensor generated in step one; It is a learnable projection network (such as an MLP). Indicates the length of the prompt. This indicates the hidden dimension of the target.

[0098] Then, the soft cue tensor P and the traffic feature tensor X are calculated. tensor Similarity or dissimilarity measures between them.

[0099] Preferably, in another embodiment, the soft cue tensor P and the traffic feature tensor X tensor The dissimilarity between them is measured by cosine distance, and the cross-modal auxiliary loss function is used. The calculation method is as follows:

[0100]

[0101] in, It is the traffic feature tensor generated in step one; It is a dimension of Learnable soft cue tensors; It is a vectorization operation. It is a vector dot product operation; This represents the L2 norm of a vector.

[0102] Preferably, in another embodiment, in order to achieve adaptive regularization, the present invention introduces a gated network. (e.g., a small MLP), see reference Figure 1 The data stream in the gating network receives traffic feature tensors. As input, the weights of the auxiliary losses are dynamically generated based on the pattern characteristics of the current input traffic flow (such as stationarity, periodicity, or volatility). The gating network dynamically generates weights based on the characteristics of the input traffic feature tensor; this is known as dynamic gating weights. The output will pass through an activation function (such as Sigmoid) to ensure that its value is within a reasonable range.

[0103] Dynamic gating weights The calculation method is as follows:

[0104]

[0105] in, It is the traffic feature tensor input to the gating network; This is a global pooling operation; It is a gated network; It is an activation function.

[0106] IV. Based on the aforementioned composite loss function, utilize the time series containing the aforementioned multivariate traffic flow. The large language model is trained using training samples of the target future traffic flow time series to optimize the trainable parameters.

[0107] During the training of the large language model, the composite loss function is minimized. The backpropagation algorithm and optimizer (such as AdamW) are used to train a large language model, and all trainable parameters in the model are optimized together.

[0108] These trainable parameters include:

[0109] Parameters in block embedding networks, learnable soft cue tensors P, LoRA parameters in large language models, projection networks Parameters and gating networks The parameters in the network, as well as the parameters in the subsequent prediction head network.

[0110] The training method, system, and device for a large language model of the present invention guide the training of the large language model by constructing a composite loss function. This loss function not only includes the main task loss for measuring prediction accuracy, but also creatively introduces two key mechanisms: 1) a cross-modal auxiliary loss term, which establishes a direct, learnable association between different modalities by calculating a similarity or dissimilarity measure between a learnable soft cue tensor and a traffic feature tensor, thereby forcing the soft cue to have instance-aware capabilities in traffic scenarios; 2) a weight dynamically generated by a gating network based on the traffic flow features input to the large language model, used to adaptively adjust the influence of the aforementioned auxiliary loss term in the total loss. Through this design, the present invention innovates the static cue fine-tuning paradigm into a dynamic, instance-aware adaptive fine-tuning process, thereby more effectively applying the powerful reasoning capabilities of the large language model to complex and ever-changing traffic flow prediction tasks, significantly improving the prediction accuracy and robustness of the trained large language model in responding to traffic emergencies.

[0111] In some embodiments, certain aspects of the above-described techniques may be implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly implemented on a non-transitory computer-readable storage medium. The software may include instructions and certain data that, when executed by one or more processors, manipulate one or more processors to perform one or more aspects of the above-described techniques. The non-transitory computer-readable storage medium may include, for example, magnetic or optical disk storage devices, solid-state storage devices such as flash memory, cache, random access memory (RAM), or other non-volatile memory devices. The executable instructions stored on the non-transitory computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats interpreted or otherwise executed by one or more processors.

[0112] Computer-readable storage media can include any storage medium or combination of storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media can include, but are not limited to, optical media (e.g., optical discs (CDs), digital versatile optical discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tapes, or magnetic hard disks), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical systems (MEMS) based storage media. Computer-readable storage media can be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., a magnetic hard disk drive), removably attached to a computing system (e.g., an optical disc or universal serial bus-based (USB) flash memory), or coupled to a computer system via a wired or wireless network (e.g., network-accessible storage (NAS)).

[0113] Please note that not all activities or elements described in the general description above are essential, a particular activity or part of the apparatus may not be essential, and one or more further activities or included elements may be performed in addition to those described. Furthermore, the order in which the activities are listed does not necessarily represent the order in which they are performed. Moreover, these concepts have been described with reference to specific embodiments. However, those skilled in the art will recognize that various modifications and changes can be made without departing from the scope of this disclosure as set forth in the following claims. Therefore, the specification and drawings are to be considered illustrative rather than restrictive, and all such modifications are included within the scope of this disclosure.

[0114] The benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, any benefits, advantages, solutions to problems, and any features that may lead to or make any benefit, advantage, or solution more apparent should not be construed as critical, essential, or essential features of any or all claims. Furthermore, the specific embodiments disclosed above are merely illustrative, as the disclosed subject matter can be modified and implemented in different but equivalent ways that would be apparent to those skilled in the art benefiting from the teachings herein. There is no intention to limit the details of the constructions or designs shown herein other than those described in the claims. Therefore, it is apparent that the specific embodiments disclosed above can be altered or modified, and all such changes are considered to be within the scope of the disclosed subject matter.

Claims

1. A method for training a large language model, characterized in that, Comprise: receiving a multivariate traffic flow time series X, the multivariate traffic flow time series X comprising at least one of historical traffic flow, vehicle speed, or road occupancy, and converting the multivariate traffic flow time series X into a fixed-dimension traffic feature tensor X tensor ; The learnable soft prompt tensor P is concatenated with the traffic feature tensor X in the sequence dimension tensor The concatenation is performed to form a large language model input sequence X in The large language model input sequence X is input into a large language model to generate a high-dimensional output containing predicted traffic state information in The main parameters of the large language model are frozen, and only a few trainable parameters are introduced through parameter efficient fine-tuning calculating a main task loss based on the high-dimensional output and the corresponding target future traffic flow time series; based on the soft prompt tensor P and the traffic feature tensor X tensor , a similarity measure or a dissimilarity measure between the two is calculated as a cross-modal auxiliary loss; based on the main task loss, cross-modal auxiliary loss, a composite loss function is constructed; training the large language model using training samples containing the multivariate traffic flow time series X and the target future traffic flow time series based on the composite loss function, and optimizing the trainable parameters; The composite loss function is: ； wherein, is the main task loss function for measuring prediction accuracy; is the cross-modal auxiliary loss function for establishing the association between the prompt and the feature; is the gating network according to the input traffic feature tensor X tensor The dynamically generated gating weight is calculated as follows: ； wherein X tensor is a traffic feature tensor input to the gating network; GlobalPool is a global pooling operation; GateNet is a gating network; is an activation function.

2. The method of claim 1, wherein the training of the large language model is performed by a plurality of processors. The calculation of the main task loss based on the high-dimensional output and the corresponding target future traffic flow time series comprises: from the high-dimensional output comprising predicted traffic state information, discarding the portion corresponding to the soft hint tensor, only keeping the portion corresponding to the traffic feature tensor X tensor the kept high-dimensional output is decoded through a prediction head network to obtain a prediction value .

3. The method of claim 1, wherein the training of the large language model is performed by a plurality of servers. based on the soft cue tensor P and the traffic feature tensor X tensor comparing the similarity measure or the dissimilarity measure between the two as a cross-modal auxiliary loss traffic feature tensor X is mapped to the same space as the soft cue tensor P by a learnable projection network tensor traffic feature tensor X is mapped to the same space as the soft cue tensor P by a learnable projection network ； wherein, denotes a global pooling operation on the sequence length dimension, X tensor is the traffic feature tensor, is a learnable projection network, denotes the prompt length, denotes the target hidden dimension; computing a similarity measure or dissimilarity measure between the soft cue tensor P and the traffic feature tensor X tensor .

4. The method of claim 3, wherein the method further comprises: The dissimilarity measure between the two is the cosine distance, the function of the cross-modal auxiliary loss The calculation method is as follows: ； where P is the soft cue tensor of dimension ; is a vectorization operation, is a vector dot product operation; denotes the L2 norm of a vector.

5. The method of claim 1, wherein the training of the large language model is performed by a plurality of servers. converting the multi-dimensional traffic flow time series X into a fixed dimension traffic feature tensor X tensor Specifically comprising: Data segmentation: Dividing a multi-source traffic flow time series X of length S into segments of a set length along the time axis. Each data block reshapes continuous traffic flow data into a shape of... tensor, It is the number of data blocks; It is the length of each data block; C is the data batch size; C is the channel dimension of the input multivariate traffic flow time series. Data block embedding: flatten all data blocks and project all data blocks to the target hidden dimension D of the large language model through a block embedding network; the block embedding network is configured to capture local time series information within each data block.

6. The method of claim 5, wherein the method further comprises: Also included is merging the channel dimension C with the data batch size B to get a traffic feature tensor of shape .

7. The method of training a large language model according to any one of claims 1-6, wherein, Also included is converting the multi-dimensional traffic flow time series X into a fixed-dimension traffic feature tensor X tensor Previously, the multi-dimensional traffic flow time series X is subjected to a reversible instance normalization operation; accordingly, after using the large language model after training to generate the final prediction result, the final prediction result is restored to the original numerical scale through the corresponding inverse normalization operation.

8. A training system of a large language model, implementing the training method of the large language model according to any one of claims 1-7, characterized in that, The training system of the large language model comprises a traffic data input representation module, a large language model inference module, a composite loss calculation module, and a model optimizer. The traffic data input representation module is configured to: receive a multivariate traffic flow time series X, and convert the multivariate traffic flow time series X into a fixed-dimension traffic feature tensor X tensor ; The large language model inference module is configured to: concatenate the soft prompt tensor P with the traffic feature tensor X in the sequence dimension tensor The large language model inference module is configured to: concatenate the soft prompt tensor P with the traffic feature tensor X in the sequence dimension The composite loss calculation module is configured to calculate a main task loss based on the high-dimensional output and the corresponding target future traffic flow time series. compute a cross-modal auxiliary loss; and construct a composite loss function based on the main task loss and the cross-modal auxiliary loss; the composite loss computation module comprises a cross-modal loss computation unit and a gating network; the cross-modal loss computation unit is configured to compute a similarity measure or a dissimilarity measure between the soft prompt tensor P and the traffic feature tensor X tensor as the cross-modal auxiliary loss; the gating network is configured to dynamically generate a gating weight for weighting the cross-modal auxiliary loss according to the traffic feature tensor X tensor ;​ The model optimizer is configured to optimize and update the trainable parameters in the large language model based on the composite loss function.

9. A computer program product comprising a computer program, characterized in that: The computer program, when executed by a processor, implements the steps of the training method of the large language model according to any one of claims 1-7.

Images