Traffic planning prediction method based on complex dialogue decomposition and progressive fine-tuning
By employing dialogue decomposition and progressive fine-tuning, the ability of large-scale language models to understand and process complex scenarios in the transportation field has been improved. This solves the problems of inaccurate output and low training efficiency in existing technologies, enabling more efficient traffic planning and management.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- UNIV OF ELECTRONICS SCI & TECH OF CHINA
- Filing Date
- 2026-01-13
- Publication Date
- 2026-06-05
AI Technical Summary
Existing large-scale language models suffer from problems such as insufficient understanding of complex scenarios, inaccurate output, and low training efficiency when applied in the transportation field.
We employ a method based on complex dialogue decomposition and progressive fine-tuning. By constructing an optimized language model for the transportation domain, including dialogue filtering, decomposition, and two-stage QLoRA fine-tuning, we improve the model's ability to understand and process complex scenarios.
It significantly improves the accuracy and efficiency of the model in traffic planning and management tasks, enhances the understanding of complex scenarios, and reduces training costs and time.
Smart Images
Figure CN122152972A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent transportation, specifically involving a traffic planning and prediction method based on complex dialogue decomposition and progressive fine-tuning. Background Technology
[0002] With the acceleration of urbanization and the increasing complexity of transportation systems, traffic congestion, environmental pollution, and inefficient traffic management have become major obstacles to sustainable urban development. These problems not only reduce residents' quality of life but also place enormous pressure on the economy and the environment. Against this backdrop, the concept of smart cities has emerged, aiming to optimize urban infrastructure, improve resource allocation, and enhance the overall efficiency of urban operations through advanced information technology and data-driven methods. Among the various components of a smart city, Intelligent Transportation Systems (ITS) play a crucial role in addressing transportation-related challenges by optimizing traffic flow, reducing congestion, and improving safety through data-driven approaches.
[0003] Traditional deep learning models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), and hybrid convolutional neural networks (HCNNs) have been successfully applied to tasks such as traffic flow prediction, traffic speed estimation, and congestion detection. However, these methods still have limitations when dealing with complex spatiotemporal dependencies and diverse traffic scenarios, especially in capturing global contextual information and long-range dependencies.
[0004] To address this issue, researchers have begun exploring the potential applications of large language models (LLMs) in the transportation domain, such as applying them to specific areas like traffic prediction. While large language models have demonstrated powerful context modeling capabilities in natural language processing (NLP) tasks, their direct application to complex traffic problems can lead to inaccurate or irrelevant outputs. Therefore, the application of LLMs in the transportation domain still faces many challenges, particularly in accurately understanding domain-specific knowledge and generating context-consistent responses when dealing with complex traffic instructions. Summary of the Invention
[0005] To address the aforementioned shortcomings in existing technologies, the traffic planning and prediction method based on complex dialogue decomposition and progressive fine-tuning provided by this invention solves the problems of insufficient understanding of complex scenarios, inaccurate output, and low training efficiency of existing large language models in the field of traffic applications.
[0006] To achieve the aforementioned objectives, the technical solution adopted by this invention is as follows: a traffic planning and prediction method based on complex dialogue decomposition and progressive fine-tuning, comprising the following steps: S1. Construct an optimized language model for the transportation domain. The specific process is as follows: Input the traffic dataset into the dialogue filtering module to extract the original complex dialogue from the traffic dataset; The original complex dialogue is input into the dialogue decomposition module, which decomposes the original complex dialogue into several sub-dialogues. ROUGE-L and BLEU are used to verify the semantic coverage of the sub-dialogues. The sub-dialogues that pass the verification are retained to construct the sub-dialogue dataset. The traffic dataset and sub-dialogue dataset are input into a two-stage progressive fine-tuning module to fine-tune the LLM using QLoRA, thereby constructing an optimized language model for the traffic domain. S2. Input the new traffic dataset into the optimized traffic domain language model and output the traffic planning prediction results.
[0007] Furthermore, in S1, the method for extracting the original complex dialogue from the traffic dataset is as follows: Calculate the complexity score of dialogue samples in the traffic dataset. If the complexity score of a dialogue sample exceeds the complexity threshold, then the dialogue sample is considered as the original complex dialogue. Among them, the calculation of the first i Complexity score of each dialogue sample The specific expression is: In the formula, Indicates the first i The length score of each dialogue sample Indicates the first i The semantic complexity score of each dialogue sample. This represents the task step score for the i-th dialogue sample.
[0008] Furthermore: In S1, the method for verifying the semantic coverage of sub-dialogues using ROUGE-L and BLEU is as follows: The original complex dialogue is decomposed into sub-dialogues and then concatenated to obtain the concatenated sub-dialogue text. ROUGE-L and BLEU are calculated based on the concatenated sub-dialogue text. The semantic coverage of the sub-dialogue is calculated by weighting ROUGE-L and BLEU. If the semantic coverage of the sub-dialogue exceeds the coverage threshold, the decomposed sub-dialogue is considered as a valid sub-dialogue.
[0009] Furthermore: the expression for calculating ROUGE-L is as follows: In the formula, This represents the length of the longest common subsequence. Indicates the length of the sample. This indicates the concatenation of sub-dialogue text. Y This indicates a primitive, complex dialogue.
[0010] Furthermore: the expression for calculating BLEU is as follows: In the formula, BP Indicates a concise punishment. p n express n Metasyntax precision, u n Indicate each n The weights of metagrams, where N represents the total number of metagrams; In the formula, This indicates the sub-dialogue concatenation text that matches the original complex dialogue. n Number of metasyntaxes Represents all of the original complex dialogue n The total number of metagrams, Indicates continuous text n A sequence of words or characters.
[0011] Furthermore: the semantic coverage of the sub-dialogue is calculated using a weighted average of ROUGE-L and BLEU. The specific expression is: .
[0012] Furthermore: In S1, the specific method for fine-tuning the LLM using QLoRA is as follows: The first stage fine-tunes the key components of the LLM using a traffic dataset using QLoRA. The second stage fine-tunes the relevant components of the LLM using a sub-dialogue dataset using QLoRA. The QLoRA adapters fine-tuned in the first and second stages are then combined to construct an optimized language model for the traffic domain.
[0013] Furthermore: During QLoRA fine-tuning, the forward pass of QLoRA fine-tuning is represented as: In the formula, This represents the quantization weight matrix of the LLM. , and Indicates the weights of the low-rank adapter. d and k This represents the input and output dimensions of an LLM. and This represents the quantization constant at different stages of quantization data. This represents the decoding process of the quantization weight matrix. Indicates the output tensor. This represents the input tensor.
[0014] Furthermore: The QLoRA fine-tuning objective is to minimize the QLoRA adapter loss function. In the first stage, the QLoRA adapter loss function is the objective of the QLoRA fine-tuning. The specific expression is: In the formula, This represents system messages within complex instructions. Indicates a query. Indicates the first generation of the response k One token, This indicates the first stage of the QLoRA adapter. This indicates that when the model parameters are and Generating the first model under the model k The conditional probability of each token Represents a traffic dataset. Indicates the first j Generate a response to a complex dialogue. Indicates the first [number] in the generated response k All previous token sequences; The QLoRA adapter loss function for fine-tuning the target in the second stage. The specific expression is: In the formula, Indicates the first i Individual dialogue datasets, Indicates the first i Sub-set j The true response of each sample Indicates the first i Sub-set j System messages for a sample Indicates the first i Sub-set j A query for a single sample, This indicates the second-stage QLoRA adapter. Indicates in model parameters and The probability distribution under the influence of the action.
[0015] Furthermore: The specific method for integrating the QLoRA adapter after the first and second stage QLoRA fine-tuning is as follows: Weighted fusion of QLoRA adapter loss functions in the first and second stages, and the fused multi-QLoRA adapter loss function. The specific expression is: In the formula, and This represents the learnable weights.
[0016] The beneficial effects of this invention are as follows: This invention provides a traffic planning and prediction method based on complex dialogue decomposition and progressive fine-tuning, solving the problems of insufficient understanding of complex scenarios, inaccurate output, and low training efficiency of existing large language models in traffic applications. It significantly improves the model's performance in tasks such as traffic planning, traffic engineering, and traffic management, providing more reliable and efficient language processing support for intelligent transportation systems. Compared with existing technologies, this invention has the following advantages: (1) Improved understanding of complex scenarios: By breaking down complex tasks in the transportation field into multiple simple sub-tasks through the dialogue decomposition module, the large language model can better understand and process complex scenarios, significantly improving the model's ability to solve complex problems in the transportation field. In transportation planning tasks, it can more accurately consider multiple factors and provide more reasonable planning solutions.
[0017] (2) Improved task processing accuracy: The dialogue filtering module is used to accurately select the original complex dialogue from the traffic dataset. The data decomposition module improves the accuracy and reliability of the sub-dialogue dataset through coverage testing. The two-stage progressive fine-tuning strategy enables the model to be effectively optimized for both complex and single tasks, enhancing the model's adaptability to various tasks in the traffic field and thus improving the accuracy of task processing.
[0018] (3) Improved training efficiency: This invention combines QLoRA technology to reduce computational costs in both stages of fine-tuning, enabling efficient optimization training of large language models even with limited hardware resources, thereby improving training efficiency and reducing training time and hardware costs. Attached Figure Description
[0019] Figure 1 This is a flowchart of the traffic planning and prediction method based on complex dialogue decomposition and progressive fine-tuning of the present invention.
[0020] Figure 2 Example image of suggestions for filtering complex dialogues. Detailed Implementation
[0021] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.
[0022] like Figure 1 As shown, in one embodiment of the present invention, the traffic planning and prediction method based on complex dialogue decomposition and progressive fine-tuning includes the following steps: S1. Construct an optimized language model for the transportation domain. The specific process is as follows: Input the traffic dataset into the dialogue filtering module to extract the original complex dialogue from the traffic dataset; The original complex dialogue is input into the dialogue decomposition module, which decomposes the original complex dialogue into several sub-dialogues. ROUGE-L and BLEU are used to verify the semantic coverage of the sub-dialogues. The sub-dialogues that pass the verification are retained to construct the sub-dialogue dataset. The traffic dataset and sub-dialogue dataset are input into a two-stage progressive fine-tuning module to fine-tune the LLM (Large Language Model) using QLoRA, thereby constructing an optimized language model for the traffic domain. S2. Input the new traffic dataset into the optimized traffic domain language model and output the traffic planning prediction results.
[0023] In this embodiment, the present invention designs a complex dialogue filtering mechanism based on multi-dimensional evaluation. The dialogue filtering module selects original complex dialogues with high semantic complexity and logical depth from the traffic dataset to ensure data quality.
[0024] Traffic datasets are stored in a specific format, typically consisting of three fields: instructions, inputs, and outputs. Given that the instructions and output fields are crucial for reflecting dialogue complexity, this embodiment considers them the core objects of complexity assessment. Specifically, quantitative assessment is performed from three dimensions: dialogue length, semantic complexity, and task steps. Dialogue length, which directly measures dialogue complexity from the perspective of character count, accounts for 0.2% of the overall assessment. Semantic complexity primarily assesses the clarity of the dialogue's meaning, the amount of background knowledge required, and the relationships and complexity between concepts and logic; this factor accounts for 0.5% of the overall assessment. Task steps reflect the difficulty of completing the task required to finish the dialogue through the number of steps; more steps indicate higher complexity, and this factor accounts for 0.3% of the overall assessment.
[0025] Based on the above evaluation factors and weights, the specific method for extracting the original complex dialogue from the traffic dataset in S1 is as follows: The complexity score of dialogue samples in the traffic dataset is calculated. If the complexity score of a dialogue sample exceeds the complexity threshold, the dialogue sample is considered as the original complex dialogue. In order to make the evaluation process more intuitive and accurate, this embodiment sets the complexity threshold to 0.7. Among them, the calculation of the first i Complexity score of each dialogue sample The specific expression is: In the formula, Indicates the first i The length score of each dialogue sample Indicates the first i The semantic complexity score of each dialogue sample. This represents the task step score for the i-th dialogue sample.
[0026] In this embodiment, an example diagram of the prompts for filtering the original complex dialogue is shown below. Figure 2 As shown.
[0027] In S1, the method for verifying the semantic coverage of sub-dialogues using ROUGE-L and BLEU is as follows: The original complex dialogue is decomposed into sub-dialogues and then concatenated to obtain the concatenated sub-dialogue text. ROUGE-L and BLEU are calculated based on the concatenated sub-dialogue text. The semantic coverage of the sub-dialogue is calculated by weighting ROUGE-L and BLEU. If the semantic coverage of the sub-dialogue exceeds the coverage threshold, the decomposed sub-dialogue is considered as a valid sub-dialogue.
[0028] In this embodiment, a dialogue decomposition module is used to decompose complex dialogues into multiple sub-dialogues to reduce the learning difficulty of the model. ROUGE-L and BLEU metrics are used to verify the semantic coverage of the sub-dialogues. Through this rigorous verification mechanism, the accuracy and reliability of the sub-dialogue dataset are improved, providing better data support for subsequent LLM fine-tuning.
[0029] In this embodiment, ROUGE-L evaluates the semantic overlap between the sub-dialogue and the original dialogue by calculating the longest common subsequence. The expression for calculating ROUGE-L is as follows: In the formula, This represents the length of the longest common subsequence. Indicates the length of the sample. This indicates the concatenation of sub-dialogue text. Y This indicates a primitive, complex dialogue.
[0030] In this embodiment, BLEU measures lexical and phrase matching between sub-dialogues and the original dialogue using n-gram precision, thereby capturing finer-grained semantic consistency. The expression for calculating BLEU is as follows: In the formula, BP Indicates a concise punishment. p n express n Metasyntax precision, u n Indicate eachn The weights of metagrams, where N represents the total number of metagrams; In the formula, Indicates the concatenation of sub-dialogue text Middle and original complex dialogue Y Matching n Number of metasyntaxes Represents the original complex dialogue Y All n The total number of metagrams, Indicates continuous text n A sequence of words or characters. For example, a 2-gram represents two consecutive words.
[0031] The semantic coverage of sub-dialogues is calculated using ROUGE-L and BLEU weighted averages. The specific expression is: .
[0032] In this embodiment, the dialogue decomposition module utilizes advanced natural language processing models such as GPT-4 to automatically decompose the selected complex dialogues into multiple sub-dialogues with clearly defined single tasks. To ensure decomposition quality, two metrics are used for verification: ROUGE-L (measuring semantic overlap, with a weight of 0.6) and BLEU (measuring lexical matching, with a weight of 0.4). The decomposed sub-dialogue set must achieve a coverage rate of at least 0.75 over the original complex dialogue to ensure that the decomposed sub-dialogues can fully encompass the core content of the original dialogue.
[0033] In S1, the specific method for fine-tuning the LLM using QLoRA is as follows: The first stage fine-tunes the key components of the LLM using a traffic dataset using QLoRA. The second stage fine-tunes the relevant components of the LLM using a sub-dialogue dataset using QLoRA. The QLoRA adapters fine-tuned in the first and second stages are then combined to construct an optimized language model for the traffic domain.
[0034] In this embodiment, to improve efficiency and simplify the process, the present invention designs a two-stage progressive fine-tuning method. The QLoRA adapter is progressively fine-tuned using a traffic dataset and a sub-dialogue dataset, combining the two stages of the LoRA adapter to improve the performance of LLM in the transportation domain.
[0035] In this embodiment, the QLoRA method is used to apply a low-rank adapter to key components of the basic LLM, such as the projection matrices of queries, keys, and values. Specifically, the QLoRA method is similar to LoRA, but it introduces several innovative techniques that make the fine-tuning process more efficient while maintaining model performance and saving resources.
[0036] During QLoRA fine-tuning, for a basic linear layer, the forward pass of QLoRA fine-tuning is represented as: In the formula, This represents the quantization weight matrix of the LLM. , and Indicates the weights of the low-rank adapter. d and k This represents the input and output dimensions of an LLM. and This represents the quantization constant at different stages of quantization data. This represents the decoding process of the quantization weight matrix. Indicates the output tensor. Represents the input tensor, where, BF 16 indicates BF16 precision.
[0037] The goal of QLoRA fine-tuning is to minimize the QLoRA adapter loss function. The QLoRA adapter loss function is the objective of QLoRA fine-tuning in the first stage. The specific expression is: In the formula, This represents system messages within complex instructions. Indicates a query. Indicates the first generation of the response k One token, This indicates the first stage of the QLoRA adapter. This indicates that when the model parameters are and Generating the first model under the model k The conditional probability of each token Represents a traffic dataset. Indicates the first j Generate a response to a complex dialogue. Indicates the first [number] in the generated response k All previous token sequences; In this embodiment, by adjusting the parameters of the QLoRA adapter, the model can be better tuned to handle complex traffic command tasks.
[0038] The QLoRA adapter loss function for fine-tuning the target in the second stage. The specific expression is: In the formula, Indicates the first i Individual dialogue datasets, Indicates the first i Sub-set j The true response of each sample Indicates the first i Sub-set j System messages for a sample Indicates the first i Sub-set j A query for a single sample, This indicates the second-stage QLoRA adapter. Indicates in model parameters and The probability distribution under the influence of the action.
[0039] In this embodiment, similar to the first stage, the QLoRA method is applied to fine-tune the relevant components of the basic LLM. By adjusting the QLoRA adapter, the model will handle each simple task more effectively.
[0040] After completing the first and second phases of fine-tuning, this embodiment merges the two QLoRA adapters into a single multi-QLoRA adapter. The fusion process is performed by a weighted combination of the loss functions of the two adapters. The specific method for fusing the QLoRA adapter after the first and second phases of QLoRA fine-tuning is as follows: Weighted fusion of QLoRA adapter loss functions in the first and second stages, and the fused multi-QLoRA adapter loss function. The specific expression is: In the formula, and This represents the learnable weights.
[0041] In this embodiment, by optimizing the fused adapter weights, the final traffic domain language model can fully leverage the advantages of two-stage fine-tuning and combine it with the ability to handle both complex and simple tasks, thereby generating more efficient and accurate responses. This progressive fine-tuning strategy ensures that the model can better understand and respond to complex traffic instructions.
[0042] In this embodiment, a specific implementation case of the present invention is provided, which performs traffic planning prediction dialogue based on the constructed traffic domain language model, as follows: Instruction(a): I heard that many problems have been encountered in the promotionprocess of urban rail transit PPP projects. What countermeasures do you have? Output(a): We have studied the investment and construction practices of urbanrail transit PPP projects, and combined with the research and analysis of urban railtransit construction cases in various places, we have put forward some countermeasures.These include: • Combining urban rail transit PPP with TOD and using the income fromcom prehensive land development to provide feedback. • Adopting the network - operation separation mode to reduce theinvestment amount requirements for a single investor and give full play to theprofessional strengths of each investor. • Handling each matter on a case - by - case basis during theimplementation process to strengthen support. • Attaching importance to the close combination of urban rail transitplanning and urban planning from the macro to the micro level. • Setting up a reasonable price adjustment mechanism. Instruction(b): Okay, I understand. Then, which mode of transportation’s demandgrowth do you think should be restricted, which mode should be encouraged for use,and which trans portation demand should be prioritized? Also, what is the definitionand measurement of traffic equity, and how do they differ from the perspectives ofmobility and accessi bility? Output(b): Under the concept of traffic equity, we need to judge which mode oftransportation’s demand growth should be restricted, which mode should be encouragedfor use, and which transportation demand should be prioritized based on the publicinterest. These include: • Comprehensively consider the roles and impacts of various transportation modes in urban traffic. • Formulate corresponding traffic plans and policies according to the actual situation ation. • Acknowledge that the understanding and measurement of trafficequity may vary from different perspectives. • Take into account the fact that the travel capabilities and travel demands of individuals or groups with different social classes, incomedifferences, and be havioral abilities are different to ensure vulnerable groups do not bear excessive additional travel costs and more people can enjoy convenient travel services. As can be seen from the above examples, in the context of traffic planning and prediction, the traffic domain language model optimized by this invention can directly handle complex traffic problems, accurately understand the specific knowledge of complex traffic instructions, and generate context-consistent responses, thereby improving the accuracy and efficiency of outputting traffic planning and prediction dialogues.
[0043] In the description of this invention, it should be understood that the terms "center," "thickness," "upper," "lower," "horizontal," "top," "bottom," "inner," "outer," and "radial," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying the relative importance or the number of technical features implicitly specified. Therefore, a feature defined by "first," "second," and "third" may explicitly or implicitly include one or more of that feature.
Claims
1. A traffic planning and forecasting method based on complex dialogue decomposition and progressive fine-tuning, characterized in that, Includes the following steps: S1. Construct an optimized language model for the transportation domain. The specific process is as follows: Input the traffic dataset into the dialogue filtering module to extract the original complex dialogue from the traffic dataset; The original complex dialogue is input into the dialogue decomposition module, which decomposes the original complex dialogue into several sub-dialogues. ROUGE-L and BLEU are used to verify the semantic coverage of the sub-dialogues. The sub-dialogues that pass the verification are retained to construct the sub-dialogue dataset. The traffic dataset and sub-dialogue dataset are input into a two-stage progressive fine-tuning module to fine-tune the LLM using QLoRA, thereby constructing an optimized language model for the traffic domain. S2. Input the new traffic dataset into the optimized traffic domain language model and output the traffic planning prediction results.
2. The traffic planning and prediction method based on complex dialogue decomposition and progressive fine-tuning according to claim 1, characterized in that, In S1, the method for extracting the original complex dialogue from the traffic dataset is as follows: Calculate the complexity score of dialogue samples in the traffic dataset. If the complexity score of a dialogue sample exceeds the complexity threshold, then the dialogue sample is considered as the original complex dialogue. Among them, the calculation of the first i Complexity score of each dialogue sample The specific expression is: In the formula, Indicates the first i The length score of each dialogue sample Indicates the first i The semantic complexity score of each dialogue sample. This represents the task step score for the i-th dialogue sample.
3. The traffic planning and prediction method based on complex dialogue decomposition and progressive fine-tuning according to claim 1, characterized in that, In S1, the method for verifying the semantic coverage of sub-dialogues using ROUGE-L and BLEU is as follows: The original complex dialogue is decomposed into sub-dialogues and then concatenated to obtain the concatenated sub-dialogue text. ROUGE-L and BLEU are calculated based on the concatenated sub-dialogue text. The semantic coverage of the sub-dialogue is calculated by weighting ROUGE-L and BLEU. If the semantic coverage of the sub-dialogue exceeds the coverage threshold, the decomposed sub-dialogue is considered as a valid sub-dialogue.
4. The traffic planning and prediction method based on complex dialogue decomposition and progressive fine-tuning according to claim 3, characterized in that, The expression for calculating ROUGE-L is as follows: In the formula, This represents the length of the longest common subsequence. Indicates the length of the sample. This indicates the concatenation of sub-dialogue text. Y This indicates a primitive, complex dialogue.
5. The traffic planning and prediction method based on complex dialogue decomposition and progressive fine-tuning according to claim 4, characterized in that, The expression for calculating BLEU is as follows: In the formula, BP Indicates a concise punishment. p n express n Metasyntax precision, u n Indicate each n The weights of metagrams, where N represents the total number of metagrams; In the formula, This indicates the sub-dialogue concatenation text that matches the original complex dialogue. n Number of metasyntaxes Represents all of the original complex dialogue n The total number of metagrams, Indicates continuous text n A sequence of words or characters.
6. The traffic planning and prediction method based on complex dialogue decomposition and progressive fine-tuning according to claim 5, characterized in that, The semantic coverage of sub-dialogues is calculated using ROUGE-L and BLEU weighted averages. The specific expression is: 。 7. The traffic planning and prediction method based on complex dialogue decomposition and progressive fine-tuning according to claim 1, characterized in that, In S1, the specific method for fine-tuning the LLM using QLoRA is as follows: The first stage fine-tunes the key components of the LLM using a traffic dataset using QLoRA. The second stage fine-tunes the relevant components of the LLM using a sub-dialogue dataset using QLoRA. The QLoRA adapters fine-tuned in the first and second stages are then combined to construct an optimized language model for the traffic domain.
8. The traffic planning and prediction method based on complex dialogue decomposition and progressive fine-tuning according to claim 7, characterized in that, During QLoRA fine-tuning, the forward pass of QLoRA fine-tuning is represented as: In the formula, This represents the quantization weight matrix of the LLM. , and Indicates the weights of the low-rank adapter. d and k This represents the input and output dimensions of an LLM. and This represents the quantization constant at different stages of quantization data. This represents the decoding process of the quantization weight matrix. Indicates the output tensor. This represents the input tensor.
9. The traffic planning and prediction method based on complex dialogue decomposition and progressive fine-tuning according to claim 8, characterized in that, The goal of QLoRA fine-tuning is to minimize the QLoRA adapter loss function. The QLoRA adapter loss function is the objective of QLoRA fine-tuning in the first stage. The specific expression is: In the formula, This represents system messages within complex instructions. Indicates a query. Indicates the first generation of the response k One token, This indicates the first stage of the QLoRA adapter. This indicates that when the model parameters are and Generating the first model under the model k The conditional probability of each token Represents a traffic dataset. Indicates the first j Generate a response to a complex dialogue. Indicates the first [number] in the generated response k All previous token sequences; The QLoRA adapter loss function for fine-tuning the target in the second stage. The specific expression is: In the formula, Indicates the first i Individual dialogue datasets, Indicates the first i Subset j The true response of each sample Indicates the first i Subset j System messages for a sample Indicates the first i Subset j A query for a sample, This indicates the second-stage QLoRA adapter. Indicates in model parameters and The probability distribution under the influence of the action.
10. The traffic planning and prediction method based on complex dialogue decomposition and progressive fine-tuning according to claim 9, characterized in that, The specific method for integrating the QLoRA adapter after the first and second stage QLoRA fine-tuning is as follows: Weighted fusion of QLoRA adapter loss functions in the first and second stages, and the fused multi-QLoRA adapter loss function. The specific expression is: In the formula, and This represents the learnable weights.