Human-cabin interaction intention personalized prediction method based on large language model

Abstract

The invention is suitable for the technical field of automobile intelligent cabins, and provides a human cabin interaction intention personalized prediction method based on a large language model, and the method comprises the steps: firstly constructing a cockpit scene data set special for driver intention prediction through combining vehicle state data, driver behavior sequence data and a data enhancement technology; secondly, a novel driver intention personalized prediction model PCockpitch-LLM is designed; and then a two-stage fine tuning strategy is adopted. In the first stage, a generalization data set which widely covers a cockpit scene is used for carrying out all-parameter fine tuning on the PCockpit-LLM. And in the second stage, the model is further finely adjusted through the personalized cockpit scene data set. According to the method, the context understanding ability and the knowledge generalization ability of the large language model are fully utilized, high accuracy and individuation of driver intention prediction are achieved, and theoretical support and technical basis are provided for the individualized active interaction function of an intelligent cabin.

Description

Technical Field

[0001] The present invention belongs to the technical field of automobile intelligent cockpits, and in particular relates to a personalized prediction method for human-cabin interaction intention based on a large language model. Background Art

[0002] The smart cockpit of a car refers to a human-machine-environment fusion space equipped with advanced software and hardware systems, and equipped with functions such as human-machine interaction, network services and scene expansion, providing drivers and passengers with a safe, intelligent, efficient and pleasant comprehensive experience. According to different functional types, the interactive functions of the smart cockpit can be divided into three dimensions: driving control, riding comfort and infotainment. Among them, the driving control function domain focuses on improving the driving quality of smart vehicles, the riding comfort function domain focuses on optimizing the riding experience, and the infotainment function domain provides rich information value-added services and entertainment experience. From the perspective of intelligence, the smart cockpit is divided into five levels from L0 to L4 in the process of transitioning from "traditional function realization" to "comprehensive intelligent cognition". Realizing active interactive execution capabilities is the key to promoting the smart cockpit to high-level intelligence. In recent years, with the rapid development of smart cockpit technology, the "active predictive interactive" cockpit has gradually replaced the traditional "passive response" cockpit. At the same time, the increasing complexity of cockpit functions and the diversification and personalization of user needs have further highlighted the importance of improving the personalized active interaction capabilities of the cockpit, which has become the core goal of realizing high-level smart cockpits.

[0003] The active interaction mechanism of the smart cockpit usually includes the following key steps: first, the intelligent system comprehensively processes and analyzes multimodal data; then, based on the current scenario, it predicts the user's possible functional operation intentions and makes a comprehensive judgment on the composition and priority of related functions; finally, it provides personalized services to users through appropriate interaction methods. In the entire active interaction process, intention prediction, as the core "brain" of the system, undertakes the important task of accurately identifying and predicting user functional needs, and is the basis for realizing the active interaction capability of the cockpit.

[0004] Traditional intention prediction technology mainly focuses on the speculation of driver behavior and needs, focusing on driving safety, comfort adjustment and interaction with infotainment systems. However, these methods are often limited to the prediction of a single task and rely on rule-driven or traditional data-driven models. This leads to the limitation that the model has poor adaptability and cannot cope with changing in-vehicle scenarios when dealing with complex, multi-task interactions. The needs in the cockpit are not only driving-related tasks, but also include seat adjustment, air conditioning control, music playback, navigation settings and other personalized functions. These diversities make it difficult for traditional methods to meet actual needs, especially when facing personalized needs.

[0005] At the same time, personalized intention prediction not only needs to consider the driver's immediate needs, but also needs to combine historical behavior, emotional state, preferences and other factors to accurately infer their behavior patterns in specific situations. This requires the model to have efficient context understanding, personalized reasoning and dynamic adaptability to meet the unique needs of each driver and improve the user experience. Summary of the invention

[0006] The purpose of an embodiment of the present invention is to provide a personalized prediction method for human-cabin interaction intention based on a large language model, aiming to solve the problems raised in the above background technology.

[0007] The embodiment of the present invention is implemented as follows: a method for personalized prediction of human-cabin interaction intention based on a large language model, comprising the following steps: Step 1: Construction of driver interaction scenario dataset; By integrating vehicle status data, driver behavior sequence data, and data enhancement technology, a cockpit scene dataset specifically for driver intention prediction is constructed. Vehicle status data includes real-time dynamic information of the vehicle while driving; driver behavior sequence data includes multi-dimensional data of the interaction between the driver and the vehicle; in addition, data enhancement technology is used to increase the diversity of data, thereby providing a data basis for personalized driver intention prediction; Step 2: Build the driver personalized intention prediction model PCockpit-LLM; The model takes the vehicle's driving status, external environment information and the driver's historical interaction data as input, and combines the driver's behavioral habits and historical preferences to generate future intention predictions with personalized content. The model captures the driver's immediate behavioral characteristics by comprehensively analyzing multimodal input data, and can perform deep learning based on historical data to analyze the personalized needs of each driver, providing highly personalized prediction results for the specific needs of different drivers. Step 3: Training and fine-tuning of the PCockpit-LLM model; A two-stage fine-tuning strategy was adopted, including full-parameter fine-tuning and adapter fine-tuning. In the first stage, a generalized dataset covering cockpit scenarios was used to fine-tune the full parameters of PCockpit-LLM, so that the model had a broad understanding of common cockpit scenarios. In the second stage, the model was further fine-tuned using a personalized cockpit scenario dataset. The personalized dataset contains the interaction information of a specific driver, and the model parameters are optimized in combination with the LoRA method.

[0008] Further technical solution, in step 1, the status information in the data set covers the driving status, cockpit status and body status of the vehicle, and all vehicle machine signals and cockpit signals are derived from the vehicle CAN bus; at the same time, a set of data preprocessing and enhancement tool libraries are designed to preprocess and enhance the original data, and finally output a text format data set that meets the training requirements, specifically: First, the sensor data in the cabin and the environment are extracted and processed to obtain the status information of the equipment in the cabin and the vehicle status data; then, the status function in the tool library is used to and behavior function Extract vehicle status information and information related to driver behavior from sensor data, and perform data enhancement processing on the above data using algorithms that convert state attributes into behavior attributes and vice versa. Finally, combine this information with system prompt information and encapsulate it into text prompts to fully describe the relationship between cockpit and vehicle status and driver behavior. The problem is defined as follows: By integrating multi-attribute cockpit perception information, a text structure with a unified format and centered on the driver is formed. Then a large language model is introduced to infer the driver's next interaction based on the information inside and outside the cabin and the driver's historical behavior sequence; The final text prompt Covering the current moment Vehicle status information and the corresponding n drivers' historical interaction information A collection of: ; The state attribute Including body condition and cockpit status : ; Body condition Including vehicle speed , Lighting status , Door status and the window status : ; Cockpit status Including seat belt status and air conditioning status : ; Air conditioning status Including air conditioning temperature, blowing mode and blowing level; Behavior Sequence The specific contents of the driver's current behavior and the previous n historical behaviors are recorded; In order to enable the model to obtain higher quality prediction task answers, a matching global feature set is designed Text instructions , It contains task objectives, input descriptions, and output instructions. The task objectives are used to guide the processing direction and generation logic of the large language model. The goal of the prompting project is to generate accurate driver behavior prediction labels; the input description is used to briefly explain what specific information will be input to the large language model, that is, the text data A brief introduction to help the model understand The content and internal logic in the output instructions include the instruction constraints on the large language model and the type of results expected to be generated by the model, which includes limiting the model prediction direction, prompting the model reasoning direction to specify the generation format of the model prediction text; By focusing on user interaction behavior, the vehicle state of the current interaction is described, and n event-oriented previous historical interaction behaviors are extracted to form a global feature set. ; After that, by using the text command Combined and packaged together as instruction text , together constitute the input of the driver interaction scenario dataset.

[0009] Further technical solution, in step 2, the constructed PCockpit-LLM model includes three parts: The encoding part of the text data tokenizer, the large language model backbone, and the full parameter fine-tuning of the LLM backbone, the adapter fine-tuning, and the decoding part of the driver behavior intention data; First, the text instructions are converted into a format that can be processed by LLM using a text data tokenizer encoder and input into LLM for processing; second, to ensure that PCockpit-Llama can accurately adapt to the current task, the model is fine-tuned using a full parameter fine-tuning method and an adapter method, and the adapter is added to the LLM backbone; finally, the prediction tags generated by LLM are decoded using a text de-tagger; the predicted driver behavior intention is hierarchically decoded, and during the decoding process, the predicted driver behavior intention is output in text format; (1) Data encoding: PCockpit-LLM input consists of text data and a piece of instruction text used to inspire a large language model constitute; Used to convert all the contents of the constructed multimodal dataset into corresponding word vector representations: ; 2) Fine-tuning of the LLM backbone: Full Parameter Fine-Tuning is a deep learning model training method that adjusts all parameters of a pre-trained model to adapt it to a specific task. In the first stage, the pre-trained model is fully fine-tuned using the driver interaction scenario dataset to ensure that the model has broad general capabilities in the driver intention prediction task, providing a basis for personalized settings for the adapter fine-tuning in the second stage; In the second stage of fine-tuning, a personalized dataset for a specific driver is extracted from the global dataset, and LoRA is used as an adapter to perform lightweight fine-tuning on the LLM backbone. The core idea of ​​the LoRA fine-tuning method is to dynamically adjust the model weights by adding a low-rank matrix to the weight matrix of the pre-trained model. This fine-tuning method allows the performance of the model to be effectively adjusted to adapt to specific tasks with only a small number of additional parameters, while keeping the weight parameters of the pre-trained model unchanged. Llama3-Instruct-8B is used as the backbone network of LLM, and Lora is introduced in each linear layer of Llama3-8B-Instruct; the input token first enters the MHA layer, in which Lora is applied to each attention matrix; the attention matrix includes: query matrix, key matrix and value matrix; Lora itself is a bypass module, which contains a lower projection matrix and an upper projection matrix. During fine-tuning, the input and output dimensions of the MHA layer remain unchanged, and the linear transformation matrix of each attention matrix is ​​frozen. Only the newly added lower projection matrix and upper projection matrix will be trained. These matrices are initialized by random Gaussian distribution or by zero matrix; the calculation process of the entire MHA mechanism is as follows: ; ; in: ; Each attention head , the linear transformation matrix, is the corresponding down-projection matrix, , , is the corresponding upper projection matrix, The transformed matrix representing the output; Then, after a residual connection and layer normalization operation, we get : ; Next, The input passes through the feedforward network layer, which consists of two fully connected layers and an activation function. In these two fully connected layers, LoRA is introduced and the same training method as above is adopted: the weight matrix of the fully connected layer is frozen, and only the lower projection matrix and the upper projection matrix are trained. After a residual connection and layer normalization operation, the output H of the transformation layer is obtained. L : ; H L It will be used as the input loop iteration of the next tramsformer, and finally the intermediate label T is obtained after adding the Lora32 layer transformer L : ; 3) Decoding part: After getting the predicted token text T L After that, the decoder is used to decode it into natural language, that is, the output text of the entire PCockpit-LLM, which represents the driver's personalized behavior predicted by the model; the entire decoding process is expressed as: .

[0010] A further technical solution is that in step 3, in the first stage of full parameter fine-tuning, all layers of the base model are fine-tuned using the global driver interaction dataset; in the second stage of adapter fine-tuning, a specific driver personalized interaction dataset is used to fine-tune all linear layers of the model after full parameter fine-tuning in the first stage, and the LoRA method is used in the fine-tuning process; the cross entropy loss is used in both stages of fine-tuning To supervise the inference process of the model, the weights of the adapter are updated by minimizing the cross entropy loss and the weight of the LLM backbone : .

[0011] The embodiment of the present invention provides a personalized prediction method for human-cabin interaction intention based on a large language model. First, by combining vehicle state data, driver behavior sequence data and data enhancement technology, a cockpit scene dataset dedicated to driver intention prediction is constructed. The dataset covers a wealth of environmental information, driving status and vehicle state data, and integrates the driver's historical interaction behavior data, providing a comprehensive and diverse data basis for model training. Secondly, a new driver intention personalized prediction model, PCockpit-LLM, is designed. The model takes vehicle driving status, external environmental information and driver's historical interaction data as input, and combines the driver's behavioral habits and historical preferences to generate future intention predictions containing personalized content. The model provides highly personalized prediction results for the specific needs of different drivers by comprehensively analyzing multimodal input data. In order to further enhance the model's ability to understand and predict the active interaction information between the driver and the cockpit, a two-stage fine-tuning strategy is adopted. In the first stage, the PCockpit-LLM is fine-tuned for all parameters using a generalized dataset that widely covers cockpit scenes, so that the model has a broad understanding of general cockpit scenes. In the second stage, the model is further fine-tuned using a personalized cockpit scene dataset. The personalized dataset contains the interaction information of specific drivers. The model parameters are efficiently optimized in combination with the LoRA method, which significantly reduces the training cost and enhances the model's ability to accurately predict the interaction behavior of specific drivers. Compared with traditional intention prediction methods, this method achieves high accuracy and personalization of driver intention prediction by making full use of the context understanding and knowledge generalization capabilities of large language models. At the same time, the proposed multi-stage fine-tuning method effectively addresses the challenges brought by the diversity of cockpit scenes and the uniqueness of driver behavior, providing theoretical support and technical foundation for the personalized active interaction function of the smart cockpit. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flowchart of a method for personalized prediction of human-cabin interaction intention based on a large language model provided by an embodiment of the present invention; Figure 2 The process of building a dataset for driver interaction scenarios; Figure 3 This is the model architecture of PCockpit-LLM. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0014] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0015] like Figure 1 As shown, a method for personalized prediction of human-cabin interaction intention based on a large language model is provided in one embodiment of the present invention, comprising the following steps: Step 1: Construction of driver interaction scenario dataset; By integrating vehicle status data, driver behavior sequence data, and data enhancement technology, a cockpit scene dataset specifically for driver intention prediction is constructed. This dataset comprehensively covers a wealth of environmental information, driving status, and vehicle dynamic status data, while integrating the driver's historical interaction behavior data, thus providing a comprehensive, detailed, and diverse data foundation for model training. Based on this global dataset, a specific driver's interaction dataset is further extracted to enhance the model's adaptability in personalized intention prediction. The construction of this dataset not only provides accurate training data support for intelligent driving systems, but also lays a solid foundation for improving the accuracy of driver intention prediction and real-time response capabilities.

[0016] Step 2: Build the driver personalized intention prediction model PCockpit-LLM; A new personalized prediction model for driver intentions, PCockpit-LLM, was designed. The model takes vehicle driving status, external environment information, and driver's historical interaction data as input, and combines the driver's behavioral habits and historical preferences to generate future intention predictions with personalized content. The model provides highly personalized prediction results for the specific needs of different drivers by comprehensively analyzing multimodal input data. This further improves the model's ability to understand and predict active interaction information between the driver and the cockpit.

[0017] Step 3: Training and fine-tuning of the PCockpit-LLM model; A two-stage fine-tuning strategy was adopted. In the first stage, a generalized dataset that widely covers cockpit scenarios was used to fine-tune all parameters of PCockpit-LLM, so that the model has a broad understanding of common cockpit scenarios. In the second stage, the model was further fine-tuned using a personalized cockpit scenario dataset. The personalized dataset contains the interaction information of a specific driver, and the model parameters are efficiently optimized in combination with the LoRA method, which significantly reduces the training cost and enhances the model's ability to accurately predict the interaction behavior of a specific driver.

[0018] As a preferred embodiment of the present invention, in step 1, the driver interaction scene data set construction process is as follows: Figure 2As shown in the figure, the vehicle-mounted signals and cockpit signals are integrated to accurately reflect the vehicle status information at the current moment and the driver's historical interaction behavior. The status information in the dataset covers the vehicle's driving status, cockpit status, and body status. All vehicle-mounted signals and cockpit signals are derived from the vehicle's CAN bus. To ensure efficient processing and utilization of data, a set of data preprocessing and enhancement tool libraries are designed. The library performs effective preprocessing and data enhancement operations on the original data, and finally outputs a text format dataset that meets the training requirements, providing a solid data foundation for subsequent model training.

[0019] First, the sensor data in the cabin and the environment are extracted and processed to obtain the status information of the equipment in the cabin and the vehicle status data. Then, the state function in the tool library is used to and behavior function Extract vehicle status information and information related to driver behavior from sensor data, and perform data enhancement processing on the above data using algorithms that convert state attributes to behavior attributes and behavior attributes to state attributes. Finally, combine this information with system prompt information and encapsulate it into text prompts to fully describe the relationship between the cockpit and vehicle status and driver behavior, providing complete data support for model fine-tuning. The problem is defined as follows: By integrating multi-attribute cockpit perception information, a text structure with a unified format centered on the driver is formed, and then a large language model is introduced to infer the driver's next interaction based on the information inside and outside the cabin and the driver's historical behavior sequence.

[0020] The final text prompt Covering the current moment Vehicle status information and the corresponding n drivers' historical interaction information A collection of: ; The state attribute Including body condition and cockpit status : ; Body condition Including vehicle speed , Lighting status , Door status and the window status : ; Cockpit status Including seat belt status and air conditioning status : ; Air conditioning status Including air conditioning temperature, blowing mode and blowing level.

[0021] Behavior Sequence The specific contents of the driver's current behavior and the previous n historical behaviors are recorded.

[0022] In order to enable the model to obtain higher quality prediction task answers, a matching global feature set is designed Text instructions , It contains the task goal, which is used to guide the processing direction and generation logic of the large language model. The main goal of the prompt project is to generate accurate driver behavior prediction labels. It is used to briefly explain what specific information will be input to the large language model, that is, the text data A brief introduction to help the model understand The main content and internal logic of the output instructions include the instruction constraints on the large language model and the expected result type generated by the model, which includes limiting the model prediction direction, prompting the model reasoning direction to specify the generation format of the model prediction text. In summary, this method is centered on user interaction behavior, describes the vehicle state of the current interaction, and extracts n event-oriented previous historical interaction behaviors, which together constitute the global feature set Such a complete state information and interaction events can not only provide an immediate understanding of the current interaction environment, but also enhance the ability to predict potential behavior patterns and state changes through the integration of historical data. Combined and packaged together as instruction text , together constitute the input of the driver interaction scenario dataset.

[0023] As a preferred embodiment of the present invention, in step 2, the architecture of the constructed PCockpit-LLM model is as follows: Figure 3 As shown, it mainly includes three parts: The text data word segmenter encoding part, the large language model backbone part, the full parameter fine-tuning for the LLM backbone, the adapter fine-tuning, and the decoding part of the driver behavior intention data.

[0024] First, the text data tokenizer is used to encode the text instructions into a format that can be processed by LLM and then input into LLM for processing. Secondly, to ensure that PCockpit-Llama can accurately adapt to the current task, the model is fine-tuned using the full parameter fine-tuning method and the adapter method, and the adapter is added to the LLM backbone. Finally, the prediction tags generated by LLM are decoded using the text de-tagger. In order to improve the efficiency of extracting reasoning results, the predicted driver behavior intention is hierarchically decoded, and during the decoding process, the predicted driver behavior intention is output in text format.

[0025] (1) Data encoding: PCockpit-LLM input consists of text data and a piece of instruction text used to inspire a large language model constitute. Used to convert all the contents of the constructed multimodal dataset into corresponding word vector representations: ; (2) Fine-tuning of the LLM main part: Full parameter fine-tuning is a deep learning model training method that adjusts all parameters of a pre-trained model to adapt it to a specific task. This method is often used in transfer learning to improve the performance of the model in new tasks by optimizing the weights of the pre-trained model on the target task. Full parameter fine-tuning enables the model to capture the characteristics of the target data more effectively, especially when the amount of data is limited, significantly improving the generalization ability and accuracy of the model.

[0026] In the first stage, the pre-trained model is fine-tuned in all parameters using the global driver interaction scenario dataset to ensure that the model has broad general capabilities in the driver intention prediction task, providing a basis for personalized settings for the adapter fine-tuning in the second stage.

[0027] In the second stage of fine-tuning, a personalized dataset for a specific driver is extracted from the global dataset, and LoRA is used as an adapter to perform lightweight fine-tuning on the LLM backbone. The core idea of ​​the LoRA fine-tuning method is to dynamically adjust the model weights by adding a low-rank matrix to the weight matrix of the pre-trained model. This fine-tuning method allows the performance of the model to be effectively adjusted to adapt to specific tasks with only a small number of additional parameters, while keeping the weight parameters of the pre-trained model unchanged.

[0028] Llama3-Instruct-8B is used as the backbone network of LLM, which contains multiple transformer layers and has excellent semantic understanding and logical reasoning capabilities. In this method, Lora is introduced in each linear layer of Llama3-8B-Instruct (including the multi-head attention layer MHA and the feedforward network layer FFN). The input token first enters the MHA layer, in which Lora is applied to each attention matrix. The attention matrix includes: query matrix, key matrix, and value matrix. Lora itself is a bypass module containing a lower projection matrix and an upper projection matrix. During fine-tuning, the input and output dimensions of the MHA layer remain unchanged, and the linear transformation matrix of each attention matrix is ​​frozen. Only the newly added lower projection matrix and upper projection matrix are trained, which are usually initialized by random Gaussian distribution or by zero matrix. The calculation process of the entire MHA mechanism is as follows: ; ; in: ; Each attention head , the linear transformation matrix, is the corresponding down-projection matrix, , , is the corresponding upper projection matrix. The transformed matrix representing the output.

[0029] Then, after a residual connection and layer normalization operation, we get : ; Next, The input passes through the feedforward network layer, which consists of two fully connected layers and an activation function. In these two fully connected layers, LoRA is introduced and the same training method as above is adopted: the weight matrix of the fully connected layer is frozen and only the lower projection matrix and the upper projection matrix are trained. After a residual connection and layer normalization operation, the output H of the transformation layer is obtained. L : ; H L It will be used as the input loop iteration of the next tramsformer, and finally the intermediate label T is obtained after adding the Lora32 layer transformer L : ; (3) Decoding part: After getting the predicted token text T L After that, the decoder is used to decode it into natural language. That is, the output text of the entire PCockpit-LLM represents the driver's personalized behavior predicted by the model. The entire decoding process can be expressed as: .

[0030] As a preferred embodiment of the present invention, in step 3, in the first stage of full parameter fine-tuning, all layers of the base model are fine-tuned using the global driver interaction dataset to ensure that the model has a wide range of general knowledge. In the second stage of adapter fine-tuning, a specific driver personalized interaction dataset is used to fine-tune all linear layers of the model after full parameter fine-tuning in the first stage, and the LoRA method is used in the fine-tuning process. The cross entropy loss is used in both stages of fine-tuning. To supervise the inference process of the model, the weights of the adapter are updated by minimizing the cross entropy loss and the weight of the LLM backbone : .

[0031] A specific example is provided below to verify this method.

[0032] (1) Implementation details: In the full parameter fine-tuning stage, 1 epoch was trained with a total batch size of 64, distributed on 3 A100 (80GB) GPUs, using the AdamW optimizer with cosine learning rate scheduling, the initial learning rate was set to 3e-5, and the target was all parameters in the LLM backbone in PCockpit-LLM. In the adapter fine-tuning stage, 3 epochs of fine-tuning were performed on 3 A100 (80GB) GPUs, the initial learning rate was set to 5e-5, the target layer was all linear layers in the llama3.1 model based on full parameter fine-tuning, the intermediate rank r = 8, and the rest of the settings were the same as the full parameter fine-tuning stage. The ratio of the dataset and the test set used in this experiment was 7:3.

[0033] (2) Evaluation indicators: For the model evaluation after full parameter fine-tuning and adaptation fine-tuning, the present invention uses four common indicators in the NLP field for evaluation, including Rouge-1, Rouge-2, Rouge-L and Bleu-4. These indicators are designed to evaluate the quality of text generation, among which BLEU4 focuses on the accuracy of n-gram, and Rouge-n considers the subtle differences in semantics and syntax. PCockpit-LLM has significantly improved the performance of driver personalized prediction compared with all baseline models at the current stage. In order to demonstrate its superiority, after the first stage of full parameter fine-tuning, we compared the intermediate model PCockpit-LLM-Full with two large language models Chat-GPT4 and Chat-GLM3 that are recognized to have strong capabilities at this stage, and then further compared the final model PCockpit-LLM after the adapter fine-tuning in the second stage with the intermediate model. The results show that in the two comparative experiments, the performance of the model PCockpit-LLM designed by the present invention has been significantly improved.

[0034] Table 1 shows the experimental results of the intermediate model Cockpit-Llama-Full obtained after fine-tuning all parameters, compared with the baseline models Chat-GLM3 and Chat-GPT4. Cockpit-Llama-Full achieved excellent results, with scores of 65.62, 63.51, 53.24, and 62.19 on the four indicators of Bleu-4, Rouge-1, Rouge-2, and Rouge-L, respectively. This shows that the performance of the model has been significantly improved after fine-tuning all parameters.

[0035] Table 1 Performance comparison of baseline methods .

[0036] Table 2 shows the experimental results of the final model after adapter fine-tuning and the intermediate model. The results show that based on the fine-tuning of all parameters, the final model PCockpit-Llama obtained by adapter fine-tuning performs as follows on the four evaluation indicators: Bleu-4 is 75.68, Rouge-1 is 68.62, Rouge-2 is 63.55, and Rouge-L is 69.38. These results further verify the accuracy and reliability of Cockpit-Llama in the driver behavior prediction task.

[0037] Table 2 Comparison of performance between the first and second stage models .

[0038] Table 3 shows the impact of different fine-tuning methods on the final performance of the model. The comparative test includes the experimental results of the frozen fine-tuning method (fine-tuning only local parameters and freezing all other parameters), LoRA fine-tuning, and full parameter fine-tuning. The results directly prove the advantages of the two-stage fine-tuning method adopted by the present invention in terms of accuracy and reliability.

[0039] Table 3 Performance comparison of fine-tuning methods .

[0040] In summary, the experimental results show that the PCockpit-LLM proposed in this invention has excellent prediction capabilities for personalized driver intentions.

[0041] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A personalized prediction method for human-cabin interaction intention based on a large language model, characterized in that: The following steps are involved: Step 1: Construction of driver interaction scenario dataset; By integrating vehicle status data, driver behavior sequence data, and data enhancement technology, a cockpit scene dataset specifically for driver intention prediction is constructed; vehicle status data includes real-time dynamic information of the vehicle while driving; driver behavior sequence data includes multi-dimensional data of the interaction between the driver and the vehicle; Increase data diversity with the help of data enhancement technology to provide a data basis for personalized driver intention prediction; Step 2: Build the driver personalized intention prediction model PCockpit-LLM; The model takes the vehicle's driving status, external environment information and the driver's historical interaction data as input, and combines the driver's behavioral habits and historical preferences to generate future intention predictions with personalized content; The model captures the driver's immediate behavioral characteristics by comprehensively analyzing multimodal input data, and can perform deep learning based on historical data to analyze the personalized needs of each driver and provide highly personalized prediction results based on the specific needs of different drivers; Step 3: Training and fine-tuning of the PCockpit-LLM model; A two-stage fine-tuning strategy was adopted, including full-parameter fine-tuning and adapter fine-tuning. In the first stage, a generalized dataset covering cockpit scenarios was used to fine-tune the full parameters of PCockpit-LLM, so that the model had a broad understanding of common cockpit scenarios. In the second stage, the model was further fine-tuned using a personalized cockpit scenario dataset. The personalized cockpit scenario dataset contains the interaction information of a specific driver, and the model parameters are optimized in combination with the LoRA method.

2. The personalized prediction method for human-cabin interaction intention based on a large language model according to claim 1 is characterized in that: In step 1, the status information in the data set covers the driving status, cockpit status and body status of the vehicle. All vehicle computer signals and cockpit signals are derived from the vehicle CAN bus. At the same time, a set of data preprocessing and enhancement tool libraries are designed to preprocess and enhance the original data, and finally output a text format data set that meets the training requirements. Specifically: First, the sensor data in the cabin and the environment are extracted and processed to obtain the status information of the equipment in the cabin and the vehicle status data; then, the status function in the tool library is used to and behavior function Extract vehicle status information and information related to driver behavior from sensor data, and perform data enhancement processing on the above data using algorithms that convert state attributes into behavior attributes and vice versa. Finally, combine this information with system prompt information and encapsulate it into text prompts to fully describe the relationship between cockpit and vehicle status and driver behavior. The problem is defined as follows: By integrating multi-attribute cockpit perception information, a text structure with a unified format and centered on the driver is formed. Then a large language model is introduced to infer the driver's next interaction based on the information inside and outside the cabin and the driver's historical behavior sequence; The final text prompt Covering the current moment Vehicle status information and the corresponding n drivers' historical interaction information A collection of: ； The state attribute Including body condition and cockpit status : ； Body condition Including vehicle speed , Lighting status , Door status and the window status : ； Cockpit status Including seat belt status and air conditioning status : ； Air conditioning status Including air conditioning temperature, blowing mode and blowing level; Behavior Sequence The specific contents of the driver's current behavior and the previous n historical behaviors are recorded; In order to enable the model to obtain higher quality prediction task answers, a matching global feature set is designed Text instructions , It contains task objectives, input descriptions, and output instructions. The task objectives are used to guide the processing direction and generation logic of the large language model. The goal of the prompting project is to generate accurate driver behavior prediction labels; the input description is used to briefly explain what specific information will be input to the large language model, that is, the text data A brief introduction to help the model understand The content and internal logic in the output instructions include the instruction constraints on the large language model and the type of results expected to be generated by the model, which includes limiting the model prediction direction, prompting the model reasoning direction to specify the generation format of the model prediction text; By focusing on user interaction behavior, the vehicle state of the current interaction is described, and n event-oriented previous historical interaction behaviors are extracted to form a global feature set. ; Afterwards, by using the text command Combined and packaged together as instruction text , together constitute the input of the driver interaction scenario dataset.

3. The personalized prediction method for human-cabin interaction intention based on a large language model according to claim 2 is characterized in that: In step 2, the constructed PCockpit-LLM model includes three parts: The encoding part of the text data tokenizer, the large language model backbone, and the full parameter fine-tuning of the LLM backbone, the adapter fine-tuning, and the decoding part of the driver behavior intention data; First, the text instructions are converted into a format that can be processed by LLM using the text data segmenter encoding and input into LLM for processing; second, to ensure that PCockpit-Llama can accurately adapt to the current task, the model is fine-tuned using the full parameter fine-tuning method and the adapter method, and the adapter is added to the LLM backbone; finally, the predicted tags generated by LLM are decoded using the text de-tagger; The predicted driver behavior intention is decoded hierarchically, and during the decoding process, the predicted driver behavior intention is output in a text format; 1) Data encoding: PCockpit-LLM input consists of text data and a piece of instruction text used to inspire a large language model constitute; Used to convert all the contents of the constructed multimodal dataset into corresponding word vector representations: ； 2) Fine-tuning of the LLM backbone: In the first stage, the pre-trained model is fine-tuned using the driver interaction scenario dataset to ensure that the model has broad general capabilities in the driver intention prediction task, providing a basis for personalized settings for the adapter fine-tuning in the second stage. In the second stage of fine-tuning, a personalized dataset for a specific driver is extracted from the global dataset, and LoRA is used as an adapter to perform lightweight fine-tuning on the LLM backbone; The core idea of ​​the LoRA fine-tuning method is to dynamically adjust the model weights by adding a low-rank matrix to the weight matrix of the pre-trained model; Use Llama3-Instruct-8B as the backbone network of LLM, and introduce Lora into each linear layer of Llama3-8B-Instruct; The input token first enters the MHA layer, in which Lora is applied to each attention matrix; The attention matrix includes: query matrix, key matrix and value matrix; Lora itself is a bypass module, which contains a down-projection matrix and an up-projection matrix. During the fine-tuning process, the input and output dimensions of the MHA layer remain unchanged, and the linear transformation matrix of each attention matrix is ​​frozen. Only the newly added down-projection matrix and up-projection matrix will be trained. These matrices are initialized by random Gaussian distribution or by zero matrix. The calculation process of the entire MHA mechanism is as follows: ； ； in: ; Each attention head , the linear transformation matrix, is the corresponding down-projection matrix, , , is the corresponding upper projection matrix, The transformed matrix representing the output; Then, after a residual connection and layer normalization operation, we get : ； Next, The input passes through the feedforward network layer, which consists of two fully connected layers and an activation function. In these two fully connected layers, LoRA is introduced and the same training method as above is adopted: the weight matrix of the fully connected layer is frozen, and only the lower projection matrix and the upper projection matrix are trained. After a residual connection and layer normalization operation, the output H of the transformation layer is obtained. L : ； H L It will be used as the input loop iteration of the next tramsformer, and finally the intermediate label T is obtained after adding the Lora32 layer transformer L : ； 3) Decoding part: After getting the predicted token text T L After that, the decoder is used to decode it into natural language, that is, the output text of the entire PCockpit-LLM, which represents the driver's personalized behavior predicted by the model; the entire decoding process is expressed as: 。 4. The personalized prediction method for human-cabin interaction intention based on a large language model according to claim 3 is characterized in that: In the step 3, in the first stage of full parameter fine-tuning, all layers of the base model are fine-tuned using the global driver interaction dataset; In the second stage of adapter fine-tuning, a specific driver personalized interaction dataset is used to fine-tune all linear layers of the model after full parameter fine-tuning in the first stage. The LoRA method is used in the fine-tuning process; cross entropy loss is used in both stages of fine-tuning. To supervise the inference process of the model, the weights of the adapter are updated by minimizing the cross entropy loss and the weight of the LLM backbone : 。

Images