Intelligent cockpit driver intention prediction method and system fusing scene semantics

Abstract

The invention discloses an intelligent cockpit driver intention prediction method and system fusing scene semantics, and the method comprises the steps: collecting cockpit interaction data and driving dynamic data, extracting a data sequence of a specified time length through a sliding window method, and constructing a driver interaction intention data set; constructing an instruction fine tuning data set of scene semantic understanding based on logic rules and manual inspection, and converting the driving dynamic data into a scene semantic text through a fine tuning large language model; training a deep learning model by using the constructed driver intention data set, and outputting the most probable driver interaction intention; according to the method, the limitation of independent modeling of a driver action sequence and spatio-temporal information in a traditional method is effectively solved, collaborative modeling of driver interaction actions and scene spatio-temporal characteristics is formed by fusing context information of a scene spatio-temporal causal relationship, and accurate prediction of interaction intentions of various services such as air conditioners, music and seats in an intelligent cabin is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent cockpit, in particular to an intelligent cockpit driver intention prediction method and system fusing scene semantics. BACKGROUND

[0002] In recent years, with the rapid development of artificial intelligence and Internet of Things technology, the automobile cockpit has gradually changed from a traditional mechanical cockpit to an intelligent cockpit, and the automobile has changed from a simple "mechanical transport tool" to an "intelligent mobile space". Under this background, the digital cockpit integrates vehicle sensor data, environmental perception data and user interaction behavior data and other multi-source data, realizes the system function of individualization of various services and naturalization of interaction, and enables the driver and passengers to enjoy more intelligent and convenient service experience. In the evolution of human-computer interaction of intelligent cockpit, the "active service" capability has become the focus of industry competition. As the core basis of active service, the driver interaction intention prediction technology predicts the potential interaction demand of users and the cockpit functions such as navigation, air conditioning and music through analyzing multi-source heterogeneous data, so as to provide support for the triggering and decision-making of active service functions.

[0003] Most of the existing driver interaction intention prediction researches construct single service function prediction model based on in-vehicle and out-vehicle images and vehicle driving state data, which has the deficiency in the generalization of intelligent cockpit multi-class service function prediction. Although the general time series prediction model performs well in multi-field classification prediction tasks, it usually takes space-time data as input in the process of interaction intention prediction task adaptation, without considering the space-time constraint information in the driving scene, and cannot analyze the causal relationship between scene characteristics and driver intention. The above limitations not only restrict the accuracy of interaction intention prediction, but also make it difficult for existing researches to adapt to the complex and changeable real driving environment.

[0004] The scene not only contains spatial dimension information such as road environment and traffic state, but also involves time sequence evolution law such as season climate and day and night illumination. Through cooperative modeling with the interactive behavior of the driver, the deep coupling relationship between the dynamic evolution of the scene and the interactive behavior can be effectively captured, thereby improving the accuracy of the intention prediction. Traditional research mainly adopts rule-based feature engineering or deep learning methods such as CNN and LSTM to extract scene semantic labels, but the combination of discrete scene labels destroys the interaction relationship between scene elements, resulting in the loss of context association. The breakthrough development of large language model (LLM) provides a new paradigm for scene semantic modeling. LLM represented by GPT4, LLaMA and GLM has a spatio-temporal correlation network constructed by a self-attention mechanism, which can effectively capture the change characteristics of driving dynamic data, thereby showing deep semantic understanding ability and context modeling ability beyond traditional methods. However, existing researches mainly focus on dialogue generation in general scenes, and there is still a technical gap in the field adaptability and spatio-temporal feature modeling of scene semantic information generation. SUMMARY

[0005] The technical problem to be solved by the present application is to overcome the above-mentioned problems existing in the prior art, and to provide an intelligent cockpit driver intention prediction method and system fusing scene semantics.

[0006] To solve the above technical problems, the present application is realized by adopting the following technical solutions, which are described in combination with the accompanying drawings as follows:

[0007] An intelligent cockpit driver intention prediction method fusing scene semantics, comprising the following steps:

[0008] Step one: collect cockpit interaction data and driving dynamic data, and use a sliding window method to extract data sequences of a specified time length to construct a driver interactive intention data set;

[0009] Step two: construct a scene semantic understanding instruction fine-tuning data set based on logical rules and manual inspection, and convert driving dynamic data into scene semantic text by fine-tuning a large language model;

[0010] Step three: establish a deep learning model containing a scene guide attention fusion mechanism, train the deep learning model using the constructed driver intention data set, and output the most possible driver interactive intention.

[0011] Further, in step one, cockpit interaction data and driving dynamic data are collected, and a sliding window method is used to extract data sequences of a specified time length to construct a driver interactive intention data set. The specific content is as follows:

[0012] The collected data consists of two parts: cockpit interaction data collected through intelligent cockpit buried point signals, including driver-cockpit interaction actions and cockpit equipment status information; driving dynamic data collected through on-board multi-source sensors and CAN bus, including vehicle driving status and environmental status information;

[0013] Filter out cockpit interaction data features and driving dynamic data features related to intent labels;

[0014] Use sliding window sampling method to construct input data samples;

[0015] The driver's action at the next moment in the time window is used as the classification label;

[0016] Construct a driver interaction intention prediction dataset.

[0017] Furthermore, the signal acquisition frequency is 0.1Hz; a total of 24-dimensional cabin interaction data features including door, window, air conditioning, seat, and music status are screened out, as well as a total of 12-dimensional driving dynamic data features including time, temperature, weather, mileage, and vehicle speed.

[0018] The sliding window sampling method with a time window length of 60 seconds is used to construct the input data sample.

[0019] Furthermore, in step 2, a fine-tuning dataset of instructions for scene semantic understanding is constructed based on logical rules and manual verification. The driving dynamic data is converted into scene semantic text by fine-tuning the large language model. The specific contents are as follows:

[0020] Build a scene classification logic rule library based on the scene element definition to verify and correct the original scene semantic text generated by the pre-trained LLM;

[0021] Then, regular expression-based pattern matching verification and manual verification are performed to ensure the accuracy and rationality of the generated text and to build a high-quality instruction fine-tuning dataset.

[0022] Low-rank adaptive LoRA technology is used to efficiently fine-tune the pre-trained model.

[0023] Furthermore, the scene semantic text S contains four key information: time S time , vehicle S veh , environment S env , user S user :

[0024] S={S time ,S veh ,S env ,S user}

[0025] Time information S time Including date division S date With time period S period ; Wherein S date Is divided into two categories: weekdays refer to Monday to Friday statutory working days; Holidays include double holidays and national statutory holidays; S period Is divided into five time periods: 00:00-05:59, 06:00-11:59, 12:00-17:59, 18:00-21:59 and 22:00-23:59 at night;

[0026] Vehicle information S veh Including current average speed S speed , driving state S state , distance traveled S travel , distance S route , road condition S traffic , driving section S road ; The S speed and S travel values of the latest time are read directly through the CAN bus; According to the Δv / Δt change coefficient, S state Is divided into six categories: parking, speed is zero and lasts for 60 seconds; Stable driving, acceleration fluctuation ≤2 m / s2; Gradual acceleration or deceleration, acceleration fluctuation is 2-4 m / s2; Frequent acceleration and deceleration, acceleration sign change ≥3 times / min; S route The distance division standard is adopted, and 50km is taken as the critical point of normal driving and long-distance driving; S traffic And S road Are obtained through vehicle navigation information, which are divided into smooth and congested, ordinary roads with design speed ≤80km / h, and expressways;

[0027] Environmental information S env Including weather conditions S wea , temperature in the car S in_car and temperature outside the car S out_car ; Among them S wea Is obtained by calling Weatherstack API according to GPS latitude and longitude information, and all weather is divided into three categories: sunny, rainy and snowy; S in_car And S out_car Are read by temperature sensor through CAN bus;

[0028] User information S user Including fatigue state S tired and distraction state S dis ; Among them S tired and S dis Are divided into the state that the driver is in or normal state, which are obtained by the driver monitoring system DMS;

[0029] By leveraging the context judgment and text generation capabilities of the large language model, we can ultimately generate a structured scene semantic text S containing all the above scene elements:

[0030] “In [S time ] of [S period ], the current weather conditions are [S wea ], outside temperature [S out_car ]℃, vehicle interior temperature [S in_car ]℃, the vehicle is [S speed The average speed is [S state ] stage, the cumulative travel time is [S travel ] and in [S route ] driving, the driver showed [S tired / S dis ]; According to the navigation road condition monitoring data, the vehicle has entered [S road ] driving, the current road conditions are [S traffic ].

[0031] ChatGLM4-9B is used as the basic model to realize the generation of scene semantic text. "You are a driving scene semantic analyst. You need to generate natural language description of driving scene semantic text based on real-time vehicle status data and environmental data" is used as the role setting. At the same time, the driving dynamic data time series fragments and data feature descriptions are combined as the input prompt text of LLM. p By building a logical rule library for scene element classification, the original scene semantic text is verified and corrected. Each rule strictly follows the above scene element definition. Then, regular expression-based pattern matching verification is performed and manually verified to ensure the accuracy and rationality of the generated text. Finally, a high-quality instruction fine-tuning dataset D is constructed. tuning ={(X p ,S)};

[0032] Use low-rank adaptation (LoRA) technology to efficiently fine-tune the basic model;

[0033] Specifically, given the pre-trained weight matrix Coupled with the low-rank decomposition matrix ΔW = BA, where And r<<min(d,k); Matrix A is initialized with random Gaussian and Matrix B is initialized with zero to ensure that ΔW is a zero matrix in the initial stage of training;

[0034] In the model inference process, the original weight W0 and the low-rank matrix ΔW act on the input features at the same time, and the output vectors are combined; in the Transformer architecture, the calculation formula of the original query Q, key K, and value V is modified as follows:

[0035]

[0036] wherein, W q0 ,W k0 ,W v0 denote the weight matrix of the initial Q, K, and V; ΔW q ,ΔW k ,ΔW v denote the weight update matrix for fine-tuning; B q ,B k ,B v ,A q ,A k ,A v denote the projection matrix under the low rank; α denotes a scaling factor for adjusting the speed of the low-rank matrix update; r denotes the rank of the low-rank matrix.

[0037] Further, in step three, a deep learning model containing a scene-guided attention fusion mechanism is established, and the deep learning model is trained using the constructed driver intention dataset to output the most likely driver interaction intention; the specific content is as follows:

[0038] The deep learning model is divided into a cockpit interaction data encoding module, a scene semantic encoding module, and an intention prediction module;

[0039] The cockpit interaction data encoding module: adopts a long short-term memory network (LSTM) as the core architecture; the LSTM updates the internal state of the memory unit at each time step t according to the current input and the hidden state of the previous time step through the memory unit and the gating mechanism; the final output hidden state contains important feature information at the current time and historical time;

[0040] The scene semantic encoding module: uses a Word2Vec model and a Tencent artificial intelligence laboratory word embedding corpus to convert scene semantic text into continuous neural word embedding representations; then, a linear layer is used to transform the text embedding vectors in dimension, obtaining scene semantic embedding vectors, so that the embedding dimension is consistent with the cockpit interaction state encoding vector dimension;

[0041] The intention prediction module:

[0042] First, the encoded cockpit interaction data and structured scene semantic vectors are used as inputs, and a scene-guided attention fusion mechanism is established to inject scene semantics as global context information into the multi-head attention calculation process, and multi-modal feature fusion is achieved by dynamically adjusting the attention weight distribution.

[0043] Subsequently, multi-head attention concatenation, residual connection and feedforward neural network calculation are performed;

[0044] Finally, a multilayer perceptron decoder with three linear layers is used to extract high-level feature representations; a Softmax function is used for the output of the last linear layer to calculate the probability distribution of various intent categories, so that the model can classify the input into a specific intent type with a probability confidence.

[0045] Further, in the cockpit interaction data encoding module, the LSTM updates the internal state of the memory unit at each time step t according to the current input and the hidden state of the previous time step; the final output hidden state contains important feature information at the current time and historical time, which can effectively analyze and model the cockpit interaction time series data and provide a high-quality encoding representation basis for subsequent intent prediction tasks;

[0046]

[0047] where X (c) represents the cockpit interaction data input at the current time step, represents the output hidden state vector at the current time step, and h represents the hidden state vector at the previous time step.

[0048] Further, the scene semantic encoding module uses a uniformly designed prompt template to generate structured scene semantic text S = [S1, S2, …, S n ] based on the step two fine-tuned large language model; to establish a semantic representation space, the scene semantic text S is converted into continuous neural word embedding representation where d p represents the embedding dimension, which takes a fixed value of 300; then the text embedding vector is dimensionally transformed to obtain the scene semantic embedding vector so that the embedding dimension is consistent with the cockpit interaction state encoding vector dimension; in terms of computational efficiency optimization, the upper limit of the length of the input text is set to 1000;

[0049]

[0050] where n represents the total number of words in the scene semantic text, S i represents the i-th word in the input text sequence, represents a trainable parameter matrix, The deviation vector is represented.

[0051] Further, the intention prediction module adopts a Transformer-based enhanced implementation, and the encoded cockpit interaction data and structured scene semantic vector are used as inputs to realize the inference of the driver interaction intention;

[0052] The scene semantics are injected into the multi-head attention calculation process as global context information, and multi-modal feature fusion is realized by dynamically adjusting the attention weight distribution; in the multi-head attention calculation of each layer, the cockpit interaction time embedding and the scene semantic embedding h text are jointly encoded into K and V, while Q remains the cockpit interaction feature encoding; the calculation formulas of Q, K, and V are formally redefined as follows:

[0053]

[0054] Among them, Q (i) ,K (i) ,V (i) represent the query, key, and value vectors of the i-th attention head, respectively; represents the trainable projection matrix of the i-th attention head; d k =d h / N is the dimension of each attention head, and N is the number of attention heads; represents the feature vector of the cockpit interaction data; h text represents the feature vector of the scene semantic text;

[0055] Subsequently, multi-head attention is concatenated and residual connected; through hierarchical attention weighting and feature fusion, the intention prediction module realizes the deep coupling of scene semantic information and interaction time features, and obtains deep time-coupled features h t ;

[0056] head i =Attn(Q (i) ,K (i) ,V (i) )

[0057] MHA(h t )=[head1;…;head N ]W o

[0058] Among them, Attn represents the attention mechanism calculation function, head i represents the output vector of the i-th attention head, and MHA(h t ) represents the output feature h t obtained after multi-head attention calculation, represents the trainable output projection matrix;

[0059] The feedforward neural network consists of two linear transformations and a ReLU activation function for non-linear transformation, as follows:

[0060] FFN(h t ) = ReLU(h t W1 + b1) W2 + b2

[0061] wherein represents the trainable weight matrix of linear transformation in the feedforward neural network; b1, b2 represents the trainable bias vector in the linear transformation; FFN(h t ) represents the feature h t updated after the calculation of the feedforward neural network; t ;

[0062] Finally, a multilayer perceptron (MLP) decoder with three linear layers is used to extract high-level feature representations; the dimensions of the hidden layers are 4096, 1024 and 256, respectively, to gradually reduce the feature dimension to capture hierarchical information, while directly adding a ReLU activation function at each layer; a Softmax function is used for the output of the last linear layer to calculate the probability distribution of various intent categories, so that the model can classify the input into a specific intent type with a probability confidence;

[0063]

[0064] wherein, represents the logits of decoding the processed hidden feature h t into C intent categories, which is an unnormalized real number vector, and each value corresponds to an intent category; MLP(h t ) i represents the logit value of the i-th intent category; c ∈ {1, 2, …, C} corresponds to each intent category, and C represents the total number of intent categories; exp(·) represents the exponential function, which converts the logits output by MLP into positive numbers; P(y = i | x) represents the probability of predicting the class i under the input x.

[0065] The intention prediction module takes the encoded cabin interaction data and structured scene semantic vector as input, injects scene semantics as global context information into the multi-head attention calculation process by establishing a scene-guided attention fusion mechanism, and realizes multi-modal feature fusion by dynamically adjusting the attention weight distribution. Subsequently, multi-head attention is connected in series, residual connection and feedforward neural network calculation are performed. Finally, a multilayer perceptron decoder with three linear layers is used to extract high-level feature representations. A Softmax function is used for the output of the last linear layer to calculate the probability distribution of various intent categories, so that the model can classify the input into a specific intent type with a probability confidence.

[0066] An intelligent cabin driver intention prediction system fusing scene semantics, comprising:

[0067] A data acquisition module is used to acquire cabin interaction data and driving dynamic data, and a sliding window method is used to extract data sequences of a specified time length to construct a driver interaction intention dataset.

[0068] A scene semantic text conversion module is used to construct an instruction fine-tuning dataset for scene semantic understanding based on logical rules and manual inspection, and convert driving dynamic data into scene semantic text by fine-tuning a large language model.

[0069] An output driver interaction intention module is used to establish a deep learning model containing a scene-guided attention fusion mechanism, train the deep learning model using the constructed driver intention dataset, and output the most likely driver interaction intention.

[0070] Compared with the prior art, the beneficial effects of the present application are:

[0071] 1. The present application proposes an intelligent cabin driver intention prediction method fusing scene semantics, effectively solving the limitations of independent modeling of driver action sequences and spatiotemporal information in traditional methods. This method forms a collaborative modeling of driver interaction actions and scene spatiotemporal features by fusing context information of scene spatiotemporal causal relationships, and can realize accurate prediction of interaction intentions of various services such as air conditioning, music, and seats in an intelligent cabin.

[0072] 2. For the double challenges in scene understanding: (1) Random fluctuations and redundant information in raw data make it difficult to extract key semantic information; (2) The scene semantic text generated directly using a pre-trained LLM has the problems of scene classification ambiguity and hallucination noise interference, which affect the performance of downstream tasks. The present application uses LoRA technology to fine-tune the pre-trained LLM to generate structured scene semantic text, thereby adaptively filtering irrelevant information and providing high-quality scene semantic input for the intention prediction task.

[0073] 3, The application constructs two new data sets: (1) a driver interaction intention data set, including time-synchronized cabin interaction data and driving dynamic data, and a driver intention label at the corresponding moment; (2) a scene semantic understanding instruction fine-tuning data set, forming a mapping association between driving dynamic data and scene semantic text. The constructed data set can support comprehensive experimental analysis and realize accurate prediction of the intelligent cabin driver interaction intention. BRIEF DESCRIPTION OF DRAWINGS

[0074] Figure 1 A driver intention prediction method flow chart for fusing scene semantics;

[0075] Figure 2 A scene element classification chart;

[0076] Figure 3 A model overall architecture chart;

[0077] Figure 4 A specific embodiment chart. DETAILED DESCRIPTION

[0078] The flow of the application will be further described in detail below in combination with the drawings.

[0079] Referring to the accompanying Figure 1 , the application proposes an intelligent cabin driver intention prediction method fusing scene semantics, including the following description flow:

[0080] Step one: collect cabin interaction data and driving dynamic data, and use a sliding window method to extract data sequences of a specified time length to construct a driver interaction intention data set. The specific content is as follows:

[0081] The data required by the application consists of two parts: cabin interaction data collected through intelligent cabin buried point signals including the interaction action of the driver and the cabin and the state information of the cabin equipment; and driving dynamic data collected through vehicle-mounted multi-source sensors and CAN bus including vehicle driving state and environmental state information, and the signal collection frequency is 0.1 Hz. Then, 24-dimensional cabin interaction data features and 12-dimensional driving dynamic data features related to the intention label are screened out.

[0082] A sliding window sampling method with a time window length of 60 seconds is used to construct input data samples and from and where T is the length of the sequence in the time window, d c and d v are the feature dimensions. The action of the driver at the next moment in the time window is taken as the classification label y t∈ {0,1,2,3}, where 0 represents no intention, and 1-3 represent music, air conditioning, and seat adjustment intentions. Then, a driver interaction intention prediction dataset D is constructed pred .

[0083]

[0084] Step two: Constructing a scene semantic understanding instruction fine-tuning dataset based on logical rules and manual inspection, and converting driving dynamic data into scene semantic text by fine-tuning a large language model LLM. The specific content is as follows:

[0085] For each driving dynamic data sample Based on LLM, a mapping relationship with scene semantic text S is constructed:

[0086] X (v) →S

[0087] The scene semantic text S contains four aspects of key information: time S time , vehicle S veh , environment S env , and user S user :

[0088] S={S time ,S veh ,S env ,S user}

[0089] The scene elements contained in each aspect of information are shown in the accompanying Figure 2 The time information S time includes date division S date and time period S period . Among them, S date is divided into two categories: weekdays refer to legal working days from Monday to Friday; holidays include weekends (Saturday and Sunday) and national statutory holidays. This division takes into account the influence of the general work and rest rules of social groups on behavior patterns. According to human daily social activities, S period is divided into five time periods: early morning (00:00-05:59), morning (06:00-11:59), afternoon (12:00-17:59), evening (18:00-21:59), and late night (22:00-23:59). This multi-granularity division method not only conforms to the characteristics of human biological clock, but also effectively captures the differences in social activities at different times of the day.

[0090] The vehicle information S veh includes the current average speed S speed , driving state S state , distance traveled S travel , and route Sroute , Road condition S traffic , Driving section S road . Among them, the latest S speed and S travel values can be directly read through the CAN bus. According to the Δv / Δt change coefficient, S state can be divided into six categories: parking (vehicle speed is zero and lasts for 60 seconds), stable driving (acceleration fluctuation ≤ 2 m / s2), gradual acceleration / deceleration (acceleration fluctuation is 2-4 m / s2), frequent acceleration / deceleration (acceleration sign change ≥ 3 times / min). S route The distance division standard is adopted, and 50 km is taken as the critical point of normal driving and long-distance driving. S traffic and S road are obtained through vehicle navigation information, which are divided into smooth and congested, ordinary road (design speed ≤ 80 km / h) and expressway.

[0091] Environmental information S env includes weather conditions S wea , vehicle interior temperature S in_car and vehicle exterior temperature S out_car . Among them, S wea is obtained by calling Weatherstack API according to GPS latitude and longitude information, and all weather is divided into three categories: sunny, rainy, and snowy; S in_car and S out_car can be read by temperature sensor through CAN bus. The change of environmental state directly affects the next intention of the driver.

[0092] User information S user includes fatigue state S tired and distraction state S dis . Among them, S tired and S dis are divided into whether the driver is in this state or normal state, which is obtained by the driver monitoring system (DMS).

[0093] Using the context judgment and text generation ability of large language model, finally generate structured scenario semantic text S containing all the above scenario elements:

[0094] "In [S time ], [S period ] is in [S wea ] weather conditions, the vehicle exterior temperature is [S out_car ]℃, the vehicle interior temperature is [S in_car ]℃, the vehicle is in [S speed ] km / h average speed, and is in [S state ] stage, has accumulated [S travel ] driving, and is in [S routeThe driver exhibits [S tired / dis According to the navigation traffic monitoring data, the vehicle has entered [S road driving, and the current road traffic conditions are [S traffic .”.

[0095] ChatGLM4-9B is used as the base model to generate scene semantic text. The role setting is "You are a driving scene semantic analyst, and you need to generate natural language description of driving scene semantic text based on real-time vehicle state data and environmental data." At the same time, the driving dynamic data time sequence segment and data feature description are used as input prompt text X p The scene semantic text generated directly by the pre-trained LLM has the problem of fuzzy classification boundary of scene elements, which makes it difficult to accurately establish the mapping relationship between scene semantics and driver's intention, and may introduce illusion noise to mislead the model reasoning. Therefore, a logical rule library of scene element classification is constructed to verify and correct the original scene semantic text, and each rule strictly follows the above scene element definition. Subsequently, regular expression matching verification and manual inspection are required to ensure the accuracy and reasonableness of the text, and finally a high-quality instruction fine-tuning dataset D tuning = {(X p , S)} is constructed. The low-rank adaptive (LoRA) technology is used to efficiently fine-tune the base model.

[0096] Specifically, given the pre-trained weight matrix is coupled with the low-rank decomposition matrix ΔW = BA, where and r << min(d, k). The matrix A is initialized with random Gaussian, and the matrix B is initialized with zero, ensuring that ΔW is a zero matrix in the initial training stage.

[0097] During model inference, the original weight W0 and the low-rank matrix ΔW act on the input features simultaneously, and the output vectors are combined. In the Transformer architecture, the calculation formula of the original Q, K, V is modified as follows:

[0098]

[0099] where is used to adjust ΔW, where α is a constant used to adjust the speed of low-rank matrix update. This method not only significantly reduces the parameter amount required for model fine-tuning, but also maintains the generalization ability of the base model, which can better adapt to the intelligent cockpit scene semantic text generation task.

[0100] Step three: establish a deep learning model containing a scene-guided attention fusion mechanism, train the deep learning model using the constructed driver intention dataset, and output the most likely driver interaction intention. The specific content is as follows:

[0101] As shown in the accompanying Figure 3 The method architecture proposed by the application can be divided into a cabin interaction data encoding module, a scene semantic encoding module, and an intention prediction module.

[0102] (1) Cabin interaction data encoding module

[0103] Considering the significant time dependence of cabin interaction data, the application adopts a long short-term memory network (Long Short-Term Memory, LSTM) as the core architecture. At each time step t, LSTM updates its internal state according to the current input and the hidden state of the previous time step through its unique memory unit and gating mechanism. The final output hidden state contains important feature information at the current time and historical times, effectively analyzing and modeling cabin interaction time series data, and providing a high-quality encoding representation basis for subsequent intention prediction tasks.

[0104]

[0105] (2) Scene semantic encoding module

[0106] This module uses a uniformly designed prompt template to generate structured scene semantic text S = [S1, S2, …, S n ] based on the LLM fine-tuning in step two. To establish a semantic representation space, the scene semantic text S is converted into continuous neural word embedding representation using the Word2Vec model and the Tencent Artificial Intelligence Laboratory word embedding corpus. Here d p represents the embedding dimension, which takes a fixed value of 300. Subsequently, the text embedding vector is dimensionally transformed through a linear layer to obtain the scene semantic embedding vector so that the embedding dimension is consistent with the cabin interaction state encoding vector dimension. In terms of computational efficiency optimization, the upper limit of the input text length is set to 1000, effectively controlling the computational complexity while ensuring semantic integrity. This continuous vector representation not only preserves rich scene semantic information, but also provides convenience for subsequent feature fusion.

[0107]

[0108] where n represents the total number of words in the scene semantic text, represents a trainable parameter matrix, represents the deviation vector.

[0109] (3) Intention prediction module

[0110] The intention prediction module adopts an enhanced implementation based on Transformer, and the encoded cockpit interaction data and structured scene semantic vector are used as input to realize the inference of the driver interaction intention.

[0111] The core innovation of this module lies in the scene-guided attention fusion mechanism, which injects scene semantics as global context information into the multi-head attention (MHA) calculation process, and realizes multi-modal feature fusion by dynamically adjusting the attention weight distribution. Specifically, in the multi-head attention calculation of each layer, the cockpit interaction time embedding and the scene semantic embedding h text are jointly encoded into K and V, while Q remains the cockpit interaction feature encoding. The calculation formulas of Q, K, and V are formally redefined as follows:

[0112]

[0113] where represents the trainable projection matrix of the i-th attention head, d k = d h / N is the dimension of each attention head, and N is the number of attention heads.

[0114] Subsequently, multi-head attention concatenation and residual connection are performed. Through hierarchical attention weighting and feature fusion, the model realizes the deep coupling of scene semantic information and interaction time features, and obtains deep time-coupled features h t .

[0115] head i = Attn(Q (i) ,K (i) ,V (i) )

[0116] MHA(h t ) = [head1;...; head N ]W o

[0117] where represents the trainable output projection matrix.

[0118] The feed-forward neural network is another important component, which is usually composed of two linear transformations and a ReLU activation function for non-linear transformation, as follows:

[0119] FFN(h t) = ReLU(h t W1+b1)W2+b2

[0120] wherein denotes the embedding dimension of the hidden layer in the feedforward neural network.

[0121] Finally, a multilayer perceptron (MLP) decoder with three linear layers is adopted to extract high-level feature representations. The dimensions of the hidden layers are 4096, 1024, and 256, respectively, to gradually reduce the feature dimension to capture hierarchical information, while a ReLU activation function is directly added at each layer. Subsequently, a Softmax function is used for the output of the last linear layer to calculate the probability distribution of various intent categories, so that the model can classify the input into a specific intent type with a probability confidence.

[0122]

[0123] wherein The final output P(y|x) is the predicted probability of each intent category.

[0124] An intelligent cockpit driver intent prediction system fusing scene semantics, comprising:

[0125] A data acquisition module is configured to acquire cockpit interaction data and driving dynamic data, extract data sequences of a specified time length using a sliding window method, and construct a driver interaction intent dataset.

[0126] A scene semantic text conversion module is configured to construct an instruction fine-tuning dataset for scene semantic understanding based on logical rules and manual inspection, and convert driving dynamic data into scene semantic text by fine-tuning a large language model.

[0127] An output driver interaction intent module is configured to establish a deep learning model containing a scene-guided attention fusion mechanism, train the deep learning model using the constructed driver intent dataset, and output the most likely driver interaction intent.

[0128] Specific implementation examples:

[0129] As shown in the accompanying Figure 4 The input cockpit interaction data X (c) is a 6*24 matrix containing information such as door status (closed), gear status (changed from D to P), window status (right front and right rear windows changed from closed to open), air conditioning status (changed from open to closed), seat heating status (changed from closed to open), etc. This data sample reflects interaction action features including vehicle stop during the journey and driver opening the window, constitutes typical pre-action for driver parking and resting, and further uses the LSTM network of the cockpit interaction data encoding module to use the formula Encode the cabin interaction data into a 6*768 time sequence feature vector

[0130] The scene semantic text expresses the key scene elements of long-distance driving. The word embedding Word2Vec model in the scene semantic encoding module uses the formula Encode the text into a 1*300 text feature vector And further transform into a 6*768 text feature vector h through the formula text .

[0131] In the intent prediction module, the cabin interaction data feature vector and the text feature vector h text are fused through the proposed scene-guided attention fusion mechanism: to obtain the fused feature vector h t . Further, the multi-head attention mechanism MHA and the feedforward neural network FFN are used to deeply embed and update the feature vector, establishing a cross-modal association between the cabin interaction data and the scene semantic information.

[0132] The final multilayer perceptron MLP and Softmax function use the formula to decode the feature vector h t , obtaining a 1*C intent prediction probability distribution value. C is 3, representing music, seat, and air conditioner, respectively. The sample intent prediction probabilities are: music 0.08; seat 0.84; air conditioner 0.08. The final accurate prediction is that the driver's next step is to adjust the seat posture for rest. The introduction of scene semantic information enhances the confidence of the intent prediction result and provides an interpretable source of intent motivation.

[0133] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any modification, equivalent replacement, and improvement within the technical scope disclosed in the present application, which is within the spirit and principles of the present application, should be covered within the protection scope of the present application. Meanwhile, the contents not described in detail in the specification are all existing technologies known to those skilled in the art.​

Claims

1. A method for predicting driver intention in an intelligent cockpit by integrating scene semantics, characterized in that: The following steps are involved: Step 1: Collect cockpit interaction data and driving dynamic data, and use the sliding window method to extract data sequences of a specified time length to construct a driver interaction intention dataset; Step 2: Build a fine-tuning dataset of instructions for scene semantic understanding based on logical rules and manual verification. Convert driving dynamic data into scene semantic text by fine-tuning the large language model. Step 3: Establish a deep learning model that includes a scene-guided attention fusion mechanism, use the constructed driver intention dataset to train the deep learning model, and output the most likely driver interaction intention.

2. The method for predicting driver intention in an intelligent cockpit by integrating scene semantics according to claim 1 is characterized by: In step 1, cockpit interaction data and driving dynamic data are collected, and a sliding window method is used to extract data sequences of a specified time length to construct a driver interaction intention dataset. The specific contents are as follows: The collected data consists of two parts: cockpit interaction data collected through intelligent cockpit buried point signals, including driver-cockpit interaction actions and cockpit equipment status information; driving dynamic data collected through on-board multi-source sensors and CAN bus, including vehicle driving status and environmental status information; Filter out cockpit interaction data features and driving dynamic data features related to intent labels; Use sliding window sampling method to construct input data samples; The driver's action at the next moment in the time window is used as the classification label; Construct a driver interaction intention prediction dataset.

3. The method for predicting driver intention in an intelligent cockpit by integrating scene semantics according to claim 2 is characterized by: The signal acquisition frequency is 0.1Hz; 24-dimensional cockpit interaction data features, including door, window, air conditioning, seat, and music status, as well as 12-dimensional driving dynamic data features, including time, temperature, weather, mileage, and speed, are screened out; The sliding window sampling method with a time window length of 60 seconds is used to construct the input data sample.

4. The method for predicting driver intention in an intelligent cockpit by integrating scene semantics according to claim 1 is characterized by: In step 2, we construct a fine-tuning dataset of instructions for scene semantic understanding based on logical rules and manual verification. We then convert driving dynamic data into scene semantic text by fine-tuning the large language model. The details are as follows: Build a scene classification logic rule library based on the scene element definition to verify and correct the original scene semantic text generated by the pre-trained LLM; Then, regular expression-based pattern matching verification and manual verification are performed to ensure the accuracy and rationality of the generated text and to build a high-quality instruction fine-tuning dataset. Low-rank adaptive LoRA technology is used to efficiently fine-tune the pre-trained model.

5. The method for predicting driver intention in an intelligent cockpit by integrating scene semantics according to claim 4 is characterized by: The scene semantic text S contains four key information: time S time , vehicle S veh , environment S env , user S user : S={S time ,S veh ,S env ,S user } Time information S time Including date divisions date With time period S period ; where S date Divided into two categories: working days refer to the statutory working days from Monday to Friday; holidays include weekends and national statutory holidays; period It is divided into five time periods: 00:00-05:59 in the morning, 06:00-11:59 in the morning, 12:00-17:59 in the afternoon, 18:00-21:59 in the evening and 22:00-23:59 in the middle of the night; Vehicle Information veh Including the current average speed S speed , driving status S state Mileage S travel 、Distance S route 、Road conditions traffic , driving section S road ; Directly read the latest S through the CAN bus speed and S travel value; according to the Δv / Δt variation coefficient, S state It is divided into six categories: parking, the speed is zero and lasts for 60 seconds; steady driving, the acceleration fluctuation is ≤2m / s2; gradual acceleration or deceleration, the acceleration fluctuation is 2-4m / s2; frequent acceleration and deceleration, the acceleration sign changes ≥3 times / min; S route The distance classification standard is adopted, with 50km as the critical point between normal driving and long-distance driving; S traffic and S road Obtained through in-vehicle navigation information, divided into smooth and congested, ordinary roads with a design speed of ≤80km / h, and expressways; Environmental Information env Including weather conditions wea , vehicle interior temperature S in_car and the outside temperature S out_cur ; where S wea Call the Weatherstack API based on GPS latitude and longitude information and classify all weather conditions into three categories: sunny, rainy, and snowy; in_car and S out_car Read by temperature sensor via CAN bus; User Information user Including fatigue state S tired and distracted state S dis ; where S tired and S dis The driver is classified as being in this state or normal state, which is obtained by the driver monitoring system DMS; By leveraging the context judgment and text generation capabilities of the large language model, we can ultimately generate a structured scene semantic text S containing all the above scene elements: "In [S time ] of [S period ], the current weather conditions are [S wea ], outside temperature [S out_car ]℃, vehicle interior temperature [S in_car ]℃, the vehicle is [S speed The average speed is [S state ] stage, the cumulative travel time is [S travel ] and in [S route ] driving, the driver showed [S tired / S dis ]; According to the navigation road condition monitoring data, the vehicle has entered [S road ] driving, the current road conditions are [S traffic ]” ChatGLM4-9B is used as the basic model to realize the generation of scene semantic text. "You are a driving scene semantic analyst. You need to generate natural language description of driving scene semantic text based on real-time vehicle status data and environmental data" is used as the role setting. At the same time, the driving dynamic data time series fragments and data feature descriptions are combined as the input prompt text of LLM. p By building a logical rule library for scene element classification, it is used to verify and correct the original scene semantic text, where each rule strictly follows the above scene element definition; Then, regular expression-based pattern matching verification is performed and manually verified to ensure the accuracy and rationality of the generated text, and finally a high-quality instruction fine-tuning dataset D is constructed. tuning ={(X p , S)}; Use low-rank adaptation (LoRA) technology to efficiently fine-tune the basic model; Specifically, given the pre-trained weight matrix Coupled with the low-rank decomposition matrix ΔW = BA, where And r<<min(d,k); Matrix A is initialized with random Gaussian and Matrix B is initialized with zero to ensure that ΔW is a zero matrix in the initial stage of training; During model inference, the original weight W0 and the low-rank matrix ΔW act on the input features simultaneously and combine the output vectors. In the Transformer architecture, the calculation formula of the original query Q, key K, and value V is modified as follows: Among them, W q0 ,W k0 ,W v0 Represents the weight matrix of the initial Q, K, V; ΔW q ,ΔW k ,ΔW v represents the weight update matrix used for fine-tuning; B q ,B k ,B v ,A q ,A k ,A v Represents the projection matrix under low rank; α represents the scaling factor, which is used to adjust the speed of updating the low rank matrix; r represents the rank of the low rank matrix.

6. The method for predicting driver intention in an intelligent cockpit by integrating scene semantics according to claim 1 is characterized by: In step three, a deep learning model with a scene-guided attention fusion mechanism is established. The deep learning model is trained using the constructed driver intention dataset to output the most likely driver interaction intention. The details are as follows: The deep learning model is divided into a cockpit interaction data encoding module, a scene semantic encoding module, and an intention prediction module; The cockpit interaction data encoding module uses a long short-term memory (LSTM) network as its core architecture. Using memory cells and a gating mechanism, the LSTM updates the internal state of the memory cell at each time step t based on the current input and the hidden state of the previous time step. The final output hidden state contains important feature information from both the current moment and previous moments. The scene semantic encoding module uses the Word2Vec model and the Tencent AI Lab word embedding corpus to convert scene semantic text into a continuous neural word embedding representation. It then transforms the text embedding vector through a linear layer to obtain a scene semantic embedding vector, aligning the embedding dimension with the cockpit interaction state encoding vector. Intent prediction module: First, the encoded cockpit interaction data and structured scene semantic vectors are used as input. By establishing a scene-guided attention fusion mechanism, the scene semantics are injected as global context information into the multi-head attention calculation process, and multimodal feature fusion is achieved by dynamically adjusting the attention weight distribution. Subsequently, multi-head attention concatenation, residual connection and feedforward neural network calculation are performed; Finally, a multilayer perceptron decoder with three linear layers is used to extract high-level feature representations; A softmax function is used on the output of the last linear layer to compute the probability distribution of various intent categories, enabling the model to classify the input as a specific intent type with probabilistic confidence.

7. The method for predicting driver intention in an intelligent cockpit by integrating scene semantics according to claim 6 is characterized by: In the cockpit interaction data encoding module, LSTM uses memory units and gating mechanisms to calculate the current input at each time step t. and the hidden value of the previous time step The state updates the internal state of the memory unit; the hidden state of the final output It contains important feature information of the current moment and historical moments, can effectively analyze and model cockpit interaction time series data, and provides a high-quality encoding representation foundation for subsequent intention prediction tasks; Among them, X (c) Represents the cockpit interaction data input at the current time step, represents the output hidden state vector of the current time step, represents the hidden state vector at the previous time step.

8. The method for predicting driver intention in an intelligent cockpit by integrating scene semantics according to claim 6 is characterized by: The scene semantic encoding module adopts a uniformly designed prompt template and generates a structured scene semantic text S = [S1, S2, ..., S n ]; To establish a semantic representation space, the scene semantic text S is converted into a continuous neural word embedding representation using the Word2Vec model and the Tencent AI Lab word embedding corpus where d p Represents the embedding dimension, which is fixed at 300; then the text embedding vector is transformed into Perform dimension transformation to obtain scene semantic embedding vector The embedding dimension is made consistent with the dimension of the cockpit interaction state encoding vector; in terms of computational efficiency optimization, the upper limit of the input text length is set to 1000; Where n represents the total number of words in the scene semantic text, S i represents the i-th word in the input text sequence, represents the trainable parameter matrix, represents the bias vector.

9. The method for predicting driver intention in an intelligent cockpit integrated with scene semantics according to claim 6, characterized in that: The intention prediction module uses an enhanced Transformer-based implementation, using encoded cockpit interaction data and structured scene semantic vectors as input to infer the driver's interaction intention. Inject scene semantics as global context information into the multi-head attention calculation process, and achieve multimodal feature fusion by dynamically adjusting the attention weight distribution; In the multi-head attention calculation of each layer, the cockpit interaction time is embedded and scene semantic embedding h text are jointly encoded into K and V, while Q retains the cockpit interaction feature encoding; the calculation formulas of Q, K, and V are formally redefined as follows: Among them, Q (i) ,K (i) ,V (i) Represent the query, key, and value vectors of the i-th attention head respectively; represents the trainable projection matrix of the i-th attention head; d k =d h / N is the dimension of each attention head, and N is the number of attention heads; The feature vector representing the cockpit interaction data; h text Feature vector representing the scene semantic text; Then, multi-head attention is connected in series and residual connection is performed; through hierarchical attention weighting and feature fusion, the intention prediction module realizes the deep coupling of scene semantic information and interaction time features, and obtains the deep moment coupling feature h t ; MHA(h t )=[head1;…;head N ]W o Among them, Attn represents the attention mechanism calculation function, head i represents the output vector of the i-th attention head, MHA(h t ) indicates the output feature h obtained after multi-head attention calculation t , represents the trainable output projection matrix; The feedforward neural network consists of two layers of linear transformation and a ReLU activation function for nonlinear transformation, as follows: FFN(h t )=ReLU(h t W1 + b1)W2+b2 in represents the trainable weight matrix of the linear transformation in the feedforward neural network; b1, b2 represent the trainable bias vector in the linear transformation; FFN(h t ) represents the feature h output by multi-head attention t Update the eigenvalue h after calculation by the feedforward neural network t ; Finally, a multi-layer perceptron (MLP) decoder with three linear layers is used to extract high-level feature representations. The hidden layers have dimensions of 4096, 1024, and 256, respectively, to progressively reduce the feature dimensionality to capture hierarchical information. ReLU activation functions are added directly to each layer. A softmax function is applied to the output of the final linear layer to calculate the probability distribution of various intent categories, enabling the model to classify the input as a specific intent type with probabilistic confidence. in, Indicates the hidden feature h after processing t Decoded into logits of C intent categories, which is an unnormalized real vector, each value corresponds to an intent category; MLP(h t ) i represents the logit value of the i-th intent category; c∈{1, 2, …, C} corresponds to each intent category, and C represents the total number of intent categories; exp(·) represents the exponential function that converts the logits output by the MLP into a positive number; P(y=i|x) represents the probability of predicting category i under input x.

10. An intelligent cockpit driver intention prediction system integrating scene semantics, characterized by: include: The data collection module is used to collect cockpit interaction data and driving dynamic data, and uses the sliding window method to extract data sequences of a specified time length to construct a driver interaction intention dataset; The module for converting scene semantic text is used to construct a fine-tuning dataset of instructions for scene semantic understanding based on logical rules and manual verification. It converts driving dynamic data into scene semantic text by fine-tuning a large language model. The driver interaction intention output module is used to establish a deep learning model that includes a scene-guided attention fusion mechanism, use the constructed driver intention dataset to train the deep learning model, and output the most likely driver interaction intention.

Images