A system and method for improving driving trust based on multimodal intelligent interaction

By collecting and analyzing the driver's multimodal data, predicting trust and recommending the optimal interaction method, the problem of insufficient adaptability of the human-computer interaction system in the autonomous driving system is solved, and the driver's trust and driving experience is significantly improved. .

CN118918558BActive Publication Date: 2024-12-03HANGZHOU DIANZI UNIV
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411408508.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-10-10
Publication Date
2024-12-03
Estimated Expiration
2044-10-10

AI Technical Summary

Technical Problem

The adaptability and intelligence of the human-computer interaction system of the existing autonomous driving system is low, and it is difficult to flexibly adjust the interaction mode according to changes in driver trust, resulting in insufficient increase in trust and hindering the widespread application of technology.

Method used

By collecting driver multimodal data (behavioral data, physiological signals and eye movement characteristics) for preprocessing and feature extraction, using large models to predict driver trust, and recommend the optimal interaction method to achieve non-invasive intelligence Melting human-computer interaction.

Benefits of technology

It significantly improves the driver's trust and safety in the autonomous driving system, reduces the driver's cognitive load, enhances the driving experience and overall driving safety. The system can continuously optimize the interaction strategy based on the driver's real-time feedback and historical data. , adapt to different drivers and situational needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918558B_ABST
    Figure CN118918558B_ABST
Patent Text Reader

Abstract

The present application belongs to the field of autonomous driving technology, and specifically relates to a system and method for improving driving trust based on multimodal intelligent interaction, the method comprising: collecting driver behavior data, physiological signals and eye movement feature data in driving scenarios with multiple human-computer interaction combinations, and performing preprocessing and feature extraction; inputting the initial data set into the pre-trained model for fine-tuning training to obtain a large model for trust prediction and intelligent interaction recommendation; updating the large model in real time based on the feedback mechanism of the memory bank and generating trust prediction results and recommended interaction methods; judging whether the results meet the requirements for improving trust, and outputting the optimal human-computer interaction method to the human-computer interaction module. The present application solves the problem in the prior art that the human-computer interaction system has a low level of intelligence and is difficult to flexibly adjust the interaction method according to changes in the driver's trust.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and specifically, relates to a system and method for improving driving confidence based on multimodal intelligent interaction. Background Art

[0002] In the current promotion and application of autonomous driving technology, users' trust in the autonomous driving system directly affects its popularization speed. As an important bridge between drivers and vehicles, the effect of the human-computer interaction system largely determines the driver's trust level. However, the existing human-computer interaction system generally has the problem of low adaptability and intelligence, and is relatively rigid. It is difficult to flexibly adjust the interaction mode according to the changes in the driver's trust, and lacks emotional intelligence. This deficiency not only reduces the driver's trust in the autonomous driving system, but also hinders the widespread application of the technology to a certain extent.

[0003] At present, although some technologies have attempted to improve the driver's sense of trust by improving the human-computer interaction method, the actual effects of these technologies are still insufficient. For example, patent CN105867595A collects the driver's voice and gesture information for recognition, and controls the human-computer interaction system accordingly to improve the interaction efficiency. Although it can improve the driver's trust to a certain extent, it requires the driver to actively participate in the interaction, which may distract attention during driving. Patent CN115027495A presents the relevant operation information of the system to the user intuitively by integrating vehicle perception information and navigation information, thereby relieving the driver's nervousness and improving trust in the system. However, this technology relies on a high level of system intelligence, which is difficult to be widely used for ordinary cars. Patent CN111813470B analyzes the text content of the human-computer interface and performs visual enhancement optimization to reduce the driver's cognitive load and improve the user experience. Although it has a certain effect on beginners, the interaction method is relatively single, the overall experience is not strong, and the technology has limited effect on non-beginners.

[0004] Therefore, in the autonomous driving smart cockpit, how to design a human-computer interaction method that can dynamically adjust according to the changes in the driver's trust, so as to improve the interaction efficiency and user experience, thereby enhancing the driver's trust in the autonomous driving system, is an important issue that needs to be solved urgently. Summary of the invention

[0005] In view of the defects of the prior art, the present invention proposes a system and method for improving driving trust based on multimodal intelligent interaction. By predicting driving trust according to the driver's physiological data and behavioral data and recommending the optimal human-computer interaction combination, non-invasive intelligent human-computer interaction is achieved, thereby implicitly improving the driver's trust in the automatic driving system, and providing ideas and basis for the human-computer interaction design of future automatic driving smart cockpits.

[0006] In one aspect, the present invention provides a method for improving driving confidence based on multi-modal intelligent interaction, comprising:

[0007] Step 1: Collect the driver's behavior data, physiological signals and eye movement feature data in various human-machine interaction combination scenarios, preprocess and extract features from the collected data, build label data based on the driver's trust score and skin galvanic response value, match the extracted feature data with the label data through timestamps, and obtain the initial data set based on driver trust prediction;

[0008] Step 2: Select the system's pre-trained model, input the initial data set obtained in step 1 into the pre-trained model for fine-tuning training, and obtain a large model for trust prediction and intelligent interactive recommendation;

[0009] Step 3: Based on step 2, a memory library of the large model is constructed, and the initial data set of step 1 is entered into the memory library. Based on the data update rules of the memory library, the training and learning data of the large model used for trust prediction and intelligent interactive recommendation in step 2 is updated in real time. Based on the feedback mechanism of the memory library, the driver's interactive experience feedback data and the system's own operation data are obtained, and the feedback data is labeled;

[0010] Step 4: Based on step 3, use the prompt template to generate trust prediction results and recommended interaction methods through the initial data set and prediction target in step 1;

[0011] Step 5: The large model determines whether the result generated by the prompt template meets the trust requirements based on the trust prediction result generated by the prompt template and the driver's actual trust score. If it does not meet the trust requirements, the large model outputs no instructions to the human-computer interaction module and inputs the analysis results into the memory library for update. If it meets the trust requirements, the large model outputs the recommended optimal human-computer interaction method instructions generated by the prompt template to the human-computer interaction module.

[0012] Step 6: The human-computer interaction module controls the corresponding human-computer interaction submodule to realize interaction with the driver according to the optimal human-computer interaction mode sent by the large model.

[0013] Furthermore, the label data in step 1 is determined by the following formula:

[0014] Label data = 0.5 × trust score + 0.5 × GSR value

[0015] Where: 0.5 represents weight; GSR represents galvanic skin response.

[0016] Furthermore, step 3 specifically includes:

[0017] Step 3.1: Classify the stored data in the memory bank, and automatically enter the initial data set in step 1 into the corresponding classification items in the memory bank according to the classification items;

[0018] Step 3.2: The data update rules of the memory database are constructed according to the update frequency and the selection of update data. The update frequency is set to real-time update after each interaction or daily scheduled batch update. The selection of update data is updated according to the data with large trust prediction error and significant driver feedback.

[0019] Step 3.3: Set the feedback mechanism of the memory bank to feedback collection and feedback processing, and collect feedback on the driver's interactive experience data, the prediction error of the system's own operation, and the interactive effect data. At the same time, mark the collected feedback data as positive feedback labels and negative feedback labels, and store them in the memory bank for model learning and optimization adjustment.

[0020] Furthermore, step 4 specifically includes:

[0021] Step 4.1: Input the problem statement: clearly state the task through direct inquiry, that is, predict the driver's trust and recommend a suitable interaction method;

[0022] Step 4.2: Input data: List the specific input data characteristics, including driving behavior data and physiological data;

[0023] Step 4.3: Output requirements: Clarify the output content that the model needs to generate, including trust scores and recommended interaction methods;

[0024] Step 4.4: Output format: Provide specific output format requirements, including "trust score" and "recommended interaction method".

[0025] Furthermore, the driving behavior data in step 1 include the number of brakes, the braking force and the steering wheel angle; the physiological signals include the mean value of the galvanic skin response, the maximum value of the galvanic skin response, the mean heart rate and the maximum heart rate.

[0026] On the other hand, the present invention also provides a system for improving driving confidence based on multimodal intelligent interaction for implementing the above method, comprising:

[0027] The data acquisition and processing module is used to obtain the driver's behavior data, physiological signals and eye movement feature data in real time during driving, and pre-process and extract features of the acquired data before sending it to the intelligent prediction learning module;

[0028] The intelligent prediction learning module is used to analyze the data sent by the data acquisition and processing module using the fine-tuned large model, generate a predicted value of driving trust, recommend a suitable interaction method based on the predicted value, and send instructions to the human-computer interaction module;

[0029] The human-computer interaction module is used to control the corresponding sub-interaction modules to perform human-computer interaction in a combined manner based on the instructions of the intelligent prediction learning module.

[0030] Furthermore, the data acquisition and processing module includes: a brake pedal sensor, a steering wheel angle sensor, an eye tracker and a physiological bracelet.

[0031] Furthermore, the brake pedal sensor is used to collect the force and frequency of the brake pedal; the steering wheel angle sensor is used to measure the rotation angle and speed of the steering wheel; the eye tracker is used to track the driver's eye movement and gaze point; the physiological bracelet is used to monitor heart rate, blood pressure, skin electrical response value and related physiological indicators in real time.

[0032] Furthermore, the intelligent prediction learning module includes: a memory bank and a large model; the memory bank is used to store the historical data of the large model's analysis and prediction of the driver; the large model is used to process and analyze the historical data stored in the memory bank and the data acquisition and processing module, and generate a predicted value of driving confidence through training on a large amount of driving data, and automatically recommend the most appropriate human-computer interaction method based on the predicted value, and send relevant instructions to the human-computer interaction module.

[0033] Furthermore, the sub-interaction module of the human-computer interaction module includes at least an atmosphere light, a vibration belt, an intelligent fragrance, a virtual image projection and a controller.

[0034] The beneficial effects of the present invention are:

[0035] First, the method of the system for improving driving trust based on multimodal intelligent interaction of the present invention can evaluate the driver's trust in real time and dynamically through multimodal data collection (including behavioral data, physiological signals and eye movement characteristics) and intelligent analysis and prediction of large models, and significantly improve the driver's trust in the automatic driving system by recommending the optimal interaction method, thereby enhancing driving safety and comfort; through the intelligent prediction and recommendation function of the system, the driver can obtain the most suitable interaction method for his current state without performing complex operations, thereby reducing the driver's cognitive load and improving the driving experience and overall driving safety;

[0036] Second, the system for improving driving trust based on multimodal intelligent interaction of the present invention realizes real-time monitoring and optimization of driver behavior through closed-loop feedback of intelligent prediction learning module, memory bank and human-computer interaction module, ensuring the accuracy and consistency of data. The data after each interaction will be automatically recorded and used to adjust the interaction strategy of the system, ensuring that the system continuously learns and improves during use, improving the accuracy of prediction and the effectiveness of interaction;

[0037] Third, the large model of the system of the present invention can effectively capture and learn the individual differences and behavior patterns of drivers through fine-tuning training, combined with historical data stored in the memory bank, and accurately predict the changing trend of driving trust. According to the prediction results, the system can automatically recommend and implement the most appropriate interaction method to meet the needs of different drivers in different situations, significantly improving the intelligence level of human-computer interaction; the large model also has a strong adaptive learning ability, and can continuously optimize the interaction strategy according to the real-time feedback and historical data of the driver, ensuring the efficiency and personalization of the human-computer interaction method, and solving the problems of rigidity and insufficient response of traditional human-computer interaction systems;

[0038] Fourth, the system of the present invention is designed with an intelligent memory library and feedback mechanism, which can effectively manage and process a large amount of driving data, update the model in real time and make optimization adjustments to ensure that the system is always in the best working state, thereby improving the accuracy and reliability of the overall interaction;

[0039] Fifth, in the preferred implementation, the present invention integrates multiple sub-interaction modules such as atmosphere lights, vibration belts, smart fragrances, and virtual image projections. The system can flexibly combine different interaction methods, which can not only enhance the interaction effect through vision, touch, and smell, but also increase the affinity and credibility of the system through virtual image projection, effectively enhancing the driver's trust and sense of security in the automatic driving system;

[0040] Sixth, in the preferred implementation, the human-computer interaction module design of the system of the present invention is highly flexible. It not only includes basic ambient lights, vibration belts, etc., but can also be expanded to more types of interactive devices according to actual needs, such as smart seat systems and smart sunshade systems. This design enables the system to adapt to the needs of different vehicle models and drivers, and has strong market promotion potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a structural schematic diagram of a system for improving driving confidence based on multimodal intelligent interaction according to an embodiment of the present invention;

[0042] Figure 2 is a flow chart of a method for improving a driving confidence system based on multi-modal intelligent interaction according to an embodiment of the present invention;

[0043] Figure 3 It is a flowchart of a data set construction method for a driving confidence enhancement system based on multimodal intelligent interaction according to an embodiment of the present invention.

[0044] Among them, 110-data acquisition and processing module; 111-brake pedal sensor; 112-steering wheel angle sensor, 113-eye tracker; 114-physiological bracelet; 120-intelligent prediction learning module; 130-human-computer interaction module; 131-atmosphere light; 132-vibration belt; 133-intelligent fragrance; 134-virtual image projection; 135-controller. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the technical solution of the present application, the present application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0046] In the description of this application, the terms "first", "second", and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance; the term "plurality" refers to two or more, unless otherwise expressly defined. Terms such as "installed", "connected", "connected", and "fixed" should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; "connected" can be a direct connection or an indirect connection through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to the specific circumstances.

[0047] In the description of the present application, it should be understood that the directions or positional relationships indicated by the terms "up", "down", "left", "right", "front", "back", etc. are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or unit referred to must have a specific direction, be constructed and operated in a specific orientation, and therefore, should not be understood as a limitation on the present application.

[0048] In the description of this specification, the description of the terms "one embodiment / method", "some embodiments / methods", "specific embodiments / methods", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments / methods or examples in a suitable manner.

[0049] The present invention records a system and method for improving driving trust based on multimodal intelligent interaction. By predicting driving trust based on the driver's physiological data and behavioral data and recommending the optimal human-computer interaction combination, non-invasive intelligent human-computer interaction is achieved, thereby implicitly improving the driver's trust in the automatic driving system, and providing ideas and basis for the human-computer interaction design of future automatic driving smart cockpits.

[0050] Example 1

[0051] As the instruction manual Figure 1 This embodiment provides a system for improving driving confidence based on multimodal intelligent interaction, including a data acquisition and processing module 110, an intelligent prediction and learning module 120, and a human-computer interaction module 130.

[0052] The data acquisition and processing module 110 is responsible for acquiring the driver's behavior data, physiological signals and eye movement characteristics during driving in real time, and preprocessing, feature selection and encoding fusion of these data, and then sending the processed data to the intelligent prediction learning module 120. After receiving the fused feature data, the intelligent prediction learning module 120 uses a fine-tuned large model for analysis to generate a predicted value of driving trust, recommends a suitable interaction method based on the predicted value, and sends instructions to the human-computer interaction module 130. The human-computer interaction module 130 controls the corresponding interactive devices according to the received instructions, and implements combined human-computer interaction to enhance the driver's experience and trust.

[0053] The system forms a closed loop through data collection, intelligent prediction and interactive feedback to achieve real-time monitoring and assistance of driver behavior, and continuously optimize human-computer interaction during driving.

[0054] Specifically, the data acquisition and processing module 110 includes a brake pedal sensor 111, a steering wheel angle sensor 112, an eye tracker 113 and a physiological bracelet 114. The brake pedal sensor 111 is installed on the brake pedal to detect the force and frequency of the driver's depressing the brake pedal. These data are helpful in judging the driver's driving behavior and state, such as emergency braking or smooth driving. The steering wheel angle sensor 112 is installed on the steering wheel shaft to measure the rotation angle and speed of the steering wheel. By analyzing the rotation of the steering wheel, the driver's steering operation and driving path can be understood. The eye tracker 113 is usually installed in front of the driver's seat to track the driver's eye movements and gaze points. For example: the number, frequency, and gaze duration of looking at the road and other non-driving behavior areas (looking at the central control screen and instrument panel). By analyzing the eye movement data, the driver's concentration and fatigue state can be evaluated. The physiological bracelet 114 is worn on the driver's wrist to monitor physiological indicators such as heart rate (HR), blood pressure, and galvanic skin response (GSR) value in real time. These physiological data help to judge the driver's emotional state such as tension, anxiety or relaxation.

[0055] The data acquisition and processing module 110 is responsible for preprocessing the collected data, including removing noise, filling missing values, and performing preliminary feature extraction. This process uses conventional technical means, and the specific steps are as follows: First, in the data preprocessing stage, the original data is denoised, filtered, and normalized to eliminate interference factors and ensure data consistency. Then, the system performs feature selection on the preprocessed data, and extracts the key features that best reflect the driver's status and behavior through correlation analysis, principal component analysis (PCA) or other machine learning algorithms. Finally, in the coding fusion stage, data of different modes are uniformly represented, and various types of feature data are fused into a comprehensive feature vector through vectorization, embedding or other coding techniques to provide high-quality input for subsequent intelligent prediction learning.

[0056] In the present invention, the data collection scope can be expanded and more types of sensors can be added to further enrich the data dimension, such as temperature sensors, humidity sensors, etc.

[0057] The intelligent prediction learning module 120 includes a memory bank and a large model. The memory bank is used to store the historical data of the large model's analysis and prediction of the driver, including past driving behavior, sensor data, and system interaction records. The large model is used to generate a predicted value of driving trust based on the historical data stored in the memory bank and the data processing and analysis of the data acquisition and processing module 110, and automatically recommend the most appropriate human-computer interaction method based on the predicted value, and send relevant instructions to the human-computer interaction module 130.

[0058] It should be noted that in order to ensure the accuracy of the prediction, the large model needs to be fine-tuned before it is deployed. The fine-tuning process includes retraining or tuning the model parameters obtained by training the large model in general scenarios according to data from specific application scenarios (such as driving style, reaction time, and acceptance of system commands, etc.). This process can better adapt to the individual differences of different drivers.

[0059] The large model of the present invention can learn the complex relationship between multimodal data and accurately predict the driver's behavior pattern and trust change trend. The memory library enhances the prediction ability and response speed of the large model. The large model combines the memory library with the comparative analysis of current data to achieve accurate prediction of the driver's trust and improve the accuracy and reliability of the prediction.

[0060] The human-computer interaction module 130 includes an ambient light 131, a vibration belt 132, an intelligent fragrance 133, a virtual image projection 134, and a controller 135. The ambient light 131 provides flashing or brightness changes of multiple colors and frequencies as a simple visual interaction for the driver. The vibration belt 132 is installed on the driver's seat or steering wheel, and can be separated from the left and right or vibrate simultaneously, with adjustable amplitude and frequency, to remind the driver's attention through vibration feedback. For example, when it is detected that the driver's attention is distracted, the vibration belt will vibrate to attract the driver's attention. The intelligent fragrance 133 adjusts the odor environment in the car through a release device, such as releasing a refreshing fragrance when the driver is tired and releasing a soothing fragrance when nervous, to help the driver maintain the best state. The virtual image projection 134 is used to project an anthropomorphic cartoon image, communicate with the driver through visual and voice interaction, and enhance the driver's trust in the decision-making intention of the automatic driving system. The controller 135 is the core component of the human-computer interaction module 130, responsible for coordinating and managing the operations of each submodule in the human-computer interaction module. The controller 135 receives information from the intelligent prediction learning module 120 and adjusts the color, frequency and brightness of the ambient light to enhance the driver's visual experience and improve their trust in the autonomous driving system. When the driver is detected to be distracted or fatigued, the controller 135 can control the vibration mode of the vibration belt (separate left and right or simultaneous vibration) as well as the intensity and frequency of the vibration, and send tactile feedback to the driver to remind him to stay focused. According to the driving environment and the driver's state, the controller 135 automatically selects and releases appropriate fragrances to adjust the odor environment in the car to help the driver stay alert or relaxed. The controller 135 manages the content and interaction method of the virtual image projection, and improves the affinity and credibility of the system through visual and voice interaction with the driver.

[0061] It should be noted that the human-computer interaction module 130 of the present invention is not limited to the atmosphere light 131, the vibration belt 132, the intelligent fragrance 133 and the virtual image projection 134. According to actual needs, other types of human-computer interaction modules can also be added, such as an intelligent seat system. According to the driver's behavior data, physiological signals and eye movement feature data, the intelligent seat system automatically controls the seat adjustment, heating, ventilation and massage functions to improve driving comfort and reduce driving fatigue; the intelligent sunshade system controls the light inside the car by automatically adjusting the transparency of the windows according to the driver's behavior data, physiological signals and eye movement feature data, providing the driver with better privacy and comfort.

[0062] The following describes the specific impact of different interaction modes on the driver's trust through specific embodiments, and explains how the system dynamically adjusts the interaction mode in different driving scenarios.

[0063] The ambient light 131 directly affects the driver's mood and attention through visual effects. For example, when the driver feels nervous or uneasy, the controller 135 controls the ambient light 131 to dim the light or use soft colors (such as blue or green) to create a relaxing environment to relieve the driver's anxiety. This visual stimulation can make the driver feel more at ease, thereby enhancing trust in the autonomous driving system. Example scenario: When driving at night, the system detects that the driver's heart rate increases, indicating that he may feel tired or nervous. At this time, the ambient light 131 is converted to a soft blue based on the control instruction of the controller 135, and gradually reduces the brightness to help the driver relax.

[0064] The vibration belt 132 enhances the driver's attention and alertness through tactile feedback. When the driver's attention is detected to be distracted, the controller 135 controls the vibration belt 132 on the steering wheel or seat to emit a short and strong vibration to remind the driver to refocus on the driving task. Different vibration modes (such as unilateral vibration or synchronous vibration) can convey different warning messages. Example scenario: When driving on a highway, the system detects that the driver's eye movement frequency decreases, indicating that his attention may be decreasing.

[0065] Smart fragrance 133 directly affects the driver's physiological and psychological state through odor stimulation. For example, when the data acquisition and processing module 110 detects that the driver's physiological indicators (such as heart rate or skin galvanic response GSR value) indicate that he is in a state of tension or stress, a fragrance with a calming effect (such as lavender or citrus fragrance) can be released to help him relax. When fatigue is detected, the controller 135 controls the smart fragrance 133 to release a fresh and refreshing fragrance (such as mint or lemon) to help the driver stay awake. Example scenario: During a long-distance drive, the data acquisition and processing module 110 monitors that the driver's heart rate is gradually decreasing, indicating that he may be beginning to feel tired. At this time, the smart fragrance 133 will release a mint aroma to boost the driver's spirits and help him stay awake.

[0066] The virtual image projection 134 enhances the affinity and credibility of the system through visual and voice interaction with the driver. An anthropomorphic virtual image can communicate with the driver in a natural and friendly way, such as reminding the driver to pay attention to road conditions, answering the driver's questions, or providing driving advice. Example scenario: When driving on a city road, the virtual image projection 134 appears in front of the driver's line of sight, reminding the driver of the upcoming intersection and informing the driver of the best driving route by voice.

[0067] In order to maximize the user experience, the system can dynamically adjust the combination of the above interaction methods according to the driver's real-time status and driving environment. When the driver is obviously tired, the human-computer interaction module 130 can dim the ambient light, start the seat vibration belt, release fresh fragrance, and give friendly reminders through virtual images. Through comprehensive multi-sensory stimulation, the driver's trust and driving safety can be effectively improved.

[0068] Comprehensive example scenario: While driving on the highway in the evening, the system detects that the driver's attention begins to decline. First, the ambient light 131 gradually adjusts the light in the car, then the vibration belt 132 of the steering wheel sends a vibration reminder, then the smart fragrance 133 releases a wisp of fresh lemon fragrance, and the virtual image projection 134 reminds the driver to take a proper rest. All these interactions effectively improve the driver's alertness and trust in the system without disturbing the driver. Through these dynamic and multimodal interaction methods, the system can not only respond to the driver's state changes in real time, but also continuously optimize his driving experience, ensuring that the driver's trust in the system is continuously improved in various driving scenarios.

[0069] Example 2

[0070] As the instruction manual Figure 2 This embodiment also provides a method for improving a driving confidence system based on multi-modal intelligent interaction, including:

[0071] Step 1: Construct the initial data set: Collect the driver's behavioral data, physiological signals and eye movement feature data in various human-machine interaction combination scenarios, preprocess and extract features from the collected data, construct label data based on the driver's trust score and skin electrical response value, match the extracted feature data with the label data through timestamps, and obtain the initial data set based on driver trust prediction.

[0072] It should be noted that the initial data set constructed in step 1 is only used for pre-training micro-training (the pre-training model is different from the large model of the intelligent prediction learning module 120) before system deployment. During subsequent driving, the data acquisition and processing module 110 proposed in the present invention will be used to collect and process data in real time, and the processed and fused data will be input into the intelligent prediction learning module 120 for predictive analysis.

[0073] As the instruction manual Figure 3 , step 1 specifically includes:

[0074] Step 1.1: The driver performs driving operations, handles emergencies, and responds to system instructions in each human-computer interaction combination. The system collects driving behavior data, physiological signals, and eye movement feature data in real time, and labels the collected data according to different human-computer interaction combinations.

[0075] The driving behavior data is ultimately expressed as vehicle status data, including: number of brakes, braking force, and steering wheel angle; physiological signals include the average skin electrical response, the maximum skin electrical response, the average heart rate, and the maximum heart rate.

[0076] Step 1.2: At the end of each driving task, the driver scores the system based on his or her trust in the system to obtain subjective trust data. The system collects the skin electrical response values ​​during driving in real time to obtain objective physiological response data. The subjective trust data and objective physiological response data are weighted and fused to obtain label data.

[0077] The label data is determined by the following formula:

[0078] Label data = 0.5 × trust score + 0.5 × GSR value

[0079] Where: 0.5 represents weight; GSR represents galvanic skin response.

[0080] It should be noted that in the initial data set, subjective trust data is used as the basic setting value, and the system reliability, safety, response speed and other aspects are evaluated based on predefined scoring criteria, which is used for fine-tuning the pre-training model before system deployment. In order to ensure the accuracy of the scoring, the system is equipped with a feedback mechanism. After the system is deployed, the mechanism allows the driver to review and adjust the previous score after each drive to ensure that the score can truly reflect the changes in his trust. At the same time, as the driver's familiarity with the system increases, the scoring criteria are dynamically adjusted to reflect the gradual changes in the driver's trust.

[0081] Step 1.3: Add a timestamp indicating the specific time point of data collection to each data point in the feature data extracted in step 1.1 and the trust label data generated in step 1.2, so that the feature data and the trust label data are preliminarily aligned. The entire driving process is divided into several time periods. In each time period, the feature data and the trust label data cover the same time range. For each time period, the timestamp of the feature data and the timestamp of the trust label data are accurately matched using a data alignment algorithm to obtain an initial data set.

[0082] Step 2: Select the system's pre-trained model, input the initial data set obtained in step 1 into the pre-trained model for fine-tuning training, and obtain a large model for trust prediction and intelligent interactive recommendation.

[0083] In this embodiment, the pre-trained model uses the Baichuan2 large model, and the large model after fine-tuning training is the Trust-Baichuan2 large model.

[0084] It should be noted that the training process in step 2 is a commonly used technical means, so it will not be described in detail here.

[0085] Step 3: Based on step 2, build a memory library of the large model, enter the initial data set of step 1 into the memory library, and based on the data update rules of the memory library, update the training and learning data of the large model used for trust prediction and intelligent interactive recommendation in step 2 in real time. Based on the feedback mechanism of the memory library, obtain the driver's interactive experience feedback data and the system's own operation data, and label the feedback data.

[0086] Specifically include:

[0087] Step 3.1: Classify the stored data in the memory bank, and automatically enter the initial data set in step 1 into the corresponding classification items in the memory bank according to the classification items.

[0088] The data stored in the memory include the input data of each interaction, the confidence of the model prediction, the recommended interaction method, the actual interaction results and the driver's feedback. The data storage format uses a database suitable for large-scale data storage and fast retrieval, such as a NoSQL database. The initial data set of step 1 includes historical data import and data formatting. Historical data includes experimental data and test data collected by the initial data set. Data formatting needs to ensure that all data is stored in a predefined format, including timestamps, input features, prediction results, recommended interaction methods and feedback.

[0089] Step 3.2: The data update rules of the memory library are constructed according to the update frequency and the selection of update data. The update frequency is set to real-time update after each interaction or daily scheduled batch update. The selection of update data is updated according to the data with large trust prediction error and significant driver feedback.

[0090] In this embodiment, real-time updating after each interaction is selected to ensure the real-time reliability of the system.

[0091] Data with large trust prediction errors refer to the trust values ​​and recommended interaction methods predicted based on the input data collected by the driver during real-time driving, which have large errors compared with the data in the original memory bank. The data of this prediction and recommendation will be updated and recorded. Data with significant driver feedback refers to the relevant behavioral signal data reflected by the driver after this prediction and recommendation, as well as the rating feedback data given by the driver, which will be updated and recorded.

[0092] Step 3.3: Set the feedback mechanism of the memory bank to feedback collection and feedback processing, and collect feedback on the driver's interactive experience data, the prediction error of the system's own operation, and the interactive effect data. At the same time, mark the collected feedback data as positive feedback labels and negative feedback labels, and store them in the memory bank for model learning and optimization adjustment.

[0093] In order to obtain the driver's interactive experience feedback data, this embodiment will collect it through a scoring system (0-10 points). The design of the scoring system includes:

[0094] Rating collection interface: After each driving task or interaction is completed, the system will automatically pop up a rating interface, asking the driver to rate their experience of this interaction. The rating range is from 0 to 10 points, 0 points means completely dissatisfied or completely untrusted, and 10 points means completely satisfied or completely trusted.

[0095] Scoring criteria: To ensure the consistency and operability of the scoring, the system will provide drivers with scoring reference criteria. For example, the system will give scoring suggestions based on multiple dimensions such as the smoothness of the interaction, the accuracy of the feedback, and the response speed of the system to help drivers make accurate scoring.

[0096] Data recording and storage: The collected rating data will be time-stamped and associated with the interaction record, and then stored in the memory. The data point of each rating will include the rating value, the corresponding driving scenario, the interaction method and other relevant contextual information.

[0097] Positive feedback: If the driver scores 7 points or above for the interaction experience, the system marks the score as positive feedback. This means that the interaction method is relatively successful and has a high degree of trust. The system uses the positive feedback data to further consolidate and optimize the existing model. For example, the system can conduct more training on these interaction methods to further improve the prediction accuracy of the model.

[0098] Negative feedback: If the driver scores 6 points or less for the interaction experience, the system will mark the score as negative feedback. This means that the interaction method did not meet expectations, and there may be insufficient trust or poor interaction experience. Negative feedback will prompt the system to retrospectively analyze the specific context of the interaction and adjust model parameters or strategies to reduce negative experiences in similar situations in the future.

[0099] Storage of feedback labels: The processed positive or negative feedback labels will be stored in the memory bank together with the original interaction data for subsequent model optimization and system adjustment. By analyzing a large amount of feedback data, the system can gradually improve its ability to predict driver trust and the accuracy of interactive recommendations.

[0100] Step 4: Based on step 3, use the prompt template to generate trust prediction results and recommended interaction methods through the initial data set and prediction target in step 1.

[0101] Specifically include:

[0102] Step 4.1: Enter the problem statement: The task is clearly stated through direct questioning, that is, predicting the driver's trust and recommending a suitable interaction method.

[0103] This problem statement guides the model to focus on the relationship between the input data and the prediction target.

[0104] Step 4.2: Input data: Specific input data features are listed, including driving behavior data (such as number of brakes, steering wheel angle) and physiological data (such as GSR value, heart rate).

[0105] In this embodiment, these input data are represented by Placeholders (such as {brake_count}, {hr_mean}). In actual applications, these Placeholders will be replaced by specific values ​​collected in real time.

[0106] Step 4.3: Output requirements: Clarify the output content that the model needs to generate, including trust scores and recommended interaction methods.

[0107] It should be noted that the output requirements are prediction targets, including high-precision trust scores and dynamic human-computer interaction method recommendations.

[0108] Specifically, outputting high-precision trust scores requires ensuring that the large model can accurately predict the driver's current trust score based on the input driving behavior and physiological data, and through training on a large amount of historical data, the model learns the relationship between different driving behaviors and physiological reactions and trust. The accuracy of the trust score should meet a specific error range (such as ±0.5 points) to ensure the practicality of the score.

[0109] Outputting dynamic human-computer interaction mode recommendations requires the big model to recommend the best combination of interaction modes based on real-time data. This recommendation should have the ability to dynamically adjust and adapt to changes in the driving environment. The recommended interaction mode should take into account the driver's current state, environmental conditions, and the availability of system resources. By listing specific input data features in the prompt, the big model can analyze the current environment and driver status and generate an optimal combination of interaction modes, which includes ambient lighting, vibration belts, smart fragrances, and virtual image projection.

[0110] Step 4.4: Output format: Provide specific output format requirements, such as "trust score" and "recommended interaction method", to ensure that the results of the model output can be directly applied to subsequent decision-making or further processing.

[0111] A specific embodiment includes:

[0112] Problem description: Please predict the driver's current driving confidence based on the following driving behavior and physiological data, and recommend the most suitable human-computer interaction method.

[0113] Input data:

[0114] - Number of brakes: {brake_count}

[0115] - Brake force: {brake_force}

[0116] - Steering wheel angle: {steering_angle}

[0117] - Number of times you gazed at the road: {road_gaze_count}

[0118] - Number of gazes at other areas: {other_gaze_count}

[0119] - Road gaze frequency: {road_gaze_frequency}

[0120] - Gaze other area frequency: {other_gaze_frequency}

[0121] - Road gaze duration: {road_gaze_duration}

[0122] - Gaze duration in other areas: {other_gaze_duration}

[0123] - Galvanic Skin Response (GSR) mean: {gsr_mean}

[0124] - Heart rate (HR) mean: {hr_mean}

[0125] Please output the following results:

[0126] 1. The driver’s current driving trust score (0-10 points).

[0127] 2. Recommended optimal human-computer interaction methods (including combinations of ambient lighting, vibration belts, smart fragrances, and virtual image projection).

[0128] Output format:

[0129] Trust score: {trust_score}

[0130] Recommended interaction method: {interaction_recommendation}

[0131] During driving, the driver-related data collected in real time will be input into the big model in the format of the prompt template. The big model performs reasoning and analysis based on this data, and outputs the prediction results and recommended interaction methods according to the requirements of the prompt template. This design ensures that the big model can process data in a standardized way in different scenarios and generate consistent and accurate prediction results.

[0132] Step 5: The large model combines the trust prediction result generated by the prompt template with the driver's actual trust score to determine whether the result generated by the prompt template meets the requirements for improving trust. If it does not meet the requirements for improving trust, the large model outputs no instructions to the human-computer interaction module and inputs the analysis results into the memory library for update. If it meets the requirements for improving trust, the large model outputs the recommended optimal human-computer interaction method instructions generated by the prompt template to the human-computer interaction module.

[0133] A specific embodiment includes:

[0134] Before trust prediction and interactive recommendation are performed based on the input data, the trust score is 7 points. In step 4, the prompt template obtains the relevant data and predicts a trust score of 9 points, which means that the requirements for improving trust are met, so the analysis results of the large model are output according to a trust score of 9 points.

[0135] Step 6: The human-computer interaction module controls the corresponding human-computer interaction submodule to realize interaction with the driver according to the optimal human-computer interaction mode sent by the large model.

[0136] Specifically, the controller of the human-computer interaction module receives data started by the large model.

[0137] The human-computer interaction submodule of this embodiment includes an ambient light 131, a vibration belt 132, an intelligent fragrance 133, and a virtual image projection 134. In this embodiment, it is assumed that the output is three interaction modes: "the ambient light flashes dark blue, the intelligent fragrance emits the fragrance of angelica, and the virtual image projection greets the driver and performs voice interaction". The controller will control the ambient light 131, the intelligent fragrance 133, and the virtual image projection 134 in the human-computer interaction module 130 to achieve intelligent interaction. After the interaction, the driver's relevant reaction data will also be recorded in the memory bank to update the data, providing a basis for the next reasoning analysis.

[0138] The method proposed in the present invention will collect and analyze the driver's multimodal data to achieve trust prediction and intelligent recommendation of human-computer interaction combinations, thereby effectively improving the driver's driving experience. By using multimodal intelligent interaction as a bridge for communication between people and vehicles, the driver's trust in the autonomous driving system is increased, which is conducive to the popularization and promotion of autonomous driving.

[0139] The above is only an embodiment of the present invention, and the common sense such as the known specific structure and characteristics in the scheme is not described in detail here. For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the attached claims rather than the above description, so it is intended to include all changes within the meaning and scope of the equivalent elements of the claims. Any figure mark in the claims should not be regarded as limiting the claims involved.

Claims

1. A method for improving driving trust based on multi-modal intelligent interaction, characterized in that: include: Step 1: Collect the driver's behavior data, physiological signals and eye movement feature data in various human-machine interaction combination scenarios, preprocess and extract features from the collected data, build label data based on the driver's trust score and skin galvanic response value, match the extracted feature data with the label data through timestamps, and obtain the initial data set based on driver trust prediction; Step 2: Select the system's pre-trained model, input the initial data set obtained in step 1 into the pre-trained model for fine-tuning training, and obtain a large model for trust prediction and intelligent interactive recommendation; Step 3: Based on step 2, a memory library of the large model is constructed, and the initial data set of step 1 is entered into the memory library. Based on the data update rules of the memory library, the training and learning data of the large model used for trust prediction and intelligent interactive recommendation in step 2 is updated in real time. Based on the feedback mechanism of the memory library, the driver's interactive experience feedback data and the system's own operation data are obtained, and the feedback data is labeled; Step 4: Based on step 3, use the prompt template to generate trust prediction results and recommended interaction methods through the initial data set and prediction target in step 1; Step 5: The large model determines whether the result generated by the prompt template meets the trust requirements based on the trust prediction result generated by the prompt template and the driver's actual trust score. If it does not meet the trust requirements, the large model outputs no instructions to the human-computer interaction module and inputs the analysis results into the memory library for update. If it meets the trust requirements, the large model outputs the recommended optimal human-computer interaction method instructions generated by the prompt template to the human-computer interaction module. Step 6: The human-computer interaction module controls the corresponding human-computer interaction submodule to realize interaction with the driver according to the optimal human-computer interaction mode sent by the large model.

2. The method of the system for improving driving trust based on multimodal intelligent interaction according to claim 1, characterized in that: The label data in step 1 is determined by the following formula: Label data = 0.5 × trust score + 0.5 × GSR value Where: 0.5 represents weight; GSR represents galvanic skin response.

3. The method of the system for improving driving trust based on multimodal intelligent interaction according to claim 1, characterized in that: Step 3 specifically includes: Step 3.1: Classify the stored data in the memory bank, and automatically enter the initial data set in step 1 into the corresponding classification items in the memory bank according to the classification items; Step 3.2: The data update rules of the memory database are constructed according to the update frequency and the selection of update data. The update frequency is set to real-time update after each interaction or daily scheduled batch update. The selection of update data is updated according to the data with large trust prediction error and significant driver feedback. Step 3.3: Set the feedback mechanism of the memory bank to feedback collection and feedback processing, and collect feedback on the driver's interactive experience data, the prediction error of the system's own operation, and the interactive effect data. At the same time, mark the collected feedback data as positive feedback labels and negative feedback labels, and store them in the memory bank for model learning and optimization adjustment.

4. The method of the system for improving driving trust based on multimodal intelligent interaction according to claim 1, characterized in that: Step 4 specifically includes: Step 4.1: Input the problem statement: clearly state the task through direct inquiry, that is, predict the driver's trust and recommend a suitable interaction method; Step 4.2: Input data: List the specific input data characteristics, including driving behavior data and physiological data; Step 4.3: Output requirements: Clarify the output content that the model needs to generate, including trust scores and recommended interaction methods; Step 4.4: Output format: Provide specific output format requirements, including "trust score" and "recommended interaction method".

5. The method of the system for improving driving trust based on multimodal intelligent interaction according to claim 1, characterized in that: The driving behavior data in step 1 include the number of brakes, the braking force and the steering wheel angle; the physiological signals include the mean value of the galvanic skin response, the maximum value of the galvanic skin response, the mean heart rate and the maximum heart rate.

6. A system for improving driving confidence based on multimodal intelligent interaction according to any one of claims 1 to 5, characterized in that: include: A data acquisition and processing module (110) is used to obtain the driver's behavior data, physiological signals and eye movement feature data during driving in real time, and to perform pre-processing and feature extraction on the obtained data before sending it to the intelligent prediction learning module (120); An intelligent prediction learning module (120) is used to analyze the data sent by the data acquisition processing module (110) using the fine-tuned large model, generate a predicted value of driving trust, recommend a suitable interaction method based on the predicted value, and send instructions to the human-computer interaction module (130); The human-computer interaction module (130) is used to control the corresponding sub-interaction modules to perform human-computer interaction in a combined manner based on the instructions of the intelligent prediction learning module (120).

7. The system for improving driving trust based on multimodal intelligent interaction according to claim 6 is characterized in that: The data acquisition and processing module (110) comprises: a brake pedal sensor (111), a steering wheel angle sensor (112), an eye tracker (113) and a physiological bracelet (114).

8. The system for improving driving trust based on multimodal intelligent interaction according to claim 7 is characterized in that: The brake pedal sensor (111) is used to collect the force and frequency of the brake pedal; the steering wheel angle sensor (112) is used to measure the rotation angle and speed of the steering wheel; the eye tracker (113) is used to track the driver's eye movement and gaze point; and the physiological bracelet (114) is used to monitor heart rate, blood pressure, skin galvanic response value and related physiological indicators in real time.

9. The system for improving driving trust based on multimodal intelligent interaction according to claim 6, characterized in that: The intelligent prediction learning module (120) includes: a memory bank and a large model; the memory bank is used to store historical data of the large model's analysis and prediction of the driver; the large model is used to process and analyze the historical data stored in the memory bank and the data of the data acquisition and processing module (110), generate a predicted value of driving confidence by training a large amount of driving data, and automatically recommend the most appropriate human-computer interaction method based on the predicted value, and send relevant instructions to the human-computer interaction module (130).

10. The system for improving driving trust based on multimodal intelligent interaction according to claim 6, characterized in that: The sub-interaction module of the human-machine interaction module (130) comprises at least an atmosphere light (131), a vibration belt (132), an intelligent fragrance (133), a virtual image projection (134), and a controller (135).

Citation Information

Patent Citations

  • Human-machine interaction mode combing voice information with gesture information and implementation device thereof

    CN105867595A

  • A method and apparatus for optimizing an initial user interface

    CN111813470B

  • Trust enhancement method and equipment for intelligent driving system

    CN115027495A

  • Automatic driving test scene generation method based on personalized driver model

    CN114896869A

  • Evaluation method and device for intelligent driving service

    CN117078051A