An intelligent vehicle cabin adaptive interaction method, device, equipment and medium

Abstract

This invention discloses an adaptive interaction method, device, equipment, and medium for intelligent vehicle cockpits. The method includes: collecting multi-source perception data, including visual perception data, voice perception data, physiological perception data, and vehicle state perception data from drivers and passengers; cleaning, time-synchronizing, and feature-level fusion processing of the collected multi-source perception data to obtain a feature vector to be identified; analyzing the feature vector based on a trained emotion recognition model to identify the behavioral and emotional state labels of drivers and passengers, generating a comprehensive risk label; matching a corresponding cockpit interaction strategy from a preset strategy library based on the comprehensive risk label, and adjusting the cockpit state based on the cockpit interaction strategy. This technical solution can achieve the goals of improving driving safety, alleviating negative emotions, and enhancing the driving experience.

Description

Technical Field

[0001] This invention relates to the field of vehicle cockpit technology, and in particular to an adaptive interaction method, device, equipment and medium for intelligent vehicle cockpits. Background Technology

[0002] With the development of automotive intelligence, the functions of in-vehicle cockpit systems are becoming increasingly rich. However, most existing intelligent cockpit systems still have many significant shortcomings in their interaction modes, failing to achieve true intelligent human-machine interaction. These systems mostly exhibit passive response characteristics, requiring users to actively issue commands via voice, touch, etc., to trigger functions, lacking the ability to proactively perceive and serve the state of drivers and passengers. Simultaneously, these systems lack emotional intelligence, failing to effectively identify and understand the real-time emotional state of drivers and passengers, making it difficult to provide appropriate care services when drivers and passengers experience fatigue, irritability, sadness, or other negative emotions. For example, if a driver is anxious due to traffic congestion, the system might still play fast-paced music, exacerbating the negative emotions. Furthermore, interaction strategies are usually preset and universal, unable to dynamically adjust based on different users or the same user's emotions at different times, resulting in low personalization and an inability to effectively identify dangerous driving states such as driver fatigue or inattention caused by intense emotions.

[0003] The aforementioned problems with existing intelligent cockpit systems prevent them from truly meeting the actual needs of drivers and passengers in human-computer interaction. This reduces the comfort and intelligent experience of cockpit use and also has an adverse impact on driving safety. Therefore, there is an urgent need in this field to develop an intelligent cockpit solution that can truly understand users, provide proactive, precise, and humanized interaction, and improve safety. Summary of the Invention

[0004] This invention provides an adaptive interaction method, device, equipment, and medium for intelligent vehicle cockpits to improve driving safety, alleviate negative emotions, and enhance the driving experience.

[0005] According to one aspect of the present invention, an adaptive interaction method for an intelligent vehicle cockpit is provided, comprising:

[0006] Collect multi-source perception data, which includes visual perception data, voice perception data, physiological perception data and vehicle status perception data of drivers and passengers.

[0007] The collected multi-source sensing data is cleaned, time-synchronized, and feature-level fused to obtain the feature vector to be identified.

[0008] The trained emotion recognition model is used to analyze the feature vector to be identified, and the behavioral labels and emotional state labels of the driver and passengers are identified to generate a comprehensive risk label.

[0009] Based on the comprehensive risk label, the corresponding cockpit interaction strategy is matched from the preset strategy library, and the cockpit status is adjusted based on the cockpit interaction strategy.

[0010] According to another aspect of the present invention, an adaptive interaction device for an intelligent vehicle cockpit is provided, comprising:

[0011] The data acquisition module is used to collect multi-source perception data, which includes visual perception data, voice perception data, physiological perception data and vehicle status perception data of drivers and passengers.

[0012] The data processing module is used to clean, synchronize time, and fuse the collected multi-source sensing data to obtain the feature vector to be identified.

[0013] The state recognition module is used to analyze the feature vector to be identified based on the trained emotion recognition model, identify the behavior labels and emotion state labels of the driver and passengers, and generate a comprehensive risk label.

[0014] The strategy matching module is used to match the corresponding cockpit interaction strategy from the preset strategy library according to the comprehensive risk label, and adjust the cockpit status based on the cockpit interaction strategy.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor;

[0017] and memory that is communicatively connected to at least one processor;

[0018] The memory stores a computer program that can be executed by at least one processor, which enables the at least one processor to execute the intelligent vehicle cockpit adaptive interaction method of any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the intelligent vehicle cockpit adaptive interaction method of any embodiment of the present invention.

[0020] The technical solution of this invention collects multi-source sensing data; cleans, synchronizes, and fuses the collected multi-source sensing data at the time level to obtain a feature vector to be identified; analyzes the feature vector based on a trained emotion recognition model to identify the behavioral and emotional state labels of the driver and passengers, generating a comprehensive risk label; matches the corresponding cabin interaction strategy from a preset strategy library based on the comprehensive risk label, and adjusts the cabin state based on the cabin interaction strategy. This solves the problems of rigid interaction modes, inability to accurately perceive the real-time state of drivers and passengers, and lack of personalization and adaptability of interaction strategies in existing intelligent vehicle cabin systems, which can easily lead to driving safety hazards and poor driving experience. It achieves accurate perception and adaptive interaction of the cabin system with the state of drivers and passengers, improves the safety and intelligence level of the driving process, and makes the cabin state highly adapted to the state of drivers and passengers, effectively optimizing the driving experience.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart of an adaptive interaction method for an intelligent vehicle cockpit provided in an embodiment of the present invention;

[0024] Figure 2 This is a structural diagram of an intelligent vehicle cockpit adaptive interaction system provided in an embodiment of the present invention;

[0025] Figure 3 A flowchart of another intelligent vehicle cockpit adaptive interaction method provided in an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of an intelligent vehicle cockpit adaptive interaction device provided in an embodiment of the present invention;

[0027] Figure 5 A schematic diagram of the structure of an electronic device for implementing the intelligent vehicle cockpit adaptive interaction method of this invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0031] In related technologies, the following are some interaction modes for intelligent cockpit systems:

[0032] Option 1 relies on a single visual modality (face facing the camera) to perceive and judge emotions, thereby adjusting the overall vehicle environment. However, Option 1, which only relies on facial recognition, will fail when the lighting is poor, the driver is wearing a mask, or there are obstructions. Using a single visual modality to judge emotions is not accurate enough; multiple dimensions such as facial expressions, voice, heart rate, and heart rate variability are needed to judge the user's true emotions.

[0033] Option 2 involves controlling the vehicle's status based on preset scenarios (fixed music, ambient lighting, etc.). However, Option 2 relies on passive command responses, and the fixed scenario mode cannot accurately match the user's actual emotions at the time.

[0034] In summary, existing smart cockpit systems have relatively rigid interaction modes, mostly based on preset commands or simple scenario triggers (such as "home mode"). They cannot perceive the real-time psychological state of drivers and passengers, and therefore cannot provide truly personalized services that "think what you think and meet your needs." For example, the system may continue to play upbeat music even when the driver is already frustrated by traffic jams; the system may fail to intervene promptly and effectively when the driver is fatigued; and the system may not proactively provide relief when passengers are bored or uncomfortable.

[0035] This invention designs an adaptive interaction scheme for vehicle cabins that can perceive the emotional state of drivers and passengers in real time and conduct intelligent and personalized interactions accordingly. It aims to use multimodal perception technology to identify the emotional state of drivers and passengers in real time and accurately, and to automatically and flexibly adjust the cabin environment and information services, thereby improving driving safety, alleviating negative emotions, and enhancing the driving experience.

[0036] Figure 1 This is a flowchart illustrating an adaptive interaction method for an intelligent vehicle cockpit according to an embodiment of the present invention. This embodiment is applicable to situations where the cockpit environment and information services are automatically and flexibly adjusted. The method can be executed by an intelligent vehicle cockpit adaptive interaction device, which can be implemented in hardware and / or software and can be configured in a vehicle. Figure 1 As shown, the method specifically includes the following steps:

[0037] S110, Collect multi-source sensing data.

[0038] Multi-source perception data comprises various types of data collected from different dimensions, used to determine the state of drivers and passengers and their driving behavior. Multi-source perception data includes visual perception data, voice perception data, physiological perception data, and vehicle status perception data. Visual perception data can be visual data reflecting the facial, eye, and head posture of drivers and passengers; voice perception data is voice-related data of drivers and passengers; physiological perception data can be data related to the physical physiological state of drivers and passengers; and vehicle status perception data can be vehicle-side data reflecting vehicle operation and driving behavior.

[0039] Specifically, by performing multi-source perception data collection operations, the collected multi-source perception data includes visual perception data, voice perception data, physiological perception data, and vehicle status perception data of drivers and passengers. Multi-dimensional data collection can ensure the comprehensiveness of status recognition and accurately identify the status of drivers and passengers.

[0040] S120. The collected multi-source sensing data is cleaned, time-synchronized, and feature-level fusion processed to obtain the feature vector to be identified.

[0041] The feature vector to be identified can be a set of feature data formed after processing multi-source perception data, and the feature vector to be identified can be used for emotion recognition model analysis.

[0042] Specifically, the collected multi-source sensing data can be cleaned, time-synchronized, and feature-level fused sequentially to obtain the feature vector to be identified.

[0043] S130. Analyze the feature vector to be identified based on the trained emotion recognition model to identify the behavioral labels and emotional state labels of the driver and passengers, so as to generate a comprehensive risk label.

[0044] Among these, the emotion recognition model is a trained model that analyzes feature vectors to identify the state of a person. Behavioral labels can be identifiers used to describe the physical actions and movements of drivers and passengers. Emotional state labels can also be identifiers used to describe the internal psychological state of drivers and passengers. A comprehensive risk label can also be an identifier, representing the driving risk level of the driver and passengers.

[0045] Specifically, based on the trained emotion recognition model, the feature vector to be identified is analyzed. First, the behavioral labels and emotional state labels of the driver and passengers are identified. Then, the two types of labels are combined to generate a comprehensive risk label. Through model analysis, the state of the driver and passengers can be accurately determined.

[0046] S140. Match the corresponding cockpit interaction strategy from the preset strategy library according to the comprehensive risk label, and adjust the cockpit state based on the cockpit interaction strategy.

[0047] The preset strategy library can be a collection of pre-stored cockpit interaction strategies, and these strategies correspond to different risk comprehensive tags. These cockpit interaction strategies can be various strategies for adjusting cockpit status.

[0048] Specifically, based on the generated comprehensive risk label, the system can match the corresponding cockpit interaction strategy from the preset strategy library, and adjust the cockpit state according to the matched cockpit interaction strategy to achieve the adaptation of the cockpit state to the state of the driver and passengers, effectively improving the driving experience and driving safety.

[0049] The technical solution of this invention collects multi-source sensing data; cleans, synchronizes, and fuses the collected multi-source sensing data at the time level to obtain a feature vector to be identified; analyzes the feature vector based on a trained emotion recognition model to identify the behavioral and emotional state labels of the driver and passengers, generating a comprehensive risk label; matches the corresponding cabin interaction strategy from a preset strategy library based on the comprehensive risk label, and adjusts the cabin state based on the cabin interaction strategy. This solves the problems of rigid interaction modes, inability to accurately perceive the real-time state of drivers and passengers, and lack of personalization and adaptability of interaction strategies in existing intelligent vehicle cabin systems, which can easily lead to driving safety hazards and poor driving experience. It achieves accurate perception and adaptive interaction of the cabin system with the state of drivers and passengers, improves the safety and intelligence level of the driving process, and makes the cabin state highly adapted to the state of drivers and passengers, effectively optimizing the driving experience.

[0050] In some possible implementations, the acquisition of multi-source perception data includes: acquiring image information of the driver's facial expressions, eye state, and head posture through a visual acquisition component as the visual perception data; acquiring the driver's voice signals and extracting acoustic features through a voice acquisition component as the voice perception data; acquiring physiological information of the driver's heart rate, heart rate variability, and skin conductance response through a physiological sensing component as the physiological perception data; and acquiring vehicle operation data reflecting driving behavior through a vehicle bus as the vehicle state perception data.

[0051] The system includes a visual acquisition component for collecting visual information from drivers and passengers; a voice acquisition component for collecting voice signals from drivers and passengers and extracting acoustic features; and a physiological sensing component for acquiring physiological information from drivers and passengers. The vehicle bus is the internal communication bus of the vehicle, used to transmit various types of vehicle operation data.

[0052] In this embodiment, when collecting multi-source perception data, image information of the driver's face, eyes, and head posture is acquired through a visual acquisition component, and this image information is used as visual perception data. The driver's voice signal is acquired through a voice acquisition component, and its acoustic features are extracted, and these extracted acoustic features are used as voice perception data. Furthermore, the driver's heart rate, heart rate variability, and skin conductance information can be acquired through a physiological sensing component, and this physiological information is used as physiological perception data. Additionally, vehicle operation data reflecting driving behavior is acquired through the vehicle bus, and this vehicle operation data is used as vehicle status perception data.

[0053] In some possible implementations, the step of cleaning, synchronizing, and fusing the collected multi-source sensing data to obtain the feature vector to be identified includes: cleaning the multi-source sensing data by removing outliers and standardizing the data; synchronizing the cleaned multi-source sensing data by aligning the time axis; and extracting and fusing heterogeneous features from the synchronized multi-source sensing data to obtain the feature vector to be identified.

[0054] Heterogeneous feature extraction and fusion can be understood as feature processing operations. Heterogeneous feature extraction and fusion can extract features from different types of perceptual data and fuse them into overall features.

[0055] Specifically, outlier removal and standardization cleaning can be performed on the multi-source sensing data, followed by time axis alignment synchronization processing on the cleaned multi-source sensing data. Finally, heterogeneous feature extraction and fusion can be performed on the synchronized multi-source sensing data to obtain the feature vector to be identified.

[0056] In some possible implementations, the step of analyzing the feature vector to be identified based on the trained emotion recognition model to identify the behavioral labels and emotional state labels of the driver and passengers in order to generate a comprehensive risk label includes: inputting the feature vector to be identified into the emotion recognition model, and parsing the behavioral labels describing the physical operations and actions of the driver and passengers through the emotion recognition model;

[0057] The emotion recognition model is used to parse out the emotion state labels that describe the internal psychological state of the driver and passengers; the driving risk level is assessed by combining the behavior labels and the emotion state labels, and the comprehensive risk label is generated based on the assessment results.

[0058] Specifically, the feature vector to be identified can be input into the emotion recognition model first. The model can then parse out the behavioral and emotional state labels of the driver and passengers. The driving risk level assessment can then be carried out by combining the parsed behavioral and emotional state labels, and a corresponding comprehensive risk label can be generated based on the assessment results.

[0059] In some possible implementations, the step of combining the behavioral tags and the emotional state tags to assess the driving risk level and generating the comprehensive risk tag based on the assessment results includes: performing feature matching on the behavioral tags and the emotional state tags according to preset tag combination rules; determining the driving risk level of the driver and passengers based on the feature matching results, wherein the driving risk level includes low risk, medium risk, high risk, and extremely high risk; and generating the corresponding comprehensive risk tag based on the different driving risk levels.

[0060] The preset label combination rules are pre-defined rules used to determine the driving risk level corresponding to the combination of behavioral labels and emotional state labels. Feature matching compares the actual behavioral labels and emotional state labels with the preset label combination rules.

[0061] Specifically, behavioral tags and emotional state tags can be matched according to preset tag combination rules. The driving risk level of the driver and passengers is then determined based on the matching results, and a corresponding comprehensive risk tag is generated based on the determined risk level. This method, based on preset rules for feature matching and level determination, makes the generation of comprehensive risk tags more objective and standardized, avoids errors caused by subjective judgment, and improves the accuracy of risk assessment.

[0062] In some possible implementations, the step of matching the corresponding cockpit interaction strategy from the preset strategy library based on the comprehensive risk label includes: matching the comprehensive risk label with strategy labels in the preset strategy library; and selecting the optimal cockpit interaction strategy corresponding to the comprehensive risk label based on the matching degree. The cockpit interaction strategy includes cockpit environment adjustment strategy, in-vehicle entertainment control strategy, and driver assistance intervention strategy.

[0063] Among them, the environmental regulation strategy is used to adjust the environmental parameters in the cabin; the entertainment control strategy is used to control the operating status of the in-vehicle entertainment system; and the driving assistance intervention strategy is used to coordinate with the driving assistance system to provide safety services.

[0064] Specifically, when matching the corresponding cockpit interaction strategy from the preset strategy library based on the comprehensive risk label, the comprehensive risk label can be matched with the strategy labels in the preset strategy library, and then the cockpit interaction strategy corresponding to the comprehensive risk label can be filtered out based on the matching degree. The cockpit interaction strategy includes environmental adjustment, entertainment control and driving assistance intervention strategies.

[0065] In some possible implementations, adjusting the cockpit state based on the cockpit interaction strategy includes: decomposing the cockpit interaction strategy into corresponding cockpit subsystem control commands; sending the control commands to the corresponding cockpit execution components; and adjusting the cockpit's environmental parameters, entertainment playback status, and driver assistance function status through the cockpit execution components to complete the adjustment of the cockpit state.

[0066] The cockpit subsystem is the functional system within the cockpit, implementing various cockpit functions. Control commands are instructions derived from the cockpit interaction strategy and can directly drive the actions of the cockpit execution components. Cockpit execution components are various types of actuators used to execute control commands and adjust the cockpit state.

[0067] Specifically, when adjusting the cockpit state based on the cockpit interaction strategy, the cockpit interaction strategy can be decomposed into control commands for the cockpit subsystems. These decomposed control commands are then sent to the corresponding cockpit execution components. The cockpit execution components adjust the state of the cockpit's environment, entertainment, and driver assistance functions to complete the cockpit state adjustment. This allows for rapid adaptation of the cockpit state to the state of the driver and passengers, effectively improving the driving experience and driving safety.

[0068] Figure 2 This is a structural diagram of an intelligent vehicle cockpit adaptive interaction system provided in an embodiment of the present invention. This embodiment is a preferred embodiment of the above embodiments, and its specific implementation can be found in the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here. Figure 2 As shown, the system specifically includes the following modules:

[0069] Multimodal perception module: Used to collect biosignals and behavioral data of drivers and passengers. This includes:

[0070] (1) Visual perception unit: The in-vehicle camera (preferably an infrared camera) is used to collect the user's facial expressions, eye state (such as blinking frequency and eyelid opening and closing) and head posture images.

[0071] (2) Speech perception unit: The microphone array is used to collect the user's speech and analyze its acoustic characteristics (such as pitch, speech rate and loudness).

[0072] (3) Physiological signal sensing unit: It obtains the user's physiological data such as heart rate and heart rate variability by communicating with embedded sensors (such as heart rate sensor, skin conductance sensor) on the steering wheel or seat and / or with wearable devices (such as smartwatches).

[0073] (4) Vehicle Status Perception Unit: Connected to the vehicle CAN bus, it acquires vehicle data that reflects driving behavior, such as rapid acceleration, emergency braking, and steering wheel angle fluctuation data.

[0074] Central processing module: Electrically connected to the multimodal perception module, used for processing data, recognizing emotions, and generating interaction strategies. This includes:

[0075] (1) Data fusion unit: used to perform time synchronization and feature-level fusion of heterogeneous data from different sensing units to form a unified feature vector.

[0076] (2) Emotion recognition unit: Built-in emotion recognition model based on deep learning (such as the combination of convolutional neural network CNN and long short-term memory network LSTM), processes the feature vector, generates the behavior label and emotion state label of the driver and passenger, and outputs the comprehensive risk label of the driver and passenger;

[0077] (3) Strategy decision-making unit: pre-stores or generates interactive strategies corresponding to different risk states online, and calls the corresponding strategy instructions according to the identified comprehensive risk state.

[0078] Adaptive execution module: electrically connected to the central processing module, used to execute the interaction strategy. This includes:

[0079] (1) Environmental Domain Control Unit: Controls the air conditioning temperature and air volume, ambient light color and brightness, and fragrance generator according to the strategy instructions output by the central processing module. For example, if it detects that the user is driving emotionally, it will automatically turn on the air conditioning and blow air, turn on the ambient light and adjust it to a warm color, and turn on the fragrance to relieve the emotions of the driver and passengers;

[0080] (2) In-vehicle entertainment control unit: Controls the in-vehicle infotainment system according to the strategy instructions output by the central processing module, such as the selection and playback of music / podcast content, and the interactive tone and content of the voice assistant. For example, if it recognizes the user's emotional driving, it will automatically turn on light music and automatically turn on the voice assistant. The voice assistant will remind the user of the risks and comfort the user verbally, thereby relieving the emotions of the driver and passengers;

[0081] Driver Assist Control Unit: In conjunction with the Advanced Driver Assistance System (ADAS), it provides safety-related services such as rest area recommendations and simplified instrument panel displays. For example, if it detects mild driver fatigue, the central processing module outputs a prompt to activate driver assistance functions and displays recommended rest areas on the navigation system (including route, distance, and other information), while simultaneously providing voice guidance.

[0082] The following is a labeling system for driver behavior and emotional state. Table 1 shows the system of behavioral labels, which describe the driver's observable physical operations and actions. Table 2 shows the emotional state labels, which describe the driver's internal psychological state.

[0083] Table 1

[0084]

[0085] Table 2

[0086]

[0087] Table 3 below shows the comprehensive risk labeling system, which combines behavioral and emotional labels to assess and classify the overall state of drivers, providing a direct basis for system intervention.

[0088] Table 3

[0089]

[0090] In some alternative embodiments, Figure 3A flowchart illustrating another intelligent vehicle cockpit adaptive interaction method provided in an embodiment of the present invention. Figure 3 As shown, the method specifically includes:

[0091] S310 continuously collects visual, voice, physiological, and vehicle behavior data of drivers and passengers through a multimodal perception module.

[0092] S320. In the central processing module, the collected multi-source data is cleaned, aligned, and feature-fused.

[0093] S330. Using the trained emotion recognition model, analyze the fused feature vector to identify the current dominant emotion state.

[0094] S340. Based on the identified emotional state, match the optimal interaction strategy from the predefined strategy library;

[0095] S350: Through the adaptive execution module, the interaction strategy is decomposed into specific control commands and sent to the corresponding cockpit subsystems for execution;

[0096] S360 continues to collect user feedback data through the perception module after executing the strategy, evaluates the effect of this interaction, and uses it to optimize the user personalization model and strategy library.

[0097] For example, when the system recognizes through the voice perception unit that the driver's voice is rapid, the tone is high and contains complaining words, while the visual perception unit detects that the driver's brow is furrowed and the corners of the mouth are downturned, and the physiological signal perception unit detects that the heart rate is increased and the skin conductance is active, indicating a sudden braking or acceleration behavior, the system determines that the driver is in an "anxious / angry" state.

[0098] The corresponding interaction strategies include:

[0099] Command Environment Domain Controller: Adjust the ambient lighting to a soothing blue or green, and increase the air conditioning fan speed.

[0100] Command the in-vehicle entertainment control unit: automatically switch to soothing light music or natural sound music, and keep the voice assistant silent or only use the simplest and calmest language to provide necessary navigation prompts.

[0101] Command the driving assistance control unit: Actively activate the traffic jam assist function to reduce driving burden.

[0102] Compared with the prior art, the present invention has at least the following technical effects:

[0103] 1) Proactivity and foresight: The system can proactively perceive its status and provide services before user commands, realizing the transformation from "people adapting to cars" to "cars adapting to people".

[0104] 2) Interaction accuracy and humanization: Through multimodal information fusion, the accuracy of emotion recognition is high; the interaction strategy is based on a deep understanding of emotions, making it more accurate and humanized, and effectively improving user satisfaction.

[0105] 3) Significantly improve safety: It can promptly identify dangerous conditions such as driver fatigue, distraction, and road rage, and intervene through environmental adjustments and proactive suggestions to prevent problems before they occur.

[0106] 4) Possesses continuous learning capabilities: Through a closed-loop feedback mechanism, the system can learn the specific reaction patterns of different users, continuously optimize interaction strategies, and achieve true personalization.

[0107] Figure 4 This is a structural schematic diagram of an adaptive interaction device for an intelligent vehicle cockpit provided in an embodiment of the present invention. Figure 4 As shown, the device includes:

[0108] The data acquisition module 410 is used to acquire multi-source perception data, wherein the multi-source perception data includes visual perception data, voice perception data, physiological perception data and vehicle status perception data of the driver and passengers.

[0109] The data processing module 420 is used to clean, synchronize time, and fuse the collected multi-source sensing data to obtain the feature vector to be identified.

[0110] The state recognition module 430 is used to analyze the feature vector to be identified based on the trained emotion recognition model, identify the behavior labels and emotion state labels of the driver and passengers, and generate a comprehensive risk label.

[0111] The strategy matching module 440 is used to match the corresponding cockpit interaction strategy from the preset strategy library according to the comprehensive risk label, and adjust the cockpit state based on the cockpit interaction strategy.

[0112] The technical solution of this invention collects multi-source sensing data; cleans, synchronizes, and fuses the collected multi-source sensing data at the time level to obtain a feature vector to be identified; analyzes the feature vector based on a trained emotion recognition model to identify the behavioral and emotional state labels of the driver and passengers, generating a comprehensive risk label; matches the corresponding cabin interaction strategy from a preset strategy library based on the comprehensive risk label, and adjusts the cabin state based on the cabin interaction strategy. This solves the problems of rigid interaction modes, inability to accurately perceive the real-time state of drivers and passengers, and lack of personalization and adaptability of interaction strategies in existing intelligent vehicle cabin systems, which can easily lead to driving safety hazards and poor driving experience. It achieves accurate perception and adaptive interaction of the cabin system with the state of drivers and passengers, improves the safety and intelligence level of the driving process, and makes the cabin state highly adapted to the state of drivers and passengers, effectively optimizing the driving experience.

[0113] In some possible implementations, the data acquisition module 410 includes:

[0114] The visual acquisition submodule is used to acquire image information of the driver's face, eyes, and head posture through the visual acquisition component, as the visual perception data;

[0115] The voice acquisition submodule is used to acquire the voice signals of drivers and passengers through the voice acquisition component and extract acoustic features as the voice perception data;

[0116] The physiological acquisition submodule is used to acquire the heart rate, heart rate variability and skin conductance information of the driver and passengers through physiological sensing components, as the physiological sensing data.

[0117] The vehicle data acquisition submodule is used to acquire vehicle operation data reflecting driving behavior through the vehicle bus, as the vehicle state perception data.

[0118] In some possible implementations, the data processing module 420 includes:

[0119] The data cleaning unit is used to perform outlier removal and standardization cleaning processes on the multi-source sensing data.

[0120] The time synchronization unit is used to perform time axis alignment synchronization processing on the cleaned multi-source sensing data;

[0121] The feature fusion unit is used to extract and fuse heterogeneous features from the synchronized multi-source sensing data to obtain the feature vector to be identified.

[0122] In some possible implementations, the state recognition module 430 includes:

[0123] The label parsing submodule is used to input the feature vector to be identified into the emotion recognition model and parse out the behavior label and the emotion state label of the driver and passenger;

[0124] The risk assessment submodule is used to combine the behavioral tags and the emotional state tags to assess the driving risk level and generate the comprehensive risk tag based on the assessment results.

[0125] In some possible implementations, the risk assessment submodule includes:

[0126] The tag matching unit is used to perform feature matching on the behavior tag and the emotional state tag according to a preset tag combination rule;

[0127] The rating generation unit is used to determine the driving risk level of the driver and passengers based on the matching results, and generate the corresponding comprehensive risk label based on the driving risk level.

[0128] In some possible implementations, the policy matching module 440 includes:

[0129] The strategy tag matching unit is used to match the comprehensive risk tag with strategy tags in a preset strategy library;

[0130] The strategy filtering unit is used to filter out the cockpit interaction strategies corresponding to the comprehensive risk label based on the matching degree. The cockpit interaction strategies include environmental adjustment, entertainment control and driving assistance intervention strategies.

[0131] In some possible implementations, the policy matching module 440 includes:

[0132] The instruction decomposition submodule is used to decompose the cockpit interaction strategy into control instructions for the cockpit subsystem.

[0133] The instruction execution submodule is used to send the control instructions to the corresponding cockpit execution components, which then adjust the cockpit's environment, entertainment, and driver assistance functions to complete the cockpit state adjustment.

[0134] The intelligent vehicle cockpit adaptive interaction device provided in the embodiments of the present invention can execute the intelligent vehicle cockpit adaptive interaction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0135] Figure 5This is a schematic diagram of the structure of an electronic device for implementing the intelligent vehicle cockpit adaptive interaction method according to embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0136] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0137] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0138] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as intelligent vehicle cockpit adaptive interaction methods.

[0139] In some embodiments, the intelligent vehicle cockpit adaptive interaction method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the intelligent vehicle cockpit adaptive interaction method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the intelligent vehicle cockpit adaptive interaction method by any other suitable means (e.g., by means of firmware).

[0140] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0141] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0144] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0145] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0146] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0147] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An adaptive interaction method for an intelligent vehicle cockpit, characterized in that, include: Collect multi-source perception data, which includes visual perception data, voice perception data, physiological perception data and vehicle status perception data of drivers and passengers. The collected multi-source sensing data is cleaned, time-synchronized, and feature-level fused to obtain the feature vector to be identified. The trained emotion recognition model is used to analyze the feature vector to be identified, and the behavioral labels and emotional state labels of the driver and passengers are identified to generate a comprehensive risk label. Based on the comprehensive risk label, the corresponding cockpit interaction strategy is matched from the preset strategy library, and the cockpit status is adjusted based on the cockpit interaction strategy.

2. The method according to claim 1, characterized in that, The collection of multi-source sensing data includes: The visual acquisition component acquires image information of the driver's and passengers' facial expressions, eye state, and head posture as the visual perception data. The voice signals of drivers and passengers are acquired by the voice acquisition component and the acoustic features are extracted as the voice perception data. Physiological information such as heart rate, heart rate variability, and skin conductance response of drivers and passengers is acquired through physiological sensing components and used as the physiological sensing data. Vehicle operation data reflecting driving behavior is acquired through the vehicle bus and used as the vehicle state perception data.

3. The method according to claim 1, characterized in that, The process of cleaning, time synchronization, and feature-level fusion of the collected multi-source sensing data to obtain the feature vector to be identified includes: The multi-source sensing data is cleaned by removing outliers and standardizing the data. The cleaned multi-source sensing data is then synchronized with time axis alignment. Heterogeneous feature extraction and fusion are performed on the synchronized multi-source sensing data to obtain the feature vector to be identified.

4. The method according to claim 1, characterized in that, The trained emotion recognition model analyzes the feature vector to be identified, and identifies the behavioral and emotional state labels of the driver and passengers to generate a comprehensive risk label, including: The feature vector to be identified is input into the emotion recognition model, and the emotion recognition model is used to parse out the behavioral labels describing the physical operations and actions of the driver and passengers. The emotion recognition model is used to parse out the emotion state labels that describe the internal psychological state of the driver and passengers; The driving risk level is assessed by combining the behavioral tags and the emotional state tags, and the comprehensive risk tag is generated based on the assessment results.

5. The method according to claim 4, characterized in that, The process of combining the behavioral tags and the emotional state tags to assess driving risk level, and generating the comprehensive risk tag based on the assessment results, includes: The behavioral tags and emotional state tags are matched according to preset tag combination rules. The driving risk level of the driver and passengers is determined based on the feature matching results. The driving risk level includes low risk, medium risk, high risk and very high risk. A corresponding comprehensive risk label is generated based on the different driving risk levels.

6. The method according to claim 1, characterized in that, The step of matching the corresponding cabin interaction strategy from the preset strategy library based on the comprehensive risk label includes: The risk composite label is matched with the strategy labels in the preset strategy library; The optimal cockpit interaction strategy corresponding to the comprehensive risk label is selected based on the matching degree. The cockpit interaction strategy includes cockpit environment adjustment strategy, in-vehicle entertainment control strategy and driving assistance intervention strategy.

7. The method according to claim 1, characterized in that, The adjustment of the cabin state based on the cabin interaction strategy includes: The cockpit interaction strategy is decomposed into corresponding cockpit subsystem control commands; The control command is sent to the corresponding cockpit execution component, which then adjusts the cockpit's environmental parameters, entertainment playback status, and driver assistance function status to complete the adjustment of the cockpit status.

8. An adaptive interaction device for an intelligent vehicle cockpit, characterized in that, include: The data acquisition module is used to collect multi-source perception data, which includes visual perception data, voice perception data, physiological perception data and vehicle status perception data of drivers and passengers. The data processing module is used to clean, synchronize time, and fuse the collected multi-source sensing data to obtain the feature vector to be identified. The state recognition module 430 is used to analyze the feature vector to be identified based on the trained emotion recognition model, identify the behavior labels and emotion state labels of the driver and passengers, and generate a comprehensive risk label. The strategy matching module is used to match the corresponding cockpit interaction strategy from the preset strategy library according to the comprehensive risk label, and adjust the cockpit status based on the cockpit interaction strategy.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the intelligent vehicle cockpit adaptive interaction method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the intelligent vehicle cockpit adaptive interaction method according to any one of claims 1-7.

Images