Intelligent cabin control method, device, equipment and program product

Abstract

The embodiment of the invention provides an intelligent cabin control method, device, equipment and program product, and the method comprises the steps: collecting the multi-modal data of a driver, and generating a cross-modal feature according to the multi-modal data; the multi-modal data comprises image data and physiological time sequence signal data, inputting the cross-modal features into a classification model, determining an output result of the classification model as a physiological state of the driver, the physiological state comprising a physiological function state and an emotional state, generating an intervention strategy according to the physiological state of the driver, and controlling the intelligent cabin according to the intervention strategy. By identifying the physiological state of the driver, the information most concerned by the driver at present is accurately judged, so that the intelligent cabin is accurately controlled.

Description

Technical Field

[0001] This application relates to the field of intelligent cockpit technology, and in particular to an intelligent cockpit control method, device, equipment and program product. Background Technology

[0002] With the rapid development of intelligent vehicle technology, the intelligent cockpit, as the core carrier of human-vehicle interaction, is gradually evolving from traditional information display and control functions to proactive perception and intelligent decision-making.

[0003] Currently, intelligent cockpit systems primarily rely on environmental data (such as vehicle speed and location information) and vehicle status (such as navigation routes and fault prompts) for interface display and function control, with their interaction logic mainly based on static preset rules.

[0004] The aforementioned intelligent cockpit adjustment method cannot adjust according to the driver's actual condition. Summary of the Invention

[0005] This application provides a smart cockpit control method, device, equipment, and program product, which achieves accurate control of the smart cockpit by determining the driver's physiological state.

[0006] Firstly, this application provides an intelligent cockpit control method, the method comprising:

[0007] Multimodal data of drivers is collected, and cross-modal features are generated based on the multimodal data; the multimodal data includes image data and physiological time-series signal data;

[0008] Cross-modal features are input into a classification model, and the output of the classification model is used to determine the driver's physiological state, which includes physiological function and emotional state.

[0009] Intervention strategies are generated based on the driver's physiological state, and the intelligent cockpit is controlled according to the intervention strategies.

[0010] Optionally, physiological time-series signal data includes heart rate, respiratory rate, and heart rate variability; multimodal data of the driver are collected, including:

[0011] The driver's heart rate and respiratory rate were collected using a PPG photoelectric sensor.

[0012] Heart rate variability was acquired using ECG electrodes;

[0013] Skin electrical response information is collected using a skin electrical response sensor;

[0014] Image data of the driver is collected using an infrared camera.

[0015] Optionally, cross-modal features can be generated from multimodal data, including:

[0016] Heart rate, respiratory rate, skin conductance response information, and heart rate variability feature information were extracted separately; the feature information included time domain features and frequency domain features.

[0017] Extract facial expression feature vectors from the driver's facial image;

[0018] By fusing feature information and facial expression feature vectors, cross-modal features are obtained.

[0019] Optionally, intervention strategies can be generated based on the driver's physiological state, including:

[0020] Intervention strategies are determined based on the driver's physiological state, current driving environment information, and the driver's personalized settings. The intervention strategies include the target intervention actions and the target execution frequency. The current driving environment includes road condition information, vehicle speed information, and weather information.

[0021] Optionally, intervention strategies can be determined based on the driver's physiological state, current driving environment information, and the driver's personalized settings, including:

[0022] Candidate intervention actions are determined based on a preset mapping table and the driver's physiological state; the preset mapping table is used to represent the correspondence between physiological state and intervention actions.

[0023] Based on current driving environment information and physiological state, the execution frequency of candidate intervention actions is adjusted;

[0024] Based on the driver's personalized settings, candidate intervention actions are screened to obtain executable target intervention actions and target execution frequencies.

[0025] Optionally, candidate intervention actions include control commands to at least one of the following: the in-cabin human-machine interface, the audio system, the ambient lighting, the seat adjustment mechanism, the air conditioning system, or the fragrance system.

[0026] Optionally, the method also includes:

[0027] Obtain the driver's historical physiological data;

[0028] The decision threshold of the classification model is adjusted based on historical physiological data.

[0029] Secondly, this application provides an intelligent cockpit control device, the device comprising:

[0030] The generation module is used to collect multimodal data of drivers and generate cross-modal features based on the multimodal data; the multimodal data includes image data and physiological time-series signal data;

[0031] The determination module is used to input cross-modal features into the classification model and determine the output of the classification model as the driver's physiological state; the physiological state includes physiological function state and emotional state.

[0032] The control module is used to generate intervention strategies based on the driver's physiological state and control the smart cockpit according to the intervention strategies.

[0033] Thirdly, this application provides an electronic device, including: at least one processor and a memory;

[0034] The memory stores instructions that the computer executes;

[0035] At least one processor executes computer execution instructions stored in memory, causing at least one processor to perform the method as described in any of the first aspects.

[0036] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method as described in any of the first aspects.

[0037] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the first aspects.

[0038] The intelligent cockpit control method, device, equipment, and program products provided in this application include: collecting multimodal data of the driver; generating cross-modal features based on the multimodal data; the multimodal data includes image data and physiological time-series signal data; inputting the cross-modal features into a classification model; determining the output of the classification model as the driver's physiological state, which includes physiological functional state and emotional state; generating an intervention strategy based on the driver's physiological state; and controlling the intelligent cockpit based on the intervention strategy. By recognizing the driver's physiological state, the method accurately determines the information that the driver needs to focus on most at the moment, thereby accurately controlling the intelligent cockpit. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0040] Figure 1 A flowchart illustrating an intelligent cockpit control method provided in an embodiment of the present invention;

[0041] Figure 2 The flowchart of another intelligent cockpit control method provided in the embodiment of the present invention;

[0042] Figure 3 A schematic diagram of a framework for determining cross-modal features provided in an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of the structure of an intelligent cockpit control device provided in an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention.

[0045] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0047] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding access points for users to choose to authorize or refuse.

[0048] Currently, smart cockpits cannot dynamically adjust according to the driver's actual state. The reason for this is that they generally rely on environmental data or vehicle data (such as vehicle speed and navigation information) and lack a real-time monitoring mechanism for the driver's personal state (such as fatigue, tension, anxiety, etc.). They cannot accurately determine the information that the driver needs to focus on most and the interface optimization strategy, and therefore cannot make intelligent responses based on the user's physiological state.

[0049] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0050] Figure 1This is a flowchart illustrating an intelligent cockpit control method provided in an embodiment of the present invention. The method includes steps S101 to S103:

[0051] Step S101: Collect multimodal data of the driver and generate cross-modal features based on the multimodal data; the multimodal data includes image data and physiological time-series signal data.

[0052] Multimodal data refers to multi-dimensional data that reflects the driver's state, collected from sensors of different sources or types. Image data is typically collected by cameras deployed in the cockpit, including facial images of the driver. Physiological time-series signal data refers to physiological indicators that change continuously over time, such as electrocardiogram signals collected by contact electrodes on the steering wheel or seat, photoplethysmography (PPG) signals collected by photoelectric sensors, or skin conductance response signals collected by dedicated sensors.

[0053] Cross-modal features refer to high-level information representations that can comprehensively characterize the driver's state, extracted by fusing raw data from different modalities.

[0054] Optionally, while the vehicle is in motion, the sensors can be activated to begin the data acquisition process, and cross-modal characteristics can be obtained through data processing to achieve control of the smart cockpit.

[0055] Step S102: Input the cross-modal features into the classification model, and determine the output of the classification model as the driver's physiological state; the physiological state includes physiological function state and emotional state.

[0056] A classification model is a trained machine learning or deep learning model whose function is to output one or more classification labels representing the driver's state based on the cross-modal features of the input.

[0057] The classification model can be a traditional machine learning model such as support vector machines or random forests, or a deep learning model such as convolutional neural networks, recurrent neural networks, or a hybrid model thereof. The training of the model depends on a labeled cross-modal feature-physiological state pairing dataset.

[0058] Optionally, a driver's physiological state can include two levels: physiological functional state and emotional state. Physiological functional state mainly refers to changes in state caused by physical fatigue, etc. Emotional state refers to psychological and emotional states such as anxiety and tension.

[0059] Optionally, the generated cross-modal feature vector can be input into a pre-trained classification model. Based on the learned patterns, the classification model analyzes and judges the feature vector, ultimately outputting one or more classification results regarding the driver's physiological state, such as fatigue or anxiety.

[0060] Step S103: Generate an intervention strategy based on the driver's physiological state, and control the smart cockpit according to the intervention strategy.

[0061] Intervention strategies are a set of specific instructions developed based on the identified physiological state of the driver to adjust the cockpit environment or interaction methods.

[0062] Optionally, based on the identified driver's physiological state, a pre-defined mapping table or a strategy generation model is consulted to generate a specific intervention strategy corresponding to the physiological state. This intervention strategy is then converted into a series of executable control commands. These commands are then used to control the relevant actuators in the smart cockpit, such as adjusting the audio output content and volume of the sound system.

[0063] After intervention, the driver's physiological state can be re-examined to assess the effectiveness of the intervention.

[0064] Through the above process, real-time perception, intelligent judgment, and proactive intervention of the driver's status are achieved, aiming to improve driving safety and passenger comfort.

[0065] Through the above process, when an abnormal driver condition (such as fatigue) is detected, the cockpit system can proactively trigger intervention measures, such as issuing prompts, providing vibration feedback, adjusting seat posture or ambient lighting, to help the driver recover to their optimal state. This can effectively improve the level of human-computer interaction intelligence in the smart cockpit, and enhance driving safety and user experience.

[0066] For example, successfully predicting and adjusting within 1-2 minutes before a rise in driver fatigue, and activating a high-profile, high-display UI mode for certain special environments (difficult roads, rain, snow, nighttime highways) to enhance driver attention. A high-profile UI is an interface display strategy that proactively captures attention in scenarios requiring high driver focus through stronger and more prominent visual and interactive means.

[0067] The intelligent cockpit control method provided in this application includes: collecting multimodal data of the driver, generating cross-modal features based on the multimodal data; the multimodal data includes image data and physiological time-series signal data; inputting the cross-modal features into a classification model; determining the output of the classification model as the driver's physiological state, which includes physiological functional state and emotional state; generating an intervention strategy based on the driver's physiological state; and controlling the intelligent cockpit based on the intervention strategy. By recognizing the driver's physiological state, the method can accurately determine the information that the driver needs to focus on most at the moment, thereby accurately controlling the intelligent cockpit.

[0068] Figure 2The flow of another intelligent cockpit control method provided in this embodiment of the invention is as follows: the driver's physiological state is monitored and it is determined whether there is an abnormality. If it is not a normal state, but a state of fatigue, anxiety, tension, etc., it is an abnormal state. If it is an abnormal state, an early warning prompt can be triggered and intervention measures can be executed.

[0069] Optionally, physiological time-series signal data includes heart rate, respiratory rate, and heart rate variability; multimodal data of the driver are collected, including:

[0070] The driver's heart rate and respiratory rate were collected using a PPG photoelectric sensor.

[0071] Heart rate variability was acquired using ECG electrodes;

[0072] Skin electrical response information is collected using a skin electrical response sensor;

[0073] Image data of the driver is collected using an infrared camera.

[0074] A PPG (Photoplethysmography) sensor is a photoplethysmography pulse wave sensor. This sensor typically works by emitting a beam of light of a specific wavelength (usually green or infrared) into the skin (such as the fingertips, wrist, or ear) and measuring changes in the intensity of reflected or transmitted light. Since the light absorption characteristics of blood change periodically with the changes in blood vessel volume caused by heartbeats, heart rate can be calculated by analyzing the received light signal. Simultaneously, because respiratory activity causes slow, periodic modulation of blood vessel volume, respiratory rate information can also be extracted by low-frequency analysis of the PPG signal.

[0075] Optionally, in an in-vehicle environment, the PPG sensor can be integrated into the steering wheel grip, seat belt buckle, or seat surface for continuous monitoring.

[0076] ECG (Electrocardiogram) electrodes are used to record the electrophysiological activity of the heart by measuring the minute potential differences generated on the skin surface due to the heart's electrical activity. By precisely calculating the minute fluctuations between consecutive heartbeats, heart rate variability (HRV) can be calculated. ECG electrodes are typically placed in contact with areas that the driver will inevitably touch, such as the surface of the steering wheel grip.

[0077] Electrodermal conductivity (EDS) sensors are used to measure changes in the electrical conductivity of the human skin surface. EDS information indicates the conductivity of human skin. The electrical conductivity of human skin changes with the activity of the autonomic nervous system, particularly the level of excitation of the sympathetic nervous system. EDS information is a physiological indicator reflecting driver stress and anxiety.

[0078] An infrared camera is an image acquisition device capable of receiving infrared light. Compared to visible light cameras, infrared cameras have the ability to produce clear images even in low-light or complete darkness. Image data can include images of the eyes to enable eye tracking.

[0079] Optionally, all of the above sensors can be activated simultaneously to acquire multimodal data, so as to facilitate the processing of the simultaneously acquired multimodal data.

[0080] By integrating multiple sensors inside the vehicle, multimodal data can be acquired.

[0081] Optionally, cross-modal features can be generated from multimodal data, including:

[0082] Heart rate, respiratory rate, skin conductance response information, and heart rate variability feature information were extracted separately; the feature information included time domain features and frequency domain features.

[0083] Extract facial expression feature vectors from the driver's facial image;

[0084] By fusing feature information and facial expression feature vectors, cross-modal features are obtained.

[0085] Figure 3 A schematic diagram of a framework for determining cross-modal features is provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the temporal and frequency domain features of each physiological time-series signal data can be extracted separately, and the facial expression feature vector of the image data can be extracted, thereby obtaining cross-modal features.

[0086] When determining cross-modal features, the data can be preprocessed to filter out noise and then standardized.

[0087] Temporal characteristics refer to statistical or morphological indicators directly calculated from the waveform of a signal changing over time. For physiological time-series signals, temporal characteristics include, but are not limited to: the mean and standard deviation of heart rate, used to measure the baseline level and variability of heart rate; the standard deviation of heart rate variability (SDNN), i.e., the standard deviation of all normal heartbeat intervals, reflecting the overall regulatory capacity of the autonomic nervous system; the root mean square difference (RMSSD) of heart rate variability, i.e., the root mean square difference of the difference between adjacent heartbeat intervals, mainly reflecting the activity of the parasympathetic nervous system; and the mean amplitude, number of responses, and rise time slope of the skin conductance response signal, which are directly related to the frequency and intensity of emotional arousal.

[0088] Frequency domain features are indicators extracted along the frequency dimension after performing spectral analysis on a signal (such as Fast Fourier Transform or Wavelet Transform). For example, the ratio of low-frequency power to high-frequency power after decomposing a heart rate variability signal is a key indicator for assessing the balance between the sympathetic and parasympathetic nervous systems; the power spectral density of skin conductance or respiratory signals in specific frequency bands (such as 0.05-0.2 Hz) is related to emotion regulation or breathing patterns.

[0089] Facial feature vectors are high-dimensional numerical vectors extracted from driver facial images using computer vision and deep learning models to quantify facial muscle movements and emotional expressions.

[0090] By fusing defined feature information with facial expression feature vectors, cross-modal features can be obtained. Fusion algorithms can employ simple feature vector concatenation or more complex mechanisms, such as attention-based models that assign appropriate weights to features from different modalities.

[0091] Through the above fusion process, quantitative indicators from internal physiological responses (such as heart rate and skin conductivity) can be combined with manifestations from external facial behavior.

[0092] By calculating cross-modal features, it is possible to achieve high-precision identification of the driver's physiological state.

[0093] Optionally, intervention strategies can be generated based on the driver's physiological state, including:

[0094] Intervention strategies are determined based on the driver's physiological state, current driving environment information, and the driver's personalized settings. The intervention strategies include the target intervention actions and the target execution frequency. The current driving environment includes road condition information, vehicle speed information, and weather information.

[0095] Optionally, when performing the method of this application, the driver may be identified first to obtain the driver's personalized settings information.

[0096] Physiological states can include normal, fatigued, anxious, and tense. Current driving environment information refers to the real-time external and internal conditions in which the vehicle is located during driving, mainly including road condition information (such as highways, congested urban roads, mountain curves, construction areas, etc.), real-time vehicle speed information, and weather information.

[0097] Personalized settings information refers to preferences related to specific drivers. These can be manually preset options such as "Do Not Disturb" or "Prefer Gentle Reminders," or settings established through long-term learning and driver intervention. Targeted intervention actions are specific operations planned to be taken in response to the currently assessed driver's physiological state, aimed at adjusting the driver's state or improving human-computer interaction. Target execution frequency refers to the number of times the selected targeted intervention action is implemented per unit of time.

[0098] Optionally, the intervention strategy can be determined based on a predefined rule base or a trained decision model, taking into account the driver's physiological state, current driving environment information, and the driver's personalized settings.

[0099] Optionally, when the driver is physically fatigued, soft music can be played and the cabin lighting can be increased to alleviate driver fatigue. When the driver is physically anxious, breathing training guidance can be initiated and the cabin volume can be reduced to alleviate driver anxiety.

[0100] Optionally, when making interventions, it is also necessary to make determinations based on the current driving environment information. For example, when driving on a highway or on a mountain curve, it is best to avoid making interventions to avoid distracting the driver and causing an accident.

[0101] Based on the above information, intervention strategies that are appropriate for the current physiological state of the driver can be accurately generated.

[0102] Optionally, intervention strategies can be determined based on the driver's physiological state, current driving environment information, and the driver's personalized settings, including:

[0103] Candidate intervention actions are determined based on a preset mapping table and the driver's physiological state; the preset mapping table is used to represent the correspondence between physiological state and intervention actions.

[0104] Based on current driving environment information and physiological state, the execution frequency of candidate intervention actions is adjusted;

[0105] Based on the driver's personalized settings, candidate intervention actions are screened to obtain executable target intervention actions and target execution frequencies.

[0106] A preset mapping table refers to a predefined data structure stored within the system that establishes a correspondence between different physiological states and a series of suggested intervention actions. For example, it can be explicitly specified that when the physiological state is fatigue, the corresponding candidate intervention actions include turning on the cabin lighting, playing energizing music, and activating the seat ventilation.

[0107] The candidate intervention actions constitute a preliminary set of strategy options, but they do not yet take into account current driving environment information and driver's personalized settings, so not all of them may be suitable for immediate execution.

[0108] At the start of the process, the strategy decision-making module receives physiological state identification results, such as fatigue. The module first queries a pre-defined mapping table and retrieves all pre-associated candidate intervention actions based on the state keyword "fatigue".

[0109] Subsequently, the module enters the frequency adjustment phase. At this time, the system synchronously acquires current driving environment information. Combining the current physiological state with the current driving environment information, the execution frequency of candidate actions is calibrated. For example, considering driving on mountain roads, the intervention frequency is reduced to avoid distracting the driver. When the driver's physiological state is anxious or tense, the intervention frequency is also reduced. Based on the above methods, a suitable target execution frequency can be determined.

[0110] Finally, the module performs personalized filtering, which can be done by accessing the driver's personalized settings for matching. Suppose the driver has selected to disable intervention action 1 in their personalized settings. Therefore, candidate intervention actions can be filtered based on this information: if a candidate intervention action includes both intervention action 1 and intervention action 2, then intervention action 2 can be identified as the target intervention action.

[0111] Through the above operations, a refined strategy generation process was achieved, from general rules to specific scenarios, and then to individual adaptation.

[0112] Optionally, candidate intervention actions include control commands to at least one of the following: the in-cabin human-machine interface, the audio system, the ambient lighting, the seat adjustment mechanism, the air conditioning system, or the fragrance system.

[0113] When candidate intervention actions are determined from a preset mapping table based on physiological state, these actions are represented internally by the system as a series of structured control commands. For example, for the "fatigue" state, the mapping table may associate a set of candidate commands, including: a command sent to the human-machine interface controller, instructing it to switch the main screen to simplified navigation mode and increase the screen brightness; a command sent to the audio system controller, instructing it to play upbeat music and adjust the volume to a medium level; a command sent to the ambient lighting controller, instructing it to gradually change the light color from warm yellow to cool blue; and a command sent to the seat adjustment device, instructing it to activate the wave-shaped massage function of the lumbar support.

[0114] The intervention action can be executed by controlling relevant hardware interfaces, such as audio, lighting, and seating.

[0115] Optionally, the method also includes:

[0116] Obtain the driver's historical physiological data;

[0117] The decision threshold of the classification model is adjusted based on historical physiological data.

[0118] Optionally, when performing the method of this application, the driver may be identified first to obtain the driver's historical physiological data.

[0119] In long-term use, a personalized state recognition benchmark can be established for each driver. Its core lies in using historical physiological data to fine-tune the general classification model locally.

[0120] Optionally, during each driving session, while collecting real-time physiological data for immediate status identification and intervention, and with the driver's consent and ensuring data security, this data, along with its corresponding timestamp and the physiological state automatically recorded or manually labeled at that time, will be archived to form the driver's historical physiological dataset. When the accumulated data reaches a certain scale, a threshold adjustment process can be initiated.

[0121] First, physiological signal fragments collected under "normal" driving conditions can be extracted from historical data, and the statistical distributions of their various characteristics (such as average heart rate and HRV SDNN) can be calculated to establish the driver's personal physiological baseline. Then, the system analyzes the physiological characteristics corresponding to historical events marked as "fatigue" and compares them with the personal baseline. Based on this analysis, the judgment thresholds in the classification model can be automatically calculated and updated.

[0122] For example, analysis revealed that the driver's heart rate variability decreased less relative to their baseline when fatigued than the general model preset. In other words, the driver's physiological response to fatigue was relatively mild. Therefore, the threshold for determining "fatigue" could be lowered accordingly, allowing the model to detect the driver's unique, mild fatigue physiological signals earlier and more sensitively.

[0123] The above methods can identify abnormal physiological states of specific drivers more accurately and earlier, providing a reliable basis for generating precise intervention strategies.

[0124] Figure 4 This is a schematic diagram of the structure of an intelligent cockpit control device provided in an embodiment of the present invention. The device includes:

[0125] The generation module 401 is used to collect multimodal data of the driver and generate cross-modal features based on the multimodal data; the multimodal data includes image data and physiological time-series signal data;

[0126] The determination module 402 is used to input cross-modal features into the classification model and determine the output of the classification model as the driver's physiological state; the physiological state includes physiological function state and emotional state.

[0127] The control module 403 is used to generate intervention strategies based on the driver's physiological state and control the smart cockpit according to the intervention strategies.

[0128] Optionally, the physiological time-series signal data includes heart rate, respiratory rate, and heart rate variability; the generation module 401 is specifically used for: when collecting multimodal data from the driver.

[0129] The driver's heart rate and respiratory rate were collected using a PPG photoelectric sensor.

[0130] Heart rate variability was acquired using ECG electrodes;

[0131] Image data of the driver is collected using an infrared camera.

[0132] Optionally, when generating cross-modal features based on multimodal data, the generation module 401 is specifically used for:

[0133] Feature information of heart rate, respiratory rate, and heart rate variability was extracted separately; the feature information included time-domain features and frequency-domain features.

[0134] Extract facial expression feature vectors from the driver's facial image;

[0135] By fusing feature information and facial expression feature vectors, cross-modal features are obtained.

[0136] Optionally, when generating an intervention strategy based on the driver's physiological state, the control module 403 is specifically used for:

[0137] Intervention strategies are determined based on the driver's physiological state, current driving environment information, and the driver's personalized settings. The intervention strategies include the target intervention actions and the target execution frequency. The current driving environment includes road condition information, vehicle speed information, and weather information.

[0138] Optionally, when determining the intervention strategy based on the driver's physiological state, current driving environment information, and the driver's personalized settings, the control module 403 is specifically used for:

[0139] Candidate intervention actions are determined based on a preset mapping table and the driver's physiological state; the preset mapping table is used to represent the correspondence between physiological state and intervention actions.

[0140] Based on current driving environment information and physiological state, the execution frequency of candidate intervention actions is adjusted;

[0141] Based on the driver's personalized settings, candidate intervention actions are screened to obtain executable target intervention actions and target execution frequencies.

[0142] Optionally, candidate intervention actions include control commands to at least one of the following: the in-cabin human-machine interface, the audio system, the ambient lighting, the seat adjustment mechanism, the air conditioning system, or the fragrance system.

[0143] Optionally, the device further includes: an adjustment module for:

[0144] Obtain the driver's historical physiological data;

[0145] The decision threshold of the classification model is adjusted based on historical physiological data.

[0146] The intelligent cockpit control device 40 provided in this embodiment of the invention can achieve the above-mentioned functions. Figure 1 The intelligent cockpit control method shown is similar in principle and technical effect, and will not be described in detail here.

[0147] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Figure 5 As shown, the electronic device provided in this embodiment includes at least one processor 501 and a memory 502. The processor 501 and the memory 502 are connected via a bus 503.

[0148] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to execute the method in the above method embodiment.

[0149] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0150] In the above Figure 5 In the illustrated embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0151] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage.

[0152] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0153] This invention also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in the above embodiments.

[0154] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the above method embodiments.

[0155] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0156] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0157] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0158] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0159] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0160] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0161] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0162] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0163] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A smart cockpit control method, characterized in that, include: Multimodal data of the driver is collected, and cross-modal features are generated based on the multimodal data; the multimodal data includes image data and physiological time-series signal data; The cross-modal features are input into the classification model, and the output of the classification model is determined as the driver's physiological state. The physiological state includes physiological functional state and emotional state; An intervention strategy is generated based on the driver's physiological state, and the smart cockpit is controlled according to the intervention strategy.

2. The method according to claim 1, characterized in that, The physiological time-series signal data includes heart rate, respiratory rate, and heart rate variability; multimodal data of the driver is collected, including: The driver's heart rate and respiratory rate were collected using a PPG photoelectric sensor. Heart rate variability was acquired using ECG electrodes; Skin electrical response information is collected using a skin electrical response sensor; Image data of the driver is acquired using an infrared camera.

3. The method according to claim 2, characterized in that, Generate cross-modal features based on the multimodal data, including: Feature information of the heart rate, respiratory rate, skin conductance information, and heart rate variability is extracted respectively; the feature information includes time-domain features and frequency-domain features. Extract the facial expression feature vector from the driver's facial image; The cross-modal features are obtained by fusing the feature information and the facial expression feature vector.

4. The method according to claim 1, characterized in that, Intervention strategies are generated based on the driver's physiological state, including: An intervention strategy is determined based on the driver's physiological state, current driving environment information, and the driver's personalized settings. The intervention strategy includes the target intervention action and the target execution frequency. The current driving environment includes road condition information, vehicle speed information, and weather information.

5. The method according to claim 4, characterized in that, The intervention strategy is determined based on the driver's physiological state, current driving environment information, and the driver's personalized settings, including: Candidate intervention actions are determined based on a preset mapping table and the driver's physiological state; the preset mapping table is used to represent the correspondence between physiological state and intervention actions. Based on the current driving environment information and the physiological state, the execution frequency of the candidate intervention actions is adjusted; Based on the driver's personalized settings, the candidate intervention actions are filtered to obtain the executable target intervention actions and target execution frequencies.

6. The method according to claim 5, characterized in that, The candidate intervention actions include control commands to at least one of the following: the in-cabin human-machine interface, the audio system, the ambient lighting, the seat adjustment device, the air conditioning system, or the fragrance system.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Obtain the driver's historical physiological data; The decision threshold of the classification model is adjusted based on the historical physiological data.

8. An intelligent cockpit control device, characterized in that, include: A generation module is used to collect multimodal data of the driver and generate cross-modal features based on the multimodal data; the multimodal data includes image data and physiological time-series signal data; The determination module is used to input the cross-modal features into the classification model and determine the output of the classification model as the driver's physiological state. The physiological state includes physiological functional state and emotional state; The control module is used to generate intervention strategies based on the driver's physiological state and control the smart cockpit according to the intervention strategies.

9. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images