Vehicle-mounted interaction system and method for adjusting driver emotion, vehicle and medium

Abstract

The invention discloses a vehicle-mounted interaction system and method for adjusting the emotion of a driver, a vehicle and a medium, and relates to the technical field of intelligent driving. The vehicle-mounted interaction system comprises an emotion sensing module, a data processing module and a feedback control module which are connected with one another, the emotion sensing module is used for acquiring multi-modal information of a driver, and the multi-modal information comprises facial data, physiological data and voice data; the data processing module is used for fusing the multi-modal information and determining an emotional state level of the driver; and the feedback control module is used for controlling operation parameters of the multi-channel actuator according to the emotional state level, and the multi-channel actuator comprises an auditory channel, a visual channel, an olfactory channel and a tactile channel. By adopting the vehicle-mounted interaction system, the emotion of the driver can be actively sensed and cooperatively adjusted, and the driving safety and comfort are effectively improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and in particular to an in-vehicle interactive system, method, vehicle, and medium for regulating driver emotions. Background Art

[0002] In modern society, cars have become an indispensable means of transportation, bringing great convenience to daily commutes. However, when drivers are under negative emotions such as fatigue, anxiety, and anger, they are more likely to make mistakes and cause traffic accidents. However, current technologies can only provide passive warnings and lack intervention capabilities. Summary of the Invention

[0003] In view of the above-mentioned defects or deficiencies in the relevant technologies, it is desired to provide an in-vehicle interactive system, method, vehicle and medium for regulating the driver's emotions, which can actively perceive and coordinately regulate the driver's emotions, effectively improving driving safety and comfort.

[0004] In a first aspect, the present application provides an in-vehicle interactive system for regulating driver emotions, the in-vehicle interactive system comprising an emotion perception module, a data processing module, and a feedback control module connected to each other;

[0005] The emotion perception module is used to obtain multimodal information of the driver, wherein the multimodal information includes facial data, physiological data and voice data; the data processing module is used to fuse the multimodal information and determine the driver's emotional state level; and the feedback control module is used to control the operating parameters of the multi-channel actuator according to the emotional state level, wherein the multi-channel actuator includes an auditory channel, a visual channel, an olfactory channel and a tactile channel.

[0006] Optionally, in some embodiments of the present application, the emotion perception module includes a camera, a physiological sensor and a microphone, the camera is specifically used to collect the facial data, the physiological sensor is specifically used to collect the physiological data, and the microphone is specifically used to collect the voice data.

[0007] Optionally, in some embodiments of the present application, the camera is arranged above the main driving seat of the vehicle; and / or, the physiological sensor is arranged on the grip area of ​​the vehicle steering wheel and the backrest of the vehicle seat; and / or, the microphone is arranged in the center of the top of the vehicle.

[0008] Optionally, the data processing module in some embodiments of the present application includes a fusion unit and a grading unit. The fusion unit is specifically used to analyze the multimodal information and comprehensively judge the emotion type by combining the corresponding weight values ​​of each data in the multimodal information. The grading unit is specifically used to obtain the emotional state level according to the emotion type.

[0009] Optionally, in some embodiments of the present application, the fusion unit is further specifically used to use a pre-trained neural network model to identify target expressions in the facial data, use a pre-trained natural language processing model to identify target keywords and tone sharpness in the voice data, and detect target physiological parameter levels in the physiological data.

[0010] Optionally, in some embodiments of the present application, the multi-channel actuator includes a player, an ambient light, a fragrance releaser, and a seat vibrator.

[0011] Optionally, in some embodiments of the present application, the feedback control module is further configured to trigger an advanced driver assistance system of the vehicle to perform linkage if the emotional state level is dangerous.

[0012] In a second aspect, the present application provides an in-vehicle interaction method for regulating driver emotions, the in-vehicle interaction method being used in any in-vehicle interaction system according to the first aspect, the in-vehicle interaction method comprising:

[0013] Acquiring multimodal information of the driver, the multimodal information including facial data, physiological data, and voice data;

[0014] fusing the multimodal information and determining the driver's emotional state level;

[0015] The operating parameters of a multi-channel actuator are controlled according to the emotional state level, wherein the multi-channel actuator includes an auditory channel, a visual channel, an olfactory channel, and a tactile channel.

[0016] In a third aspect, the present application provides a vehicle, comprising the in-vehicle interactive system described in any one of the first aspects.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the in-vehicle interaction method described in the second aspect.

[0018] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0019] The embodiments of the present application provide an in-vehicle interactive system, method, vehicle, and medium for regulating driver emotions. The system actively obtains multimodal information of the driver in real time, including facial, physiological, and voice data, and then fuses the multimodal information to accurately determine the driver's emotional state level. The operating parameters of the multi-channel actuator are dynamically controlled based on the emotional state level, and the driver's emotions are coordinated and regulated from the aspects of hearing, vision, smell, and touch, thereby effectively improving driving safety and comfort. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A structural block diagram of an in-vehicle interactive system for regulating driver emotions provided in an embodiment of the present application;

[0022] Figure 2 A structural block diagram of another in-vehicle interactive system for regulating driver emotions provided in an embodiment of the present application;

[0023] Figure 3 A structural block diagram of another in-vehicle interactive system for regulating driver emotions provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of feedback control logic provided in an embodiment of the present application;

[0025] Figure 5 A schematic flow chart of an in-vehicle interaction method for regulating driver emotions provided in an embodiment of the present application;

[0026] Figure 6 A structural block diagram of a vehicle provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0028] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0029] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of this application can be combined with each other. Figures 1 to 6 The in-vehicle interactive system, method, vehicle and medium for regulating driver emotions provided by the embodiments of the present application are described in detail.

[0030] Please refer to Figure 1, which is a structural block diagram of an in-vehicle interactive system for regulating driver emotions provided in an embodiment of the present application. The in-vehicle interactive system 10 includes an emotion perception module 101, a data processing module 102, and a feedback control module 103 that are interconnected. In actual use, the emotion perception module 101 can obtain multimodal information of the driver, and the multimodal information includes facial data, physiological data, and voice data. The data processing module 102 can fuse the multimodal information and determine the driver's emotional state level. And, the feedback control module 103 can control the operating parameters of the multi-channel actuator according to the emotional state level, and the multi-channel actuator includes an auditory channel, a visual channel, an olfactory channel, and a tactile channel.

[0031] In some embodiments of the present application, Figure 2 As shown, the emotion perception module 101 includes but is not limited to a camera 1011, a physiological sensor 1012, and a microphone 1013. Among them, the camera 1011 can collect facial data, and the facial data may include facial expressions such as frowning and blinking, as well as head posture. The physiological sensor 1012 can collect physiological data, and the physiological data may include heart rate, skin conductivity (Galvanic Skin Response, GSR) and body surface temperature, while the microphone 1013 can collect voice data. Furthermore, the camera 1011 can be set above the main driving seat of the vehicle, such as above the vehicle steering wheel or above the vehicle A-pillar; and / or, the physiological sensor 1012 can be set in the grip area of ​​the vehicle steering wheel and the backrest of the vehicle seat; and / or, the microphone 1013 can be set in the center of the top of the vehicle.

[0032] In some embodiments of the present application, Figure 3As shown, the data processing module 102 includes, but is not limited to, a fusion unit 1021 and a grading unit 1022. The fusion unit 1021 can analyze multimodal information and determine the emotion type by combining the weight values ​​corresponding to each data in the multimodal information. For example, the weight value corresponding to facial data can be 0.4, the weight value corresponding to physiological data can be 0.3, and the weight value corresponding to voice data can be 0.3. Emotion types can include anger, fatigue, tension, and relaxation. The grading unit 1022 can determine the emotional state level based on the emotion type. For example, the emotional state level of anger and fatigue can be displayed as dangerous (red), the emotional state level of tension can be displayed as warning (yellow), and the emotional state level of relaxation can be displayed as normal (green). Furthermore, the fusion unit 1021 can also use a pre-trained neural network model to identify target expressions in facial data. For example, the neural network model can be a convolutional neural network (CNN) model, which is trained using labeled samples. The target expressions can be drooping corners of the mouth and closed eyelids, etc., drooping corners of the mouth indicate anger, and closed eyelids indicate fatigue; use a pre-trained natural language processing (NLP) model to identify target keywords and tone sharpness in speech data. For example, the natural language processing model can be a bidirectional encoder representations from transformers (BERT) model, which is trained using large-scale unlabeled corpus. The target keywords can be daily swear words and negative words such as "so annoying" and "so tired"; and detect the levels of target physiological parameters in physiological data, such as heart rate variability (HRV) and skin conductivity peak.

[0033] In some embodiments of the present application, the multi-channel actuators include but are not limited to a player, an ambient light, a fragrance dispenser, and a seat vibrator. The player can play music and voice assistants, the ambient light can adjust the color temperature and brightness, the fragrance dispenser can release different scents with adjustable concentrations, and the nozzle of the fragrance dispenser is set at the vehicle air-conditioning outlet. The seat vibrator can provide massage and emergency warning vibration. For example Figure 4As shown, when the emotional state level is dangerous, the ambient light switches to a cold blue light, the fragrance dispenser releases a 70% concentration of mint fragrance to refresh the mind, the seat vibrator turns on high-frequency tapping vibration, the player plays fast-paced light music, and a synchronous voice reminder "You are nervous and it is recommended to switch to automatic driving mode" is issued; when the emotional state level is warning, the ambient light gradually changes to warm yellow, the fragrance dispenser releases a 30% concentration of light lavender fragrance to relieve the mood, and the player plays natural white noise; and when the emotional state level is normal, the ambient light remains neutral white, and the fragrance dispenser maintains a default lemongrass fragrance of 10%. Furthermore, the feedback control module 103 can also trigger the vehicle's Advanced Driving Assistance System (ADAS) to work together when the emotional state level is dangerous, such as forcing lane keeping to be turned on, thereby reducing the risk of human error.

[0034] The in-vehicle interactive system for regulating driver emotions provided in the embodiments of the present application actively obtains the driver's multimodal information in real time. The multimodal information includes facial, physiological, and voice dimensional data, and then fuses the multimodal information to accurately determine the driver's emotional state level. The operating parameters of the multi-channel actuator are dynamically controlled based on the emotional state level, and the driver's emotions are coordinated and regulated from the aspects of hearing, vision, smell, and touch, thereby effectively improving driving safety and comfort.

[0035] Based on the above embodiments, the present invention provides a vehicle-mounted interaction method for regulating the driver's emotions. The vehicle-mounted interaction method can be applied to Figures 1 to 4 The vehicle-mounted interactive system 10 of the corresponding embodiment. Figure 5 , which is a flow chart of an in-vehicle interaction method for regulating driver emotions provided by an embodiment of the present application. The in-vehicle interaction method specifically includes the following steps:

[0036] S101, obtaining multimodal information of the driver, where the multimodal information includes facial data, physiological data, and voice data.

[0037] Exemplarily, the embodiment of the present application can use the camera 1011 in the emotion perception module 101 to collect facial data, which includes facial expressions such as frowning and blinking, as well as head posture, etc., use the physiological sensor 1012 in the emotion perception module 101 to collect physiological data, which includes heart rate, skin conductivity and body surface temperature, etc., and use the microphone 1013 in the emotion perception module 101 to collect voice data.

[0038] S102: Fusing the multimodal information and determining the driver's emotional state level.

[0039] For example, the embodiment of the present application can utilize the fusion unit 1021 in the data processing module 102 to analyze multimodal information and determine the emotion type by comprehensively considering the corresponding weight values ​​of each data in the multimodal information. For example, the corresponding weight value of facial data is 0.4, the corresponding weight value of physiological data is 0.3, and the corresponding weight value of voice data is 0.3. The emotion types include anger, fatigue, tension, and relaxation. In addition, the grading unit 1022 in the data processing module 102 is utilized to obtain the emotional state level according to the emotion type. For example, the emotional state level of anger and fatigue is dangerous and is displayed in red, the emotional state level of tension is warning and is displayed in yellow, and the emotional state level of relaxation is normal and is displayed in green. For example, if the fusion unit 1021 detects that the driver blinks frequently, corresponding to PERCLOS (the proportion of time the eyes are closed per unit time) ≥ 0.3, and at the same time, the heart rate variability is reduced and the voice recognition detects yawning, it indicates that the driver has a tendency to drive fatigued and requires immediate intervention. For another example, if the fusion unit 1021 detects that the driver is frowning and the corners of his mouth are drooping, and the skin conductivity peak value suddenly increases, and the voice recognition detects daily swear words, it indicates that the driver has a tendency to road rage and requires timely and proactive intervention.

[0040] S103 , controlling operating parameters of a multi-channel actuator according to the emotional state level, where the multi-channel actuator includes an auditory channel, a visual channel, an olfactory channel, and a tactile channel.

[0041] For example, if the driver is prone to driving fatigue, the ambient lighting switches to cool white (6500K), the fragrance dispenser releases a 70% concentration of mint fragrance, the seat vibrator starts to vibrate in a wave pattern, and the player plays rock music. If there is no improvement within five minutes, the player will quickly announce, "We have searched for the nearest rest area for you." Another example is if the driver is prone to road rage, the ambient lighting gradually changes to pink-purple, the fragrance dispenser releases a 30% concentration of light lavender fragrance, the player plays ASMR (Autonomous Sensory Meridian Response) natural sound effects, and a gentle voice prompts, "Please take a deep breath and maintain a safe distance."

[0042] It should be noted that, for the description of the same steps and contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0043] The in-vehicle interaction method for regulating driver emotions provided in the embodiments of the present application actively obtains the driver's multimodal information in real time. The multimodal information includes facial, physiological, and voice dimensional data, and then fuses the multimodal information to accurately determine the driver's emotional state level. The operating parameters of the multi-channel actuator are dynamically controlled based on the emotional state level, and the driver's emotions are coordinated and regulated from the aspects of hearing, vision, smell, and touch, thereby effectively improving driving safety and comfort.

[0044] As another aspect, the present application provides a vehicle. Figure 6 , which is a structural block diagram of a vehicle provided in an embodiment of the present application, the vehicle 20 may include Figures 1 to 4 The in-vehicle interactive system 10 of the corresponding embodiment.

[0045] As another aspect, the present invention provides a computer-readable storage medium for storing program code for executing the aforementioned Figure 5 Any implementation of the in-vehicle interaction method of the corresponding embodiment.

[0046] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0047] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. On the other hand, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms. The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0048] In addition, the functional modules in the various embodiments of the present application may be integrated into a single processing unit, or each module may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated units may be implemented in the form of hardware or in the form of software functional units. If the integrated units are implemented in the form of software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium.

[0049] Based on this understanding, the technical solution of the present application, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the in-vehicle interaction method of each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0050] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0051] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An in-vehicle interactive system for regulating driver emotions, characterized in that: The in-vehicle interactive system (10) comprises an emotion perception module (101), a data processing module (102) and a feedback control module (103) which are interconnected; The emotion perception module (101) is used to obtain multimodal information of the driver, the multimodal information including facial data, physiological data and voice data; the data processing module (102) is used to fuse the multimodal information and determine the driver's emotional state level; and the feedback control module (103) is used to control the operating parameters of a multichannel actuator according to the emotional state level, the multichannel actuator including an auditory channel, a visual channel, an olfactory channel and a tactile channel.

2. The in-vehicle interactive system according to claim 1, characterized in that: The emotion perception module (101) includes a camera (1011), a physiological sensor (1012) and a microphone (1013), wherein the camera (1011) is specifically used to collect the facial data, the physiological sensor (1012) is specifically used to collect the physiological data, and the microphone (1013) is specifically used to collect the voice data.

3. The in-vehicle interactive system according to claim 2, characterized in that: The camera (1011) is arranged above the main driving seat of the vehicle; and / or, the physiological sensor (1012) is arranged at the grip area of ​​the vehicle steering wheel and the backrest of the vehicle seat; and / or, the microphone (1013) is arranged at the center of the top of the vehicle.

4. The in-vehicle interactive system according to claim 1, characterized in that: The data processing module (102) includes a fusion unit (1021) and a grading unit (1022). The fusion unit (1021) is specifically used to analyze the multimodal information and determine the emotion type by combining the weight values ​​corresponding to each data in the multimodal information. The grading unit (1022) is specifically used to obtain the emotional state level according to the emotional type.

5. The in-vehicle interactive system according to claim 4, characterized in that: The fusion unit (1021) is further specifically configured to use a pre-trained neural network model to identify target expressions in the facial data, use a pre-trained natural language processing model to identify target keywords and intonation sharpness in the voice data, and detect target physiological parameter levels in the physiological data.

6. The in-vehicle interactive system according to any one of claims 1 to 5, characterized in that: The multi-channel actuator includes a player, an ambient light, a fragrance releaser and a seat vibrator.

7. The in-vehicle interactive system according to claim 6, characterized in that: The feedback control module (103) is further configured to trigger the vehicle's advanced driver assistance system to perform linkage if the emotional state level is dangerous.

8. A vehicle-mounted interactive method for regulating driver emotions, characterized in that: The in-vehicle interaction method is used in the in-vehicle interaction system (10) according to any one of claims 1 to 7, and the in-vehicle interaction method comprises: Acquiring multimodal information of the driver, the multimodal information including facial data, physiological data, and voice data; fusing the multimodal information and determining the driver's emotional state level; The operating parameters of a multi-channel actuator are controlled according to the emotional state level, wherein the multi-channel actuator includes an auditory channel, a visual channel, an olfactory channel, and a tactile channel.

9. A vehicle, characterized in that: The vehicle (20) includes the in-vehicle interactive system (10) according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the in-vehicle interaction method according to claim 8.

Images