Face information-based driver emotion intelligent interaction method, system and device

Abstract

The invention relates to a driver emotion intelligent interaction method based on face information. The method comprises the steps that firstly, the face information is collected under the multi-view, multi-angle and multi-illumination conditions, and a deep learning method is used for model training to generate an expression recognition model; then real-time acquisition and expression recognition of face information of a driver are carried out in real time through a multi-view, multi-angle and multi-illumination camera; according to the current emotional state of the driver, various information prompts including light, music, seats and the like are carried out, and the emotion of the driver is adjusted; and finally, according to expression recognition of the driver reminded by various kinds of information, judging whether adjustment and intervention of additional factors need to be carried out again until the emotion of the driver tends to be stable. On the basis of automatically recognizing the emotion of the driver in real time, interactive recognition of the emotion of the driver is realized through intelligent intervention of external factors, so that the emotion stability of the driver can be ensured, and road accidents can be reduced.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent control technology, and in particular relates to a driver emotion intelligent interaction method, system and device based on facial information. Background Art

[0002] At present, with the rapid development of the digital economy, the core part of the car cockpit based on artificial intelligence technology - human-computer interaction will undergo tremendous changes. The car has been upgraded from a single driving and riding tool to a consumer-centric "smart mobile space". This space is expected to become the third basic living space in addition to residential and office spaces.

[0003] Regardless of how technology advances, safety is an eternal topic in the automotive society, and the safety of drivers and passengers is of paramount importance. This system mainly uses software and hardware technologies such as sensors, cameras, face recognition, artificial intelligence, and big data analysis to improve image quality and recognition accuracy, and to achieve real-time monitoring of driver status. It also identifies and reminds dangerous driving behaviors based on the constructed evaluation method, improves driving safety, and protects the safety of life and property of the people and economic stability. At the same time, through the development of intelligent control systems for driver and passenger emotion recognition and information interaction, the interaction between the in-cabin driver and passenger and the in-car atmosphere is achieved, and the driver and passenger are affected by adjusting brightness and color according to their emotions, making them feel relaxed and comfortable, or excited and inspired, and improving the intuitive driving experience. It can also improve the driver's cognitive ability, which helps to increase the safety factor of driving.

[0004] The research and development and industrialization of the project can not only use technical means to strengthen the supervision of drivers' unsafe behaviors and enhance drivers' safety awareness, but also effectively reduce and control the occurrence of unsafe behaviors of drivers, control the occurrence of accidents, avoid and reduce casualties and property losses; at the same time, it also brings excellent value experience to drivers and passengers. It is an optimal demonstration point for big data applications and contains huge commercial value and social value. Summary of the invention

[0005] The technical problem to be solved by the present invention is to design a method, system and device for intelligent interaction of driver emotions based on facial information, which utilizes intelligent interactive adjustment of various information including lights, music, seats, etc. to effectively control the driver's negative emotions, thereby improving driving safety, protecting the lives and property of the people and economic stability, and solving existing technical problems.

[0006] In order to solve the above technical problems, the present invention provides a driver emotional intelligent interaction method based on face information, comprising the following steps:

[0007] Step S1: Collect facial information under multiple viewing angles, multiple angles, and multiple lighting conditions, and use deep learning methods to train the model to generate an expression recognition model.

[0008] Step S2: The driver's facial information is collected in real time under multiple viewing angles, multiple angles, and multiple lighting conditions through a camera, and the driver's facial expression is recognized in real time.

[0009] Step S3: Determine whether the driver's current emotional state is a negative emotion based on the recognition result:

[0010] If so, the driver's mood is adjusted by controlling the lights, music and seats in the car;

[0011] If not, return to step S2.

[0012] Step S4: After the adjustment is completed, the driver's expression is recognized again, and it is determined whether multiple adjustments and interventions of external factors are needed according to the recognition results until the driver's emotions tend to be stable.

[0013] Step S5: During the driving process, steps S2 to S4 are continuously executed in sequence until the driving process ends.

[0014] Furthermore, in step S1, the multi-view facial expression information collection method is: using the camera equipment arranged at the front end of the cockpit of each type of motor vehicle and the camera equipment arranged at the side of each type of motor vehicle to collect various facial expression messages including anger, disgust, fear, happiness, surprise, sadness and neutrality.

[0015] Furthermore, in step S1, the method for collecting multi-angle facial expression information is: based on the multi-viewing angle arrangement of camera equipment, various facial expression messages including anger, disgust, fear, happiness, surprise, sadness and neutrality are collected from different angles including the biased camera.

[0016] Furthermore, in step S1, the method for collecting multi-illumination facial expression information is: on the basis of multiple viewing angles and multiple angles, various facial expression information including anger, disgust, fear, happiness, surprise, sadness and neutrality are collected under various conditions including no sunlight on the face in sunny weather, sunlight on the face in sunny weather, partial sunlight on the face in sunny weather, strong backlight on the face in sunny weather, interior lights on the face in weak light, street lights on the face in weak light, headlights from the opposite side on the face in weak light, dark conditions, and dark conditions with weak light on the face.

[0017] Furthermore, the collection of facial information also includes:

[0018] Multi-gender and multi-age experimental population: Under the premise of a male-to-female ratio of 7:3, the age groups of 18-30 years old: 31-40 years old: 41-50 years old: 51-60 years old: 61-70 years old: 71-80 years old were selected with a ratio of 12:16:8:4:2:1.

[0019] Furthermore, in step S1, the miniaturized lightweight facial expression recognition model based on multi-scale feature fusion and global attention specifically includes:

[0020] The dilated spatial pyramid pooling is used to capture local details and global context information of the image, which includes multiple parallel dilated convolutions with different sampling rates.

[0021] The feature fusion module with a dual-branch structure accurately captures the facial features of different facial expressions by fusing feature maps at different levels;

[0022] The compression excitation channel attention module strengthens important features and suppresses unimportant features, enhancing the feature representation ability of convolutional neural networks.

[0023] Furthermore, in step S3, the method of intervening and regulating the driver's emotions includes: automatically and intelligently performing various adjustments, including the color of the interior lights, the intensity of the interior lights, the duration of the interior lights, the type of music played, the time of playing music, the up and down height of the driving seat, and the front and rear distance of the driving seat.

[0024] Furthermore, in step S4, if the driver's emotions tend to calm down within 3 to 5 seconds, the intelligent adjustments corresponding to the current various reminders are maintained; if the driver's emotions cannot calm down within 3 to 5 seconds, various intelligent adjustments for emotional adjustment are continued until the driver's emotions calm down.

[0025] The present invention also provides a driver emotion intelligent interaction system based on face information, comprising:

[0026] The face information collection module is used to collect face information for model training and to collect driver information in real time for identification;

[0027] Model learning module, which uses deep learning methods to train models and generate expression recognition models;

[0028] The expression recognition module is used to recognize the driver's expression collected in real time and output the result;

[0029] A first judgment module is used to judge the recognized driver's emotion;

[0030] Emotion regulation module, used to regulate the driver's emotions;

[0031] The second judgment module is used to judge the driver's emotions after emotion adjustment.

[0032] The present invention also provides a driver emotion intelligent interaction device based on face information, comprising:

[0033] at least one processor; and

[0034] at least one memory in communication with the processor;

[0035] Among them, the memory stores instructions that can be executed by the processor, and the instructions are executed by the processor to enable the device to perform the aforementioned driver emotional intelligent interaction method based on facial information.

[0036] Beneficial effects of the present invention:

[0037] The intelligent interactive method and system of driver emotions based on facial information of the present invention realizes interactive identification of driver emotions through intelligent intervention of external factors on the basis of real-time automatic identification of driver emotions, and realizes effective control of driver's negative emotions through intelligent interactive adjustment of various information including lights, music, seats, etc., which is conducive to ensuring the emotional stability of the driver, thereby improving driving safety, and is more conducive to reducing road accidents and protecting the safety of people's lives and property and economic stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The specific implementation of the present invention will be further explained below in conjunction with the accompanying drawings.

[0039] Figure 1 A flowchart of the driver's emotional intelligence interaction method based on facial information of the present invention;

[0040] Figure 2 A schematic diagram of the intelligent vehicle condition adjustment and interactive emotion recognition process in the invention;

[0041] Figure 3 The block diagram of the driver emotional intelligent interaction system based on facial information of the present invention. DETAILED DESCRIPTION

[0042] Example 1

[0043] Combination Figure 1 and Figure 2 , a driver emotional intelligence interaction method based on face information of this embodiment includes the following steps:

[0044] Step S1: Collect facial information under multiple viewing angles, multiple angles, and multiple lighting conditions, and use deep learning methods to train the model to generate an expression recognition model.

[0045] Preferably, in step S1 of the present embodiment, the method for collecting multi-view facial expression information is as follows: using the camera equipment arranged at the front end of the cockpit of each type of motor vehicle and the camera equipment arranged at the side of each type of motor vehicle respectively, to collect various facial expression messages including anger, disgust, fear, happiness, surprise, sadness and neutrality.

[0046] Preferably, in step S1 of the present embodiment, the method for collecting multi-angle facial expression information is: based on the multi-viewing angle arrangement of camera equipment, various facial expression messages including anger, disgust, fear, happiness, surprise, sadness and neutrality are collected from different angles including the biased camera.

[0047] Specifically, in this embodiment, various facial expression messages are collected from various angles including 0°, 10°, 30° and 40° towards the camera.

[0048] Preferably, in step S1 of the present embodiment, the method for collecting multi-illumination facial expression information is: on the basis of multiple viewing angles and multiple angles, respectively, under various conditions including no sunlight on the face in sunny weather, sunlight on the face in sunny weather, partial sunlight on the face in sunny weather, strong backlight on the face in sunny weather, interior lights on the face in weak light, street lights on the face in weak light, headlights from the opposite side on the face in weak light, dark conditions, and dark conditions with weak light on the face, to collect various facial expression information including anger, disgust, fear, happiness, surprise, sadness and neutrality.

[0049] In this embodiment, preferably, in step S1, collecting facial information further includes:

[0050] Multi-gender and multi-age experimental population: Under the premise of a male-to-female ratio of 7:3, the age groups of 18-30 years old: 31-40 years old: 41-50 years old: 51-60 years old: 61-70 years old: 71-80 years old were selected with a ratio of 12:16:8:4:2:1.

[0051] In this embodiment, preferably, in step S1, the miniaturized lightweight facial expression recognition model based on multi-scale feature fusion and global attention specifically includes:

[0052] The dilated spatial pyramid pooling is used to capture local details and global context information of the image, which includes multiple parallel dilated convolutions with different sampling rates.

[0053] The dual-branch feature fusion module accurately captures the facial features of different facial expressions by fusing feature maps at different levels, thereby improving the accuracy and generalization of the model;

[0054] The compression excitation channel attention module strengthens important features and suppresses unimportant features, enhancing the feature representation ability of the convolutional neural network, thereby improving model performance.

[0055] Step S2: The driver's facial information is collected in real time under multiple viewing angles, multiple angles, and multiple lighting conditions through a camera, and the driver's facial expression is recognized in real time.

[0056] Step S3: Determine whether the driver's current emotional state is a negative emotion based on the recognition result:

[0057] If so, the driver's mood is adjusted by controlling the lights, music and seats in the car;

[0058] If not, return to step S2.

[0059] Preferably, in step S3 of the present embodiment, the method for intervening and regulating the driver's emotions includes: automatically and intelligently performing various adjustments, including the color of the interior lights, the intensity of the interior lights, the duration of the interior lights, the type of music played, the time of playing music, the up and down height of the driving seat, and the front and back distance of the driving seat.

[0060] Specifically, in step S3 of the present embodiment, information reminders are provided while adjusting the driver's emotions.

[0061] Step S4: After the adjustment is completed, the driver's expression is recognized again, and it is determined whether multiple adjustments and interventions of external factors are needed according to the recognition results until the driver's emotions tend to be stable.

[0062] Preferably, in this embodiment, in step S4, if the driver's emotions tend to calm down within 3 to 5 seconds, the intelligent adjustments corresponding to the current various reminders are maintained; if the driver's emotions cannot calm down within 3 to 5 seconds, various intelligent adjustments for emotional adjustment are continued until the driver's emotions calm down.

[0063] Step S5: During the driving process, steps S2 to S4 are continuously executed in sequence until the driving process ends.

[0064] Example 2

[0065] Combination Figure 3 , a driver emotional intelligent interaction system based on face information of this embodiment includes:

[0066] The face information collection module is used to collect face information for model training and to collect driver information in real time for identification;

[0067] Model learning module, which uses deep learning methods to train models and generate expression recognition models;

[0068] The expression recognition module is used to recognize the driver's expression collected in real time and output the result;

[0069] A first judgment module is used to judge the recognized driver's emotion;

[0070] Emotion regulation module, used to regulate the driver's emotions;

[0071] The second judgment module is used to judge the driver's emotions after emotion adjustment.

[0072] Preferably, this embodiment further includes a control module, which is used to control the aforementioned modules to execute relevant steps according to the method in Example 1.

[0073] Example 3

[0074] This embodiment provides a driver emotion intelligent interaction device based on face information, including:

[0075] at least one processor; and

[0076] at least one memory in communication with the processor;

[0077] Among them, the memory stores instructions that can be executed by the processor, and the instructions are executed by the processor to enable the device to execute the driver's emotional intelligent interaction method based on facial information in Example 1.

[0078] Many specific details are described in the above description to facilitate a full understanding of the present invention. However, the above description is only a preferred embodiment of the present invention. The present invention can be implemented in many other ways different from those described herein, so the present invention is not limited to the specific implementation disclosed above. At the same time, any person familiar with the art can make many possible changes and modifications to the technical solution of the present invention using the methods and technical contents disclosed above without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Claims

1. A driver emotional intelligent interaction method based on face information, characterized by: The steps include: Step S1: Collecting facial information under multiple viewing angles, multiple angles, and multiple lighting conditions, and using deep learning methods to train the model to generate an expression recognition model; Step S2: collecting the driver's facial information in real time under multiple viewing angles, multiple angles, and multiple lighting conditions through a camera, and performing real-time expression recognition of the driver; Step S3: Determine whether the driver's current emotional state is a negative emotion based on the recognition result: If so, the driver's mood is adjusted by controlling the lights, music and seats in the car; If not, return to step S2; Step S4: After the adjustment is completed, the driver's expression is recognized again, and it is determined whether multiple external factors need to be adjusted and intervened according to the recognition result until the driver's emotion tends to be stable; Step S5: During the driving process, steps S2 to S4 are continuously executed in sequence until the driving process ends.

2. The driver emotional intelligence interaction method based on face information according to claim 1 is characterized by: In step S1, the multi-view facial expression information collection method is: using the camera equipment arranged at the front end of the cockpit of each type of motor vehicle and the camera equipment arranged at the side of each type of motor vehicle to collect various facial expression messages including anger, disgust, fear, happiness, surprise, sadness and neutrality.

3. The driver emotional intelligence interaction method based on face information according to claim 1 is characterized by: In step S1, the method for collecting multi-angle facial expression information is: based on the multi-angle arrangement of camera equipment, various facial expression messages including anger, disgust, fear, happiness, surprise, sadness and neutrality are collected from different angles including the biased camera.

4. The driver emotional intelligence interaction method based on face information according to claim 1 is characterized by: In step S1, the method for collecting multi-illumination facial expression information is: on the basis of multiple viewing angles and multiple angles, respectively, under various conditions including no sunlight on the face in sunny weather, sunlight on the face in sunny weather, partial sunlight on the face in sunny weather, strong backlight on the face in sunny weather, interior lights on the face in weak light, street lights on the face in weak light, headlights from the opposite side on the face in weak light, dark conditions, and dark conditions with weak light on the face, to collect various facial expression information including anger, disgust, fear, happiness, surprise, sadness and neutrality.

5. The driver emotional intelligent interaction method based on face information according to claim 1 is characterized by: The collection of facial information also includes: Multi-gender and multi-age experimental population: Under the premise of a male-to-female ratio of 7:3, the age groups of 18-30 years old: 31-40 years old: 41-50 years old: 51-60 years old: 61-70 years old: 71-80 years old were selected with a ratio of 12:16:8:4:2:

1.

6. The driver emotional intelligence interaction method based on face information according to claim 1 is characterized by: In step S1, a miniaturized lightweight facial expression recognition model based on multi-scale feature fusion and global attention specifically includes: The dilated spatial pyramid pooling is used to capture local details and global context information of the image, which includes multiple parallel dilated convolutions with different sampling rates. The feature fusion module with a dual-branch structure accurately captures the facial features of different facial expressions by fusing feature maps at different levels; The compression excitation channel attention module strengthens important features and suppresses unimportant features, enhancing the feature representation ability of convolutional neural networks.

7. The driver emotional intelligence interaction method based on face information according to claim 1 is characterized by: In step S3, the method of intervening and adjusting the driver's emotions includes: automatically and intelligently performing various adjustments, including the color of the interior lights, the intensity of the interior lights, the duration of the interior lights, the type of music played, the time of playing music, the up and down height of the driver's seat, and the front and back distance of the driver's seat.

8. The driver emotional intelligence interaction method based on face information according to claim 1 is characterized by: In step S4, if the driver's emotions tend to calm down within 3 to 5 seconds, the intelligent adjustments corresponding to the current various reminders are maintained; if the driver's emotions cannot calm down within 3 to 5 seconds, various intelligent adjustments for emotional adjustment are continued until the driver's emotions calm down.

9. A driver emotional intelligent interaction system based on facial information, characterized by: include: The face information collection module is used to collect face information for model training and to collect driver information in real time for identification; Model learning module, which uses deep learning methods to train models and generate expression recognition models; The expression recognition module is used to recognize the driver's expression collected in real time and output the result; A first judgment module is used to judge the recognized driver's emotion; Emotion regulation module, used to regulate the driver's emotions; The second judgment module is used to judge the driver's emotions after emotion adjustment.

10. A driver emotional intelligent interaction device based on facial information, characterized in that: include: at least one processor; as well as at least one memory in communication with the processor; Wherein, the memory stores instructions executable by the processor, and the instructions are executed by the processor so that the device executes the driver emotional intelligent interaction method based on facial information as described in any one of claims 1-8.

Images