Driver emotion recognition method and system based on multi-information fusion

Abstract

The invention is suitable for the technical field of automatic driving, and provides a driver emotion recognition method and system based on multi-information fusion, and the system comprises an expression emotion level obtaining module which is used for obtaining a driver image, calculating the three-dimensional key point information of a face according to the driver image, and storing the three-dimensional key point information of the face; performing facial expression analysis based on the three-dimensional key point information, and performing grading processing to obtain expression emotion grades; the voice emotion level acquisition module is used for acquiring voice information by monitoring a microphone in the vehicle, performing emotion recognition on the voice information through a BERT model to obtain a driver emotion sequence, and calculating an expression emotion level of the driver based on the driver emotion sequence; and a vehicle driving information acquisition module and a final emotion score module. The emotional state of the driver can be identified more accurately, and guidance and data support are provided for safe driving of the driver.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving, and more particularly to a driver emotion recognition method and system based on multi-information fusion. Background Art

[0002] With the rapid development of autonomous driving technology, cars are gradually evolving from simple means of transportation into intelligent mobile spaces. However, at the current stage, autonomous driving technology is still in its infancy, and in complex environments and edge cases, human drivers are still required to take over the vehicle. Therefore, the emotional state of the driver is crucial for driving safety. The driver's emotions directly affect driving behavior and decision-making ability. Negative emotions, such as anger, anxiety, fatigue, etc., will reduce the driver's attention, judgment, and operation ability, leading to an increased risk of traffic accidents.

[0003] Research shows that fatigue driving is one of the important causes of traffic accidents, and its danger is comparable to that of drunk driving. Currently, the methods for identifying driver emotions mainly include the following: physiological signal recognition, facial expression recognition, speech recognition, vehicle driving information analysis, etc. Each of the above methods has its own advantages and disadvantages, but there is a high possibility of misjudgment due to single information recognition. For example, physiological signal recognition is easily affected by environmental factors, facial expression recognition may not be able to recognize hidden emotions, speech recognition is easily interfered by noise, and vehicle driving information analysis may be affected by driving experience and road conditions. Therefore, a driver emotion recognition method and system based on multi-information fusion are proposed to solve the above problems. Summary of the Invention

[0004] Aiming at the deficiencies of the existing technology, the purpose of the present invention is to provide a driver emotion recognition method and system based on multi-information fusion to solve the problems existing in the above background art.

[0005] The present invention is implemented as follows. A driver emotion recognition system based on multi-information fusion, the system includes: An expression emotion level acquisition module, configured to acquire a driver image, calculate three-dimensional key point information of the face based on the driver image, perform facial expression analysis based on the three-dimensional key point information, and perform grading processing to obtain an expression emotion level; A voice emotion level acquisition module, configured to acquire voice information by listening to the in-vehicle microphone, perform emotion recognition on the voice information through a BERT model to obtain a driver emotion sequence, and calculate the expression emotion level of the driver based on the driver emotion sequence; A vehicle driving information acquisition module, configured to acquire the throttle information, brake information, and steering wheel information of the vehicle itself; The final emotion score module is used for a BP neural network introducing a self-attention mechanism, obtaining an emotion score based on the expression emotion level, expression emotion level, throttle information, brake information, and steering wheel information, and outputting it.

[0006] As a further solution of the present invention: the expression emotion level acquisition module includes: An image acquisition unit, used to acquire a driver image through a monitoring camera set in the vehicle; A face recognition unit, used to obtain a face bounding box and a confidence level according to the driver image by using a two-stage model, and obtaining three-dimensional key point information for the face area in the face bounding box through the two-stage model; An expression emotion acquisition unit, used to calculate an expression emotion sequence by using the MediapipeBlendshapePrediction model for the three-dimensional key point information; An expression emotion grading unit, used to calculate the driver's expression emotion level according to the obtained expression emotion sequence.

[0007] As a further solution of the present invention: the voice emotion level acquisition module includes: An in-vehicle voice acquisition unit, used to acquire the driver's voice information by listening to the microphone for making or answering calls in the vehicle; A natural language processing unit, used to perform language emotion recognition and analysis through a fine-tuned and trained BERT model; A language emotion acquisition unit, used to obtain the driver's emotion sequence by parsing the output of the BERT model; A language emotion grading unit, used to calculate the driver's language emotion level according to the obtained driver's emotion sequence.

[0008] As a further solution of the present invention: the vehicle driving information acquisition module includes: A throttle monitoring unit, used to subscribe to the throttle pedal opening in the chassis information to obtain throttle information; A brake monitoring unit, used to subscribe to the brake pedal opening in the chassis information to obtain brake information; A steering wheel monitoring unit, used to subscribe to the steering wheel angle and angular acceleration in the chassis information to obtain steering wheel information.

[0009] As a further solution of the present invention: the final emotion score module includes: An emotion score regression unit, through a BP neural network introducing a self-attention mechanism, taking the expression emotion level, language emotion level, throttle information, brake information, and steering wheel information as model inputs, autonomously optimizing the weight of each input through model training, finally fixing the model parameters, and obtaining the emotion score by calculating the model output with real-time input; An emotion score output unit, configured to obtain the emotion score calculated and processed by the emotion score regression unit.

[0010] Another object of the present invention is to provide a driver emotion recognition method based on multi-information fusion. The method includes the following steps: Obtain a driver image, calculate three-dimensional key point information of the face based on the driver image, perform facial expression analysis based on the three-dimensional key point information, and perform grading processing to obtain an expression emotion level; Obtain voice information by listening to the in-vehicle microphone, perform emotion recognition on the voice information through a BERT model to obtain a driver emotion sequence, and calculate the expression emotion level of the driver based on the driver emotion sequence; Obtain the throttle information, brake information, and steering wheel information of the vehicle itself; Introduce a BP neural network with a self-attention mechanism, obtain an emotion score based on the expression emotion level, expression emotion level, throttle information, brake information, and steering wheel information, and output it.

[0011] As a further solution of the present invention: The step of obtaining the driver image, calculating three-dimensional key point information of the face based on the driver image, performing facial expression analysis based on the three-dimensional key point information, and performing grading processing to obtain an expression emotion level specifically includes: Obtain a driver image through a monitoring camera set in the vehicle; Obtain a face bounding box and a confidence level according to the driver image by using a two-stage model, and obtain three-dimensional key point information of the face area in the face bounding box through the two-stage model; Calculate a sequence of expression emotions through a MediapipeBlendshapePrediction model for the three-dimensional key point information; Calculate the driver's expression emotion level according to the obtained sequence of expression emotions.

[0012] As a further solution of the present invention: The step of obtaining voice information by listening to the in-vehicle microphone, performing emotion recognition on the voice information through a BERT model to obtain a driver emotion sequence, and calculating the expression emotion level of the driver based on the driver emotion sequence specifically includes: Obtain the driver's voice information by listening to the microphone for making or receiving calls in the vehicle; Perform language emotion recognition and analysis through a fine-tuned and trained BERT model; Parse the output of the BERT model to obtain a driver emotion sequence; Calculate the driver's language emotion level according to the obtained driver emotion sequence.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: Through multi-data fusion and machine learning algorithms, the present invention comprehensively considers the driver's facial expressions, in-vehicle voice, and vehicle driving information, reducing the influence of environmental and human factors encountered in driver emotion recognition using traditional single information sources. For example, inaccurate facial expression recognition caused by lighting and facial occlusion, and inaccurate analysis of vehicle driving information due to different driving styles of drivers and road conditions. With the collaborative effect of four modules, namely the facial expression emotion level acquisition module, the voice emotion level acquisition module, the vehicle driving information acquisition module, and the final emotion score module, it effectively overcomes the disadvantages of low recognition accuracy, easy misjudgment, and poor anti-interference ability when using traditional single information sources for driver emotion recognition, and can more accurately identify the driver's emotional state, providing guidance and data support for the driver's safe driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 FIG. is a schematic structural diagram of a driver emotion recognition system based on multi-information fusion.

[0015] Figure 2 FIG. is a flowchart of a driver emotion recognition method based on multi-information fusion.

[0016] Figure 3 FIG. is a flowchart of obtaining the facial expression emotion level in the driver emotion recognition method based on multi-information fusion.

[0017] Figure 4 FIG. is a flowchart of obtaining the voice emotion level in the driver emotion recognition method based on multi-information fusion. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0019] The following describes the specific implementation of the present invention in detail with reference to specific embodiments.

[0020] As Figure 1 shown, the embodiment of the present invention provides a driver emotion recognition system based on multi-information fusion, and the system includes: A facial expression emotion level acquisition module 100, configured to acquire a driver image, calculate three-dimensional key point information of the face based on the driver image, perform facial expression analysis based on the three-dimensional key point information, and perform grading processing to obtain a facial expression emotion level; The voice emotion level acquisition module 200 is used to obtain voice information by listening to the in-vehicle microphone, and perform emotion recognition on the voice information through the BERT model to obtain the driver's emotion sequence, and calculate the driver's facial expression emotion level based on the driver's emotion sequence; The vehicle driving information acquisition module 300 is used to obtain the throttle information, brake information, and steering wheel information of the vehicle itself; The final emotion score module 400 is used to introduce a BP neural network with a self-attention mechanism, and obtain and output an emotion score according to the facial expression emotion level, facial expression emotion level, throttle information, brake information, and steering wheel information.

[0021] It should be noted that the methods for identifying the driver's emotion mainly include the following: physiological signal recognition, facial expression recognition, speech recognition, vehicle driving information analysis, etc. Each of the above methods has its own advantages and disadvantages, but there is a high possibility of misjudgment due to single information recognition.

[0022] In the embodiment of the present invention, through multi-data fusion and machine learning algorithms, the present invention comprehensively considers the driver's facial expression, in-vehicle voice, and vehicle driving information, reduces the influence of environmental and human factors encountered in traditional single-information-source driver emotion recognition, such as inaccurate facial expression recognition caused by light and facial occlusion, inaccurate vehicle driving information analysis caused by different driving styles and road conditions of different drivers, etc. By using the four modules of the facial expression emotion level acquisition module, voice emotion level acquisition module, vehicle driving information acquisition module, and final emotion score module in cooperation, it effectively overcomes the disadvantages of low recognition accuracy, easy misjudgment, and poor anti-interference ability when using traditional single information sources to identify the driver's emotion, and can more accurately identify the driver's emotion state, providing guidance and data support for the driver's safe driving.

[0023] As Figure 1 shown, as a preferred embodiment of the present invention, the facial expression emotion level acquisition module 100 includes: The image acquisition unit 101 is used to obtain the driver's image through a monitoring camera set in the vehicle; The facial recognition unit 102 is used to obtain the face bounding box and confidence according to the driver's image by using a two-stage model, and obtain three-dimensional key point information for the face area in the face bounding box through the two-stage model; The facial expression emotion acquisition unit 103 is used to calculate the sequence of facial expression emotions by using the MediapipeBlendshapePrediction model for the three-dimensional key point information; The facial expression emotion grading unit 104 is used to calculate the driver's facial expression emotion level according to the obtained sequence of facial expression emotions.

[0024] In the embodiments of the present invention, the acquisition of the expression and emotion level includes functions of acquiring the real-time image of the driver, facial three-dimensional key point recognition, acquiring the expression and emotion, and acquiring the expression and emotion level. Among them, the function of acquiring the real-time image of the driver is obtained through the camera of the in-vehicle driver monitoring system; the facial three-dimensional key point recognition is realized by a two-stage model. First, the acquired driver image is scaled to a size of 192*192, and then the image is converted into the RGB format and sent to the MediapipeFace Detection model for calculation to obtain the face bounding box and confidence level output by the model. Subsequently, the face area in the face bounding box is scaled to a size of 256*256 and sent to the MediapipeFaceMesh model for calculation to obtain 478 three-dimensional key point information of the face; the function of acquiring the expression and emotion is to select the two-dimensional information of 146 key points on the face and send it to the MediapipeBlendshapePrediction model for calculation to obtain a sequence containing 52 expression coefficients, representing the possibility of each expression; the function of acquiring the expression and emotion level is to calculate the driver's expression and emotion level according to the possibility of each expression. The expression and emotion level is divided into ten levels. The higher the level, the more positive the driver's expression and emotion. The lower the level, the more depressed or irritable the driver's expression and emotion.

[0025] As Figure 1 shown, as a preferred embodiment of the present invention, the voice emotion level acquisition module 200 includes: An in-vehicle voice acquisition unit 201, configured to acquire the voice information of the driver by listening to the microphone for making or answering calls in the vehicle; A natural language processing unit 202, configured to perform language emotion recognition and analysis through a fine-tuned and trained BERT model; A language emotion acquisition unit 203, configured to obtain the driver emotion sequence by parsing the output of the BERT model; A language emotion grading unit 204, configured to calculate the driver's language emotion level according to the obtained driver emotion sequence.

[0026] In the embodiments of the present invention, the acquisition of the speech emotion level includes functions of acquiring in-vehicle speech information, natural language processing, acquiring language emotion, and acquiring the language emotion level. Among them, the function of acquiring in-vehicle speech information is obtained through the in-vehicle call microphone. The natural language processing function performs language emotion recognition through a fine-tuned and trained BERT model. First, the speech content is converted into text content, and then preprocessing operations such as recombination and cleaning are performed on the text content. Subsequently, the content is encoded and input into the BERT model for language emotion analysis; the function of acquiring language emotion obtains a sequence containing 20 driver emotion coefficients by parsing the output of the BERT model, representing the likelihood of each driver's language emotion; the function of acquiring the language emotion level calculates the driver's language emotion level according to the likelihood of each language emotion. The language emotion level is divided into ten levels. The higher the level, the more positive the driver's language emotion is, and the lower the level, the more depressed or irritable the driver's language emotion is.

[0027] As Figure 1 shown, as a preferred embodiment of the present invention, the vehicle driving information acquisition module 300 includes: An accelerator monitoring unit 301 for subscribing to the accelerator pedal opening in the chassis information to obtain accelerator information; A brake monitoring unit 302 for subscribing to the brake pedal opening in the chassis information to obtain brake information; A steering wheel monitoring unit 303 for subscribing to the steering wheel angle and angular acceleration in the chassis information to obtain steering wheel information.

[0028] In the embodiments of the present invention, the acquisition methods of the accelerator information, brake information, and steering wheel information are easily achievable in the prior art. The current vehicle electronic control system is very developed, and there can be many sources for obtaining these information.

[0029] As Figure 1 shown, as a preferred embodiment of the present invention, the final emotion score module 400 includes: An emotion score regression unit 401, through a BP neural network introducing a self-attention mechanism, takes the facial expression emotion level, language emotion level, accelerator information, brake information, and steering wheel information as model inputs, autonomously optimizes the weight of each input through model training, finally fixes the model parameters, and calculates the model output through real-time input to obtain the emotion score; An emotion score output unit 402 for obtaining the emotion score calculated and processed by the emotion score regression unit.

[0030] In the embodiments of the present invention, the final emotion score includes an emotion score regression function and an output emotion score function. Among them, the emotion score regression function is implemented by introducing a BP neural network with a self-attention mechanism. The facial expression emotion level, language emotion level, throttle information, brake information, and steering wheel information are used as model inputs. Each input weight is autonomously optimized through model training, and finally the model parameters are fixed. In subsequent calculations, the emotion score is obtained by inputting in real time and calculating the model output. The emotion score is out of 100. The higher the score, the more positive or stable the driver's emotion. The lower the score, the more depressed or irritable the driver's emotion. The output emotion score function is to obtain the emotion score of the emotion score regression.

[0031] As Figure 2 shown, the embodiments of the present invention also provide a driver emotion recognition method based on multi-information fusion. The method includes the following steps: S100, obtain a driver image, calculate the three-dimensional key point information of the face based on the driver image, and perform facial expression analysis and grading processing based on the three-dimensional key point information to obtain the facial expression emotion level; S200, obtain voice information by listening to the in-vehicle microphone, perform emotion recognition on the voice information through a BERT model to obtain the driver emotion sequence, and calculate the facial expression emotion level of the driver based on the driver emotion sequence; S300, obtain the throttle information, brake information, and steering wheel information of the vehicle itself; S400, introduce a BP neural network with a self-attention mechanism, obtain the emotion score based on the facial expression emotion level, facial expression emotion level, throttle information, brake information, and steering wheel information, and output it.

[0032] In the embodiments of the present invention, through multi-data fusion and machine learning algorithms, the present invention comprehensively considers the driver's facial expressions, in-vehicle voice, and vehicle driving information, reducing the influence of environmental and human factors encountered in driver emotion recognition using traditional single information sources, such as inaccurate facial expression recognition due to light and facial occlusion, and inaccurate analysis of vehicle driving information due to different driving styles and road conditions of different drivers. With the cooperation of four modules: the facial expression emotion level acquisition module, the voice emotion level acquisition module, the vehicle driving information acquisition module, and the final emotion score module, it effectively overcomes the disadvantages of low recognition accuracy, easy misjudgment, and poor anti-interference ability when using traditional single information sources for driver emotion recognition, and can more accurately identify the driver's emotional state, providing guidance and data support for the driver's safe driving.

[0033] As Figure 3As shown, as a preferred embodiment of the present invention, the steps of obtaining a driver image, calculating three-dimensional key point information of the face based on the driver image, performing facial expression analysis based on the three-dimensional key point information, and performing grading processing to obtain an expression emotion level specifically include: S101, obtaining a driver image through a monitoring camera arranged inside the vehicle; S102, obtaining a face bounding box and a confidence level according to the driver image by using a two-stage model, and obtaining three-dimensional key point information for the face region in the face bounding box through the two-stage model; S103, calculating a sequence of expression emotions for the three-dimensional key point information through the MediapipeBlendshapePrediction model; S104, calculating a driver expression emotion level according to the obtained sequence of expression emotions.

[0034] As Figure 4 shown, as a preferred embodiment of the present invention, the steps of obtaining voice information by listening to an in-vehicle microphone, performing emotion recognition on the voice information through a BERT model to obtain a driver emotion sequence, and calculating a driver's expression emotion level based on the driver emotion sequence specifically include: S201, obtaining the driver's voice information by listening to the microphone for making or answering calls inside the vehicle; S202, performing language emotion recognition and analysis through a fine-tuned and trained BERT model; S203, parsing the output of the BERT model to obtain a driver emotion sequence; S204, calculating a driver language emotion level according to the obtained driver emotion sequence.

[0035] The above only describes the preferred embodiments of the present invention in detail, and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

[0036] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown in sequence according to the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limitation, and these steps can be executed in other orders. Moreover, at least a part of the steps in each embodiment may include multiple sub-steps or multiple stages. These sub-steps or stages do not necessarily have to be executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages does not necessarily have to be sequential, but can be executed alternately or in turn with at least a part of other steps or sub-steps or stages of other steps.

[0037] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0038] After considering the specification and the disclosure of the embodiments, those skilled in the art will readily conceive of other embodiments of the present disclosure. The present application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and the embodiments are only regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the claims.

Claims

1. A driver emotion recognition system based on multi-information fusion, characterized in that The system includes: An expression and emotion level acquisition module, which is used to acquire a driver image, calculate three-dimensional key point information of the face based on the driver image, perform facial expression analysis based on the three-dimensional key point information, and perform grading processing to obtain an expression and emotion level; A voice emotion level acquisition module, which is used to acquire voice information by listening to the in-vehicle microphone, perform emotion recognition on the voice information through a BERT model to obtain a driver emotion sequence, and calculate the driver's expression and emotion level based on the driver emotion sequence; A vehicle driving information acquisition module, which is used to acquire the throttle information, brake information, and steering wheel information of the vehicle itself; A final emotion score module, which is used to introduce a BP neural network with a self-attention mechanism, obtain an emotion score based on the expression and emotion level, expression and emotion level, throttle information, brake information, and steering wheel information, and output it.

2. The driver emotion recognition system based on multi-information fusion according to claim 1, wherein, The expression and emotion level acquisition module includes: An image acquisition unit, which is used to acquire a driver image through a monitoring camera installed in the vehicle; A face recognition unit, which is used to obtain a face bounding box and a confidence level according to the driver image by using a two-stage model, and obtain three-dimensional key point information of the face area in the face bounding box through the two-stage model; An expression and emotion acquisition unit, which is used to calculate an expression and emotion sequence by using a MediapipeBlendshapePrediction model for the three-dimensional key point information; An expression and emotion grading unit, which is used to calculate the driver's expression and emotion level according to the obtained expression and emotion sequence.

3. The driver emotion recognition system based on multi-information fusion according to claim 1, characterized in that The voice emotion level acquisition module includes: An in-vehicle voice acquisition unit, which is used to acquire the driver's voice information by listening to the microphone for making or answering calls in the vehicle; A natural language processing unit, which is used to perform language emotion recognition and analysis through a fine-tuned and trained BERT model; A language emotion acquisition unit, which is used to obtain a driver emotion sequence by parsing the output of the BERT model; A language emotion grading unit, which is used to calculate the driver's language emotion level according to the obtained driver emotion sequence.

4. The driver emotion recognition system based on multi-information fusion according to claim 1, characterized in that, The vehicle driving information acquisition module includes: A throttle monitoring unit, which is used to subscribe to the throttle pedal opening in the chassis information to obtain throttle information; A brake monitoring unit, which is used to subscribe to the brake pedal opening in the chassis information to obtain brake information; A steering wheel monitoring unit, which is used to subscribe to the steering wheel angle and angular acceleration in the chassis information to obtain steering wheel information.

5. The driver emotion recognition system based on multi-information fusion according to claim 1, characterized in that, The final emotion score module includes: An emotion score regression unit, which introduces a BP neural network with a self-attention mechanism, takes the expression and emotion level, language emotion level, throttle information, brake information, and steering wheel information as model inputs, autonomously optimizes the weights of each input through model training, finally fixes the model parameters, and calculates the model output through real-time input to obtain an emotion score; An emotion score output unit, which is used to acquire the emotion score calculated and processed by the emotion score regression unit.

6. A driver emotion recognition method based on multi-information fusion, characterized in that, The method includes the following steps: Acquire a driver image, calculate three-dimensional key point information of the face based on the driver image, perform facial expression analysis based on the three-dimensional key point information, and perform grading processing to obtain an expression and emotion level; Obtain voice information by listening to the in-vehicle microphone, perform emotion recognition on the voice information through the BERT model to obtain the driver's emotion sequence, and calculate the driver's facial expression emotion level based on the driver's emotion sequence; Obtain the vehicle's own throttle information, brake information, and steering wheel information; Introduce a BP neural network with a self-attention mechanism, and obtain and output the emotion score according to the facial expression emotion level, facial expression emotion level, throttle information, brake information, and steering wheel information.

7. The driver emotion recognition method based on multi-information fusion according to claim 6, characterized in that The steps of obtaining the driver's image, calculating the three-dimensional key point information of the face based on the driver's image, performing facial expression analysis based on the three-dimensional key point information, and performing grading processing to obtain the facial expression emotion level specifically include: Obtain the driver's image through a monitoring camera installed in the vehicle; Use a two-stage model to obtain the face bounding box and confidence according to the driver's image, and obtain the three-dimensional key point information of the face region in the face bounding box through the two-stage model; Calculate the sequence of facial expression emotions through the MediapipeBlendshapePrediction model for the three-dimensional key point information; Calculate the driver's facial expression emotion level according to the obtained sequence of facial expression emotions.

8. The driver emotion recognition method based on multi-information fusion according to claim 6, characterized in that The steps of obtaining voice information by listening to the in-vehicle microphone, performing emotion recognition on the voice information through the BERT model to obtain the driver's emotion sequence, and calculating the driver's facial expression emotion level based on the driver's emotion sequence specifically include: Obtain the driver's voice information by listening to the microphone for making or receiving calls in the vehicle; Perform language emotion recognition analysis through a fine-tuned and trained BERT model; Parse the output of the BERT model to obtain the driver's emotion sequence; Calculate the driver's language emotion level according to the obtained driver's emotion sequence.

Images