A method, device, equipment and readable storage medium for monitoring driver's emotions

Abstract

The present application provides a method, device, equipment and readable storage medium for monitoring driver emotions, the method comprising: recognizing the driver's facial image through a preset emotion classification model to obtain multiple emotion categories and corresponding multiple confidence scores; determining whether the driver's current driving road is a bumpy and / or congested road; when the driver's current driving road is determined to be a bumpy and / or congested road, determining the number of bumps and / or congestions generated by the driver on the current driving road; and weighting the multiple confidence scores corresponding to the multiple emotion categories according to the number of bumps and / or congestions to obtain an emotion monitoring result. The method can achieve the effect of accurately monitoring the driver's emotions.

Description

Technical Field

[0001] The present application relates to the field of assisted intelligent driving, and more specifically, to a method, apparatus, device and readable storage medium for monitoring driver emotions. Background Art

[0002] In intelligent driving assistance systems, the driver's emotional state has an important impact on driving safety and driving experience. In the existing driver facial expression recognition process, the driver's emotions are mainly recognized through his facial expression, specifically by taking a facial image of the driver and determining the driver's expression through image key point recognition.

[0003] However, the driver's emotions are often not reflected through facial expressions. There are many other factors that can affect the driver's emotions. Determining the driver's emotions through a single driver's facial expression is often not accurate enough.

[0004] Therefore, how to accurately monitor the driver's emotions is a technical problem that needs to be solved. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method for monitoring the driver's emotions. The technical solution of the embodiments of the present application can achieve the effect of accurately monitoring the driver's emotions.

[0006] In a first aspect, an embodiment of the present application provides a method for monitoring a driver's emotions, comprising: recognizing a driver's facial image through a preset emotion classification model to obtain multiple emotion categories and corresponding multiple confidence scores; determining whether the driver's current driving road is a bumpy and / or congested road; when it is determined that the driver's current driving road is a bumpy and / or congested road, determining the number of bumps and / or congestion times the driver has experienced on the current driving road; and weighting multiple confidence scores corresponding to multiple emotion categories by the number of bumps and / or congestion times to obtain an emotion monitoring result.

[0007] In the above embodiment of the present application, by monitoring road bumps and traffic congestion, and combining this information with the facial expression recognition result, the influence of the external environment on the driver's emotions can be comprehensively considered, so as to more accurately detect the driver's emotional state. This method can achieve the effect of accurately monitoring the driver's emotions.

[0008] In some embodiments, after determining whether the driver's current driving road is a bumpy and / or congested road, it also includes: when it is determined that the driver's current driving road is not a bumpy and / or congested road, taking the emotion category corresponding to the highest score among multiple confidence scores as the emotion monitoring result.

[0009] In the above-mentioned embodiment of the present application, when it is determined that the current road the driver is traveling on is not a bumpy and / or congested road, the driver's emotions can be accurately determined directly by recognizing the driver's facial image.

[0010] In some embodiments, determining whether the driver's current driving road is a bumpy and / or congested road includes: performing semantic segmentation on the video image of the current driving road through a semantic segmentation model to obtain multiple object categories; calculating the proportion of the number of pixels corresponding to the multiple object categories in the total pixels of the current driving road; when the pothole pixels or vehicle pixels corresponding to the multiple object categories are greater than or equal to a preset threshold, the current driving road is a bumpy and / or congested road; when the pothole pixels and vehicle pixels corresponding to the multiple object categories are both less than the preset threshold, the current driving road is not a bumpy or congested road.

[0011] In the above-mentioned embodiments of the present application, the proportion of pixels of different categories of objects in the image can be determined by the proportion of pixels of different categories of objects relative to the entire captured image, and then the preset threshold can be used to accurately judge whether the current driving road is a bumpy or congested road.

[0012] In some embodiments, determining the number of bumps and / or congestion times generated by the driver on the current driving road includes: calculating the distance of each vertical displacement and each horizontal displacement of the driver's face on the current driving road through a key point detection algorithm; determining the number of times the vertical displacement distance is greater than or equal to a vertical threshold, and obtaining the number of bumps; determining the number of times the horizontal displacement distance is greater than or equal to a horizontal threshold, and obtaining the number of congestion times.

[0013] In the above embodiments of the present application, the bumpy and congested conditions can also be accurately judged by the moving distance, moving direction and number of movements of the driver's face.

[0014] In some embodiments, multiple confidence scores corresponding to multiple emotion categories are weighted by the number of bumps and / or the number of congestion times to obtain emotion monitoring results, including: determining weighting factors corresponding to multiple emotion categories according to the corresponding proportions of the number of bumps and / or the number of congestion times; performing weighted calculations on multiple confidence scores according to initial weight factors corresponding to the multiple emotion categories and weighting factors corresponding to the multiple emotion categories to obtain multiple final confidence scores corresponding to the multiple emotion categories.

[0015] In the above-mentioned embodiments of the present application, weighting factors of different emotions can be determined according to the number of bumps and / or congestion times, and then the expression recognition confidence scores can be weighted and calculated, so as to accurately obtain the confidence scores corresponding to the weighted different emotion categories.

[0016] In some embodiments, after obtaining the emotion monitoring result by weighting multiple confidence scores corresponding to multiple emotion categories according to the number of bumps and / or congestion times, it also includes: taking the emotion category corresponding to the highest score among the multiple final confidence scores as the driver's final emotion; and controlling the vehicle terminal to take corresponding intervention measures according to the final emotion.

[0017] In the above embodiments of the present application, the vehicle can be controlled to intervene by judging the user's emotions, thereby achieving the effect of alleviating the driver's emotions.

[0018] In some embodiments, before identifying the driver's facial image through a preset emotion classification model to obtain multiple emotion categories and corresponding multiple confidence scores, it also includes: obtaining multiple facial image information and corresponding emotion annotation information of different drivers; training the basic classification model through multiple facial image information and corresponding emotion annotation information to obtain an emotion classification model.

[0019] In the above-mentioned embodiments of the present application, the emotion classification model can be trained by using emotion images of different drivers, and the emotion classification model obtained by training can be used to accurately identify the driver's emotions.

[0020] In a second aspect, an embodiment of the present application provides a device for monitoring a driver's emotions, comprising:

[0021] A recognition module, used to recognize the driver's facial image through a preset emotion classification model to obtain multiple emotion categories and corresponding multiple confidence scores;

[0022] A first determination module, used to determine whether the driver's current driving road is a bumpy and / or congested road;

[0023] A second determination module is used to determine the number of bumps and / or congestion times caused by the driver on the current driving road when it is determined that the driver's current driving road is a bumpy and / or congested road;

[0024] The emotion monitoring module is used to perform weighted processing on multiple confidence scores corresponding to multiple emotion categories according to the number of bumps and / or congestion times to obtain emotion monitoring results.

[0025] Optionally, the device further comprises:

[0026] The third determination module is used for, after the first determination module determines whether the driver's current driving road is a bumpy and / or congested road, when it is determined that the driver's current driving road is not a bumpy and / or congested road, to use the emotion category corresponding to the highest score among multiple confidence scores as the emotion monitoring result.

[0027] Optionally, the first determining module is specifically used for:

[0028] The semantic segmentation model is used to perform semantic segmentation on the video image of the current driving road to obtain multiple object categories;

[0029] Calculate the proportion of the number of pixels corresponding to multiple object categories in the total number of pixels on the current driving road;

[0030] When the pothole pixels or vehicle pixels corresponding to multiple object categories are greater than or equal to a preset threshold, the current driving road is a bumpy and / or congested road;

[0031] When the pothole pixels and vehicle pixels corresponding to multiple object categories are all smaller than a preset threshold, the current driving road is not a bumpy or congested road.

[0032] Optionally, the second determining module is specifically used for:

[0033] The key point detection algorithm is used to calculate the distance of each vertical displacement and each horizontal displacement of the driver's face on the current driving road;

[0034] Determine the number of times that the distance of the vertical displacement is greater than or equal to the vertical threshold, and obtain the number of bumps;

[0035] Determine the number of times that the distance of the horizontal displacement is greater than or equal to the horizontal threshold, and obtain the number of congestion times.

[0036] Optionally, the emotion monitoring module is specifically used for:

[0037] Determine weighting factors corresponding to multiple emotion categories according to corresponding proportions of bump times and / or congestion times;

[0038] According to the initial weight factors corresponding to the multiple emotion categories and the weighting factors corresponding to the multiple emotion categories, the multiple confidence scores are weightedly calculated to obtain multiple final confidence scores corresponding to the multiple emotion categories.

[0039] Optionally, the device further comprises:

[0040] A control module, configured for the emotion monitoring module to perform weighted processing on multiple confidence scores corresponding to multiple emotion categories according to the number of bumps and / or congestion times, and after obtaining the emotion monitoring result, use the emotion category corresponding to the highest score among the multiple final confidence scores as the final emotion of the driver;

[0041] Based on the final emotion, the vehicle terminal is controlled to take corresponding intervention measures.

[0042] Optionally, the device further comprises:

[0043] A training module, used for the recognition module to obtain multiple facial image information and corresponding emotion annotation information of different drivers before recognizing the driver's facial image through a preset emotion classification model to obtain multiple emotion categories and corresponding multiple confidence scores;

[0044] The basic classification model is trained through multiple facial image information and corresponding emotion annotation information to obtain an emotion classification model.

[0045] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method provided in the first aspect are performed.

[0046] In a fourth aspect, an embodiment of the present application provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the method provided in the first aspect are performed.

[0047] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A flowchart of a method for monitoring driver emotions provided in an embodiment of the present application;

[0050] Figure 2 A flowchart of an implementation method for monitoring driver emotions provided in an embodiment of the present application;

[0051] Figure 3 A schematic block diagram of a device for monitoring driver emotions provided in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of the structure of a device for monitoring driver emotions provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0054] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0055] First, some terms involved in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0056] ‌Dlib‌ is an open source library written in C++ that provides a range of functions related to machine learning, numerical computing, graphical model algorithms, image processing, and other fields. The Dlib model performs well in face detection, key point detection, image processing, and other fields, and is particularly suitable for applications such as face recognition, beauty, and 2D / 3D modeling. The model provides a variety of pre-trained models, including face key point detection models, face recognition models, etc. These models are based on deep learning technologies such as convolutional neural networks (CNN) and residual neural networks (ResNet) to achieve high-precision facial feature extraction and recognition.

[0057] ‌OpenCV,‌ is a model based on the computer vision library OpenCV, which uses OpenCV's functions and algorithms for image processing and computer vision tasks. OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library that contains many common algorithms for image processing and computer vision, such as feature detection, image segmentation, object detection, face recognition, etc. OpenCV models can be applied to various scenarios, including but not limited to real-time video analysis, face recognition, object tracking, image enhancement, etc.

[0058] The ‌DeepLab model‌ is a deep learning model that is mainly used for image semantic segmentation tasks. The DeepLab model has gone through multiple versions, from v1 to v3+, and each version has made improvements and optimizations in solving the problems encountered in image semantic segmentation.

[0059] ‌The SegNet model is a semantic segmentation model based on a convolutional neural network. It uses an encoder-decoder symmetric structure to achieve end-to-end pixel-level image segmentation. ‌ The main innovation of the SegNet model lies in its upsampling method in the decoding stage. By using the maximum pooling index during downsampling in the encoding stage, the need for learning in the upsampling process is eliminated, thereby better preserving boundary feature information. This structure enables SegNet to efficiently and accurately complete pixel-by-pixel classification in semantic segmentation tasks.

[0060] This application is applied to the scenario of assisted intelligent driving. The specific scenario is to monitor road bumps and traffic congestion, and combine this information with facial expression recognition results to accurately monitor the driver's emotions, and assist the driver in driving the vehicle by controlling vehicle intervention.

[0061] In intelligent driving assistance systems, the driver's emotional state has an important impact on driving safety and driving experience. In the existing driver facial expression recognition process, the driver's emotions are mainly identified through his facial expressions, specifically by taking a facial image of the driver and determining the driver's expression through image key point recognition. However, the driver's emotions are often not reflected through facial expressions. There are many other factors that can affect the driver's emotions. Determining the driver's emotions through a single driver's expression is often not accurate enough.

[0062] To this end, the present application recognizes the driver's facial image through a preset emotion classification model to obtain multiple emotion categories and corresponding multiple confidence scores; determines whether the driver's current driving road is a bumpy and / or congested road; when it is determined that the driver's current driving road is a bumpy and / or congested road, determines the number of bumps and / or congestion times the driver has experienced on the current driving road; and weights the multiple confidence scores corresponding to the multiple emotion categories by the number of bumps and / or congestion times to obtain emotion monitoring results. By monitoring road bumps and traffic congestion, and combining this information with the facial expression recognition results, the impact of the external environment on the driver's emotions can be comprehensively considered, thereby more accurately detecting the driver's emotional state. This method can achieve the effect of accurately monitoring the driver's emotions.

[0063] In the embodiment of the present application, the execution entity may be a driver emotion monitoring device in a driver emotion monitoring system. In actual applications, the driver emotion monitoring device may be an electronic device such as a terminal device and a server, which is not limited here.

[0064] Combine the following Figure 1 The method for monitoring driver emotions in an embodiment of the present application is described in detail.

[0065] Please see Figure 1 , Figure 1 A flowchart of a method for monitoring driver emotions provided in an embodiment of the present application is shown in FIG. Figure 1 The method of monitoring driver emotions shown includes:

[0066] Step 110: Recognize the driver's facial image through a preset emotion classification model to obtain multiple emotion categories and corresponding multiple confidence scores.

[0067] Among them, the emotion categories include: happy, angry, calm and anxious, etc., and can also include sad, upset, happy and impatient. The facial image can be the current facial image of the driver or multiple consecutive facial images obtained in real time or at a fixed time. The facial image can be obtained by taking pictures with the vehicle camera or by capturing the video recorded by the camera. Before recognizing the facial image, it can be pre-processed, denoised, contrast enhanced, and converted into a grayscale image.

[0068] In some embodiments of the present application, before recognizing the driver's facial image through a preset emotion classification model to obtain multiple emotion categories and corresponding multiple confidence scores, Figure 1 The method shown also includes: obtaining multiple facial image information and corresponding emotion annotation information of different drivers; training a basic classification model through the multiple facial image information and corresponding emotion annotation information to obtain an emotion classification model.

[0069] In the above process, the present application can train the emotion classification model through the emotion images of different drivers, and the emotion classification model obtained by training can accurately identify the driver's emotions.

[0070] The image information includes at least one facial image of a driver, and the emotion annotation information includes multiple emotion categories corresponding to the facial image and scores corresponding to each emotion category. The basic classification model can adopt a key point detection model, such as Dlib and OpenCV.

[0071] Step 120: Determine whether the driver's current driving road is a bumpy and / or congested road.

[0072] Among them, bumpy roads may be some pothole-prone roads, and congested roads may be construction congestion or excessive vehicle congestion.

[0073] In some embodiments of the present application, after determining whether the driver's current driving road is a bumpy and / or congested road, Figure 1 The method shown also includes: when it is determined that the driver's current driving road is not a bumpy and / or congested road, taking the emotion category corresponding to the highest score among the multiple confidence scores as the emotion monitoring result.

[0074] In the above process, when it is determined that the driver's current driving road is not a bumpy and / or congested road, the present application can directly and accurately determine the driver's emotions by recognizing the driver's facial image.

[0075] For example, there are four emotion categories, namely happy, angry, calm and anxious, and the corresponding confidence scores are 0.1, 0.2, 0.3 and 0.4. The emotion category with the highest score is anxiety.

[0076] In some embodiments of the present application, determining whether the driver's current driving road is a bumpy and / or congested road includes: performing semantic segmentation on the video image of the current driving road through a semantic segmentation model to obtain multiple object categories; calculating the proportion of the number of pixels corresponding to the multiple object categories in the total pixels of the current driving road; when the pothole pixels or vehicle pixels corresponding to the multiple object categories are greater than or equal to a preset threshold, the current driving road is a bumpy and / or congested road; when the pothole pixels and vehicle pixels corresponding to the multiple object categories are both less than the preset threshold, the current driving road is not a bumpy or congested road.

[0077] In the above process, the present application can determine the proportion of different categories of objects in the image through the proportion of pixels of different categories of objects relative to the pixels of the entire captured image, and then accurately judge whether the current driving road is a bumpy or congested road through a preset threshold.

[0078] Among them, the semantic segmentation model can adopt DeepLab and SegNet, etc. Object categories include potholes, vehicles, trees, and flat roads, etc. The preset threshold can be set according to needs.

[0079] Optionally, the proportion of the number of pixels corresponding to potholes and congested vehicles in the total number of pixels on the current road can be obtained by the following formula:

[0080] ;

[0081] ;

[0082] in, and They respectively represent the proportion of the number of pixels corresponding to potholes and congested vehicles in the total number of pixels on the current road.

[0083] Step 130: When it is determined that the current driving road of the driver is a bumpy and / or congested road, determine the number of bumps and / or congestion times generated by the driver on the current driving road.

[0084] In some embodiments of the present application, determining the number of bumps and / or congestion times generated by the driver on the current driving road includes: calculating the distance of each vertical displacement and each horizontal displacement of the driver's face on the current driving road through a key point detection algorithm; determining the number of times the vertical displacement distance is greater than or equal to a vertical threshold, and obtaining the number of bumps; determining the number of times the horizontal displacement distance is greater than or equal to a horizontal threshold, and obtaining the number of congestion times.

[0085] During the above process, the present application can also accurately judge the bumpy and congested conditions by the moving distance, moving direction and number of movements of the driver's face.

[0086] The key point detection algorithm may use an existing key point detection algorithm or train an existing key point detection model with the driver's facial image to obtain a model containing the key point detection algorithm. The vertical threshold and the horizontal threshold may be set as required.

[0087] Optionally, the distance of each vertical displacement and each horizontal displacement of the driver's face on the current driving road calculated by the key point detection algorithm can be obtained by the following formula:

[0088] ;

[0089] ;

[0090] in, and Respectively represent the amplitude of the driver's facial movement in the vertical and horizontal directions, and Represent the vertical displacement distance and the horizontal displacement distance respectively. Indicates the time interval of movement. or When the vertical threshold and the horizontal threshold are respectively greater than or equal to each other, it is considered that there is a bump or congestion; otherwise, there is no bump or congestion.

[0091] Step 140: weighting multiple confidence scores corresponding to multiple emotion categories according to the number of bumps and / or the number of congestion to obtain an emotion monitoring result.

[0092] In some embodiments of the present application, multiple confidence scores corresponding to multiple emotion categories are weighted by the number of bumps and / or the number of congestion times to obtain emotion monitoring results, including: determining weighting factors corresponding to multiple emotion categories according to the corresponding proportions of the number of bumps and / or the number of congestion times; performing weighted calculations on multiple confidence scores according to the initial weight factors corresponding to the multiple emotion categories and the weighting factors corresponding to the multiple emotion categories to obtain multiple final confidence scores corresponding to the multiple emotion categories.

[0093] In the above process, the present application can determine the weighting factors of different emotions according to the number of bumps and / or congestion times, and then perform weighted calculation on the expression recognition confidence score, so as to accurately obtain the confidence scores corresponding to the weighted different emotion categories.

[0094] According to the corresponding proportions of the number of bumps and / or the number of congestion, the weighting factors corresponding to the multiple emotion categories can be obtained by the following formula:

[0095] w 总 =w+n 颠簸 (w 颠簸 -1) +n 拥堵 (w 拥堵 -1);

[0096] ;

[0097] Among them, w represents the initial weight factor of different categories of emotions, which can be 1. The weight factor in the case of bumpy and congested is w 颠簸 and w 拥堵 , the total weight w 总 , n 颠簸 and n 拥堵 Represent the number of bumps and congestion respectively. Represents the weighting factors corresponding to different emotions. The sum of the weights corresponding to different emotions is 1. Indicates the emotion count thresholds corresponding to different emotions, which can be set according to needs. In addition, It can also be set according to demand or based on historical experience.

[0098] Optionally, weighted calculation is performed on multiple confidence scores to obtain multiple final confidence scores corresponding to multiple emotion categories, which can be obtained by the following formula:

[0099] ;

[0100] Among them, P' 情绪 Represents multiple confidence scores corresponding to multiple emotion categories, Represents the weighting factors corresponding to different emotions, P 情绪Represents multiple final confidence scores corresponding to multiple emotion categories. 情绪 The scores can be normalized.

[0101] For example, there are four emotions: happy, angry, calm, and anxious, and multiple confidence scores are: P' 高兴 , P' 愤怒 , P' 平静 and P' 焦虑 , the weighting factors corresponding to different emotions are: , multiple final confidence scores are: P 高兴 , P 愤怒 , P 平静 and P 焦虑 The total score P 总 =P 高兴+ P 愤怒 +P 平静 +P 焦虑 , each score after normalization: , , and And the sum of all scores is 1.

[0102] In some embodiments of the present application, after weighting multiple confidence scores corresponding to multiple emotion categories by the number of bumps and / or congestion times to obtain the emotion monitoring result, Figure 1 The method shown also includes: taking the emotion category corresponding to the highest score among multiple final confidence scores as the driver's final emotion; and controlling the vehicle terminal to take corresponding intervention measures based on the final emotion.

[0103] In the above process, the present application can control the vehicle to intervene by judging the user's emotions, thereby achieving the effect of alleviating the driver's emotions.

[0104] For example, providing corresponding feedback and intervention measures based on emotional states (such as playing soothing music and adjusting the in-car environment to relieve anxiety and anger) to improve the driver's mood and enhance driving experience and safety.

[0105] In the above Figure 1In the process shown, the present application recognizes the driver's facial image through a preset emotion classification model to obtain multiple emotion categories and corresponding multiple confidence scores; determines whether the driver's current driving road is a bumpy and / or congested road; when it is determined that the driver's current driving road is a bumpy and / or congested road, determines the number of bumps and / or congestion times the driver has experienced on the current driving road; and weights the multiple confidence scores corresponding to the multiple emotion categories by the number of bumps and / or congestion times to obtain emotion monitoring results. By monitoring road bumps and traffic congestion, and combining this information with the facial expression recognition results, the impact of the external environment on the driver's emotions can be comprehensively considered, thereby more accurately detecting the driver's emotional state. This method can achieve the effect of accurately monitoring the driver's emotions.

[0106] Combine the following Figure 2 The implementation method of monitoring driver emotions in an embodiment of the present application is described in detail.

[0107] Please see Figure 2 , Figure 2 A flowchart of a method for monitoring driver emotions provided in an embodiment of the present application, such as Figure 2 The implementation method shown for monitoring driver emotions includes:

[0108] Step 201: Import vehicle interior and exterior data.

[0109] Specifically: external road images and internal driver face images are acquired through cameras.

[0110] Step 202: Inputting cabin expression data.

[0111] Specific: Input driver's face image.

[0112] Step 203: Initial facial expression recognition.

[0113] Specifically: the driver's facial image is recognized by a preset emotion classification model to obtain multiple emotion categories and corresponding multiple confidence scores.

[0114] Step 204: Input extravehicular environment information.

[0115] Specific: Input external road image.

[0116] Step 205: Image semantic segmentation.

[0117] Specifically: semantic segmentation is performed on the video image of the current driving road through a semantic segmentation model to obtain multiple object categories.

[0118] Step 206: Identify bumps and congestion.

[0119] Specifically: calculate the proportion of the number of pixels corresponding to multiple object categories in the total pixels of the current driving road; when the pothole pixels or vehicle pixels corresponding to multiple object categories are greater than or equal to a preset threshold, the current driving road is a bumpy and / or congested road; when the pothole pixels and vehicle pixels corresponding to multiple object categories are both less than the preset threshold, the current driving road is not a bumpy or congested road.

[0120] Step 207: Whether there is road bumps or congestion.

[0121] Specifically: determine whether the current road is a bumpy and / or congested road, if yes, proceed to step 208, otherwise directly proceed to step 212.

[0122] Step 208: Driver key point detection.

[0123] Specifically: The movement of the driver's face is detected through a key point detection algorithm.

[0124] Step 209: Driver bumping and congestion identification.

[0125] Specifically: calculate the distance of each vertical displacement and each horizontal displacement of the driver's face on the current driving road.

[0126] Step 210: Counting the number of times and calculating the weighted value.

[0127] Specifically: determine the number of times that the vertical displacement distance is greater than or equal to the vertical threshold, and obtain the number of bumps; determine the number of times that the horizontal displacement distance is greater than or equal to the horizontal threshold, and obtain the number of congestion.

[0128] Step 211: Emotion confidence weighting.

[0129] Specifically: weighted processing is performed on multiple confidence scores corresponding to multiple emotion categories according to the number of bumps and / or the number of congestion.

[0130] Step 212: Output the expression recognition result.

[0131] Specifically: according to the corresponding proportions of the number of bumps and / or the number of congestion times, weighting factors corresponding to multiple emotion categories are determined; according to the initial weight factors corresponding to the multiple emotion categories and the weighting factors corresponding to the multiple emotion categories, multiple confidence scores are weightedly calculated to obtain multiple final confidence scores corresponding to the multiple emotion categories.

[0132] Step 213: Vehicle-computer function linkage.

[0133] Specific: Play relevant videos and music according to emotions or control the speed and brakes of the vehicle, etc.

[0134] also, Figure 2The specific methods and steps shown can be found in Figure 1 The method shown will not be described in detail here.

[0135] Previous article Figure 1-Figure 2 A method for monitoring driver emotions is described below. Figure 3-Figure 4 Describe a device for monitoring driver emotions.

[0136] Please refer to Figure 3 , is a schematic block diagram of a device 300 for monitoring driver emotions provided in an embodiment of the present application. The device 300 may be a module, program segment or code on an electronic device. The device 300 is similar to the above Figure 1 The method embodiment corresponds to and can be executed Figure 1 The various steps involved in the method embodiment and the specific functions of the device 300 can be found in the description below. To avoid repetition, the detailed description is appropriately omitted here.

[0137] Optionally, the device 300 includes:

[0138] The recognition module 310 is used to recognize the driver's facial image through a preset emotion classification model to obtain multiple emotion categories and corresponding multiple confidence scores;

[0139] A first determination module 320, configured to determine whether the current driving road of the driver is a bumpy and / or congested road;

[0140] The second determination module 330 is used to determine the number of bumps and / or congestion times generated by the driver on the current driving road when it is determined that the current driving road of the driver is a bumpy and / or congested road;

[0141] The emotion monitoring module 340 is used to perform weighted processing on multiple confidence scores corresponding to multiple emotion categories according to the number of bumps and / or the number of congestion times to obtain emotion monitoring results.

[0142] Optionally, the device further comprises:

[0143] The third determination module is used for, after the first determination module determines whether the driver's current driving road is a bumpy and / or congested road, when it is determined that the driver's current driving road is not a bumpy and / or congested road, to use the emotion category corresponding to the highest score among multiple confidence scores as the emotion monitoring result.

[0144] Optionally, the first determining module 320 is specifically configured to:

[0145] The video image of the current driving road is semantically segmented through a semantic segmentation model to obtain multiple object categories; the proportion of the number of pixels corresponding to the multiple object categories in the total pixels of the current driving road is calculated; when the pothole pixels or vehicle pixels corresponding to the multiple object categories are greater than or equal to a preset threshold, the current driving road is a bumpy and / or congested road; when the pothole pixels and vehicle pixels corresponding to the multiple object categories are both less than the preset threshold, the current driving road is not a bumpy or congested road.

[0146] Optionally, the second determining module 330 is specifically configured to:

[0147] The key point detection algorithm is used to calculate the distance of each vertical displacement and each horizontal displacement of the driver's face on the current driving road; the number of times the vertical displacement distance is greater than or equal to the vertical threshold is determined to obtain the number of bumps; the number of times the horizontal displacement distance is greater than or equal to the horizontal threshold is determined to obtain the number of congestion times.

[0148] Optionally, the emotion monitoring module 340 is specifically used for:

[0149] According to the corresponding proportions of the number of bumps and / or the number of congestion times, weighting factors corresponding to multiple emotion categories are determined; according to the initial weight factors corresponding to the multiple emotion categories and the weighting factors corresponding to the multiple emotion categories, multiple confidence scores are weightedly calculated to obtain multiple final confidence scores corresponding to the multiple emotion categories.

[0150] Optionally, the device further comprises:

[0151] A control module is used for the emotion monitoring module to perform weighted processing on multiple confidence scores corresponding to multiple emotion categories based on the number of bumps and / or congestion times, and after obtaining the emotion monitoring results, the emotion category corresponding to the highest score among the multiple final confidence scores is used as the driver's final emotion; and according to the final emotion, the vehicle terminal is controlled to take corresponding intervention measures.

[0152] Optionally, the device further comprises:

[0153] The training module is used for obtaining multiple facial image information and corresponding emotion annotation information of different drivers before the recognition module recognizes the driver's facial image through a preset emotion classification model to obtain multiple emotion categories and corresponding multiple confidence scores; the basic classification model is trained through the multiple facial image information and the corresponding emotion annotation information to obtain the emotion classification model.

[0154] Please refer to Figure 44 is a schematic block diagram of a device for monitoring driver emotions provided in an embodiment of the present application. The device may include a memory 410 and a processor 420. Optionally, the device may also include: a communication interface 430 and a communication bus 440. The device is similar to the above Figure 1 The method embodiment corresponds to and can be executed Figure 1 The various steps involved in the method embodiment and the specific functions of the device can be found in the description below.

[0155] Specifically, the memory 410 is used to store computer-readable instructions.

[0156] Processor 420 is used to process the readable instructions stored in the memory and can execute Figure 1 The steps in the method.

[0157] The communication interface 430 is used for signaling or data communication with other node devices, for example, for communication with a server or a terminal, or for communication with other device nodes, but the embodiments of the present application are not limited thereto.

[0158] The communication bus 440 is used to realize direct connection and communication among the above components.

[0159] The communication interface 430 of the device in the embodiment of the present application is used to communicate with other node devices for signaling or data. The memory 410 can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The memory 410 can also be at least one storage device located away from the aforementioned processor. The memory 410 stores computer-readable instructions. When the computer-readable instructions are executed by the processor 420, the electronic device executes the aforementioned Figure 1 The method process shown. The processor 420 can be used on the device 300 and used to perform the functions in the present application. Exemplarily, the above-mentioned processor 420 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and the embodiments of the present application are not limited thereto.

[0160] The embodiment of the present application also provides a readable storage medium, when the computer program is executed by a processor, Figure 1 The method process in the method embodiment shown is executed by the electronic device.

[0161] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0162] In summary, the embodiments of the present application provide a method, device, electronic device and readable storage medium for monitoring driver emotions, the method comprising: recognizing the driver's facial image through a preset emotion classification model to obtain multiple emotion categories and corresponding multiple confidence scores; determining whether the driver's current driving road is a bumpy and / or congested road; when it is determined that the driver's current driving road is a bumpy and / or congested road, determining the number of bumps and / or congestions generated by the driver on the current driving road; and weighting the multiple confidence scores corresponding to the multiple emotion categories according to the number of bumps and / or congestions to obtain an emotion monitoring result. This method can achieve the effect of accurately monitoring the driver's emotions.

[0163] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0164] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0165] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0166] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0167] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0168] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for monitoring driver emotions, characterized in that: include: Recognize the driver's facial image through a preset emotion classification model to obtain multiple emotion categories and corresponding multiple confidence scores; determining whether the current driving road of the driver is a bumpy and / or congested road; When it is determined that the current driving road of the driver is a bumpy and / or congested road, determining the number of bumps and / or congestion times generated by the driver on the current driving road; Performing weighted processing on a plurality of confidence scores corresponding to the plurality of emotion categories according to the number of bumps and / or the number of congestion, to obtain an emotion monitoring result; The determining of the number of bumps and / or congestion times caused by the driver on the current driving road comprises: calculating the distance of each vertical displacement and each horizontal displacement of the driver's face on the current driving road by a key point detection algorithm; determining the number of times the vertical displacement distance is greater than or equal to a vertical threshold, and obtaining the number of bumps; determining the number of times the horizontal displacement distance is greater than or equal to a horizontal threshold, and obtaining the number of congestion times; The emotion monitoring result is obtained by weighting the multiple confidence scores corresponding to the multiple emotion categories through the number of bumps and / or the number of congestion times, including: determining the weighting factors corresponding to the multiple emotion categories according to the corresponding proportions of the number of bumps and / or the number of congestion times; and weighting the multiple confidence scores according to the initial weight factors corresponding to the multiple emotion categories and the weighting factors corresponding to the multiple emotion categories to obtain multiple final confidence scores corresponding to the multiple emotion categories.

2. The method according to claim 1, characterized in that After determining whether the current driving road of the driver is a bumpy and / or congested road, the method further includes: When it is determined that the current driving road of the driver is not a bumpy and / or congested road, the emotion category corresponding to the highest score among the multiple confidence scores is used as the emotion monitoring result.

3. The method according to claim 1 or 2, characterized in that: The determining whether the current driving road of the driver is a bumpy and / or congested road comprises: Performing semantic segmentation on the video image of the current driving road by using a semantic segmentation model to obtain multiple object categories; Calculate the proportion of the number of pixels corresponding to the multiple object categories in the total number of pixels on the current driving road; When the pothole pixels or vehicle pixels corresponding to the multiple object categories are greater than or equal to a preset threshold, the current driving road is a bumpy and / or congested road; When the pothole pixels and vehicle pixels corresponding to the multiple object categories are all smaller than a preset threshold, the current driving road is not a bumpy or congested road.

4. The method according to claim 3, characterized in that: After performing weighted processing on the multiple confidence scores corresponding to the multiple emotion categories based on the number of bumps and / or the number of congestion to obtain the emotion monitoring result, the method further includes: Using the emotion category corresponding to the highest score among the multiple final confidence scores as the final emotion of the driver; According to the final emotion, the vehicle terminal is controlled to take corresponding intervention measures.

5. The method according to claim 1 or 2, characterized in that: Before recognizing the driver's facial image by using a preset emotion classification model to obtain a plurality of emotion categories and a plurality of corresponding confidence scores, the method further includes: Obtain multiple facial image information and corresponding emotion annotation information of different drivers; The basic classification model is trained using the multiple facial image information and the corresponding emotion annotation information to obtain the emotion classification model.

6. A device for monitoring driver's emotions, characterized in that: include: A recognition module, used to recognize the driver's facial image through a preset emotion classification model to obtain multiple emotion categories and corresponding multiple confidence scores; A first determination module, used to determine whether the current driving road of the driver is a bumpy and / or congested road; A second determination module, for determining the number of bumps and / or congestion times generated by the driver on the current driving road when it is determined that the current driving road of the driver is a bumpy and / or congested road; An emotion monitoring module, configured to perform weighted processing on a plurality of confidence scores corresponding to the plurality of emotion categories according to the number of bumps and / or the number of congestion, to obtain an emotion monitoring result; The second determination module is specifically used to: calculate the distance of each vertical displacement and each horizontal displacement of the driver's face on the current driving road through a key point detection algorithm; determine the number of times the vertical displacement distance is greater than or equal to the vertical threshold, and obtain the number of bumps; determine the number of times the horizontal displacement distance is greater than or equal to the horizontal threshold, and obtain the number of congestion; The emotion monitoring module is specifically used to: determine the weighting factors corresponding to the multiple emotion categories according to the corresponding proportions of the number of bumps and / or the number of congestion times; perform weighted calculation on the multiple confidence scores according to the initial weight factors corresponding to the multiple emotion categories and the weighting factors corresponding to the multiple emotion categories to obtain multiple final confidence scores corresponding to the multiple emotion categories.

7. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method according to any one of claims 1 to 5 are executed.

8. A computer-readable storage medium, characterized in that: include: A computer program, when the computer program is run on a computer, causes the computer to execute the method according to any one of claims 1 to 5.

Images