Driver emotion recognition method, device and system and vehicle
By directly obtaining target offsets from facial images to identify the driver's emotional state, the problem of the cumbersome process of the expression model is solved, the efficiency of emotional recognition is improved, and emotional comfort is provided through digital people and music, which improves driving safety and user experience.
Patent Information
- Application Number
- CN202411014214.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-22
AI Technical Summary
The process of determining the expression model in the prior art is cumbersome, resulting in low driver emotion recognition efficiency.
By obtaining the driver's current facial image and the target offset in the preset facial image, the driver's emotional state is directly determined without training the expression model, using eyebrow marking lines, lower eye socket edge marking lines, lip marking lines and eyebrows The offset of the spacing to identify emotions of natural states, anger, shock, excitement, and sadness.
Improves the efficiency and accuracy of emotional recognition, and provides emotional comfort to drivers by adjusting the shape of digital people and playing matching music, improving driving safety and user experience.
Smart Images

Figure CN119007167A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of vehicle safety technology, and in particular to a driver emotion recognition method, device, system and vehicle. Background Art
[0002] In the smart cockpit segment, the Driver Monitoring System (DMS) collects the driver's facial information to realize the driver's identity recognition, driver fatigue monitoring, driver attention monitoring, and dangerous driving behavior monitoring functions, and promptly issues alarms to improve driving safety.
[0003] At present, in order to further improve the intelligent experience of automobiles, the driver's emotions can be identified by using a large expression model (artificial intelligence model) based on the collected facial information of the driver, and music can be played to soothe the driver's emotions based on the identified emotions. However, the above method requires a large amount of data to train the large expression model, and the process of determining the large expression model is cumbersome, which affects the efficiency of emotion recognition. Summary of the invention
[0004] In view of this, the present invention provides a method, device, system and vehicle for identifying a driver's emotion, so as to solve the problem that the process of determining a large expression model is complicated and the efficiency of emotion recognition is low.
[0005] In a first aspect, the present invention provides a method for identifying a driver's emotions, the method comprising: acquiring a current facial image of the driver; determining a target offset based on the current facial image and a preset facial image of the driver, wherein the target offset is an offset of a target feature in the current facial image relative to a target feature in the preset facial image, and the target feature comprises at least one of an eyebrow marking line, a lower orbital edge marking line, a lip marking line, and an eyebrow distance; determining the driver's emotional state based on the target offset, wherein the emotional state comprises a natural state, anger, shock, excitement, and sadness.
[0006] The method for recognizing driver emotions provided in this embodiment, after obtaining the current facial image of the driver, determines the target offset based on the current facial image and the preset facial image of the driver, and then determines the driver's emotional state based on the target offset. It is unnecessary to train a large facial expression model, and the driver's emotional state can be determined directly through the target offset, thereby improving the efficiency of emotion recognition.
[0007] In an optional embodiment, the preset facial image is a facial image of the driver in a natural state, and a target offset is determined based on the current facial image and the preset facial image of the driver, including: comparing the target features of the current facial image with the target features of the preset facial image to determine the target offset, wherein the target offset includes the offset angle of the eyebrow marking line, the offset angle of the lower edge of the eye socket marking line, the offset angle of the lip marking line and the deviation value of the eyebrow distance.
[0008] In an optional embodiment, the emotional state of the driver is determined according to the target offset, including: when -5°≤θ1≤5°, -5°≤θ2≤5°, 0°≤θ3≤3° and 0mm≤D≤3mm, the emotional state of the driver is determined to be a natural state, wherein θ1 represents the offset angle of the eyebrow marking line, θ2 represents the offset angle of the lip marking line, θ3 represents the offset angle of the lower edge of the eye socket marking line, and D represents the deviation value of the eyebrow distance; when 5°<1≤10° and D≥5mm , the driver’s emotional state is determined to be angry; when 5°<1≤10° or -10°≤θ1<-5°, 0mm≤D≤3mm, 0°≤θ3≤10° and the lips are in an open state, the driver’s emotional state is determined to be shocked; when 5°<1≤10°, 5°<2≤10°, -5°≤θ3<0° and 0mm≤D≤3mm, the driver’s emotional state is determined to be excited; when -10°≤θ2<-5°, the driver’s emotional state is determined to be sad.
[0009] In this embodiment, the emotional state of the driver is determined by comparing the target offset with the preset offset interval, which can identify emotions more quickly and accurately.
[0010] In an optional embodiment, after determining the emotional state of the driver based on the target offset, the method further includes: adjusting the form of the digital human on the vehicle according to the emotional state of the driver, and triggering the digital human to have a conversation with the driver, wherein the form includes gestures and facial expressions.
[0011] In this embodiment, after the driver's emotional state is determined, the driver's emotional state can be transmitted to the on-board large model. The on-board large model can be used to adjust the form of the digital human and trigger the digital human to have a conversation with the driver, thereby providing emotional value to the driver and giving the driver the comfort and companionship he desires, thereby improving driving safety and enhancing the user's driving experience.
[0012] In an optional implementation, according to the emotional state of the driver, the form of the digital human on the vehicle is adjusted, and the digital human is triggered to have a conversation with the driver, including: when the emotional state is anger, the digital human's gesture action is adjusted to a predefined first gesture action, and the digital human's facial expression is adjusted to a predefined first facial expression, so that the digital human presents a soothing and caring expression; when the emotional state is shock, the digital human's gesture action is adjusted to a predefined second gesture action, and the digital human's facial expression is adjusted to a predefined second facial expression, so that the digital human presents a curious and caring expression; when the emotional state is excitement, the digital human's gesture action is adjusted to a predefined third gesture action, and the digital human's facial expression is adjusted to a predefined third facial expression, so that the digital human presents a happy expression; when the emotional state is sadness, the digital human's gesture action is adjusted to a predefined fourth gesture action, and the digital human's facial expression is adjusted to a predefined fourth facial expression, so that the digital human presents a sad and caring expression.
[0013] In an optional implementation, the method further includes: selecting music that matches the driver's emotional state, and controlling the voice module to play the music.
[0014] In this embodiment, after determining the driver's emotional state, the voice module is controlled to play music that matches the driver's emotional state, which can provide comfort and companionship to the driver, improve driving safety, and provide the driver with a better driving experience.
[0015] In an optional embodiment, the vehicle is bound to a target driver, and before determining the target offset based on a current facial image and a preset facial image of the driver, the method also includes: determining whether the driver is the target driver; determining the target offset based on the current facial image and the preset facial image of the driver, including: when the driver is the target driver, determining the target offset based on the current facial image and the preset facial image of the driver.
[0016] In a second aspect, the present invention provides a device for identifying a driver's emotions, the device comprising: an acquisition module for acquiring a current facial image of the driver; a first determination module for determining a target offset based on the current facial image and a preset facial image of the driver, wherein the target offset is an offset of a target feature in the current facial image relative to a target feature in the preset facial image, and the target feature comprises at least one of an eyebrow marking line, a lower edge of an eye socket marking line, a lip marking line, and an eyebrow distance; a second determination module for determining the driver's emotional state based on the target offset, wherein the emotional state comprises a natural state, anger, shock, excitement, and sadness.
[0017] In an optional embodiment, the first determination module includes: a first determination unit, used to compare the target features of the current facial image with the target features of the preset facial image, and determine the target offset, wherein the target offset includes the offset angle of the eyebrow marking line, the offset angle of the lower edge of the eye socket marking line, the offset angle of the lip marking line and the deviation value of the eyebrow distance.
[0018] In an optional implementation, the second determination module 903 includes: a second determination unit, used to determine the driver's emotional state as a natural state when -5°≤θ1≤5°, -5°≤θ2≤5°, 0°≤θ3≤3° and 0mm≤D≤3mm, wherein θ1 represents the offset angle of the eyebrow marking line, θ2 represents the offset angle of the lip marking line, θ3 represents the offset angle of the lower edge of the eye socket marking line, and D represents the deviation value of the eyebrow distance; a third determination unit, used to determine the driver's emotional state as a natural state when 5°<1≤10° and D≥5mm. the fourth determination unit is used to determine the driver's emotional state as anger when 5°<1≤10° or -10°≤θ1<-5°, 0mm≤D≤3mm, 0°≤θ3≤10° and the lips are in an open state; the fifth determination unit is used to determine the driver's emotional state as excitement when 5°<1≤10°, 5°<2≤10°, -5°≤θ3<0° and 0mm≤D≤3mm; the sixth determination unit is used to determine the driver's emotional state as sadness when -10°≤θ2<-5°.
[0019] In an optional embodiment, the device further includes: an adjustment module, which is used to adjust the form of the digital human on the vehicle according to the driver's emotional state, and trigger the digital human to have a conversation with the driver, wherein the form includes gestures and facial expressions.
[0020] In an optional implementation, the adjustment module includes: a first adjustment unit, which is used to adjust the digital human's gestures to a predefined first gesture, and the digital human's facial expression to a predefined first facial expression when the emotional state is anger, so that the digital human presents a soothing and caring expression; a second adjustment unit, which is used to adjust the digital human's gestures to a predefined second gesture, and the digital human's facial expression to a predefined second facial expression when the emotional state is shock, so that the digital human presents a curious and caring expression; a third adjustment unit, which is used to adjust the digital human's gestures to a predefined third gesture, and the digital human's facial expression to a predefined third facial expression when the emotional state is excitement, so that the digital human presents a happy expression; a fourth adjustment unit, which is used to adjust the digital human's gestures to a predefined fourth gesture, and the digital human's facial expression to a predefined fourth facial expression when the emotional state is sadness, so that the digital human presents a sad and caring expression.
[0021] In an optional embodiment, the device further includes: a control module for selecting music that matches the driver's emotional state and controlling the voice module to play the music to adjust the driver's emotions.
[0022] In an optional embodiment, the vehicle is bound to a target driver, and the device also includes: a third determination module, used to determine whether the driver is the target driver; the first determination module includes: a seventh determination unit, used to determine the target offset based on the current facial image and the preset facial image of the driver when the driver is the target driver.
[0023] In a third aspect, the present invention provides a driver emotion recognition system, the system comprising a driver monitoring system, a data judgment module, a vehicle-mounted large model and a digital human; the driver monitoring system is used to collect the current facial image of the driver; the data judgment module is used to obtain the current facial image of the driver from the driver monitoring system, to determine the target offset according to the current facial image and the preset facial image of the driver, to determine the emotional state of the driver according to the target offset, and to adjust the form of the digital human through the vehicle-mounted large model according to the emotional state of the driver, and to trigger the digital human to have a conversation with the driver, wherein the target offset is the offset of the target feature in the current facial image relative to the target feature in the preset facial image, the target feature includes at least one of an eyebrow annotation line, a lower edge of the eye socket annotation line, a lip annotation line and an eyebrow distance, the emotional state includes natural state, anger, shock, excitement and sadness, and the form includes gestures and facial expressions.
[0024] In a fourth aspect, the present invention provides a vehicle, comprising: a driver emotion recognition device according to the second aspect or any corresponding embodiment thereof, or a driver emotion recognition system according to the third aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0026] Figure 1 is a flow chart of a method for identifying driver emotions according to an embodiment of the present invention;
[0027] Figure 2 is a schematic diagram of an eyebrow marking line according to an embodiment of the present invention;
[0028] Figure 3 is a schematic diagram of a marking line of the lower edge of the orbit according to an embodiment of the present invention;
[0029] Figure 4 is a schematic diagram of a lip marking line according to an embodiment of the present invention;
[0030] Figure 5 is a flow chart of another method for identifying driver emotions according to an embodiment of the present invention;
[0031] Figure 6 is a schematic diagram of a preset offset interval according to an embodiment of the present invention;
[0032] Figure 7 is a flow chart of another method for identifying driver emotions according to an embodiment of the present invention;
[0033] Figure 8 is a schematic diagram of an interactive dialogue according to an embodiment of the present invention;
[0034] Fig. 9 is a structural block diagram of a device for identifying driver emotions according to an embodiment of the present invention;
[0035] Fig.10 4 is a structural block diagram of a driver emotion recognition system according to an embodiment of the present invention. DETAILED DESCRIPTION
[0036] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0037] The method for identifying driver emotions provided by the present invention can be used in the intelligent cockpit module of a vehicle to identify the driver's emotional state, and can also be used in other fields that require identification of user emotions, which is not specifically limited by the present invention. After identifying the driver's emotional state, the intelligent cockpit module can give feedback based on the driver's emotional state, achieve "empathy" with the companion, improve driving safety, and provide users with a richer experience.
[0038] Specifically, the smart cockpit is an important part of the development of vehicle intelligence, integrating advanced information technology, Internet technology, and artificial intelligence technology, aiming to provide a safer, more comfortable, and personalized driving and riding experience. The DMS in the smart cockpit segment can collect the driver's facial expressions, eye movements, head posture and other physiological and behavioral characteristics in real time through devices such as cameras or sensors installed in the cab, and then use image processing, biometrics, machine learning and other technologies for analysis and processing to achieve driver identity recognition, driver fatigue monitoring, driver attention monitoring, and dangerous driving behavior monitoring functions. The combination of the smart cockpit and DMS not only improves the safety performance of the vehicle, but also enhances the user experience through highly personalized services and intelligent assistance functions.
[0039] In order to solve the problem of low efficiency in emotion recognition, the present invention provides a method for recognizing driver's emotions. The method recognizes the driver's emotional state through the target offset between the driver's current facial image and a preset facial image. There is no need to train a large expression model, which simplifies the recognition process and can improve recognition efficiency.
[0040] According to an embodiment of the present invention, an embodiment of a method for identifying a driver's emotion is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a device for identifying a driver's emotion in a vehicle's intelligent cockpit, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0041] In this embodiment, a method for identifying a driver's emotion is provided, which can be used in the above-mentioned device for identifying a driver's emotion in the intelligent cockpit of the vehicle. The device for identifying a driver's emotion can be a processor or a controller, etc. Figure 1is a flow chart of a method for identifying driver emotions according to an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:
[0042] Step S101, obtaining the current facial image of the driver.
[0043] The current facial image is a facial image of the driver acquired at the current moment. Facial features in the facial image include but are not limited to lips, eyes, and mouth.
[0044] For example, during driving of the vehicle, the current facial image of the driver may be acquired through a vehicle-mounted camera disposed in the cab, or may be acquired from a driver monitoring system (DMS).
[0045] Step S102, determining a target offset according to the current facial image and a preset facial image of the driver.
[0046] The target offset is the offset of the target feature in the current facial image relative to the target feature in the preset facial image, and the target feature includes at least one of the eyebrow marking line, the lower edge of the eye socket marking line, the lip marking line and the eyebrow distance.
[0047] Specifically, the eyebrow marking line can be a line reflecting the direction of the driver's eyebrows, the lower edge marking line of the eye socket can be a line reflecting the direction of the driver's lower eyelid, the lip marking line can be a line segment formed by the two corners of the mouth or the junction line between the upper lip and the lower lip, and the eyebrow spacing is the distance between the eyebrows of the driver.
[0048] For example, the eyebrow marking line can be as follows Figure 2 As shown in the first line segment 21 and the second line segment 22 in the figure, the first line segment 21 can be specifically the eyebrow annotation line of the preset facial image, and the second line segment 22 can be specifically the eyebrow annotation line collected from the current facial image. The eyebrow distance can be Figure 2 The distance from point A to point B.
[0049] The lower orbital margin can be marked as follows Figure 3 As shown in the fourth line segment 31 or the fifth line segment 32, the fourth line segment 31 may specifically be a marking line of the lower edge of the eye socket of a preset facial image, and the fifth line segment 32 may specifically be a marking line of the lower edge of the eye socket collected from the current facial image.
[0050] The lip marking line can be Figure 4 As shown in the sixth line segment 41 and the seventh line segment 42 in the figure, the sixth line segment 41 may specifically be a lip annotation line of a preset facial image, and the seventh line segment 42 may specifically be a lip annotation line collected from the current facial image.
[0051] For example, the target offset can be obtained by Figure 2 The included angle θ1 between the first line segment 21 and the second line segment 22 in Figure 3 The included angle θ3 between the fourth line segment 31 or the fifth line segment 32 in Figure 4 The angle θ2 between the sixth line segment 41 and the seventh line segment 42 in the image, and the difference between the eyebrow distance of the preset facial image and the eyebrow distance of the current facial image are represented.
[0052] Specifically, a target feature is preset on a facial image and is pre-stored in a recognition device for driver emotions in a vehicle intelligent cockpit. Exemplarily, the target feature on the preset facial image can be manually annotated by a user, or automatically annotated and stored by an annotation model. The annotation model can be an artificial intelligence model (such as DeepFace, etc.) for identifying facial features. Among them, the target feature in the current facial image can be determined based on the annotation model.
[0053] Exemplarily, the preset facial image may be a facial image of the driver in a natural state, and the preset facial image may also include facial images of the driver in various emotional states (natural state, anger, shock, excitement and sadness).
[0054] Step S103, determining the driver's emotional state according to the target offset.
[0055] Among them, emotional states include natural state, anger, shock, excitement and sadness.
[0056] For example, when the preset facial image is only the facial image of the driver in a natural state, the emotional state of the driver at the current moment can be judged based on the threshold information set for the driver's lips, eyes and eyebrows. This method is described in detail below and will not be described in detail here.
[0057] Exemplarily, when the preset facial images include facial images in various emotional states (natural state, anger, shock, excitement and sadness), the current facial image is compared with the facial image in each preset emotional state to obtain 5 target offsets, and the emotional state of the facial image corresponding to the minimum value of the 5 target offsets is the emotional state of the driver at the current moment.
[0058] The method for recognizing driver emotions provided in this embodiment, after obtaining the current facial image of the driver, determines the target offset based on the current facial image and the preset facial image of the driver, and then determines the driver's emotional state based on the target offset. It is unnecessary to train a large facial expression model, and the driver's emotional state can be determined directly through the target offset, thereby improving the efficiency of emotion recognition.
[0059] In this embodiment, a method for identifying a driver's emotion is provided, which can be used in the above-mentioned device for identifying a driver's emotion in the vehicle intelligent cockpit. Figure 5 FIG. 1 is a flow chart of another method for identifying driver emotions according to an embodiment of the present invention. Figure 5 As shown, the method comprises the following steps:
[0060] Step S501, obtaining the current facial image of the driver.
[0061] For details, please see Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.
[0062] Step S502, comparing the target feature of the current facial image with the target feature of the preset facial image to determine the target offset.
[0063] Specifically, in this embodiment, the preset facial image is the facial image of the driver in a natural state, and the above step S502 is Figure 1 A specific implementation of the illustrated embodiment.
[0064] The target offset includes the offset angle of the eyebrow marking line, the offset angle of the lower edge of the eye socket marking line, the offset angle of the lip marking line and the deviation value of the eyebrow distance.
[0065] Specifically, the offset angle of the eyebrow annotation line refers to the angle between the eyebrow annotation line of the current facial image and the eyebrow annotation line of the preset facial image, for example, Figure 2 The angle θ1 shown, where θ1 is a positive value if the eyebrow annotation line of the current facial image is offset upward relative to the eyebrow annotation line of the preset facial image; and a negative value if the eyebrow annotation line of the current facial image is offset downward relative to the eyebrow annotation line of the preset facial image.
[0066] The offset angle of the lower orbital edge annotation line refers to the angle between the lower orbital edge annotation line of the current facial image and the eyebrow annotation line of the preset facial image. For example, Figure 3 The angle θ3 shown, where θ3 is a positive value if the lower orbital edge marking line of the current facial image is offset upward relative to the lower orbital edge marking line of the preset facial image; and θ3 is a negative value if the lower orbital edge marking line of the current facial image is offset downward relative to the lower orbital edge marking line of the preset facial image.
[0067] The offset angle of the lip annotation line refers to the angle between the lip annotation line of the current facial image and the lip annotation line of the preset facial image, for example, Figure 4The angle θ2 shown is, where θ2 is a positive value if the lip annotation line of the current facial image is offset upward relative to the lip annotation line of the preset facial image; and θ2 is a negative value if the lip annotation line of the current facial image is offset downward relative to the lip annotation line of the preset facial image.
[0068] Specifically, the deviation value of the eyebrow spacing refers to the absolute value of the difference between the eyebrow spacing of the current facial image and the eyebrow spacing of the preset facial image. For example, if the eyebrow spacing of the preset facial image is 20 mm and the eyebrow spacing of the current facial image is 15 mm, then the deviation value of the eyebrow spacing is 5 mm.
[0069] Exemplarily, after obtaining the target features of the driver's current facial image and the target features of a preset facial image, the target features of the current facial image and the target features of the preset facial image can be extracted, and the target features of the current facial image and the target features of the preset facial image can be compared under the same benchmark to determine the target offset.
[0070] Step S503: determining the driver's emotional state according to the target offset.
[0071] For example, after determining the target offset, the target offset is compared with a threshold condition between preset offsets to determine the driver's emotional state, wherein the preset offset is determined by the staff based on experience or big data and is pre-configured in the driver's emotion recognition device.
[0072] Specifically, the above step S503 includes the following steps a1 to a5:
[0073] Step a1, when -5°≤θ1≤5°, -5°≤θ2≤5°, 0°≤θ3≤3° and 0mm≤D≤3mm, the driver's emotional state is natural.
[0074] Among them, θ1 represents the offset angle of the eyebrow annotation line, θ2 represents the offset angle of the lip annotation line, θ3 represents the offset angle of the lower edge of the eye socket annotation line, and D represents the deviation value of the eyebrow distance.
[0075] Step a2: When 5°<1≤10° and D≥5mm, the driver's emotional state is angry.
[0076] Step a3, when 5°<1≤10° or -10°≤θ1<-5°, 0mm≤D≤3mm, 0°≤θ3≤10° and the lips are in an open state, the driver's emotional state is shock;
[0077] Step a4, when 5°<1≤10°, 5°<2≤10°, -5°≤θ3<0°, and 0mm≤D≤3mm, the driver's emotional state is excitement;
[0078] Step a5: when -10°≤θ2<-5°, the driver's emotional state is sad.
[0079] Specifically, Figure 6 The left side of displays multiple preset offset intervals. After the target offset is determined, the target offset is compared with the multiple offset intervals, and the offset interval matching the target offset is screened out from the multiple offset intervals. Then, based on the offset interval matching the target offset, the emotional state of the driver at the current moment is determined.
[0080] For example, if θ1=10°, θ3=10°, D=0 mm, and the lips are in an open state (the maximum distance between the upper lip and the lower lip is greater than a preset distance), it can be determined that the driver's emotional state at the current moment is shock.
[0081] Step S504, selecting music that matches the driver's emotional state, and controlling the voice module to play it.
[0082] Among them, the types of music include slow, cheerful, bright, dynamic and / or lyrical.
[0083] Specifically, if the driver's emotional state is determined to be sad, the voice module is controlled to play gentle music; if the driver's emotional state is determined to be excited or natural, the voice module is controlled to play cheerful or dynamic music; if the driver's emotional state is angry, the voice module is controlled to play bright music; if the driver's emotional state is shocked (surprised), the voice module is controlled to play cheerful or lyrical music.
[0084] The method for identifying driver emotions provided in this embodiment determines the driver's emotional state by comparing the target offset with the preset offset interval, and can identify emotions more quickly and accurately. After determining the driver's emotional state, the voice module is controlled to play music that matches the driver's emotional state, which can provide comfort and companionship to the driver, improve driving safety, and provide the driver with a better driving experience.
[0085] In this embodiment, a method for identifying a driver's emotion is provided, which can be used in the above-mentioned device for identifying a driver's emotion in the vehicle intelligent cockpit. Figure 7 FIG. 4 is a flow chart of another method for identifying driver emotions according to an embodiment of the present invention. Figure 7 As shown, the method comprises the following steps:
[0086] Step S701, obtaining the current facial image of the driver.
[0087] For details, please see Figure 1Step S101 of the illustrated embodiment will not be described in detail here.
[0088] Step S702, determining whether the driver is the target driver.
[0089] Specifically, the vehicle is bound to the target driver, and whether the driver is the target driver can be determined based on the acquired current facial image and the preset facial image.
[0090] Step S703, when the driver is the target driver, a target offset is determined according to the current facial image and a preset facial image of the driver.
[0091] Specifically, different users have different facial features. To ensure the accuracy of emotion recognition, emotion recognition is performed only when the driver is a target driver, and is not performed when the driver is not a target driver.
[0092] Step S704: determining the driver's emotional state according to the target offset.
[0093] For details, please see Figure 5 Step S503 of the illustrated embodiment will not be described in detail here.
[0094] Step S705: adjust the form of the digital human on the vehicle according to the driver's emotional state, and trigger the digital human to communicate with the driver.
[0095] Among them, form includes gestures and facial expressions.
[0096] Specifically, after determining the driver's emotional state, the driver's emotion recognition device can adjust the form of the digital human on the vehicle based on the driver's emotional state through the vehicle-mounted large model and trigger the digital human to communicate with the driver.
[0097] Among them, the on-board big model refers to a large-scale language and cognitive model based on artificial intelligence applied to the automobile intelligent system, which has a series of capabilities such as spatial environment understanding, user status perception, multi-modal command parsing, multi-round logical dialogue, content generation, etc., and then enables a series of functions including emotion perception, intelligent assistant, emotion-based dialogue, creative content generation, personalized interaction, etc., which can enhance the intelligent experience of automobiles. The on-board big model in this embodiment is a conventional on-board big model currently installed in vehicles, which is not specifically described in the present invention.
[0098] For example, the digital human's form can be adjusted by the digital human action control unit in the driver's emotion recognition system, and the digital human can communicate with the driver by the language module in the driver's emotion recognition system.
[0099] Specifically, the above step S705 includes steps b1 to b4:
[0100] Step b1, when the emotional state is anger, the digital human's gestures are adjusted to a predefined first gesture, and the digital human's facial expression is adjusted to a predefined first facial expression, so that the digital human presents a soothing and caring look.
[0101] Step b2, when the emotional state is shock, the digital human's gesture is adjusted to a predefined second gesture, and the digital human's facial expression is adjusted to a predefined second facial expression, so that the digital human presents a curious and concerned look.
[0102] Step b3, when the emotional state is excitement, the digital human's gestures are adjusted to a predefined third gesture, and the digital human's facial expression is adjusted to a predefined third facial expression, so that the digital human presents a happy expression.
[0103] Step b4, when the emotional state is sadness, the digital human's gestures are adjusted to a predefined fourth gesture, and the digital human's facial expression is adjusted to a predefined fourth facial expression, so that the digital human presents a sad and caring look.
[0104] For example, Figure 8 As shown, after the digital human's form is adjusted to a soothing and concerned expression, a conversation between the digital human and the driver is triggered. For example, it can be "What happened to make you so angry? Can you tell (digital human wake-up word) me and see if I can help you?" and "Let me play a song "Don't be Angry" for you".
[0105] After the digital human's appearance is adjusted to an expression of curiosity and concern, a conversation between the digital human and the driver is triggered. For example, it can be "Don't be afraid~ I'm with you" and "But can you tell me what happened? (Digital human wake-up word) I'm curious too" and so on.
[0106] After adjusting the digital human's appearance to a happy expression, a dialogue between the digital human and the driver is triggered. For example, it can be "Today is a really good day. Let me guess what makes you happy" and "The four great joys in life are rain after a long drought, meeting an old friend in a foreign land, the wedding night, and passing the imperial examination. Which one is yours? Can you give me some hints?" etc.
[0107] After adjusting the digital human's form to a sad and caring expression, a conversation between the digital human and the driver is triggered. For example, it can be "You don't seem very happy. Would you like to chat?" and "I know you are in a bad mood. Let's listen to some calming music together. I will always be with you. I hope you don't keep immersing yourself in negative emotions. Let's overcome them together."
[0108] In this embodiment, after the driver's emotional state is determined, the driver's emotional state can be transmitted to the on-board large model. The on-board large model can be used to adjust the form of the digital human and trigger the digital human to have a conversation with the driver, thereby providing emotional value to the driver and giving the driver the comfort and companionship he desires, thereby improving driving safety and enhancing the user's driving experience.
[0109] In this embodiment, a device for identifying driver emotions is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0110] This embodiment provides a device for identifying driver emotions. Fig. 9 As shown, including:
[0111] An acquisition module 901 is used to acquire a current facial image of the driver;
[0112] A first determination module 902 is used to determine a target offset according to a current facial image and a preset facial image of the driver, wherein the target offset is an offset of a target feature in the current facial image relative to a target feature in the preset facial image, and the target feature includes at least one of an eyebrow marking line, a lower orbital edge marking line, a lip marking line, and an eyebrow distance;
[0113] The second determination module 903 is used to determine the driver's emotional state according to the target offset, wherein the emotional state includes natural state, anger, shock, excitement and sadness.
[0114] In some optional implementations, the first determining module 902 includes:
[0115] The first determination unit is used to compare the target features of the current facial image with the target features of the preset facial image to determine the target offset, wherein the target offset includes the offset angle of the eyebrow marking line, the offset angle of the lower edge of the eye socket marking line, the offset angle of the lip marking line and the deviation value of the eyebrow distance.
[0116] In some optional implementations, the second determining module 903 includes:
[0117] The second determination unit is used to determine the emotional state of the driver as a natural state when -5°≤θ1≤5°, -5°≤θ2≤5°, 0°≤θ3≤3° and 0mm≤D≤3mm, wherein θ1 represents the offset angle of the eyebrow marking line, θ2 represents the offset angle of the lip marking line, θ3 represents the offset angle of the lower edge of the eye socket marking line, and D represents the deviation value of the eyebrow distance;
[0118] a third determination unit, configured to determine the driver's emotional state as anger when 5°<1≤10° and θ≥5mm;
[0119] a fourth determination unit, configured to determine the driver's emotional state as shock when 5°<1≤10° or -10°≤θ1<-5°, 0mm≤D≤3mm, 0°≤θ3≤10°, and the lips are in an open state;
[0120] a fifth determining unit, configured to determine the driver's emotional state as excitement when 5°<1≤10°, 5°<2≤10°, -5°≤θ3<0°, and 0mm≤D≤3mm;
[0121] The sixth determination unit is configured to determine the driver's emotional state as sadness when -10°≤θ2<-5°.
[0122] In some optional embodiments, the device further comprises:
[0123] The adjustment module is used to adjust the form of the digital human in the vehicle according to the driver's emotional state and trigger the digital human to communicate with the driver, where the form includes gestures and facial expressions.
[0124] In some optional implementations, the adjustment module includes:
[0125] The first adjustment unit is used to adjust the digital human's gestures to a predefined first gesture, and adjust the digital human's facial expression to a predefined first facial expression, so that the digital human presents a soothing and caring expression, when the emotional state is anger;
[0126] The second adjustment unit is used to adjust the digital human's gesture to a predefined second gesture, and adjust the digital human's facial expression to a predefined second facial expression, so that the digital human presents a curious and concerned expression, when the emotional state is shock;
[0127] A third adjustment unit is used to adjust the digital human's gestures to a predefined third gesture, and adjust the digital human's facial expression to a predefined third facial expression, so that the digital human presents a happy expression, when the emotional state is excitement;
[0128] The fourth adjustment unit is used to adjust the digital human's gestures to a predefined fourth gesture, and adjust the digital human's facial expression to a predefined fourth facial expression when the emotional state is sad, so that the digital human presents a sad and caring expression.
[0129] In some optional embodiments, the device further comprises:
[0130] The control module is used to select music that matches the driver's emotional state and control the voice module to play to adjust the driver's emotions.
[0131] In some optional embodiments, the vehicle is bound to the target driver, and the device further comprises:
[0132] A third determination module, used to determine whether the driver is a target driver;
[0133] The first determining module 902 includes:
[0134] The seventh determination unit is used to determine the target offset according to the current facial image and the preset facial image of the driver when the driver is the target driver.
[0135] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0136] The driver emotion recognition device in this embodiment is presented in the form of a functional unit, where the unit refers to an application specific integrated circuit (ASIC) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0137] The present invention also provides a driver emotion recognition system, such as Fig.10 As shown, the system includes a driver monitoring system 1001, a data judgment module 1002, a vehicle-mounted large model 1003 and a digital human 1004.
[0138] Among them, the driver monitoring system 1001 is used to collect the current facial image of the driver; the data judgment module 1002 is used to obtain the current facial image of the driver from the driver monitoring system 1001 through the vehicle system library 1005, and is used to determine the target offset according to the current facial image and the preset facial image of the driver, and is used to determine the emotional state of the driver according to the target offset, and is used to adjust the shape of the digital human through the vehicle-mounted large model according to the emotional state of the driver, and trigger the digital human to communicate with the driver.
[0139] Among them, the target offset is the offset of the target feature in the current facial image relative to the target feature in the preset facial image, the target feature includes at least one of the eyebrow marking line, the lower edge of the eye socket marking line, the lip marking line and the eyebrow distance, the emotional state includes natural state, anger, shock, excitement and sadness, and the form includes gestures and facial expressions.
[0140] Exemplarily, the data judgment module 1002 in this embodiment may be the above-mentioned driver emotion recognition device.
[0141] An embodiment of the present invention further provides a vehicle having the driver emotion recognition device or driver emotion recognition system according to any one of the above embodiments.
[0142] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the present invention.
Claims
1. A method for identifying driver emotions, characterized in that: The method comprises: Get the current facial image of the driver; Determine a target offset according to the current facial image and the preset facial image of the driver, wherein the target offset is an offset of a target feature in the current facial image relative to a target feature in the preset facial image, and the target feature includes at least one of an eyebrow marking line, a lower orbital edge marking line, a lip marking line, and an eyebrow distance; The driver's emotional state is determined according to the target offset, wherein the emotional state includes natural state, anger, shock, excitement and sadness.
2. The method according to claim 1, characterized in that The preset facial image is a facial image of the driver in a natural state, and the target offset is determined according to the current facial image and the preset facial image of the driver, including: The target features of the current facial image and the target features of the preset facial image are compared to determine the target offset, wherein the target offset includes an offset angle of an eyebrow marking line, an offset angle of a lower orbital edge marking line, an offset angle of a lip marking line, and a deviation value of an eyebrow distance.
3. The method according to claim 2, characterized in that Determining the driver's emotional state according to the target offset includes: In the case of -5°≤θ1≤5°, -5°≤θ2≤5°, 0°≤θ3≤3° and 0mm≤D≤3mm, the emotional state of the driver is determined to be a natural state, wherein θ1 represents the offset angle of the eyebrow marking line, θ2 represents the offset angle of the lip marking line, θ3 represents the offset angle of the lower edge of the eye socket marking line, and D represents the deviation value of the eyebrow distance; In the case of 5°<1≤10° and D≥5mm, determining the driver's emotional state as angry; In the case where 5°<1≤10° or -10°≤θ1<-5°, 0mm≤D≤3mm, 0°≤θ3≤10° and the lips are in an open state, determining the emotional state of the driver as shocked; In the case of 5°<1≤10°, 5°<2≤10°, -5°≤θ3<0°, and 0mm≤D≤3mm, determining the emotional state of the driver as excitement; In the case of -10°≤θ2<-5°, the driver's emotional state is determined to be sad.
4. The method according to any one of claims 1 to 3, characterized in that After determining the emotional state of the driver according to the target offset, the method further includes: According to the emotional state of the driver, the form of the digital human on the vehicle is adjusted, and the digital human is triggered to have a conversation with the driver, wherein the form includes gestures and facial expressions.
5. The method according to claim 4, characterized in that The adjusting the form of the digital human on the vehicle according to the emotional state of the driver and triggering the digital human to have a conversation with the driver includes: When the emotional state is anger, the digital human's gesture is adjusted to a predefined first gesture, and the digital human's facial expression is adjusted to a predefined first facial expression, so that the digital human presents a soothing and caring look; When the emotional state is shock, the digital human's gesture is adjusted to a predefined second gesture, and the digital human's facial expression is adjusted to a predefined second facial expression, so that the digital human presents a curious and concerned look; When the emotional state is excitement, the digital human's gesture is adjusted to a predefined third gesture, and the digital human's facial expression is adjusted to a predefined third facial expression, so that the digital human presents a happy expression; When the emotional state is sadness, the digital human's gesture is adjusted to a predefined fourth gesture, and the digital human's facial expression is adjusted to a predefined fourth facial expression, so that the digital human presents a sad and caring look.
6. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Select music that matches the driver's emotional state and control the voice module to play it.
7. The method according to any one of claims 1 to 3, characterized in that The vehicle is bound to a target driver, and before determining the target offset according to the current facial image and the preset facial image of the driver, the method further includes: determining whether the driver is the target driver; The step of determining a target offset according to the current facial image and a preset facial image of the driver includes: In a case where the driver is a target driver, a target offset is determined according to the current facial image and a preset facial image of the driver.
8. A driver emotion recognition device, characterized in that: The device comprises: An acquisition module, used for acquiring a current facial image of the driver; A first determination module is used to determine a target offset according to the current facial image and the preset facial image of the driver, wherein the target offset is an offset of a target feature in the current facial image relative to a target feature in the preset facial image, and the target feature includes at least one of an eyebrow marking line, a lower orbital edge marking line, a lip marking line, and an eyebrow distance; The second determination module is used to determine the emotional state of the driver according to the target offset, wherein the emotional state includes natural state, anger, shock, excitement and sadness.
9. A driver emotion recognition system, characterized in that: The system includes a driver monitoring system, a data judgment module, a large vehicle model and a digital human; The driver monitoring system is used to collect the current facial image of the driver; The data judgment module is used to obtain the current facial image of the driver from the driver monitoring system, to determine the target offset according to the current facial image and the preset facial image of the driver, to determine the emotional state of the driver according to the target offset, and to adjust the form of the digital human through the vehicle-mounted large model according to the emotional state of the driver, and to trigger the digital human to have a dialogue with the driver, wherein the target offset is the offset of the target feature in the current facial image relative to the target feature in the preset facial image, the target feature includes at least one of the eyebrow annotation line, the lower edge of the eye socket annotation line, the lip annotation line and the eyebrow distance, the emotional state includes natural state, anger, shock, excitement and sadness, and the form includes gestures and facial expressions.
10. A vehicle, characterized in that: include: The driver emotion recognition device as claimed in claim 8, or the driver emotion recognition system as claimed in claim 9.
