Vehicle user emotion recognition method, device, system, vehicle and medium

Abstract

The present disclosure relates to a method, device, system, vehicle and medium for identifying emotions of vehicle users, and relates to the field of vehicle technology. It includes: determining the user emotions of users in the vehicle according to multimodal emotion-related information corresponding to the vehicle, wherein the multimodal emotion-related information includes at least two emotion-related information associated with the user emotions, wherein at least one of the emotion-related information is emotion-related information corresponding to the area where the user is located. By using the method, device, system, vehicle and medium for identifying emotions of vehicle users proposed in the present disclosure, user emotions can be obtained from at least two dimensions, making the determined user emotions more accurate.

Description

Technical Field

[0001] The present disclosure relates to the field of vehicle technology, and in particular, to a method, device, system, vehicle and medium for identifying an emotion of a vehicle user. Background Art

[0002] During the driving process, the extreme emotions of the driver and passengers will affect the driver's normal operation and judgment. If an emergency occurs, the driver may be unable to make timely judgments and take timely measures, thereby increasing the risk of traffic accidents.

[0003] In the related art, there is a driver emotion detection function (Drivers Emotion Detection), which is used to detect and identify the driver's emotional state and determine whether the emotional state will affect the normal driving of the vehicle. However, the driver emotion detection function usually performs a single-dimensional detection based on the driver's image, resulting in low accuracy and reliability of the detected driver's user emotion. In addition, the related art is less comprehensive in the recognition of in-vehicle user emotions. Summary of the invention

[0004] The purpose of the present disclosure is to provide a vehicle user's emotion recognition method, device, system, vehicle and medium device to more accurately calculate the user's emotions and provide the user with an effective emotion soothing strategy.

[0005] In order to achieve the above object, the present disclosure provides a method for identifying an emotion of a vehicle user, comprising:

[0006] The user emotion of a user in the vehicle is determined based on multimodal emotion-related information corresponding to the vehicle, wherein the multimodal emotion-related information includes at least two emotion-related information associated with the user emotion, wherein at least one of the emotion-related information is emotion-related information corresponding to an area where the user is located.

[0007] Optionally, the multimodal emotion-related information includes at least two of the user's voice information, the user's facial image information, and the vehicle's driving information.

[0008] Optionally, the voice information is emotion-related information corresponding to the area where the user is located, and / or the face image information is emotion-related information corresponding to the area where the user is located.

[0009] Optionally, the user emotion is determined based on a plurality of emotion features obtained according to the multimodal emotion-related information, the plurality of emotion features comprising at least two of a language emotion feature, a fatigue detection feature, a facial emotion feature, and a driving emotion feature;

[0010] The language emotion feature is determined based on the voice information; the fatigue detection feature and / or the facial emotion feature is determined based on the face image information; and the driving emotion feature is determined based on the driving information.

[0011] Optionally, the fatigue detection feature includes a feature for characterizing the degree of opening or closing of the user's eyes.

[0012] Optionally, the user emotion is determined based on multiple fusion features, and the multiple fusion features are obtained by fusing the multiple emotion features with latent features based on a cross-attention mechanism, and the latent features are obtained by fusing the multiple emotion features through low-nodality.

[0013] Optionally, the user emotion is determined based on a multimodal feature obtained by concatenating the multiple fusion features.

[0014] Optionally, the user emotion is obtained after a first convolutional layer processes the multimodal feature; wherein the first convolutional layer is trained by associated multimodal feature samples and user emotion samples.

[0015] Optionally, the voice information is determined from first voice information collected from an area where the user is located, and the area where the user is located is determined based on the facial image information.

[0016] Optionally, the method further comprises:

[0017] Using an emotion soothing strategy corresponding to the user's emotion, the on-board equipment on the vehicle is controlled to soothe the user.

[0018] Optionally, the vehicle-mounted device includes a vehicle-mounted device corresponding to an area where the user is located.

[0019] Optionally, the emotion soothing strategy is determined based on one or more of the user's emotion, the user's identity information, and the area where the user is located, and the user identity information is determined based on the first voiceprint information in the voice information and / or the first facial information in the facial image information.

[0020] Optionally, the user identity information is the identity information corresponding to the target voiceprint information that matches the first voiceprint information among multiple pre-stored second voiceprint information; and / or, the user identity information is the identity information corresponding to the target facial information that matches the first facial information among multiple pre-stored second facial information.

[0021] Optionally, the emotion soothing strategy includes an emotion soothing method corresponding to the user emotion, and the emotion soothing method is determined based on one or more of the user emotion, the user identity information, and the area where the user is located. The emotion soothing method includes at least one of a music soothing method, a video soothing method, an aroma soothing method, a conversation soothing method, and a light soothing method.

[0022] Optionally, the area where the user is located is a driving area or a non-driving area;

[0023] The emotional soothing method matched to the driving area includes at least one of a music soothing method and a smell soothing method;

[0024] The emotion soothing method matched in the non-driving area includes at least one of a music soothing method, a video soothing method, a smell soothing method, a light soothing method and a conversation soothing method.

[0025] Optionally, the emotion soothing strategy also includes emotion regulation content that the vehicle-mounted device needs to output when a corresponding emotion soothing method is adopted.

[0026] Optionally, the emotion regulation content includes first emotion regulation content that is triggered to be output by an onboard device on the vehicle when the user is in the user emotion within a historical time and adopts a corresponding emotion soothing method.

[0027] Optionally, the emotion regulation content includes third emotion regulation content that is filtered out from second emotion regulation content and can soothe the user's emotions, and the second emotion regulation content includes emotion regulation content that is triggered by the user within a historical time period and output by an onboard device on the vehicle.

[0028] Optionally, the facial image information is obtained by identifying a second passenger image of the vehicle, and the second passenger image is obtained by adjusting the brightness of the first passenger image; the brightness uniformity of the second passenger image is higher than that of the first passenger image.

[0029] Optionally, the second passenger image is obtained by performing shading adjustment on the first passenger image using an image shading processing model.

[0030] In order to achieve the above-mentioned purpose, the present disclosure provides an electronic device, including a processor, wherein the processor is connected to a memory, and the processor is used to execute the computer program in the memory to implement the steps of the vehicle user emotion recognition method proposed in the present disclosure.

[0031] In order to achieve the above-mentioned purpose, the present disclosure provides an emotion recognition system for a vehicle user, including the electronic device proposed in the present disclosure.

[0032] In order to achieve the above-mentioned objectives, the present disclosure provides a vehicle, including an electronic device proposed by the present disclosure or an emotion recognition system of a vehicle user proposed by the present disclosure.

[0033] To achieve the above objectives, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a vehicle user emotion recognition method proposed in the present disclosure.

[0034] Through the above technical solution, the user's emotions can be determined from multiple dimensions from multimodal emotion-related information. When any emotion-related information is missing or has low credibility, the user's emotions can be determined through other emotion-related information, so that the user's emotions can be analyzed, which improves the accuracy of the user's emotion analysis. When multiple emotion-related information is not missing, the emotion-related information of multiple dimensions is combined to make the obtained user emotions more accurate. In addition, since at least one emotion-related information corresponds to the area where the user is located, the above technical solution can not only identify the driver's emotions, but also the emotions of passengers in other areas, which improves the comprehensiveness of in-car user emotion recognition and helps to consider the impact of passenger emotions on the driver's operation and judgment.

[0035] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0037] Figure 1 It is a flowchart of the steps of a method for identifying an emotion of a vehicle user proposed according to an exemplary embodiment.

[0038] Figure 2 It is a flowchart of the steps of a method for identifying an emotion of a vehicle user proposed according to an exemplary embodiment.

[0039] Figure 3 It is a flowchart of steps for extracting language emotion features according to an exemplary embodiment.

[0040] Figure 4 It is a flowchart of the steps of extracting fatigue detection features according to an exemplary embodiment.

[0041] Figure 5 The present invention is a flowchart of steps for extracting facial emotion features according to an exemplary embodiment.

[0042] Figure 6It is a flowchart of the steps of extracting driving emotion features according to an exemplary embodiment.

[0043] Figure 7 is a schematic diagram of a multimodal fusion model proposed according to an exemplary embodiment.

[0044] Figure 8 The diagram is a schematic diagram of an emotion monitoring device controlling an in-vehicle device according to an exemplary embodiment.

[0045] Fig. 9 The present invention is a flowchart of the steps of processing a first passenger image to obtain a second passenger image according to an exemplary embodiment.

[0046] Fig.10 is a block diagram of a device for identifying an emotion of a vehicle user according to an exemplary embodiment.

[0047] Fig.11 is a block diagram of an electronic device according to an exemplary embodiment.

[0048] Fig.12 is a block diagram of a vehicle according to an exemplary embodiment. DETAILED DESCRIPTION

[0049] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0050] Figure 1 It is a flowchart of a method for identifying an emotion of a vehicle user proposed by the present disclosure according to an exemplary embodiment. The method for identifying an emotion of a vehicle user includes the following steps.

[0051] In step S11, the user emotion of the user is determined according to the multimodal emotion-related information corresponding to the vehicle.

[0052] The multimodal emotion-related information includes at least two emotion-related information associated with the user's emotion, wherein at least one of the emotion-related information is emotion-related information corresponding to the area where the user is located.

[0053] Optionally, the multimodal emotion-related information includes at least two of the user's voice information, the user's facial image information, and the vehicle's driving information.

[0054] Voice information is voice information used to express user emotions. Voice information used to express user emotions can be extracted from the voice signal emitted by the user. The voice information includes information such as the tone, speech speed, energy, and voice content of the voice emitted by the user. The higher the tone, the more excited the user is; the faster the speech speed, the more excited the user is; the greater the energy, the more excited the user is; the words contained in the voice content are positive, which means the user is more excited; the words contained in the voice content are negative, which means the user is more negative. For example, if the voice content contains the word "happy", it means the user is excited.

[0055] Among them, the voice information is the voice information sent from the area where the user is located. The area where the user is located is determined based on the face image information. The area where the user is located can be determined based on the face image information, and then the voice information of the area where the user is located is collected.

[0056] For example, the image acquisition device may be used to capture passenger images in the vehicle cabin, and the facial image information of each user in the passenger image may be obtained through the passenger image, and then the area where the user is located may be determined based on the facial image information. For example, if there is a user in the driving area of ​​the main driving area, and there is no user in the co-pilot area and the rear seat area, then the area where the user is located may be determined to be the main driving area based on the facial image information.

[0057] Facial image information is image information used to express user emotions. Facial image information used to express user emotions can be extracted from the collected passenger images. The facial image information includes key point information of the user's face, and the key point information includes facial information such as the mouth, eyes, eyebrows and forehead.

[0058] Driving information is vehicle driving information used to express user emotions, such as vehicle speed, acceleration, yaw rate, vehicle position, steering wheel steering angle, etc. For example, when the user is in extreme emotions, at least one of the vehicle's speed, acceleration, yaw rate, and steering wheel steering angle will be significantly increased.

[0059] For example, when the user does not send out any voice information, the user's emotion is determined based on the facial image information and the driving information.

[0060] Optionally, the voice information is emotion-related information corresponding to the area where the user is located, and / or the face image information is emotion-related information corresponding to the area where the user is located.

[0061] In some scenarios, when the user does not send a voice message, if the user is determined to be angry based on the facial image information, and the driving information determines that the distance between the user's vehicle and other vehicles is less than a preset distance, it can be determined that the user is currently angry because of the congested road.

[0062] For example, when the determined facial image information cannot identify the user's emotion, the user's emotion is determined based on the voice information and the driving information.

[0063] In some scenarios, when the user has no facial expression, if the user's anxiety is determined based on the voice information sent by the user, and there is a traffic light in front of the vehicle through driving information, it can be determined that the user is currently anxious because of waiting for the traffic light ahead.

[0064] For example, when the user does not send out any voice information and the user's emotion cannot be identified based on the facial image information, the user's emotion is determined based on the driving information.

[0065] In some scenarios, when the user does not send a voice message and has no facial expression, it means that the user's language and expression do not show obvious emotional tendencies. At this time, if the vehicle speed is determined to be greater than the preset speed based on the driving information, it indicates that the user is currently emotionally excited.

[0066] In the related technology, there is a driver emotion detection function (Drivers Emotion Detection), which is used to detect and identify the driver's emotional state and determine whether the emotional state will affect the normal driving of the vehicle. However, the driver emotion detection function usually determines the user's emotions based on a single dimension, and the accuracy of the calculated user emotions is low.

[0067] For example, facial analysis technology can be used to detect the driver's face in real time, and the driver's emotional state can be calculated based on the driver's facial features. However, due to individual differences between users, some users' emotions have reached extreme states, but their facial emotions are not obvious, and there is no obvious expression on the user's face, making it difficult for the driver's emotion detection function to detect the driver's user emotions.

[0068] For example, the driver's voice can be detected through voice recognition technology, and the emotion category contained in the driver's speech can be identified based on the driver's voice. However, if the driver's face shows extreme emotions such as excitement, anger or fatigue, if the driver does not speak, the driver emotion detection function cannot recognize the driver's user emotion.

[0069] It can be seen that the user emotions calculated by using a single-dimensional solution to determine the user emotions may not be accurate.

[0070] The present disclosure proposes a method for identifying emotions of vehicle users, which calculates the user emotions from at least two dimensions of three dimensions: voice information, face image information, and driving information.

[0071] When the user's emotions cannot be identified through facial image information, the user's emotions are identified through voice information and driving information. For example, when the user has no facial expression, the user's emotions are judged through the voice information and driving information sent by the user; when the user's emotions cannot be identified through voice information, the user's emotions are identified through facial image information and driving information. For example, when the user does not speak, the user's emotions are judged through the user's facial expression and driving information; when the user's emotions cannot be judged through facial image information and voice information, the user's emotions are judged through driving information. For example, when the user has neither facial expression nor speaks, the vehicle's driving information is used to judge whether the user is driving the vehicle in an extreme manner, such as whether the driving speed is too fast, whether the driving acceleration is large, and whether the yaw angular velocity is large, so as to judge the user's emotions.

[0072] Through the above technical solution, user emotions are determined from at least two dimensions of voice information, facial image information and driving information. When any one of the information is missing, the user emotions can be determined through other information, so that the user emotions can be analyzed. Or when all three information are present, the information of the three dimensions can be combined to make the obtained user emotions more accurate.

[0073] The following introduces an exemplary embodiment involved in the present disclosure, which is used to explain the process of obtaining user emotions from at least two dimensions of three dimensions: voice information, face image information and driving information, and includes the following steps.

[0074] (1) The language emotion feature is determined based on the language information, and the language emotion feature can be obtained based on the speech information.

[0075] Language emotion features are features used to represent emotions such as anger, happiness, frustration, excitement, etc. expressed in the user's voice.

[0076] Optionally, the language emotion feature may be obtained based on at least one of the information contained in the voice information, such as voice content, voice pitch, voice energy, and voice speed.

[0077] See also Figure 3 As shown, taking the example of obtaining language emotion features based on the voice content contained in the voice information, the user voices in various areas can be collected through an audio collection device, and then the user voices can be pre-processed such as filtering to obtain voice information; then the voice information is converted into text information; and based on the text information, the language emotion features can be obtained.

[0078] The audio collection device may be a microphone arranged in multiple areas in the vehicle, and the microphone is installed above the A-pillar or B-pillar on the left and right inner sides of the vehicle to collect voice signals in different areas. The area in the vehicle includes the main driving area and the non-main driving area in the vehicle. The main driving area includes the front driving area and the co-pilot area of ​​the vehicle, and the non-main driving area may be the rear seat area.

[0079] An exemplary solution for obtaining language emotion features from text information is: using ASR (Automatic Speech Recognition) to convert speech information into text information; then preprocessing the text information, such as removing stop words, punctuation marks, lowercase, stem extraction and part-of-speech restoration in the text information, to obtain processed text information; and then using a deep learning model (Transformer) as Figure 7 The language emotion feature extractor in the text information extracts the language emotion features from the processed text information. The deep learning network can be a self-attention mechanism neural network trained on the CMU-MOSEI (CarnegieMellon University Multimodal Sentiment and Emotion Intensity, multimodal dataset) training set. The multimodal dataset includes the emotion category annotation of the speaker's language.

[0080] (2) The fatigue detection feature and / or facial emotion feature is determined based on the facial image information. The fatigue detection feature and / or facial emotion feature can be obtained based on the facial image information.

[0081] The fatigue detection feature is a feature used to indicate fatigue or non-fatigue expressed by the user's facial expression.

[0082] An exemplary solution for obtaining fatigue detection features based on facial image information is: Figure 4 As shown, fatigue detection features can be extracted based on facial image information, and the fatigue detection features include features for characterizing the degree of eye opening and closing of the user.

[0083] The face image can be input into the face 68 key point detection function in the Python (programming language) function package dlib (tool library). The key point detection function selects the key points representing the user's eyes. The key point detection function is Figure 7 The fatigue detection feature extractor in the image processing unit is used; the distance between the upper and lower eyelids of each eye is calculated, and the degree of eye opening and closing is obtained through the distance between the upper and lower eyelids; the degree of eye opening and closing is input into the feature extraction of one-dimensional convolution for feature mapping, and the degree of eye opening and closing is mapped from one space to another space to obtain fatigue detection features. It can be understood that the larger the distance between the upper and lower eyelids, the greater the degree of eye opening and closing.

[0084] Passenger images can be collected by an image acquisition device. The image acquisition device can be installed above the front windshield of the cabin and can cover and collect passenger images of various areas in the cabin; the passenger images are then segmented into facial images of different users.

[0085] Among them, a threshold can be set to determine whether the human eyes are closed, and the degree of opening and closing of the user's eyes can be continuously monitored. When the degree of opening and closing of the human eyes is less than the threshold and the duration is greater than the preset duration, it is determined that the user's eyes are in a closed state for a long time, rather than a blinking state, and then it is determined that the user is in a fatigue state.

[0086] Among them, the degree of opening and closing of the human eye can be reflected by the aspect ratio of the human eye (EAR). The larger the aspect ratio, the lower the degree of opening and closing of the human eye.

[0087] Facial emotion features are features used to represent emotions such as anger, happiness, frustration, excitement, etc. expressed by the user's facial expressions.

[0088] An exemplary solution for obtaining facial emotion features based on facial image information is: Figure 5 As shown, facial emotion features in the face image are extracted based on the face image information.

[0089] The face image can be input into the facial emotion recognition model to obtain the facial emotion features. The facial emotion recognition model can use the convolutional neural network (SqueezeNet) trained on KMU-FED (Facial Emotion Classification Dataset). The facial emotion recognition model is Figure 7 The facial emotion feature extractor shown.

[0090] (3) The driving emotion feature is determined based on the driving information, and the driving emotion feature can be obtained based on the driving information.

[0091] The driving emotion feature is used to represent the user's emotions such as anger, calmness, etc. indicated based on the driving information of the vehicle after the user controls the vehicle.

[0092] An exemplary solution for obtaining driving emotion characteristics according to the driving information is: Figure 6 As shown in the figure, the driving information of the vehicle in the driving state is collected, and the driving information includes parameters such as the vehicle's position, speed, acceleration, steering wheel steering angle, yaw angular velocity, yaw angular acceleration and lane keeping; then the driving information feature extractor based on SqueezeNet performs feature mapping on the driving information, maps the driving information from one space to another space, and obtains the driving emotion characteristics.

[0093] (4) The user emotion feature is determined based on a plurality of emotion features obtained based on the multimodal emotion-related information, wherein the plurality of emotion features include at least two of language emotion features, fatigue detection features, facial emotion features, and driving emotion features. The user emotion can be determined based on at least two of the language emotion features, the fatigue detection features, the facial emotion features, and the driving emotion features.

[0094] Optionally, see Figure 7 As shown, the text information converted from the voice information is input into the language emotion feature extractor to obtain the language emotion feature; the face image information is respectively input into the fatigue detection feature extractor and the facial emotion feature extractor to obtain the fatigue detection feature and the facial emotion feature respectively; the driving information is input into the driving information extractor to obtain the driving emotion feature; and then the language emotion feature, fatigue detection feature, facial emotion feature and driving emotion feature are fused to obtain the user emotion.

[0095] See also Figure 2 and Figure 7 As shown, an exemplary scheme for fusing language emotion features, fatigue detection features, facial emotion features and driving emotion features to obtain user emotions is: inputting language emotion features, fatigue detection features, facial emotion features and driving emotion features into a multimodal fusion model to obtain user emotions.

[0096] The multimodal fusion model is trained by the following method: the multimodal fusion model is trained using facial image information samples, text information samples and driving information samples labeled with user emotion samples. For example, facial image information samples, text information samples and driving information samples labeled with anger emotions can be input into the multimodal fusion model to determine whether the user emotion output by the multimodal fusion model is anger emotion. If not, continue to input the next group of facial image information samples, text information samples and driving information samples with anger emotions into the multimodal fusion model. If not, repeat the above steps until the user emotion output by the multimodal fusion model is anger emotion.

[0097] Similarly, the above training method is used to train other emotions such as excitement, calmness, happiness, fatigue, etc., so that the multimodal fusion model can recognize different user emotions based on facial image information, text information and driving information.

[0098] Through the above technical solution, the user's emotions can be obtained by combining at least two of the four emotional features of the user's language emotional features, fatigue detection features, facial emotional features and driving emotional features, which accurately determine the user's current emotions from at least two dimensions. Of course, if the user does not currently make a voice, the language emotional features cannot be extracted. The multimodal fusion model can also calculate the user's emotions based on the three emotional features of fatigue detection features, facial emotional features and driving emotional features. Similarly, in the case of missing emotional features, the multimodal fusion model can also calculate the user's emotions through the remaining emotional features.

[0099] The following is an exemplary solution for obtaining user emotions based on four emotion features, namely, language emotion features, fatigue detection features, facial emotion features, and driving emotion features, using a multimodal fusion model, including the following steps.

[0100] (1) The latent feature is obtained by fusing the multiple emotion features through low-level fusion. The language emotion feature, the fatigue detection feature, the facial emotion feature and the driving emotion feature can be fused to obtain the latent feature.

[0101] Optionally, see Figure 7 As shown, the multimodal fusion model includes a low-rank fusion module, which can input language emotion features, fatigue detection features, facial emotion features and driving emotion features into the low-rank fusion module to obtain latent features.

[0102] The low-rank fusion module is used to remove redundant information of each emotional feature while retaining useful information. Therefore, the language emotional features, fatigue detection features, facial emotional features and driving emotional features are input into the low-rank fusion module to remove redundant information in each emotional feature and convert the multimodal emotional features into an updated emotional feature set. The updated emotional feature set is the latent feature, which is a combination of emotional features from different modalities.

[0103] (2) The multiple fused features are obtained by fusing the multiple emotion features with the latent features based on the cross-attention mechanism. The latent features can be fused with the language emotion features, the fatigue detection features, the facial emotion features and the driving emotion features to obtain the multiple fused features.

[0104] The multimodal fusion model includes a cross-attention mechanism. An exemplary scheme for fusing latent features with language emotion features, fatigue detection features, facial emotion features and driving emotion features is: the latent features, language emotion features, fatigue detection features, facial emotion features and driving emotion features can be input into the cross-attention mechanism, and the latent features can be fused with the language emotion features, fatigue detection features, facial emotion features and driving emotion features through the cross-attention mechanism to obtain multiple fused features.

[0105] Optionally, before the latent features are respectively fused with the language emotion features, fatigue detection features, facial emotion features and driving emotion features, the latent features can be first convolved through a second convolutional layer to unify the emotion feature mappings of each dimension contained in the latent features to the same dimension, thereby obtaining the latent features with unified dimensions.

[0106] It can be understood that after the cross-attention mechanism fuses the latent features after the dimensions are unified with the language emotion features, fatigue detection features, facial emotion features and driving emotion features, four fusion features, namely the first fusion feature, the second fusion feature, the third fusion feature and the fourth fusion feature, will be obtained respectively.

[0107] The first fusion feature obtained not only considers the language emotion feature, but also the emotion features of other dimensions. On the one hand, the latent features complement the language emotion features, making the emotion expressed by the first fusion feature more comprehensive; on the other hand, in the absence of language emotion features, emotion features of other modalities can be used to supplement them, making the user emotion obtained more robust; on the third hand, the multimodal fusion model can combine emotion features from different modalities to learn more complex relationships.

[0108] The second fusion feature obtained not only considers the fatigue detection feature, but also the emotional features of other dimensions. On the one hand, the latent features complement the fatigue detection features, making the emotions expressed by the second fusion feature more comprehensive; on the other hand, in the absence of fatigue detection features, emotional features of other modalities can be used to supplement them, making the user emotions more robust; on the third hand, the multimodal fusion model can combine emotional features from different modalities to learn more complex relationships.

[0109] The third fusion feature obtained not only considers facial emotion features, but also emotion features of other dimensions. On the one hand, the latent features complement the facial emotion features, making the emotions expressed by the third fusion feature more comprehensive; on the other hand, in the absence of facial emotion features, emotion features of other modalities can be used to supplement them, making the user emotions obtained more robust; on the third hand, the multimodal fusion model can combine emotion features from different modalities to learn more complex relationships.

[0110] The fourth fusion feature obtained not only considers the driving emotion feature, but also the emotion features of other dimensions. On the one hand, the latent features complement the driving emotion features, making the emotions expressed by the fourth fusion feature more comprehensive; on the other hand, in the absence of driving emotion features, emotion features of other modalities can be used to supplement them, making the user emotions obtained more robust; on the third hand, the multimodal fusion model can combine emotion features from different modalities to learn more complex relationships.

[0111] (3) The user emotion is obtained based on multiple fused features, and the user emotion can be obtained according to the multiple fused features.

[0112] Optionally, after the cross-attention mechanism obtains multiple fused features, the multiple fused features can be mapped to a one-dimensional space, and then the multiple fused features mapped to the one-dimensional space are spliced ​​to obtain multimodal features; and then the user emotion is obtained based on the multimodal features.

[0113] Among them, an exemplary scheme for obtaining user emotions based on multimodal features includes: user emotions are obtained after the multimodal features are processed by the first convolution layer, and the multimodal features can be input into the first convolution layer to obtain the user emotions; wherein, the first convolution layer is obtained by training multimodal feature samples and user emotion samples, and the first convolution layer is used to learn the association and weight between the multimodal feature samples.

[0114] See also Figure 7 As shown in the figure, the multimodal fusion model includes four feature extractors, a low-rank fusion module, a second convolutional layer, a cross-attention mechanism, a feature mapping module and a first convolutional layer from right to left. After inputting text information, facial image information and driving information into the four feature extractors, language emotion features, fatigue detection features, facial emotion features and driving emotion features are obtained respectively. These four emotion features are input into the low-rank fusion module to obtain latent features; then the latent features are input into the second convolutional layer to obtain latent features after unified dimension; then the latent features after unified dimension are input into the four cross-attention mechanisms to obtain four fusion features respectively; then the four fusion features are input into the feature mapping module to map the four fusion features into one-dimensional space; then the four fusion features mapped into one-dimensional space are spliced ​​to obtain multimodal features; finally, the multimodal features are input into the first convolutional layer to obtain user emotions.

[0115] Among them, when training the multimodal fusion model, multimodal feature samples (multimodal feature samples include face image information samples, text information samples and driving information samples) and user emotion samples are used to train the multimodal fusion model. For example, when the user emotion output by the first convolution layer is not a user emotion sample, the network parameters of the first convolution layer and the second convolution layer are updated, and the multimodal feature samples are re-input to determine whether the user emotion output by the first convolution layer is a user emotion sample. If not, the above steps are repeatedly performed until the user emotion output by the first convolution layer is the same as the pre-labeled user emotion sample. The network parameters of the first convolution layer and the second convolution layer include the weights of each network layer in the first convolution layer and the second convolution layer.

[0116] Through the above technical solution, the user's emotions are analyzed by fusing language emotion features, fatigue detection features, facial emotion features and driving emotion features. On the one hand, when some emotion features are missing or unreliable, they can be supplemented by other emotion features to analyze the user's emotions; on the other hand, by combining emotion features from different dimensions and different modalities, the multimodal fusion model can learn the relationship between more complex emotion features, thereby accurately capturing the user's emotions.

[0117] The following describes an exemplary embodiment of the above steps involved in the present disclosure, which is used to explain an emotional soothing scheme for the user after obtaining the user's emotions, including:

[0118] Using an emotion soothing strategy corresponding to the user's emotion, the on-board equipment on the vehicle is controlled to soothe the user.

[0119] The vehicle-mounted equipment includes the vehicle's music player, central control screen, display screen, ambient light, scent distribution device, and dialogue device. The music player is used to play music to soothe the user's emotions; the central control screen and display screen are used to display videos to divert the user's attention and soothe the user's emotions; the ambient light is used to display different lights to soothe the user's emotions; the scent distribution device is used to distribute perfume, air freshener and other scents to soothe the user's emotions; the dialogue device is used to communicate with the user to divert the user's attention and soothe the user's emotions.

[0120] For example, taking a music player as an example, if the user is angry, the emotion soothing strategy may be to control the music player to play soothing music, thereby soothing the user's angry emotion.

[0121] Optionally, the vehicle-mounted device includes a vehicle-mounted device corresponding to an area where the user is located.

[0122] For example, if it is identified that the area where the user is located is the main driving area, the vehicle-mounted equipment in the main driving area will be controlled with an emotion soothing strategy to soothe the emotions of the user of the vehicle in the main driving area.

[0123] Through the above technical solution, after the user's emotions are determined, the vehicle-mounted equipment will be controlled to execute an emotion soothing strategy corresponding to the user's emotions to soothe the user's emotions and reduce the possibility of dangerous driving by the user under extreme emotions.

[0124] The following introduces an exemplary embodiment of the present disclosure involved in the above steps, which is used to explain that the emotion soothing strategy is determined based on one or more of the user's emotion, the user's user identity information, and the area where the user is located.

[0125] The emotion soothing strategy includes emotion soothing methods, emotion regulation content and emotion soothing areas. The emotion soothing methods include at least one of music soothing method, video soothing method, smell soothing method, light soothing method and dialogue soothing method; the emotion regulation content includes the content output under the emotion soothing method; the emotion soothing area includes the area where the user is located.

[0126] User emotions can determine the emotion soothing method and emotion regulation content. For example, if a user is angry, soothing music, which is the emotion regulation content, can be used to regulate the user's emotions.

[0127] The user's user identity information can determine the emotion soothing method and / or emotion regulation content. For example, the user's user identity information is stored in the vehicle's memory, and the user identity information has previously triggered the emotion regulation content of soothing music under the vehicle's music soothing mode. When the same user is in extreme emotions, the emotion regulation content of soothing music under the music soothing mode can be used to regulate the user's emotions.

[0128] The area where the user is located can determine the emotional soothing method, and the area where the user is located includes the driving area or the non-driving area, and the driving area includes the main driving area and the co-pilot area, or the driving area includes the main driving area. For example, if the area where the user is located is the main driving area, music soothing method or smell soothing method will be used to adjust the user's emotions; if the area where the user is located is the non-main driving area, at least one of video soothing method, smell soothing method, light soothing method and dialogue soothing method will be used to adjust the user's emotions.

[0129] (1) The area where the user is located is determined based on the facial image information, and the area where the user is located can be determined based on the facial image information.

[0130] The image acquisition device may be used to acquire images of passengers in the vehicle cabin, and facial image information of each user in the passenger image may be obtained through the passenger image, and then the area where the user is located may be determined based on the facial image information.

[0131] For example, if there is a user in the driving area of ​​the main driving area, but there is no user in the co-pilot area and the rear seat area, then the area where the user is located will be determined to be the main driving area based on the facial image information.

[0132] (2) Compare the first voiceprint information generated in the area where the user is located with the multiple second voiceprint information pre-recorded, and filter out the target voiceprint information that matches the first voiceprint information; or compare the first face information in the area where the user is located with the multiple second face information pre-recorded, and filter out the target face information that matches the first face information; the target face information and / or target voiceprint information is the user identity information.

[0133] For example, multiple second voiceprint information of user A, user B and user C can be pre-recorded in the vehicle. If the area where the user is located is the driving area and the first voiceprint information is generated in the driving area, the first voiceprint information will be compared with the multiple second voiceprint information, and the target voiceprint information consistent with the first voiceprint information will be screened out as the second voiceprint information of user A, thereby determining that user A is speaking in the area where the user is located.

[0134] (3) Determine the user identifier corresponding to the user identity information.

[0135] The user identifier and user identity information can be pre-entered into the vehicle system, and then a correspondence between the user identifier and the user identity information can be established. After the user identity information is obtained, the user identifier corresponding to the user identity information can be determined based on the correspondence, thereby identifying the user who is speaking.

[0136] By comparing the first voiceprint information of the user's area with multiple second voiceprint information, and / or comparing the first face information of the user's area with multiple second face information, the user identity information of the user's area can be determined, thereby distinguishing users in different user areas.

[0137] For example, if the area where the user is located includes the driving area, the left rear seat area and the passenger area, then the user identity of the user who speaks in the driving area can be identified, the user identity information of the user who speaks in the left rear seat area can be identified, and the user identity information of the user who speaks in the passenger area can also be identified, thereby achieving the distinction of user identity information in different areas. It can be understood that the user identifier represents the user identity.

[0138] (4) Controlling the vehicle-mounted equipment in the area where the target user is located using an emotion soothing strategy corresponding to the user emotion of the target user; the target user is a user marked with a user identification corresponding to the target identity information.

[0139] For example, the target user may be comforted by controlling the vehicle-mounted device based on an emotion comfort strategy corresponding to the user emotion of the target user.

[0140] For example, if there is a user in the driving area, but no user in other areas, then the driving area is the area where the user is located. Then, the first voiceprint information emitted by the user in the driving area can be collected and compared with multiple pre-recorded second voiceprint information to determine that it is consistent with the second voiceprint information of user A. Then, it can be determined that the user who speaks in the driving area is user A; then the emotion soothing strategy corresponding to the user emotion of user A is used to control the vehicle-mounted equipment in the driving area to release the emotion regulation method to soothe the emotions of user A in the driving area.

[0141] For another example, if there is a user in the co-pilot area, but no user in other areas, then the co-pilot area is the area where the user is located. The first voiceprint information emitted by the user in the co-pilot area can be collected and compared with multiple pre-recorded second voiceprint information to determine that it is consistent with the second voiceprint information of user B. Then, it can be determined that the user who spoke in the co-pilot area is user B, and then the emotion soothing strategy corresponding to the user emotion of user B can be used to control the vehicle-mounted equipment in the co-pilot area to release the emotion regulation method to soothe the emotions of user B in the co-pilot area.

[0142] Through the above technical solution, the area where users in multiple areas are located can be identified first; then the identity information of the users in the area where the users are located can be determined, and the corresponding emotional soothing strategy for the users can be determined based on the identity information of the users, so as to soothe the users in a targeted manner. By distinguishing the identities of the speaking users in different target areas, on the one hand, the multimodal emotion-related information of the distinguished different users can be input into the sub-multimodal fusion model to obtain the user emotions of different users, so as to determine the emotional states of different users; on the other hand, in response to the user emotions of different users, emotional soothing strategies suitable for users in different user areas can be adopted, soothing users in different user areas can be targeted, and the effectiveness of emotional soothing can be improved.

[0143] The following is an exemplary scheme of an emotion soothing strategy including a first emotion regulation content, which includes the following steps.

[0144] (1) Determine a first emotion regulation content that is triggered when the target user is in the user emotion during a historical time.

[0145] The first emotion regulation content is the regulation content released by the vehicle-mounted equipment to soothe the user's emotions. The emotion regulation content includes music, video, smell, light and dialogue. The music is played by a music player, the video is played by a video player, the smell is the fragrance or air freshener emitted by the odor dispensing device, the light is lit by the lighting device, and the dialogue can be a dialogue device, such as the voice emitted by a virtual elf for communicating with the user.

[0146] The first emotion regulation content is the emotion regulation content output by the vehicle-mounted device of the vehicle when the user in the area where the user is located is in a user emotion that triggers the vehicle-mounted device on the vehicle.

[0147] (2) Controlling the in-vehicle device in the area where the user is located to output the first emotion regulation content.

[0148] It is understandable that since the user is in different areas in the cabin during different time periods, there is no need to consider the area where the user is located when detecting the first emotion regulation content triggered by the user. What is obtained is the first emotion regulation content triggered when the user is in the user emotion; and when soothing the user's emotions, due to the differences in the areas where the user is located, in order to provide different first emotion regulation content to users in different areas in a targeted manner, the area where the user is located will be considered, and the vehicle-mounted equipment in the user's area will be controlled to release the first emotion regulation content.

[0149] For example, if user A located in the main driving area is in an angry mood during historical time and triggers the music player in the main driving area to play music A, it means that user A tends to play music A to relieve anger when he is angry in the past. At this time, there is no need to consider the area where the user is located; when the user identity in the rear seat area (non-main driving area) is identified as user A, and the multimodal fusion model calculates that the user emotion in the rear seat area is angry, the music player in the rear seat area is controlled to play music A to soothe the user emotion of user A. At this time, it is necessary to consider that the area where the user is located is the rear seat area.

[0150] For another example, if user B located in the co-pilot area is in an angry mood during the historical time, and the odor diffusion device in the main driving area is triggered to release a fresh fragrance, it means that user B tends to smell fresh air to relieve his anger when he is in an angry mood; when the user in the rear seat area is identified as user B, and the multimodal fusion model calculates that the user emotion in the rear seat area is angry, the odor diffusion device in the rear seat area will also be controlled to release a fresh fragrance to soothe the user's emotions.

[0151] For another example, if user C in the front passenger seat area was in a sad mood during the historical period and triggered the order of a funny video, it means that user C tends to divert his attention by watching funny videos when he is in a sad mood; when the user in the rear seat area is identified as user C, and the multimodal fusion model calculates that the user emotion in the rear seat area is sad, the headrest screen in the rear seat area will also be controlled to play a funny video to divert the user's attention and soothe the user's emotion.

[0152] For another example, if user D located in the front passenger seat area triggered the dialogue device when he was in a sad mood during the historical period, it means that user D tends to talk to the dialogue device to divert his attention when he is in a sad mood; when the user in the rear seat area is identified as user D, and the multimodal fusion model calculates that the user emotion in the rear seat area is sad, the dialogue device in the rear seat area will also be controlled to actively talk to user D to divert the user's attention and soothe the user's emotions.

[0153] Through the above technical solution, the first emotion regulation content triggered by the user in the past when he was in extreme emotions can be determined. When the same user is currently in the same extreme emotion, the vehicle-mounted device will be controlled to release the same first emotion regulation content to soothe the user's emotions, thereby improving the effectiveness of the emotion soothing strategy. On the one hand, not only the user's emotions are taken into account, but also the usage habits of different users for vehicle-mounted devices when regulating their emotions are taken into account, so that the released first emotion regulation content can be more in line with user habits and can more effectively soothe user's emotions; on the other hand, since users in different regions are distinguished, users in different regions can be targeted at soothing their emotions with the first emotion regulation content that is in line with user habits and user emotions, thereby reducing the impact on users in other regions.

[0154] The third emotion regulation content included in the emotional soothing strategy is introduced below, including the following steps.

[0155] (1) Determine the second emotion regulation content triggered by the target user in the historical time.

[0156] The second emotion regulation content is the emotion regulation content output by the on-board device of the vehicle when the user triggers the on-board device on the vehicle.

[0157] Different from the first emotion regulation content, which is the emotion regulation content released when the target user is in extreme emotions, the second emotion regulation content is the emotion regulation content that the target user has triggered in the past. For example, the first emotion regulation content can be the emotion regulation content that the target user has triggered when he was in anger in the past; the second emotion regulation content is all the emotion regulation content that the target user has triggered in the past.

[0158] (2) Filtering out third emotion regulation content from the second emotion regulation content, where the third emotion regulation content is emotion regulation content used to soothe the user's emotions.

[0159] The type of emotion regulation content that can soothe the user's emotions can be pre-configured in the vehicle system.

[0160] For example, with respect to music, soothing music can be pre-configured to soothe anger; quiet music can be used to soothe anxiety; cheerful music can be used to soothe sadness, and so on.

[0161] For videos, you can pre-configure funny videos to soothe sad emotions; cooking videos to soothe anxious emotions, etc.

[0162] As for the smell, you can configure grass-scented fragrance to soothe anger; lemon-scented fragrance to soothe fatigue, etc.; floral fragrance to soothe sadness, etc.

[0163] As for lighting, you can configure lights with a faster flashing frequency to soothe sad emotions; slowly changing lights can be used to soothe angry emotions or tired emotions, etc.

[0164] For dialogue, you can configure a relaxed and happy dialogue to soothe sad emotions; use storytelling dialogue to soothe tired emotions, etc.

[0165] The second emotion regulation content is emotion regulation content that conforms to the user's habits, and the third emotion regulation content selected from the second emotion regulation content is not only emotion regulation content that conforms to the user's habits, but also emotion regulation content that can soothe the user's current emotions.

[0166] For example, the second emotion regulation content triggered by user A in historical time includes videos and conversations. When the multimodal fusion model identifies that user A is in a sad mood, any one of the elements of funny videos in the video and light-hearted conversations in the conversation will be output to soothe the user's sad emotions. The funny videos and any one of the elements of light-hearted conversations in the conversation not only conform to the user's previous user habits, but also can soothe the user's sad emotions.

[0167] (3) Controlling the in-vehicle device in the area where the user is located to output the third emotion regulation content.

[0168] For example, if there are users in both the left and right rear seat areas of the rear seat area, then the left and right rear seat areas are the user areas, and the first voiceprint information emitted from the areas where the users are located in the left and right rear seat areas can be collected respectively, and the first voiceprint information of the two user areas can be compared with multiple second voiceprint information to determine that user A is in the left rear seat area and user B is in the right rear seat area.

[0169] If it is determined that the user emotion of user A is anger, and the second emotion regulation content previously triggered by the user includes music; if it is determined that the user emotion of user B is sadness, and the second emotion regulation content previously triggered by the user includes fragrance, the music player in the left rear seat area where user A is located will be controlled to play soothing music to appease the anger of user A, and the scent dispensing device in the right rear seat area where user B is located will be controlled to emit flower fragrance to appease the sadness of user B.

[0170] It can be understood that, in the above-mentioned first emotion regulation content and the second emotion regulation content, for the main driving area, at least one of music and smell will be used to regulate the user emotions in the main driving area, and video playback or strong sound and light changes cannot be used to regulate the user emotions in the main driving area, so as to avoid affecting the safe driving of the vehicle; for the emotion regulation content of the non-main driving area, including at least one of music, video, smell, light and dialogue, dialogue can be used preferentially to soothe the emotions of users in the back row and divert users' attention.

[0171] Through the above technical scheme, on the first hand, the second emotion regulation content that conforms to the user's habits can be determined first, and then the third emotion regulation content that can soothe the user's emotions can be screened out from the second emotion regulation content that conforms to the user's habits, so that the third emotion regulation content can not only soothe the user's emotions, but also conform to the user's habits, thereby improving the effectiveness of emotion soothing; on the second hand, since the second emotion regulation content that conforms to the user's habits may have multiple types, such as music, fragrance, conversation, etc., each time the user is in the same extreme emotion, the third emotion regulation content that can soothe the user's extreme emotion will be randomly selected from a variety of second emotion regulation contents. For example, the last time the user was in an angry mood, fragrance was used to soothe the user's emotions, and the next time the user is in an angry mood, music may be used to soothe the user's emotions. The randomness of the screened third emotion regulation content is relatively strong, thereby improving the user's freshness experience.

[0172] The following is an exemplary solution for obtaining facial image information. Fig. 9 As shown, the exemplary scheme is used to explain the processing of the passenger image captured by the image acquisition device to obtain facial image information with uniform brightness and darkness, including the following steps.

[0173] (1) Acquire an image of a first passenger in the vehicle.

[0174] The first passenger image in the vehicle may be acquired by an image acquisition device, where the first passenger image is an original passenger image acquired in the vehicle.

[0175] Optionally, the pixel values ​​in the first passenger image may be normalized to obtain a processed first passenger image.

[0176] (2) The brightness of the first passenger image is adjusted to obtain an adjusted second passenger image.

[0177] The brightness uniformity of the second passenger image is higher than that of the first passenger image. The first passenger image can be input into an image brightness processing model (Self-Calibrated Illumination, SCI) to obtain the second passenger image.

[0178] Among them, the image shading processing model can be trained through the following steps: multiple passenger image training sets can be input into the image shading processing model, and then the low-light enhancement performance output by the image shading processing model is determined; if the low-light enhancement performance does not meet the low-light enhancement standard, multiple passenger image training sets are re-input into the image shading processing model for training until the image shading processing model can output passenger images with high shading uniformity.

[0179] The low-light enhancement performance refers to the performance of increasing the brightness of dark areas in an image and reducing the brightness of bright areas in an image.

[0180] (3) Identifying facial image information of each user in the second passenger image.

[0181] The faces of the users in the second passenger image are recognized to obtain the facial image information of the users, so as to distinguish the users subsequently, thereby identifying the emotional states of the different users and soothing their emotions in a targeted manner.

[0182] At present, when a user is sitting in a car cabin, the brightness outside the car may be high while the brightness inside the car is low. At this time, the brightness of the first passenger image obtained is low, and it is difficult to accurately capture the user's facial image information in the first passenger image; it may also be low outside and high inside. At this time, the first passenger image obtained may be overexposed and unable to capture more detailed facial image information. Therefore, the brightness of the first passenger image can be adjusted, and the dark area in the first passenger image can be enhanced and the bright area can be darkened, so that the brightness of the image is more uniform, so as to obtain more facial image information.

[0183] Through the above technical solution, the brightness of the acquired first passenger image can be adjusted to obtain a second passenger image with more uniform brightness. Then, based on the second passenger image with more uniform brightness, more facial image information can be captured, thereby accurately distinguishing users in different areas and accurately identifying the facial emotional characteristics of different users.

[0184] Fig.10 1 is a block diagram of a vehicle user's emotion recognition device proposed by an exemplary embodiment of the present disclosure. The vehicle user's emotion recognition device 1000 includes: an emotion recognition module 1010 .

[0185] The emotion recognition module 1010 is configured to determine the user emotion of the user in the vehicle based on the multimodal emotion-related information corresponding to the vehicle, wherein the multimodal emotion-related information includes at least two emotion-related information associated with the user emotion, wherein at least one of the emotion-related information is emotion-related information corresponding to the area where the user is located.

[0186] Optionally, the multimodal emotion-related information includes at least two of the user's voice information, the user's facial image information, and the vehicle's driving information.

[0187] Optionally, the voice information is emotion-related information corresponding to the area where the user is located, and / or the face image information is emotion-related information corresponding to the area where the user is located.

[0188] Optionally, the user emotion is determined based on a plurality of emotion features obtained according to the multimodal emotion-related information, the plurality of emotion features comprising at least two of a language emotion feature, a fatigue detection feature, a facial emotion feature, and a driving emotion feature;

[0189] The language emotion feature is determined based on the voice information; the fatigue detection feature and / or the facial emotion feature is determined based on the face image information; and the driving emotion feature is determined based on the driving information.

[0190] Optionally, the fatigue detection feature includes a feature for characterizing the degree of opening or closing of the user's eyes.

[0191] Optionally, the user emotion is determined based on multiple fusion features, and the multiple fusion features are obtained by fusing the multiple emotion features with latent features based on a cross-attention mechanism, and the latent features are obtained by fusing the multiple emotion features through low-nodality.

[0192] Optionally, the user emotion is determined based on a multimodal feature obtained by concatenating the multiple fusion features.

[0193] Optionally, the user emotion is obtained after a first convolutional layer processes the multimodal feature; wherein the first convolutional layer is trained by associated multimodal feature samples and user emotion samples.

[0194] Optionally, the voice information is determined from first voice information collected from an area where the user is located, and the area where the user is located is determined based on the facial image information.

[0195] Optionally, the vehicle user's emotion recognition device 1000 includes:

[0196] The soothing module is configured to control the vehicle-mounted equipment on the vehicle to soothe the user using an emotion soothing strategy corresponding to the user's emotion.

[0197] Optionally, the vehicle-mounted device includes a vehicle-mounted device corresponding to an area where the user is located.

[0198] Optionally, the emotion soothing strategy is determined based on one or more of the user's emotion, the user's identity information, and the area where the user is located, and the user identity information is determined based on the first voiceprint information in the voice information and / or the first facial information in the facial image information.

[0199] Optionally, the user identity information is the identity information corresponding to the target voiceprint information that matches the first voiceprint information among multiple pre-stored second voiceprint information; and / or, the user identity information is the identity information corresponding to the target facial information that matches the first facial information among multiple pre-stored second facial information.

[0200] Optionally, the emotion soothing strategy includes an emotion soothing method corresponding to the user emotion, and the emotion soothing method is determined based on one or more of the user emotion, the user identity information, and the area where the user is located. The emotion soothing method includes at least one of a music soothing method, a video soothing method, an aroma soothing method, a conversation soothing method, and a light soothing method.

[0201] Optionally, the area where the user is located is a driving area or a non-driving area;

[0202] The emotional soothing method matched to the driving area includes at least one of a music soothing method and a smell soothing method;

[0203] The emotion soothing method matched in the non-driving area includes at least one of a music soothing method, a video soothing method, a smell soothing method, a light soothing method and a conversation soothing method.

[0204] Optionally, the emotion soothing strategy also includes emotion regulation content that the vehicle-mounted device needs to output when a corresponding emotion soothing method is adopted.

[0205] Optionally, the emotion regulation content includes first emotion regulation content that is triggered to be output by an onboard device on the vehicle when the user is in the user emotion within a historical time and adopts a corresponding emotion soothing method.

[0206] Optionally, the emotion regulation content includes third emotion regulation content that is filtered out from second emotion regulation content and can soothe the user's emotions, and the second emotion regulation content includes emotion regulation content that is triggered by the user within a historical time period and output by an onboard device on the vehicle.

[0207] Optionally, the facial image information is obtained by identifying a second passenger image of the vehicle, and the second passenger image is obtained by adjusting the brightness of the first passenger image; the brightness uniformity of the second passenger image is higher than that of the first passenger image.

[0208] Optionally, the second passenger image is obtained by performing shading adjustment on the first passenger image using an image shading processing model.

[0209] Based on the same inventive concept, the present disclosure also provides an electronic device, Fig.11 1 is a block diagram of an electronic device 1100 according to an exemplary embodiment. Fig.11 As shown, the electronic device 1100 may include: a processor 1101 , a memory 1102 . The electronic device 1100 may also include one or more of a multimedia component 1103 , an input / output (I / O) interface 1104 , and a communication component 1105 .

[0210] The processor 1101 is used to control the overall operation of the electronic device 1100 to complete all or part of the steps in the above-mentioned vehicle user emotion recognition method. The memory 1102 is used to store various types of data to support the operation of the electronic device 1100, and these data may include, for example, instructions for any application or method used to operate on the electronic device 1100, and application-related data, such as contact data, sent and received messages, pictures, audio, video, etc. The memory 1102 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, magnetic disk or optical disk. The multimedia component 1103 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone, which is used to receive external audio signals. The received audio signal may be further stored in the memory 1102 or sent through the communication component 1105. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 1104 provides an interface between the processor 1101 and other interface modules, and the above-mentioned other interface modules may be keyboards, mice, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 1105 is used for wired or wireless communication between the electronic device 1100 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 1105 may include: Wi-Fi module, Bluetooth module, NFC module.

[0211] In an exemplary embodiment, the electronic device 1100 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned vehicle user emotion recognition method.

[0212] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned vehicle user emotion recognition method are implemented. For example, the computer-readable storage medium may be the above-mentioned memory 1102 including program instructions, and the above-mentioned program instructions may be executed by the processor 1101 of the electronic device 1100 to complete the above-mentioned vehicle user emotion recognition method.

[0213] For example, see Figure 8 As shown, the electronic device may be Figure 8 The emotion monitoring device in the embodiment can also be a chip capable of running an AI model, which is used to receive driving information, facial image information and voice information, and analyze at least two of the three to obtain user emotions.

[0214] Based on the same inventive concept, the present disclosure also proposes an emotion recognition system for vehicle users, including the electronic device proposed above, and / or on-board equipment on the vehicle. The electronic device is used to control the on-board equipment on the vehicle to release an emotion regulation method that is consistent with the user's habits and emotions, so as to more effectively soothe the user's emotions.

[0215] Optionally, see Figure 8 As shown, the vehicle user emotion recognition system proposed in the present disclosure includes an information collection device, an emotion monitoring device, an emotion soothing device (vehicle-mounted equipment) and a vehicle control device.

[0216] Among them, the information acquisition device includes an image acquisition device, an audio acquisition device and a vehicle status sensor. The image acquisition device can be a camera installed in the cockpit to collect image information. The audio acquisition device can be a microphone installed in the cockpit to collect voice information. The vehicle status sensor can be used to read vehicle speed, acceleration and other information.

[0217] The emotion monitoring device is used to receive the driving information, face image information and voice information output by the information collection device, and analyze the three to obtain the user's emotion; then send a control signal to the bus to soothe the user's emotion according to the user's emotion.

[0218] The emotion soothing device includes a music player, an ambient light, a display screen, a central control screen, a conversation device, and a scent dispensing device, etc., which is used to receive control signals from the bus and respond to the control signals to play music, display lights, play videos, have conversations, and distribute fragrance, etc.

[0219] The vehicle control device is used to receive a control signal from the bus and control the vehicle to drive smoothly based on the control signal, thereby reducing the risk of dangerous driving caused by extreme emotions of the user.

[0220] Based on the same inventive concept, please refer to Fig.12 As shown, the present disclosure also proposes a vehicle, which is equipped with the above-mentioned electronic device or the above-mentioned vehicle user's emotion recognition system.

[0221] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0222] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0223] In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A method for identifying emotions of vehicle users, It is characterized in that include: Determining a user emotion of a user in the vehicle according to multimodal emotion-related information corresponding to the vehicle, wherein the multimodal emotion-related information includes at least two emotion-related information associated with the user emotion, wherein at least one of the emotion-related information is emotion-related information corresponding to a region where the user is located; The user emotion is determined based on a multimodal feature obtained by splicing multiple fusion features, wherein the multiple fusion features are obtained by fusing multiple emotion features with latent features based on a cross-attention mechanism, and the latent features are obtained by fusing the multiple emotion features through low-order fusion; the multiple emotion features include language emotion features, fatigue detection features, facial emotion features, and driving emotion features; the multiple emotion features are obtained based on multimodal emotion-related information, and the multimodal emotion-related information includes the user's voice information, the user's face image information, and the vehicle's driving information; Among them, the language emotion feature is determined based on the voice information; the fatigue detection feature and / or the facial emotion feature is determined based on the face image information; and the driving emotion feature is determined based on the driving information.

2. The method according to claim 1, It is characterized in that The voice information is emotion-related information corresponding to the area where the user is located, and / or the face image information is emotion-related information corresponding to the area where the user is located.

3. The method according to claim 1, It is characterized in that The fatigue detection feature includes a feature for characterizing the degree of eye opening or closing of the user.

4. The method according to claim 1, It is characterized in that The user emotion is obtained after the first convolution layer processes the multimodal feature; wherein the first convolution layer is trained by associated multimodal feature samples and user emotion samples.

5. The method according to claim 1, It is characterized in that The voice information is determined from first voice information collected from an area where the user is located, and the area where the user is located is determined based on the facial image information.

6. The method according to claim 1, It is characterized in that The method further comprises: Using an emotion soothing strategy corresponding to the user's emotion, the on-board equipment on the vehicle is controlled to soothe the user.

7. The method according to claim 6, It is characterized in that The vehicle-mounted equipment includes the vehicle-mounted equipment corresponding to the area where the user is located.

8. The method according to claim 6, It is characterized in that The emotion soothing strategy is determined based on one or more of the user's emotion, the user's identity information, and the area where the user is located, and the user identity information is determined based on the first voiceprint information in the voice information and / or the first face information in the face image information.

9. The method according to claim 8, It is characterized in that The user identity information is the identity information corresponding to the target voiceprint information that matches the first voiceprint information among multiple pre-stored second voiceprint information; and / or, the user identity information is the identity information corresponding to the target facial information that matches the first facial information among multiple pre-stored second facial information.

10. The method according to claim 6, It is characterized in that The emotion soothing strategy includes an emotion soothing method corresponding to the user's emotion, and the emotion soothing method is determined based on one or more of the user's emotion, user identity information, and the area where the user is located. The emotion soothing method includes at least one of a music soothing method, a video soothing method, an aroma soothing method, a conversation soothing method, and a light soothing method.

11. The method according to claim 10, It is characterized in that The area where the user is located is a driving area or a non-driving area; The emotional soothing method matched to the driving area includes at least one of a music soothing method and a smell soothing method; The emotion soothing method matched in the non-driving area includes at least one of a music soothing method, a video soothing method, a smell soothing method, a light soothing method and a conversation soothing method.

12. The method according to claim 10, It is characterized in that The emotion soothing strategy also includes emotion regulation content that the vehicle-mounted device needs to output when using a corresponding emotion soothing method.

13. The method according to claim 12, It is characterized in that The emotion regulation content includes first emotion regulation content that is triggered to be output by an onboard device on the vehicle when the user is in the user emotion within a historical time and adopts a corresponding emotion soothing method.

14. The method according to claim 12, It is characterized in that The emotion regulation content includes third emotion regulation content that is selected from second emotion regulation content and can soothe the user's emotions, and the second emotion regulation content includes emotion regulation content that is triggered by the user to output by an onboard device on the vehicle within a historical time.

15. The method according to claim 1, It is characterized in that The facial image information is obtained by identifying the second passenger image of the vehicle, and the second passenger image is obtained by adjusting the brightness of the first passenger image; the brightness uniformity of the second passenger image is higher than that of the first passenger image.

16. The method according to claim 15, It is characterized in that The second passenger image is obtained by performing shading adjustment on the first passenger image using an image shading processing model.

17. An electronic device, It is characterized in that The method comprises a processor connected to a memory, wherein the processor is used to execute a computer program in the memory to implement the steps of the method according to any one of claims 1 to 16.

18. A vehicle user's emotion recognition system, It is characterized in that An electronic device comprising the electronic device of claim 17.

19. A vehicle, It is characterized in that An emotion recognition system comprising the electronic device of claim 17 or the vehicle user of claim 18.

20. A computer-readable storage medium, It is characterized in that A computer program is stored thereon, characterized in that when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 16 are implemented.

Images