In-vehicle user emotion recognition method and device, vehicle and medium

Abstract

The invention provides an in-vehicle user emotion recognition method and device, a vehicle and a medium. The method comprises the steps that initial emotion features of M users in the vehicle are acquired; according to the initial emotion feature of the target user and the initial emotion features of the N other users, determining an initial emotion influence weight of each of the N other users on the target user; and determining individual emotion probability distribution of the target user according to the initial emotion feature of each other user in the N other users and the initial emotion influence weight of each other user on the target user. In the embodiment of the invention, the emotion influence weight of other users on the target user is determined according to the emotion characteristics of the target user and the other users, and then the individual emotion probability distribution of the target user is determined according to the emotion characteristics of the other users and the emotion influence weight of the target user, namely the influence of the other users on the emotion. The accuracy of the recognition result and the scene adaptability are improved, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of in-vehicle intelligent interaction technology, and in particular to an in-vehicle user emotion recognition method, device, vehicle and medium. Background Technology

[0002] With the development of automotive intelligence, in-vehicle intelligent interaction technology has received widespread attention. In-vehicle intelligent interaction systems typically provide personalized services by sensing the state of passengers, and emotion recognition is an important way to achieve intelligent interaction.

[0003] During vehicle operation, it is possible to collect emotional expression signals such as facial expressions, language and body movements of users inside the vehicle, extract emotional features from the user's emotional expression signals, identify the user's emotional type based on the emotional features, and then provide interactive feedback that meets the user's needs based on the user's emotional type.

[0004] However, the emotional states of multiple users often influence each other. Emotional recognition based on the emotional characteristics of a single user ignores the influence of the emotional states of other users, resulting in insufficient accuracy and scenario adaptability of the recognition results, which affects the user experience.

[0005] It should be noted that the information disclosed in the background section of this application is intended only to enhance the understanding of the general background of this application, and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0006] This application provides a method, device, vehicle, and medium for recognizing user emotions in a vehicle, which helps to solve the problems of insufficient accuracy and scene adaptability of recognition results and the impact on user experience caused by ignoring the influence of other users' emotions.

[0007] In a first aspect, embodiments of this application provide an in-vehicle user emotion recognition method, including: Obtain the initial emotional features of M users inside the vehicle, where M≥2; Based on the initial emotional characteristics of the target user and the initial emotional characteristics of N other users, determine the initial emotional influence weight of each of the N other users on the target user. The target user is any user among the M users, and the other users are users other than the target user among the M users, where N≥1. Based on the initial emotional characteristics of each of the N other users and the initial emotional influence weight of each other user on the target user, the individual emotional probability distribution of the target user is determined. The individual emotional probability distribution of the target user is used to characterize the probability of the target user corresponding to each emotional type.

[0008] In one possible implementation, after determining the individual emotional probability distribution of the target user, the method further includes: Based on the initial emotional characteristics of each of the M users, and the influence weights of the other users on the initial emotions of each user, the overall emotional probability distribution inside the vehicle is determined. The overall emotional probability distribution is used to characterize the probability of each emotional type in the vehicle as a whole.

[0009] In one possible implementation, determining the individual emotional probability distribution of the target user based on the initial emotional characteristics of each of the N other users and the initial emotional influence weight of each other user on the target user includes: Based on the initial emotional characteristics of each of the N other users and the initial emotional influence weight of each other user on the target user, the current layer emotional characteristics of the target user are determined; Based on the current emotional characteristics of the target user and the current emotional characteristics of N other users, determine the influence weight of each of the N other users on the current emotional characteristics of the target user. Based on the current layer sentiment features of each of the N other users and the weight of each other user’s influence on the current layer sentiment of the target user, determine the next layer sentiment features of the target user; Based on the next layer of emotional characteristics of the target user, determine the individual emotional probability distribution of the target user.

[0010] In one possible implementation, determining the influence weight of each of the N other users on the current emotional state of the target user, based on the current emotional state characteristics of the target user and the current emotional state characteristics of N other users, includes: According to the formula: Determine the current-layer sentiment influence weight of any one of the N other users on the target user; Among them, h i L h represents the current layer of sentiment characteristics for the target user. j L For any other user's current layer sentiment feature, e ij L Let W1 be the weight of the current layer sentiment influence of any other user on the target user, a be the first weight matrix, a be the influence vector, LeakyReLU be the first activation function, and contact be the feature concatenation function.

[0011] In one possible implementation, determining the next-layer sentiment characteristics of the target user based on the current-layer sentiment characteristics of each of the N other users and the influence weight of each other user on the current-layer sentiment of the target user includes: According to the formula: Determine the next layer of emotional characteristics for the target user; Among them, h j L For any other user's current layer sentiment feature, e ij L W1 represents the weight of the current layer sentiment influence of any other user on the target user, and W2 represents the second weight matrix. i L+1 ELU is the next layer of emotional features for the target user, and ELU is the second activation function.

[0012] In one possible implementation, after determining the individual emotional probability distribution of the target user, the method further includes: If the distribution entropy of the individual emotional probability distribution of the target user is greater than or equal to the first preset distribution entropy threshold, then distribution entropy feedback compensation is obtained. Based on distribution entropy feedback compensation, the initial emotional characteristics of the target user, and the initial emotional characteristics of N other users, determine the emotional impact weight of each of the N other users on the target user. Based on the initial emotional characteristics of each of the N other users and the emotional influence weight of each other user on the target user's feedback, the individual emotional probability distribution of the target user's feedback is determined.

[0013] In one possible implementation, obtaining the initial emotional features of M users inside the vehicle includes: By performing face recognition processing, scaling processing, rotation processing, translation processing and / or illumination normalization processing on single-frame acquired images, single-frame face images of M users inside the vehicle are determined; The initial emotional features of the M users in the vehicle are obtained by extracting emotional features from at least one frame of facial images of the M users within a preset time period.

[0014] Secondly, embodiments of this application also provide a passenger-to-passenger interaction emotion recognition device, comprising: The emotion feature acquisition module is used to acquire the initial emotion features of M users in the vehicle, where M≥2; The emotional influence weight determination module is used to determine the initial emotional influence weight of each of the N other users on the target user based on the initial emotional characteristics of the target user and the initial emotional characteristics of N other users. The target user is any user among the M users, and the other users are users other than the target user among the M users, where N≥1. The emotion probability distribution determination module is used to determine the individual emotion probability distribution of the target user based on the initial emotion characteristics of each of the N other users and the initial emotion influence weight of each other user on the target user. The individual emotion probability distribution of the target user is used to characterize the probability of the target user corresponding to each emotion type.

[0015] Thirdly, embodiments of this application provide a vehicle, including: A controller, wherein the controller is configured to be used in any of the first aspects of the method.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a controller, implements the method described in any one of the first aspects.

[0017] In this embodiment, the emotional influence weight of other users on the target user is determined based on the emotional characteristics of the target user and other users. Then, based on the emotional characteristics of other users and their emotional influence weight on the target user, the individual emotional probability distribution of the target user is determined. When identifying the target user's emotions, the influence of other users' emotions is taken into account, which improves the accuracy and scene adaptability of the identification results and enhances the user experience. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a structural diagram illustrating an application scenario provided in an embodiment of this application. Figure 2 A flowchart illustrating an in-vehicle user emotion recognition method provided in an embodiment of this application; Figure 3 A flowchart illustrating another in-vehicle user emotion recognition method provided in an embodiment of this application; Figure 4A flowchart illustrating another in-vehicle user emotion recognition method provided in an embodiment of this application; Figure 5 A flowchart illustrating another in-vehicle user emotion recognition method provided in an embodiment of this application; Figure 6 A flowchart illustrating another in-vehicle user emotion recognition method provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an in-vehicle user emotion recognition device provided in an embodiment of this application; Figure 8 This is a structural schematic diagram of a vehicle provided in an embodiment of this application. Detailed Implementation

[0020] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0021] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0022] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0023] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0024] With the development of automotive intelligence, in-vehicle intelligent interaction technology has received widespread attention. In-vehicle intelligent interaction systems typically provide personalized services by sensing the state of passengers, and emotion recognition is an important way to achieve intelligent interaction.

[0025] See Figure 1 This is a structural diagram illustrating an application scenario provided in an embodiment of this application. Figure 1 As shown, the vehicle 100 includes a sensor 101, a controller 102, and an actuator 103.

[0026] During the operation of vehicle 100, sensor 101 can collect emotional expression signals such as facial expressions, language and body movements of users in the vehicle. Controller 102 can extract emotional features from the user's emotional expression signals, identify the user's emotional type based on the user's emotional features, and then provide interactive feedback that matches the user's emotional type through actuator 103.

[0027] Among them, the sensor 101 includes, but is not limited to, image acquisition devices and audio acquisition devices; the controller 102 includes, but is not limited to, microcontroller units (MCUs) and system-on-chips (SOCs); and the actuator 103 includes, but is not limited to, audio playback devices, visual display devices, or environmental control devices.

[0028] Specifically, the image acquisition device can be a camera deployed inside the vehicle cabin, including a high-definition camera or an infrared camera suitable for low-light environments, used to capture the user's facial expressions and body movements; the audio acquisition device can be a microphone array used to collect the user's voice signals; of course, multiple or more sensors can be selected to jointly acquire the user's emotional expression signals according to actual needs, for example, multiple cameras at different angles can be set in multiple locations inside the vehicle cabin, such as in the roof or rearview mirror, to collect the emotional expression signals of the front-seat users and the rear-seat users respectively, or cameras and microphone arrays can be used simultaneously to acquire the user's emotional expression signals, etc.

[0029] Furthermore, audio playback devices can be speaker units of a vehicle audio system, used to play specific types of music, provide voice prompts, or offer sound feedback; visual display devices can include a central control screen, instrument panel, or head-up display, capable of adjusting interface colors, pushing visual content, or displaying customized information; environmental control devices can be air conditioning systems, ambient lighting, seat vibration motors, etc. Of course, these actuators can operate independently or work in conjunction to provide interactive feedback tailored to the user's emotional type.

[0030] However, the emotional states of multiple users often influence each other. Emotional recognition based on the emotional characteristics of a single user ignores the influence of the emotional states of other users, resulting in insufficient accuracy and scenario adaptability of the recognition results, which affects the user experience.

[0031] To address the aforementioned issues, this application provides an in-vehicle user emotion recognition method, which helps to solve the problems of insufficient accuracy and scene adaptability of recognition results and the impact on user experience caused by ignoring the influence of other users' emotions.

[0032] See Figure 2 This is a flowchart illustrating an in-vehicle user emotion recognition method provided in an embodiment of this application. It can be applied to... Figure 1The application scenarios shown are as follows: Figure 2 As shown, it mainly includes the following steps.

[0033] S201: Obtain the initial emotional features of M users in the car, where M≥2.

[0034] Understandably, during vehicle operation, it's possible to determine a single user's initial emotional characteristics. Specifically, sensors can acquire the user's emotional expression signals, and a pre-defined feature extraction algorithm is typically used to identify and quantify features related to the user's emotional state within these signals, thereby determining the user's initial emotional characteristics. A single user's emotional state is usually transformed into a structured data representation, such as a numerical vector, which represents the user's initial emotional characteristics.

[0035] Furthermore, the vehicle can also determine the initial emotional characteristics of multiple users. In this embodiment, the emotional expression signals of at least two users can be extracted to determine the initial emotional characteristics corresponding to different users, i.e., to obtain the initial emotional characteristics of M users in the vehicle, where M≥2. It should be noted that the M users in the vehicle can be all users in the vehicle, or a subset of users selected according to preset rules, i.e., some users. For example, the user subset can be selected based on the location and number of users in the vehicle, or according to the needs of the interaction scenario, namely the driver and the passenger. Of course, those skilled in the art can also select other preset conditions to determine the M users according to actual needs, and this embodiment does not impose specific limitations on this.

[0036] S202: Based on the initial emotional characteristics of the target user and the initial emotional characteristics of N other users, determine the initial emotional influence weight of each of the N other users on the target user.

[0037] Wherein, the target user is any one of the M users, and the other users are users other than the target user among the M users, and N≥1.

[0038] Understandably, for any given other user, the initial emotional influence weight of that other user on the target user can be determined by the correlation between the initial emotional characteristics of the target user and the other user. This initial emotional influence weight can be a numerical value used to quantify the degree to which the emotional state of that other user influences the emotional state of the target user.

[0039] In this embodiment, the initial emotional influence weight of a specific other user on the target user can be determined by the feature similarity between the initial emotional features of the target user and the specific other user. Specifically, the degree of similarity between the emotional states of the target user and the specific other user can be assessed by calculating the similarity between their initial emotional feature vectors. For example, this can be achieved using the formula: The cosine similarity between the initial sentiment features of two users is determined and used as the weight for the initial sentiment influence of other users on the target user. Where h i 0 h represents the initial emotional characteristics of the target users. j 0 Let S(i,j) be the initial sentiment characteristics of a specific other user, and S(i,j) be the cosine similarity between the initial sentiment characteristics of the two users. Generally speaking, the higher the feature similarity, the greater the weight of the sentiment influence of that specific other user on the target user.

[0040] Of course, those skilled in the art can also use other reasonable correlation analysis methods to determine the initial emotional influence weights, such as learning the interaction relationship between two features and outputting weight values ​​through a pre-trained neural network model, or determining the influence intensity based on the relative positional relationship of feature vectors in a preset emotional space, etc. The embodiments of this application do not impose specific limitations on this.

[0041] Furthermore, for multiple other users, the initial emotional influence weight of each of the N other users on the target user can be determined based on the initial emotional characteristics of the target user and the initial emotional characteristics of the N other users. The N other users can be all users in the vehicle other than the target user, or they can be a subset of other users selected based on specific conditions or randomly.

[0042] S203: Determine the individual emotional probability distribution of the target user based on the initial emotional characteristics of each of the N other users and the initial emotional influence weight of each other user on the target user.

[0043] The individual emotional probability distribution of the target user is used to characterize the probability of the target user corresponding to each emotional type.

[0044] It should be noted that the initial sentiment characteristics of each other user correspond to their initial sentiment influence weights on the target user. The initial sentiment influence weights of each other user can be normalized so that the sum of all weights is one, thereby accurately obtaining the individual sentiment probability distribution of the target user.

[0045] Understandably, the individual sentiment probability distribution of the target user can also be determined based on the initial sentiment characteristics of each of the N other users and the normalized initial sentiment influence weight of each other user on the target user. A weighted aggregation method is typically used to determine the individual sentiment probability distribution of the target user.

[0046] Of course, those skilled in the art can also choose other methods according to the actual situation to determine the individual emotional probability distribution of the target user through the initial emotional characteristics of other users and the corresponding initial emotional influence weights. For example, the initial emotional characteristics of the target user can be used and corresponding weights can be assigned to them; or a graph neural network structure, such as a two-layer graph attention network, can be used to consider the emotional characteristics of all users and their mutual influence relationships through the message passing mechanism between nodes, and finally output the individual emotional probability distribution of the target user.

[0047] Furthermore, the individual emotion probability distribution of a target user typically refers to a set of probability values, where each probability value corresponds to a preset emotion type. The sum of all probability values ​​is usually one, used to characterize the likelihood that the target user belongs to each emotion type. The individual emotion type of a target user can also be determined based on the individual emotion probability distribution. For example, the emotion type with the highest probability value can be selected as the final identification result, or a probability threshold can be set to filter the possible emotion type set.

[0048] In this embodiment, corresponding interactive feedback can be provided based on the individual emotional probability distribution or emotional type of the target user. A privacy protection module can also be included; for example, explicit privacy settings options can be provided, allowing users to choose the degree of participation in emotion recognition and the data retention period. All emotional expression signal collection and emotion recognition are typically completed locally to enhance user trust.

[0049] In one possible implementation, the initial emotional characteristics of all users in the vehicle can be obtained. Based on the initial emotional characteristics of each user in the vehicle and the initial emotional characteristics of all other users in the vehicle, the initial emotional influence weight of each other user on the target user can be determined. Based on the initial emotional characteristics of each other user and the initial emotional influence weight of each other user on the target user, the individual emotional probability distribution of the target user can be determined.

[0050] Specifically, all M users in the vehicle, or each user among all users in the vehicle, can be considered a node, and the node features are the user's initial emotional features; the initial emotional influence weights can be represented by the edges connecting the nodes, and can also be represented by an M... The adjacency matrix e of M, where the matrix elements e ijDescribe the initial emotional influence weight of user j on user i.

[0051] See Figure 3 This is a flowchart illustrating another in-vehicle user emotion recognition method provided in an embodiment of this application. Figure 3 As shown, in Figure 2 Based on the method embodiment shown, the following steps are further included after step S203.

[0052] S301: Determine the overall emotional probability distribution inside the vehicle based on the initial emotional characteristics of each of the M users and the influence weights of the other users on the initial emotions of each user.

[0053] The overall emotion probability distribution is used to characterize the probability of each emotion type in the vehicle interior.

[0054] Understandably, the individual emotional probability distribution of each of the M users can also be determined based on the initial emotional characteristics of each user among the M users, and the influence weights of the other users on each user's initial emotions. This can be referred to... Figure 2 The method embodiments shown are not described in detail here for the sake of brevity.

[0055] Furthermore, the overall emotional probability distribution within the vehicle can be determined based on the individual emotional probability distribution of each of the M users. Of course, those skilled in the art can also choose other methods to determine the overall emotional probability distribution within the vehicle based on each user's initial emotional characteristics and the influence weights of other users' initial emotions on that user, depending on the actual situation. For example, based on each user's initial emotional characteristics and the influence weights of other users' initial emotions on that user, the output emotional characteristics of each user can be determined. These output emotional characteristics can then be processed through global average pooling and a fully connected layer to obtain the overall emotional characteristics, thereby determining the overall emotional probability distribution.

[0056] Similarly, the overall emotion probability distribution is typically a set of probability values, where each probability value corresponds to a preset emotion type. The sum of all probability values ​​is usually one, representing the likelihood that the in-vehicle users as a whole belong to each emotion type. The overall emotion type of the in-vehicle users can also be determined based on the overall emotion probability distribution.

[0057] See Figure 4 This is a flowchart illustrating another in-vehicle user emotion recognition method provided in an embodiment of this application. Figure 4 As shown, in Figure 2 Based on the method embodiment shown, step S203 specifically includes the following steps.

[0058] S401: Determine the current layer emotional characteristics of the target user based on the initial emotional characteristics of each of the N other users and the initial emotional influence weight of each other user on the target user.

[0059] In practical applications, the weighted sentiment characteristics of other users towards the target user can be determined based on their initial sentiment characteristics and the initial sentiment influence weights of those other users. Furthermore, the current-layer sentiment characteristics of the target user are determined based on the weighted sentiment characteristics of all N other users. Of course, when multiple other users exist, the initial sentiment influence weights can be normalized first.

[0060] Similar to the initial sentiment features, the current layer sentiment features are typically represented as vectors with the same dimensions as the initial sentiment features. However, they not only include the target user's own sentiment information but also incorporate the sentiment influence of other users. These current layer sentiment features can serve as input features for determining the individual sentiment probability distribution of the target user in subsequent steps.

[0061] S402: Based on the current emotional characteristics of the target user and the current emotional characteristics of N other users, determine the influence weight of each of the N other users on the current emotional characteristics of the target user.

[0062] It is understandable that, similar to determining the current emotional characteristics of the target user, the current emotional characteristics of each of the N other users can be determined. That is, each of the other users can be used as the target user, and steps S202 and S401 can be repeated to determine the current emotional characteristics of each of the N other users.

[0063] Furthermore, for any specific other user, the influence weight of that other user on the target user's current emotional level is typically determined based on the target user's current emotional level characteristics and the current emotional level characteristics of that specific other user. In this way, the influence weight of each of the N other users on the target user's current emotional level can be determined based on the target user's current emotional level characteristics and the current emotional level characteristics of N other users.

[0064] In one possible implementation, according to the formula: Determine the current-layer sentiment influence weight of any one of the N other users on the target user; Among them, h i L h represents the current layer of sentiment characteristics for the target user. j L For any other user's current layer sentiment feature, e ij LLet W1 be the weight of the current layer sentiment influence of any other user on the target user, a be the first weight matrix, a be the influence vector, LeakyReLU be the first activation function, and contact be the feature concatenation function.

[0065] Specifically, for any other user, the current layer sentiment features of the target user can be concatenated with the current layer sentiment features of that other user to form a combined feature vector. Subsequently, this combined feature vector undergoes a linear transformation and is processed by an activation function, finally mapped to a scalar value. The first weight matrix W1 is a trainable weight matrix with dimensions matching the combined feature vector. A linear transformation is performed on the combined feature vector using the first weight matrix W1, and a LeakyReLU activation function is applied to the transformed features to prevent complete neuron deactivation while maintaining the non-linear transformation. The influence vector a is a trainable vector with dimensions matching the linearly transformed features. The activated features are mapped to a scalar score, e. ij L This refers to the influence weight of other users on the current layer of the target user's sentiment, reflecting the degree of influence of other users j on the target user i after considering the complex interaction relationships between features.

[0066] S403: Determine the next layer of sentiment features for the target user based on the current layer sentiment features of each of the N other users and the influence weight of each other user on the current layer sentiment of the target user.

[0067] Similar to step S401, in practical applications, the weighted sentiment features of other users on the target user can be determined based on their current sentiment features and the weights of their influence on the target user's current sentiment features. Further, the next-layer sentiment features of the target user are determined based on the weighted sentiment features of all N other users. Of course, in cases with multiple other users, the weights of each current-layer sentiment influence can be normalized first. The next-layer sentiment features are typically represented as a vector with the same dimensions as the current-layer sentiment features and the initial sentiment features.

[0068] In one possible implementation, according to the formula: Determine the next layer of emotional characteristics for the target user; Among them, h j L For any other user's current layer sentiment feature, e ij L Let W1 be the weight of the current layer sentiment influence of any other user on the target user, and W2 be the second weight matrix. i L+1ELU is the next layer of emotional features for the target user, and ELU is the second activation function.

[0069] Specifically, the current-layer sentiment features of each other user are linearly transformed using the second weight matrix W2, and then the transformed features are assigned the corresponding current-layer sentiment influence weight e. ij L After weighting, the next layer of sentiment feature representation h for the target user is obtained by summing all weighted features and applying the ELU activation function. i (L+1) .

[0070] S404: Determine the individual emotional probability distribution of the target user based on the next layer of emotional characteristics of the target user.

[0071] In practical applications, the next layer of sentiment features of the target user can be used as input and processed by a sentiment classifier. This sentiment classifier typically includes a fully connected layer and a normalization function. The fully connected layer maps the sentiment features to a dimension equal to the number of sentiment types, and the normalization function converts this mapping result into probability values.

[0072] In one specific implementation, the normalization function can be the softmax function. Through the softmax function, the system outputs a set of probability values, where each probability value corresponds to a preset sentiment type, and the sum of all probability values ​​is one. This set of probability values ​​constitutes the individual sentiment probability distribution of the target user.

[0073] Of course, those skilled in the art can also use other applicable classifier structures or normalization methods to determine the probability distribution of individual emotions. For example, a nonlinear classifier combining multiple fully connected layers can be used, or a sigmoid function can be used to handle multi-label emotion classification scenarios. This application does not impose specific limitations on these methods.

[0074] It should be noted that steps S401 to S403 in this embodiment are only one layer of emotional feature processing, and a second or even more layers of emotional feature processing can be set.

[0075] In other words, determining the individual emotional probability distribution of the target user based on the next layer of emotional characteristics can include the following specific steps.

[0076] Based on the next-level emotional characteristics of the target user and the next-level emotional characteristics of N other users, determine the weight of the influence of each of the N other users on the next-level emotional characteristics of the target user. Based on the next-level sentiment features of each of the N other users and the weight of each other user’s influence on the next-level sentiment of the target user, the next-level sentiment features of the target user are determined. Based on the next layer of emotional characteristics of the target user, determine the individual emotional probability distribution of the target user.

[0077] In this way, in a multi-layered structure, the processing method of each layer is similar to that of a single-layer processing unit, but its input features use the output results of the previous layer. Specifically, the sentiment features of the next layer obtained from the processing of the first layer are used as the input of the current layer sentiment features for the processing of the second layer, and so on. Through this hierarchical processing, sentiment features are propagated and interacted with among users in multiple rounds.

[0078] Understandably, modeling the emotional relationships between users is achieved through iterative computation using a multi-layered structure. The total number of layers, L, can be set according to the specific application requirements. For example, L can be set to an integer between 2 and 4 to strike a balance between model complexity and computational efficiency. The choice of the number of layers is typically related to the number of users in the vehicle and the complexity of their emotional interactions.

[0079] See Figure 5 This is a flowchart illustrating another in-vehicle user emotion recognition method provided in an embodiment of this application. Figure 5 As shown, in Figure 2 Based on the method embodiment shown, the following steps are further included after step S203.

[0080] S501: If the distribution entropy of the individual emotional probability distribution of the target user is greater than or equal to the first preset distribution entropy threshold, then obtain distribution entropy feedback compensation.

[0081] In practical applications, distribution entropy is typically used to measure the uncertainty of a probability distribution, i.e., the degree of dispersion of an individual's emotional probability distribution. When the probability is evenly distributed across multiple emotional types, the distribution entropy is high, indicating that the recognition result is highly uncertain and the emotional probability distribution is highly dispersed. When the probability is concentrated in a specific emotional type, the distribution entropy is low, indicating that the recognition result is highly certain and the emotional probability distribution is low.

[0082] Furthermore, a pre-set first distribution entropy threshold can be used to determine whether a compensation mechanism needs to be activated for the recognition result. When the distribution entropy of the individual emotion probability distribution of the target user, or the overall emotion probability distribution in some possible implementations, is greater than or equal to the first pre-set distribution entropy threshold, it indicates that the model's recognition result has significant uncertainty in the current interaction scenario.

[0083] For example, it can be done through the formula: Determine the distribution entropy of the individual sentiment probability distribution of the target user, where C is the total number of sentiment types in the individual sentiment probability distribution of the target user, and y c Let c be the probability of emotion type c in the individual emotion probability distribution of the target user.

[0084] Specifically, distribution entropy feedback compensation is usually used to adjust the parameters or compensation mechanism of the identification mechanism, which may include adjusting the parameters of the loss function, the model weight update strategy, or the correction coefficient that affects the weight calculation method.

[0085] S502: Based on the distribution entropy feedback compensation, the initial emotional characteristics of the target user, and the initial emotional characteristics of N other users, determine the feedback emotional influence weight of each of the N other users on the target user.

[0086] It is understandable that, based on the distribution entropy feedback compensation, and combined with the initial emotional characteristics of the target user and the initial emotional characteristics of N other users, the emotional influence weight of each other user on the target user can be re-determined, i.e., the feedback emotional influence weight.

[0087] S503: Determine the individual emotional probability distribution of the target user's feedback based on the initial emotional characteristics of each of the N other users and the emotional influence weight of each other user on the target user's feedback.

[0088] It is understandable that the initial emotional characteristics of each other user correspond to the weight of their emotional impact on the target user's feedback. The emotional impact weights of each other user's feedback can be normalized, and a weighted aggregation method can typically be used to determine the individual emotional probability distribution of the target user's feedback.

[0089] Of course, those skilled in the art can also choose other methods according to the actual situation to determine the individual emotional probability distribution of the target user's feedback by using the initial emotional characteristics of other users and the corresponding feedback emotional influence weights. For example, the target user's initial emotional characteristics can be used and corresponding weights can be assigned to them; or a graph neural network structure, such as a two-layer graph attention network, can be used to simultaneously consider the emotional characteristics of all users and their mutual influence relationships through the message passing mechanism between nodes, and finally output the individual emotional probability distribution of the target user's feedback.

[0090] See Figure 6 This is a flowchart illustrating another in-vehicle user emotion recognition method provided in an embodiment of this application. Figure 6 As shown, in Figure 2 Based on the method embodiment shown, step S201 specifically includes the following steps.

[0091] S601: By performing face recognition processing, scaling processing, rotation processing, translation processing and / or illumination normalization processing on a single frame of acquired image, determine the single frame face images of M users inside the vehicle.

[0092] In practical applications, raw single-frame images of the vehicle interior can be acquired using image acquisition devices deployed within the vehicle cabin. These devices typically include one or more cameras, and their installation locations usually cover the driver's seat, front passenger seat, and rear seats.

[0093] It can perform face recognition processing, scaling processing, rotation processing, translation processing and / or illumination normalization processing on single-frame acquired images to determine single-frame face images of M users inside the vehicle.

[0094] Specifically, scaling usually refers to adjusting the face area to a uniform pixel size; rotation usually refers to performing in-plane or out-of-plane angle correction on the face image; translation usually refers to adjusting the spatial position of the face in the image frame; and illumination normalization usually refers to applying a contrast-limited adaptive histogram to enhance the local contrast of the image and improve image quality.

[0095] This can be achieved through the following formula: The single-frame acquired image is scaled, rotated, and translated, where (x0, y0) are the coordinates of the acquired image, (x1, y1) are the coordinates of the processed image, s is the scaling factor, θ is the rotation angle, and t is the rotation angle. x and t y This represents the translation amount.

[0096] S602: By extracting emotional features from at least one frame of a single-frame facial image of M users in the vehicle within a preset time period, the initial emotional features of the M users in the vehicle are obtained.

[0097] Understandably, a preset time period typically refers to a period of time ending at the moment of in-vehicle user emotion recognition. The length of the preset time period can be set according to actual needs. Of course, it can also be a period of time with a certain interval from the moment of in-vehicle user emotion recognition. Within the preset time period, a single frame of face image can be selected according to preset rules or randomly, or multiple frames or even all frames of single frame face images can be selected within the preset time period.

[0098] Furthermore, feature extraction can be performed on single-frame facial images. For example, a pre-trained convolutional neural network can be used to process a single-frame image to determine a fixed-dimensional feature vector, which serves as the user's initial emotional feature. For multi-frame single-frame facial images, features can be extracted from each frame and stacked temporally to obtain the user's temporal feature sequence; this feature sequence is then input into a recurrent neural network (such as an LSTM network) or a temporal convolutional network to determine the user's initial emotional feature.

[0099] This application also provides an in-vehicle user emotion recognition device, see [link]. Figure 7 This is a schematic diagram of the structure of an in-vehicle user emotion recognition device provided in an embodiment of this application. Figure 7 As shown, the in-vehicle user emotion recognition device 700 includes an emotion feature acquisition module 701, an emotion influence weight determination module 702, and an emotion probability distribution determination module 703.

[0100] The emotion feature acquisition module 701 is used to acquire the initial emotion features of M users in the vehicle, where M≥2; The emotional influence weight determination module 702 is used to determine the initial emotional influence weight of each of the N other users on the target user based on the initial emotional characteristics of the target user and the initial emotional characteristics of N other users. The target user is any user among the M users, and the other users are users other than the target user among the M users, where N≥1. The emotion probability distribution determination module 703 is used to determine the individual emotion probability distribution of the target user based on the initial emotion characteristics of each of the N other users and the initial emotion influence weight of each other user on the target user. The individual emotion probability distribution of the target user is used to characterize the probability of the target user corresponding to each emotion type.

[0101] In one possible implementation, the emotion feature acquisition module 701 is further used to determine the single-frame face images of M users in the vehicle by performing face recognition processing, scaling processing, rotation processing, translation processing and / or illumination normalization processing on the single-frame acquired image. The initial emotional features of the M users in the vehicle are obtained by extracting emotional features from at least one frame of facial images of the M users within a preset time period.

[0102] In one possible implementation, the emotion probability distribution determination module 703 is further configured to determine the overall emotion probability distribution inside the vehicle based on the initial emotion characteristics of each of the M users and the influence weights of the other users on the initial emotions of each user. The overall emotion probability distribution is used to characterize the probability of each emotion type in the vehicle as a whole.

[0103] In one possible implementation, the emotion probability distribution determination module 703 is further configured to determine the current layer emotion characteristics of the target user based on the initial emotion characteristics of each of the N other users and the initial emotion influence weight of each other user on the target user; Based on the current emotional characteristics of the target user and the current emotional characteristics of N other users, determine the influence weight of each of the N other users on the current emotional characteristics of the target user. Based on the current layer sentiment features of each of the N other users and the weight of each other user’s influence on the current layer sentiment of the target user, determine the next layer sentiment features of the target user; Based on the next layer of emotional characteristics of the target user, determine the individual emotional probability distribution of the target user.

[0104] In one possible implementation, the emotion probability distribution determination module 703 is further configured to determine the emotion probability distribution according to the formula: Determine the current-layer sentiment influence weight of any one of the N other users on the target user; According to the formula: Determine the next layer of emotional characteristics for the target user; Among them, h i L h represents the current layer of sentiment characteristics for the target user. j L For any other user's current layer sentiment feature, e ij L Let W1 be the first weight matrix, W2 be the second weight matrix, a be the influence vector, LeakyReLU be the first activation function, contact be the feature concatenation function, and h be the influence vector. i L+1 ELU is the next layer of emotional features for the target user, and ELU is the second activation function.

[0105] In one possible implementation, the emotion probability distribution determination module 703 is further configured to obtain distribution entropy feedback compensation if the distribution entropy of the individual emotion probability distribution of the target user is greater than or equal to a first preset distribution entropy threshold. Based on distribution entropy feedback compensation, the initial emotional characteristics of the target user, and the initial emotional characteristics of N other users, determine the emotional impact weight of each of the N other users on the target user. Based on the initial emotional characteristics of each of the N other users and the emotional influence weight of each other user on the target user's feedback, the individual emotional probability distribution of the target user's feedback is determined.

[0106] Corresponding to the above embodiments, this application also provides a vehicle.

[0107] See Figure 8 This is a structural schematic diagram of a vehicle provided in an embodiment of this application. Figure 8 As shown, vehicle 800 includes controller 801.

[0108] The controller 801 is configured to perform some or all of the steps in the method embodiment.

[0109] For details regarding the embodiments of this application, please refer to the description of the above method embodiments. For the sake of brevity, these details will not be repeated here.

[0110] Corresponding to the above embodiments, this application also provides a computer-readable storage medium, wherein the computer-readable storage medium may store a program, and when the program runs, it can control the device where the computer-readable storage medium is located to execute some or all of the steps in the above method embodiments. In specific implementation, the computer-readable storage medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0111] For details regarding the embodiments of this application, please refer to the description of the above method embodiments. For the sake of brevity, these details will not be repeated here.

[0112] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0113] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0114] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus, controller, and computer storage medium can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0115] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. An in-vehicle user emotion recognition method, characterized by, The method comprises the following steps: obtaining initial emotional features of M users in a vehicle, M≥2; determining an initial emotional influence weight of each of N other users on the target user according to the initial emotional feature of the target user and the initial emotional features of the N other users, the target user being any one of the M users, the other users being the users other than the target user among the M users, N≥1; determining an individual emotional probability distribution of the target user according to the initial emotional feature of each of the N other users and the initial emotional influence weight of each of the N other users on the target user, the individual emotional probability distribution of the target user being used to represent a probability of each emotional type corresponding to the target user.

2. The method of claim 1, wherein, After determining the individual emotional probability distribution of the target user, the method further comprises the following steps: determining an overall emotional probability distribution in the vehicle according to the initial emotional feature of each of the M users and the initial emotional influence weight of the corresponding other users on each of the M users, the overall emotional probability distribution being used to represent a probability of each emotional type corresponding to the overall vehicle.

3. The method of claim 1, wherein, The method of determining the individual emotional probability distribution of the target user comprises the following steps: determining a current layer emotional feature of the target user according to the initial emotional feature of each of the N other users and the initial emotional influence weight of each of the N other users on the target user; determining a current layer emotional influence weight of each of the N other users on the target user according to the current layer emotional feature of the target user and the current layer emotional features of the N other users; determining a next layer emotional feature of the target user according to the current layer emotional feature of each of the N other users and the current layer emotional influence weight of each of the N other users on the target user; determining the individual emotional probability distribution of the target user according to the next layer emotional feature of the target user.

4. The method of claim 3, wherein, The method of determining the current layer emotional influence weight of each of the N other users on the target user according to the current layer emotional feature of the target user and the current layer emotional features of the N other users comprises the following steps: determining a current layer sentiment influence weight of any of the N other users on the target user according to the formula: ​ wherein h i L is the current layer emotion feature of the target user, h j L is the current layer emotion feature of any other user, e ij L is the current layer emotion influence weight of any of the other users on the target user, W1is a first weight matrix, a is an influence vector, LeakyReLU is a first activation function, and contact is a feature concatenation function.

5. The method of claim 3, wherein, The method of determining the next layer emotional feature of the target user according to the current layer emotional feature of each of the N other users and the current layer emotional influence weight of each of the N other users on the target user comprises the following steps: According to the formula: determine the next layer emotional features of the target user; wherein h j L is a current layer emotion feature of any other user, e ij L is a current layer emotion influence weight of any said other user on said target user, W2 is a second weight matrix, h i L+1 is a next layer emotion feature of said target user, ELU is a second activation function.

6. The method of claim 1, wherein, After determining the individual emotional probability distribution of the target user, the method further comprises the following steps: if the distribution entropy of the individual emotional probability distribution of the target user is greater than or equal to a first preset distribution entropy threshold, obtaining a distribution entropy feedback compensation; determining a feedback emotional influence weight of each of the N other users on the target user according to the distribution entropy feedback compensation, the initial emotional feature of the target user and the initial emotional features of the N other users; According to the initial emotional feature of each of the N other users and the feedback emotional influence weight of each of the N other users on the target user, the feedback individual emotional probability distribution of the target user is determined.

7. The method of claim 1, wherein, The initial emotional features of the M users in the vehicle are acquired, including: The single-frame face image of the M users in the vehicle is determined by performing face recognition processing, scaling processing, rotation processing, translation processing and / or illumination normalization processing on the single-frame collected image. The initial emotional features of the M users in the vehicle are acquired by performing emotional feature extraction on at least one single-frame face image of the M users in the vehicle in a preset time period.

8. An inter-passenger interaction emotion recognition device, characterized by, The method comprises: An emotional feature acquisition module is configured to acquire the initial emotional features of the M users in the vehicle, M≥2. An emotional influence weight determination module is configured to determine the initial emotional influence weight of each of the N other users on the target user according to the initial emotional features of the target user and the initial emotional features of the N other users, the target user being any one of the M users, the other users being the users other than the target user among the M users, and N≥1. An emotional probability distribution determination module is configured to determine the individual emotional probability distribution of the target user according to the initial emotional features of each of the N other users and the initial emotional influence weight of each of the N other users on the target user, the individual emotional probability distribution of the target user being used to represent the probability of each emotional type corresponding to the target user.

9. A vehicle characterized by comprising: The controller is configured to perform the method of any one of claims 1-7. The computer-readable storage medium comprises a stored program, wherein the computer-readable storage medium controls the device where the computer-readable storage medium is located to perform the method of any one of claims 1-7 when the program is running.

10. A computer-readable storage medium, characterized in that, ​

Images