Emotional computing method and system for vehicle-mounted robots based on width learning algorithm

Abstract

The present invention discloses an emotion calculation method and system for an on-vehicle robot based on a width learning algorithm, the method comprising: training a width learning algorithm model based on user data of a vehicle driver; transmitting the user data, the vehicle's surrounding environment data and the vehicle CAN data to the width learning algorithm model and the HMM emotion calculation model at the same time; predicting the driver's behavior habits through the width learning algorithm model to understand the driver's intention; transmitting the understood driver's intention data to the HMM emotion calculation model for data preprocessing, converting the user data, the vehicle's surrounding environment data, the vehicle CAN data and the driver's intention data into mathematical expression information that can be understood by the HMM emotion calculation model; performing driver emotion recognition; establishing an HMM emotion calculation model and correcting the HMM emotion calculation model; and outputting emotion expression data that meets the driver's needs.

Description

Technical Field

[0001] The present invention belongs to the field of vehicle-mounted artificial intelligence, and in particular relates to an emotion computing method and system for a vehicle-mounted robot. Background Art

[0002] After the clear concept of in-vehicle robots was proposed, in order to provide sufficiently humane emotional services, the ability to observe words and expressions and to interact emotionally with the driver has become a core functional requirement.

[0003] Some existing technical solutions have integrated multimodal interactive emotional solutions that combine voice and images. The on-board robot can perform intelligent emotional calculations based on the voice and facial expression characteristics of the people in the car, so as to interact with people emotionally. For example, it can understand the user's mood and provide corresponding proactive services if the user is in a bad mood.

[0004] Current emotional computing technology solutions mainly focus on multimodal information processing in the user dimension, performing emotional computing based on user voice, expression, action and other characteristics. This method can read, hear or understand user needs.

[0005] However, the application scenarios of in-vehicle robots are not just about chatting and communicating with users. Their applications run through the entire driving environment of users. In addition to emotion analysis based on expression recognition and intelligent voice systems, and then providing corresponding anthropomorphic emotional output, the driving environment and vehicle status are also important input factors for the emotional calculation of in-vehicle robots. The existing emotional calculation dimensions of in-vehicle robots are too single, and the robot's emotional output is often stiff, with a single "pleasing" experience. After long-term use, users will inevitably feel bored because of the single and unchanging way of expressing their emotions, and cannot achieve a good experience. Summary of the invention

[0006] In order to overcome the deficiencies of the above-mentioned prior art, the purpose of the present invention is to provide a vehicle-mounted robot multimodal emotion computing system based on a width learning algorithm, which can continuously learn and continuously improve the accuracy of emotion computing, and make the robot's emotional expression more humanized, making human-computer interaction natural and smooth.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] An emotion calculation method for a vehicle-mounted robot based on a width learning algorithm comprises the following steps:

[0009] S1. Performing width learning algorithm model training on the user data of vehicle drivers;

[0010] S2, transmitting the user data, the surrounding environment data of the vehicle and the automobile CAN data of the vehicle simultaneously to the width learning algorithm model and the HMM emotion calculation model that complete the model training;

[0011] S3, predicting the driver's behavior habits through the width learning algorithm model to understand the driver's intention;

[0012] S4, the understood driver intention data is also transmitted to the HMM emotion calculation model, and data preprocessing is performed to convert the user data, the surrounding environment data of the vehicle, the vehicle CAN data data and the understood driver intention data into mathematical expression information that can be understood by the HMM emotion calculation model;

[0013] S5, extracting characteristic data related to the driver and performing driver emotion recognition;

[0014] S6, establishing an HMM emotion calculation model, taking the vehicle CAN data, surrounding environment data and driver intention as sensitive factor indicators, and correcting the HMM emotion calculation model;

[0015] S7. Output emotional expression data that meets the driver's needs.

[0016] Furthermore, the feature data in S5 includes facial features, acoustic features, and text features, and the feature data is extracted by using an image feature extraction method, an acoustic feature extraction method, and a text feature extraction method.

[0017] Furthermore, the user emotion recognition in S5 includes identifying whether the user's emotion is positive, negative or neutral, specifically including six basic emotion elements: happiness, surprise, sadness, anger, disgust and fear.

[0018] Furthermore, the HMM sentiment calculation model is a mathematical model of Hidden Markov Model.

[0019] Furthermore, the establishment of the HMM sentiment calculation model described in S6 includes the following steps:

[0020] Acquiring an emotion signal, including user data of a vehicle driver, surrounding environment data of the vehicle, and vehicle CAN data of the vehicle;

[0021] Process the above data and combine it with the robot's basic emotion data to generate the robot's emotion intensity vector;

[0022] Entering the HMM emotion calculation model to perform emotion transfer operation;

[0023] Get the emotion sequence of the onboard robot;

[0024] Generate emotional expression data for the in-vehicle robot;

[0025] Drive the robot's facial expressions and motion mechanisms to generate the vehicle-mounted robot's external emotional expressions.

[0026] Furthermore, the processing of the above data and combining it with the robot's basic emotional data to generate a robot emotion intensity vector specifically includes: using basic emotional elements as the intrinsic expression of the vehicle-mounted robot's emotions to characterize the robot's emotional nature, using intensity for data description of each emotional element, the intensity has a decay process, and finally tends to a calm neutral emotional state to generate a robot emotion vector. In the process of generating the robot emotion vector, factors affecting the emotion intensity also include user data of the vehicle driver, the vehicle's surrounding environment data, and the vehicle's car CAN data; external stimuli, internal stimuli and the robot emotion vector are used together to calculate the robot emotion intensity vector, and I is used to represent the robot emotion intensity vector, IT = (I1T + I2T + I3T +, ..., + INT) represents the emotion intensity of the vehicle-mounted robot at time T, and IiT represents the intensity of the i-th basic emotion element of the vehicle-mounted robot at time T; the concept of emotion intensity does not exist in the robot's neutral emotional state, that is, when the intensity of all basic emotion elements is as low as a certain threshold, the robot's emotion is calm.

[0027] The entering of the HMM emotion calculation model and the performing of emotion transfer operations specifically include: the robot has a variety of basic emotion states, which can be transferred to each other, and the transfer of emotion states is affected by the probability distribution of emotion and the emotion state transfer matrix; after all elements in the emotion intensity vector are calculated, they are input into the HMM emotion calculation model to participate in the emotion transfer operation.

[0028] Furthermore, the expression and action mechanism of the driving robot generates the external emotional expression form of the on-board robot, specifically including: the on-board robot generates corresponding emotional behavior according to the emotional state and the corresponding emotional intensity, and transmits it to the on-board robot's emotional expression system, and the emotional expression system matches the corresponding expression, posture, and action from the robot's emotional expression database.

[0029] The step S1 of performing width learning algorithm model training on the user data of the vehicle driver specifically includes the following steps:

[0030] Acquiring user data of a vehicle driver, surrounding environment data of the vehicle, and vehicle CAN data of the vehicle;

[0031] Process the above data to obtain the input data matrix of emotion recognition based on width learning;

[0032] Processing the input data matrix using a width learning engine;

[0033] Identify the user's emotional state and store it in the user emotion database;

[0034] Form an initial in-vehicle robot user emotion recognition model.

[0035] The emotion computing system of the vehicle-mounted robot based on the width learning algorithm of the present invention comprises:

[0036] A width learning algorithm model training module, which is used to perform width learning algorithm model training based on user data of vehicle drivers;

[0037] A data acquisition module, which is used to simultaneously transmit the user data, the surrounding environment data of the vehicle, and the automobile CAN data of the vehicle to the width learning algorithm model and the HMM emotion calculation model that complete the model training;

[0038] A width learning algorithm model engine, which is used to predict the driver's behavior habits and understand the driver's intentions through the width learning algorithm model;

[0039] An HMM sentiment computing model data acquisition module is used to transmit the understood driver intention data to the HMM sentiment computing model, perform data preprocessing, and convert the user data, the surrounding environment data of the vehicle, the vehicle CAN data data and the understood driver intention data into mathematical expression information that can be understood by the HMM sentiment computing model;

[0040] Emotion recognition module, used to extract characteristic data related to the driver and perform driver emotion recognition;

[0041] A correction module, which is used to establish an HMM emotion calculation model, and uses the automobile CAN data, surrounding environment data and driver intention as sensitive factor indicators to correct the HMM emotion calculation model;

[0042] The emotion expression output module is used to output emotion expression data that meets the driver's needs.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] The present invention provides a user emotion recognition method with high efficiency and high accuracy, which can realize the training and recognition of emotion recognition models based on local hardware;

[0045] The user's emotional state, external environment state, and vehicle state are fully considered as stimulus factors for emotional intensity;

[0046] The emotional state of the in-vehicle robot is expressed through multiple dimensions of expression, posture, and action, making the in-vehicle robot more humanized and providing a better user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, wherein:

[0048] Figure 1 It is a principle block diagram of an emotion calculation method of a vehicle-mounted robot based on a width learning algorithm according to an embodiment of the present invention;

[0049] Figure 2 is a flow chart of establishing an HMM sentiment calculation model according to an embodiment of the present invention;

[0050] Figure 3 is a flow chart of emotion recognition training based on a width learning algorithm according to an embodiment of the present invention;

[0051] Figure 4 This is the emotional computing expression process of the embodiment of the present invention. DETAILED DESCRIPTION

[0052] This embodiment provides an emotion calculation method for a vehicle-mounted robot based on a width learning algorithm. Figure 1 As shown, the following steps are included:

[0053] S1. Performing width learning algorithm model training on the user data of vehicle drivers;

[0054] S2, transmitting the user data, the surrounding environment data of the vehicle and the automobile CAN data of the vehicle simultaneously to the width learning algorithm model and the HMM emotion calculation model that complete the model training;

[0055] S3, predicting the driver's behavior habits through the width learning algorithm model to understand the driver's intention;

[0056] S4, the understood driver intention data is also transmitted to the HMM emotion calculation model, and data preprocessing is performed to convert the user data, the surrounding environment data of the vehicle, the vehicle CAN data data and the understood driver intention data into mathematical expression information that can be understood by the HMM emotion calculation model;

[0057] S5, extracting characteristic data related to the driver and performing driver emotion recognition;

[0058] S6, establishing an HMM emotion calculation model, taking the vehicle CAN data, surrounding environment data and driver intention as sensitive factor indicators, and correcting the HMM emotion calculation model;

[0059] S7. Output emotional expression data that meets the driver's needs.

[0060] The user data includes the driver's facial expressions, voice input, head posture, etc.

[0061] The vehicle's surrounding environment data includes weather conditions, traffic conditions, and network conditions in the area where the vehicle is located.

[0062] The feature data in S5 includes facial features, acoustic features, and text features, and the feature data are extracted by an image feature extraction method, an acoustic feature extraction method, and a text feature extraction method. The image feature extraction method and the acoustic feature extraction method may use a camera. The acoustic feature extraction method may use a microphone.

[0063] Moreover, user emotion recognition in S5 includes identifying whether the user's emotion is positive, negative or neutral, specifically including six basic emotion elements such as happiness, surprise, sadness, anger, disgust and fear.

[0064] The HMM sentiment calculation model described in this embodiment is a mathematical model of Hidden Markov Model.

[0065] Specifically, the HMM sentiment calculation model described in S6 is established, such as Figure 2 and Figure 4 As shown, the following steps are included:

[0066] Acquiring an emotion signal, including user data of a vehicle driver, surrounding environment data of the vehicle, and vehicle CAN data of the vehicle;

[0067] Process the above data and combine it with the robot's basic emotional data to generate a robot emotion intensity vector: use basic emotional elements as the emotional internal representation of the vehicle-mounted robot to characterize the robot's emotional essence, use intensity to describe the data of each emotional element, the intensity has a decay process, and finally tends to a calm neutral emotional state to generate a robot emotion vector. In the process of generating the robot emotion vector, factors affecting the emotion intensity also include the user data of the vehicle driver, the surrounding environment data of the vehicle, and the car CAN data of the vehicle; external stimulation, internal stimulation and the robot emotion vector are used together to calculate the robot emotion intensity vector, and I is used to represent the robot emotion intensity vector, IT = (I1T + I2T + I3T +, ..., + INT) represents the emotion intensity of the vehicle-mounted robot at time T, and IiT represents the intensity of the i-th basic emotion element of the vehicle-mounted robot at time T; the concept of emotion intensity does not exist in the neutral emotional state of the robot, that is, when the intensity of all basic emotion elements is as low as a certain threshold, the robot's emotion is calm;

[0068] Enter the HMM emotion calculation model and perform emotion transfer operation: the robot has multiple basic emotion states, which can be transferred to each other. The transfer of emotion state is jointly affected by the probability distribution of emotion and the emotion state transfer matrix; after all elements in the emotion intensity vector are calculated, they are input into the HMM emotion calculation model to participate in the emotion transfer operation;

[0069] Get the emotion sequence of the onboard robot;

[0070] Generate emotional expression data for the in-vehicle robot;

[0071] Drive the robot's facial expressions and motion mechanisms to generate the vehicle-mounted robot's external emotional expression form: the vehicle-mounted robot generates corresponding emotional behaviors based on the emotional expression data and transmits them to the vehicle-mounted robot's emotional expression system. The emotional expression system matches the corresponding expressions, postures, and actions from the robot's emotional expression database.

[0072] like Figure 3 As shown, the width learning algorithm model training based on the user data of the vehicle driver described in S1 specifically includes the following steps:

[0073] Acquiring user data of a vehicle driver, surrounding environment data of the vehicle, and vehicle CAN data of the vehicle;

[0074] Process the above data to obtain the input data matrix of emotion recognition based on width learning;

[0075] Processing the input data matrix using a width learning engine;

[0076] Identify the user's emotional state and store it in the user emotion database;

[0077] Form an initial in-vehicle robot user emotion recognition model.

[0078] More specifically, the width learning algorithm model training includes the following steps:

[0079] Collect a large amount of user data from drivers and construct training data for the algorithm model: divide the user data from a large amount of drivers into a training dataset X Train, a validation dataset X Incre, and a test dataset X test, and perform z-score normalization and sparse representation on the training dataset XTrain, the validation dataset X Incre, and the test dataset X test;

[0080] Process the training data to obtain the original input matrix and the original output matrix, and construct a mapping feature point matrix: first, the input training data is sequentially subjected to z-score standardization and normalization processing to obtain an input training data matrix, and then the input training data matrix is ​​subjected to z-score normalization and sparse representation using a width learning method, and the input training data is subjected to z-score normalization processing and then augmented, and then feature nodes are generated to construct a mapping feature point matrix; the specific algorithm is: when constructing the mapping feature point matrix, the training data matrix obtained by the training data set X Train is subjected to z-score normalization and sparse representation using a width learning method to generate feature nodes Z i =φ(XW ei +β ei ), i = 1, ..., n, and mark the feature layer as Z i =[Z1,…,Z i ], where W ei is a random weight matrix of appropriate dimension, generated by a random weight matrix ωe with Gaussian distribution, i represents the number of iterations;

[0081] The mapping feature point matrix is ​​used to construct an enhanced point matrix after processing: the mapping feature point matrix is ​​normalized and sparsely represented by the width learning method, enhanced nodes are generated, and the enhanced point matrix is ​​constructed; in the specific algorithm, when constructing the enhanced point matrix, the enhanced node H is directly calculated based on the mapping feature point matrix. j =φ(Z i W hj +β hj ), j = 1, ..., n, and mark the enhancement layer as H j =[H1,…,H j ];

[0082] The weight matrix from the input layer to the output layer is obtained by using pseudo-inverse calculation and ridge regression algorithm to complete the training of the algorithm model: first, the actual content of the user data is defined as the label vector Y, which is known data; then the above feature layer and enhancement layer are merged into A = [Z|H], and the vertical line indicates that the feature layer and enhancement layer are merged into one row, and the weight W = A is calculated using pseudo-inverse and ridge regression algorithm. -1 Y; Finally, after the initial training of the model is completed, the validation data set XIncre and the test data set Xtest are used to verify and debug the model's fitting and data generalization capabilities. The model training is completed after the expected indicators are met.

[0083] This embodiment also provides an emotion computing system for a vehicle-mounted robot based on a width learning algorithm, which includes:

[0084] A width learning algorithm model training module, which is used to perform width learning algorithm model training based on user data of vehicle drivers;

[0085] A data acquisition module, which is used to simultaneously transmit the user data, the surrounding environment data of the vehicle, and the automobile CAN data of the vehicle to the width learning algorithm model and the HMM emotion calculation model that complete the model training;

[0086] A width learning algorithm model engine, which is used to predict the driver's behavior habits and understand the driver's intentions through the width learning algorithm model;

[0087] An HMM sentiment computing model data acquisition module is used to transmit the understood driver intention data to the HMM sentiment computing model, perform data preprocessing, and convert the user data, the surrounding environment data of the vehicle, the vehicle CAN data data and the understood driver intention data into mathematical expression information that can be understood by the HMM sentiment computing model;

[0088] Emotion recognition module, used to extract characteristic data related to the driver and perform driver emotion recognition;

[0089] A correction module, which is used to establish an HMM emotion calculation model, and uses the automobile CAN data, surrounding environment data and driver intention as sensitive factor indicators to correct the HMM emotion calculation model;

[0090] The emotion expression output module is used to output emotion expression data that meets the driver's needs.

[0091] The above is only a preferred embodiment of the invention and does not limit the invention in any form. Any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. An emotional computing method for a vehicle-mounted robot based on a width learning algorithm, characterized in that : Includes the following steps: S1, performing width learning algorithm model training on the user data of the vehicle driver, the surrounding environment data of the vehicle and the vehicle CAN data of the vehicle, wherein the user data includes the driver's facial expression, voice input and head posture; S2, transmitting the user data, the surrounding environment data of the vehicle and the automobile CAN data of the vehicle simultaneously to the width learning algorithm model and the HMM emotion calculation model that complete the model training; S3, predicting the driver's behavior habits through the width learning algorithm model to understand the driver's intention; S4, the understood driver intention data is also transmitted to the HMM emotion calculation model, and data preprocessing is performed to convert the user data, the surrounding environment data of the vehicle, the vehicle CAN data and the understood driver intention data into mathematical expression information that can be understood by the HMM emotion calculation model; S5, extracting feature data related to the driver and performing user emotion recognition of the driver; S6, establishing an HMM emotion calculation model, taking the vehicle CAN data, surrounding environment data and driver intention as sensitive factor indicators, and correcting the HMM emotion calculation model; S7. Output emotional expression data that meets the driver's needs.

2. The method according to claim 1, characterized in that: The feature data in S5 includes facial features, acoustic features, and text features, and the feature data is extracted by using an image feature extraction method, an acoustic feature extraction method, and a text feature extraction method.

3. The method according to claim 1, characterized in that: The user emotion recognition in S5 includes recognizing the user's emotions including six basic emotion elements: happiness, surprise, sadness, anger, disgust, and fear.

4. The method according to claim 1, characterized in that: The HMM sentiment calculation model is a mathematical model of Hidden Markov Model.

5. The method according to claim 1, characterized in that: The establishment of the HMM sentiment calculation model described in S6 includes the following steps: Acquiring an emotion signal, including user data of a vehicle driver, surrounding environment data of the vehicle, and vehicle CAN data of the vehicle; Process the above data and combine it with the robot's basic emotion data to generate the robot's emotion intensity vector; Entering the HMM emotion calculation model to perform emotion transfer operation; Get the emotion sequence of the onboard robot; Generate emotional expression data for the in-vehicle robot; Drive the robot's facial expressions and motion mechanisms to generate the vehicle-mounted robot's external emotional expressions.

6. The method according to claim 5, characterized in that: The processing of the above data and combining it with the basic emotion data of the robot to generate the robot emotion intensity vector specifically includes: using basic emotion elements as the emotional internal representation of the vehicle-mounted robot to characterize the emotional nature of the robot, using intensity for data description of each emotion element, the intensity has a decay process, and finally tends to a calm neutral emotion state to generate the robot emotion vector. In the process of generating the robot emotion vector, factors affecting the emotion intensity also include user data of the vehicle driver, the surrounding environment data of the vehicle, and the car CAN data of the vehicle; external stimulation, internal stimulation and the robot emotion vector are used together to calculate the robot emotion intensity vector, and I is used to represent the robot emotion intensity vector, IT=(I1T+I2T+I3T+,……,+INT) represents the emotion intensity of the vehicle-mounted robot at time T, and IiT represents the intensity of the i-th basic emotion element of the vehicle-mounted robot at time T; the concept of emotion intensity does not exist in the neutral emotion state of the robot, that is, when the intensity of all basic emotion elements is as low as a certain threshold, the robot's emotion is calm.

7. The method according to claim 5, characterized in that: The entering of the HMM emotion calculation model and the performing of emotion transfer operations specifically include: the robot has a variety of basic emotion states, which can be transferred to each other, and the transfer of emotion states is affected by the probability distribution of emotion and the emotion state transfer matrix; after all elements in the emotion intensity vector are calculated, they are input into the HMM emotion calculation model to participate in the emotion transfer operation.

8. The method according to claim 5, characterized in that: The expression and action mechanism of the driving robot generates the external emotional expression form of the on-board robot, specifically including: the on-board robot generates corresponding emotional behavior according to the emotional state and the corresponding emotional intensity, and transmits it to the emotional expression system of the on-board robot, and the emotional expression system matches the corresponding expression, posture and action from the robot emotional expression database.

9. The method according to claim 1, characterized in that: The step S1 of performing width learning algorithm model training on the user data of the vehicle driver specifically includes the following steps: Acquiring user data of a vehicle driver, surrounding environment data of the vehicle, and vehicle CAN data of the vehicle; Process the above data to obtain the input data matrix of emotion recognition based on width learning; Processing the input data matrix using a width learning engine; Identify the user's emotional state and store it in the user emotion database; Form an initial in-vehicle robot user emotion recognition model.

10. An emotional computing system for a vehicle-mounted robot based on a width learning algorithm, characterized in that: It includes: A width learning algorithm model training module, which is used to perform width learning algorithm model training based on user data of the vehicle driver, surrounding environment data of the vehicle, and automobile CAN data of the vehicle; A data acquisition module, which is used to simultaneously transmit the user data, the surrounding environment data of the vehicle, and the automobile CAN data of the vehicle to the width learning algorithm model and the HMM emotion calculation model that complete the model training; A width learning algorithm model engine, which is used to predict the driver's behavior habits and understand the driver's intentions through the width learning algorithm model; An HMM sentiment computing model data acquisition module is used to transmit the understood driver intention data to the HMM sentiment computing model, perform data preprocessing, and convert the user data, the surrounding environment data of the vehicle, the vehicle CAN data and the understood driver intention data into mathematical expression information that can be understood by the HMM sentiment computing model; Emotion recognition module, used to extract characteristic data related to the driver and perform driver emotion recognition; A correction module, which is used to establish an HMM emotion calculation model, and uses the automobile CAN data, surrounding environment data and driver intention as sensitive factor indicators to correct the HMM emotion calculation model; The emotion expression output module is used to output emotion expression data that meets the driver's needs.

Images