Emotion analysis processing method, system and equipment for humanoid robot and medium

Abstract

The invention discloses an emotion analysis processing method for a humanoid robot, and the method comprises the steps: obtaining first multi-modal data of a user, and carrying out the preprocessing of the first multi-modal data, and obtaining second multi-modal data; performing time sequence synchronization and feature extraction on the second multi-modal data to obtain voice feature data, posture feature data and visual feature data; according to the voice feature data and the posture feature data, performing self-attention feature fusion on the visual feature data to obtain fusion features; inputting the fusion features into an emotion classification model for emotion recognition to obtain emotion state information; and inputting the emotional state information into an emotional interaction generation model for emotional interaction generation to obtain emotional interaction information. According to the method, by improving the accuracy and robustness of emotion recognition in the man-machine intelligent interaction process, the matching degree of the generated interaction content and the user emotion can be improved, and the user experience is improved. The invention relates to the technical field of emotion analysis.

Description

Technical Field

[0001] The present invention relates to the field of emotion analysis technology, and in particular to an emotion analysis processing method, system, device and medium for a humanoid robot. Background Art

[0002] With the rapid development of artificial intelligence technology, human-computer intelligent interaction has become one of the key areas of people's attention.

[0003] At present, the existing technology mainly recognizes human emotions in the process of human-computer intelligent interaction through single-modal data, and then generates interactive content for humanoid robots to execute based on the recognized human emotional information. The human emotions recognized by this method are relatively one-sided, and the accuracy and robustness of emotion recognition are unsatisfactory, which makes the generated interactive content less matched with user emotions and the user experience is poor.

[0004] Therefore, the problems existing in the prior art still need to be solved and optimized. Summary of the invention

[0005] The purpose of the present invention is to solve one of the technical problems existing in the related art to at least a certain extent.

[0006] To this end, an object of an embodiment of the present invention is to provide a method, system, device and medium for emotion analysis and processing of a humanoid robot, wherein the method can improve the matching degree between the generated interactive content and the user's emotions and improve the user experience by improving the accuracy and robustness of emotion recognition during human-computer intelligent interaction.

[0007] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present application include:

[0008] In a first aspect, an embodiment of the present application provides a method for analyzing and processing emotions of a humanoid robot, comprising:

[0009] Acquire first multimodal data of a user, and preprocess the first multimodal data to obtain second multimodal data;

[0010] Performing time synchronization and feature extraction on the second multimodal data to obtain third multimodal data, wherein the third multimodal data includes voice feature data, posture feature data, and visual feature data;

[0011] According to the speech feature data and the posture feature data, performing self-attention feature fusion on the visual feature data to obtain a fusion feature;

[0012] Inputting the fused features into the current emotion classification model to perform emotion recognition and obtain emotion state information;

[0013] The emotional state information is input into the current emotional interaction generation model to generate emotional interaction and obtain emotional interaction information.

[0014] In addition, the method according to the above embodiment of the present application may also have the following additional technical features:

[0015] Further, in one embodiment of the present application, preprocessing the first multimodal data to obtain second multimodal data includes:

[0016] Performing data denoising on the first multimodal data to obtain intermediate denoised data;

[0017] The intermediate denoised data is subjected to data normalization to obtain the second multimodal data.

[0018] Further, in one embodiment of the present application, performing time series synchronization and feature extraction on the second multimodal data to obtain third multimodal data includes:

[0019] Acquire voice modality data, posture modality data, and visual modality data in the second multimodal data;

[0020] According to the voice modality data and the posture modality data, time series matching is performed on the visual modality data to obtain time series matching information, wherein the time series matching information is used to characterize a time mapping relationship between the voice modality data, the posture modality data, and the visual modality data;

[0021] According to the timing matching information, timing synchronization is performed on the second multimodal data to obtain timing synchronization data, where the timing synchronization data includes voice synchronization data, posture synchronization data, and visual synchronization data;

[0022] Extracting frequency spectrum features from the speech synchronization data to obtain speech feature data, and extracting facial features from the visual synchronization data to obtain visual feature data;

[0023] Performing posture feature extraction on the posture synchronization data to obtain the posture feature data.

[0024] Further, in one embodiment of the present application, the self-attention feature fusion is performed on the visual feature data according to the voice feature data and the posture feature data to obtain the fusion feature, including:

[0025] Performing a linear transformation on the speech feature data, the posture feature data and the visual feature data to obtain a multimodal feature vector set, wherein the multimodal feature vector set includes a speech feature vector, a posture feature vector and a visual feature vector;

[0026] Performing matrix calculation on the multimodal feature vector set to obtain a query key value matrix set corresponding to the multimodal feature vector set;

[0027] Multi-head self-attention calculation is performed on the query key-value matrix set to obtain the fusion feature.

[0028] Further, in one embodiment of the present application, the emotion classification model includes a first network framework and a second network framework, and the step of inputting the fusion feature into the current emotion classification model for emotion recognition to obtain the emotion state information includes:

[0029] Inputting the fused features into the first network framework to extract spatial features to obtain spatial features;

[0030] Inputting the spatial features into the second network framework to extract emotional time-varying features to obtain emotional spatiotemporal features;

[0031] The emotional spatiotemporal features are classified and identified to obtain the emotional state information.

[0032] Further, in one embodiment of the present application, the emotion interaction generation model includes a pre-trained language framework and a third network framework, and the emotion state information is input into the current emotion interaction generation model to generate emotion interaction, and the emotion interaction information is obtained, including:

[0033] Obtaining a preset action library and context information corresponding to the user,

[0034] Inputting the context information and the emotional state information into the pre-trained language framework to perform language generation to obtain language interaction information;

[0035] Inputting the action library, the emotional state information and the language interaction information into the third network framework to perform action learning and generation to obtain action interaction information;

[0036] The emotion interaction information is obtained according to the language interaction information and the action interaction information.

[0037] Furthermore, in the embodiment of the present application, the method further includes:

[0038] Obtain users’ emotional feedback data;

[0039] According to a preset reward function and the emotional feedback data, the quality of the emotional interaction information is evaluated to obtain a reward evaluation value;

[0040] According to the reward evaluation value, feedback optimization is performed on the current emotion interaction generation model to obtain an optimized emotion interaction generation model;

[0041] The method further comprises:

[0042] Acquire historical emotion information and historical interaction data of the user;

[0043] According to the historical emotion information and the historical interaction data, the current emotion classification model is fine-tuned and optimized to obtain an optimized emotion classification model.

[0044] In a second aspect, the embodiment of the present application provides a humanoid robot emotion analysis and processing system, comprising:

[0045] A first processing unit, configured to obtain first multimodal data of a user, and preprocess the first multimodal data to obtain second multimodal data;

[0046] a second processing unit, configured to perform time synchronization and feature extraction on the second multimodal data to obtain third multimodal data, wherein the third multimodal data includes voice feature data, posture feature data, and visual feature data;

[0047] A third processing unit is used to perform self-attention feature fusion on the visual feature data according to the voice feature data and the posture feature data to obtain a fusion feature;

[0048] A fourth processing unit, configured to input the fusion feature into a current emotion classification model to perform emotion recognition and obtain emotion state information;

[0049] The fifth processing unit is used to input the emotional state information into the current emotional interaction generation model to generate emotional interaction and obtain emotional interaction information.

[0050] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0051] at least one processor;

[0052] at least one memory for storing at least one program;

[0053] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0054] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a program executable by a processor, and the program executable by the processor is used to implement the above method when executed by the processor.

[0055] The advantages and benefits of the present application will be partially given in the following description, and partially become apparent from the following description, or be understood through the practice of the present application:

[0056] The embodiment of the present application discloses a method, system, device and medium for emotion analysis and processing of a humanoid robot, wherein the method obtains the first multimodal data of the user, and pre-processes the first multimodal data to obtain the second multimodal data; performs time synchronization and feature extraction on the second multimodal data to obtain the third multimodal data, and the third multimodal data includes voice feature data, posture feature data and visual feature data; performs self-attention feature fusion on the visual feature data according to the voice feature data and the posture feature data to obtain fusion features; inputs the fusion features into the current emotion classification model for emotion recognition to obtain emotion state information; inputs the emotion state information into the current emotion interaction generation model for emotion interaction generation to obtain emotion interaction information. By performing self-attention feature fusion on the language feature data, posture feature data and visual feature data, the method can more comprehensively understand the multi-dimensional emotion clues, improve the accuracy and robustness of emotion recognition, and thus improve the matching degree between the generated interactive content and the user's emotion, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the embodiments of the present application or the drawings of the related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly expressing some embodiments of the technical solutions of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 A schematic diagram of a flow chart of a method for analyzing and processing emotions of a humanoid robot provided in an embodiment of the present application;

[0059] Figure 2 A schematic diagram of the structural framework of a humanoid robot emotion analysis and processing system provided in an embodiment of the present application;

[0060] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limitations on the present application. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0063] At present, the existing technology mainly uses single-modal data to identify human emotions in the process of human-machine intelligent interaction, and then generates interactive content for humanoid robots to execute based on the identified human emotional information. However, since single-modal emotion recognition is difficult to fully understand multi-dimensional emotional clues, it is often unable to accurately capture the synergy between different signals. The human emotions identified by this method are relatively one-sided, and the accuracy and robustness of emotion recognition are unsatisfactory, which makes the generated interactive content less matched with user emotions and the user experience poor. In addition, there is a part of the existing technology that uses simple feature splicing or weighted averaging to realize emotion recognition of multi-modal data. This method is difficult to effectively capture the high-order correlation between different modal data, which limits the accuracy and robustness of emotion recognition to a certain extent.

[0064] It is worth mentioning that when the existing technology performs emotion recognition on unimodal data or multimodal data, it usually adopts a general emotion recognition model. This general emotion recognition model is difficult to adapt to the individual differences of different users and performs poorly in personalized emotion recognition, resulting in inaccurate recognition results, which makes the generated interactive content less compatible with the user's emotions and the user experience poor. In addition, the existing technology often relies on a preset fixed language and action library in the process of human-computer intelligent interaction. It lacks the ability to flexibly generate dynamic interactive content and cannot generate personalized interactive content based on the actual emotional state. This makes the human-computer interaction appear mechanical and repetitive, making it difficult to effectively improve the user's emotional identification and acceptance, and the user experience is poor.

[0065] In view of this, an embodiment of the present invention provides a method for emotion analysis and processing of a humanoid robot, wherein the method can more comprehensively understand multi-dimensional emotion clues, effectively capture high-order correlations between multimodal data, and improve the accuracy and robustness of emotion recognition by performing self-attention feature fusion on language feature data, posture feature data, and visual feature data; furthermore, the method is based on an emotion interaction generation model including a pre-trained language framework and a third network framework, and can generate flexible and dynamic interactive content, can improve the degree of personalization in the process of human-computer intelligent interaction, can effectively enhance the user's emotional identification and acceptance, and provide a better user experience; in addition, the method is based on real-time feedback optimization of the emotion interaction generation model, and real-time fine-tuning training of the emotion classification model, and can effectively improve the individual differences of the emotion recognition model and the emotion interaction generation model for different users, and can perform more personalized emotion recognition and emotion interaction generation for different users, and provide a better user experience.

[0066] Reference Figure 1 In an embodiment of the present application, a method for analyzing and processing emotions of a humanoid robot includes:

[0067] Step 110: Acquire first multimodal data of the user, and preprocess the first multimodal data to obtain second multimodal data;

[0068] In the embodiment of the present application, the user may be a human who interacts intelligently with the humanoid robot, and the humanoid robot may be a robot used in the fields of intelligent services, medical care, educational guidance, etc. The first multimodal data may be the original voice data, original posture data and original visual data of the user in the current state, wherein the original voice data may be acquired by a sound acquisition device, specifically by a microphone array, and the original voice data contains the acoustic characteristics of the user such as pitch, volume, and speech speed; the original visual data may be acquired by a camera device, specifically by obtaining the user's facial expression through a high-definition camera; the original posture data may be acquired by a depth sensor (such as Kinect) or other motion capture devices, and the original posture data includes the user's body posture, body language, etc. In addition, the preprocessing in step 110 may be at least one of data cleaning, data standardization, dimensionality reduction, data balancing, data normalization, etc., on the first multimodal data to obtain the second multimodal data.

[0069] In some embodiments, the step 110 of preprocessing the first multimodal data to obtain second multimodal data includes:

[0070] A1. performing data denoising on the first multimodal data to obtain intermediate denoised data;

[0071] A2. Perform data standardization on the intermediate denoised data to obtain the second multimodal data.

[0072] In the embodiment of the present application, since the first multimodal data includes original voice data, original posture data and original visual data, the data denoising in step A1 can be respectively performing voice signal denoising operations on the original voice data, performing image denoising operations on the original visual data and performing data precision improvement operations on the original posture data, so as to obtain intermediate denoised data. Step A2 can be to standardize the intermediate denoised data of each modality, unify the dimensions of each modal data, reduce the data deviation between different modal data, and thus obtain the second multimodal data.

[0073] Step 120: performing time synchronization and feature extraction on the second multimodal data to obtain third multimodal data, wherein the third multimodal data includes voice feature data, posture feature data, and visual feature data;

[0074] In an embodiment of the present application, the timing synchronization and feature extraction in step 120 may be to time align each modal data (such as speech modal data, posture modal data or visual modal data) in the second multimodal data, maintain the consistency of each modal data in the time dimension, and then extract the data features of each modal data after time alignment, so as to obtain the third multimodal data.

[0075] In some embodiments, the step 120 of performing time series synchronization and feature extraction on the second multimodal data to obtain third multimodal data includes:

[0076] B1. Acquire voice modality data, posture modality data, and visual modality data in the second multimodal data;

[0077] B2. performing time series matching on the visual modality data according to the voice modality data and the posture modality data to obtain time series matching information, wherein the time series matching information is used to characterize a time mapping relationship between the voice modality data, the posture modality data and the visual modality data;

[0078] B3. Performing timing synchronization on the second multimodal data according to the timing matching information to obtain timing synchronization data, where the timing synchronization data includes voice synchronization data, posture synchronization data, and visual synchronization data;

[0079] B4, extracting spectrum features from the voice synchronization data to obtain voice feature data, and extracting facial features from the visual synchronization data to obtain visual feature data;

[0080] B5. Extract posture features from the posture synchronization data to obtain the posture feature data.

[0081] In an embodiment of the present application, the time series alignment in step B2 is used to align the visual sequences of speech modal data, posture modal data and visual modal data. First, data timestamps can be added to the speech modal data, posture modal data and visual modal data respectively, and the data timestamps are used to characterize the acquisition time points of the corresponding modal data; then, based on the dynamic time warping (DTW) algorithm, the best matching path (i.e., time mapping relationship) between each modal data is obtained, and the timing matching information is determined based on the best matching path.

[0082] Exemplarily, for voice modality data and gesture modality data, the equivalent expression of the best matching path may be:

[0083]

[0084] Where X is the time series of speech modality data; Y is the time series of posture modality data; DTW(X,Y) is the best matching path between speech modality data and posture modality data; x i is the i-th time data point in the time series of speech modality data; y j is the jth time data point in the time series of the posture modal data; d(x i ,y j ) is the distance metric.

[0085] It is understandable that the specific measurement method of the distance metric can be any one of the Euclidean distance, Manhattan distance, Mahalanobis distance, etc. The best matching path of the remaining two modal data can be simply obtained based on the above content, and this application will not repeat it here. In addition, the timing synchronization in step B3 can be based on the obtained time mapping relationship, and each modal data is resampled or interpolated respectively, and the data points of each modal data are ensured to correspond in time by generating a synchronized time series, so as to obtain voice synchronization data, posture synchronization data and visual synchronization data.

[0086] It should be noted that the spectral feature extraction in step B4 can be to extract the spectral features of the speech synchronization data through Mel frequency cepstrum coefficients (MFCC) to obtain the speech feature data; or, the spectral feature extraction in step B4 can be to extract the pitch features of the speech synchronization data through the autocorrelation function or the simplified cepstrum method while extracting the spectral features, extract the energy features of the speech synchronization data by calculating the energy of the speech signal in a short time window, and determine the obtained spectral features, pitch features and energy features as the speech feature data. In addition, the facial feature extraction in step B4 can be to extract facial key points in the visual synchronization data using a convolutional neural network (CNN), and calculate specific facial action units based on the extracted facial key points, which reflect the movement of facial muscles, and then determine the obtained facial key points and facial action units as the visual feature data.

[0087] It is worth mentioning that the posture feature extraction in step B5 can be based on the posture estimation algorithm to obtain the user's specific posture data such as skeleton joint coordinates, joint angles and motion trajectory, and then perform zero-mean unit variance normalization and principal component analysis (PCA) dimensionality reduction on these specific posture data in turn to obtain posture feature data.

[0088] Step 130: performing self-attention feature fusion on the visual feature data according to the speech feature data and the posture feature data to obtain a fusion feature;

[0089] In an embodiment of the present application, the self-attention feature fusion in step 130 can be based on the self-attention mechanism to deeply fuse the feature data of each modality, and generate fusion features by capturing the interactive relationship between the modal data. The fusion features contain richer emotional information, which can enable the subsequent sentiment classification model to focus on important modal features and understand multi-dimensional emotional clues more comprehensively.

[0090] In some embodiments, the step 130 of performing self-attention feature fusion on the visual feature data according to the voice feature data and the posture feature data to obtain a fusion feature includes:

[0091] C1. performing a linear transformation on the speech feature data, the posture feature data and the visual feature data to obtain a multimodal feature vector set, wherein the multimodal feature vector set includes a speech feature vector, a posture feature vector and a visual feature vector;

[0092] C2. performing matrix calculation on the multimodal feature vector set to obtain a query key value matrix set corresponding to the multimodal feature vector set;

[0093] C3. Perform multi-head self-attention calculation on the query key-value matrix set to obtain the fusion feature.

[0094] In an embodiment of the present application, step C1 can be a linear transformation of the feature vectors of speech feature data, the feature vectors of posture feature data, and the feature vectors of visual feature data. Specifically, the feature vectors of speech feature data, the feature vectors of posture feature data, and the feature vectors of visual feature data can be mapped to a unified feature space, and the mapped speech feature vectors, posture feature vectors, and visual feature vectors can be determined as a multimodal feature vector set.

[0095] It is understandable that step C2 can be based on each vector element in the multimodal feature vector set, and the query (Query) matrix, key (Key) matrix and value (Value) matrix corresponding to the multimodal feature vector set are calculated, and then the query key value matrix set is obtained based on the calculated query matrix, key matrix and value matrix. Step C3 can be to adopt a multi-head mechanism, divide the query matrix, key matrix and value matrix in the query key value matrix set into several self-attention heads, and calculate the weighted attention based on several self-attention heads, and then splice the obtained weighted attention to obtain the fusion feature.

[0096] It should be noted that the multi-head self-attention in the embodiment of the present application can learn the correlation between different modal data from different subspaces, which can effectively enhance the expressive ability of the subsequent sentiment classification model; at the same time, the weighted fusion based on weighted attention can retain the important information between the modal data, avoid information loss, and improve the accuracy and robustness of emotion recognition.

[0097] Step 140: input the fusion feature into the current emotion classification model to perform emotion recognition and obtain emotion state information;

[0098] In an embodiment of the present application, step 140 may be to input the fused features into the current latest emotion classification model for emotion recognition, so as to obtain the emotional state information of the user in the current state, wherein the current latest emotion classification model may be an emotion classification model obtained by continuous learning and optimization during the process of human-computer intelligent interaction between the user and the humanoid robot, which can enable the humanoid robot to quickly adapt to the differences in emotional expression of different users, thereby improving the accuracy of emotion recognition and the degree of personalization of subsequent emotional interaction information generation, thereby improving the user experience.

[0099] In some embodiments, the step 140, the emotion classification model includes a first network framework and a second network framework, and the step of inputting the fusion feature into the current emotion classification model for emotion recognition to obtain the emotion state information includes:

[0100] D1. Inputting the fusion feature into the first network framework to extract spatial features to obtain spatial features;

[0101] D2, inputting the spatial features into the second network framework to extract the emotion time change features to obtain the emotion spatiotemporal features;

[0102] D3. Classify and identify the emotional spatiotemporal features to obtain the emotional state information.

[0103] In the embodiment of the present application, the emotion classification model is used to accurately identify the user's current emotional state from the input fusion features, and the emotional state can be happiness, sadness, anger, etc. In addition, the first network framework in the embodiment of the present application can be a convolutional neural network (CNN), and the second network framework can be a long short-term memory network (LSTM).

[0104] It is understandable that step D1 may be to extract spatial features from fused features through a convolutional neural network, and the spatial features include spatial feature information such as spatial changes in facial expressions and the structure of body postures, which can be captured specifically through convolution operations. Step D2 may be to capture the temporal dynamic changes of emotions through a long short-term memory network, and the temporal dynamic changes may be changes in user emotions over time during a conversation or interaction, and specifically may be to process the input spatial features through a long short-term memory network to obtain emotional spatiotemporal features.

[0105] It should be noted that after the emotional spatiotemporal features are extracted, the classification recognition in step D3 can firstly combine the extracted emotional spatiotemporal features through a fully connected layer, and then estimate the probability distribution of the emotional categories corresponding to the combined emotional spatiotemporal features based on the Softmax classifier, and determine the emotional category with the highest probability distribution value as the emotional state information.

[0106] Step 150: input the emotional state information into the current emotional interaction generation model to generate emotional interaction and obtain emotional interaction information.

[0107] In an embodiment of the present application, step 150 may be inputting the emotional state information into the current latest emotional interaction generation model to generate emotional interaction, so as to obtain emotional interaction information corresponding to the emotional state information of the user in the current state, wherein the emotional interaction information may include the language response and action response of the humanoid robot, etc. The current latest emotional interaction generation model may be an emotional interaction generation model obtained by continuous learning and optimization during the process of human-computer intelligent interaction between the user and the humanoid robot, which can enable the humanoid robot to quickly adapt to the differences in emotional expression of different users, thereby improving the personalization of emotional interaction information generation and improving user experience.

[0108] In some embodiments, the emotion interaction generation model includes a pre-trained language framework and a third network framework, and the step 150 of inputting the emotion state information into the current emotion interaction generation model to generate emotion interaction to obtain emotion interaction information includes:

[0109] E1. Obtain a preset action library and context information corresponding to the user.

[0110] E2. Inputting the context information and the emotional state information into the pre-trained language framework to generate language and obtain language interaction information;

[0111] E3, inputting the action library, the emotional state information and the language interaction information into the third network framework to perform action learning and generation to obtain action interaction information;

[0112] E4. Obtain the emotion interaction information according to the language interaction information and the action interaction information.

[0113] In an embodiment of the present application, the action library in step E1 can be pre-set manually, and the action library records a number of actions that can be performed by the humanoid robot; the context information can be the conversation history between the user and the humanoid robot, user preferences, situational information, etc., wherein the conversation history can be the conversation content of the user in the current human-computer intelligent interaction process, or it can be the conversation content of the user in the historical human-computer intelligent interaction process and the current human-computer intelligent interaction process, and the present application does not impose any restrictions on this.

[0114] It is understandable that the pre-trained language framework can be a large pre-trained language model. There are many specific model types, which will not be described in detail in this application. Step E2 can first convert the emotional state information into an emotional embedding vector, and then input the emotional embedding vector and context information into the pre-trained language framework to obtain language interaction information corresponding to the user's current emotional state.

[0115] It should be noted that the third network framework in step E3 can be a generative adversarial network (GAN) or a variational autoencoder (VAE), etc., which is used to generate body movements and facial expressions that match the language interaction information and emotional state information. Specifically, step E3 can first determine the corresponding emotional language embedding vector based on the emotional state information and language interaction information, and then input the emotional language embedding vector and several action features in the action library into the third network framework for action learning generation, so as to obtain the action that is most consistent with the emotional state information and language interaction information, and determine the action as the action interaction information. Step E4 can be based on the obtained language interaction information and action interaction information, and combine to obtain emotional interaction information for execution by the humanoid robot.

[0116] In some embodiments, the method further comprises:

[0117] F1. Obtain user's emotional feedback data;

[0118] F2. According to a preset reward function and the emotional feedback data, a quality assessment is performed on the emotional interaction information to obtain a reward assessment value;

[0119] F3. According to the reward evaluation value, feedback optimization is performed on the current emotion interaction generation model to obtain an optimized emotion interaction generation model;

[0120] In an embodiment of the present application, after each round of human-computer intelligent interaction, the humanoid robot can also collect the user's emotional feedback data in real time. The emotional feedback data can be the user's tone, user expression, user action, etc.; then determine the reward evaluation value of the emotional interaction information based on the preset reward function and the emotional feedback data. The preset reward function can be the reward function in the reinforcement learning mechanism, and the reward evaluation value can be the reward signal in the reinforcement learning mechanism; specifically, if the user's emotional feedback data is positive emotional feedback, then the reward evaluation value obtained after quality evaluation can be a positive reward signal; or, if the user's emotional feedback data is negative emotional feedback, then the reward evaluation value obtained after quality evaluation can be a negative reward signal.

[0121] It can be understood that step F3 can be to use the reward evaluation value as a reward signal and feed it back to the current emotion interaction generation model through a reinforcement learning mechanism, so as to optimize the emotion interaction generation strategy of the emotion interaction generation model, thereby obtaining an optimized emotion interaction generation model, and the optimized emotion interaction generation model can be used as the current emotion interaction generation model used in the next round of human-computer intelligent interaction.

[0122] It should be noted that a certain human-machine intelligent interaction process between a user and a humanoid robot usually includes multiple rounds of human-machine intelligent interaction. The embodiment of the present application is based on steps F1 to F3, which can enable the emotional interaction generation model to dynamically adjust the emotional interaction generation strategy according to the user's real-time feedback, which can continuously improve the interaction effect of the humanoid robot in the human-machine intelligent interaction process, can achieve natural and personalized emotional interaction, and effectively improve the user experience. Exemplarily, if the current human-machine intelligent interaction is the third round of human-machine intelligent interaction, the current emotional interaction generation model can be obtained after feedback optimization of the emotional interaction generation model in the second round of human-machine intelligent interaction, and specifically can be obtained by feedback optimization of the emotional interaction generation model corresponding to the second round of human-machine intelligent interaction based on the reward evaluation value corresponding to the second round of human-machine intelligent interaction. The example of this application is for illustration only and does not limit this application.

[0123] The method further comprises:

[0124] G1. Obtain the user's historical emotion information and historical interaction data;

[0125] G2. According to the historical sentiment information and the historical interaction data, fine-tune and optimize the current sentiment classification model to obtain an optimized sentiment classification model.

[0126] In an embodiment of the present application, the historical emotional information may be the emotional information of the user in the previous rounds of human-computer intelligent interaction during the current human-computer intelligent interaction process, and the historical interaction information is similar to the historical emotional information. The model fine-tuning optimization in step G2 may first be based on a meta-learning mechanism or a few-sample learning technique, by using the MAML (Model-Agnostic Meta-Learning) framework to fine-tune the parameters of the current latest emotion classification model based on the historical emotional information and historical interaction information in the previous rounds of human-computer intelligent interaction, so that the emotion classification model is more sensitive to the emotional expression of the current specific user, thereby obtaining an optimized emotion classification model, which is similar to the content of the aforementioned optimized emotion interaction generation model, and can be simply deduced, and this application will not go into details here.

[0127] The following describes in detail an emotion analysis and processing system for a humanoid robot proposed in accordance with an embodiment of the present application with reference to the accompanying drawings.

[0128] Reference Figure 2 , a humanoid robot emotion analysis and processing system proposed in an embodiment of the present application comprises:

[0129] The first processing unit 101 is used to obtain first multimodal data of a user, and preprocess the first multimodal data to obtain second multimodal data;

[0130] A second processing unit 102 is used to perform time synchronization and feature extraction on the second multimodal data to obtain third multimodal data, where the third multimodal data includes voice feature data, posture feature data, and visual feature data;

[0131] A third processing unit 103 is used to perform self-attention feature fusion on the visual feature data according to the voice feature data and the posture feature data to obtain a fusion feature;

[0132] The fourth processing unit 104 is used to input the fusion feature into the current emotion classification model to perform emotion recognition and obtain emotion state information;

[0133] The fifth processing unit 105 is used to input the emotional state information into the current emotional interaction generation model to generate emotional interaction and obtain emotional interaction information.

[0134] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0135] Reference Figure 3 , the embodiment of the present application further provides an electronic device, including:

[0136] at least one processor 201;

[0137] At least one memory 202, used to store at least one program;

[0138] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the above method embodiment.

[0139] Similarly, it can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0140] The embodiment of the present application further provides a computer-readable storage medium, in which a program executable by the processor 201 is stored. The program executable by the processor 201 is used to implement the above-mentioned method embodiment when executed by the processor 201.

[0141] Similarly, the contents of the above method embodiments are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0142] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part described as a larger operation is performed independently.

[0143] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0144] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0145] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0146] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0147] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0148] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0149] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0150] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Technical personnel familiar with the field can make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A method for analyzing and processing emotions of a humanoid robot, characterized in that: include: Acquire first multimodal data of a user, and preprocess the first multimodal data to obtain second multimodal data; Performing time synchronization and feature extraction on the second multimodal data to obtain third multimodal data, wherein the third multimodal data includes voice feature data, posture feature data, and visual feature data; According to the speech feature data and the posture feature data, performing self-attention feature fusion on the visual feature data to obtain a fusion feature; Inputting the fused features into the current emotion classification model to perform emotion recognition and obtain emotion state information; The emotional state information is input into the current emotional interaction generation model to generate emotional interaction and obtain emotional interaction information.

2. The method according to claim 1, characterized in that: The preprocessing of the first multimodal data to obtain second multimodal data includes: Performing data denoising on the first multimodal data to obtain intermediate denoised data; The intermediate denoised data is subjected to data normalization to obtain the second multimodal data.

3. The method according to claim 1, characterized in that The performing time series synchronization and feature extraction on the second multimodal data to obtain third multimodal data includes: Acquire voice modality data, posture modality data, and visual modality data in the second multimodal data; According to the voice modality data and the posture modality data, time series matching is performed on the visual modality data to obtain time series matching information, wherein the time series matching information is used to characterize a time mapping relationship between the voice modality data, the posture modality data, and the visual modality data; According to the timing matching information, timing synchronization is performed on the second multimodal data to obtain timing synchronization data, where the timing synchronization data includes voice synchronization data, posture synchronization data, and visual synchronization data; Extracting frequency spectrum features from the speech synchronization data to obtain speech feature data, and extracting facial features from the visual synchronization data to obtain visual feature data; Performing posture feature extraction on the posture synchronization data to obtain the posture feature data.

4. The method according to claim 1, characterized in that: The step of performing self-attention feature fusion on the visual feature data according to the voice feature data and the posture feature data to obtain a fusion feature comprises: Performing a linear transformation on the speech feature data, the posture feature data and the visual feature data to obtain a multimodal feature vector set, wherein the multimodal feature vector set includes a speech feature vector, a posture feature vector and a visual feature vector; Performing matrix calculation on the multimodal feature vector set to obtain a query key value matrix set corresponding to the multimodal feature vector set; Multi-head self-attention calculation is performed on the query key-value matrix set to obtain the fusion feature.

5. The method according to claim 1, characterized in that The emotion classification model includes a first network framework and a second network framework, and the fusion feature is input into the current emotion classification model for emotion recognition to obtain emotion state information, including: Inputting the fused features into the first network framework to extract spatial features to obtain spatial features; Inputting the spatial features into the second network framework to extract emotional time-varying features to obtain emotional spatiotemporal features; The emotional spatiotemporal features are classified and identified to obtain the emotional state information.

6. The method according to claim 1, characterized in that The emotion interaction generation model includes a pre-trained language framework and a third network framework, and the emotion state information is input into the current emotion interaction generation model to generate emotion interaction, thereby obtaining emotion interaction information, including: Obtaining a preset action library and context information corresponding to the user, Inputting the context information and the emotional state information into the pre-trained language framework to perform language generation to obtain language interaction information; Inputting the action library, the emotional state information and the language interaction information into the third network framework to perform action learning and generation to obtain action interaction information; The emotion interaction information is obtained according to the language interaction information and the action interaction information.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Obtain users’ emotional feedback data; According to a preset reward function and the emotional feedback data, the quality of the emotional interaction information is evaluated to obtain a reward evaluation value; According to the reward evaluation value, feedback optimization is performed on the current emotion interaction generation model to obtain an optimized emotion interaction generation model; The method further comprises: Acquire historical emotion information and historical interaction data of the user; According to the historical emotion information and the historical interaction data, the current emotion classification model is fine-tuned and optimized to obtain an optimized emotion classification model.

8. A humanoid robot emotion analysis and processing system, characterized in that: include: A first processing unit, configured to obtain first multimodal data of a user, and preprocess the first multimodal data to obtain second multimodal data; a second processing unit, configured to perform time synchronization and feature extraction on the second multimodal data to obtain third multimodal data, wherein the third multimodal data includes voice feature data, posture feature data, and visual feature data; A third processing unit is used to perform self-attention feature fusion on the visual feature data according to the voice feature data and the posture feature data to obtain a fusion feature; A fourth processing unit, configured to input the fusion feature into a current emotion classification model to perform emotion recognition and obtain emotion state information; The fifth processing unit is used to input the emotional state information into the current emotional interaction generation model to generate emotional interaction and obtain emotional interaction information.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.

Images