Emotion interaction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an emotion interaction method and device, electronic equipment and a storage medium, the method is applied to a central control screen, and the method comprises the following steps: determining facial features, voice features and physiological features of a user; performing feature classification on the facial features, the voice features and the physiological features to obtain a plurality of classified emotion features; determining feature vectors and emotion prediction results respectively corresponding to the plurality of emotion features; determining a target emotion according to the feature vectors corresponding to the plurality of emotion features and the emotion prediction result; and performing corresponding interaction according to the target emotion. Therefore, the central control screen can perform emotion interaction with the user to meet the demand of the user, and the central control screen determines the target emotion according to the feature vectors corresponding to the plurality of emotion features and the emotion prediction result, so that the accuracy of the target emotion is improved, the interaction is further more accurate, and the use experience of the user is improved.

Description

Technical Field

[0001] The present invention relates to the field of information interaction technology, and in particular to an emotion interaction method, an emotion interaction device, an electronic device and a computer-readable storage medium. Background Art

[0002] With the development of artificial intelligence technology, the scenarios of human-computer interaction are becoming more and more diverse, and are also gradually evolving towards a more natural and humanized direction. As an advanced form of human-computer interaction, emotional interaction provides users with more intimate and personalized services. However, the human-computer emotional interaction technology in related technologies is more applied to specific scenarios, such as intelligent customer service, virtual assistants, etc., and its application on smart central control screens is not yet mature. Summary of the invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide an emotion interaction method, an emotion interaction device, an electronic device and a computer-readable storage medium that overcome the above problems or at least partially solve the above problems.

[0004] In order to solve the above problems, an embodiment of the present invention discloses an emotional interaction method, which is applied to a central control screen and includes:

[0005] Determine the user's facial, voice, and physiological characteristics;

[0006] Performing feature classification on the facial features, the voice features, and the physiological features to obtain a plurality of classified emotion features;

[0007] Determine feature vectors and emotion prediction results corresponding to the plurality of emotion features respectively;

[0008] Determining a target emotion according to the feature vectors corresponding to the plurality of emotion features and the emotion prediction results;

[0009] Corresponding interactions are performed according to the target emotion.

[0010] Optionally, determining the target emotion according to the feature vectors respectively corresponding to the plurality of emotion features and the emotion prediction results comprises:

[0011] Merging the feature vectors corresponding to the multiple emotional features to obtain a target emotional feature vector;

[0012] Fusing the emotion prediction results corresponding to the multiple emotion features to obtain an emotion fusion prediction result;

[0013] Prediction is performed based on the target emotion feature vector and the emotion fusion prediction result to obtain the target emotion.

[0014] Optionally, determining the feature vectors and the emotion prediction results respectively corresponding to the plurality of emotion features comprises:

[0015] Determine feature vectors corresponding to the plurality of emotional features respectively;

[0016] The feature vectors are predicted respectively to obtain emotion prediction results corresponding to the multiple emotion features respectively.

[0017] Optionally, the central control screen is connected to a smart home device; and the corresponding interaction according to the target emotion includes:

[0018] According to the target emotion, determining a control instruction corresponding to the target emotion;

[0019] The smart home device is controlled according to the control instruction.

[0020] Optionally, performing corresponding interaction according to the target emotion includes:

[0021] According to the target emotion, interactive speech and recommended content corresponding to the target emotion are determined, and the interactive speech and the recommended content are output to the user; the recommended content is a way to control the emotion of the user and a way to control the physiological health of the user.

[0022] Optionally, the central control screen is provided with a camera and a microphone; the central control screen is connected to a wearable device; and the determining of the user's facial features, voice features, and physiological features includes:

[0023] Acquire the user's facial expression through the camera, and determine the user's facial features through the facial expression;

[0024] Acquiring voice information of the user through the microphone, and determining voice features of the user through the voice information;

[0025] Acquire the physiological signal of the user sent by the wearable device, and determine the physiological characteristics of the user through the physiological signal.

[0026] Optionally, determining the facial features of the user through the facial expression includes:

[0027] A convolutional neural network is used to extract features of the facial expression to obtain facial features of the user.

[0028] Optionally, determining the voice feature of the user through the voice information includes:

[0029] A preset speech feature extraction model is used to extract features from the speech information to obtain the speech features of the user; the preset speech feature extraction model extracts features from sample speech information through a Mel-frequency cepstral coefficient algorithm to obtain sample speech features, and a pre-trained model is trained through the sample speech features to obtain the preset speech feature extraction model.

[0030] Optionally, determining the physiological characteristics of the user through the physiological signal includes:

[0031] Preprocessing the physiological signal to obtain a preprocessed physiological signal;

[0032] Performing normalization processing on the preprocessed physiological signal to obtain a normalized physiological signal;

[0033] Feature extraction is performed on the normalized physiological signal to obtain the physiological features of the user.

[0034] The embodiment of the present invention further provides an emotional interaction device, which is applied to a central control screen and includes:

[0035] A first determination module, used to determine the user's facial features, voice features and physiological features;

[0036] A classification module, used for classifying the facial features, the voice features and the physiological features to obtain a plurality of classified emotion features;

[0037] A second determination module is used to determine feature vectors and emotion prediction results corresponding to the plurality of emotion features respectively;

[0038] A third determination module, configured to determine a target emotion according to the feature vectors corresponding to the plurality of emotion features and the emotion prediction results;

[0039] The interaction module is used to perform corresponding interactions according to the target emotion.

[0040] Optionally, the third determining module includes:

[0041] A merging submodule, used for merging the feature vectors corresponding to the multiple emotional features to obtain a target emotional feature vector;

[0042] A fusion submodule, used for fusing the emotion prediction results corresponding to the multiple emotion features to obtain an emotion fusion prediction result;

[0043] The first prediction submodule is used to make predictions based on the target emotion feature vector and the emotion fusion prediction result to obtain the target emotion.

[0044] Optionally, the second determining module includes:

[0045] A first determination submodule is used to determine feature vectors corresponding to the plurality of emotional features respectively;

[0046] The second prediction submodule is used to predict the feature vectors respectively to obtain emotion prediction results corresponding to the multiple emotion features respectively.

[0047] Optionally, the central control screen is connected to a smart home device; the interaction module includes:

[0048] A second determination submodule is used to determine a control instruction corresponding to the target emotion according to the target emotion;

[0049] The control submodule is used to control the smart home device according to the control instruction.

[0050] Optionally, the interaction module includes:

[0051] The output submodule is used to determine the interactive voice and recommended content corresponding to the target emotion according to the target emotion, and output the interactive voice and the recommended content to the user; the recommended content is a way to control the user's emotion and a way to control the user's physiological health.

[0052] Optionally, the central control screen is provided with a camera and a microphone; the central control screen is connected to a wearable device; and the first determination module includes:

[0053] A first acquisition submodule, used to acquire the user's facial expression through the camera, and determine the user's facial features through the facial expression;

[0054] A second acquisition submodule, configured to acquire voice information of the user through the microphone, and determine voice features of the user through the voice information;

[0055] The third acquisition submodule is used to acquire the physiological signal of the user sent by the wearable device, and determine the physiological characteristics of the user through the physiological signal.

[0056] Optionally, the first acquisition submodule includes:

[0057] The first extraction unit is used to extract features of the facial expression using a convolutional neural network to obtain facial features of the user.

[0058] Optionally, the second acquisition submodule includes:

[0059] The second extraction unit is used to use a preset voice feature extraction model to extract features from the voice information to obtain the voice features of the user; the preset voice feature extraction model uses a Mel-frequency cepstral coefficient algorithm to extract features from sample voice information to obtain sample voice features, and trains a pre-trained model using the sample voice features to obtain the preset voice feature extraction model.

[0060] Optionally, the third acquisition submodule includes:

[0061] A preprocessing unit, used for preprocessing the physiological signal to obtain a preprocessed physiological signal;

[0062] A normalization unit, used for performing normalization processing on the preprocessed physiological signal to obtain a normalized physiological signal;

[0063] The third extraction unit is used to extract features from the normalized physiological signal to obtain the physiological features of the user.

[0064] The present invention also discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned emotional interaction method when executing the computer program.

[0065] The present invention also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned emotional interaction method are implemented.

[0066] The embodiments of the present invention include the following advantages:

[0067] In the embodiment of the present invention, the central control screen first determines the facial features, voice features and physiological features of the user, and then classifies the facial features, voice features and physiological features to obtain multiple emotional features after classification, and then determines the feature vectors and emotional prediction results corresponding to the multiple emotional features, and finally determines the target emotion according to the feature vectors and emotional prediction results corresponding to the multiple emotional features, and performs corresponding interaction according to the target emotion, so that the central control screen can emotionally interact with the user to meet the needs of the user, and the central control screen determines the target emotion according to the feature vectors and emotional prediction results corresponding to the multiple emotional features, thereby improving the accuracy of the target emotion, further making the interaction more accurate, and increasing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 is a flowchart of a method for emotional interaction provided by an embodiment of the present invention;

[0069] Figure 2This is a structural block diagram of a central control screen provided by an embodiment of the present invention;

[0070] Figure 3 It is a structural block diagram of an emotional interaction device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0071] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0072] The human-computer emotional interaction technology in the related technology is more applied to specific scenarios, such as intelligent customer service, virtual assistants, etc., and its application on the intelligent central control screen is not yet mature. In order to solve the above technical problems, the present invention provides an emotional interaction method, the core concept of which is that the central control screen first determines the user's facial features, voice features and physiological features, and then classifies the facial features, voice features and physiological features to obtain multiple emotional features after classification, and then determines the feature vectors and emotional prediction results corresponding to the multiple emotional features, and finally determines the target emotion according to the feature vectors and emotional prediction results corresponding to the multiple emotional features, and interacts accordingly according to the target emotion, so that the central control screen can interact emotionally with the user to meet the user's needs, and the central control screen determines the target emotion according to the feature vectors and emotional prediction results corresponding to the multiple emotional features, which improves the accuracy of the target emotion, further makes the interaction more accurate, and increases the user's experience.

[0073] Reference Figure 1 , shows a flow chart of the steps of an emotional interaction method provided by an embodiment of the present invention, the method is applied to a central control screen, and the method may specifically include the following steps:

[0074] Step 101, determining the user's facial features, voice features, and physiological features.

[0075] In an embodiment of the present invention, when a user interacts with the central control screen, the central control screen can determine the user's facial features, voice features, and physiological features.

[0076] In one embodiment, the central control screen is provided with a camera and a microphone; the central control screen is connected to the wearable device; determining the facial features, voice features and physiological features of the user may include: obtaining the user's facial expressions through the camera, and determining the user's facial features through the facial expressions; obtaining the user's voice information through the microphone, and determining the user's voice features through the voice information; obtaining the user's physiological signals sent by the wearable device, and determining the user's physiological characteristics through the physiological signals.

[0077] Specifically, the central control screen in the present invention may be provided with a camera and a microphone, and the user may wear a wearable device, such as a wearable bracelet, etc. The wearable device is used to detect the user's physiological signals, and the wearable device is connected to the central control screen, and sends the detected user's physiological signals to the central control screen in real time. When the user interacts with the central control screen, the central control screen can obtain the user's facial expressions during the interaction according to the camera, obtain the user's voice information during the interaction according to the microphone, obtain the user's physiological characteristics during the interaction according to the wearable device, and then determine the user's facial features through facial expressions, determine the user's voice features through voice information, and determine the user's physiological characteristics through physiological signals.

[0078] like Figure 2 As shown, a structural block diagram of a central control screen provided by an embodiment of the present invention is shown. A camera and a microphone are provided on the central control screen. The user wears a wearable device. The central control screen and the wearable device are connected via WIFI (Wireless Fidelity) or BT (Bluetooth) to transmit physiological signals detected by the wearable device. The camera obtains the user's facial expression, and the microphone obtains the user's voice information. The central control screen analyzes the facial expression, the voice information, and the physiological signals to obtain the user's facial features, voice features, and physiological features.

[0079] In one embodiment, determining the facial features of a user through facial expressions may include: extracting features of facial expressions using a convolutional neural network to obtain the facial features of the user.

[0080] Specifically, the central control screen in the present invention has a built-in emotion recognition module, and a convolutional neural network is provided in the emotion recognition module. After the central control screen obtains the facial expression, the convolutional neural network can be used to extract features of the facial expression to obtain the facial features of the user.

[0081] In the present invention, the convolutional neural network is a deep learning model, which is particularly suitable for processing image data. In terms of facial feature extraction, the convolutional neural network can automatically learn useful features from the image, which can be used for tasks such as facial recognition, expression analysis, and age estimation. The core components of the convolutional neural network include convolutional layers, pooling layers, and fully connected layers: Convolutional layer: The input image is convolved through a convolution kernel (filter) to extract local features (such as edges, textures, etc.). Pooling layer: Downsample the output of the convolution layer to reduce the data dimension while retaining important features. Fully connected layer: Map the extracted features to the final output (such as classification results). The process of facial feature extraction: (1) Input image preprocessing: Adjust the facial image to a fixed size (such as 224x224 pixels). Normalize pixel values ​​(such as scaling pixel values ​​to [0,1] or [-1,1]). Align the face (such as using face detection and alignment algorithms to ensure that the positions of facial features in the image are consistent). (2) Convolutional layer feature extraction: The convolution layer extracts local features of the image through multiple convolution kernels. The low-level convolutional layers extract low-level features such as edges and textures. The middle-level convolutional layers extract intermediate features such as eyes, noses, and mouths. The high-level convolutional layers extract high-level features such as the overall facial structure. The size and number of convolution kernels determine the complexity and expressiveness of the features. (3) Pooling layer dimensionality reduction: The pooling layer (such as maximum pooling or average pooling) downsamples the output of the convolutional layer to reduce the data dimension while retaining important features. The pooling operation helps improve the robustness of the model and reduce overfitting. (4) Fully connected layer output features: After multiple layers of convolution and pooling, the feature map is flattened into a one-dimensional vector. The fully connected layer maps these features to the final output (such as a facial feature vector).

[0082] The following are some commonly used deep learning models for facial feature extraction: VGGFace: A facial feature extraction model based on the VGG network. FaceNet: Directly learn facial feature embedding using triplet loss. DeepFace: A facial recognition model proposed by Facebook that combines CNN and traditional feature extraction methods. ArcFace: An angle-based loss function for learning more discriminative facial features. MobileFaceNet: A lightweight facial feature extraction model designed specifically for mobile devices.

[0083] In one embodiment, determining the user's voice features through voice information can include: using a preset voice feature extraction model to extract features from the voice information to obtain the user's voice features; using the preset voice feature extraction model to extract features from sample voice information through a Mel-frequency cepstral coefficient algorithm to obtain sample voice features, and training a pre-trained model through the sample voice features to obtain a preset voice feature extraction model.

[0084] Mel-frequency cepstral coefficient algorithm is a feature extraction method used in audio signal processing and speech recognition. It can extract features related to human auditory perception from audio signals and is particularly suitable for tasks such as speech recognition, speaker recognition, and music classification.

[0085] Specifically, in the present invention, a preset voice feature extraction model is provided in the central control screen, and the preset voice feature extraction model is trained in the following manner: a large amount of audio and voice data is collected, and then the audio and voice data is labeled and classified according to the acoustic features to obtain sample voice information, and the sample voice features are extracted by the Mel-frequency cepstral coefficient algorithm to obtain sample voice features, and the sample voice features are segmented and recombined into different feature vectors according to the sample voice features, and these feature vectors are sequenced to obtain sequence data, and the sequence data is processed by the pre-trained model to complete the training of the pre-trained model, so as to obtain the preset voice feature extraction model. The pre-trained model can be a recurrent neural network or a long short-term memory network. The specific pre-trained model can be selected according to actual needs, and the present invention does not make specific limitations here. After the central control screen obtains the user's voice information, the preset voice feature extraction model can be used to extract features from the voice information to obtain the user's voice features.

[0086] Recurrent Neural Networks (RNNs) are a type of neural network specifically designed for processing sequence data. Unlike traditional feedforward neural networks, RNNs have an internal memory mechanism that can capture the temporal dependencies in sequence data. This makes RNNs perform well when processing time series data (such as speech, text, etc.). The core idea of ​​RNNs is to process sequence data through a loop structure. At each time step, RNNs receive the current input and the hidden state of the previous time step, and output the hidden state and output of the current time step. The advantages of RNNs are: Processing sequence data: It can capture the temporal dependencies in the sequence. Shared parameters: The model has fewer parameters and is computationally efficient. Flexibility: It can process sequence data of any length.

[0087] Long Short-Term Memory (LSTM) is a special type of recurrent neural network (RNN) that aims to solve the gradient vanishing and gradient exploding problems in standard RNNs, thereby better capturing long-term dependencies in sequence data. LSTM controls the flow of information by introducing a "gate" mechanism (input gate, forget gate, output gate) to selectively retain or forget information. This mechanism enables LSTM to effectively process long sequence data and capture long-term dependencies. Structure of LSTM: The core of LSTM is the cell state, which is like a conveyor belt that runs through the entire sequence and only adds or deletes information through gate operations. (1) Cell State: The cell state CtCt is the memory part of the LSTM and is responsible for storing long-term information. Through the operation of the gate, the cell state can transfer information between different time steps. (2) Gate Mechanism: The LSTM contains three gates: Forget Gate: Determines which information needs to be deleted from the cell state. Input Gate: Determines which new information needs to be added to the cell state. Output Gate: Determines which information in the cell state needs to be output. The characteristics of LSTM networks are: (1) Long-term dependency. LSTM can effectively capture long-term dependencies in long sequences through cell states and gate mechanisms. (2) Gradient problem relief. The gate mechanism of LSTM networks makes it easier for gradients to remain stable during backpropagation, thereby alleviating the gradient vanishing and gradient exploding problems. (3) Flexibility. LSTM networks can process sequence data of any length and can adjust the gate operation according to task requirements. The advantages of LSTM networks are: Long-term dependency: It can capture long-term dependencies in long sequences. Gradient problem relief: The gate mechanism alleviates the gradient vanishing and gradient exploding problems. Flexibility: It can process sequence data of any length.

[0088] In one embodiment, determining the physiological characteristics of a user through physiological signals may include: preprocessing the physiological signals to obtain preprocessed physiological signals; normalizing the preprocessed physiological signals to obtain normalized physiological signals; and extracting features from the normalized physiological signals to obtain the physiological characteristics of the user.

[0089] Specifically, in the present invention, after the central control screen acquires the physiological signal of the user sent by the wearable device, it can first pre-process the physiological signal of the user to obtain the pre-processed physiological signal. For example, after the central control screen acquires the physiological signal of the user, it can filter out the environmental noise and other interference signals in the physiological signal to obtain the pre-processed physiological signal. Then the central control screen normalizes the pre-processed physiological signal to obtain the normalized physiological signal. For example, the central control screen adjusts the baseline of the pre-processed physiological signal and performs value normalization. Finally, the central control screen extracts features from the normalized physiological signal to obtain the physiological features of the user.

[0090] In the present invention, filtering out environmental noise and other interference signals in physiological signals is a key step in physiological signal processing. Physiological signals (such as electrocardiogram (ECG), electroencephalogram (EEG), electromyogram (EMG), etc.) are usually interfered by various noises, including power frequency noise, baseline drift, electromyographic interference, motion artifacts, etc. In order to obtain high-quality physiological signals, preprocessing is usually required, which mainly includes the following steps: 1. Main steps of preprocessing: (1) Signal acquisition: Use a sensor with a high signal-to-noise ratio to collect physiological signals. Ensure that the sensor is in good contact with the skin to reduce contact noise. (2) Filtering: Filtering is the core step of noise removal. Commonly used filtering methods include: Low-pass filtering: Remove high-frequency noise (such as electromyographic interference). High-pass filtering: Remove low-frequency noise (such as baseline drift). Bandpass filtering: Remove high-frequency and low-frequency noise at the same time. Notch filtering: Remove noise of a specific frequency (such as 50Hz or 60Hz power frequency noise). (3) Detrending: Remove trend items in the signal (such as baseline drift). Method: Use high-pass filtering or polynomial fitting. (4) Noise reduction: Use noise reduction algorithms to remove random noise. Methods: wavelet transform, empirical mode decomposition (EMD), independent component analysis (ICA), etc. (5) Artifact removal: Remove motion artifacts, electrooculogram artifacts, etc. Methods: ICA, template matching, threshold method, etc. (6) Resampling: If the signal sampling rate is too high, downsampling can be performed to reduce the amount of data. Method: Downsampling after low-pass filtering. 2. Common noise and its processing methods: (1) Power frequency noise: Source: power line interference. Processing method: notch filter (such as 50Hz or 60Hz notch). Use differential amplifier to reduce common mode interference. (2) Baseline drift: Source: breathing, poor electrode contact, etc. Processing method: high-pass filter (such as 0.5Hz-1Hz). Wavelet transform to remove low-frequency components. (3) Electromyographic interference: Source: muscle activity. Processing method: low-pass filter (such as 30Hz-50Hz). Use ICA to separate electromyographic signals. (4) Motion artifacts: Source: electrode movement or poor skin contact. Processing method: Use ICA to separate artifacts. Re-fix the electrodes. (5) Random noise: Source: sensor noise, environmental noise, etc. Processing method: Wavelet transform noise reduction. Mean filter or median filter. 3. Common filters: (1) Low-pass filter: Purpose: Remove high-frequency noise (such as electromyographic interference). Design: Butterworth filter, Chebyshev filter, etc. Cut-off frequency: Select according to signal characteristics (such as the cut-off frequency of ECG is usually 30Hz-50Hz). (2) High-pass filter: Purpose: Remove low-frequency noise (such as baseline drift). Design: Butterworth filter, Chebyshev filter, etc. Cut-off frequency: Select according to signal characteristics (such as the cut-off frequency of ECG is usually 0.5Hz-1Hz). (3) Band-pass filter: Purpose: Remove high-frequency and low-frequency noise at the same time. Design: Butterworth filter, Chebyshev filter, etc.Frequency range: Select according to the signal characteristics (e.g. the frequency range of ECG is usually 0.5Hz-50Hz). (4) Notch filter: Purpose: Remove noise of a specific frequency (e.g. 50Hz or 60Hz power frequency noise). Design: Band-stop filter. Notch frequency: 50Hz or 60Hz. 4. Advanced noise reduction methods: (1) Wavelet transform: Principle: Decompose the signal into components of different scales and remove the noise component. Advantages: Can effectively remove random noise and baseline drift. Application: Noise reduction of ECG, EEG and other signals. (2) Empirical mode decomposition: Principle: Decompose the signal into multiple intrinsic mode functions (IMFs) and remove the noise component. Advantages: Applicable to nonlinear and non-stationary signals. Application: Noise reduction of EEG, ECG and other signals. (3) Independent component analysis: Principle: Decompose the signal into independent components and separate noise and useful signals. Advantages: Can effectively remove artifacts (e.g. electrooculographic artifacts, electromyographic interference). Application: Artifact removal of EEG, ECG and other signals. 5. Preprocessed signal analysis: Preprocessed physiological signals can be used for further analysis and processing, such as: Feature extraction: Extracting the time domain, frequency domain, or time-frequency domain features of the signal. Classification: Using machine learning or deep learning methods to classify signals (such as arrhythmia detection). Pattern recognition: Identifying specific patterns in the signal (such as heart rate variability analysis).

[0091] Step 102, performing feature classification on the facial features, the voice features and the physiological features to obtain a plurality of classified emotion features.

[0092] In an embodiment of the present invention, after determining the facial features, voice features and physiological features of the user, the central control screen can perform feature classification on the facial features, voice features and physiological features, thereby obtaining a plurality of emotion features after classification.

[0093] Step 103: Determine feature vectors and emotion prediction results corresponding to the plurality of emotion features respectively.

[0094] In an embodiment of the present invention, facial features, voice features and physiological features are classified on the central control screen to obtain a plurality of classified emotional features, and then feature vectors and emotional prediction results corresponding to the plurality of emotional features can be determined.

[0095] In one embodiment, determining feature vectors and emotion prediction results corresponding to multiple emotion features respectively may include: determining feature vectors corresponding to multiple emotion features respectively; predicting the feature vectors respectively to obtain emotion prediction results corresponding to the multiple emotion features respectively.

[0096] Specifically, in the present invention, the central control screen can first determine the feature vectors corresponding to multiple emotional features, and then perform emotional prediction on the multiple feature vectors respectively, so as to obtain emotional prediction results corresponding to the multiple emotional features. For example: the multiple emotional features are emotional feature 1, emotional feature 2, emotional feature 3, and emotional feature 4. It is possible to first determine feature vector 1 corresponding to emotional feature 1, feature vector 2 corresponding to emotional feature 2, feature vector 3 corresponding to emotional feature 3, and feature vector 4 corresponding to emotional feature 4, and then perform emotional prediction on feature vector 1 to obtain emotional prediction result 1 corresponding to emotional feature 1, perform emotional prediction on feature vector 2 to obtain emotional prediction result 2 corresponding to emotional feature 2, perform emotional prediction on feature vector 3 to obtain emotional prediction result 3 corresponding to emotional feature 3, and perform emotional prediction on feature vector 4 to obtain emotional prediction result 4 corresponding to emotional feature 4.

[0097] Step 104: determining a target emotion according to the feature vectors respectively corresponding to the plurality of emotion features and the emotion prediction results.

[0098] In an embodiment of the present invention, after determining the feature vectors and emotion prediction results corresponding to multiple emotional features, the central control screen can determine the target emotion based on the feature vectors and emotion prediction results corresponding to the multiple emotional features.

[0099] In one embodiment, determining a target emotion based on feature vectors and emotion prediction results corresponding to a plurality of emotion features may include: merging feature vectors corresponding to a plurality of emotion features to obtain a target emotion feature vector; fusing emotion prediction results corresponding to a plurality of emotion features to obtain an emotion fusion prediction result; and performing prediction based on the target emotion feature vector and the emotion fusion prediction result to obtain the target emotion.

[0100] Specifically, when determining the target emotion, the central control screen can first merge the feature vectors corresponding to multiple emotional features to obtain the target emotional feature vector, for example, merge feature vector 1 corresponding to emotional feature 1, feature vector 2 corresponding to emotional feature 2, feature vector 3 corresponding to emotional feature 3, and feature vector 4 corresponding to emotional feature 4 to obtain the target emotional feature vector A. Fusion is performed on the emotional prediction results corresponding to multiple emotional features to obtain an emotional fusion prediction result, for example, fusion is performed on the emotional prediction result 1 corresponding to emotional feature 1, the emotional prediction result 2 corresponding to emotional feature 2, the emotional prediction result 3 corresponding to emotional feature 3, and the emotional prediction result 4 corresponding to emotional feature 4 to obtain an emotional fusion prediction result a. Finally, prediction is performed based on the target emotional feature vector and the emotional fusion result to obtain the target emotion, for example, prediction is performed based on the target emotional feature vector A and the emotional fusion result a, and the target emotion obtained is the emotional state of the user during the interaction.

[0101] Step 105: Perform corresponding interaction according to the target emotion.

[0102] In an embodiment of the present invention, after determining the target emotion of the user, the central control screen can perform corresponding interactions according to the target emotion.

[0103] In one embodiment, the central control screen is connected to the smart home device; corresponding interaction is performed according to the target emotion, including: determining a control instruction corresponding to the target emotion according to the target emotion; and controlling the smart home device according to the control instruction.

[0104] Specifically, after determining the user's target emotion, the central control screen can determine the control instruction corresponding to the target emotion from multiple preset control instructions based on the target emotion, and control the smart home devices according to the control instruction. For example, when it is determined that the user's target emotion is in a happy state, the smart speaker can be controlled to play a cheerful song according to the target emotion of the happy state.

[0105] In one embodiment, corresponding interactions are performed according to the target emotion, including: determining interactive voice and recommended content corresponding to the target emotion according to the target emotion, and outputting the interactive voice and recommended content to the user; the recommended content is a way to control the user's emotions and a way to control the user's physiological health.

[0106] Specifically, after determining the user's target emotion, the central control screen can determine the interactive voice and recommended content corresponding to the target emotion according to the target emotion, and output the interactive voice and recommended content to achieve interaction with the user. For example, if the target emotion is an unhappy state, then the corresponding soothing voice can be determined to be output to the user for interaction, and the corresponding recommended content on how to adjust the mood can also be determined to be output to the user for interaction. For another example, if the target emotion is a nervous state, then the corresponding interactive voice and recommended content on how to control the heartbeat can be determined to be output to the user for interaction.

[0107] In the embodiment of the present invention, the facial features, voice features and physiological features of the user are determined; the facial features, voice features and physiological features are feature classified to obtain multiple emotional features after classification; the feature vectors and emotional prediction results corresponding to the multiple emotional features are determined; the target emotion is determined according to the feature vectors and emotional prediction results corresponding to the multiple emotional features; and the corresponding interaction is performed according to the target emotion. Thus, the central control screen can interact with the user emotionally to meet the needs of the user, and the central control screen determines the target emotion according to the feature vectors and emotional prediction results corresponding to the multiple emotional features, thereby improving the accuracy of the target emotion, further making the interaction more accurate, and increasing the user's experience.

[0108] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0109] Reference Figure 3 , shows a structural block diagram of an emotional interaction device provided by an embodiment of the present invention, the device is applied to a central control screen, and the device may specifically include the following modules:

[0110] A first determination module 301 is used to determine the facial features, voice features and physiological features of the user;

[0111] A classification module, used for classifying the facial features, the voice features and the physiological features to obtain a plurality of classified emotion features;

[0112] A second determination module 302, used to determine feature vectors and emotion prediction results corresponding to the plurality of emotion features respectively;

[0113] A third determination module 303 is used to determine a target emotion according to the feature vectors corresponding to the plurality of emotion features and the emotion prediction results;

[0114] The interaction module 304 is used to perform corresponding interactions according to the target emotion.

[0115] In one embodiment, the third determining module 303 includes:

[0116] A merging submodule, used for merging the feature vectors corresponding to the multiple emotional features to obtain a target emotional feature vector;

[0117] A fusion submodule, used for fusing the emotion prediction results corresponding to the multiple emotion features to obtain an emotion fusion prediction result;

[0118] The first prediction submodule is used to make predictions based on the target emotion feature vector and the emotion fusion prediction result to obtain the target emotion.

[0119] In one embodiment, the second determining module 302 includes:

[0120] A first determination submodule is used to determine feature vectors corresponding to the plurality of emotional features respectively;

[0121] The second prediction submodule is used to predict the feature vectors respectively to obtain emotion prediction results corresponding to the multiple emotion features respectively.

[0122] In one embodiment, the central control screen is connected to a smart home device; the interaction module 304 includes:

[0123] A second determination submodule is used to determine a control instruction corresponding to the target emotion according to the target emotion;

[0124] The control submodule is used to control the smart home device according to the control instruction.

[0125] In one embodiment, the interaction module 304 includes:

[0126] The output submodule is used to determine the interactive voice and recommended content corresponding to the target emotion according to the target emotion, and output the interactive voice and the recommended content to the user; the recommended content is a way to control the user's emotion and a way to control the user's physiological health.

[0127] In one embodiment, the central control screen is provided with a camera and a microphone; the central control screen is connected to a wearable device; the first determination module 301 includes:

[0128] A first acquisition submodule, used to acquire the user's facial expression through the camera, and determine the user's facial features through the facial expression;

[0129] A second acquisition submodule, configured to acquire voice information of the user through the microphone, and determine voice features of the user through the voice information;

[0130] The third acquisition submodule is used to acquire the physiological signal of the user sent by the wearable device, and determine the physiological characteristics of the user through the physiological signal.

[0131] In one embodiment, the first acquisition submodule includes:

[0132] The first extraction unit is used to extract features of the facial expression using a convolutional neural network to obtain facial features of the user.

[0133] In one embodiment, the second acquisition submodule includes:

[0134] The second extraction unit is used to use a preset voice feature extraction model to extract features from the voice information to obtain the voice features of the user; the preset voice feature extraction model uses a Mel-frequency cepstral coefficient algorithm to extract features from sample voice information to obtain sample voice features, and trains a pre-trained model using the sample voice features to obtain the preset voice feature extraction model.

[0135] In one embodiment, the third acquisition submodule includes:

[0136] A preprocessing unit, used for preprocessing the physiological signal to obtain a preprocessed physiological signal;

[0137] A normalization unit, used for performing normalization processing on the preprocessed physiological signal to obtain a normalized physiological signal;

[0138] The third extraction unit is used to extract features from the normalized physiological signal to obtain the physiological features of the user.

[0139] In the embodiment of the present invention, the first determination module is used to determine the user's facial features, voice features and physiological features; the classification module is used to perform feature classification on the facial features, voice features and physiological features to obtain multiple emotional features after classification; the second determination module is used to determine the feature vectors and emotional prediction results corresponding to the multiple emotional features; the third determination module is used to determine the target emotion according to the feature vectors and emotional prediction results corresponding to the multiple emotional features; the interaction module is used to perform corresponding interaction according to the target emotion. Thus, the central control screen can perform emotional interaction with the user to meet the user's needs, and the central control screen determines the target emotion according to the feature vectors and emotional prediction results corresponding to the multiple emotional features, thereby improving the accuracy of the target emotion, further making the interaction more accurate, and increasing the user's experience.

[0140] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0141] An embodiment of the present invention further provides an electronic device, including:

[0142] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned emotional interaction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0143] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned emotional interaction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0144] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0145] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0146] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0147] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0149] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0150] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0151] The above is a detailed introduction to an emotion interaction method, an emotion interaction device, an electronic device and a computer-readable storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An emotional interaction method, characterized in that: The method is applied to the central control screen, and the method includes: Determine the user's facial, voice, and physiological characteristics; Performing feature classification on the facial features, the voice features, and the physiological features to obtain a plurality of classified emotion features; Determine feature vectors and emotion prediction results corresponding to the plurality of emotion features respectively; Determining a target emotion according to the feature vectors corresponding to the plurality of emotion features and the emotion prediction results; Corresponding interactions are performed according to the target emotion.

2. The emotional interaction method according to claim 1, characterized in that: The step of determining the target emotion according to the feature vectors and the emotion prediction results respectively corresponding to the plurality of emotion features comprises: Merging the feature vectors corresponding to the multiple emotional features to obtain a target emotional feature vector; Fusing the emotion prediction results corresponding to the multiple emotion features to obtain an emotion fusion prediction result; Prediction is performed based on the target emotion feature vector and the emotion fusion prediction result to obtain the target emotion.

3. The emotional interaction method according to claim 1, characterized in that: The determining of the feature vectors and the emotion prediction results respectively corresponding to the plurality of emotion features comprises: Determine feature vectors corresponding to the plurality of emotional features respectively; The feature vectors are predicted respectively to obtain emotion prediction results corresponding to the multiple emotion features respectively.

4. The emotional interaction method according to claim 1, characterized in that: The central control screen is connected to the smart home device; and the corresponding interaction according to the target emotion includes: According to the target emotion, determining a control instruction corresponding to the target emotion; The smart home device is controlled according to the control instruction.

5. The emotional interaction method according to claim 1, characterized in that: The performing corresponding interaction according to the target emotion includes: According to the target emotion, interactive speech and recommended content corresponding to the target emotion are determined, and the interactive speech and the recommended content are output to the user; the recommended content is a way to control the emotion of the user and a way to control the physiological health of the user.

6. The emotional interaction method according to claim 1, characterized in that: The central control screen is provided with a camera and a microphone; The central control screen is connected to the wearable device; the determining of the facial features, voice features and physiological features of the user includes: Acquire the user's facial expression through the camera, and determine the user's facial features through the facial expression; Acquiring voice information of the user through the microphone, and determining voice features of the user through the voice information; Acquire the physiological signal of the user sent by the wearable device, and determine the physiological characteristics of the user through the physiological signal.

7. The emotional interaction method according to claim 6, characterized in that: Determining the facial features of the user through the facial expression includes: A convolutional neural network is used to extract features of the facial expression to obtain facial features of the user.

8. The emotional interaction method according to claim 6, characterized in that: The determining the voice feature of the user through the voice information includes: A preset speech feature extraction model is used to extract features from the speech information to obtain the speech features of the user; the preset speech feature extraction model extracts features from sample speech information through a Mel-frequency cepstral coefficient algorithm to obtain sample speech features, and a pre-trained model is trained through the sample speech features to obtain the preset speech feature extraction model.

9. The emotional interaction method according to claim 6, characterized in that: Determining the physiological characteristics of the user through the physiological signal includes: Preprocessing the physiological signal to obtain a preprocessed physiological signal; Performing normalization processing on the preprocessed physiological signal to obtain a normalized physiological signal; Feature extraction is performed on the normalized physiological signal to obtain the physiological features of the user.

10. An emotional interaction device, characterized in that: The device is applied to a central control screen, and the device comprises: A first determination module, used to determine the user's facial features, voice features and physiological features; A classification module, used for classifying the facial features, the voice features and the physiological features to obtain a plurality of classified emotion features; A second determination module is used to determine feature vectors and emotion prediction results corresponding to the plurality of emotion features respectively; A third determination module, configured to determine a target emotion according to the feature vectors corresponding to the plurality of emotion features and the emotion prediction results; The interaction module is used to perform corresponding interactions according to the target emotion.

11. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of the emotional interaction method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the emotional interaction method as described in any one of claims 1 to 9 are implemented.

Images