Intelligent accompanying human-type robot based on multi-modal emotion interaction and sensing method

Abstract

The invention relates to the technical field of robot perception, in particular to an intelligent accompanying human-type robot based on multi-mode emotion interaction and a perception method, and the method comprises the following steps: collecting facial expression and body posture data of a user through a visual perception unit, collecting intonation, speed, volume and keyword data through a voice perception unit, and carrying out the recognition of the speech, speed, volume and keyword data; data preprocessing and feature extraction are carried out; secondly, carrying out time sequence modeling on the extracted visual and voice features based on a recurrent neural network, and generating visual and voice time sequence feature vectors; the method comprises the steps that firstly, time sequence features are extracted, then, a multi-mode fusion model based on an attention mechanism is adopted, the time sequence features are fused into a comprehensive emotion feature vector used for emotion state classification, and finally, personalized interaction content such as voice, expressions and limb actions matched with the emotion state of a user is generated according to an emotion classification result. According to the method, the accuracy of robot emotion perception and the naturalness of man-machine interaction are remarkably improved, and the method is suitable for multi-scene application of education, medical treatment, old-age care and the like.

Description

Technical Field

[0001] The present invention relates to the field of robot perception technology, and in particular to an intelligent companion humanoid robot and a perception method based on multimodal emotional interaction. Background Art

[0002] With the rapid development of artificial intelligence and robotics, the application of intelligent robots in social life is becoming more and more popular. In companion robots, perception methods are the key technology for robots to understand the user's emotional state, action intentions and environmental changes, and are an important basis for realizing intelligent interaction.

[0003] Existing perception methods still have significant technical deficiencies when dealing with the recognition and analysis of emotional states in complex scenarios. First, most perception methods rely too much on single-modal data and fail to fully utilize the complementarity of multimodal information such as vision and speech, resulting in low reliability and accuracy of perception results. Second, in terms of time series modeling, existing methods often ignore the dynamic changes in the user's emotional state and cannot accurately capture the temporal correlation contained in multimodal data. In addition, the personalized interaction design for different emotional states is still relatively rough, and fails to generate accurate and natural feedback based on the emotion classification results. Summary of the invention

[0004] The present invention provides an intelligent companion humanoid robot and a perception method based on multimodal emotional interaction.

[0005] A robot perception method comprises the following steps:

[0006] S1, multimodal data collection: collect the user's visual data through the visual perception unit on the robot, and collect the user's voice data through the voice perception unit;

[0007] S2, data preprocessing: preprocessing the collected visual data and speech data, and extracting visual features and speech features;

[0008] S3, time series feature modeling: Based on the extracted visual features and speech features, a recurrent neural network is used to perform time series modeling to generate visual time series feature vectors and speech time series feature vectors;

[0009] S4, multimodal feature fusion: through the multimodal fusion model based on the attention mechanism, the visual temporal feature vector and the speech temporal feature vector are fused into a comprehensive emotional feature vector;

[0010] S5, emotional state classification: Based on the comprehensive emotional feature vector, the deep learning classification model is used to classify the user's emotional state;

[0011] S6, personalized interaction: Based on the emotional state classification results, generate personalized interactions that match the user's emotional state.

[0012] Optionally, the S1 includes:

[0013] S11, visual data collection: collecting the user's visual data through the visual perception unit on the robot, wherein the visual data includes facial expressions and body postures;

[0014] S12, voice data collection: collecting the user's voice data through the voice perception unit on the robot, the voice data including intonation, speaking speed, volume and keywords;

[0015] S13, data synchronization: time stamp the visual data and voice data.

[0016] Optionally, S2 includes:

[0017] S21, visual data preprocessing: preprocessing the collected visual data, including denoising and image enhancement;

[0018] S22, voice data preprocessing: preprocessing the collected voice data, including frame segmentation and noise reduction;

[0019] S23, visual feature extraction: extracting visual features based on the preprocessed visual data;

[0020] S24, speech feature extraction: extracting speech features based on the preprocessed speech data.

[0021] Optionally, the S3 includes:

[0022] S31, constructing a time series data set: arranging the extracted visual feature vectors and speech feature vectors in chronological order to construct a visual time series data set and a speech time series data set;

[0023] S32, Time Series Feature Modeling: Use recurrent neural networks to model visual and speech time series data to capture the dynamic changes of emotions in the time dimension;

[0024] S33, time series feature output: modeling the visual time series dataset and the speech time series dataset respectively through a recurrent neural network model to generate a visual time series feature vector and a speech time series feature vector;

[0025] S34, time series feature storage: storing the generated visual time series feature vector and speech time series feature vector, retaining the timestamp information, and ensuring that the time axes of the visual and speech time series features are aligned.

[0026] Optionally, the S4 includes:

[0027] S41, weight calculation: receiving the generated visual temporal feature vector and speech temporal feature vector, and calculating the weights of the visual features and the speech features through the attention mechanism model;

[0028] S42, feature weighting: weighted calculation of visual and speech features according to attention weights;

[0029] S43, feature fusion: fusing the weighted visual features and speech features to generate a comprehensive emotion feature vector;

[0030] S44, output comprehensive feature vector: use the generated comprehensive emotion feature vector as multimodal emotion representation.

[0031] Optionally, the S5 includes:

[0032] S51, input comprehensive emotion feature vector: receiving the generated comprehensive emotion feature vector as input data of the classification model;

[0033] S52, classification model: classify the emotional state according to the input comprehensive emotional feature vector through the pre-trained multi-layer perceptron classification model;

[0034] S53, classification output: output the emotion category according to the input comprehensive emotion feature vector, the categories include happiness, sadness and anger;

[0035] S54, final classification decision: According to the output probability distribution, the emotion category with the highest probability is selected as the final classification result.

[0036] Optionally, the S6 includes:

[0037] S61, receiving the emotional state classification result: receiving the output emotional state classification result;

[0038] S62, selecting a personalized interaction strategy: selecting an adaptive interaction strategy from a predefined interaction strategy library according to the classification result;

[0039] S63. Generate interactive content: Generate personalized interactive content based on the selected interactive strategy.

[0040] Optionally, S6 further includes:

[0041] S64, real-time interaction optimization: adjust the interactive content according to the user's immediate feedback;

[0042] S65, interactive content output: outputting the generated personalized interactive content through the robot interactive unit, including playing voice content, displaying facial expressions, and performing body movements.

[0043] An intelligent companion humanoid robot based on multimodal emotional interaction uses one of the above-mentioned robot perception methods to perform multimodal emotional interaction.

[0044] Beneficial effects of the present invention:

[0045] The present invention, through multimodal data collection and data preprocessing steps, makes full use of the comprehensive characteristics of visual data and voice data, extracts multimodal emotional features with high recognition, and introduces time series modeling based on recurrent neural networks, accurately captures the dynamic change trend of the user's emotional state, and improves the modeling ability of complex emotional evolution processes. At the same time, the time series features of visual and voice modalities are weighted and fused through the attention mechanism, effectively overcoming the limitations of single modality in emotion recognition, and the generated comprehensive emotional feature vector comprehensively represents the user's overall emotional state.

[0046] In this invention, the multi-layer perceptron classification model realizes accurate classification of emotional states, including joy, sadness and anger, based on comprehensive emotional feature vectors, and generates voice, expression and body movement responses that match the user's emotional state in combination with the classification results. The interactive content is meticulous and targeted. In addition, real-time adjustment of interaction strategies (such as voice intonation and body movement amplitude) further enhances the naturalness and flexibility of human-computer interaction, providing users with a caring emotional companionship experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A schematic diagram of a method flow chart of an embodiment of the present invention;

[0049] Figure 2 Schematic diagram of S3 process of an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. At the same time, it is explained here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments, and those skilled in the art may also adopt other alternatives to implement some known technologies; and the accompanying drawings are only for more specific description of the embodiments, and are not intended to specifically limit the present invention.

[0051] It should be noted that the references to "one embodiment", "an embodiment", "an exemplary embodiment", "some embodiments" and the like in the specification indicate that the embodiments described may include specific features, structures or characteristics, but not every embodiment may include the specific features, structures or characteristics. In addition, when a specific feature, structure or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the art to implement such feature, structure or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0052] In general, a term can be understood, at least in part, from its use in context. For example, depending, at least in part, on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending, at least in part, on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0053] like Figure 1-Figure 2 As shown, a robot perception method includes the following steps:

[0054] S1, multimodal data collection: collect the user's visual data through the visual perception unit on the robot, and collect the user's voice data through the voice perception unit;

[0055] S2, data preprocessing: preprocessing the collected visual data and speech data, and extracting visual features and speech features;

[0056] S3, time series feature modeling: Based on the extracted visual features and speech features, a recurrent neural network is used to perform time series modeling to generate visual time series feature vectors and speech time series feature vectors;

[0057] S4, multimodal feature fusion: through the multimodal fusion model based on the attention mechanism, the visual temporal feature vector and the speech temporal feature vector are fused into a comprehensive emotional feature vector;

[0058] S5, emotional state classification: Based on the comprehensive emotional feature vector, the deep learning classification model is used to classify the user's emotional state;

[0059] S6, personalized interaction: Based on the emotional state classification results, generate personalized interactions that match the user's emotional state.

[0060] S1 includes:

[0061] S11, visual data collection: collecting the user's visual data through the visual perception unit on the robot, the visual data including facial expressions and body postures, the steps include;

[0062] S111, Facial Expression Capture: Use RGB-D camera to obtain 2D image and depth information of the user’s face;

[0063] Perform face detection on the collected images and extract facial key points;

[0064] Based on the facial area detected by key points, the facial expression recognition algorithm is used to extract the user's expression features, including subtle changes in expression such as smiling, frowning, and opening the mouth;

[0065] S112, body posture collection: using a depth camera to collect the user's body posture information;

[0066] Based on the human skeleton tracking algorithm, the 3D position and angle changes of the user's key body parts (such as head, arms, torso and legs) are extracted;

[0067] Record the user's overall body movement characteristics, such as standing, sitting, waving and other dynamic behaviors;

[0068] S12, voice data collection: collect the user's voice data through the voice perception unit on the robot. The voice data includes intonation, speaking speed, volume and keywords. The steps include:

[0069] S121, intonation collection: perform fundamental frequency analysis on the user's voice signal and calculate the fundamental frequency of each frame of voice to describe the change pattern of the intonation.

[0070] S122, speech rate collection: count the number of speech pronunciation units of the user in a unit time, and calculate the speech rate value per second (words per second, WPS) using a speech segmentation algorithm.

[0071] S123, volume acquisition: calculate the short-time energy E(t) of the speech signal, the formula is:

[0072]

[0073] Where x[n] is the sample value of the speech signal, and N is the number of sampling points in the frame;

[0074] S124, keyword collection: using a speech recognition algorithm to transcribe the speech content and extract keywords from the speech;

[0075] S13, data synchronization: timestamp the visual data and voice data to ensure that the data of different modalities are aligned on the same timeline and provide synchronized input for subsequent steps.

[0076] S2 includes:

[0077] S21, visual data preprocessing: preprocessing the collected visual data, including denoising and image enhancement, specifically including:

[0078] Denoising: Use median filtering to remove random noise while retaining image edge details, and use bilateral filtering to smooth image texture and reduce the interference of high-frequency noise on subsequent processing;

[0079] Image enhancement: Histogram equalization is used to improve the contrast of the image and enhance the visual details of key areas. Gamma correction is applied to the depth image to optimize the brightness distribution and ensure the consistency of the collected data under different lighting conditions.

[0080] S22, voice data preprocessing: preprocessing the collected voice data, including frame segmentation and noise reduction, specifically including:

[0081] Frame processing: The speech signal is divided into frames of 25ms in length, and the frame shift is set to 10ms to extract short-time features. Each frame contains sufficient frequency information to ensure the temporal resolution of the features.

[0082] Noise reduction processing: Use spectrum subtraction method to remove background noise, and filter algorithm based on signal-to-noise ratio to enhance the clarity of speech signals and retain the main speech frequency components;

[0083] S23, visual feature extraction: extracting visual features based on the preprocessed visual data, specifically including:

[0084] Facial expression feature extraction: Use the face detection algorithm to locate the facial area, use the pre-trained convolutional neural network to extract facial feature points, perform geometric analysis on the facial feature points, and generate feature vectors that describe micro-expressions (such as smiles and frowns);

[0085] Body posture feature extraction: Use the skeleton tracking algorithm to identify the user's key body parts (such as shoulders, elbows, wrists, etc.), and extract the user's dynamic posture features based on the three-dimensional position and angle of the joint points to generate a feature vector that describes the body movement;

[0086] S24, speech feature extraction: extracting speech features based on the preprocessed speech data, specifically including:

[0087] Intonation feature extraction: Calculate the fundamental frequency F0 through spectrum analysis to describe the change pattern of intonation over time;

[0088] The fundamental frequency calculation formula is:

[0089] Among them, T pitch is the fundamental pitch period of speech;

[0090] Speech rate feature extraction: Count the number of pronunciation units per second to measure speech rate;

[0091] Tone feature extraction: Calculate the Mel frequency cepstral coefficient (MFCC), which is expressed as:

[0092]

[0093] Where X[n] is the amplitude of the spectrum, N is the number of filters, and k is the order of the cepstral coefficients;

[0094] The preprocessed visual data and speech data generate visual feature vectors describing facial expressions and body postures, and speech feature vectors describing intonation, speaking speed, and tone, respectively, providing high-quality input data for subsequent steps.

[0095] S3 includes:

[0096] S31, constructing a time series data set: arranging the extracted visual feature vectors and speech feature vectors in chronological order to construct a visual time series data set and a speech time series data set;

[0097] The visual time series dataset is represented as:

[0098] X vis =[x vis,1 ,x vis,2 ,…,x vis,T ];

[0099] Among them, x vis,t represents the visual feature vector at time t, T represents the total length of the time series, t∈1,2,...,T;

[0100] The speech time series dataset is represented as:

[0101] X aud =[x aud,1 ,x aud,2 ,…,x aud,T ];

[0102] Among them, x aud,t Represents the speech feature vector at time t, t∈1,2,...,T;

[0103] S32, Time Series Feature Modeling: Use recurrent neural networks to model visual and speech time series data to capture the dynamic changes of emotions in the time dimension;

[0104] The model hidden state update formula is expressed as:

[0105] h t =f(W h h t-1 +Wx x t + b);

[0106] Among them, h t represents the hidden state at time t, x t represents the input features (visual or speech features) at time t, W h , W x is the weight matrix of the model, b is the bias vector, and f is the activation function (such as tanh);

[0107] S33, time series feature output: The visual time series dataset X is respectively vis and speech time series dataset X aud Modeling is performed to generate visual temporal feature vectors and speech temporal feature vectors;

[0108] The visual temporal feature vector is expressed as:

[0109] H vis =[h vis,1 ,h vis,2 ,…,h vis,T ];

[0110] Among them, h vis,t Represents the visual temporal feature vector at time t;

[0111] The speech time series feature vector is expressed as:

[0112] H aud =[h aud,1 ,h aud,2 ,…,h aud,T ];

[0113] Among them, h aud,t The speech temporal feature vector representing time t;

[0114] S34, temporal feature storage: the generated visual temporal feature vector H vis and speech time series feature vector H aud Store and retain timestamp information to ensure the time axis alignment of visual and speech timing features, providing input for subsequent multimodal feature fusion;

[0115] Based on visual features and speech features, complete visual temporal feature vectors and speech temporal feature vectors are generated, providing an accurate data basis for the robot to perceive the dynamic changes of user emotions.

[0116] S4 includes:

[0117] S41, weight calculation: receiving the generated visual temporal feature vector H vis and speech time series feature vector Haud , the weights of visual features and speech features are calculated through the attention mechanism model, and the weight calculation formula is:

[0118]

[0119] Among them, w vis 、w aud is the weight vector of visual and speech features, b vis ,b aud is the bias term, α vis and α aud is the weight matrix of visual and speech features, indicating the contribution of each temporal feature;

[0120] S42, feature weighting: weighted calculation of visual and speech features according to attention weights, the formula is:

[0121] F vis =α vis ·H vis ;

[0122] F aud =α aud ·H aud ;

[0123] Among them, F vis and F aud Represent the weighted visual feature and speech feature matrices respectively;

[0124] S43, feature fusion: The weighted visual features and speech features are fused to generate a comprehensive emotional feature vector H fusion , the fusion formula is:

[0125] H fusion =[F vis ; F aud ];

[0126] Among them, [·;·] represents the vector concatenation operation;

[0127] S44, output comprehensive feature vector: the generated comprehensive emotion feature vector H fusion As a multimodal emotion representation, its timestamp information is retained to ensure the temporal consistency of subsequent emotion state classification. fusion Characterizes the user's overall emotional state, including the dynamic changes of visual and voice modalities in the temporal dimension;

[0128] The multimodal fusion model based on the attention mechanism fully considers the contribution of visual and speech features. The generated comprehensive emotional feature vector accurately describes the user's overall emotional state, providing a comprehensive foundation for subsequent emotional state classification.

[0129] S5 includes:

[0130] S51, input comprehensive emotion feature vector: receive the generated comprehensive emotion feature vector H fusion , as input data for the classification model;

[0131] S52, classification model: Through the pre-trained multi-layer perceptron classification model, according to the input comprehensive emotional feature vector H fusion ,To classify the emotional state, the structure of the multi-layer perceptron classification model includes input layer, hidden layer and output layer;

[0132] Model expression: comprehensive sentiment feature vector H fusion For nonlinear transformation, the formula is:

[0133] z=fW1·H fusion +b1);

[0134] Where W1 is the weight matrix of the first layer, b1 is the bias vector of the first layer, and f(·) is the activation function (such as ReLU);

[0135] The output layer generates the probability distribution of emotion categories, the formula is:

[0136] o=softmax(W out ·z+b out );

[0137] Among them, o represents the probability distribution of emotion category, W out is the weight matrix of the output layer, b out is the bias term of the output layer, and softmax(·) is a normalization function used to convert the output value into a probability distribution;

[0138] S53, classification output: according to the input comprehensive emotional feature vector H fusion , output emotion categories, including happiness, sadness and anger;

[0139] The probability distribution o of the classification result represents the confidence of each emotion category, for example

[0140] o=[P 愉悦 ,P 悲伤 ,P 愤怒 ];

[0141] Among them, P 愉悦、 P 悲伤和 P 愤怒 Respectively represent the probability of the corresponding emotion category;

[0142] S54, final classification decision: According to the output probability distribution o, the emotion category with the highest probability is selected as the final classification result, which is expressed as:

[0143] C = arg max ( o ) ;

[0144] Among them, C is the final sentiment category label, and arg max is the maximum value.

[0145] S6 includes:

[0146] S61, receiving the emotional state classification result: receiving the output emotional state classification result C;

[0147] S62, selecting a personalized interaction strategy: According to the classification result C, selecting an adaptive interaction strategy from a predefined interaction strategy library, specifically including:

[0148] Pleasure state:

[0149] Choose positive interaction strategies, such as voice responses like “I’m glad to see you’re so happy!”;

[0150] Generate cheerful body gestures, such as waving or smiling emoticons;

[0151] Sad state:

[0152] Choose comforting interaction strategies, such as a voice response of “Is there anything I can do for you?”;

[0153] Generate gentle body language, such as leaning forward slightly to show concern;

[0154] Angry state:

[0155] Choose calming strategies, such as voice response "Please take a deep breath, let's relax together";

[0156] Generate stable body movements, such as maintaining a neutral stance and avoiding overly intense movements;

[0157] S63. Generate interactive content: Generate personalized interactive content based on the selected interactive strategy, including:

[0158] Speech content generation: Use speech synthesis units to generate specific speech responses, with intonation adjusted according to the emotional state:

[0159] Pleasant state: rising tone and slightly faster speaking speed;

[0160] Sad state: flat tone and slow speech speed;

[0161] Angry state: neutral tone, moderate speed.

[0162] Expression generation: Generate expressions that match the emotional state using the robot's facial display unit:

[0163] Pleasant state: smiling;

[0164] Sad state: gentle expression;

[0165] Angry state: calm expression;

[0166] Limb motion generation: Generate specific movements through the robot motion control unit:

[0167] Joyful state: waving easily;

[0168] Sad state: body leans forward slightly;

[0169] Angry state: Maintain a stable neutral posture.

[0170] The S6 also includes:

[0171] S64, real-time interaction optimization: adjust the interactive content based on the user's immediate feedback, for example:

[0172] If the user shows a change in emotional state (e.g., from sadness to joy), dynamically adjust the interaction strategy;

[0173] S65, interactive content output: outputting the generated personalized interactive content through the robot interactive unit, including playing voice content, displaying facial expressions, and performing body movements.

[0174] An intelligent companion humanoid robot based on multimodal emotional interaction uses one of the above-mentioned robot perception methods to perform multimodal emotional interaction.

[0175] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0176] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A robot perception method, characterized in that: The following steps are involved: S1, multimodal data collection: collect the user's visual data through the visual perception unit on the robot, and collect the user's voice data through the voice perception unit; S2, data preprocessing: preprocessing the collected visual data and speech data, and extracting visual features and speech features; S3, time series feature modeling: Based on the extracted visual features and speech features, a recurrent neural network is used to perform time series modeling to generate visual time series feature vectors and speech time series feature vectors; S4, multimodal feature fusion: through the multimodal fusion model based on the attention mechanism, the visual temporal feature vector and the speech temporal feature vector are fused into a comprehensive emotional feature vector; S5, emotional state classification: Based on the comprehensive emotional feature vector, the deep learning classification model is used to classify the user's emotional state; S6, personalized interaction: Based on the emotional state classification results, generate personalized interactions that match the user's emotional state.

2. A robot perception method according to claim 1, characterized in that: The S1 includes: S11, visual data collection: collecting the user's visual data through the visual perception unit on the robot, wherein the visual data includes facial expressions and body postures; S12, voice data collection: collecting the user's voice data through the voice perception unit on the robot, the voice data including intonation, speaking speed, volume and keywords; S13, data synchronization: time stamp the visual data and voice data.

3. A robot perception method according to claim 2, characterized in that: The S2 includes: S21, visual data preprocessing: preprocessing the collected visual data, including denoising and image enhancement; S22, voice data preprocessing: preprocessing the collected voice data, including frame segmentation and noise reduction; S23, visual feature extraction: extracting visual features based on the preprocessed visual data; S24, speech feature extraction: extracting speech features based on the preprocessed speech data.

4. A robot perception method according to claim 3, characterized in that: The S3 includes: S31, constructing a time series data set: arranging the extracted visual feature vectors and speech feature vectors in chronological order to construct a visual time series data set and a speech time series data set; S32, Time Series Feature Modeling: Use recurrent neural networks to model visual and speech time series data to capture the dynamic changes of emotions in the time dimension; S33, time series feature output: modeling the visual time series dataset and the speech time series dataset respectively through a recurrent neural network model to generate a visual time series feature vector and a speech time series feature vector; S34, time series feature storage: storing the generated visual time series feature vector and speech time series feature vector.

5. A robot perception method according to claim 4, characterized in that: The S4 includes: S41, weight calculation: receiving the generated visual temporal feature vector and speech temporal feature vector, and calculating the weights of the visual features and the speech features through the attention mechanism model; S42, feature weighting: weighted calculation of visual and speech features according to attention weights; S43, feature fusion: fusing the weighted visual features and speech features to generate a comprehensive emotion feature vector; S44, output comprehensive feature vector: use the generated comprehensive emotion feature vector as multimodal emotion representation.

6. A robot perception method according to claim 5, characterized in that: The S5 includes: S51, input comprehensive emotion feature vector: receiving the generated comprehensive emotion feature vector as input data of the classification model; S52, classification model: classify the emotional state according to the input comprehensive emotional feature vector through the pre-trained multi-layer perceptron classification model; S53, classification output: output the emotion category according to the input comprehensive emotion feature vector, the categories include happiness, sadness and anger; S54, final classification decision: According to the output probability distribution, the emotion category with the highest probability is selected as the final classification result.

7. A robot perception method according to claim 6, characterized in that: The S6 includes: S61, receiving the emotional state classification result: receiving the output emotional state classification result; S62, selecting a personalized interaction strategy: selecting an adaptive interaction strategy from a predefined interaction strategy library according to the classification result; S63. Generate interactive content: Generate personalized interactive content based on the selected interactive strategy.

8. A robot perception method according to claim 7, characterized in that: The S6 further includes: S64, real-time interaction optimization: adjust the interactive content according to the user's immediate feedback; S65, interactive content output: outputting the generated personalized interactive content through the robot interactive unit, including playing voice content, displaying facial expressions, and performing body movements.

9. Intelligent companion humanoid robot based on multimodal emotional interaction, characterized in that: Use the perception method as described in any one of claims 1 to 8 to perform multimodal emotional interaction.

Images