Method for behavior expression of humanoid robot based on audio-visual emotion and environment perception

Abstract

The application discloses a kind of based on audiovisual sense emotion and environment perception humanoid robot behavior expression method, including interactive object facial emotion perception method, interactive object voice emotion perception method, cross-modal decision-making strategy based on audiovisual sense emotion and environment perception and humanoid robot behavior expression strategy.The method of the application overcomes the vulnerability of single channel perception in complex environment through interactive object facial emotion perception, voice emotion perception, cross-modal decision-making and humanoid robot behavior expression, dynamically adjusts the weight of visual perception and auditory perception in decision-making, realizes the dynamic self-adaptive identification of interactive object emotion in complex environment, and improves the emotional interaction and scene adaptability of human-computer interaction or human-machine collaboration.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction and affective computing, specifically a method for expressing humanoid robot behavior based on visual and auditory emotion and environmental perception. It mainly uses visual and auditory perception of user and environmental information, and integrates visual and auditory modal data to achieve dynamic emotion recognition and cross-modal decision-making, thereby driving robot behavior expression and realizing adaptive interaction between humans and robots. Background Technology

[0002] With the rapid development of theories and technologies in robotics, affective computing, and human-computer interaction, humanoid robots are increasingly being applied in various scenarios such as teaching, companionship, entertainment, and service, interacting or collaborating with humans. Humanoid robots should possess the ability to perceive emotions and express corresponding behaviors, thereby improving the understanding of intentions and work efficiency during interactions or collaborations with humans.

[0003] Currently, humanoid robots primarily focus on two aspects of emotion perception: vision and hearing. These involve perceiving facial or vocal features to classify emotions. However, these single-modal emotion perception systems generally suffer from environmental adaptability deficiencies. Visual perception is highly dependent on lighting conditions; when ambient light is insufficient, the failure rate of extracting key facial features such as eye contours and micro-expressions increases dramatically, leading to a surge in emotion misjudgment. While auditory perception is not affected by lighting, it is extremely sensitive to background noise. When the signal-to-noise ratio decreases, features of the speech signal, such as fundamental frequency and formants, may shift or be lost. This single-modal limitation is particularly pronounced in complex working environments such as nighttime and public areas, severely restricting the robot's perception capabilities.

[0004] Some existing emotion perception technologies have introduced bimodal fusion mechanisms of visual and auditory perception, but there is still room for improvement in their adaptive perception capabilities in the face of environmental disturbances. For example, the visual-audio fusion method with a fixed weight allocation strategy is insufficiently adaptable to dynamically changing environments. When lighting conditions significantly decrease, leading to a deterioration in visual data quality, or when background noise suddenly increases and interferes with auditory signals, systems using a fixed weight allocation strategy struggle to adjust the contribution of the corresponding modality in a timely and accurate manner, thus affecting emotion perception capabilities. Furthermore, in complex, unstructured environments with varying characteristics, cross-modal emotion classification conflicts are a key issue that urgently needs in-depth exploration. For example, when the emotion classification result perceived by visual information is "anger," but the emotion classification result perceived by auditory information is "happiness," such inconsistent emotion classification results may lead to undesirable outputs if only one classification result is used, different emotions are weighted and averaged, or a majority vote is conducted. This could result in errors in human-robot interaction, the generation of meaningless intermediate emotions causing confusion in the robot's response logic, etc.

[0005] In summary, existing technologies still face challenges in addressing the instability of single-modal perception, the insufficient flexibility of multimodal fusion strategies, and cross-modal decision-making conflicts. Therefore, to solve these problems, a method for expressing humanoid robot behavior based on visual and auditory emotion and environmental perception is proposed. This method breaks through the traditional static weight framework and constructs a cross-modal decision-making mechanism, thereby achieving more natural and effective emotional communication and interaction between humans and humanoid robots. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method for expressing humanoid robot behavior based on visual and auditory emotion and environmental perception, which addresses the shortcomings of the prior art. This method overcomes the vulnerability of single-channel perception in complex environments by perceiving facial emotion, speech emotion, cross-modal decision-making, and expressing humanoid robot behavior. It dynamically adjusts the weight of visual and auditory perception in decision-making, realizes dynamic adaptive recognition of the emotion of interactive objects in complex environments, and improves the emotional interactivity and scene adaptability of human-computer interaction or human-computer collaboration.

[0007] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0008] A method for expressing humanoid robot behavior based on visual and auditory emotion and environmental perception includes a method for perceiving facial emotions of interactive objects, a method for perceiving voice emotions of interactive objects, a cross-modal decision-making strategy based on visual and auditory emotion and environmental perception, and a humanoid robot behavior expression strategy.

[0009] Methods for facial emotion perception of interactive objects include:

[0010] S101. Image Acquisition and Preprocessing: Acquire image streams, perform face detection and facial landmark localization on each frame, and preprocess the face regions.

[0011] S102, Adaptive Image Enhancement: Perform adaptive image enhancement on the preprocessed image;

[0012] S103, Preliminary estimation of visual emotion probability: The enhanced image from S102 is input into a lightweight convolutional neural network for processing. The lightweight convolutional neural network model is pre-trained using a facial expression database. The lightweight convolutional neural network model outputs a preliminary probability distribution of 7 emotion categories, namely anger, fear, happiness, sadness, surprise, disgust, and neutral emotion probability distribution.

[0013] S104, Visual Cognitive Uncertainty Quantification: During the inference phase of the lightweight convolutional neural network model, the Dropout layer in the network is kept active. Subsequently, the same enhanced face image from S102 is repeatedly input into the lightweight convolutional neural network model for T random forward propagations. Since the neurons that are randomly deactivated are different in each propagation, this process will produce a set {P} containing T slightly different emotion probability vectors. v (1) , P v (2) , …, P v (T)};

[0014] S105, Visual Modal Probability Output: Based on the set of T sentiment probability vectors generated in S104, calculate the final, smoothed visual sentiment probability vector and a quantized uncertainty score. Together, they constitute the output of the visual modality;

[0015] Methods for perceiving speech emotions in interactive objects include:

[0016] S201. Voice signal acquisition and processing: Acquire audio signals and process them;

[0017] S202. Speech Feature Extraction and Dimensionality Reduction: Extract the acoustic cepstral features, short-time energy, formants, and fundamental frequency of the audio, and perform feature fusion to form a speech emotion feature vector; and perform dimensionality reduction processing on the speech emotion feature vector;

[0018] S203, Model Training: Obtain the speech emotion feature vectors corresponding to the audio in the CASIA speech emotion database using the method in S202. Train the obtained speech emotion feature vectors using the random forest algorithm to establish a random forest model for recognizing speech emotion categories. The emotion classification of the random forest model includes anger, fear, happiness, sadness, surprise, and neutrality.

[0019] S204. Preliminary estimation of acoustic emotion probability: The speech emotion feature vector extracted from the audio signal obtained in S201 using the method in S202 is input into the random forest model established in S203 to obtain the acoustic emotion probability vector P of the audio signal. a ;

[0020] S205, Acoustic Emotion Uncertainty Quantification: Based on Acoustic Emotion Probability Vector P a Calculate the acoustic cognitive uncertainty score U a ;

[0021] Cross-modal decision-making strategies based on visual and auditory emotion and environmental perception include:

[0022] S301, Data Stream Time Domain Alignment: Through a unified timestamp system, ensure that the visual data frames output by S105 and the audio data frames output by S205 are precisely aligned in time.

[0023] S302, Quantification of Environmental Uncertainty: Define visual environmental distrust levels respectively. Distrust of auditory environment ;

[0024] S303. Establish a cross-modal unified identification framework; the cross-modal unified identification framework is defined as a superset of the emotion categories that can be identified in the visual and auditory modalities, denoted as Θ, and Θ = {anger, fear, happiness, sadness, surprise, disgust, neutral}, which contains a total of 7 mutually exclusive single emotions.

[0025] S304. Establish a two-layer uncertainty framework, the first layer of which is... Used to assess cognitive uncertainty; the second layer in the two-layer uncertainty framework. , Environmental uncertainty used to assess the impact of the external environment on the quality of visual and auditory data;

[0026] S305. Generate basic probability allocation under the cross-modal unified identification framework, that is, combine the two-layer uncertainty framework of S304 to convert the probability output of visual modality and auditory modality into basic probability allocation under the unified identification framework Θ described in S303.

[0027] S306. Based on the degree of conflict between vision and hearing, the basic probability assignments of the visual modality and the basic probability assignments of the auditory modality are combined to generate a fused basic probability assignment (BPA).

[0028] S307. Based on the fused Basic Probability Allocation (BPA) and the degree of conflict, perform gating and collaborative decision-making to output the final sentiment category.

[0029] Humanoid robot behavior expression strategies include:

[0030] S401, Final Emotion Decision: S301-S307 running on the PC output the final emotion category;

[0031] S402, Data Packet Encoding and Transmission: The PC will encode the final emotion category into a standard data packet and send the data packet to the robot via the network;

[0032] S403, Command Reception and Parsing: The robot receives data packets from the PC and parses out the emotion commands corresponding to the emotion categories;

[0033] S404, Robot Action Execution: Based on the parsed emotional instructions and the preset emotional action mapping table, the robot drives its own servo system to execute the corresponding limb actions that express the emotion.

[0034] As a further improvement of the present invention, the interactive object facial emotion perception method specifically includes:

[0035] S101. Image Acquisition and Preprocessing: A binocular camera with infrared illumination is used to acquire image streams and transmit them to the PC in real time. On the PC, face detection and facial key point localization are performed on each frame of the image, and the face area is scaled to 48×48 pixels. RGB channel alignment and pixel value normalization to the range are performed.

[0036] S102, Adaptive Image Enhancement: Perform channel-wise histogram analysis on the preprocessed image and control contrast stretching through dynamic thresholding;

[0037] S103. Preliminary estimation of visual emotion probability: The enhanced image from S102 is input into a lightweight convolutional neural network for processing. This lightweight convolutional neural network model is pre-trained using the FER-2013 facial expression database and categorized into seven emotion classes based on the database: anger, fear, happiness, sadness, surprise, disgust, and neutral. The lightweight convolutional neural network model extracts and classifies features through its two sets of convolutional units and fully connected layers at the ends, and finally outputs a preliminary probability distribution of the seven emotion classes, i.e., the emotion probability vector P, using the Softmax function. va ;P va ={p an , p f , p ha , p fe , p su , p d , p ne}, where p an , p f ,p ha , p fe , p su , p d , p ne The probability distributions of emotions are anger, fear, joy, sadness, surprise, disgust, and neutral, respectively.

[0038] Both sets of convolutional units include 3×3 convolutional layers, batch normalization layers, ReLU layers, and 2×2 pooling layers;

[0039] S104, Visual Cognition Uncertainty Quantification: During the inference phase of the lightweight convolutional neural network model, the Dropout layer in the network is kept active, with a dropout rate p of 0.5. Subsequently, the same enhanced face image from S102 is repeatedly input into the lightweight convolutional neural network model for T random forward propagations, where T is 50. Since the neurons randomly deactivated in each propagation are different, this process will generate a set {P} containing T slightly different emotion probability vectors. v (1) , P v (2) , …, P v (T)};

[0040] S105, Visual Modal Probability Output: Based on the set of T probability vectors generated in S104, calculate the final, smoothed visual emotion probability vector and a quantified uncertainty score. The two together constitute the visual modal output.

[0041] S1051. Take the arithmetic mean of the probability vectors obtained from the T random forward propagations in S104 to obtain the final visual emotion probability vector. For each sentiment category in S103, its average probability Where c represents the emotion category in S103, Let T represent the average probability that the model prediction is sentiment category c, obtained from T random forward propagations. Let c be the probability that the model predicts the emotion category c during the t-th forward propagation; the average probability combination of all emotion categories constitutes the visual emotion probability vector. ;

[0042] S1052. The uncertainty score U is obtained by averaging the variances of all sentiment categories. v :

[0043]

[0044] Where C represents the number of emotion categories; emotion categories c = 1, 2, 3...C, and represent anger, fear, happiness, sadness, surprise, disgust and neutrality respectively.

[0045] As a further improvement of the present invention, the interactive object voice emotion perception method specifically includes:

[0046] S201. Voice signal acquisition and processing: Audio signals are acquired using a microphone array and then sequentially amplified, filtered for noise reduction, and digitally processed before being transmitted to the PC in real time.

[0047] S202. Speech Feature Extraction and Dimensionality Reduction: The audio signal is preprocessed to extract the acoustic cepstral features, short-time energy, formants, and fundamental frequency, and the features are fused to form a speech emotion feature vector; PCA principal component analysis is used for dimensionality reduction.

[0048] S203, Model Training: Obtain the speech emotion feature vectors corresponding to the audio in the CASIA speech emotion database using the method in S202. Train the obtained speech emotion feature vectors using the random forest algorithm to establish a random forest model for recognizing speech emotion categories. The emotion classification of the random forest model includes anger, fear, happiness, sadness, surprise, and neutrality.

[0049] S2031. The random forest model consists of N decision trees. For a given input acoustic feature, each decision tree in the random forest, from the 1st to the Nth tree, independently outputs a sentiment classification vote.

[0050] set up Let represent the voting result of the i-th decision tree. To quantify the degree of disagreement among the decision trees within the random forest model, the vote of each decision tree is considered as an independent observation.

[0051] S2032, Calculate the acoustic emotion probability vector This allows for a comprehensive evaluation of the voting results with decision trees;

[0052] For any emotion category c a The corresponding acoustic emotion probability The calculation method is as follows:

[0053]

[0054] Where I(·) is an indicator function, which determines the vote of the i-th tree when the condition within the parentheses is true. For the emotion category Its value is 1 if it is positive and 0 otherwise; acoustic emotion probability vector It is a set of acoustic emotion probabilities corresponding to anger, fear, joy, sadness, surprise, and neutrality;

[0055] S204. Preliminary estimation of acoustic emotion probability: The speech emotion feature vector extracted from the audio signal obtained in S201 using the method in S202 is input into the random forest model established in S203 to obtain the acoustic emotion probability vector P of the audio signal. a ;

[0056] S205, Quantification of Acoustic Emotional Uncertainty: Quantifying the Acoustic Cognitive Uncertainty Score U aDefined as the average sample variance of voting results across six emotion categories; this acoustic cognitive uncertainty score U a The formula used to measure the dispersion of voting results among N decision trees in a random forest is as follows:

[0057]

[0058] As a further improvement to the present invention, the cross-modal decision-making strategy based on visual and auditory emotion and environmental perception includes:

[0059] S301, Data Stream Time Domain Alignment: Through a unified timestamp system, ensure that the visual data frames output by S105 and the audio data frames output by S205 are precisely aligned in time.

[0060] S302, Quantification of Environmental Uncertainty: Define visual environmental distrust levels respectively. Distrust of auditory environment ;

[0061] S3021. Obtain the current ambient brightness L by reading the camera's own exposure parameters; visual environment distrust level. for:

[0062]

[0063] In the formula, L0 is the preset brightness threshold; k is the kurtosis parameter; when the ambient light intensity is much lower than L0, A value close to 1 indicates a high degree of distrust of visual evidence.

[0064] S3022. Calculating auditory environment distrust based on speech signal-to-noise ratio calculated using microphone array. The calculation formula is:

[0065]

[0066] In the formula, The current speech signal-to-noise ratio is calculated based on the microphone array. The preset signal-to-noise ratio threshold is defined by k, where k is the kurtosis parameter; when the signal-to-noise ratio decreases significantly, A value approaching 1 indicates a high degree of distrust in auditory evidence.

[0067] S303. Establish a cross-modal unified identification framework; the cross-modal unified identification framework is defined as a superset of the emotion categories that can be identified in the visual and auditory modalities, denoted as Θ, and Θ = {anger, fear, happiness, sadness, surprise, disgust, neutral}, which contains a total of 7 mutually exclusive single emotions.

[0068] S304. Establish a two-layer uncertainty framework, the first layer of which is... Used to assess cognitive uncertainty; the second layer in the two-layer uncertainty framework. , Environmental uncertainty used to assess the impact of the external environment on the quality of visual and auditory data;

[0069] S305. Generate basic probability allocation under the cross-modal unified identification framework, that is, combine the two-layer uncertainty framework described in S304 to convert the probability outputs of visual and auditory modalities into basic probability allocation (BPA) under the unified identification framework Θ described in S303.

[0070] S3051, Visual BPA Generation: For any single sentiment hypothesis in the recognition frame Θ The quality of the beliefs it is endowed with The probability is output by S105. Uncertainty score U v Visual environment distrust level calculated by S3021 The result is calculated using the following formula:

[0071]

[0072] The total uncertainty is quantified into remaining beliefs, and these beliefs are uniformly assigned to the belief quality of the universal set Θ, which represents complete uncertainty. The belief quality is:

[0073]

[0074] The visual modality basic probability assignment (BPA) is determined by the quality of belief assigned to the seven individual sentiment hypotheses in the identification frame Θ. And the quality of belief assigned to the universal set Θ, which represents complete uncertainty. Together constitute;

[0075] S3052, Auditory BPA Generation: Define the set of emotion categories in S203 as follows Emotional category set Any emotion category is c a For any sentiment category in the identification framework Θ The corresponding auditory belief quality , is the probability output by S2032. Acoustic cognitive uncertainty score U output by S205 a Auditory environment distrust calculated by S3022 The results are calculated as follows:

[0076] like ,but:

[0077]

[0078] like ∉ ,Right now For the purpose of aversion, then:

[0079]

[0080] The remaining beliefs that were not assigned to any specific emotion are quantified and uniformly assigned to the belief quality of the universal set Θ, which represents complete uncertainty. The belief quality is:

[0081]

[0082] Auditory modality basic probability assignment (BPA) is the quality of belief assigned to six single emotion hypotheses that are identifiable in the auditory modality. A belief quality explicitly set to zero And the quality of belief assigned to the universal set Θ, which represents complete uncertainty. Together constitute;

[0083] S306. Using Dempster-Shafer evidence theory, the combined rule evidence fusion combines the basic probability assignments of the visual modality and the basic probability assignments of the auditory modality based on the degree of conflict between the visual and auditory modalities to generate a fused basic probability assignment (BPA).

[0084] S3061. Combine the visual modality basic probability assignment (BPA) and the auditory modality basic probability assignment (BPA) to generate a fused basic probability assignment (BPA); for any subset in the recognition frame Θ... The formula for calculating the quality of beliefs after fusion is as follows:

[0085]

[0086] In the formula, For any subset of the frame Θ to be identified, it represents an sentiment hypothesis;

[0087] For the integrated BPA in the affective hypothesis The assigned belief value, that is, the emotion of the interactive object after the fusion of audiovisual information, is... Ultimate level of trust;

[0088] and The variable in formula S3061 is used to aggregate all evidence. During the calculation of S3061, all possible [evidence] will be traversed. and The combination;

[0089] The emotion representing any non-zero belief from the visual BPA, according to the definition in S3051, has a probability of being recognized within the frame of reference. The seven individual emotions and the complete set representing utter uncertainty There are a total of 8 types;

[0090] The emotion representing any non-zero belief from auditory BPA, according to the definition in S3052, has the probability of being any of the six single emotions identifiable by the auditory modality plus the entire set representing complete uncertainty. There are a total of 7 types;

[0091] For visual BPA The assigned belief value is calculated by step S3051.

[0092] For hearing BPA The assigned belief value is calculated by step S3052.

[0093] The conflict coefficient is used to measure the degree of conflict between visual and auditory evidence, and its calculation method is defined by S3062;

[0094] To perform evidence fusion, the combination rule formula of S3061 takes the visual basic probability assignment (BPA) generated by S3051 and the auditory basic probability assignment (BPA) generated by S3052 as direct inputs. The specific substitution process is as follows: [Formula details omitted]. The value of the term is directly taken from the calculation output of S3051; the formula in The value of the term is directly taken from the calculation output of S3052; by traversing all terms that satisfy the condition... By combining the conditions and multiplying and summing the corresponding belief quality values, the quantitative fusion of cross-modal evidence can be achieved.

[0095] The final output of S3061 is the fused basic probability distribution BPA, denoted as... It is assigned to the identification framework Each individual emotion and the entire series It is constituted by the quality of one's own beliefs;

[0096] S3062. To measure the degree of conflict between two sources of evidence, the conflict coefficient is used. The calculation formula is:

[0097]

[0098] When K=1, it indicates that the evidence is completely conflicting, which is considered the highest level of conflict;

[0099] S307. Based on the fused Basic Probability Allocation (BPA) and Conflict Coefficient (K), perform gating and collaborative decision-making to output the final sentiment category.

[0100] As a further improvement to the present invention, S307 specifically includes:

[0101] S3071, Gating Decision: Gating decision is made based on the fused BPA structure and the conflict coefficient K;

[0102] S30711, Low-conflict decision path: If the inter-modal conflict coefficient K is lower than the low-conflict threshold, and the fused basic probability assignment... If all three of the following belief-dominant conditions are met simultaneously, the fusion decision is deemed credible, and the sentiment category with the highest belief value is adopted. As the final output:

[0103] (a) Maximum belief threshold condition: Belief quality of the emotion category with the highest belief value. Not lower than a preset minimum confidence threshold;

[0104] (b) Confidence odds ratio condition: highest belief value With the second highest belief value The ratio is not lower than a preset advantage ratio threshold;

[0105] (c) Global uncertainty condition: the quality of beliefs assigned to the entire set Θ of the identification framework It is below a preset global uncertainty limit;

[0106] S30712, High-Conflict Decision Path: If any of the following occurs, the current fusion result is determined to be unreliable, and the collaborative decision-making strategy of S308 is activated:

[0107] (a) Conflict of evidence: intermodal conflict coefficient Equal to or exceeding the conflict threshold;

[0108] (b) Decision fuzziness: Basic probability assignment after fusion when the value is below the conflict threshold. Failed to meet any of the belief-dominant conditions defined in Clause S3071;

[0109] S3072, Collaborative Decision Making: Examine the calculation results of S3062 over a past period and make S3071 decisions.

[0110] As a further improvement to the present invention, the humanoid robot behavior expression strategy specifically includes:

[0111] S401, Final Emotion Decision: S301-S307 running on the PC output the final emotion category;

[0112] S402, Data Packet Encoding and Transmission: Encode the emotion category into a standard data packet and, as a TCP / IP client, send the data packet over the network to the specified IP address and port of the robot's built-in Raspberry Pi;

[0113] S403, Command Reception and Parsing: The TCP / IP server program running on the Raspberry Pi listens for and receives data packets from the PC and parses out the emotional commands;

[0114] S404, Robot Action Execution: Based on the parsed emotional instructions, the Raspberry Pi sends corresponding low-level control commands to the robot serially. According to the preset emotional action mapping table, it drives the servo system to make the robot perform the corresponding limb movements that express the emotion.

[0115] The beneficial effects of this invention are as follows:

[0116] (1) This invention overcomes the vulnerability of single-channel perception in complex environments by integrating information from both visual and auditory modalities. An environmental uncertainty quantification mechanism is introduced to evaluate the current lighting conditions and background noise level in real time, and dynamically adjust the weights of visual and auditory perception in decision-making accordingly, so that the robot can dynamically adapt to environmental changes and improve the robot's emotion recognition ability in real and varied scenarios such as dim lighting and noisy environments.

[0117] (2) This invention establishes a two-layer uncertainty framework (i.e., a dual uncertainty framework), which quantifies environmental uncertainty and cognitive uncertainty of visual and auditory emotion recognition models, respectively. This two-layer uncertainty modeling provides a more comprehensive and refined basis for subsequent evidence fusion, making the entire decision-making process more rigorous and robust.

[0118] (3) This invention combines Dempster-Shafer evidence to fuse visual and auditory information, quantifies the degree of conflict between visual perception and auditory perception, and makes cross-modal decisions based on the degree of conflict, thereby greatly improving the reliability of the final decision and the rationality of the interaction.

[0119] (4) This invention constructs a unified identification framework covering visual and auditory emotion categories, and uniformly maps the outputs of models trained in different emotion spaces to this framework for basic probability allocation, thereby solving the common problem of heterogeneous classification spaces in multimodal perception systems. For modalities lacking a specific emotion category, the belief quality of that category is explicitly set to zero, which accurately expresses the modality's "ignorance" of this proposition, rather than providing negative evidence. This method not only theoretically guarantees the rigor of subsequent DS evidence fusion, but also avoids fusion errors or information loss caused by category mismatch in practice. Attached Figure Description

[0120] Figure 1 This invention provides a two-layer uncertainty framework and a cross-modal decision-making flowchart. Detailed Implementation

[0121] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0122] A method for expressing humanoid robot behavior based on visual and auditory emotion and environmental perception includes a method for perceiving facial emotions of interactive objects, a method for perceiving voice emotions of interactive objects, a cross-modal decision-making strategy based on visual and auditory emotion and environmental perception, and a humanoid robot behavior expression strategy.

[0123] The interactive object facial emotion perception method aims to extract emotion classification results from visual information and quantify its inherent cognitive uncertainty to evaluate the model's own prediction results, providing key weight information for subsequent cross-modal decision-making.

[0124] Methods for facial emotion perception of interactive objects include:

[0125] S101. Image Acquisition and Preprocessing: A binocular camera (1920×1080 resolution) with infrared illumination is used to acquire image streams and transmit them to the PC in real time. On the PC, face detection and face key point localization are performed on each frame of the image, and the face area is scaled to 48×48 pixels. RGB channel alignment and pixel value normalization to the range are performed.

[0126] S102. Adaptive Image Enhancement: Perform channel-wise histogram analysis on the preprocessed image and control contrast stretching through dynamic thresholding to improve the visibility of facial features in low-light or changing lighting environments.

[0127] S103. Preliminary estimation of visual emotion probability: The enhanced image from S102 is input into a lightweight convolutional neural network (CNN) for processing. This lightweight CNN model is pre-trained using the FER-2013 facial expression database and categorized into seven emotion classes based on the database: anger, fear, happiness, sadness, surprise, disgust, and neutral. The CNN model extracts and classifies features through its two sets of convolutional units (including 3×3 convolutional layers, batch normalization layers, ReLU layers, and 2×2 pooling layers) and the fully connected layer at the end. Finally, the Softmax function outputs a preliminary probability distribution of the seven emotion classes, i.e., the emotion probability vector P. va ;P va ={p an , p f , p ha , p fe , p su , p d , p ne}, where p an , p f , p ha , p fe , p su , p d , p ne The probability distributions of emotions are anger, fear, joy, sadness, surprise, disgust, and neutral, respectively.

[0128] S104, Visual Cognitive Uncertainty Quantification: To quantify the cognitive uncertainty of the model's prediction of the current input image, this step employs Monte Carlo Dropout technology: During the inference phase of the lightweight convolutional neural network model, the Dropout layer (dropout rate p=0.5) in the network is kept active. Subsequently, the same enhanced face image from S102 is repeatedly input into the lightweight convolutional neural network model for T (T=50) random forward propagation. Since the neurons randomly deactivated in each propagation are different, this process will generate a set {P} containing T slightly different emotion probability vectors. v (1) , P v (2) , …, P v (T)};

[0129] S105, Visual Modal Probability Output: Based on the set of T probability vectors generated in S104, calculate the final, smoothed visual emotion probability vector and a quantified uncertainty score. The two together constitute the visual modal output.

[0130] S1051. Take the arithmetic mean of the probability vectors obtained from the T random forward propagations in S104 to obtain the final visual emotion probability vector. For each sentiment category in S103, its average probability Where c represents the emotion category in S103, Let T represent the average probability that the model prediction is sentiment category c, obtained from T random forward propagations. Let c be the probability that the model predicts the emotion category c during the t-th forward propagation; the average probability combination of all emotion categories constitutes the visual emotion probability vector. ;

[0131] S1052. The cognitive uncertainty of the model is quantified by calculating the prediction variance of T probability vectors, and the uncertainty score U is obtained by averaging the variances of all emotion categories. v :

[0132]

[0133] Where C represents the number of emotion categories; emotion categories c = 1, 2, 3...C, and represent anger, fear, happiness, sadness, surprise, disgust and neutrality respectively.

[0134] The interactive object speech emotion perception method aims to extract emotion classification results from the speech information of interactive objects, and use the random forest model to quantify the cognitive uncertainty of speech emotion information, so as to provide key weight information for subsequent cross-modal decision making.

[0135] Methods for perceiving speech emotions in interactive objects include:

[0136] S201. Voice signal acquisition and processing: Audio signals are acquired using a microphone array and then sequentially amplified, filtered for noise reduction, and digitally processed before being transmitted to the PC in real time.

[0137] S202. Speech Feature Extraction and Dimensionality Reduction: The audio signal is preprocessed to extract the acoustic cepstral features, short-time energy, formants, and fundamental frequency, and the features are fused to form a speech emotion feature vector; PCA principal component analysis is used for dimensionality reduction.

[0138] S203, Model Training: Obtain the speech emotion feature vectors corresponding to the audio in the CASIA speech emotion database using the method in S202. Train the obtained speech emotion feature vectors using the random forest algorithm to establish a random forest model for recognizing speech emotion categories. The emotion classification of the random forest model includes anger, fear, happiness, sadness, surprise, and neutrality.

[0139] S2031. The random forest model consists of N decision trees. For a given input acoustic feature, each decision tree in the random forest (from the 1st to the Nth) independently outputs a sentiment classification vote.

[0140] set up Let represent the voting result of the i-th decision tree. To quantify the degree of disagreement among the decision trees within the random forest model, the vote of each decision tree is considered as an independent observation.

[0141] S2032, Calculate the acoustic emotion probability vector This allows for a comprehensive evaluation of the voting results with decision trees;

[0142] For any emotion category c a The corresponding acoustic emotion probability The calculation method is as follows:

[0143]

[0144] Where I(·) is an indicator function, which determines the vote of the i-th tree when the condition within the parentheses is true. For the emotion category Its value is 1 if it is positive and 0 otherwise; acoustic emotion probability vector It is a set of acoustic emotion probabilities corresponding to anger, fear, joy, sadness, surprise, and neutrality;

[0145] S204. Preliminary estimation of acoustic emotion probability: The speech emotion feature vector extracted from the audio signal obtained in S201 using the method in S202 is input into the random forest model established in S203 to obtain the acoustic emotion probability vector P of the audio signal. a ;

[0146] S205, Quantification of Acoustic Emotional Uncertainty: Quantifying the Acoustic Cognitive Uncertainty Score U a Defined as the average sample variance of voting results across six emotion categories; this acoustic cognitive uncertainty score U a The formula used to measure the dispersion of voting results among N decision trees in a random forest is as follows:

[0147]

[0148] Ua The higher the value, the greater the voting disagreement among the decision trees, the higher the volatility of the voting results, and the lower the cognitive confidence of this emotion recognition.

[0149] The cross-modal decision-making strategy based on visual and auditory emotion and environmental perception uses Dempster-Shafer (DS) evidence theory to make cross-modal decisions based on information from visual and auditory modalities with uncertainty measures.

[0150] Cross-modal decision-making strategies based on visual and auditory emotion and environmental perception include:

[0151] S301, Data Stream Temporal Alignment: Through a unified timestamp system, ensure that the visual data frames output by S105 and the audio data frames output by S205 are precisely aligned in time, providing synchronous input for subsequent fusion;

[0152] S302, Quantification of Environmental Uncertainty: Define visual environmental distrust levels respectively. Distrust of auditory environment Assess the quality of perceived information about the external environment, thereby serving as a dynamic adjustment factor in the evidence fusion process;

[0153] S3021. Calculating visual environment distrust based on ambient brightness detected by a light sensor. The ambient brightness L is obtained by reading the camera's own exposure parameters; the following formula is used to avoid sudden weight changes:

[0154]

[0155] In the formula, L0 is the preset brightness threshold (L0=50 lux), representing the lowest acceptable light level; k is the kurtosis parameter, controlling the abruptness of the transition; both parameters are determined empirically through performance calibration of specific camera hardware; when the ambient light intensity is much lower than L0, A value close to 1 indicates a high degree of distrust of visual evidence.

[0156] S3022. Calculating auditory environment distrust based on speech signal-to-noise ratio calculated using microphone array. The calculation formula is:

[0157]

[0158] In the formula, The current speech signal-to-noise ratio is calculated based on the microphone array. The preset signal-to-noise ratio threshold ( =20dB), representing the lowest acceptable signal-to-noise ratio level; k is the kurtosis parameter; when the signal-to-noise ratio decreases significantly... A value approaching 1 indicates a high degree of distrust in auditory evidence.

[0159] S303. Establish a cross-modal unified identification framework; the cross-modal unified identification framework is defined as a superset of the emotion categories that can be identified in the visual and auditory modalities, denoted as Θ, and Θ = {anger, fear, happiness, sadness, surprise, disgust, neutral}, which contains a total of 7 mutually exclusive single emotions.

[0160] S304. Establish a two-layer uncertainty framework, such as Figure 1 As shown. The first layer in the two-layer uncertainty framework. Used to assess cognitive uncertainty; the second layer in the two-layer uncertainty framework. , Environmental uncertainty used to assess the impact of the external environment on the quality of visual and auditory data, i.e., the reliability of the input data itself;

[0161] S305. Generate basic probability assignments under the cross-modal unified identification framework, that is, combine the two-layer uncertainty framework described in S304 to convert the probability outputs of the visual and auditory modalities into basic probability assignments (BPA) under the unified identification framework Θ described in S303.

[0162] S3051, Visual BPA Generation: For any single sentiment hypothesis in the recognition frame Θ The quality of the beliefs it is endowed with The probability is output by S1052. The uncertainty fraction U output by S1052 v Visual environment distrust level calculated by S3021 The result is calculated using the following formula:

[0163]

[0164] After assigning corresponding trust levels to each specific emotion, the overall uncertainty in the system (originating from the uncertainty of the model itself) remains. Distrust of ambient lighting This is quantified into the remaining belief component. This portion of beliefs is then uniformly assigned to the identification framework representing the entire set of all possibilities. This inherently corresponds to a state where "current evidence is insufficient to make a clear judgment" (or a state of "ignorance"). Its belief quality is:

[0165]

[0166] The final output of this step is the Basic Probability Assignment (BPA) of the visual modality, which is determined by the belief quality assigned to the seven individual sentiment hypotheses in the recognition frame Θ. And the quality of belief assigned to the universal set Θ, which represents complete uncertainty. Together constitute;

[0167] S3052, Auditory BPA Generation: Define the set of emotion categories in S203 as follows Emotional category set Any emotion category is c a For any sentiment category in the identification framework Θ The corresponding auditory belief quality , is the probability output by S2032. Acoustic cognitive uncertainty score U output by S205 a Auditory environment distrust calculated by S3022 The results are calculated as follows:

[0168] like ,but:

[0169]

[0170] like ∉ ,Right now For the purpose of aversion, then:

[0171]

[0172] Similar to the visual modality, this incorporates the uncertainty of the auditory model. Distrust of environmental noise Subsequently, the remaining beliefs that were not assigned to any specific emotion were quantified and uniformly assigned to the identification framework representing the state of "insufficient current evidence to make a clear judgment" (or the state of "ignorance"). Its belief quality is:

[0173]

[0174] The final output of this step is the auditory modality basic probability assignment (BPA) under a unified 7-class emotion recognition framework Θ. It consists of the belief quality assigned to the 6 single emotion hypotheses that can be identified in the modality, an explicit zeroed belief quality (corresponding to "disgust"), and the belief quality assigned to the entire set Θ representing complete uncertainty.

[0175] S306. Using Dempster-Shafer evidence theory, the combined rule evidence fusion combines the basic probability assignments of the visual modality and the basic probability assignments of the auditory modality based on the degree of conflict between the visual and auditory modalities to generate a fused basic probability assignment (BPA).

[0176] S3061. Combine the visual modality basic probability assignment (BPA) and the auditory modality basic probability assignment (BPA) to generate a fused basic probability assignment (BPA): For any subset in the identification frame Θ The formula for calculating the quality of beliefs after fusion is as follows:

[0177]

[0178] In the formula, For any subset of the identification frame Θ, representing an sentiment hypothesis; based on the basic probability allocation generation method described in S305 of this invention, the following is calculated using this formula: Its possibility is the identification framework The seven individual emotions (anger, fear, joy, sadness, surprise, disgust, and neutrality) and the complete set representing utter uncertainty. There are a total of 8 types;

[0179] For the integrated BPA in the affective hypothesis The assigned belief value, that is, the system's emotional response to the interactive object after the fusion of audiovisual information, is... Ultimate level of trust;

[0180] and The variable in formula S3061 is used to aggregate all evidence. During the calculation of S3061, all possible [evidence] will be traversed. and The combination;

[0181] The emotion representing any non-zero belief from the visual BPA, according to the definition in S3051, has a probability of being recognized within the frame of reference. The seven individual emotions (anger, fear, joy, sadness, surprise, disgust, and neutrality) and the complete set representing utter uncertainty. There are a total of 8 types;

[0182] The emotion representing any non-zero belief from the auditory BPA, according to the definition in S3052, since the auditory model does not include the "disgust" category, its probability is the six single emotions (anger, fear, happiness, sadness, surprise, neutral) identifiable by the auditory modality and the entire set representing complete uncertainty. There are a total of 7 types;

[0183] For visual BPA The assigned belief value represents the visual modality's association with "emotion as..." The support level for this judgment is calculated in step S3051.

[0184] For hearing BPA The assigned belief value represents the auditory modality's response to "emotion as..." The support level for this judgment is calculated in step S3052.

[0185] The conflict coefficient is used to measure the degree of conflict between visual and auditory evidence, and its calculation method is defined by S3062;

[0186] To perform evidence fusion, the combination rule formula of S3061 takes the visual basic probability assignment (BPA) generated by S3051 and the auditory basic probability assignment (BPA) generated by S3052 as direct inputs. The specific substitution process is as follows: [Formula details omitted]. The value of the term is directly taken from the calculation output of S3051; for example, if For "happy", then The value is the one calculated in S3051. ;like For the universal set Θ, its value is the value calculated by S3051. Similarly, in the formula... The value of the term is directly taken from the calculation output of S3052; by traversing all terms that satisfy the condition... By combining the conditions and multiplying and summing the corresponding belief quality values, the quantitative fusion of cross-modal evidence can be achieved.

[0187] The final output of S3061 is the fused basic probability distribution BPA, denoted as... It is assigned to the identification framework Each individual emotion and the entire series The quality of their own beliefs collectively constitutes a quantitative and unified evidentiary basis for subsequent cross-modal decision-making (S307);

[0188] S3062. To measure the degree of conflict between two sources of evidence, the conflict coefficient is used. The calculation formula is:

[0189]

[0190] When K=1, it indicates that the evidence is completely conflicting. Under the framework of this invention, this will be regarded as the highest level of conflict and handled by the high-conflict decision path of S3072.

[0191] S307. Based on the fused Basic Probability Allocation (BPA) and Conflict Coefficient (K), perform gating and collaborative decision-making to output the final sentiment category.

[0192] S3071, Gating Decision: Gating decision is made based on the fused BPA structure and the conflict coefficient K;

[0193] S30711, Low-conflict decision path: If the inter-modal conflict coefficient K is lower than the low-conflict threshold (K<0.5), and the basic probability allocation after fusion is... If all three of the following belief-dominant conditions are met simultaneously, the fusion decision is deemed credible, and the sentiment category with the highest belief value is adopted. As the final output:

[0194] (a) Maximum belief threshold condition: Belief quality of the emotion category with the highest belief value. Not lower than a preset minimum confidence threshold ( ≥0.4);

[0195] (b) Confidence odds ratio condition: highest belief value With the second highest belief value The ratio is not lower than a preset advantage ratio threshold. / ≥1.5);

[0196] (c) Global uncertainty condition: the quality of beliefs assigned to the entire set Θ of the identification framework Below a preset global uncertainty limit ( <0.5);

[0197] S30712, High-Conflict Decision Path: If any of the following occurs, the system determines that the current fusion result is unreliable and initiates the collaborative decision-making strategy of S308:

[0198] (a) Conflict of evidence: intermodal conflict coefficient Equal to or exceeding the conflict threshold (K≥0.5);

[0199] (b) Decision ambiguity: Basic probability assignment after fusion when the value is below the conflict threshold (K<0.5). Failed to meet any of the belief-dominant conditions defined in Clause S3071;

[0200] S3072, Collaborative Decision Making: Check the calculation results of S3062 in the past 10 seconds and make a decision in S3071.

[0201] The humanoid robot behavior expression strategy is responsible for translating the emotional category of the final decision into the robot's physical actions, and implementing this through a specified hardware link.

[0202] Humanoid robot behavior expression strategies specifically include:

[0203] S401, Final Emotion Decision: The S301-S307 decision strategy modules running on the PC output the final emotion category;

[0204] S402, Data Packet Encoding and Transmission: Encode the emotion category into a standard data packet and, as a TCP / IP client, send the data packet over the network to the specified IP address and port of the robot's built-in Raspberry Pi;

[0205] S403, Command Reception and Parsing: The TCP / IP server program running on the Raspberry Pi listens for and receives data packets from the PC and parses out the emotional commands;

[0206] S404, Robot Action Execution: Based on the parsed emotional instructions, the Raspberry Pi sends corresponding low-level control commands to the robot serially. According to the preset emotion-action mapping table, it drives the servo system to make the robot perform corresponding limb movements that express the emotion.

[0207] The scope of protection of this invention includes, but is not limited to, the above embodiments. The scope of protection of this invention is defined by the claims. Any substitutions, modifications, or improvements to this technology that are easily conceived by those skilled in the art fall within the scope of protection of this invention.

Claims

1. A method for expressing the behavior of humanoid robots based on visual and auditory emotion and environmental perception, characterized in that, This includes methods for perceiving facial emotions of interactive objects, methods for perceiving voice emotions of interactive objects, cross-modal decision-making strategies based on visual and auditory emotions and environmental perception, and behavioral expression strategies for humanoid robots. Methods for facial emotion perception of interactive objects include: S101. Image Acquisition and Preprocessing: Acquire image streams, perform face detection and facial landmark localization on each frame, and preprocess the face regions. S102, Adaptive Image Enhancement: Perform adaptive image enhancement on the preprocessed image; S103, Preliminary estimation of visual emotion probability: The enhanced image from S102 is input into a lightweight convolutional neural network for processing. The lightweight convolutional neural network model is pre-trained using a facial expression database. The lightweight convolutional neural network model outputs a preliminary probability distribution of 7 emotion categories, namely anger, fear, happiness, sadness, surprise, disgust, and neutral emotion probability distribution. S104, Visual Cognition Uncertainty Quantification: During the inference phase of the lightweight convolutional neural network model, the Dropout layer in the network is kept active. Then, the same enhanced face image from S102 is repeatedly input into the lightweight convolutional neural network model for T random forward propagations. Since the neurons randomly deactivated in each propagation are different, this process will generate a set {P} containing T slightly different emotion probability vectors. v (1) , P v (2) , …,P v (T) }; S105, Visual Modal Probability Output: Based on the set of T sentiment probability vectors generated in S104, calculate the final, smoothed visual sentiment probability vector and a quantized uncertainty score. Together, they constitute the output of the visual modality; specifically: S1051. Take the arithmetic mean of the probability vectors obtained from the T random forward propagations in S104 to obtain the final visual emotion probability vector. For each sentiment category in S103, its average probability , where c represents the emotion category in S103, c=1,2,3……C, and respectively represent anger, fear, happiness, sadness, surprise, disgust and neutrality, and C represents the number of emotion categories; Let T represent the average probability that the model prediction is sentiment category c, obtained from T random forward propagations. Let c be the probability that the model predicts the emotion category c during the t-th forward propagation; the average probability combination of all emotion categories constitutes the visual emotion probability vector. ; S1052. The uncertainty score U is obtained by averaging the variances of all sentiment categories. v : Methods for perceiving speech emotions in interactive objects include: S201. Voice signal acquisition and processing: Acquire audio signals and process them; S202. Speech Feature Extraction and Dimensionality Reduction: Extract the acoustic cepstral features, short-time energy, formants, and fundamental frequency of the audio, and perform feature fusion to form a speech emotion feature vector; and perform dimensionality reduction processing on the speech emotion feature vector; S203, Model Training: Using the method in S202, obtain the speech emotion feature vectors corresponding to the audio in the CASIA speech emotion database. Train the obtained speech emotion feature vectors using the random forest algorithm to establish a random forest model for recognizing speech emotion categories. The emotion classification of the random forest model includes anger, fear, happiness, sadness, surprise, and neutrality; specifically: S2031. The random forest model consists of N decision trees. For a given input acoustic feature, each decision tree in the random forest, from the 1st to the Nth tree, independently outputs a sentiment classification vote. set up Let represent the voting result of the i-th decision tree. To quantify the degree of disagreement among the decision trees within the random forest model, the vote of each decision tree is considered as an independent observation. S2032, Calculate the acoustic emotion probability vector This allows for a comprehensive evaluation of the voting results with decision trees; For any emotion category c a The corresponding acoustic emotion probability The calculation method is as follows: Where I(·) is an indicator function, which determines the vote of the i-th tree when the condition within the parentheses is true. For the emotion category Its value is 1 if it is positive and 0 otherwise; acoustic emotion probability vector It is a set of acoustic emotion probabilities corresponding to anger, fear, joy, sadness, surprise, and neutrality; S204. Preliminary estimation of acoustic emotion probability: The speech emotion feature vector extracted from the audio signal obtained in S201 using the method in S202 is input into the random forest model established in S203 to obtain the acoustic emotion probability vector P of the audio signal. a ; S205, Acoustic Emotion Uncertainty Quantification: Based on Acoustic Emotion Probability Vector P a Calculate the acoustic cognitive uncertainty score U a ; The acoustic cognitive uncertainty score U a Defined as the average sample variance of voting results across six emotion categories; this acoustic cognitive uncertainty score U a The formula used to measure the dispersion of voting results among N decision trees in a random forest is as follows: Cross-modal decision-making strategies based on visual and auditory emotion and environmental perception include: S301, Data Stream Time Domain Alignment: Through a unified timestamp system, ensure that the visual data frames output by S105 and the audio data frames output by S205 are precisely aligned in time. S302, Quantification of Environmental Uncertainty: Define visual environmental distrust levels respectively. Distrust of auditory environment ;in, S3021. Obtain the current ambient brightness L by reading the camera's own exposure parameters; visual environment distrust level. for: In the formula, L0 is the preset brightness threshold; k is the kurtosis parameter; when the ambient light intensity is much lower than L0, A value close to 1 indicates a high degree of distrust of visual evidence. S3022. Calculating auditory environment distrust based on speech signal-to-noise ratio calculated using microphone array. The calculation formula is: In the formula, The current speech signal-to-noise ratio is calculated based on the microphone array. The preset signal-to-noise ratio threshold is defined by k, where k is the kurtosis parameter; when the signal-to-noise ratio decreases significantly, A value approaching 1 indicates a high degree of distrust in auditory evidence. S303. Establish a cross-modal unified identification framework; the cross-modal unified identification framework is defined as a superset of the emotion categories that can be identified in the visual and auditory modalities, denoted as Θ, and Θ = {anger, fear, happiness, sadness, surprise, disgust, neutral}, which contains a total of 7 mutually exclusive single emotions. S304. Establish a two-layer uncertainty framework, the first layer of which is... Used to assess cognitive uncertainty; the second layer in the two-layer uncertainty framework. , Environmental uncertainty used to assess the impact of the external environment on the quality of visual and auditory data; S305. Generate the basic probability allocation under the cross-modal unified identification framework, that is, combine the two-layer uncertainty framework of S304 to convert the probability outputs of the visual and auditory modalities into the basic probability allocation under the unified identification framework Θ described in S303; specifically: S3051, Visual Modal Basic Probability Assignment (BPA) Generation: For any single sentiment hypothesis in the recognition frame Θ The quality of the beliefs it is endowed with The probability is output by S105. Uncertainty score U v Visual environment distrust level calculated by S3021 The result is calculated using the following formula: The total uncertainty is quantified into remaining beliefs, and these beliefs are uniformly assigned to the belief quality of the universal set Θ, which represents complete uncertainty. The belief quality is: The visual modality basic probability assignment (BPA) is determined by the quality of belief assigned to the seven individual sentiment hypotheses in the identification frame Θ. And the quality of belief assigned to the universal set Θ, which represents complete uncertainty. Together constitute; S3052, Auditory Modal Basic Probability Assignment (BPA) Generation: Define the set of emotion categories in S203 as follows: Emotional category set Any emotion category is c a For any sentiment category in the identification framework Θ The corresponding auditory belief quality , is the probability output by S2032. Acoustic cognitive uncertainty score U output by S205 a Auditory environment distrust calculated by S3022 The results are calculated as follows: like ,but: like ∉ ,Right now For the purpose of aversion, then: The remaining beliefs that were not assigned to any specific emotion are quantified and uniformly assigned to the belief quality of the universal set Θ, which represents complete uncertainty. The belief quality is: Auditory modality basic probability assignment (BPA) is the quality of belief assigned to six single emotion hypotheses that are identifiable in the auditory modality. A belief quality explicitly set to zero And the quality of belief assigned to the universal set Θ, which represents complete uncertainty. Together constitute; S306. Based on the degree of conflict between vision and hearing, the basic probability assignments of the visual modality and the basic probability assignments of the auditory modality are combined to generate a fused basic probability assignment (BPA). S307. Based on the fused Basic Probability Allocation (BPA) and the degree of conflict, perform gating and collaborative decision-making to output the final sentiment category. Humanoid robot behavior expression strategies include: S401, Final Emotion Decision: S301-S307 running on the PC output the final emotion category; S402, Data Packet Encoding and Transmission: The PC will encode the final emotion category into a standard data packet and send the data packet to the robot via the network; S403, Command Reception and Parsing: The robot receives data packets from the PC and parses out the emotion commands corresponding to the emotion categories; S404, Robot Action Execution: Based on the parsed emotional instructions and the preset emotional action mapping table, the robot drives its own servo system to execute the corresponding limb actions that express the emotion.

2. The method for expressing humanoid robot behavior based on visual and auditory emotion and environmental perception according to claim 1, characterized in that, The facial emotion perception method for interactive objects specifically includes: S101. Image Acquisition and Preprocessing: A binocular camera with infrared illumination is used to acquire image streams and transmit them to the PC in real time. On the PC, face detection and facial key point localization are performed on each frame of the image, and the face area is scaled to 48×48 pixels. RGB channel alignment and pixel value normalization to the range are performed. S102, Adaptive Image Enhancement: Perform channel-wise histogram analysis on the preprocessed image and control contrast stretching through dynamic thresholding; S103. Preliminary estimation of visual emotion probability: The enhanced image from S102 is input into a lightweight convolutional neural network for processing. This lightweight convolutional neural network model is pre-trained using the FER-2013 facial expression database and categorized into seven emotion classes based on the database: anger, fear, happiness, sadness, surprise, disgust, and neutral. The lightweight convolutional neural network model extracts and classifies features through its two sets of convolutional units and fully connected layers at the ends, and finally outputs a preliminary probability distribution of the seven emotion classes, i.e., the emotion probability vector P, using the Softmax function. va ;P va ={p an , p f , p ha , p fe , p su , p d , p ne }, where p an , p f , p ha , p fe , p su , p d , p ne The probability distributions of emotions are anger, fear, joy, sadness, surprise, disgust, and neutral, respectively. Both sets of convolutional units include 3×3 convolutional layers, batch normalization layers, ReLU layers, and 2×2 pooling layers; S104, Visual Cognition Uncertainty Quantification: During the inference phase of the lightweight convolutional neural network model, the Dropout layer in the network is kept active, with a dropout rate p of 0.

5. Then, the same enhanced face image from S102 is repeatedly input into the lightweight convolutional neural network model for T random forward propagations, where T is 50. Since the neurons randomly deactivated in each propagation are different, this process will generate a set {P} containing T slightly different emotion probability vectors. v (1) , P v (2) , …, P v (T) }; S105, Visual Modal Probability Output: Based on the set of T probability vectors generated in S104, calculate the final, smoothed visual emotion probability vector and a quantified uncertainty score. The two together constitute the visual modal output.

3. The method for expressing humanoid robot behavior based on visual and auditory emotion and environmental perception according to claim 1, characterized in that, The method for perceiving the voice emotion of the interactive object specifically includes: S201. Voice signal acquisition and processing: Audio signals are acquired using a microphone array and then sequentially amplified, filtered for noise reduction, and digitally processed before being transmitted to the PC in real time. S202. Speech Feature Extraction and Dimensionality Reduction: The audio signal is preprocessed to extract the acoustic cepstral features, short-time energy, formants, and fundamental frequency, and the features are fused to form a speech emotion feature vector; PCA principal component analysis is used for dimensionality reduction. S203, Model Training: Obtain the speech emotion feature vectors corresponding to the audio in the CASIA speech emotion database using the method in S202. Train the obtained speech emotion feature vectors using the random forest algorithm to establish a random forest model for recognizing speech emotion categories. The emotion classification of the random forest model includes anger, fear, happiness, sadness, surprise, and neutrality. S204. Preliminary estimation of acoustic emotion probability: The speech emotion feature vector extracted from the audio signal obtained in S201 using the method in S202 is input into the random forest model established in S203 to obtain the acoustic emotion probability vector P of the audio signal. a ; S205, Quantification of Acoustic Emotion Uncertainty.

4. The method for expressing humanoid robot behavior based on visual and auditory emotion and environmental perception according to claim 1, characterized in that, The cross-modal decision-making strategy based on visual and auditory emotion and environmental perception includes: S301, Data Stream Time Domain Alignment: Through a unified timestamp system, ensure that the visual data frames output by S105 and the audio data frames output by S205 are precisely aligned in time. S302, Quantification of Environmental Uncertainty: S303. Establish a cross-modal unified identification framework; the cross-modal unified identification framework is defined as a superset of the emotion categories that can be identified in the visual and auditory modalities, denoted as Θ, and Θ = {anger, fear, happiness, sadness, surprise, disgust, neutral}, which contains a total of 7 mutually exclusive single emotions. S304. Establish a two-layer uncertainty framework, the first layer of which is... Used to assess cognitive uncertainty; the second layer in the two-layer uncertainty framework. , Environmental uncertainty used to assess the impact of the external environment on the quality of visual and auditory data; S305. Generate basic probability allocation under the cross-modal unified identification framework, that is, combine the two-layer uncertainty framework described in S304 to convert the probability outputs of visual and auditory modalities into basic probability allocation (BPA) under the unified identification framework Θ described in S303. S306. Using Dempster-Shafer evidence theory, the combined rule evidence fusion combines the basic probability assignments of the visual modality and the basic probability assignments of the auditory modality based on the degree of conflict between the visual and auditory modalities to generate a fused basic probability assignment (BPA). S3061. Combine the visual modality basic probability assignment (BPA) and the auditory modality basic probability assignment (BPA) to generate a fused basic probability assignment (BPA); for any subset in the recognition frame Θ... The formula for calculating the quality of beliefs after fusion is as follows: In the formula, For any subset of the frame Θ to be identified, it represents an sentiment hypothesis; For the integrated BPA in the affective hypothesis The assigned belief value, that is, the emotion of the interactive object after the fusion of audiovisual information, is... Ultimate level of trust; and The variable in formula S3061 is used to aggregate all evidence. During the calculation of S3061, all possible [evidence] will be traversed. and The combination; The emotion representing any non-zero belief from the visual BPA, according to the definition in S3051, has a probability of being recognized within the frame of reference. The seven individual emotions and the complete set representing utter uncertainty There are a total of 8 types; The emotion representing any non-zero belief from auditory BPA, according to the definition in S3052, has the probability of being any of the six single emotions identifiable by the auditory modality plus the entire set representing complete uncertainty. There are a total of 7 types; For visual BPA The assigned belief value is calculated by step S3051. For hearing BPA The assigned belief value is calculated by step S3052. The conflict coefficient is used to measure the degree of conflict between visual and auditory evidence, and its calculation method is defined by S3062; To perform evidence fusion, the combination rule formula of S3061 takes the visual basic probability assignment (BPA) generated by S3051 and the auditory basic probability assignment (BPA) generated by S3052 as direct inputs. The specific substitution process is as follows: [Formula details omitted]. The value of the term is directly taken from the calculation output of S3051; the formula in The value of the term is directly taken from the calculation output of S3052; by traversing all terms that satisfy the condition... By combining the conditions and multiplying and summing the corresponding belief quality values, the quantitative fusion of cross-modal evidence can be achieved. The final output of S3061 is the fused basic probability distribution BPA, denoted as... It is assigned to the identification framework Each individual emotion and the entire series It is constituted by the quality of one's own beliefs; S3062. To measure the degree of conflict between two sources of evidence, the conflict coefficient is used. The calculation formula is: When K=1, it indicates that the evidence is completely conflicting, which is considered the highest level of conflict; S307. Based on the fused Basic Probability Allocation (BPA) and Conflict Coefficient (K), perform gating and collaborative decision-making to output the final sentiment category.

5. The method for expressing humanoid robot behavior based on visual and auditory emotion and environmental perception according to claim 4, characterized in that, The aforementioned S307 specifically includes: S3071, Gating Decision: Gating decision is made based on the fused BPA structure and the conflict coefficient K; S30711, Low-conflict decision path: If the inter-modal conflict coefficient K is lower than the low-conflict threshold, and the fused basic probability assignment... If all three of the following belief-dominant conditions are met simultaneously, the fusion decision is deemed credible, and the sentiment category with the highest belief value is adopted. As the final output: (a) Maximum belief threshold condition: Belief quality of the emotion category with the highest belief value. Not lower than a preset minimum confidence threshold; (b) Confidence odds ratio condition: highest belief value With the second highest belief value The ratio is not lower than a preset advantage ratio threshold; (c) Global uncertainty condition: the quality of beliefs assigned to the entire set Θ of the identification framework It is below a preset global uncertainty limit; S30712, High-Conflict Decision Path: If any of the following occurs, the current fusion result is determined to be unreliable, and the collaborative decision-making strategy of S308 is activated: (a) Conflict of evidence: intermodal conflict coefficient Equal to or exceeding the conflict threshold; (b) Decision fuzziness: Basic probability assignment after fusion when the value is below the conflict threshold. Failed to meet any of the belief-dominant conditions defined in Clause S3071; S3072, Collaborative Decision Making: Examine the calculation results of S3062 over a past period and make S3071 decisions.

6. The method for expressing humanoid robot behavior based on visual and auditory emotion and environmental perception according to claim 1, characterized in that, The humanoid robot behavior expression strategy specifically includes: S401, Final Emotion Decision: S301-S307 running on the PC output the final emotion category; S402, Data Packet Encoding and Transmission: Encode the emotion category into a standard data packet and, as a TCP / IP client, send the data packet over the network to the specified IP address and port of the robot's built-in Raspberry Pi; S403, Command Reception and Parsing: The TCP / IP server program running on the Raspberry Pi listens for and receives data packets from the PC and parses out the emotional commands; S404, Robot Action Execution: Based on the parsed emotional instructions, the Raspberry Pi sends corresponding low-level control commands to the robot serially. According to the preset emotional action mapping table, it drives the servo system to make the robot perform the corresponding limb movements that express the emotion.

Images