Robot interaction system and interaction method based on machine learning and emotion calculation

Abstract

The invention discloses a robot interaction system and method based on machine learning and emotion calculation, and the method comprises the steps: a robot collects the voice, facial expression, body language and physiological signal data of a user through a multi-mode perception device, and carries out the noise elimination, signal enhancement and feature extraction of the data; utilizing a deep neural network and an emotion classification algorithm to identify the emotion state of the user, identifying mixed emotion and performing label classification; a personalized emotion model is established through machine learning in combination with historical emotion data of the user, an emotion change rule is reflected, and long-term learning and adaptation are carried out; based on the sentiment analysis result, a response conforming to the user sentiment state is generated; through user feedback and interaction, the system continuously optimizes emotion understanding and reaction ability and updates emotion archives; emotion computing tasks are shared through edge computing or cloud computing resources, and real-time response to complex emotion computing tasks is ensured.

Description

Technical Field

[0001] The present invention belongs to the field of robot interaction in machine learning and affective computing, and particularly relates to a robot interaction system and an interaction method based on machine learning and affective computing. Background Art

[0002] A robot interaction system based on machine learning and affective computing is an intelligent system that integrates artificial intelligence technology, emotion recognition, and machine learning algorithms, aiming to enable robots to understand and respond to human emotions. Machine learning is a method of artificial intelligence that enables robots or systems to learn from experience through data and algorithms, gradually improving their task execution capabilities; affective computing refers to the ability of a computer to recognize, understand, simulate, and express emotions; a robot interaction system based on machine learning and affective computing usually has the characteristics of emotion perception and understanding, emotion response, and personalized interaction; through affective computing, a customer service robot can adjust its service strategy according to the user's emotion and provide more personalized support; in the care of the elderly or patients, an emotion perception robot can judge the emotional state of the patient and provide comfort or assistance in a timely manner; a robot can adjust its teaching method according to the emotional changes of students to improve the teaching effect.

[0003] However, although the robot interaction system based on machine learning and affective computing has made significant progress in many fields, robots in the prior art usually identify user emotions through means such as voice and facial expressions and give pre-determined responses, lacking sufficient adaptability and emotional depth, resulting in an unnatural and unsmooth interaction experience for users. Summary of the Invention

[0004] Aiming at the deficiencies in the prior art, the purpose of the present invention is to provide a robot interaction system and an interaction method based on machine learning and affective computing, achieving higher emotion recognition accuracy and more natural emotion responses through technical means such as multi-modal emotion perception, deep learning of emotion responses, personalized emotion modeling, and edge computing optimization.

[0005] The technical solution adopted by the present invention to solve its technical problems is as follows:

[0006] A robot interaction system based on machine learning and affective computing, comprising:

[0007] An emotion perception module: used to extract emotion information from the multi-modal data of the user and fuse different signals using a deep learning algorithm;

[0008] An emotion understanding module: based on a deep neural network and an emotion analysis algorithm, this module deeply analyzes the multi-modal emotion data obtained from the perception module, identifies the emotional state of the user, and processes complex emotion mixtures;

[0009] Personalized Emotion Modeling Module: Based on the user's historical emotion data and combined with reinforcement learning methods, it gradually optimizes the robot's ability to understand and respond to different users' emotions, and conducts personalized emotional interactions;

[0010] Emotion Generation and Response Module: Based on the analysis results of the Emotion Understanding Module, it generates responses that match the user's emotional state through a deep emotion generation model;

[0011] Edge Computing and Resource Optimization Module: By combining edge computing technologies, it distributes difficult-to-parse emotion computing tasks to edge devices or cloud platforms;

[0012] Preferably, the Emotion Perception Module includes sub-modules for speech recognition, facial expression recognition, body language analysis, and physiological signal acquisition; the Emotion Understanding Module conducts long-term tracking of emotions and gradually builds the user's emotion model; the emotion generation process of the Emotion Generation and Response Module is based on emotion diversity and situational adaptability, enabling the robot to adjust the intonation, tone, and content of its responses according to the user's emotional fluctuations.

[0013] A robot interaction method based on machine learning and emotion computing includes the following steps:

[0014] The robot uses multi-modal perception devices to collect the user's speech, facial expressions, body language, and physiological signal data in real-time to form emotion data input;

[0015] Preprocess the collected emotion data, including noise elimination, signal enhancement, feature extraction, and fuse emotion data from different sources;

[0016] Through a deep neural network and emotion classification algorithm, identify the user's specific emotional state, classify it into specific emotion labels, and at the same time identify the mixed state of complex emotions;

[0017] Based on the user's historical emotion data, combine machine learning algorithms to establish a personalized emotion model, reflect the laws of the user's emotional changes, and continuously learn and adapt in the long term;

[0018] Based on the emotion analysis results and combined with the user's personalized emotion model, generate responses that match the user's emotional state;

[0019] Through continuous user feedback and interactions, the system gradually learns and optimizes its emotion understanding and response capabilities, continuously updates the user's emotion profile, and distributes emotion computing tasks through an edge computing platform or cloud computing resources to ensure that the system can continuously respond in real-time under complex emotion computing tasks.

[0020] Preferably, the method by which the robot uses multi-modal perception devices to collect the user's speech, facial expressions, body language, and physiological signal data in real-time to form emotion data input is:

[0021] Assume that a voice signal passes through a speech recognition and emotion analysis model to obtain an emotion feature vector V specch , which includes speech features such as emotional tone, intonation, volume, and speech rate. The formula is:

[0022] V specch = f specch (x specch )

[0023] where x specch is the original voice signal, and f specch is the mapping function for speech feature extraction and emotion analysis. The generated V specch is an emotion-related feature vector;

[0024] Facial expressions are converted into an expression feature vector V face through facial recognition technology. This vector includes the type and intensity of the expression. The formula is:

[0025] V face = f face (x face )

[0026] where x face is the facial image data, and f face is the mapping function for facial expression feature extraction;

[0027] Body language is extracted as a feature vector V body through motion capture technology, including the user's posture, movement amplitude, and speed. The formula is:

[0028] V body = f body (x body )

[0029] where x body is the input data of body movements, and f body is the function for action or posture recognition;

[0030] Physiological signals are collected by physiological signal monitoring devices and converted into a feature vector V bio through signal processing, including the user's physiological state data, such as emotional fluctuations, tension, or relaxation indicators. The formula is:

[0031] V bio = f bio (x bio )

[0032] where x bio is the physiological signal data, and f bio is the mapping function for physiological signal analysis;

[0033] The feature vector V of the above multimodal data specch , V face , V body , and V bio Fusion is performed to obtain a comprehensive emotional data vector V cmotion , this fusion process uses weighted averaging, splicing, or fusion methods based on deep learning, and the formula is:

[0034] V cmotion =λ1V specch +λ2V face +λ3V body +λ4V bio

[0035] Among them, λ1, λ2, λ3, and λ4 are the weight coefficients of each mode.

[0036] As a preferred method, the collected emotional data is preprocessed, including noise elimination, signal enhancement, feature extraction, and the emotional data from different sources are fused as follows:

[0037] Noise elimination is processed by using filters or signal denoising methods. For the speech signal x specch , assuming y specch is the denoised speech signal:

[0038]

[0039] in, For noise cancellation operation;

[0040] Signal enhancement improves the effect of sentiment analysis by increasing the clarity of the signal. specch , the enhanced signal z specch By enhancing the algorithm ε, we can obtain:

[0041] z specch =ε(y specch )

[0042] Among them, ε is the signal enhancement algorithm;

[0043] For data on speech, facial expressions, body language, and physiological signal modalities, different feature extraction methods are used. The speech signal x specch Extract features such as Mel frequency cepstral coefficients, fundamental frequency, and speech rate to form a feature vector V specch :

[0044] V specch =f specch (z specch )

[0045] Among them, f specch It is the speech feature extraction function;

[0046] Facial expression is detected by facial feature points and emotion classification model to extract features, and a facial expression feature vector V is obtained. face :

[0047] V face =f face (x face )

[0048] The characteristics of body language are extracted from posture recognition and motion analysis to obtain a feature vector V body :

[0049] V face =f face (x face )

[0050] Physiological signals are analyzed by signal analysis to extract the corresponding physiological feature vector V bio :

[0051] V bio =f bio (x bio )

[0052] After obtaining the feature vector of each modality, the emotional data from different sources are then fused using weighted summation, splicing, or more complex deep learning methods. Assuming that the feature vector of each modality is V specch , V face , V body , V bio , the weight of each modality is λ1, λ2, λ3, λ4, then the fused emotional data vector V cmotion for:

[0053] V cmotion =λ1V specch +λ2V face +λ3V body +λ4V bio

[0054] Among them, λ1, λ2, λ3, and λ4 are learned through training and represent the relative importance of each mode;

[0055] Another common fusion method is to concatenate the feature vectors of each modality:

[0056] V cmotion =[V specch , V face , V body , V bio ]

[0057] In addition to weighted summation and concatenation, a deep neural network or a long short-term memory network model is also used for fusion. By learning the relationships between different modalities, emotional data is fused. Suppose there is a deep neural network model f fusion , then the fusion process is as follows:

[0058] V cmotio = f fusion (V specch , V face , V body , V bio )

[0059] where f fusion is a deep neural network obtained through training, which is used to learn the fusion rules from multiple modalities;

[0060] The fused emotional data vector V cmotion is used as input and passed to the sentiment analysis model, the emotion recognition model, or for other tasks.

[0061] Preferably, through a deep neural network and a sentiment classification algorithm, the specific emotional state of the user is identified and classified into specific emotional labels. At the same time, the method for identifying the mixed state of complex emotions is as follows:

[0062] The process of data preprocessing and feature extraction is similar to that mentioned above. Feature vectors are extracted through multimodal information. Suppose a fused feature vector V fusion is obtained, which contains all the information related to emotions and is in the form of:

[0063] V fusion = [V specch , V face , V body , V bio

[0064] where V specch , V facc , V body , V bio are feature vectors obtained through their respective feature extraction methods and are fused to obtain V fusion as input;

[0065] A deep neural network is used to train and classify the fused feature vector V fusion . Suppose there is a multi-layer perceptron model whose structure includes an input layer, a hidden layer, and an output layer. Suppose V fusion is the input. Each layer of the DNN model is calculated through weighted sum, bias, and activation function for the input feature vector V fusion :​

[0066] h1 = W1V fusion + b1

[0067] The activation function of the hidden layer uses ReLU:

[0068] h2 = ReLU(W2h1 + b2)

[0069] Finally, for the probability distribution of the output sentiment label, the softmax function is used to convert the output into the probability of each sentiment category:

[0070]

[0071] where g = c represents the sentiment label c, and P(y = c|V fusion ) is the probability belonging to the label c;

[0072] For sentiment classification, according to the output probability distribution, the sentiment label with the highest probability is selected as the prediction result. Suppose there are C sentiment categories C = {c1, c2,..., c C}, the prediction result is the label with the highest probability:

[0073]

[0074] The complex sentiment state is a mixture of multiple basic sentiments. When identifying mixed sentiments, the following method is adopted:

[0075] In multi-label classification, the output of the sentiment network is not only a single label, but the probability value of each sentiment category;

[0076] For the probability of each sentiment category c i , the Sigmoid activation function is used:

[0077] P(y = c i V fusion ) = σ(W i h2 + b i )

[0078] where is the Sigmoid function, and the output value represents the existence probability of this sentiment category.

[0079] Preferably, according to the user's historical sentiment data, a personalized sentiment model is established by combining machine learning algorithms to reflect the law of the user's sentiment change, and the method for long-term learning and adaptation is:

[0080] Construct appropriate input features by preprocessing the user's historical sentiment data. Suppose there is a time series dataset where each time step contains the user's sentiment data and auxiliary features, and these data are used to form a feature vector of the time series:

[0081] X t =[f1(t), f2(t),..., f N (t)]

[0082] where, X t represents the feature vector at time step t, f i (t) is the value of the i-th feature at time step t, and N is the total number of features;

[0083] The LSTM consists of an input gate, a forget gate, and an output gate. Its core is to control the transmission and update of information. At each time step t, the output h t and the hidden state c t are updated as follows:

[0084] Forget gate f t : f t =σ(W f ·[h t1 , X t +b f );

[0085] Input gate i t ; i t =σ(W i ·[h t1 , X t +b i );

[0086] Candidate memory cell

[0087] Update memory cell c t :

[0088] Output gate o t : o t =σ(W o ·[h t1 , X t +b o );

[0089] Hidden state h t : h t =o t ·tanh(c t );

[0090] By training the LSTM model, predict the user's sentiment state y at time step t t:

[0091] y i = softmax(W y ·h t + b y )

[0092] where W y is the weight matrix of the output layer, b y is the bias term, and the softmax function outputs a probability distribution over different sentiment labels;

[0093] Reinforcement learning is used to adjust the sentiment model so that the model can be updated in the long term according to real-time feedback;

[0094] State S t : The current sentiment prediction and historical sentiment information of the model;

[0095] Action A t : The adjustment or update strategy of the sentiment model;

[0096] Reward R t : The reward based on user feedback;

[0097] At each step, the model selects an action according to the current state and obtains a reward, and uses Q-learning to update the Q value:

[0098]

[0099] where α is the learning rate, γ is the discount factor, and R t is the current reward;

[0100] Between time steps t and t+1, the weights W and the bias b are adjusted according to the actual feedback:

[0101]

[0102] where η is the learning rate and L is the loss function, representing the difference between the model prediction and the actual sentiment label.

[0103] Preferably, the method for generating a response that conforms to the user's sentiment state based on the sentiment analysis result and in combination with the user's personalized sentiment model is:

[0104] Identify the user's current sentiment state through sentiment analysis. The user's current sentiment state S t , the intensity of the sentiment Intensity t , and the sentiment analysis output formula is:

[0105] S t = Happy, Intensity t = 0.8

[0106] Combined with the user's personalized emotion model, the user's emotional historical data is combined with the current emotional state to predict or adjust the user's emotional changes. Assuming the user's emotional historical data is {S1, S2,..., S i}, an individualized emotion adjustment factor α is output through the model t , which represents the personalized adjustment value of the current emotional state and reflects the long-term change trend of the user's emotion:

[0107] α t = f(S1, S2,..., S t )

[0108] where f is a function based on the user's historical emotion data;

[0109] Based on the output of the sentiment analysis and the personalized emotion model, a response that matches the user's emotional state is generated according to the current emotional state and the adjustment factor;

[0110] Assume there is a set of predefined response libraries, and each response is related to a specific emotional state: <L

[0111] Response R for happiness happy ;

[0112] Response R for sadness sorow ;

[0113] Response R for anger angry ;

[0114] According to the current emotional state S t and the emotional intensity Intensity t to select the most appropriate response, and the response selection formula is as follows:

[0115] R t = SelectResponse(S t , Intensity t )

[0116] where SelectResponse(S t , Intensity t ) is a function to select the most matching response according to the user's emotional state and intensity;

[0117] In addition to selecting the most matching response, fine-tuning is performed through the personalized emotion model. Based on the historical emotion data and the current personalized factor α t , the tone or diction of the response is adjusted to make the response more in line with the user's personalized needs. The formula for this adjustment is expressed as:

[0118] R adjusted = R t + α t ·ΔR

[0119] Wherein, R t is the basic response;

[0120] α t is the personalized emotion adjustment factor;

[0121] ΔR is the fine-tuning based on the current emotion state;

[0122] In summary, the method for generating a response that conforms to the user's emotion state is summarized into the following steps:

[0123] Emotion analysis: Analyze the user's current emotion state S t and emotion intensity Intensity t ;

[0124] Personalized emotion adjustment factor: Obtain the adjustment factor α through the personalized emotion model t ;

[0125] Response selection: Select the most appropriate response R t from the predefined response library, and make adjustments according to the emotion state S i and emotion intensity Intensity t ;

[0126] Personalized adjustment: Fine-tune the response according to the personalized factor α t to generate the final response.

[0127] Preferably, through continuous user feedback and interaction, the system gradually learns and optimizes the emotion understanding and response capabilities, continuously updates the user's emotion profile, and distributes the emotion calculation tasks through the edge computing platform or cloud computing resources. The method to ensure the system to continuously perform real-time response under complex emotion calculation tasks is:

[0128] After each user feedback, the system updates the user's emotion profile according to the new feedback information. Assuming that the user's emotion state can be represented as a vector S t , and the feedback information is F t , the system adjusts the emotion understanding model according to the feedback information, and the update process can be represented by the following formula:

[0129] S t+1 = S t + ΔS t

[0130] Wherein, S t+1 is the updated emotion state vector, and ΔS t represents according to the user feedback Ft The adjustment of the emotional state made, this adjustment ΔS t is updated by the system through an online learning algorithm;

[0131] Update the emotional state vector in the way of Q-learning or reinforcement learning:

[0132]

[0133] where Q(S t , A t ) is the quality value of a certain emotional response behavior A t in the current emotional state;

[0134] α is the learning rate;

[0135] R t is the reward obtained based on user feedback;

[0136] γ is the discount factor,

[0137] max A’ Q(S t+1 , A′) is the maximum Q value obtained in the next emotional state S t+1 ;

[0138] With the optimization of the emotional understanding ability, the user's emotional profile is continuously updated, recording historical emotional data, user feedback, and emotional intensity information. The emotional profile is represented as a multi-dimensional emotional vector C t , and each dimension represents the intensity of a certain emotional category;

[0139] After each user feedback, the emotional profile will be updated in the following way:

[0140] C t+1 = C t + ΔC t

[0141] where ΔC t represents the change in the emotional profile caused by user feedback;

[0142] To ensure that the system maintains real-time response when processing complex emotional computing tasks, the emotional computing tasks are distributed to the edge computing platform and cloud computing resources. The emotional computing tasks are divided into low-latency tasks and high-computation task types;

[0143] The decision basis for task allocation is dynamically adjusted through a load balancing algorithm. Assuming that the system allocates tasks to edge devices or cloud platforms according to the complexity and latency requirements of the tasks, the task allocation strategy is expressed as:

[0144] Task t=AssignTask(C t , Q t , Resources)

[0145] Wherein, C t is the current sentiment profile;

[0146] Q t is the computational complexity of the task;

[0147] Resources are the available computational resources;

[0148] AssignTask is a task allocation function responsible for selecting a suitable platform for processing according to the computational complexity and resource conditions;

[0149] To ensure that the system responds in real time under complex sentiment computing tasks, the collaborative work of edge computing and cloud computing is utilized, and the computing process is described by the following formula:

[0150] Response t =f(Task t , Time t , Resources)

[0151] Wherein, Response t is the sentiment response generated by the system at time t;

[0152] Task t is the current sentiment computing task;

[0153] Time t is the time requirement of the task;

[0154] Resources are the computational resources scheduled by the system.

[0155] Another technical problem to be solved by the present invention is to provide an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the robot interaction system and interaction method based on machine learning and sentiment computing as described in any of the above are implemented.

[0156] Another technical problem to be solved by the present invention is to provide a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the robot interaction system and interaction method based on machine learning and sentiment computing are implemented.

[0157] The beneficial effects of the present invention are:

[0158] By collecting voice, facial expressions, body language, and physiological signal data in real-time, the robot can comprehensively understand the user's emotional state from multiple perspectives; by processing and analyzing emotional data through deep neural networks, it can identify more complex emotional patterns, including mixed emotions; by analyzing the user's historical emotional data through machine learning algorithms, the robot can establish a personalized emotional model to understand the long-term trends and changing patterns of the user's emotions; based on the emotional analysis results and the personalized emotional model, the robot can generate responses that are more in line with the user's current emotional state; through the combination of edge computing and cloud computing, the system can process complex emotional computing tasks while ensuring real-time performance; by continuously collecting user feedback and optimizing the emotional understanding and response capabilities, the system can gradually improve the accuracy of emotional recognition and the quality of responses to users; emotional recognition and personalized responses make the robot appear more human and understanding, enhancing the user's trust and emotional connection. BRIEF DESCRIPTION OF THE DRAWINGS

[0159] Figure 1 FIG. is a schematic flowchart of a robot interaction system based on machine learning and emotional computing according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0160] The principles and features of the present invention will be described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention. The present invention will be described more specifically by way of example in the following paragraphs. The advantages and features of the present invention will be clearer according to the following description and the claims.

[0161] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used in the description of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0162] Embodiment

[0163] The technical solution adopted by the present invention to solve its technical problems is as follows:

[0164] A robot interaction system based on machine learning and emotional computing, comprising:

[0165] An emotion perception module: used to extract emotion information from the user's multi-modal data and fuse different signals using deep learning algorithms;

[0166] An emotion understanding module: based on a deep neural network and an emotion analysis algorithm, this module deeply analyzes the multi-modal emotion data obtained from the perception module, identifies the user's emotional state, and processes complex emotion mixtures;

[0167] Personalized Emotion Modeling Module: Based on the user's historical emotion data and combined with reinforcement learning methods, it gradually optimizes the robot's ability to understand and respond to different users' emotions and conducts personalized emotional interactions;

[0168] Emotion Generation and Response Module: Based on the analysis results of the Emotion Understanding Module, it generates responses that match the user's emotional state through a deep emotion generation model;

[0169] Edge Computing and Resource Optimization Module: By combining edge computing technology, it distributes difficult-to-parse emotion computing tasks to edge devices or cloud platforms;

[0170] Combined with multi-modal perception and deep learning technologies, the system can more accurately identify and analyze the user's emotional state; through reinforcement learning and historical emotion data, the system can gradually optimize emotional interactions according to the user's needs and provide personalized emotional experiences; the introduction of edge computing technology ensures that the system can still respond in real time when performing complex emotion calculations, avoiding delays or incorrect responses; the deep emotion generation model can generate responses that fit the user's emotional state, making the interaction more resonant and user-friendly, enhancing the user experience; the system will continuously learn and optimize based on the user's interaction feedback, making the accuracy of emotion recognition and response higher and higher.

[0171] The Emotion Perception Module includes sub-modules for speech recognition, facial expression recognition, body language analysis, and physiological signal acquisition; the Emotion Understanding Module conducts long-term tracking of emotions and gradually builds a user's emotion model; the emotion generation process of the Emotion Generation and Response Module is based on emotion diversity and situational adaptability, enabling the robot to adjust the intonation, tone, and content of its responses according to the user's emotional fluctuations.

[0172] Through multi-dimensional perception means such as speech, facial expressions, body language, and physiological signals, the robot can accurately capture the user's emotional changes from different angles, reducing the errors caused by a single perception method; the long-term tracking and personalized emotion model of the emotion understanding module enable the robot to gradually optimize its understanding of the user's emotions according to the unique emotional expressions of each user. This long-term tracking enables the robot to no longer be limited to one-time interactions, but to be able to "remember" the user's emotional characteristics and provide more appropriate responses; the emotion generation module adjusts the response based on the user's emotional state and specific situation, enabling the robot to flexibly handle different emotional scenarios. This flexibility and the diversity of emotional responses make the user's interaction experience more natural and comfortable, enhancing emotional resonance; the robot's responses are not just mechanical, but interactive with emotional depth, capable of making corresponding adjustments as the user's emotions fluctuate, which increases the robot's "emotional intelligence" and makes its interaction with the user more humanized; through the continuous optimization and personalized adjustment of the emotion model, the robot can establish an emotional connection with the user. Over time, the robot's emotion understanding and response capabilities will become more accurate, making the user feel more understood and cared for during each interaction.

[0173] A robot interaction method based on machine learning and emotion computing includes the following steps:

[0174] The robot uses multi-modal perception devices to collect the user's speech, facial expressions, body language, and physiological signal data in real time, forming an emotional data input;

[0175] Preprocess the collected emotional data, including noise elimination, signal enhancement, feature extraction, and fuse the emotional data from different sources;

[0176] Through a deep neural network and an emotion classification algorithm, identify the user's specific emotional state, classify it into specific emotion labels, and at the same time identify the mixed state of complex emotions;

[0177] According to the user's historical emotional data, combine machine learning algorithms to establish a personalized emotion model, reflect the laws of the user's emotional changes, and continuously learn and adapt in the long term;

[0178] Based on the emotion analysis results, combine the user's personalized emotion model to generate a response that conforms to the user's emotional state;

[0179] Through continuous user feedback and interaction, the system gradually learns and optimizes its emotion understanding and response capabilities, continuously updates the user's emotional profile, and distributes the emotion computing tasks through an edge computing platform or cloud computing resources to ensure that the system can continuously perform real-time responses under complex emotion computing tasks.

[0180] Multi-modal perception and data fusion enable the robot to obtain user emotion information from multiple perspectives, thus more accurately identifying the user's emotional state; the personalized emotion model enables the robot to better adapt to the emotional needs of each user. As the interaction between the user and the robot increases, the robot's understanding of the user's emotions will continue to deepen and optimize, providing more personalized services; the robot can dynamically adjust the tone, content, and manner of response according to the user's emotional state, making the interaction with the user more natural and fluent; through continuous learning and adaptation based on user feedback, the system can optimize the emotion recognition and response strategies, gradually improving the accuracy of its emotion computing and the response quality; with the help of cloud computing and edge computing platforms, the system can efficiently process complex emotion computing tasks, ensuring real-time response capabilities; through emotion computing, the robot can better understand the user's emotional needs and provide a more user-friendly interaction experience; through long-term learning and optimization, the robot can not only understand the current emotional state of the user, but also predict possible future emotional changes and prepare appropriate responses in advance.

[0181] The method for the robot to form emotional data input by using multi-modal perception devices to collect the user's voice, facial expressions, body language, and physiological signal data in real time is as follows:

[0182] Suppose the voice signal obtains an emotional feature vector V through a speech recognition and emotion analysis model specch , which contains the speech features of emotional tone, intonation, volume, and speech rate. The formula is:

[0183] V specch = f specch (x specch )

[0184] Among them, x specch is the original voice signal, and f specch is the mapping function for speech feature extraction and emotion analysis. The generated V specch is an emotion-related feature vector;

[0185] The facial expression is converted into an expression feature vector V through facial recognition technology face , which contains the type and intensity of the expression. The formula is:

[0186] V face = f face (x face )

[0187] Among them, x face is the facial image data, and f face is the mapping function for facial expression feature extraction;

[0188] The body language is the feature vector V extracted through motion capture technologybody , including the user's posture, movement amplitude and speed, with the formula:

[0189] V body = f body (x body )

[0190] where x body is the input data of the body movement, and f body is the function for action or posture recognition;

[0191] Physiological signals are collected by a physiological signal monitoring device and converted into a feature vector V bio , which contains the user's physiological state data, such as emotional fluctuations, tension or relaxation indicators, with the formula:

[0192] V bio = f bio (x bio )

[0193] where x bio is the physiological signal data, and f bio is the mapping function for physiological signal analysis;

[0194] Fuse the feature vectors V specch , V face , V body , and V bio of the above multi-modal data to obtain a comprehensive emotional data vector V cmotion . This fusion process uses weighted average, concatenation, or a deep learning-based fusion method, with the formula:

[0195] V cmotion = λ1V specch + λ2V face + λ3V body + λ4V bio

[0196] where λ1, λ2, λ3, and λ4 are the weight coefficients of each modality.

[0197] Through the fusion of multi-modal data, the system can integrate information from various aspects to conduct more accurate sentiment analysis; the system can establish a personalized sentiment model based on the user's sentiment data history and real-time responses, thereby providing a tailored interactive experience; the robot can adjust its responses according to the user's sentiment state, making the interaction with the robot more natural and emotional; by using cloud computing and edge computing resources, the system can quickly process large-scale multi-modal sentiment data, ensuring real-time response even under complex sentiment computing tasks, and thus providing a latency-free user experience; as the user interaction continues, the system makes sentiment computing increasingly accurate and adaptable by continuously learning and optimizing its sentiment understanding and response capabilities; the system can not only respond to a single sentiment state but also handle various complex mixed sentiment situations.

[0198] The method for preprocessing the collected sentiment data, including noise elimination, signal enhancement, feature extraction, and fusing sentiment data from different sources is as follows:

[0199] Noise elimination is processed using a filter or signal denoising method. For the speech signal x specch , assuming y specch is the denoised speech signal:

[0200]

[0201] where, is the noise elimination operation;

[0202] Signal enhancement improves the effect of sentiment analysis by enhancing the clarity of the signal. For the speech signal y specch , the enhanced signal z specch is obtained by processing through the enhancement algorithm ε:

[0203] z specch = ε(y specch )

[0204] where, ε is the signal enhancement algorithm;

[0205] For data of speech, facial expressions, body language, and physiological signal modalities, different feature extraction methods are adopted. For the speech signal x specch features such as Mel-frequency cepstral coefficients, fundamental frequency, speech rate features are extracted to form a feature vector V specch :

[0206] V specch = f specch (z specch )

[0207] where, f specch is the speech feature extraction function;

[0208] Facial expressions extract features through facial feature point detection and an emotion classification model to obtain a facial expression feature vector V face :

[0209] V face = f face (x face )

[0210] The features of body language are extracted from pose recognition and motion analysis to obtain a feature vector V body :

[0211] V face = f face (x face )

[0212] Physiological signals extract corresponding physiological feature vectors V through signal analysis methods bio :

[0213] V bio = f bio (x bio )

[0214] After obtaining the feature vectors of each modality, next, the emotion data from different sources are fused, and weighted summation, concatenation, or more complex deep learning methods are used for fusion. Assume that the feature vectors of each modality are V specch , V face , V body , V bio , and the weights of each modality are λ1, λ2, λ3, λ4. Then the fused emotion data vector V cmotion is:

[0215] V cmotion = λ1V specch + λ2V face + λ3V body + λ4V bio

[0216] where λ1, λ2, λ3, λ4 are obtained through training and learning and represent the relative importance of each modality;

[0217] Another common fusion method is to concatenate the feature vectors of each modality:

[0218] V cmotion = [V specch , V facc , V body , V bio

[0219] ​In addition to weighted summation and concatenation, a deep neural network or a long short-term memory network model is also used for fusion, and the emotional data is fused by learning the relationships between different modalities. Suppose there is a deep neural network model f fusion , then the fusion process is as follows:

[0220] V cmotion = f fusion (V specch , V face , V body , V bio )

[0221] where f fusion is a deep neural network obtained through training and is used to learn the fusion rules from multiple modalities;

[0222] The fused emotional data vector V cmotion is used as input and passed to the sentiment analysis model, the emotion recognition model, or for other tasks.

[0223] Noise cancellation and signal enhancement can effectively improve the quality of data, feature extraction can extract more useful information, and modality fusion can integrate the advantages of different modalities, ultimately improving the accuracy of sentiment analysis; by fusing data from speech, facial expressions, body language, and physiological signals, the system can comprehensively understand the user's emotional state from multiple perspectives, making emotion recognition more accurate and delicate; through the deep learning model, the system can automatically adapt to the emotional expressions of different users and different situations, improving the robustness and flexibility of the system; multi-modal fusion enables emotion calculation not to be limited to the changes in a single signal, but to dynamically adapt to the emotional changes of users and provide more personalized emotional feedback.

[0224] Through the deep neural network and the sentiment classification algorithm, the specific emotional state of the user is identified and classified into specific emotion labels. At the same time, the method for identifying the mixed state of complex emotions is as follows:

[0225] The process of data preprocessing and feature extraction is similar to that mentioned above. Feature vectors are extracted through multi-modal information. Suppose a fused feature vector V fusion is obtained, and this vector contains all the information related to emotions, in the form of:

[0226] V fusion = |V specch , V facc , V body , V bio

[0227] where V specch , V face , V body , V​bio Feature vectors obtained through their respective feature extraction methods are fused to obtain V fusion as the input;

[0228] A deep neural network is used to train and classify the fused feature vector V fusion For example, there is a multi-layer perceptron model whose structure includes an input layer, a hidden layer, and an output layer. Assuming V fusion is the input, each layer of the DNN model is calculated through weighted sum, bias, and activation function. The input feature vector V fusion :

[0229] h1 = W1V fusion + b1

[0230] The activation function of the hidden layer uses ReLU:

[0231] h2 = ReLU(W2h1 + b2)

[0232] Finally, the probability distribution of the sentiment label is output, and the softmax function is used to convert the output into the probability of each sentiment category:

[0233]

[0234] where y = c represents the sentiment label c, and P(y = c|V fusion ) is the probability belonging to the label c;

[0235] For sentiment classification, according to the output probability distribution, the sentiment label with the highest probability is selected as the prediction result. Assuming there are C sentiment categories C = {c1, c2,..., c C}, the prediction result is the label with the highest probability:

[0236]

[0237] A complex sentiment state is a mixture of multiple basic sentiments. When identifying mixed sentiments, the following method is used:

[0238] In multi-label classification, the sentiment network output is not only a single label but the probability value of each sentiment category;

[0239] For the probability of each sentiment category c i , the Sigmoid activation function is used:

[0240] P(y = c i |V fusion ) = σ(W i h2 + b i )

[0241] where, is the Sigmoid function, and the output value represents the existence probability of this emotion category.

[0242] Multimodal data (such as speech, facial expressions, body language, and physiological signals) contains rich emotion information. By fusing this information, a more comprehensive emotion representation can be obtained, thereby improving the accuracy and robustness of emotion analysis; Deep neural networks (DNNs) can automatically learn complex feature representations from large amounts of data without the need for manual feature design. This enables the model to capture deeper emotion information, especially suitable for processing non-linear and high-dimensional data; Through multi-label classification methods, the model can not only identify single emotion states but also identify mixed states of complex emotions. This is crucial for emotion analysis in practical applications (such as user emotion feedback) because human emotions are often diverse and mixed; This solution can expand emotion categories as needed, adjust the model structure, or adopt other types of deep learning architectures (such as LSTM, CNN) for more complex emotion modeling. In addition, the Sigmoid activation function enables multi-label classification to effectively handle the coexistence of multiple emotions.

[0243] The method of establishing a personalized emotion model based on the user's historical emotion data, combining machine learning algorithms, reflecting the law of the user's emotion changes, and learning and adapting in the long term is as follows:

[0244] By preprocessing the user's historical emotion data to construct appropriate input features. Suppose there is a time series dataset, where each time step contains the user's emotion data and auxiliary features, and these data are formed into a feature vector of the time series:

[0245] X t = [f1(t), f2(t),..., f N (t)]

[0246] where, X t represents the feature vector at time step t, f i (t) is the value of the i-th feature at time step t, and N is the total number of features;

[0247] LSTM consists of an input gate, a forget gate, and an output gate. Its core is to control the transmission and update of information. At each time step t, the output h t and the hidden state c t of LSTM are updated as follows:

[0248] Forget gate f t : f t = σ(W f ·[h t1 , X t + b f );

[0249] Input gate i t : i t = σ(W i ·[h t1 , X t +b i );

[0250] Candidate memory unit

[0251] Update memory unit c t :

[0252] Output gate o t : o t = σ(W o ·[h t1 , X t +b o );

[0253] Hidden state h t : h t = o t ·tanh(c t );

[0254] By training the LSTM model, predict the user's emotional state y at time step t t :

[0255] u i = softmax(W y ·h t +b y )

[0256] where, W y is the weight matrix of the output layer, b y is the bias term, and the softmax function outputs a probability distribution over different emotion labels;

[0257] Adopt reinforcement learning to adjust the emotion model so that the model can be updated in the long term according to real-time feedback;

[0258] State S t : The current emotion prediction and historical emotion information of the model;

[0259] Action A t : The adjustment or update strategy of the emotion model;

[0260] Reward R t : The reward based on user feedback;

[0261] At each step, the model selects an action based on the current state and receives a reward, and updates the Q-value using Q-learning:

[0262]

[0263] where α is the learning rate, γ is the discount factor, and R t is the current reward;

[0264] Between time steps t and t+1, the weights w and biases b are adjusted according to the actual feedback:

[0265]

[0266] where η is the learning rate and L is the loss function, representing the difference between the model prediction and the actual sentiment label.

[0267] By using the user's historical sentiment data and the LSTM network, the model can capture the long-term dependencies of the user's sentiment changes and establish a personalized sentiment prediction model; through reinforcement learning, the model can continuously learn and adapt to the user's sentiment changes in actual applications, accumulating more feedback over time, enabling the model to continuously optimize and adapt to the user's sentiment fluctuations in the long term; through the user's real-time feedback, the model can adjust the parameters according to the actual sentiment label and prediction error, thereby enhancing the accuracy of sentiment prediction; different from traditional static sentiment models, this solution enables the model to not only be frozen after one-time training, but to continuously update and improve according to real-time user feedback through reinforcement learning and an adaptive adjustment mechanism; this solution has broad potential in a variety of applications, including sentiment analysis, personalized recommendation, intelligent customer service and other systems, and can effectively improve the interaction quality and user experience with users.

[0268] The method for generating a response that conforms to the user's sentiment state based on the sentiment analysis result and in combination with the user's personalized sentiment model is as follows:

[0269] Identify the user's current sentiment state through sentiment analysis, the user's current sentiment state St, and the intensity of the sentiment Intensity t , and the sentiment analysis output formula is:

[0270] S t = Happy, Intensity t = 0.8

[0271] In combination with the user's personalized sentiment model, combine the user's sentiment historical data with the current sentiment state to predict or adjust the user's sentiment changes. Assume the user's sentiment historical data is {S1, S2,..., S t}, and output a personalized sentiment adjustment factor α through the model t, this factor represents the personalized adjustment value of the current emotional state, reflecting the long-term change trend of the user's emotion:

[0272] α t = f(S1, S2,..., S t )

[0273] where f is a function based on the user's historical emotion data;

[0274] Based on the output of the sentiment analysis and the personalized emotion model, generate a response that conforms to the user's emotional state according to the current emotional state and the adjustment factor;

[0275] Assume there is a set of predefined response libraries, and each response is related to a specific emotional state:

[0276] The response R to happiness happy ;

[0277] The response R to sadness sorrow ;

[0278] The response R to anger angry ;

[0279] According to the current emotional state S t and the emotional intensity Intensity t to select the most appropriate response, and the selection formula of the response is as follows:

[0280] R t = SelectResponse(S l , Intensity t )

[0281] where SelectResponse(S t , Intensity t ) is a function to select the most matching response according to the user's emotional state and intensity;

[0282] In addition to selecting the most matching response, fine-tune through the personalized emotion model, and based on the historical emotion data and the current personalized factor α t , adjust the tone or diction of the response to make the response more in line with the user's personalized needs, and the formula for this adjustment is expressed as:

[0283] R adjusted = R t + a t ·ΔR

[0284] where R t is the basic response;

[0285] α tis a personalized emotion adjustment factor;

[0286] ΔR is the fine-tuning based on the current emotional state;

[0287] In summary, the method for generating a response that conforms to the user's emotional state is summarized in the following steps:

[0288] Emotion analysis: Analyze the user's current emotional state S t and emotional intensity Intensity t ;

[0289] Personalized emotion adjustment factor: Obtain the adjustment factor α through the personalized emotion model t ;

[0290] Response selection: Select the most appropriate response R from the predefined response library t , and adjust it according to the emotional state S t and emotional intensity Intensity t ;

[0291] Personalized adjustment: Fine-tune the response according to the personalized factor αt to generate the final response.

[0292] Through emotion analysis and the personalized emotion model, the system can understand the user's emotional state and emotional intensity, and select the most appropriate response based on this information; the personalized adjustment factor enables the model to adjust the tone or wording of the response according to the user's emotional history and current emotional state; through the personalized emotion model, the system can learn and adapt to the user's emotional changes over time; the predefined response library provides the system with diverse response options. Combining emotion analysis with personalized adjustment, the system can select the most suitable response for the current emotional state and further fine-tune it to make the response more natural; through precise emotion matching and personalized adjustment, users will feel that the system understands and respects their emotions, greatly enhancing the user experience, especially in application scenarios such as intelligent customer service and virtual assistants; since the model is optimized and adjusted based on the user's historical emotional data, it can continuously learn according to user feedback to ensure that the quality of the response improves over time.

[0293] Through continuous user feedback and interaction, the system gradually learns and optimizes its emotion understanding and response capabilities, continuously updates the user's emotion profile, and shares the emotion calculation tasks through the edge computing platform or cloud computing resources. The method to ensure that the system can continuously perform real-time responses under complex emotion calculation tasks is as follows:

[0294] After each user feedback, the system updates the user's emotion profile according to the new feedback information. Assuming that the user's emotional state can be represented as a vector S t , and the feedback information is Ft , the system adjusts the emotion understanding model according to the feedback information, and the updating process can be expressed by the following formula:

[0295] S t+1 =S t +ΔS t

[0296] Among them, S t+1 is the updated emotional state vector, ΔS t According to user feedback F t The adjustment of the emotional state, this adjustment ΔS t Updated by the system through an online learning algorithm;

[0297] Update the emotional state vector using Q-learning or reinforcement learning:

[0298]

[0299] Among them, Q(S t , A t ) is an emotional response behavior A in the current emotional state t The quality value of

[0300] α is the learning rate;

[0301] R t Rewards based on user feedback;

[0302] γ is the discount factor,

[0303] max A′ Q(S t+1 , A′) is the next emotional state S t+1 The maximum Q value obtained under

[0304] As the ability to understand emotions is optimized, the user's emotional profile is constantly updated, recording historical emotional data, user feedback, and emotional intensity information. The emotional profile is represented as a multidimensional emotional vector C t , each dimension represents the intensity of a certain emotion category;

[0305] After each user feedback, the sentiment profile will be updated in the following ways:

[0306] C t+1 =C t +ΔC t

[0307] Where, ΔC t Indicates the changes in sentiment profile caused by user feedback;

[0308] To ensure that the system maintains real-time response when processing complex emotion computing tasks, the emotion computing tasks are distributed to the edge computing platform and cloud computing resources. The emotion computing tasks are divided into low-latency tasks and high-computation task types;

[0309] The decision basis for task allocation is dynamically adjusted through a load balancing algorithm. Assuming that the system allocates tasks to edge devices or cloud platforms according to the complexity and latency requirements of the tasks, the task allocation strategy is expressed as:

[0310] Task t =AssignTask(C t ,Q t ,Resources)

[0311] where C t is the current emotion profile;

[0312] Q t is the computational complexity of the task;

[0313] Resources are the available computing resources;

[0314] AssignTask is a task allocation function responsible for selecting a suitable platform for processing according to the computational complexity and resource conditions;

[0315] To ensure the system's real-time response under complex emotion computing tasks, the collaborative work of edge computing and cloud computing is utilized, and the computing process is described by the following formula:

[0316] Response i =f(Task i ,Time t ,Resources)

[0317] where Response t is the emotion response generated by the system at time t;

[0318] Task t is the current emotion computing task;

[0319] Time t is the time requirement of the task;

[0320] Resources are the computing resources scheduled by the system.

[0321] Through online learning and reinforcement learning, the system can continuously update the emotion understanding model based on user feedback, and then optimize the emotion response strategy; the user's emotion profile is updated after each feedback, which can record information such as the user's historical emotion data and emotion intensity; allocating emotion computing tasks to the edge computing and cloud computing platforms can effectively solve the problems of real-time performance and computing complexity; by dynamically adjusting task allocation through the load balancing algorithm, the system can flexibly select the appropriate platform according to the computing complexity and latency requirements of the tasks, ensuring real-time response under various emotion computing tasks; through sharing computing tasks, the collaborative work of edge computing and cloud computing can not only improve the efficiency of the system in processing tasks, but also effectively reduce the burden on a single platform, avoiding latency problems caused by resource bottlenecks; through continuous learning and optimization, the system can more accurately understand user emotions and make corresponding responses in a timely manner, thereby enhancing the user's experience satisfaction.

[0322] This embodiment also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the robot interaction system and interaction method based on machine learning and emotion computing as described above.

[0323] This embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the robot interaction system and interaction method based on machine learning and emotion computing as described above.

[0324] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0325] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the system is divided into different functional units or modules to complete all or part of the functions described above.

[0326] The above embodiments of the present invention do not limit the protection scope of the present invention. The implementation manners of the present invention are not limited thereto. All kinds of modifications, substitutions, or changes made to the above structure of the present invention according to the above content of the present invention, in accordance with the common general knowledge and customary means in the art, without departing from the above basic technical idea of the present invention, shall fall within the protection scope of the present invention.

Claims

1. A robot interaction system based on machine learning and sentiment computing, characterized in that, It includes: Emotion Perception Module: It is used to extract emotion information from the user's multimodal data and fuse different signals using deep learning algorithms. Emotion Understanding Module: Based on deep neural networks and emotion analysis algorithms, this module deeply analyzes the multimodal emotion data obtained from the perception module, identifies the user's emotional state, and processes complex emotion mixtures. Personalized Emotion Modeling Module: According to the user's historical emotion data and combined with reinforcement learning methods, it gradually optimizes the robot's emotion understanding and response capabilities for different users and conducts personalized emotional interactions. Emotion Generation and Response Module: Based on the analysis results of the emotion understanding module, it generates responses that match the user's emotional state through a deep emotion generation model. Edge Computing and Resource Optimization Module: By combining edge computing technologies, it distributes difficult-to-analyze emotion computing tasks to edge devices or cloud platforms.

2. The robot interaction system based on machine learning and sentiment computing according to claim 1, characterized in that, The emotion perception module includes sub-modules for speech recognition, facial expression recognition, body language analysis, and physiological signal acquisition; the emotion understanding module conducts long-term tracking of emotions and gradually builds the user's emotion model; the emotion generation and response module's emotion generation process is based on emotion diversity and situational adaptability, enabling the robot to adjust the intonation, tone, and content of its response according to the user's emotional fluctuations.

3. A robot interaction method based on machine learning and sentiment computing, characterized in that, It includes the following steps: The robot uses multimodal perception devices to collect the user's speech, facial expressions, body language, and physiological signal data in real time to form emotion data input. Preprocess the collected emotion data, including noise elimination, signal enhancement, feature extraction, and fuse emotion data from different sources. Through deep neural networks and emotion classification algorithms, identify the user's specific emotional state, classify it into specific emotion labels, and at the same time identify the mixed state of complex emotions. According to the user's historical emotion data, combine machine learning algorithms to establish a personalized emotion model, reflect the laws of the user's emotional changes, and conduct long-term learning and adaptation. Based on the emotion analysis results and combined with the user's personalized emotion model, generate responses that match the user's emotional state. Through continuous user feedback and interactions, the system gradually learns and optimizes its emotion understanding and response capabilities, continuously updates the user's emotion profile, and distributes emotion computing tasks through edge computing platforms or cloud computing resources to ensure that the system can continuously respond in real time under complex emotion computing tasks.

4. The robot interaction method based on machine learning and sentiment computing according to claim 3, characterized in that The method by which the robot uses multimodal perception devices to collect the user's speech, facial expressions, body language, and physiological signal data in real time to form emotion data input is: Suppose a speech signal passes through a speech recognition and sentiment analysis model to obtain a sentiment feature vector V spccch , which contains speech features such as sentiment tone, intonation, volume, and speech rate. The formula is as follows: V spccch = f spccch (r spccch ) Among them, x spccch is the original voice signal, f spccch is the mapping function for voice feature extraction and sentiment analysis, and the generated V spccch is a sentiment-related feature vector; Facial expressions are converted into expression feature vectors V through facial recognition technology facc , which contains the type and intensity of the expression, and the formula is: V facc = f facc (x facc ) where x facc is facial image data, and f facc is a mapping function for facial expression feature extraction; Feature vector V extracted from body language through motion capture technology body , including the user's posture, movement amplitude, and speed, with the formula: V body = f body (x body ) where x body is the input data of the body movement, and f body is the function for action or pose recognition; Physiological signals are collected by a physiological signal monitoring device and transformed into a feature vector V through signal processing bio , which contains the user's physiological state data, such as emotional fluctuations, tension or relaxation indicators, and the formula is: V bio = f bio (x bio ) where x bio is physiological signal data, and f bio is a mapping function for physiological signal analysis; Fuse the feature vectors V spccch of the above-mentioned multimodal data, V face of the above-mentioned multimodal data, V lxxly of the above-mentioned multimodal data, and V bio of the above-mentioned multimodal data to obtain a comprehensive sentiment data vector V cmotion by using weighted average, concatenation, or a deep learning-based fusion method. The formula is as follows: V cmotion = λV spccch + λ2V facc + λ3V body + λ4V bio Among them, λ1, λ2, λ3, λ4 are the weight coefficients of each modality.

5. The robot interaction method based on machine learning and sentiment computing according to claim 4, characterized in that, The method for preprocessing the collected emotion data, including noise elimination, signal enhancement, feature extraction, and fusing emotion data from different sources is: Noise cancellation is processed using filters or signal denoising methods. For the speech signal x spccch , assume y spccch is the denoised speech signal: Among them, is a noise cancellation operation; Signal enhancement improves the effect of sentiment analysis by enhancing the clarity of the signal. For the speech signal y spccch , the enhanced signal z spccch is obtained by processing through the enhancement algorithm ε: z spccch = ε(y spccch ) Among them, ε is the signal enhancement algorithm. For data of speech, facial expressions, body language, and physiological signal modalities, different feature extraction methods are adopted. For the speech signal x spccch Extract features such as Mel-frequency cepstral coefficients, fundamental frequency, speech rate features, and form a feature vector V spccch : V spccch = f spccch (z spccch ) where f spccch is a voice feature extraction function; Facial expressions extract features through facial feature point detection and an emotion classification model to obtain a facial expression feature vector V face : V facc = f facc (x facc ) The features of body language are used to extract a feature vector V from posture recognition and motion analysis body : V facc = f facc (x facc ) The physiological signal extracts the corresponding physiological feature vector V through a signal analysis method bio : V bio = f bio (x bio ) After obtaining the feature vectors of each modality, the sentiment data from different sources are then fused, using weighted summation, concatenation, or more complex deep learning methods for fusion. Assume that the feature vectors of each modality are V spccch , V facc , V body , V bio , and the weights of each modality are λ1, λ2, λ3, λ4. Then the fused sentiment data vector V cmotion is: V cmotion = λ1V spccch + λ2V facc + λ3V body + λ4V bio Among them, λ1, λ2, λ3, λ4 are obtained through training and learning and represent the relative importance of each modality. Another common fusion method is to concatenate the feature vectors of each modality: V cmotion = [V spccch , V facc , V body , V bio ​ In addition to weighted summation and concatenation, a deep neural network or a long short-term memory network model is also used for fusion, and the emotional data is fused by learning the relationships between different modalities. Suppose there is a deep neural network model f fusion , then the fusion process is as follows: V cmotion = f fusion (V spccch , V facc , V body , V bio ) Among them, f fusion is a deep neural network obtained through training and is used to learn the fusion rule from multiple modalities; The fused emotional data vector V cmotion Is used as input and passed to an emotion analysis model, an emotion recognition model, or for other tasks.

6. The robot interaction method based on machine learning and sentiment computing according to claim 5, characterized in that, The method for identifying the specific emotional state of a user through a deep neural network and an emotion classification algorithm, classifying it into specific emotion labels, and simultaneously identifying the mixed state of complex emotions is as follows: The process of data preprocessing and feature extraction is similar to that mentioned above. Feature vectors are extracted through multimodal information. Suppose a fused feature vector V is obtained by extracting features from different modalities. fusion , which contains all the emotion-related information in the form of: V fusion = [V spccch , V facc , V body , V bio ​ Among them, V spccch , V facc , V lxxly , V bio are feature vectors obtained through their respective feature extraction methods, and after fusion, V fusion is used as the input; Use a deep neural network to train and classify the fused feature vector V fusion For training and classification, assume there is a multi-layer perceptron model with an input layer, a hidden layer, and an output layer. Assume V fusion is the input. Each layer of the DNN model is calculated through weighted sum, bias, and activation function. The input feature vector V fusion : h1 = W1V fusion + b1 The activation function of the hidden layer uses ReLU: h2 = ReLU(W2h1 + b2) Finally, the probability distribution of the emotion labels is output, and the softmax function is used to convert the output into the probability of each emotion category: Among them, y = c represents the emotion label c, and P(y = c|V fusion ) belongs to the probability of the label being c; Emotional classification selects the emotional label with the highest probability as the prediction result according to the output probability distribution. Suppose there are C emotional categories C = {c1, c2,..., c C}, and the prediction result is the label with the highest probability: The complex emotional state is a mixture of multiple basic emotions. When identifying the mixed emotions, the following method is used: In multi-label classification, the output of the emotion network is not only a single label, but the probability value for each emotion category; For each sentiment category c i The probability of is calculated using the sigmoid activation function: P(y = c i | V fusion ) = σ(W i h2 + b i ) Among them, is the Sigmoid function, and the output value represents the existence probability of this sentiment category.

7. The robot interaction method based on machine learning and sentiment computing according to claim 6, wherein The method for establishing a personalized emotion model based on the user's historical emotion data, combined with machine learning algorithms, to reflect the law of the user's emotional changes and to learn and adapt in the long term is as follows: By preprocessing the user's historical emotion data to construct appropriate input features, assuming there is a time series data set, where each time step contains the user's emotion data and auxiliary features, and these data are composed into a feature vector of the time series: X t = [f1(t), f2(t),..., f N (t)] Among them, X t represents the feature vector at time step t, and f i (t) is the value of the i-th feature at time step t, and N is the total number of features; The LSTM consists of an input gate, a forget gate, and an output gate. Its core is to control the transmission and update of information. At each time step t, the output h of the LSTM t and the hidden state c t are updated as follows: Forgotten gate f t : f t = σ(W f ·[h t1 , X t +b f ); Input gate i t : i t = σ(W i ·[h t1 , X t + b i ); Candidate memory unit Update memory cell c t : Output gate o t : o t = σ(W o · [h t1 , X t + b o ); Hidden state h t : h t = o t ·tanh(c t ); Predict the user's emotional state y at time step t by training an LSTM model t : y t = softmax(W y ·h t + b y ) Among them, W y is the weight matrix of the output layer, and b y is the bias term. The softmax function outputs a probability distribution over different sentiment labels; Reinforcement learning is used to adjust the emotion model so that the model can be updated in the long term according to real-time feedback; State S t : The current sentiment prediction and historical sentiment information of the model; Action A t : Adjustment or update strategy for the emotion model; Reward R t : Reward based on user feedback; At each step, the model selects an action according to the current state and obtains a reward, and the Q value is updated using Q-learning: where α is the learning rate, γ is the discount factor, and R t is the current reward; Between time steps t and t + 1, the weights W and biases b are adjusted according to the actual feedback: Among them, η is the learning rate, and L is the loss function, which represents the difference between the model prediction and the actual emotion label.

8. The robot interaction method based on machine learning and sentiment computing according to claim 7, wherein The method for generating a response that conforms to the user's emotional state based on the emotion analysis result and combined with the user's personalized emotion model is as follows: Identify the user's current emotional state through sentiment analysis, where the user's current emotional state is S t , the intensity of the emotion Intensity t , and the sentiment analysis output formula is: S t = Happy, Intensity t = 0.8 Combined with the user's personalized emotion model, the user's emotional historical data is combined with the current emotional state to predict or adjust the user's emotional changes. Assume that the user's emotional historical data is {S1, S2,..., S t}, and a personalized emotion adjustment factor α t is output through the model. This factor represents the personalized adjustment value of the current emotional state and reflects the long-term change trend of the user's emotion: α t = f(S1, S2,..., S t ) Among them, f is a function based on the user's historical emotion data; Based on the output of the emotion analysis and the personalized emotion model, a response that conforms to the user's emotional state is generated according to the current emotional state and the adjustment factor; Assume there is a set of predefined response libraries, and each response is related to a specific emotional state: Response R to pleasure happy ; Response to sadness R sorrow ; Response R to Anger angry ; According to the current emotional state S t and the emotional intensity Intensity t to select the most appropriate response. The response selection formula is as follows: R t = SelectResponse(S t , Intensity t ) Among them, SelectResponse(S t , Intensity t ) is a function that selects the most suitable response according to the user's emotional state and intensity; In addition to selecting the most matching response, fine-tuning is performed through a personalized sentiment model, based on historical sentiment data and the current personalized factor α t , to adjust the tone or wording of the response to make it more in line with the personalized needs of the user. The formula for this adjustment is expressed as: R adjustcd = R t + α t ·ΔR wherein, R t is the basic response; α t is a personalized emotion adjustment factor; ΔR is the fine-tuning based on the current emotional state; To sum up, the method for generating a response that conforms to the user's emotional state is summarized into the following steps: Sentiment Analysis: Analyze the user's current sentiment state S t and sentiment intensity Intensity t ; Personalized emotion adjustment factor: Obtain the adjustment factor α through the personalized emotion model t ; Response Selection: Select the most appropriate response R from a predefined response library t and adjust it according to the emotional state S t and the emotional intensity Intensity t ; Personalized adjustment: According to the personalized factor α t Fine-tune the response to generate the final response.

9. The robot interaction method based on machine learning and sentiment computing according to claim 8, characterized in that Through continuous user feedback and interaction, the system gradually learns and optimizes the emotion understanding and response capabilities, continuously updates the user's emotion profile, and distributes the emotion calculation tasks through the edge computing platform or cloud computing resources to ensure that the system can continuously perform real-time responses under complex emotion calculation tasks. The method is as follows: After each user feedback, the system updates the user's sentiment profile based on the new feedback information. Assume that the user's sentiment state can be represented as a vector S t , and the feedback information is F t . The system adjusts the sentiment understanding model according to the feedback information, and the update process can be expressed by the following formula: S t+1 = S t + ΔS t Among them, S t+1 is the updated emotional state vector, and ΔS t represents the adjustment of the emotional state made according to the user feedback F t This adjustment ΔS t is updated by the system through an online learning algorithm; The emotion state vector is updated using the method of Q-learning or reinforcement learning: Among them, Q(S t , A t ) is the quality value of an emotional response behavior A t in the current emotional state; α is the learning rate; R t is the reward obtained based on user feedback; γ is the discount factor, max A′ Q(S t|1 , A′) is the maximum Q value obtained in the next emotional state S t|1 ; As the emotional understanding ability is optimized, the user's emotional profile is continuously updated, recording historical emotional data, user feedback, and emotional intensity information. The emotional profile is represented as a multi-dimensional emotional vector C t , where each dimension represents the intensity of a certain emotional category; After each user feedback, the emotion profile will be updated in the following way: C t+1 = C t + ΔC t where ΔC t represents the change in the sentiment profile caused by user feedback; To ensure that the system maintains real-time response when processing complex emotion calculation tasks, the emotion calculation tasks are distributed to the edge computing platform and cloud computing resources. The emotion calculation tasks are divided into low-latency tasks and high-computation task types; The decision basis for task allocation is dynamically adjusted through a load balancing algorithm. Assume that the system allocates tasks to edge devices or cloud platforms according to the complexity and latency requirements of the tasks. The task allocation strategy is expressed as: Task t = AssignTask(C t , Q t , Resources) Among them, C t is the current sentiment file; Q t is the computational complexity of the task; Resources are the available computing resources; AssignTask is a task assignment function responsible for selecting a suitable platform for processing according to the computational complexity and resource conditions; To ensure the system's real-time response under complex emotion computing tasks, the collaborative work of edge computing and cloud computing is utilized, and the computing process is described by the following formula: Response t = f(Task t , Time t , Resources) Among them, Response t is the emotional response generated by the system at time t; Task t is the current emotion computing task; Time t It is the time requirement for the task; Resources are the computing resources that the system can schedule.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the robot interaction method based on machine learning and emotion computing as described in any one of claims 3-9; A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the robot interaction method based on machine learning and emotion computing as described in any one of claims 3-.

Images