Robot emotion accompanying system based on deep learning

Abstract

The invention relates to the technical field of robot emotion accompanying, and discloses a robot emotion accompanying system based on deep learning. The system comprises an emotional state recognition module, an emotional demand analysis module and an emotional interaction strategy generation module. The emotional state recognition module extracts facial micro-expression features, voice spectrum features and body movement track features based on the user interaction data flow and performs multi-modal feature fusion to generate a user emotional state vector; the emotion demand analysis module retrieves an emotion demand knowledge graph based on the vector, matches an emotion demand tag, calculates an emotion demand compactness index, and generates a demand analysis result; and the emotion interaction strategy generation module calls the basic strategy template, adjusts the strategy parameter weight according to the demand closeness index, and generates a personalized emotion interaction strategy instruction set. The system can comprehensively and accurately identify the user emotion, accurately analyze the emotion demand, generate a personalized interaction strategy, and improve the emotion accompanying quality and the user experience.

Description

Technical Field

[0001] This invention relates to the field of robotic emotional companionship technology, specifically a deep learning-based robotic emotional companionship system. Background Technology

[0002] Traditional robotic emotional companionship systems often rely on single-modal information collection and analysis, such as judging a user's emotional state solely through speech recognition or simply categorizing emotions based on facial expressions. This single-modal approach has significant limitations. In real-world interaction scenarios, users' emotional expressions are often multi-dimensional; facial micro-expressions, tone of voice, and body language all convey rich emotional information. Relying solely on single-modal information makes it difficult to comprehensively and accurately capture a user's true emotional state, easily leading to biased emotion recognition results and consequently affecting the quality of subsequent emotional companionship services.

[0003] Existing systems lack effective knowledge support and quantitative assessment methods in the emotional needs analysis phase. Most systems can only make a preliminary judgment on a user's emotional state, failing to deeply analyze the specific emotional needs behind those states or to rationally prioritize different emotional needs based on their urgency. This forces systems to use generic templates when generating emotional interaction strategies, unable to flexibly adjust them according to the user's individual emotional needs and the intensity of those needs. For example, when a user simultaneously exhibits anxiety and loneliness, existing systems struggle to accurately determine which emotional need is more urgent, thus failing to prioritize targeted support services. This results in a lack of personalization and specificity in the interaction process, making it difficult to meet the diverse emotional support needs of users.

[0004] In the application of deep learning technology, existing systems have failed to fully realize the effective fusion of multimodal features and the dynamic adjustment of policy parameters. Multimodal feature fusion is a key step in improving the accuracy of emotion recognition, but existing fusion methods mostly remain at the level of simple feature splicing, failing to fully explore the intrinsic relationships between different modal features. As a result, the fused features cannot comprehensively reflect the user's emotional information. In the emotional interaction strategy generation stage, the policy parameters of existing systems are mostly fixed settings, unable to be adjusted in real time according to changes in the user's emotional needs. This makes the generated interaction strategies inflexible and difficult to adapt to the emotional companionship needs of different users in different scenarios, further limiting the practical value and user experience of robotic emotional companionship systems. Summary of the Invention

[0005] The purpose of this invention is to provide a deep learning-based robotic emotional companionship system to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides a deep learning-based robotic emotional companionship system, the system comprising: The emotion state recognition module extracts facial micro-expression features, speech spectrum features, and body movement trajectory features based on user interaction data streams. It then performs multimodal feature fusion on the facial micro-expression features, speech spectrum features, and body movement trajectory features to generate a user emotion state vector. The emotional needs analysis module retrieves emotional state nodes in the emotional needs knowledge graph based on the user's emotional state vector, matches associated emotional needs tags, calculates the emotional needs density index based on the number of matched emotional needs tags and the strength of association, and generates user emotional needs analysis results. The emotional interaction strategy generation module, based on the user's emotional needs analysis results, calls the basic strategy template in the emotional interaction strategy library, adjusts the strategy parameter weights according to the emotional needs density index, and generates a personalized emotional interaction strategy instruction set.

[0007] Preferably, the process of obtaining the user's emotional state vector is as follows: Based on the video frame sequence in the user interaction data stream, the activation intensity of facial muscle motion units is extracted by a three-dimensional convolutional neural network to generate facial micro-expression feature vectors. Based on the audio signal in the user interaction data stream, the speech spectrum feature matrix is ​​extracted by Mel frequency cepstral coefficient transform. Based on depth images in user interaction data streams, limb movement trajectory feature sequences are extracted using a skeletal joint tracking algorithm. The facial micro-expression feature vector, speech spectrum feature matrix, and limb movement trajectory feature sequence are input into a multimodal feature fusion network to generate a user emotional state vector.

[0008] Preferably, the process for obtaining the emotional need intensity index is as follows: Based on the user's emotional state vector, node similarity is calculated in the emotional needs knowledge graph, and related emotional state nodes with similarity exceeding the threshold are filtered out. Extract the set of emotional need tags corresponding to the associated emotional state nodes, and count the frequency and duration of each emotional need tag in historical interactions; The emotional need tag weights are calculated based on the frequency and duration of occurrence, and combined with the similarity scores of associated emotional state nodes, an emotional need density index is generated.

[0009] Preferably, the generation process of the personalized emotional interaction strategy instruction set is as follows: Based on the results of the emotional needs analysis, the basic strategy template with the highest matching degree is retrieved from the emotional interaction strategy library; Extract the set of adjustable parameters from the basic strategy template, and calculate the adjustment range of each parameter based on the emotional need density index; The adjusted parameter values ​​are injected into the strategy template execution engine to generate a personalized emotional interaction strategy instruction set.

[0010] Preferably, the system further includes: The long-term emotional pattern modeling module constructs an emotional state transition graph over time based on multiple user emotional state vectors and calculates the emotional state transition probability matrix. Based on the emotional state transition probability matrix, periodic emotional fluctuation patterns are identified, and a long-term emotional profile of the user is generated.

[0011] Preferably, the process of obtaining the user's long-term emotional profile is as follows: The multiple continuously collected user emotional state vectors are subjected to time series alignment processing to construct an emotional state transition edge set; The frequency of occurrence of each transition path in the set of emotional state transition edges is counted, and the normalized transition probability is calculated. Based on the normalized transition probabilities, an emotional state transition probability matrix is ​​constructed, and periodic paths with transition probabilities exceeding a threshold are extracted to generate a long-term emotional feature profile of the user.

[0012] Preferably, the system further includes: The multi-robot collaboration module establishes a cross-robot emotional state consistency verification mechanism based on the user emotional state vectors collected by multiple robots. When the difference in emotional state across robots exceeds the fault tolerance threshold, a multi-source data re-fusion process is triggered to generate a corrected user emotional state vector.

[0013] Preferably, the process of obtaining the corrected user emotional state vector is as follows: Collect the user emotional state vectors reported by multiple robots, and calculate the cosine similarity matrix between the vectors; Identify abnormal vectors in the cosine similarity matrix whose differences exceed the fault tolerance threshold, and re-collect multimodal interaction data for the corresponding time period; The newly collected multimodal interaction data is weighted and fused to generate a corrected user sentiment state vector.

[0014] Preferably, the system further includes: The emotional interaction effect evaluation module calculates the strategy execution effect score based on the trend of user emotional state vector change after implementing the personalized emotional interaction strategy instruction set. The performance score of the strategy is fed back to the emotional interaction strategy library to update the utility weight of the corresponding strategy template.

[0015] Preferably, the calculation process for the strategy execution effect score is as follows: Record the difference in the user's emotional state vector before and after implementing the personalized emotional interaction strategy instruction set; Based on the direction and magnitude of the difference, and combined with the typical response patterns in the user's long-term emotional profile, a strategy execution effectiveness score is calculated. The strategy execution performance score is associated with and stored with the strategy template identifier for subsequent strategy optimization decisions.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This deep learning-based robotic emotional companionship system optimizes the entire process from emotional information acquisition to interaction strategy generation by incorporating emotional state recognition, emotional needs analysis, and emotional interaction strategy generation modules. This provides users with more accurate and personalized emotional companionship services. In terms of emotional state recognition, the system overcomes the limitations of traditional single-modal recognition. It simultaneously extracts facial micro-expression features, speech spectrum features, and body movement trajectory features from the user interaction data stream, and performs multi-modal feature fusion to generate a user emotional state vector. This multi-modal feature fusion approach fully integrates emotional information from different dimensions, comprehensively capturing the emotional details conveyed by the user during interaction. It avoids the emotional recognition bias caused by incomplete single-modal information acquisition, making the generated user emotional state vector more accurately reflect the user's true emotional state, laying a reliable foundation for subsequent emotional needs analysis and interaction strategy generation.

[0017] In the emotional needs analysis phase, the system, based on the generated user emotional state vector, retrieves emotional state nodes from the emotional needs knowledge graph and matches them with associated emotional needs tags. Simultaneously, it calculates an emotional needs density index based on the number of matched emotional needs tags and the strength of the association. The introduction of the emotional needs knowledge graph provides rich knowledge support for emotional needs analysis, effectively uncovering the specific emotional needs corresponding to the user's emotional state. The calculation of the emotional needs density index enables a quantitative assessment of the urgency of different emotional needs. In this way, the system can not only clearly identify the user's specific emotional needs but also rationally prioritize different emotional needs, avoiding the problem of existing systems being unable to accurately determine the urgency of emotional needs. This makes the emotional needs analysis results more practical and instructive, providing a precise basis for the generation of subsequent personalized interaction strategies.

[0018] In terms of generating emotional interaction strategies, the system calls upon basic strategy templates from the emotional interaction strategy library based on the results of emotional needs analysis. It then adjusts the weights of strategy parameters according to the emotional needs density index to generate a personalized emotional interaction strategy instruction set. This strategy generation method breaks the limitations of existing systems that use fixed strategy templates. By dynamically adjusting the weights of strategy parameters, the generated interaction strategies closely match the user's personalized emotional needs and the density of those needs. For example, when the user's emotional needs density index is high, the system can adjust strategy parameters to prioritize providing more targeted companionship content and interaction methods. When the user's emotional needs change, the system can also adjust strategy parameters promptly to ensure that the interaction strategy always matches the user's current emotional needs. This personalized interaction strategy generation mechanism effectively improves the targeting and flexibility of emotional companionship services, meeting the emotional companionship needs of different users in different scenarios, thereby enhancing the user experience and satisfaction during the interaction process.

[0019] The entire system is supported by deep learning technology, fully leveraging its advantages in multimodal feature fusion, emotional need knowledge graph retrieval, and strategy parameter adjustment to achieve efficient collaboration and data flow between modules. Compared to existing systems, this system significantly improves in terms of emotion recognition accuracy, depth of emotional need analysis, and personalization of interaction strategies. It is better able to adapt to complex and ever-changing emotional companionship scenarios, providing a more feasible solution for the practical application of robotic emotional companionship technology. This will help promote the technological development and widespread application of robotic emotional companionship, providing high-quality and efficient services to more people in need of emotional companionship. Attached Figure Description

[0020] Figure 1 This is a timing diagram of the deep learning-based robotic emotional companionship system described in this invention. Figure 2 A flowchart for the process of obtaining user emotional state vectors; Figure 3 A flowchart for obtaining the emotional need density index. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Please see Figure 1This invention provides a deep learning-based robot emotional companionship system. The system comprises the collaborative operation of three core modules: an emotional state recognition module, an emotional needs analysis module, and an emotional interaction strategy generation module, which together construct a complete emotional intelligent interaction system.

[0023] The emotion state recognition module handles various data streams generated by the user during the interaction process, encompassing multimodal information such as voice, text, and facial expressions. Leveraging advanced feature extraction techniques and multimodal feature fusion algorithms, this module generates a vector comprehensively reflecting the user's current emotional state, providing foundational data support for subsequent emotion needs analysis. The emotion needs analysis module receives this emotion state vector and performs a deep search using a pre-constructed emotion needs knowledge graph. This knowledge graph contains rich emotion needs and corresponding tags. The module uses a matching algorithm to find the need tags that best match the user's current emotional state and further calculates the emotion needs density index. This index reflects the specificity and urgency of the user's emotional needs, ultimately outputting a detailed analysis of the user's emotion needs. The emotion interaction strategy generation module, based on the analysis results of the first two modules, calls a preset basic strategy template. To ensure the generated strategy is more personalized and targeted, the module adjusts the parameter weights in the strategy template according to the user's specific emotional needs, generating a customized set of emotion interaction strategy instructions. This set of instructions guides the system to conduct more natural and considerate emotional support interactions with the user.

[0024] Example 1: See Figure 2The process of acquiring user emotional state vectors begins with the synchronous acquisition and preprocessing of multimodal user interaction data streams. Video streams, audio streams, and depth image data are aligned using a timestamp alignment mechanism to ensure that multimodal information is on the same time base. Video frame sequences are captured at a rate of 30 frames per second with a resolution of 1920x1080 pixels. Each video frame is sent to a preprocessing pipeline for face region detection and normalization. Face detection uses a cascaded classifier based on Haar features. The detected face regions are cropped and scaled to a standardized size of 224x224 pixels, and then grayscale and histogram equalization are performed to enhance image contrast. The preprocessed frame sequence constitutes a three-dimensional data cube as input for subsequent feature extraction. Facial micro-expression feature extraction is achieved through a deep 3D convolutional neural network architecture. This network consists of four 3D convolutional layers and three 3D max pooling layers arranged alternately. The convolutional kernel size is designed to be 3x3x3 to capture both spatial and temporal features simultaneously. The first convolutional layer uses 64 filters, and the number of filters in each subsequent layer doubles. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The pooling layers use a 2x2x2 kernel size to gradually reduce the dimension of the feature map. The network ends with a global average pooling layer to compress the feature map into a 256-dimensional feature vector. This vector encodes the activation intensity and dynamic pattern of facial muscle motor units in the temporal dimension.

[0025] The audio signal processing module operates in parallel. The raw audio input is pre-emphasized to compensate for high-frequency components, followed by frame segmentation with a frame length of 25 milliseconds and a frame shift of 10 milliseconds. Each frame signal is weighted by a Hamming window and then subjected to a Fast Fourier Transform (FFT) to obtain the spectrum. The Mel filter bank contains 40 triangular bandpass filters, covering the critical bandwidth of human auditory perception. The logarithmic energy output is subjected to a Discrete Cosine Transform (DCT) to obtain 39-dimensional Mel frequency cepstral coefficients. The final constructed speech spectrum feature matrix contains temporal and feature dimensions, with the number of rows corresponding to the number of time frames and the number of columns corresponding to the order of the cepstral coefficients. Depth image data processing relies on a skeletal joint tracking algorithm. The Intel RealSense depth sensor provides a depth map stream of 30 frames per second. The skeletal tracking algorithm first extracts the human contour through background subtraction and noise filtering, and then uses a random forest algorithm to regress and calculate the three-dimensional coordinates of 18 major joints. The joint coordinate sequence is smoothed by a Kalman filter to eliminate jitter noise. The final generated limb motion trajectory feature sequence contains the position, velocity, and acceleration information of each joint. These motion parameters are organized into a time series array.

[0026] The multimodal feature fusion network employs a hierarchical attention mechanism for feature integration. Facial micro-expression feature vectors are first projected onto a 512-dimensional latent space through a fully connected layer. The speech spectral feature matrix is ​​encoded into a context-aware representation of the same dimension through a bidirectional long short-term memory network. The limb movement trajectory feature sequence is extracted for temporal patterns through a one-dimensional convolutional network and then mapped to the same 512-dimensional space. The feature representations of the three modalities are input into a cross-attention module, which calculates the correlation weights between modalities and generates a weighted fusion representation. Finally, the fusion representation is dimensionality-reduced through two fully connected layers to generate a 512-dimensional user emotional state vector. The video processing thread, audio processing thread, and deep data processing thread execute in parallel. The feature fusion thread starts computation immediately upon receiving features from each modality. The system manages the data flow through a circular buffer to ensure temporal consistency. Processing latency is controlled to the millisecond level through hardware acceleration and algorithm optimization. The final output user emotional state vector is encapsulated in JSON format with attached timestamp metadata for subsequent modules to call and analyze.

[0027] Example 2: See Figure 3 The process of obtaining the emotional need closeness index begins with the interactive matching of the user's emotional state vector and the emotional need knowledge graph. The emotional need knowledge graph is stored and maintained using a graph database. Its nodes represent various emotional state categories, and edges represent semantic associations and transition relationships between states. Each emotional state node contains a high-dimensional vector representation and its metadata attributes. The metadata covers emotional type labels, intensity value ranges, and descriptions of common triggering situations. The system calculates the cosine similarity between the input user emotional state vector and all node vectors in the knowledge graph. The similarity score is obtained through vector inner product and norm normalization. After sorting and filtering, the calculation results retain the associated emotional state nodes with similarity exceeding a preset threshold, forming a candidate node set for subsequent processing.

[0028] The emotional need tags corresponding to the associated emotional state nodes are extracted from the attribute fields of the graph database. Each node is associated with a primary need tag and several secondary need tags. The tag types include various categories such as emotional support, information needs, and behavioral guidance. The system accesses the interaction history database to query the frequency and duration of these tags in recent interactions. The frequency statistics are normalized based on the number of occurrences within the sliding time window, while the duration statistics are calculated by accumulating the total duration of tag activation and calculating the average value. These statistical values ​​are standardized and used as the basic input parameters for weight calculation.

[0029] The calculation process for the weight of emotional need tags integrates frequency and duration factors. The frequency factor reflects the urgency of the need, while the duration factor reflects the intensity of the need's persistence. These two factors are weighted linearly to generate an initial weight value. This initial weight is then weighted and fused with the similarity score of the corresponding emotional state node. The resulting emotional need density index is a normalized value between 0 and 1; a higher value indicates a greater need for immediate response. The entire calculation process employs a pipelined architecture, with intermediate results cached in an in-memory database to improve real-time processing performance. The personalized emotional interaction strategy instruction set generation module initiates a strategy retrieval process upon receiving the emotional need analysis results. The emotional interaction strategy library uses a graph structure for storage and management. Nodes represent basic strategy templates, and edges represent the applicability and conversion conditions between strategies. Each strategy template contains an executable instruction set and an adjustable parameter set, covering multiple dimensions such as voice output parameters, action execution parameters, and environmental adjustment parameters. The system retrieves strategies based on the matching degree between emotional need tags and strategy objectives. The matching degree calculation comprehensively considers the semantic similarity of the tags and historical performance evaluation data.

[0030] The adjustment of strategy parameter weights is dynamically mapped based on the emotional demand density index. A higher density index corresponds to a more significant parameter adjustment range, such as the increase range of the fundamental frequency offset in the voice intonation parameter, the expansion degree of the action amplitude parameter, and the reduction ratio of the response delay parameter. Each adjustable parameter has a preset adjustment range and mapping function. The mapping function uses a linear interpolation method to convert the density index into a specific parameter value. The adjusted parameter values ​​are injected into the strategy template execution engine after validity verification. The strategy template execution engine deeply integrates the adjusted parameter values ​​with the template instruction set. The engine adopts an interpreted execution architecture design, parsing the strategy instructions one by one and instantiating the parameter values. The generated personalized emotional interaction strategy instruction set contains complete execution elements such as time-series arranged action sequences, voice content text and its output parameters, and environmental device control commands. The instruction set is encapsulated in a structured data format and distributed to each execution unit through a message queue mechanism. The entire generation process ensures end-to-end real-time performance and ensures that the low latency requirements of actual interaction scenarios are met.

[0031] Taking a typical home environment as an example, after finishing a day's work, user Xiao Li sits on the sofa and interacts with an emotional companion robot. The robot captures subtle changes in Xiao Li's facial expressions through its built-in camera, records fluctuations in his voice tone through its microphone, and records the movement trajectory of his posture and gestures through its depth sensor. This multimodal data is transmitted in real time to the system's emotional state recognition module for processing. The emotional state recognition module performs synchronous analysis on the collected data stream. The video stream detects micro-expression features such as an increased frequency of Xiao Li's drooping mouth and deepened wrinkles between his eyebrows. Audio analysis shows that his speech rate is 0.4 times slower than usual and his fundamental frequency has dropped by 15 Hz. Skeletal tracking data shows that his body posture is in a contracted state and his hand gestures have decreased by 60%. The emotional state vector generated after multimodal fusion of these features shows a significant increase in the score of the sadness dimension. After receiving the vector, the emotional needs analysis module begins to perform a matching search in the emotional needs knowledge graph. The knowledge graph contains hundreds of emotional state nodes and their associated needs tags. The system calculates that the node with the highest similarity to the current vector is the "depressed mood" state node, which is associated with three main needs tags: "emotional comfort", "distraction" and "seeking companionship".

[0032] The system further queried the performance data of these demand tags in Xiao Li's historical interactions. The "emotional comfort" tag appeared 3.2 times per week for an average duration of 15 minutes in the past 30 days, the "distraction" tag appeared 2.1 times per week for an average duration of 12 minutes, and the "seeking companionship" tag appeared 4.3 times per week for an average duration of 25 minutes. After weighting these statistics with the node similarity score, the "seeking companionship" tag obtained the highest closeness index of 0.86. The emotional interaction strategy generation module selects the corresponding basic strategy template based on the closeness index. The system retrieved the companionship enhancement strategy template numbered ST-042 from the strategy library. This template contains a set of parameters in three dimensions: voice interaction, action execution, and environmental adjustment. The original parameter settings of the template include a voice volume of 70%, a speech rate of 120 words / minute, a body forward tilt angle of 15 degrees, and an ambient light brightness of 300 lux. The parameter adjustment process dynamically adjusts the parameters based on a density index of 0.86. The voice volume is increased to 85% to enhance presence, the speech rate is reduced to 100 words / minute to create a soothing atmosphere, the body lean angle is increased to 25 degrees to express attentiveness, and the ambient lighting is adjusted to 400 lux to create a warm atmosphere. These adjustments are positively correlated with the density index.

[0033] The strategy execution engine injects the adjusted parameters into the template to generate the final instruction set. This set includes specific instructions such as "play soothing music at 85% volume," "express concern and greetings at a rate of 100 words per minute," "perform a 25-degree forward lean," and "adjust ambient lighting to 400 lux." These instructions are arranged according to temporal logic to ensure a natural and smooth interaction. The system continuously monitors Xiao Li's emotional state changes and fine-tunes the execution parameters based on real-time feedback. When it detects that the user's facial muscles begin to relax and the voice frequency gradually rises, the system appropriately reduces the intensity of the companionship to avoid over-intervention. This dynamic adjustment mechanism ensures the appropriateness and effectiveness of emotional support. After the strategy execution is complete, the system records all data from this interaction into the database, including the initial emotional state vector, the required tags used, the strategy template number, parameter adjustment records, and the final effect evaluation. This data will be used to optimize future emotional needs analysis and strategy generation processes, forming a closed-loop system for continuous improvement.

[0034] Example 3: The long-term sentiment pattern modeling module constructs a time-dependent sentiment state transition model based on continuously collected user sentiment state vectors. This module first performs timestamp alignment and data cleaning on the input sentiment state vector sequence. The cleaning process includes removing outliers and filling in missing values. Outlier detection uses a statistical distribution-based outlier detection method, and missing value filling uses time series interpolation techniques. The processed vector sequence is organized at fixed time intervals to form a regular sentiment state time series data. The construction of the sentiment state transition graph uses an unsupervised clustering method to classify the sentiment state vectors. The clustering algorithm selected is a density-based spatial clustering algorithm, which can automatically determine the number of clusters and handle noisy data. Each cluster center represents a typical sentiment state, and the cluster label serves as the node identifier in the transition graph. The transition edges between nodes are established by analyzing state changes at continuous time points, and the weight of the transition edges is initialized to the number of observations of the state transition.

[0035] The emotional state transition probability matrix is ​​obtained by counting the frequency of transition edges and then normalizing it. The specific calculation method is as follows:

[0036] in: Indicates from state Transition to state The probability, Record from state Transition to state The number of observations, where the denominator represents the number of observations from the state. The probability matrix, which represents the sum of observations of transitions to all possible states, comprehensively depicts the transition patterns and evolutionary trends between different emotional states.

[0037] The identification of periodic emotional fluctuation patterns employs spectral analysis, using discrete Fourier transform to extract frequency features from the emotional state sequence. Periodic patterns corresponding to significant frequency components are identified as potential emotional fluctuation cycles. These patterns are associated with time factors such as daily routines and weekly activities. The system automatically records and labels the formation conditions and triggering environments of these periodic patterns. The generation of a user's long-term emotional profile integrates static features and dynamic patterns. Static features include the distribution characteristics and frequency of emotional states, while dynamic patterns encompass state transition patterns and periodic fluctuations. The profile content is stored in a structured data format, including an emotional state transition probability matrix, a list of significant periodic patterns, and typical emotional sequence patterns. This profile is incrementally updated with the continuous ingestion of new data, using a sliding window mechanism to maintain its timeliness. The updating and optimization process of the emotional state transition model employs an online learning mechanism. Newly arrived emotional state vectors participate in the real-time updating of model parameters. Model updates consider a time decay factor, with recent data receiving higher weights. This design allows the model to adapt to the dynamic changes in user emotional characteristics while maintaining its ability to remember long-term patterns. The entire modeling process is fully automated, requiring no manual intervention or parameter adjustments.

[0038] The long-term emotional pattern modeling module began collecting and analyzing user Xiao Wang's emotional state data for 30 consecutive days. The system recorded his emotional state vectors daily during his evening rest period. These vectors contained eight dimensions of emotional features, representing basic emotional states such as joy, sadness, anger, and surprise. Data collection was fixed between 7 PM and 9 PM to ensure consistency of the time series. In the data preprocessing stage, the collected emotional state vectors were cleaned and standardized. The system detected missing values ​​in the data from days 12 and 23 due to sensor anomalies, which were filled using time series interpolation. The average of the two days before and after the missing values ​​was used to replace them. All vectors underwent z-score standardization to eliminate the influence of dimensions, forming a well-organized time series dataset. Cluster analysis used an improved density clustering algorithm to group the 30 emotional state vectors. The algorithm automatically identified five cluster centers, labeled as "calm state," "pleasant state," "depressed state," "anxious state," and "excited state," respectively. Each cluster center represents a typical emotional pattern, and the clustering results serve as the node basis for the emotional state transition graph.

[0039] The establishment of state transition edges is achieved by analyzing changes in emotional state at adjacent time points. The system found 8 transitions from "calm state" to "pleasant state," 6 transitions from "pleasant state" to "calm state," and 3 transitions from "calm state" to "depressed state." Each transition edge records the number of transitions and the time of occurrence, constructing a complete emotional state transition network. Transition probability calculation is based on normalized processing of state transition frequencies. The transition probability from "calm state" to "pleasant state" is 0.42, and the transition probability from "calm state" to "depressed state" is 0.16. All transition probabilities form a 5×5 transition probability matrix, with rows and columns corresponding to the five emotional states, and each element representing the probability of transitioning from a row state to a column state. Periodic analysis uses Fourier transform to detect the regularity of emotional fluctuations. The system found that the "depressed state" occurs significantly more frequently on Sunday evenings than at other times, while the "pleasant state" has the highest probability of occurrence on Friday evenings. These periodic patterns highly coincide with users' work-rest rhythms, showing obvious temporal regularity. The generation of long-term emotional profiles integrates static features and dynamic patterns. The profile includes coordinate vectors of five cluster centers, a state transition probability matrix, a list of major periodic patterns, and typical emotional sequence patterns. The "calm state," being the most frequent, has a weight of 0.35 in the profile, while other states are assigned weights based on their frequency. The profile update mechanism uses a sliding window method to maintain timeliness. The system retains data from the most recent 30 days for updates, automatically removing the oldest data and adding the latest data daily, recalculating all statistics and pattern features. This design ensures the profile always reflects the user's latest emotional characteristics. In the application phase, the system uses this profile to predict user emotional change trends. When a user enters a "calm state," the system predicts a high probability of transitioning to a "pleasant state" based on the transition probability matrix, thus implementing appropriate emotional support strategies to enhance this positive transition. Simultaneously, when a periodic "depressed state" is detected, emotional support measures are initiated in advance. From data collection to profile updates, no manual intervention is required. The system periodically generates visual reports showcasing the distribution, transition patterns, and periodic characteristics of emotional states, helping to understand the user's emotional change patterns.

[0040] Example 4: The operation of the multi-robot collaborative module is based on a distributed system architecture. Three emotional companion robots (identified as K3, K7, and K12) interact with the user simultaneously in a home environment. Each robot is equipped with the same sensor array and processing unit, collecting the user's emotional state vector once per second and uploading it to a coordination server via a wireless network. The server maintains the vector set within the latest time window for consistency verification. See Table 1 for details. Emotional state vector data reported by each robot at any given time and similarity calculation results: Table 1: Data Records for Consistency Verification of Emotional State Vectors of Multiple Robots

[0041] The coordination server calculates the cosine similarity matrix among all reported vectors. The matrix shows that the similarity between the vectors of robot K12 and the vectors of other robots is lower than the fault tolerance threshold of 0.75. The system automatically marks the vectors reported by K12 as outliers and records the time period corresponding to the outlier. (in (For time tolerance window). The exception handling process triggers a multi-source data re-acquisition mechanism, coordinating the server to send instructions to all robots to re-upload the original multimodal data for the exception period. Robot K3 uploads 15 frames of video sequence, audio clips and depth map data for that period, robot K7 uploads 13 frames of multimodal data, and robot K12 uploads 16 frames of sensor data. All data packets are accompanied by timestamps accurate to milliseconds.

[0042] The weighted fusion process first assesses the confidence level of each robot's sensors, based on historical accuracy, signal quality, and environmental interference. Robot K3, due to recent good maintenance, receives a confidence weight of 0.92; robot K7, due to slight microphone noise, receives a weight of 0.85; and robot K12, due to minor lens damage, receives a weight of 0.78. The re-fusion process employs a confidence-based weighted average algorithm, weighting the multimodal features at each time point. For video feature fusion, facial feature point coordinates from high-confidence robots are prioritized; for audio feature fusion, the spectral characteristics of high-confidence robots are emphasized; and for skeletal joint data fusion, a weighted average coordinate is used.

[0043] The corrected user sentiment state vector is generated through iterative optimization. The first round of fusion generates an initial vector [0.76, -0.14, 0.38]. The second round of fusion further optimizes it to [0.77, -0.16, 0.41] based on spatiotemporal consistency constraints. The final output corrected vector is [0.78, -0.15, 0.40], which is highly consistent with the original vectors of K3 and K7. The consistency verification mechanism performs a final verification of the corrected result, calculating the similarity between the corrected vector and the original vectors of each robot. The similarity with the K3 vector is 0.94, the similarity with the K7 vector is 0.89, and the similarity with the K12 vector is 0.72. After confirming the reliability of the corrected result, the system broadcasts it to all robot nodes. After receiving the correction vector, the robot node updates its local state record. The K12 robot starts a self-calibration program to adjust the sensor parameters and records the abnormal event in the error log for subsequent analysis. After the overall system state returns to consistency, it continues to operate normally. The time from abnormal detection to correction completion is less than 500 milliseconds. During this period, it maintains continuous interaction with the user. The user is unaware of the internal adjustment process of the system, thus achieving seamless multi-robot collaborative service.

[0044] Example 5: The emotional interaction effect evaluation module initiates a monitoring process after the personalized emotional interaction strategy instruction set is executed. The system continuously collects user facial expressions, voice signals, and body movement data. This data undergoes the same feature extraction pipeline as before strategy execution to generate new user emotional state vectors for effect comparison and analysis. The data collection period covers the entire response time after strategy execution, ensuring that the dynamic changes in user emotional state are captured. The calculation of the strategy execution effect score is based on multi-dimensional change analysis. The system records the difference in emotional state vectors before and after strategy implementation. The difference is calculated using vector subtraction to obtain the change in each dimension. The analysis process not only focuses on the magnitude of numerical changes but also considers the degree of matching between the direction of change and the user's long-term emotional characteristics. For example, some users may exhibit delayed responses or adverse reactions to specific types of interaction strategies.

[0045] The user's long-term emotional profile provides an important benchmark in this process. The typical response patterns recorded in the profile help distinguish between normal fluctuations and strategy-induced changes. The system matches the current change pattern with historical patterns based on similarity. Changes with high matching scores are considered more reliable performance indicators. At the same time, the system considers the user's recent emotional baseline level to avoid misjudging natural emotional fluctuations as strategy effects. The scoring algorithm comprehensively considers immediate changes and lasting impacts. Immediate changes are reflected in the short-term change in emotional state after strategy execution, while lasting impacts are assessed by extending the observation window to evaluate persistent emotional changes. The two are combined in a weighted manner to ensure that the score reflects both the effect of rapid response and the value of long-term impact. The scoring results are normalized to a standard range of -1 to 1.

[0046] The strategy template utility weights are updated using a dynamic adjustment mechanism. New and historical scores jointly determine the magnitude of weight changes, with recent scores typically having a higher impact. However, the system also retains long-term performance records to prevent over-adjustment. The weight update formula includes a decay factor and learning rate parameters to ensure the system can adapt quickly to changes while maintaining stability. Performance evaluation data is linked to the strategy template identifier and stored in a time-series database, recording complete context information for each execution, including execution timestamps, environmental conditions, user status, and other metadata. This data provides rich decision-making support for subsequent strategy selection, forming a closed-loop learning system. The strategy selection decision process is based on the updated utility weights. When selecting a strategy, the system comprehensively considers the weight score and its matching degree with the current context. High-weight strategies receive priority selection, but a certain probability is reserved for exploring other strategies to prevent getting trapped in local optima. The decision algorithm balances utilization and exploration. From data collection to weight updates, no manual intervention is required. The system's real-time nature ensures timely strategy optimization, and the low-latency design ensures that every user interaction benefits from continuously improving companionship strategies, forming an ever-evolving emotional interaction ecosystem.

[0047] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0048] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A deep learning-based robotic emotional companionship system, characterized in that, include: The emotion state recognition module extracts facial micro-expression features, speech spectrum features, and body movement trajectory features based on user interaction data streams. It then performs multimodal feature fusion on the facial micro-expression features, speech spectrum features, and body movement trajectory features to generate a user emotion state vector. The emotional needs analysis module retrieves emotional state nodes in the emotional needs knowledge graph based on the user's emotional state vector, matches associated emotional needs tags, calculates the emotional needs density index based on the number of matched emotional needs tags and the strength of association, and generates user emotional needs analysis results. The emotional interaction strategy generation module, based on the user's emotional needs analysis results, calls the basic strategy template in the emotional interaction strategy library, adjusts the strategy parameter weights according to the emotional needs density index, and generates a personalized emotional interaction strategy instruction set.

2. The deep learning-based robotic emotional companionship system according to claim 1, characterized in that, The process of obtaining the user's emotional state vector is as follows: Based on the video frame sequence in the user interaction data stream, the activation intensity of facial muscle motion units is extracted by a three-dimensional convolutional neural network to generate facial micro-expression feature vectors. Based on the audio signal in the user interaction data stream, the speech spectrum feature matrix is ​​extracted by Mel frequency cepstral coefficient transform. Based on depth images in user interaction data streams, limb movement trajectory feature sequences are extracted using a skeletal joint tracking algorithm. The facial micro-expression feature vector, speech spectrum feature matrix, and limb movement trajectory feature sequence are input into a multimodal feature fusion network to generate a user emotional state vector.

3. The deep learning-based robotic emotional companionship system according to claim 1, characterized in that, The process of obtaining the emotional need density index is as follows: Based on the user's emotional state vector, node similarity is calculated in the emotional needs knowledge graph, and related emotional state nodes with similarity exceeding the threshold are filtered out. Extract the set of emotional need tags corresponding to the associated emotional state nodes, and count the frequency and duration of each emotional need tag in historical interactions; The emotional need tag weights are calculated based on the frequency and duration of occurrence, and combined with the similarity scores of associated emotional state nodes, an emotional need density index is generated.

4. The deep learning-based robotic emotional companionship system according to claim 1, characterized in that, The generation process of the personalized emotional interaction strategy instruction set is as follows: Based on the results of the emotional needs analysis, the basic strategy template with the highest matching degree is retrieved from the emotional interaction strategy library; Extract the set of adjustable parameters from the basic strategy template, and calculate the adjustment range of each parameter based on the emotional need density index; The adjusted parameter values ​​are injected into the strategy template execution engine to generate a personalized emotional interaction strategy instruction set.

5. The deep learning-based robotic emotional companionship system according to claim 1, characterized in that, Also includes: The long-term emotional pattern modeling module constructs an emotional state transition graph over time based on multiple user emotional state vectors and calculates the emotional state transition probability matrix. Based on the emotional state transition probability matrix, periodic emotional fluctuation patterns are identified, and a long-term emotional profile of the user is generated.

6. The deep learning-based robotic emotional companionship system according to claim 5, characterized in that, The process of obtaining the user's long-term emotional profile is as follows: The multiple continuously collected user emotional state vectors are subjected to time series alignment processing to construct an emotional state transition edge set; The frequency of occurrence of each transition path in the set of emotional state transition edges is counted, and the normalized transition probability is calculated. Based on the normalized transition probabilities, an emotional state transition probability matrix is ​​constructed, and periodic paths with transition probabilities exceeding a threshold are extracted to generate a long-term emotional feature profile of the user.

7. The deep learning-based robotic emotional companionship system according to claim 1, characterized in that, Also includes: The multi-robot collaboration module establishes a cross-robot emotional state consistency verification mechanism based on the user emotional state vectors collected by multiple robots. When the difference in emotional state across robots exceeds the fault tolerance threshold, a multi-source data re-fusion process is triggered to generate a corrected user emotional state vector.

8. The deep learning-based robotic emotional companionship system according to claim 7, characterized in that, The process of obtaining the corrected user sentiment state vector is as follows: Collect the user emotional state vectors reported by multiple robots, and calculate the cosine similarity matrix between the vectors; Identify abnormal vectors in the cosine similarity matrix whose differences exceed the fault tolerance threshold, and re-collect multimodal interaction data for the corresponding time period; The newly collected multimodal interaction data is weighted and fused to generate a corrected user sentiment state vector.

9. The deep learning-based robotic emotional companionship system according to claim 1, characterized in that, Also includes: The emotional interaction effect evaluation module calculates the strategy execution effect score based on the trend of user emotional state vector change after implementing the personalized emotional interaction strategy instruction set. The performance score of the strategy is fed back to the emotional interaction strategy library to update the utility weight of the corresponding strategy template.

10. The deep learning-based robotic emotional companionship system according to claim 9, characterized in that, The calculation process for the strategy execution effect score is as follows: Record the difference in the user's emotional state vector before and after implementing the personalized emotional interaction strategy instruction set; Based on the direction and magnitude of the difference, and combined with the typical response patterns in the user's long-term emotional profile, a strategy execution effectiveness score is calculated. The strategy execution performance score is associated with and stored with the strategy template identifier for subsequent strategy optimization decisions.

Images