Intelligent robot for accompanying and taking care of the elderly based on emotion recognition and method

Abstract

This application discloses an intelligent companion robot and method for the elderly based on emotion recognition, relating to the field of robot control. It first uses video and audio data to make a preliminary judgment on the elderly person's external behavior, identifying the presence of negative states. Then, it compares real-time collected physiological data with a pre-stored personalized physiological baseline model for the user, thereby obtaining a quantified physiological abnormality score. Finally, through a gating fusion mechanism, this physiological abnormality score is used as the key judgment criterion: if the score significantly deviates from the baseline, it is determined to be physiological discomfort, triggering corresponding health monitoring actions; conversely, it is confirmed as an emotional problem, and emotional reassurance is performed. This method effectively avoids the risk of misjudging physiological discomfort as negative emotions, enabling the robot to deconstruct the true causes behind the appearance, thereby making accurate and safe decisions.

Description

Technical Field

[0001] This application relates to the field of robot control, and more specifically, to an intelligent companion robot and method for the elderly based on emotion recognition. Background Technology

[0002] With the increasing global trend of population aging and the rapid development of artificial intelligence technology, intelligent companion robots that can provide emotional interaction and spiritual comfort are becoming an important direction for addressing the challenges of social aging. For robots to truly understand and serve the elderly, the core lies in their ability to accurately identify the user's state, especially their emotional state, in order to provide appropriate feedback and actions. Therefore, building an efficient and reliable intelligent robot control system capable of deeply understanding the emotional needs of the elderly has become an inevitable requirement for the development of this field.

[0003] However, most existing emotion recognition solutions rely on single or limited modal information, such as capturing facial expressions and body postures through cameras or analyzing voice tone through microphones. These methods suffer from a critical and dangerous flaw when dealing with complex scenarios: they struggle to effectively distinguish between outward manifestations of negative emotions (such as sadness or depression) and similar manifestations caused by physical discomfort (such as sudden pain or heart problems). For example, if an elderly person is in severe pain, their brow furrowed and they fall silent, a robot relying solely on visual and auditory signals is highly likely to misjudge this as deep depression and initiate pre-set emotional soothing programs such as playing music or telling jokes. This erroneous intervention not only fails to alleviate the elderly person's suffering but may also exacerbate their agitation due to noise interference. More seriously, it could delay crucial assistance, leading to irreversible consequences. This confusion arising from signal ambiguity (i.e., multiple internal states leading to similar external manifestations) is a technological bottleneck that current intelligent companion robots must overcome as they leap from emotional companionship to health monitoring. Furthermore, even when some studies attempt to incorporate physiological signals such as heart rate and skin conductance as auxiliary diagnostic criteria, they often overlook individual differences and contextual dynamism. For example, if the system does not establish personalized physiological data baselines for specific users under different activities (such as after exercise or at rest), it is easy to misjudge normal fluctuations as abnormalities, leading to frequent false alarms.

[0004] Therefore, effectively integrating audiovisual signals with personalized physiological signals to establish a system that can accurately decipher the true causes behind the negative states of the elderly has become the key to improving the service quality, reliability, and safety of smart companion robots. Summary of the Invention

[0005] To address the fundamental problem of traditional audiovisual emotion recognition failing to distinguish between negative emotions and physiological discomfort due to signal ambiguity, this application provides a control method for an emotional recognition-based intelligent companion robot for the elderly. The method includes: acquiring the user ID, raw video data, raw audio data, and raw physiological data of the target elderly individual; loading a personalized physiological baseline model from a database based on the user ID; performing initial screening of negative states based on the raw video and audio data to obtain initial screening results and expression feature vectors; performing anomaly detection on the raw physiological data based on the physiological baseline to obtain physiological anomaly scores; performing gating fusion and state attribution on the physiological anomaly scores and expression feature vectors to obtain a final state determination result; and generating control actions to be executed by the robot based on the final state determination result.

[0006] According to another aspect of this application, an emotional recognition-based intelligent companion robot for the elderly is provided, comprising: a target elderly object data acquisition module for acquiring the target elderly object's user ID, raw video data, raw audio data, and raw physiological data; a physiological baseline model loading module for loading a user-personalized physiological baseline model from a database based on the target elderly object's user ID; a negative state initial screening module for performing negative state initial screening based on the raw video data and raw audio data to obtain negative state initial screening results and expression feature vectors; a physiological abnormality score calculation module for performing anomaly detection on the raw physiological data based on the physiological baseline to obtain physiological abnormality scores; a final state determination module for performing gating fusion and state attribution on the physiological abnormality scores and expression feature vectors to obtain a final state determination result; and a robot control action generation module for generating control actions to be executed by the robot based on the final state determination result.

[0007] Compared with existing technologies, this application provides an intelligent companion robot and method for the elderly based on emotion recognition. It addresses the fundamental problem in traditional audiovisual emotion recognition—the inability to distinguish between negative emotions and physical discomfort due to signal ambiguity—by introducing a personalized physiological data dimension. Specifically, it first uses video and audio data to make a preliminary judgment on the elderly person's external behavior, identifying the presence of negative states. Then, it compares real-time collected physiological data with a pre-stored personalized physiological baseline model for the user, obtaining a quantified physiological abnormality score. Finally, through a gating fusion mechanism, this physiological abnormality score is used as the key judgment criterion: if the score significantly deviates from the baseline, it is determined to be physical discomfort, triggering corresponding health monitoring actions; conversely, it is confirmed as an emotional problem, and emotional reassurance is performed. This method effectively avoids the risk of misjudging physical discomfort as negative emotions, enabling the robot to deconstruct the true causes behind the appearance, thereby making accurate and safe decisions. Attached Figure Description

[0008] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings.

[0009] Figure 1 This is a flowchart of a control method for an emotion-recognition-based intelligent companion robot for the elderly, according to an embodiment of this application.

[0010] Figure 2 This is a data flow diagram of a control method for an emotion-recognition-based intelligent companion robot for the elderly according to an embodiment of this application.

[0011] Figure 3 This is a flowchart of step 3 in the control method of the intelligent elderly care robot based on emotion recognition according to an embodiment of this application.

[0012] Figure 4 This is a flowchart illustrating the logical execution of step 5 in the control method of an emotion-based intelligent companion robot for the elderly according to an embodiment of this application.

[0013] Figure 5 This is a block diagram of an emotion-based intelligent companion robot for the elderly according to an embodiment of this application. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. It should be understood that the drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] Based on the main problems in the above-mentioned background technology, this application proposes a control method for an intelligent companion robot for the elderly based on emotion recognition. Figure 1 This is a flowchart of a control method for an emotion-recognition-based intelligent companion robot for the elderly, according to an embodiment of this application. Figure 2 This is a schematic diagram of the data flow of a control method for an emotion-recognition-based intelligent companion robot for the elderly, according to an embodiment of this application. Figure 1 and Figure 2As shown, the control method for an emotion-based intelligent companion robot for the elderly according to an embodiment of this application includes: Step 1, obtaining the user ID, original video data, original audio data, and original physiological data of the target elderly person; Step 2, loading a personalized physiological baseline model of the user from the database based on the user ID of the target elderly person; Step 3, performing a preliminary screening of negative states based on the original video data and original audio data to obtain a preliminary screening result and an expression feature vector; Step 4, performing anomaly detection based on the physiological baseline on the original physiological data to obtain a physiological anomaly score; Step 5, performing gating fusion and state attribution on the physiological anomaly score and expression feature vector to obtain a final state determination result; Step 6, generating control actions to be executed by the robot based on the final state determination result.

[0016] In step 1, the user ID, raw video data, raw audio data, and raw physiological data of the target elderly person are acquired. It should be understood that, in order to accurately safeguard the physical and mental health of the elderly, intelligent companion robots need to go beyond simple surface interaction and deeply understand the essence of the user's state. Existing technologies rely solely on audiovisual signals, which can easily misinterpret severe signs of physiological discomfort as ordinary negative emotions, leading to disastrous consequences. To overcome this bottleneck and achieve a qualitative leap from emotional companionship to health monitoring, the first step of this control method is to construct a comprehensive, multi-dimensional information acquisition framework. Therefore, in order to simultaneously collect various raw data reflecting the user's external performance, internal physiological state, and individual identity, it is necessary to acquire the user ID, raw video data, raw audio data, and raw physiological data of the target elderly person. This provides a solid and indispensable data foundation for subsequent personalized, high-precision state differentiation and attribution analysis.

[0017] In an exemplary embodiment of this application, step 1 is implemented as follows: The process is initiated when the companion robot enters its working state. When it detects a target elderly person entering its service area, it begins the data collection process. First, to ensure personalized analysis, it is necessary to obtain the target elderly person's unique user identification identifier, i.e., the user ID. This user ID is a unique character code pre-entered and stored in the robot's local or cloud database, such as SN20230815LI. It is associated with a specific user, such as all their personal information, preferences, and historical health data. The user ID can be obtained by the elderly person logging in via a touchscreen, or by the robot directly identifying a registered user through a facial recognition module and automatically retrieving their user ID.

[0018] Simultaneously with acquiring the user ID, the robot's onboard sensing devices begin operating. Raw video data is acquired via two wide-angle, high-resolution monocular cameras mounted on a movable gimbal on the robot's head. These cameras continuously capture a 1920x1080 pixel color video stream at 30 frames per second. This video stream comprehensively records rich visual information, including the elderly person's facial expressions, body movements, and surrounding environment. Simultaneously, a circular microphone array integrated into the robot's chest acquires raw audio data. This array can beamform sounds from different directions, effectively suppressing environmental noise. With a sampling rate of 48 kHz and a sampling depth of 24 bits, it converts the captured sound signals into a digitized raw pulse-code modulation (PCM) audio stream. This audio data not only contains the elderly person's speech content but, more importantly, preserves paralinguistic information such as intonation, speech rate, and volume variations.

[0019] Raw physiological data is acquired through a wearable health monitoring device worn by the elderly and continuously connected to the companion robot via Bluetooth Low Energy 5.0. This device can be a smart wristband or a medical-grade chest patch that fits snugly against the skin. It integrates multiple sensors to monitor a range of key physiological indicators in real time. Specifically, it uses a photoplethysmography (PPG) sensor to collect heart rate (beats / minute) and heart rate variability (calculated using the standard deviation of consecutive cardiac cycles) every five seconds; a skin conductance sensor to continuously measure skin conductance levels (in micro-Siemens) at a frequency of ten times per second to reflect emotional arousal; and a thermistor sensor to measure body surface temperature every minute. The continuous time-series data generated by these sensors—heart rate, heart rate variability, skin conductance, and body temperature—constitutes the raw physiological data stream. This data is wirelessly transmitted to the companion robot and strictly aligned with video and audio data collected at the same time, ensuring temporal synchronization of all modal data. Ultimately, at any given analysis point, the controller obtains a complete data packet, including a unique user ID, a piece of raw video data, a piece of raw audio data, and a set of corresponding raw physiological data points.

[0020] In step 2, a personalized physiological baseline model is loaded from the database based on the user ID of the target elderly individual. Specifically, this personalized physiological baseline model represents the statistical distribution of the user's physiological data in different contexts. Correspondingly, the interpretation of physiological signals is highly susceptible to interference from individual differences and contextual changes. For example, an increase in heart rate after a walk is a normal physiological response for an elderly person, but if compared to the standard at rest, it may be incorrectly judged as abnormal. This lack of personalized and contextualized reference is the core reason for the high false alarm rate and low reliability of existing health monitoring programs. To ensure the accuracy of physiological data analysis and avoid misjudgments caused by individual differences and normal activities, this application retrieves a personalized physiological baseline model specifically designed for the current user and covering different life situations from the database, based on the user's unique identifier. This provides a dynamic and highly personalized benchmark for subsequent physiological abnormality detection and is a prerequisite for achieving accurate state attribution.

[0021] In an exemplary embodiment of this application, step 2 is implemented as follows: If the user ID obtained in the previous step is SN20230815LI, the controller uses this ID as a query instruction to send a request to the user model database stored in the robot's local non-volatile memory. The database retrieves and returns a user-specific physiological baseline model that uniquely corresponds to the ID. This model is a structured dataset that represents the statistical distribution of physiological data calibrated for the user in different life situations. Specifically, the model uses a series of predefined situational labels as indexes, such as resting state, after light activity, after a meal, and during sleep. These situational judgments can be automatically identified by the robot based on auxiliary information such as time and activity status (determined by the accelerometer).

[0022] Under each context label, a key parameter describing the user's physiological state in that context—a multivariate Gaussian distribution—is stored: a mean vector and a covariance matrix. The dimension of the mean vector corresponds to the dimension of the collected physiological data features, representing the average normal level of various physiological indicators for the user in that context. The covariance matrix is ​​a square matrix that not only describes the fluctuation range of each physiological indicator (the diagonal elements are the variance) but also captures the interrelationships between these indicators (the off-diagonal elements are the covariance). For example, for user SN20230815LI, the loaded model data structure may contain a set of parameters corresponding to the resting state: such as the resting mean vector [70, 58, 1.2], corresponding to a heart rate of 70 beats / minute, heart rate variability of 58 milliseconds, and skin conductance of 1.2 microSiemens, respectively, and a 3x3 resting covariance matrix. Additionally, it includes another set of parameters after mild activity: such as the mean vector after activity = [92, 45, 2.1] and the corresponding covariance matrix after activity.

[0023] It is worth noting that the parameters of these models were obtained through a period of calibration during the initial setup phase. During the calibration period of several weeks, physiological data of the user was continuously collected under the guidance of a robot or medical personnel in different defined scenarios. For example, the user was guided to sit quietly for fifteen minutes each morning, and all physiological data samples collected during this period were marked as resting state; data collected within ten minutes after returning from an afternoon walk were marked as mild activity. After calibration, statistical calculations were performed on the large number of data samples collected in each scenario. Using methods such as maximum likelihood estimation, the mean vector and covariance matrix belonging to each scenario were calculated. These calculated parameters were permanently stored in the database and bound to the user ID, SN20230815LI, thus forming a personalized baseline model for that user. After execution, the user's complete personalized physiological baseline model was loaded into the processing unit's cache.

[0024] In step 3, based on the original video and audio data, a preliminary screening of negative states based on audiovisual signals is performed to obtain the preliminary screening results and expressive feature vectors. It is understandable that before making a final judgment on the elderly person's condition, it is necessary to first capture any potential abnormal signals from their external, observable behavior. Human emotions and states are expressed in parallel through multiple channels such as facial expressions, body language, and tone of voice; information from a single modality is easily subject to biased or erroneous interpretation. Therefore, before introducing high-precision physiological data for in-depth attribution analysis, it is necessary to first fuse the audiovisual signals to construct a comprehensive expressive feature description. Based on this, a preliminary screening of negative states based on audiovisual signals is performed to achieve a highly sensitive preliminary screening by processing video and audio data streams in parallel and effectively fusing them. This determines whether there are currently noteworthy negative states and simultaneously generates a quantitative feature vector that can finely characterize the external manifestation of the state, providing crucial contextual information for subsequently distinguishing its underlying causes.

[0025] In one exemplary embodiment of this application, Figure 3 This is a flowchart of step 3 in the control method of an emotion-recognition-based intelligent companion robot for the elderly according to an embodiment of this application. Figure 3As shown, step 3, based on the original video data and original audio data, performs a preliminary screening of negative states based on audiovisual signals to obtain the preliminary screening result and expression feature vector, including: step 31, performing parallel audiovisual modal feature extraction on the original video data and original audio data to obtain visual feature vector and audio feature vector; step 32, performing feature fusion on the visual feature vector and audio feature vector to obtain the expression feature vector; step 33, inputting the expression feature vector into a pre-trained binary classifier to obtain the preliminary screening result of the negative state, wherein the preliminary screening result of the negative state is used to indicate whether it is a negative state.

[0026] In the above exemplary embodiment, the implementation process of step 3 is as follows: In step 31, the controller initiates two independent processing flows in parallel to extract audiovisual modal features. In an exemplary embodiment of this application, step 31, performing parallel audiovisual modal feature extraction on the original video data and original audio data to obtain visual feature vectors and audio feature vectors, includes: step 311, inputting the original video data into a face detection model to obtain a region of interest (ROI); step 312, inputting the original video data into a pose estimation module to obtain a set of body keypoint coordinates; step 313, inputting the ROI into a convolutional neural network for facial expression analysis to obtain a facial emotion embedding vector; step 314, calculating a pose feature vector based on the set of body keypoint coordinates; and step 315, concatenating the facial emotion embedding vector and the pose feature vector to obtain a visual feature vector.

[0027] Specifically, step 311 is executed first. Each frame of the original video data is input into the face detection model to obtain accurate regions of interest (ROIs) for the face. The face detection model used in this embodiment is a deep-optimized multi-task cascaded convolutional neural network. This model consists of three sequentially cascaded sub-networks with increasing complexity: the proposal network (P-Net), the refinement network (R-Net), and the output network (O-Net). These three sub-networks work together to achieve efficient detection and accurate localization of faces in a coarse-to-fine manner. The internal parameters of these three networks, including the weights of all convolutional kernels and the weights and biases of fully connected layers, are pre-trained through supervised learning on a large-scale public dataset containing millions of labeled face locations. When a 1920x1080 pixel original video image is input into the model, firstly, in order to detect faces of different sizes, the original image is scaled into an image pyramid. The image pyramid consists of a series of images of varying sizes. Starting from the original size, it iteratively shrinks the images multiple times with a preset scaling factor, such as 0.709, until the image size is smaller than a certain threshold, such as 12x12 pixels. Then, the first sub-network, the proposal network, begins its work. It is a fully convolutional network containing three convolutional layers and two max-pooling layers, with a lightweight structure that can quickly scan each layer of the image pyramid using a sliding window. For each location the window scans, the proposal network outputs two results: a confidence score for the face in that region, and a preliminary bounding box regression vector used to correct the initial position and size of the window. This stage generates a massive number of highly overlapping candidate face bounding boxes. To reduce redundancy, a non-maximum suppression (NMS) technique is employed. This technique sorts all candidate boxes according to their confidence scores and iteratively removes other boxes whose overlap with the highest-confidence box exceeds a preset threshold, such as 0.7. After processing and initial filtering by the proposal network, the number of candidate boxes is significantly reduced. Next, all candidate regions that passed the first stage of screening are cropped from the original image and uniformly scaled to a fixed size, such as 24x24 pixels, before being batch-input into the second sub-network, the refinement network. The refinement network has a deeper and more complex convolutional layer structure than the proposal network, containing three convolutional layers and two max-pooling layers, with a fully connected layer at the end, thus possessing stronger discriminative power. It performs a more thorough analysis of each input candidate image patch, outputting a confidence score for the region as a face and a more accurate bounding box regression vector. This step filters out a large number of false positives, such as background regions with textures similar to faces. Similarly, after the refinement network output, non-maximum suppression is performed again, using a stricter overlap threshold, such as 0.5, to further merge bounding boxes and refine the candidate set.Finally, candidate regions with high confidence that have passed the refinement network test are cropped and scaled to a larger standard size, such as 48x48 pixels, and then input into the last and most complex sub-network of the cascaded structure—the output network. The output network has the deepest structure and the most parameters, containing four convolutional layers, three max-pooling layers, and a larger fully connected layer. It performs the final and most refined discrimination and localization. It not only outputs the final face confidence score and bounding box regression vector, but in some configurations, it can also simultaneously output the coordinates of facial key points, such as the center of the eyes, the tip of the nose, and the corners of the mouth. In this embodiment, its high-precision bounding box output is mainly utilized. After the output network finishes processing, a final non-maximum suppression is performed to obtain the final face detection result. For example, for a given image frame, after the cascaded processing of the three-level network described above, the final output may be a unique, high-confidence bounding box coordinate, in the form of [850, 320, 1070, 540]. These four values ​​represent the x and y coordinates of the top-left and bottom-right corners of the bounding box, respectively. The final region of interest (ROI) for the face in this image frame is a 220x220 pixel rectangular image block cropped from the original 1920x1080 pixel image based on this set of coordinates. This image block clearly contains the user's face and excludes all background interference.

[0028] Next, step 312 is executed. Similarly, the set of coordinates of the user's body key points is detected for each frame of the original video data. The pose estimation module used in this embodiment is a deep learning-based, bottom-up analysis strategy. It is a two-branch, multi-stage convolutional neural network. Unlike the top-down method that first detects the human body bounding box and then analyzes the pose within the box, the bottom-up strategy first detects all possible human body key points, such as all wrists and all shoulders, across the entire image range. Then, through the learned inter-part connectivity, these independent key points are assembled into complete skeletons of different individuals. This has better robustness for handling multi-person scenes or situations where the human body is occluded, and is also more computationally efficient. The neural network parameters of this module, including the weights and biases in all its convolutional layers, pooling layers, and activation functions, are pre-trained on a large-scale public dataset containing finely annotated human body key points. When a 1920x1080 pixel original video image is input into the pose estimation module, it first undergoes forward propagation through a shared base network based on the first ten layers of the VGG-19 network. The basic network consists of a series of convolutional and pooling layers. It inherits the classic stacked design of the first ten layers of the VGG-19 network, extracting visual features from simple edges to complex textures by alternating modules of multiple convolutional layers with two to three 3x3 kernels and a max-pooling layer. To meet the high-precision spatial location requirements of pose estimation tasks, the basic network has been adaptively improved, for example, by removing some subsequent pooling layers, ensuring that the final output feature map maintains rich semantic information while also possessing high spatial resolution. This basic network is responsible for extracting low- and mid-level visual features of the image, generating a feature map that serves as the common input to two subsequent parallel branches. These two parallel branches are the core of the module, both structurally represented as a stack of convolutional layers. That is, each branch receives the shared feature map from the basic network and processes it through several convolutional layers with kernel sizes of 3x3 and 1x1. For example, a branch might consist of three 3x3 convolutional layers and a final 1x1 convolutional layer. This final 1x1 convolutional layer plays a crucial role in dimensionality reduction and mapping, with its output channel number precisely set to the number of predicted spectra. The first branch predicts the part confidence map, and the output channel number of its last 1x1 convolutional layer is equal to the number of keypoint types (e.g., eighteen), thus generating a set of heatmaps. In the heatmap corresponding to the left elbow, the brightness value of each pixel in the image represents the confidence that the point is the left elbow. Therefore, the appearance of a high-brightness peak area in the image indicates that the network is highly confident in detecting a left elbow at that location.The second branch predicts the part affinity field. Its last 1x1 convolutional layer has twice the number of output channels of the limb connection type (because each connection is a two-dimensional vector requiring two channels to represent its x and y components), learning the connections between different keypoints. For example, the left forearm connects the left elbow and left wrist. This branch generates a set of two-dimensional vector fields for this limb connection, where vectors within the left forearm region roughly point from the left elbow to the left wrist. This vector field encodes the limb's orientation and position information, providing crucial clues for correctly connecting detected individual joints. The prediction process for these two branches is implemented through a multi-stage refinement architecture. In the first stage, the feature maps extracted by the base network are input to both branches, generating the first set of preliminary confidence maps and part affinity fields. In each subsequent stage, the prediction results generated in the previous stage are concatenated with the original feature maps and then used again as input to the two branches in the current stage. This iterative refinement process allows the network to continuously refine and optimize the prediction results in subsequent stages using contextual information from previous stages, thereby obtaining accurate localization and connection information. After all stages of processing to obtain the final part confidence maps and part affinity fields, the analysis and assembly stages begin. First, non-maximum suppression is applied to each confidence map to find the most likely candidate location for each keypoint. This results in a list containing all detected keypoints in the image, but these are discrete. Next, the part affinity field is used to assemble these discrete points into a skeleton. For any two candidate keypoints, such as a candidate left elbow and a candidate left wrist, a line integral is calculated along the vector field direction of the corresponding limb in the part affinity field on the line segment connecting them. A high integral value indicates that the two points are highly likely to belong to the same limb of the same person. By calculating this integral value for all possible connections, the problem is transformed into a bipartite graph matching problem that finds the maximum weight. Solved using a greedy algorithm or a more complex optimization algorithm, the complete skeleton of the user in the image can finally be constructed. The final output is the set of coordinates of the body keypoints describing the skeleton. For a model using eighteen keypoints, its output is a set of eighteen coordinate pairs, such as: {(x_nose, y_nose), (x_neck, y_neck), (x_left shoulder, y_left shoulder), ..., (x_right ankle, y_right ankle)}. These coordinate values ​​precisely describe the user's body pose in the current video frame.

[0029] Next, step 313 is executed. This embodiment uses a deep residual network such as ResNet-50 for facial expression analysis. This network, due to its fifty-layer depth and unique residual learning structure, exhibits excellent performance in image feature extraction tasks. Its network architecture mainly consists of the following parts: First, an initial convolutional layer uses a large 7x7 convolutional kernel to perform preliminary feature extraction on the input 224x224 pixel image to capture a large range of contour and texture information. Then, a max-pooling layer is used to reduce the resolution of the feature map and enhance the invariance of the features. The main body of the network consists of four consecutive stages containing different numbers of residual blocks. The core idea of ​​a residual block is to introduce shortcut connections, which allow the input signal to directly skip two to three layers of convolutional processing within the block and be added element-wise with the outputs of these layers. This greatly alleviates the gradient vanishing problem common in deep networks, making it possible to train a very deep network. In the ResNet-50 architecture of this embodiment, these four stages are stacked with 3, 4, 6, and 3 residual blocks, respectively. As network depth increases, the spatial size of the feature maps is downsampled level by level, while the number of channels increases accordingly, thus achieving layer-by-layer refinement of features from low to high level and from concrete to abstract. After the last residual block, the network does not use a traditional fully connected layer to flatten the feature maps, but instead uses a global average pooling layer. This layer calculates the average value of all pixel values ​​for each feature map output from the last convolutional layer (e.g., 2048 feature maps), thereby compressing each two-dimensional feature map into a single value. Thus, a high-dimensional feature map tensor is transformed into a 2048-dimensional vector. Finally, this 2048-dimensional vector is fed into a fully connected layer for linear transformation and mapping to the final 512-dimensional embedding space. All parameters of this ResNet-50 network, including the weight matrices and bias vectors in all its convolutional and fully connected layers, are obtained through pre-training on a facial expression dataset. During execution, the regions of interest (ROIs) of a face are fed into this pre-trained ResNet-50 network. Image data is sequentially passed through an initial convolutional layer and four stages of residual blocks, during which features are continuously abstracted and refined. Finally, after a global average pooling layer, the image information is condensed into a 2048-dimensional vector, which is then linearly transformed by a final fully connected layer, ultimately outputting a 512-dimensional floating-point vector. This vector, for example [0.21, -0.88, 0.05, ..., 0.43], is a quantitative description of the user's current facial expression, i.e., a facial emotion embedding vector.

[0030] Then proceed to step 314. First, calculate the static pose descriptor. This feature aims to capture the user's body posture at the current instant. A key descriptor is the torso lean angle. Calculating this angle requires determining a vector representing the torso and a vector representing the vertical direction. The starting point of the torso vector is defined as the center point of the coordinates of the left and right hip keypoints, i.e., M_hip = ((x_left hip + x_right hip) / 2, (y_left hip + y_right hip) / 2), and the ending point is the coordinates of the neck keypoint (x_neck, y_neck). This yields the torso vector V_torso = (x_neck - M_hip_x, y_neck - M_hip_y). In the image coordinate system, the vertically upward vector can be defined as V_vertical = (0, -1). Using the vector dot product formula, the torso lean angle θ_torso = arccos((V_torso · V_vertical) / (|V_torso| * |V_vertical|)). A larger angle value indicates a greater degree of forward or backward lean. Another important static descriptor is the head droop angle. This angle is defined as the angle between the head vector and the torso vector. The head vector starts at the neck keypoint (x_neck, y_neck) and ends at the nose keypoint (x_nose, y_nose), i.e., V_head = (x_nose - x_neck, y_nose - y_neck). The head droop angle θ_head = arccos((V_head·V_torso) / (|V_head|*|V_torso|)). This angle effectively quantifies whether the head is in a drooping state. In addition, a total of sixteen joint angles are calculated, including the left and right arm flexion angles (calculated from the coordinates of the shoulder, elbow, and wrist) and the left and right leg flexion angles (calculated from the coordinates of the hip, knee, and ankle). Next, the dynamic activity descriptor is calculated. To capture the user's body movement amplitude, time-series information is needed. The controller caches the set of all keypoint coordinates within the most recent second (i.e., thirty frames). The body movement amplitude is defined as the average displacement of all eighteen keypoints within the most recent fifteen frames (i.e., half a second). For each keypoint i, its coordinates in the current frame t are P_i(t), and its coordinates 15 frames ago are P_i(t-15). The displacement of this keypoint is d_i = ||P_i(t) - P_i(t-15)||, which is the Euclidean distance between the two points. The amplitude of body activity A = (Σd_i) / 18, where i ranges from 1 to 18. A higher A value indicates that the user's recent body activity has been more intense, while a value close to zero indicates that the user is in a static state. Finally, all the calculated static and dynamic descriptors are combined to form the final posture feature vector. For example, the sixteen joint angle values, the torso forward tilt angle, the head droop angle, and the instantaneous velocities of the eighteen keypoints (calculated by the coordinate difference between the current frame and the previous frame, a total of 36 values) are concatenated. To maintain the consistency of the vector dimensions, this embodiment selects the instantaneous velocities of the eighteen keypoints in the x and y directions (a total of 36 values) as the final 36-dimensional posture feature vector.For example, if the coordinates of Li Dama's left wrist move from (910, 650) in the previous frame to (912, 651) in the current frame, then its corresponding velocity component is (2, 1). Concatenating the velocity components of all eighteen keypoints forms a 36-dimensional vector. This vector not only implicitly contains information about the range of body movement but also includes the direction and relative speed of movement of different body parts, enabling a precise depiction of the user's dynamic posture.

[0031] Finally, step 315 is executed. The facial emotion embedding vector and the pose feature vector are concatenated end-to-end in memory. To ensure consistency, the concatenation order is fixed: the 36-dimensional pose feature vector is appended completely to the end of the 512-dimensional facial emotion embedding vector. This newly generated 548-dimensional vector is the final visual feature vector. It encodes both the user's facial expression details and full-body posture dynamics in a unified vector space, forming a comprehensive quantitative description of the current visual state.

[0032] It is important to emphasize that the complete processing flow from steps 311 to 315 described above is executed independently for each frame of the original video data stream. This generates a continuous sequence of 548-dimensional visual feature vectors synchronized with the video frame rate. To obtain a stable and representative single feature vector from the visual performance during this period, a temporal pooling operation is also performed. Specifically, the controller maintains a sliding window of length N frames (e.g., corresponding to the last two seconds of sixty frames), the length of which is determined based on empirical analysis and experimental verification of typical emotional and physiological state change cycles. The controller then calculates the element-level average of the 548-dimensional visual feature vectors of all frames within the window. A sequence of sixty vectors is aggregated into a single 548-dimensional vector. This time-smoothed vector effectively filters out noise and momentary, unrepresentative actions or expressions that may exist in a single frame, and more accurately reflects the user's overall visual state over a recent period. Therefore, this averaged vector is the final output of the entire visual modality feature extraction step.

[0033] Parallel to visual processing is the audio feature extraction path. First, the continuous raw audio stream is segmented. The controller uses a two-second sliding analysis window to extract audio segments, moving the window forward one second at a time to ensure continuity of analysis and overlap of temporal information. Each extracted two-second audio segment is further divided into a series of short audio frames, each 25 milliseconds long with a 10-millisecond step. Subsequent Low-Level Acoustic Features (LLD) are calculated on these short frames. Feature extraction consists of two parallel sub-processes: prosodic feature set extraction and spectral feature set extraction. In the prosodic feature set extraction process, the focus is primarily on features related to speech melody. For each 25-millisecond short frame, its fundamental frequency (F0) and energy are calculated. The fundamental frequency is calculated using an algorithm based on the autocorrelation function, which effectively estimates the pitch of voiced segments. The energy is obtained by calculating the sum of the squares of the amplitudes of all sample points within the frame. Thus, for a two-second audio clip (containing approximately 200 short frames), two curves changing over time are obtained: the fundamental frequency profile and the energy profile. The spectral feature set extraction process primarily focuses on features related to timbre and articulation. The core is the extraction of Mel frequency cepstral coefficients (MFCCs). For each 25-millisecond short frame, the processing includes: first, passing the image through a Hamming window function to reduce spectral leakage; then, performing a Fast Fourier Transform to obtain the frame's spectrum; next, passing the spectrum through a Mel filter bank consisting of forty triangular bandpass filters to simulate the nonlinear frequency perception characteristics of the human ear; taking the logarithm of the output energy of each filter; and finally, performing a Discrete Cosine Transform, taking the first thirteen coefficients as the MFCC for that frame. To capture the dynamic changes in the spectral features, the first and second differences of these thirteen MFCC coefficients are also calculated. Therefore, for each short frame, a 39-dimensional spectral feature vector is obtained: 13 MFCCs + 13 first-order differences + 13 second-order differences. After calculating the underlying acoustic features of all short frames, these time series need to be statistically aggregated to generate a fixed-dimensional feature vector that represents the entire two-second audio segment. The controller applies a set of statistical functions to the fundamental frequency profile, energy profile, and 39 spectral feature profiles, including mean, standard deviation, maximum, minimum, 25th percentile, and 75th percentile. For example, for the fundamental frequency profile, six statistical values, such as mean and standard deviation, are calculated over two seconds. The same operation is performed on the energy profile. Six statistical values ​​are also calculated for each of the 39 spectral features. Finally, all the calculated statistical features are concatenated to form the final audio feature vector. Specifically, the statistical values ​​from the prosodic features are concatenated with the statistical values ​​from the spectral features to obtain the final audio feature vector, which comprehensively quantifies the prosodic and timbre characteristics of the user's voice over the most recent two seconds.

[0034] In step 32, the fusion process is implemented through a specific neural network module containing linear projection and attention mechanisms. First, because the input visual and audio feature vectors have different dimensions, directly weighting and summing or concatenating them would cause information asymmetry. Therefore, they need to be projected into a unified, common feature space. This process is accomplished through two independent linear projection layers. The visual feature vector is input into a fully connected layer, which linearly transforms it into a 512-dimensional vector using a weight matrix and a bias vector. Similarly, the audio feature vector is also input into another independent fully connected layer, which also transforms it into a 512-dimensional vector using a weight matrix and a corresponding bias vector. Next, to achieve dynamic weighted fusion, an attention network is introduced to calculate the relative importance of the two modalities at the current time step. This attention network receives the original visual and audio feature vectors as input, first concatenating them to form an input vector. This vector is then fed into a small feedforward neural network containing a 64-neuron hidden layer using the ReLU activation function and an output layer with 2 neurons. The two neurons in the output layer correspond to the raw attention scores for the visual and audio modalities, respectively. These two scores are then normalized using a softmax function, generating two weight values ​​that sum to 1: the visual weight and the audio weight. For example, at a given moment, if a user's facial expression shows pain (significant visual feature) but remains silent (insignificant audio feature), the attention network might calculate a visual weight of 0.85 and an audio weight of 0.15. Conversely, if the user is expressionless but groans in pain, the weights might become a visual weight of 0.20 and an audio weight of 0.80. Finally, the calculated weights are used to perform a weighted sum of the two vectors after linear projection. The final representation feature vector = (visual weight * projected visual feature vector) + (audio weight * projected audio feature vector). It is worth noting that all parameters in this fusion module, including the weight matrices and bias vectors of the two linear projection layers, as well as the weights and biases within the attention network, are jointly trained end-to-end with the binary classifier in step 33 as part of the overall negative state screening model.

[0035] In step 33, a preliminary screening of negative states is performed on the expression feature vector. In an exemplary embodiment of this application, step 33, inputting the expression feature vector into a pre-trained binary classifier to obtain the preliminary screening result of the negative states, includes: step 331, the pre-trained binary classifier processes the expression feature vector using the following formula to obtain the probability value of the negative states, the formula being: ,in, To express the feature vector, It is the transpose of the weight vector. For bias terms, It is the sigmoid activation function. It is the probability value of the negative state; step 332, based on the comparison between the probability value of the negative state and the preset threshold, the initial screening result of the negative state is generated.

[0036] Specifically, this step involves a pre-trained binary classifier. In this embodiment, the classifier takes the form of a logistic regression model and contains only a single weight vector. and a bias term Among them, the weight vector The dimension of the bias term is the same as the dimension of the input feature vector; This is a scalar. These two parameters are obtained through supervised training on a public sentiment database containing a large amount of labeled audiovisual data. In practice, the representation feature vector obtained in step 32 is input into the classifier. The classifier first performs a linear computation, namely, calculating the transpose of the weight vector. The dot product of the eigenvectors and the bias term is then added. The result of this calculation is a single scalar value. This scalar value is then fed into a standard sigmoid activation function to convert it into a probability value between 0 and 1. The probability value output. This visually represents how likely the controller is to determine that the user is in a negative state based on the current audiovisual information. Finally, the calculated probability value of the negative state is... The result is compared with a pre-set judgment threshold to obtain the final binary initial screening result. This threshold is set to 0.5, but can be adjusted according to the different requirements of sensitivity and specificity in the application scenario. For example, to ensure that no potential risks are missed, the threshold can be appropriately lowered to 0.45. In this embodiment, the threshold is set to 0.5. If the calculated... If the value is greater than 0.5, the initial screening result for a negative state is determined to be a negative state, and the process proceeds to the next step; otherwise, if... If the result is less than or equal to 0.5, it is a non-negative state. For example, if the input feature vector is calculated to obtain... The value is 0.92. Since this value is greater than 0.5, the controller outputs a preliminary screening result indicating a negative state. This preliminary screening result is a key trigger signal for the subsequent in-depth analysis process: if the result is determined to be a negative state, the entire process will proceed to the next step and start the analysis of physiological data; conversely, if the result is determined to be a non-negative state, the current analysis process will terminate, and the controller will return to continuous monitoring.

[0037] In step 4, anomaly detection based on a physiological baseline is performed on the raw physiological data to obtain a physiological anomaly score. It should be understood that relying solely on externally observable audiovisual signals to judge the state of an elderly person has inherent limitations, as it cannot distinguish between behavioral manifestations caused by genuine negative emotions and similar appearances caused by underlying physiological emergencies (such as cardiac discomfort or dizziness). Raw physiological data streams, such as instantaneous readings of heart rate or body temperature, are also difficult to provide clear criteria for judgment if they are detached from individualized health benchmarks and the current context. To achieve in-depth analysis from state appearance to state attribution, an indicator that can objectively measure the stability of a user's internal physiological state needs to be introduced. Therefore, this application utilizes a previously loaded, highly personalized physiological baseline model to perform a statistically significant distance calculation on the currently collected physiological data, thereby generating a physiological anomaly score that can accurately quantify the degree to which the current physiological state deviates from the individual's normal range.

[0038] In an exemplary embodiment of this application, step 4, performing anomaly detection on the raw physiological data based on a physiological baseline to obtain a physiological anomaly score, includes: step 41, extracting a mean vector and a covariance matrix from a user-personalized physiological baseline model; step 42, extracting physiological features from the raw physiological data to obtain a physiological feature vector; and step 43, processing the physiological feature vector based on the mean vector and the covariance matrix using the following formula to obtain the physiological anomaly score, wherein the formula is: ,in, It is the mean vector. Let covariance matrix be the variance matrix. For physiological feature vectors, It is a score for physiological abnormalities.

[0039] In the above exemplary embodiment, step 4 is implemented as follows: In step 41, the corresponding statistical parameters need to be extracted from the user's personalized physiological baseline model according to the current situation. The controller determines that the user is currently in a resting state by using a built-in activity recognition module, such as judging based on accelerometer data or a time module, such as judging whether it is nighttime sleep time. Accordingly, the controller retrieves and extracts the mean vector and covariance matrix associated with the resting state label from the loaded model. For example, the extracted mean vector is μ_resting = [70, 58, 1.2], which corresponds to the average normal level of heart rate, heart rate variability and skin conductance, respectively. At the same time, the extracted 3x3 resting covariance matrix has diagonal elements representing the normal fluctuation range (variance) of each of these three indicators, and off-diagonal elements describing the interrelationship (covariance) between them.

[0040] Meanwhile, in step 42, the controller extracts physiological features from the continuous raw physiological data from the wearable device. To obtain a stable feature that represents the recent state, the controller analyzes the data using a sliding window of 30 seconds, the duration of which is determined based on experimental optimization on a prior dataset. It calculates the average of all heart rate readings collected within the window, the average of heart rate variability indices (such as the standard deviation of adjacent heartbeat intervals), and the average of skin conductance values. For example, in the most recent 30 seconds, the user's average heart rate is calculated to be 95 beats / minute, the average heart rate variability is 30 milliseconds, and the average skin conductance is 2.5 microsiemens. These three values ​​are combined into a three-dimensional physiological feature vector, namely [95, 30, 2.5].

[0041] Next, in step 43, the physiological abnormality score is calculated based on the extracted baseline model parameters, namely μ_resting and μ_resting and μ_resting covariance matrices, and the currently calculated physiological feature vector. This calculation uses the Mahalanobis distance formula. The calculation process is broken down as follows: First, the current physiological feature vector is calculated. With baseline mean vector The difference vector between We obtain [95-70,30-58,2.5-1.2]=[25,-28,1.3]. Then, we calculate the inverse of the covariance matrix Σ_rest. Then, the transpose of the difference vector is... Inverse of the resting covariance matrix and the difference vector Perform matrix multiplication. The result of this operation is a scalar representing the square of the Mahalanobis distance. Finally, take the square root of this result to obtain the final physiological abnormality score. For example, the calculated score is 4.7. This score is not a physical unit, but a statistical distance that takes into account the degree of deviation of various physiological indicators and their correlation. A score close to 0 indicates that the current physiological state is very consistent with the individual's normal pattern at rest, while a score significantly greater than 0, such as 3 according to calibration data, can be considered abnormal and indicates that the physiological state has deviated significantly from the expected pattern.

[0042] In particular, traditional processing relies entirely on isolated analysis of physiological data, resulting in a purely physiological score that completely ignores the user's outward expressions, posture, and voice. This information fragmentation makes it particularly vulnerable in critical states. For example, when sensors generate slight noise or the user's physiological indicators are on the edge of normal fluctuations, an isolated physiological score cannot confirm an anomaly with the help of the user's pained expression, nor can it suppress noise with the user's calm posture. To build a more intelligent and context-aware anomaly assessment mechanism, this modal barrier must be broken down. Therefore, to avoid using single-modal physiological features to calculate anomaly scores, this application uses a unified vector that deeply integrates expressive and physiological features, thereby generating a more robust and accurate physiological anomaly score that reflects cross-modal collaborative evidence.

[0043] Based on this, in a preferred exemplary embodiment of this application, step 4, performing anomaly detection on the original physiological data based on a physiological baseline to obtain a physiological anomaly score, includes: extracting a mean vector and a covariance matrix from the user's personalized physiological baseline model; and extracting physiological features from the original physiological data to obtain a physiological feature vector. Specifically, these two steps are the same as in the above embodiments and therefore will not be described in detail.

[0044] The response bias projection vector of the expressed feature vector is obtained by projecting the response bias onto the baseline of the physiological feature vector based on the mean vector and covariance matrix. ,in, It is a feature vector. To calculate the reciprocal of each eigenvalue, It is a dot product by position. It is matrix multiplication. This is an expressive feature response bias projection vector. It should be understood that a single expressive feature vector only describes a user's external behavior, while a single physiological feature vector only describes their internal state. The two exist in different feature spaces and cannot be directly compared or fused. This application transforms the expressive feature vector into an expressive feature response within the context of the current physiological bias through a response bias projection operation. Specifically, it first calculates the Mahalanobis distance correlation between the current physiological feature vector and its personal baseline, defined by the mean vector and covariance matrix. This quantity not only reflects the magnitude of the physiological abnormality but also considers the correlation between different physiological indicators due to the introduction of the covariance matrix. Then, this quantified physiological bias is used to modulate or reshape the original expressive feature vector. Finally, a new expressive feature response bias projection vector is generated, whose shape and strength directly reflect the degree of correlation between external behavior and internal physiological abnormalities. If the physiological state is stable, the norm of this projection vector approaches zero, equivalent to silencing the expressive features; if the physiological state is significantly abnormal, this projection vector will highlight those expressive feature dimensions highly correlated with the physiological abnormality pattern, thus providing visual and auditory evidence with a clear physiological context for subsequent fusion.

[0045] Cross-modal information response fusion is performed on the expressive feature response bias projection vector and the physiological feature vector to obtain the fused expressive physiological feature vector, i.e.: ,in, This is the fused physiological feature vector. Correspondingly, after obtaining the projection vector of the expression feature response bias, although it already contains the context of the physiological state, the direct information contained in the original physiological feature vector itself (such as the specific value of heart rate) is equally crucial. If only the projection vector is used, this part of the original physiological information will be lost. Therefore, it is necessary to finally integrate the expression feature information modulated by the physiological context with the original physiological feature information to form a single, comprehensive fused physiological feature vector that can simultaneously represent the intrinsic correlation between the two modalities. This fusion process aims to achieve the relevance alignment of cross-modal information; that is, when measuring the cross-modal context representation, the abnormal state of one modality is used to measure the information cues contained in the other modality. For example, if a certain physiological deviation (such as a sudden increase in heart rate) frequently occurs simultaneously with a certain facial expression (such as fear) in historical data, this fusion mechanism can strengthen their representational correlation in the latent space. A fused physiological feature vector is generated that can synergistically reflect the user's true state, especially in critical and ambiguous scenarios. For example, in cases of minor physiological abnormalities but strong signs of pain in expression, the fused features can significantly amplify the abnormal signal, thereby improving the accuracy of the final judgment. Simultaneously, this fusion also enhances robustness to single-modal noise. For instance, when a heart rate sensor generates erroneous readings due to poor contact, if the user's expression, voice, and posture are all very calm, the fused features will suppress the negative impact of this noise signal on the final decision.

[0046] Physiological anomaly scores are obtained based on the fused physiological feature vector. In other words, a true abnormal state is often the result of the synergistic effect of intrinsic physiological indicators and extrinsic behavioral manifestations. By calculating scores from the fused physiological feature vector, the assessment process can simultaneously consider physiological readings and the user's facial expressions and postures, significantly enhancing the reliability of the scores. For example, a falsely high heart rate reading caused by sensor noise will be suppressed because the fused vector includes the user's calm facial expressions, resulting in a lower, more realistic anomaly score. Conversely, a physiological indicator at a critical value will be amplified because the fused vector includes the user's distressed facial expressions, resulting in a higher anomaly score that can trigger an alert promptly. Specifically, this is achieved through a pre-trained regression model, which can be a simple linear regressor or a small feedforward neural network with several hidden layers. The input layer dimension of this model is... The dimensions are the same. In this embodiment, the regression model is specifically implemented as a pre-trained feedforward neural network, whose input layer strictly matches the dimension of the fused physiological feature vector. Following the input layer, the network architecture contains two fully connected hidden layers, designed to abstract and refine the deep features most relevant to the severity of physiological abnormalities layer by layer. The first hidden layer is configured with 256 neurons, and the second hidden layer further compresses the feature dimension to 64 neurons. Each neuron in these two hidden layers uses a rectified linear unit (ReLU) as its activation function. The network terminates with an output layer consisting of only one neuron, whose sole task is to output the final scalar physiological abnormality score. To ensure that the output value can cover any real number range to accurately represent different degrees of abnormality, the output neuron uses a linear activation function, i.e., it is not weighted by the input and does not perform any nonlinear compression, thus directly generating unbounded, continuous regression predictions. In particular, the regression model is pre-trained using supervised learning on a multimodal dataset labeled with the severity of abnormalities. During execution, the fused physiological feature vector is input into the model, undergoes forward propagation, and ultimately outputs a single scalar value. For example, the input might be a vector representing a slightly elevated heart rate and a pained expression. The model may output a high anomaly score, such as 4.2.

[0047] In step 5, the physiological abnormality score and expression feature vector are gated and fused, and state attribution is performed to obtain the final state determination result. Correspondingly, in the controller's processing flow, two key dimensions of information have been gathered at this point: one is the expression feature vector captured through audiovisual signals, describing the user's external performance; the other is the physiological abnormality score, which quantifies the degree to which the user's internal physiological state deviates from their personal baseline. Both of these pieces of information point to the user possibly being in a negative state, but whether the root cause is a simple emotional problem or an urgent physiological discomfort remains unclear. To resolve this core ambiguity and trigger the most appropriate subsequent response, a final decision-making and attribution mechanism is needed. Therefore, this application constructs a gated fusion logic that prioritizes objective physiological data, first excluding or confirming urgent physiological situations, and then, based on this, performing a more refined emotional attribution on non-physiological abnormality negative states, thereby obtaining a unique, clear, and reliable final state determination result.

[0048] In one exemplary embodiment of this application, Figure 4 This is a flowchart illustrating the logical execution of step 5 in the control method for an emotion-recognition-based intelligent companion robot for the elderly, according to an embodiment of this application. Figure 4As shown, step 5, which involves gating and fusing the physiological abnormality score and the expression feature vector to obtain the final state determination result, includes: step 51, in response to the physiological abnormality score being greater than a preset threshold, determining the final state determination result as physiological discomfort; step 52, in response to the physiological abnormality score being less than or equal to a preset threshold, inputting the expression feature vector into a multi-classifier used to distinguish different negative emotions to obtain the final state determination result.

[0049] In the above exemplary embodiment, step 5 is implemented as follows: The core of this gating structure is a preset physiological abnormality threshold. This threshold is a key, predetermined scalar value used to define the boundary between normal fluctuations and significant abnormalities in physiological states. The threshold is set based on statistical analysis of users' historical physiological data or general population health data. Since this embodiment uses Mahalanobis distance as the physiological abnormality score, which follows a chi-square distribution, a distance value corresponding to a relatively small p-value, i.e., the significance level, which is an indicator used to measure the statistical significance of an observation result, such as p < 0.01, is selected as the threshold to ensure that only statistically improbable physiological deviations are judged as abnormal. In this embodiment, the threshold is set to 3.0.

[0050] When this step is executed, the physiological abnormality score is first compared to the preset threshold of 3.0. This triggers one of two mutually exclusive execution paths. The first path, namely the implementation of step 51, occurs when the physiological abnormality score is greater than the preset threshold. For example, if the physiological abnormality score is 4.7, since 4.7 is greater than 3.0, the controller determines that there is a significant physiological abnormality. In this case, the controller will not perform any further analysis on the expressive feature vector, but will directly determine the final state as physiological discomfort. This determination result will be output immediately to trigger the highest priority response, such as sending an alert to the preset emergency contact.

[0051] The second path, namely step 52, occurs when the physiological abnormality score is less than or equal to a preset threshold. For example, at another time point, if the user's physiological abnormality score is 1.5, since 1.5 is less than or equal to 3.0, the controller determines that their physiological state is within the normal fluctuation range of their personal baseline. At this time, the controller will activate the multi-classifier stored therein, which is used to distinguish different negative emotions, and use the expression feature vector as the input to the classifier. This multi-classifier is a pre-trained feedforward neural network, whose specific architecture includes a 512-dimensional input layer, two hidden layers with 256 and 128 neurons respectively, both using the Modified Linear Unit (ReLU) as the activation function, and an output layer with 3 neurons. The 3 neurons of the output layer correspond to three predefined negative emotion categories: sadness, anger, and anxiety. The output layer uses the Softmax function as the activation function, thus outputting a probability distribution representing the current state belonging to these three emotion categories. All weights and bias parameters of this classifier are pre-trained through supervised learning on an audiovisual sentiment database that has been finely labeled with negative emotions. When the feature vector is input into the classifier, after forward propagation through the network, the output layer generates a three-dimensional probability vector. For example, the output might be [0.85, 0.10, 0.05], representing an 85% probability of being judged as sad, a 10% probability of being judged as angry, and a 5% probability of being judged as anxious, respectively. The controller selects the category with the highest probability value as the final judgment. In this example, since sadness has the highest probability, the final state judgment is determined to be sad.

[0052] In particular, the previously used gated judgment logic treats physiological and emotional issues as two mutually exclusive and sequentially judged independent events, which contradicts the reality that physiological and emotional expressions are often interconnected and mutually causal. For example, irritability may lead to an increased heart rate, while physical pain may trigger a painful expression. Treating these as independent evidence can easily lead to misjudgments in critical and ambiguous states. To overcome this information loss, it is crucial to establish a collaborative reasoning mechanism that can reflect the dynamic correlation across modal contexts. Therefore, this application constructs an end-to-end deep fusion decision model that no longer evaluates any modality in isolation. It provides a way for physiological and expressive features to query and corroborate each other, and performs a one-time, holistic joint state determination based on the final generated unified fusion features, thereby achieving a more accurate and reliable final state attribution.

[0053] Based on this, in a preferred exemplary embodiment of this application, step 5 includes: obtaining the final state determination result based on the fused physiological feature vector. That is, to achieve collaborative reasoning based on comprehensive evidence and completely avoid misjudgments caused by information fragmentation, a single classifier trained on the fused feature space is used. The decision-making process no longer judges physiology first and then emotion, but directly evaluates the joint state of physiology and emotion as a whole. This end-to-end decision-making method can learn more complex and non-linear correlation patterns between the two modalities. Ultimately, it can accurately handle ambiguous states that are difficult to judge under a single modality, significantly improving the judgment accuracy in complex scenarios such as mild physiological abnormalities with strong expression or significant physiological abnormalities with implicit expression. Specifically, this is achieved through a pre-trained multi-class classifier. This classifier is a feedforward neural network whose input layer dimension matches the dimension of the fused physiological feature vector, and its output layer has N neurons, where N is the predefined total number of categories for the final state (e.g., 4 categories: physical discomfort, sadness, anger, and anxiety), and uses a Softmax function to output the probability of each category. The classifier is trained end-to-end on a multimodal dataset labeled with the final state. During execution, the fused physiological feature vector is input into the classifier, and the output layer generates an N-dimensional probability vector, for example, [0.95, 0.02, 0.01, 0.02]. The controller selects the class with the highest probability value as the final decision. In this example, since physiological discomfort has the highest probability, the final state is determined to be physiological discomfort.

[0054] In step 6, based on the final state determination result, the control actions to be executed by the robot are generated. That is, after all the complex perception, analysis, and decision-making in the preceding steps, the controller has arrived at a highly condensed final determination regarding the user's current state. However, this conclusion itself is still an intangible numerical or textual label within the controller, and has not yet had any impact on the external world. Therefore, the final example is to provide effective companionship and support through the robot's proactive behavior, transforming intelligent perception into meaningful physical interaction. To complete the closed loop from understanding to action, this abstract determination result needs to be transformed into concrete, executable robot instructions. Therefore, to act as the final execution layer of the entire control process, it is responsible for accurately mapping the determined user state to one or a series of predefined robot behavior strategies best suited to the current situation, thereby transforming the analysis results into the robot's actual companionship and intervention actions.

[0055] In an exemplary embodiment of this application, step 6 is implemented as follows: This step relies on a built-in, structured state-action mapping unit. The core of this state-action mapping unit is a pre-configured policy library, which is essentially a lookup table or decision tree. This library uniquely associates the final state determination result with a carefully designed, multi-step action sequence. The content of this library is pre-set and configured jointly by elderly care experts, psychologists, and the user's family members based on care knowledge and the user's personal preferences to ensure the professionalism, safety, and personalization of the robot's response.

[0056] When the controller executes this step, it uses the input final state judgment result as a query index, matches it in the policy library, and executes the corresponding action sequence. Two specific implementation scenarios are listed below: Scenario 1: If the input final state judgment result is the string "physiological discomfort," the controller matches this entry in the policy library and immediately initiates the highest priority emergency health alarm action sequence. This sequence includes the following parallel control action instructions: 1. Send an instruction to the communication module, requesting it to immediately dial the first emergency contact preset in the user's SN20230815LI personal profile, such as their child's phone number 138********. 2. Send a text-to-speech instruction to the speech synthesis module, stating: "Are you feeling unwell? I have detected significant abnormalities in your physiological data and am calling your emergency contact." Simultaneously, send an instruction to the audio amplifier module to adjust the volume to a preset high level, such as 85 decibels, to ensure the user can hear clearly. 3. Send an instruction to the screen display module, requesting it to switch the screen background to a striking red and highlight an emergency cross icon. 4. Send navigation commands to the motor controller to drive the robot to move to a safe distance of one meter from the user, and adjust the camera angle to continuously observe the user's status.

[0057] Scenario 2: If the final state result from the input is the string "sad," the controller matches this entry in the policy library and initiates an emotional soothing action sequence. This sequence is designed to focus on gentle companionship and guidance, specifically including: 1. Sending a low-speed movement command to the motor controller, driving the robot to slowly and silently approach the user to avoid startling them. 2. Sending a command to the screen display module to switch the robot's virtual facial expression to a gentle, caring expression, such as slightly drooping eyebrows and a calm mouth. 3. Sending an emotionally optimized text-to-speech command to the speech synthesis module, such as a slower speech rate and softer tone, saying: "You look a little sad. Would you like me to chat with you, or play your favorite Peking Opera?" 4. If no explicit refusal command is received from the user within a preset 30 seconds, a command is sent to the media playback module to start playing the Peking Opera audio file marked as the user's favorite from their personal preference library at a moderate volume.

[0058] In summary, the control method for an emotion-recognition-based intelligent companion robot for the elderly, based on embodiments of this application, is explained. It addresses the fundamental problem in traditional audiovisual emotion recognition—the inability to distinguish between negative emotions and physical discomfort due to signal ambiguity—by introducing a personalized physiological data dimension. Specifically, firstly, video and audio data are used to make a preliminary judgment on the elderly person's external behavior to identify the presence of a negative state. Then, real-time collected physiological data is compared with a pre-stored personalized physiological baseline model for the user to obtain a quantified physiological abnormality score. Finally, a gating fusion mechanism uses this physiological abnormality score as the key judgment criterion: if the score significantly deviates from the baseline, it is determined to be physical discomfort, triggering corresponding health monitoring actions; otherwise, it is confirmed as an emotional problem, and emotional reassurance is performed. This method effectively avoids the risk of misjudging physical discomfort as negative emotions, enabling the robot to deconstruct the true causes behind the appearance, thereby making accurate and safe decisions.

[0059] Figure 5 This is a block diagram of an emotion-recognition-based intelligent companion robot for the elderly according to an embodiment of this application. Figure 5As shown, the elderly intelligent companion robot 100 based on emotion recognition according to an embodiment of this application includes: a target elderly object data acquisition module 110, used to acquire the user ID, original video data, original audio data and original physiological data of the target elderly object; a physiological baseline model loading module 120, used to load a user-personalized physiological baseline model from the database based on the user ID of the target elderly object; a negative state screening module 130, used to perform negative state screening based on audiovisual signals based on the original video data and original audio data to obtain negative state screening results and expression feature vectors; a physiological abnormality score calculation module 140, used to perform abnormal detection based on physiological baseline on the original physiological data to obtain physiological abnormality scores; a final state determination module 150, used to perform gating fusion and state attribution on the physiological abnormality scores and expression feature vectors to obtain a final state determination result; and a robot control action generation module 160, used to generate control actions to be performed by the robot based on the final state determination result.

[0060] Here, those skilled in the art will understand that the specific operations of each step in the above-described intelligent companion robot for the elderly based on emotion recognition have been referenced above. Figures 1 to 4 The control method for the emotion-based intelligent companion robot for the elderly has been described in detail, and therefore, its repeated description will be omitted.

Claims

1. A control method for an intelligent companion robot for the elderly based on emotion recognition, characterized in that, include: Obtain the user ID, raw video data, raw audio data, and raw physiological data of the target elderly population; Based on the user ID of the target elderly population, load the user's personalized physiological baseline model from the database; Based on the original video data and original audio data, a preliminary screening of negative states based on audiovisual signals is performed to obtain the preliminary screening results and expression feature vectors of negative states. Anomaly detection based on a physiological baseline is performed on the raw physiological data to obtain a physiological anomaly score. This includes: extracting the mean vector and covariance matrix from the user's personalized physiological baseline model; extracting physiological features from the raw physiological data to obtain physiological feature vectors; and projecting the expression feature vectors onto the baseline of the physiological feature vectors based on the response bias vector and covariance matrix to obtain the expression feature response bias projection vector, i.e.: ; in, It is the mean vector. Let covariance matrix be the variance matrix. For physiological feature vectors, It is a feature vector. To calculate the reciprocal of each eigenvalue, It is a dot product by position. It is matrix multiplication. It is the projection vector of the expression feature response bias; cross-modal information response fusion is performed on the projection vector of the expression feature response bias and the physiological feature vector to obtain the fused expression physiological feature vector; based on the fused expression physiological feature vector, the physiological abnormality score is obtained; Gated fusion and state attribution are performed on physiological abnormality scores and expression feature vectors to obtain the final state determination result; Based on the final state determination result, the control actions to be executed by the robot are generated.

2. The control method for the intelligent elderly care robot based on emotion recognition according to claim 1, characterized in that, The user-personalized physiological baseline model is a statistical distribution of users' physiological data in different contexts.

3. The control method for the intelligent elderly care robot based on emotion recognition according to claim 1, characterized in that, Based on the original video and audio data, a preliminary screening of negative states based on audiovisual signals is performed to obtain the preliminary screening results and expression feature vectors, including: Parallel audiovisual modal feature extraction is performed on the original video data and the original audio data to obtain visual feature vectors and audio feature vectors; The visual feature vector and the audio feature vector are fused to obtain the expression feature vector; The expression feature vector is input into a pre-trained binary classifier to obtain the initial screening result of the negative state, which is used to indicate whether it is a negative state.

4. The control method for the intelligent elderly care robot based on emotion recognition according to claim 3, characterized in that, Parallel audiovisual modal feature extraction is performed on the original video and audio data to obtain visual and audio feature vectors, including: The raw video data is input into the face detection model to obtain the region of interest for the face; The raw video data is input into the pose estimation module to obtain a set of body keypoint coordinates; The region of interest of the face is input into a convolutional neural network for facial expression analysis to obtain a facial emotion embedding vector; Calculate the pose feature vector based on the set of body keypoint coordinates; The facial emotion embedding vector and the pose feature vector are concatenated to obtain the visual feature vector.

5. The control method for the intelligent elderly care robot based on emotion recognition according to claim 3, characterized in that, The feature vector is input into a pre-trained binary classifier to obtain the initial screening result of the negative states, including: The pre-trained binary classifier processes the expressed feature vector using the following formula to obtain the negative state probability value: ;in, To express the feature vector, It is the transpose of the weight vector. For bias terms, It is the sigmoid activation function. It is the probability value of a negative state; the initial screening result of the negative state is generated based on the comparison between the probability value of the negative state and a preset threshold.

6. The control method for the intelligent elderly care robot based on emotion recognition according to claim 1, characterized in that, Gated fusion and state attribution are performed on physiological abnormality scores and expression feature vectors to obtain the final state determination result, including: If the physiological abnormality score is greater than a preset threshold, the final state determination result is determined to be physiological discomfort. In response to a physiological abnormality score being less than or equal to a preset threshold, the expression feature vector is input into a multi-classifier used to distinguish different negative emotions to obtain the final state determination result.

7. A smart companion robot for the elderly based on emotion recognition, characterized in that, include: The target elderly object data acquisition module is used to acquire the user ID, raw video data, raw audio data, and raw physiological data of the target elderly object; The physiological baseline model loading module is used to load a user-personalized physiological baseline model from the database based on the user ID of the target elderly object. The negative state screening module is used to perform negative state screening based on audiovisual signals based on the original video data and original audio data to obtain the negative state screening results and expression feature vectors. The physiological abnormality score calculation module is used to perform abnormality detection on raw physiological data based on a physiological baseline to obtain a physiological abnormality score. This includes: extracting the mean vector and covariance matrix from the user's personalized physiological baseline model; extracting physiological features from the raw physiological data to obtain physiological feature vectors; and projecting the expression feature vectors onto the baseline of the physiological feature vectors based on the mean vector and covariance matrix to obtain the expression feature response deviation projection vector, i.e.: ; in, It is the mean vector. Let covariance matrix be the variance matrix. For physiological feature vectors, It is a feature vector. To calculate the reciprocal of each eigenvalue, It is a dot product by position. It is matrix multiplication. It is the projection vector of the expression feature response bias; cross-modal information response fusion is performed on the projection vector of the expression feature response bias and the physiological feature vector to obtain the fused expression physiological feature vector; based on the fused expression physiological feature vector, the physiological abnormality score is obtained; The final state determination module is used to perform gating fusion and state attribution on physiological abnormality scores and expression feature vectors to obtain the final state determination result. The robot control action generation module is used to generate control actions to be performed by the robot based on the final state determination result.

Images