Healthcare accompanying robot based on multi-modal facial feature fusion

Abstract

The invention relates to the field of health care robots, and particularly discloses a health care accompanying robot based on multi-modal facial feature fusion, which is characterized in that a facial expression video stream of an elderly object is acquired and refined preprocessing is carried out, and then fine features of each area of the face are captured; then, the fine features of all the areas of the face are intelligently recognized, and higher weights are given to the face micro expressions; and then capturing the time dynamic state of the expression, and further focusing on the key moment and mode of expression change in combination with a time attention module so as to generate a more accurate final emotion vector. And finally, an appropriate action instruction is generated for the robot based on the emotion vector, so that the defect that a traditional method depends on a single macro expression and ignores a contradictory signal is overcome, and accurate and unambiguous understanding and response to the emotion state of the elderly user are achieved.

Description

Technical Field

[0001] The present application relates to the field of health care robots, and more specifically, to a health care companion robot based on multimodal facial feature fusion. Background Art

[0002] With the increasingly aging global population, healthcare companion robots, as an emerging intelligent service device, show great potential in improving the quality of life of the elderly and alleviating the burden of elderly care in society. However, for healthcare companion robots to truly fulfill their companionship role, accurately understanding the true emotional state of elderly users is crucial. When expressing emotions, elderly people often display expressions that are inconsistent with their true feelings for various reasons (such as a desire to avoid causing trouble for others or a habit of concealing emotions). For example, they may habitually smile even when feeling lost or sad. This ambiguity and contradiction in emotional expression poses a severe challenge to traditional emotion recognition technology, potentially leading to inappropriate interactive responses from robots, which in turn impairs the user experience and even has a negative impact.

[0003] Existing healthcare companion robots or emotion recognition systems often rely on the recognition of macro-expressions (such as a raised mouth corner or frown) or are limited to single-modal facial feature analysis. However, this single or shallow emotion recognition approach struggles to effectively capture deeper, more nuanced emotional cues, such as micro-expressions, subtle eye changes, eyebrow movements, and blink rate. When macro-expressions conflict with micro-expressions (for example, a happy macro-expression on the mouth, while the eyes or eyebrows convey neutral or sad micro-expressions), systems that rely solely on macro-expressions often selectively ignore these conflicting signals, leading to misjudgments of the user's true emotions. This misjudgment not only fails to meet the deeper emotional needs of elderly users, but can also exacerbate their negative emotions through inappropriate interactions, severely limiting the practical application of healthcare companion robots.

[0004] Therefore, how to effectively integrate features from different facial regions and levels, especially to resolve conflicts between macro-expressions and micro-expressions, so as to achieve accurate and unambiguous recognition of the emotional state of elderly users, is a key technical problem that needs to be urgently solved in the field of health care companion robots. Summary of the Invention

[0005] In order to solve the above technical problems, this application is proposed.

[0006] According to one aspect of the present application, a health care companion robot based on multimodal facial feature fusion is provided, which includes: an elderly object facial expression video stream acquisition module, which is used to obtain the facial expression video stream of the target elderly object collected by the camera of the health care companion robot; a facial expression preprocessing module, which is used to preprocess the facial expression video stream to obtain a facial expression frame sequence; a facial expression spatial feature extraction module, which is used to extract facial expression spatial features of each facial expression frame in the facial expression frame sequence to obtain a facial expression spatial feature map sequence; a facial expression spatial saliency module, which is used to input the facial expression spatial feature map sequence into a spatial attention module to obtain a facial expression spatial saliency feature vector sequence; a facial expression hidden encoding module, which is used to perform temporal dynamic encoding on the facial expression spatial saliency feature vector sequence to obtain a facial expression hidden state encoding vector sequence; an emotion vector generation module, which is used to input the facial expression hidden state encoding vector sequence into the temporal attention module to obtain a final emotion vector; and an instruction generation module, which is used to generate robot action instructions based on the final emotion vector.

[0007] Compared to existing technologies, this application provides a health and wellness companion robot based on multimodal facial feature fusion, aiming to address the ambiguity and contradictions in emotional expressions of the elderly, as well as the inaccurate recognition issues of existing systems. This solution captures a video stream of facial expressions of elderly subjects and performs sophisticated preprocessing. It then utilizes a facial expression spatial feature extraction module to capture subtle features of various facial regions. Key to this is the facial expression spatial saliency module, which intelligently identifies and assigns higher weight to micro-expressions (such as subtle changes in the eyes and eyebrows), effectively distinguishing true emotions even when they conflict with macro-expressions of the mouth (such as a forced smile). Subsequently, a facial expression hidden encoding module captures the temporal dynamics of expressions. Combined with a temporal attention module, it further focuses on key moments and patterns of expression changes, thereby generating a more accurate final emotion vector. Finally, based on this emotion vector, a command generation module generates appropriate action commands for the robot, overcoming the traditional methods' reliance on single macro-expressions and neglect of conflicting signals, achieving accurate and unambiguous understanding and response to the elderly user's emotional state. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0009] Figure 1This is a block diagram of a health care companion robot based on multimodal facial feature fusion according to an embodiment of the present application.

[0010] Figure 2 This is a data flow diagram of a health care companion robot based on multimodal facial feature fusion according to an embodiment of the present application.

[0011] Figure 3 This is a block diagram of the facial expression preprocessing module in the health care companion robot based on multimodal facial feature fusion according to an embodiment of the present application.

[0012] Figure 4 This is a block diagram of a facial expression spatial saliency module in a healthcare companion robot based on multimodal facial feature fusion according to an embodiment of the present application.

[0013] Figure 5 This is a block diagram of a facial expression hidden coding module in a health care companion robot based on multimodal facial feature fusion according to an embodiment of the present application. DETAILED DESCRIPTION

[0014] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. While the drawings illustrate certain embodiments of the present disclosure, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0015] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in a different order and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0016] In order to overcome the above-mentioned difficulties raised in the background technology, this application is proposed. Figure 1 This is a block diagram of a health care companion robot based on multimodal facial feature fusion according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of a health care companion robot based on multimodal facial feature fusion according to an embodiment of the present application. Specifically, Figure 1 and Figure 2As shown, the health care companion robot 100 based on multimodal facial feature fusion according to the embodiment of the present application includes: an elderly object facial expression video stream acquisition module 110, which is used to obtain the facial expression video stream of the target elderly object collected by the camera of the health care companion robot; a facial expression preprocessing module 120, which is used to preprocess the facial expression video stream to obtain a facial expression frame sequence; a facial expression spatial feature extraction module 130, which is used to extract facial expression spatial features of each facial expression frame in the facial expression frame sequence to obtain a facial expression spatial feature map sequence; a facial expression spatial saliency module 140, which is used to input the facial expression spatial feature map sequence into a spatial attention module to obtain a facial expression spatial saliency feature vector sequence; a facial expression hidden encoding module 150, which is used to perform temporal dynamic encoding on the facial expression spatial saliency feature vector sequence to obtain a facial expression hidden state encoding vector sequence; an emotion vector generation module 160, which is used to input the facial expression hidden state encoding vector sequence into the temporal attention module to obtain a final emotion vector; and an instruction generation module 170, which is used to generate robot action instructions based on the final emotion vector. In particular, this solution can transform complex emotional results derived from in-depth analysis of a single visual modality into highly contextualized robotic behavior that leverages multiple modalities, including auditory, kinesthetic, and visual perception. This shift from single-modal perception to multimodal expression eliminates the monotony and rigidity of the robot's responses, bringing them closer to natural human interaction, significantly enhancing the realism and effectiveness of wellness and companionship services.

[0017] Specifically, the elderly subject facial expression video stream acquisition module 110 is configured to acquire a video stream of the target elderly subject's facial expressions, captured by the health and wellness companion robot's camera. It is understood that during the interaction between the health and wellness companion robot and the elderly user, accurately capturing and understanding the user's emotional state is key to effective companionship. However, for various reasons, such as a reluctance to burden their children or out of habit, elderly people often display facial expressions that are inconsistent with their true feelings when interacting with the robot. For example, they may habitually smile even when feeling lost or sad. This ambiguity and contradiction in emotional expression makes it difficult for the robot to determine the user's true emotions based solely on simple conversations or preset programs. To be able to deeply analyze these subtle and complex facial expression changes, distinguish between forced smiles and genuine joy, and further identify potential anxiety or sadness, the health and wellness companion robot must first be able to acquire the user's facial expression information. Therefore, acquiring the target elderly subject's facial expression video stream, captured by the health and wellness companion robot's camera, is the foundation for all subsequent emotion recognition and intelligent interaction, providing the robot with the raw data input for perceiving the user's emotions.

[0018] In one specific embodiment, the module 110 for capturing the facial expression video stream of an elderly subject operates as follows: The healthcare companion robot is equipped with a high-resolution visual sensor, i.e., a camera. This camera is strategically mounted on the robot's head or chest and has a certain degree of mechanical freedom, enabling it to tilt and rotate horizontally based on the interaction scenario and the target elderly subject's position, ensuring stable and clear capture of the target elderly subject's facial area.

[0019] When the robot activates companion mode or receives user interaction, the camera module is activated and enters operation. The camera first performs a series of self-tests to ensure that its optical components, such as the lens, are unobstructed, the sensor chip is functioning properly, and the communication link with the main processing unit is unobstructed. The camera then begins continuously capturing image data within its field of view at a preset frame rate and resolution. For example, a frame rate of 30 frames per second and a resolution of 1920x1080 pixels can be set to ensure smooth video streaming and rich image detail. To adapt to dynamic changes in indoor lighting conditions, the camera's built-in image signal processor (ISP) runs automatic exposure and white balance algorithms in real time. The automatic exposure algorithm dynamically adjusts the sensor's exposure time, gain, and aperture based on the current scene's brightness to ensure moderate image brightness and avoid overexposure or underexposure of facial areas, which could lead to loss of detail. The white balance algorithm adjusts the gain of the three primary colors (red, green, and blue) according to the scene's color temperature to ensure faithful reproduction of facial skin tones and expressions.

[0020] The collected raw image data is transmitted as a digital signal to the robot's main processing unit via a high-speed data interface, such as USB 3.0 or MIPI CSI. During transmission, to optimize data bandwidth and facilitate subsequent storage and processing, the raw image data undergoes real-time video compression encoding, such as using the H.264 or H.265 encoding format. After receiving the encoded data stream, the main processing unit decapsulates and decodes it using a corresponding decoder, restoring it to a continuous, uncompressed sequence of image frames. This continuous sequence of image frames constitutes the facial expression video stream.

[0021] Specifically, the facial expression preprocessing module 120 is used to preprocess the facial expression video stream to obtain a facial expression frame sequence. Accordingly, the original video stream data usually contains a large amount of redundant information, such as background clutter, lighting changes, and continuous frames that may be irrelevant to facial expression analysis. Directly performing in-depth analysis on these unprocessed raw data will not only cause a huge computational burden and reduce processing efficiency, but may also introduce noise, affecting the accuracy of subsequent feature extraction. In addition, the deep learning model used for facial expression recognition has strict requirements on the format, size and pixel distribution of input data. In order to improve processing efficiency, reduce noise interference, and provide high-quality, standardized input data for subsequent feature extraction and emotion recognition, the collected facial expression video stream needs to be preprocessed. This series of preprocessing operations aims to accurately locate and extract key facial areas from the continuous video stream and convert them into a unified format, thereby laying a solid foundation for subsequent in-depth analysis.

[0022] In particular, in a specific embodiment, Figure 3 FIG is a block diagram of a facial expression preprocessing module in a health care companion robot based on multimodal facial feature fusion according to an embodiment of the present application. Figure 3 As shown, the facial expression preprocessing module 120 includes: a facial expression sampling unit 121, which is used to sample the facial expression video stream at a fixed frequency to obtain a facial expression sampling frame sequence; a face area detection unit 122, which is used to perform face detection on each facial expression sampling frame in the facial expression sampling frame sequence to obtain a face area image sequence; and a face area processing unit 123, which is used to grayscale, size normalize and standardize each face area image in the face area image sequence to obtain the facial expression frame sequence.

[0023] In this embodiment, the facial expression preprocessing module 120 is implemented as follows: First, after the health care companion robot acquires the facial expression video stream of the elderly, it requires preliminary processing to improve the efficiency and accuracy of subsequent analysis. Because the original video frame rate is as high as 30 frames per second and contains a large amount of redundant information, direct processing increases the computational burden and introduces noise, affecting feature extraction and emotion recognition. To preserve key facial expressions and reduce data volume, a fixed-frequency sampling strategy should be adopted to reduce redundancy and computational pressure while providing efficient and concise input for subsequent refined analysis. The facial expression sampling unit 121 receives the facial expression video stream from the camera acquisition module. This video stream consists of a series of continuous image frames and has a fixed original frame rate, such as 30 frames per second. This unit contains a frame counter and sampling interval calculation logic. When processing the video stream, the frame counter increments from the starting frame of the video stream. Simultaneously, based on the preset fixed sampling frequency and the frame rate of the original video stream, it calculates the frame interval for sampling. For example, if the original video stream has a frame rate of 30 frames per second and the preset fixed sampling frequency is 5 frames per second, the sampling interval is calculated as 30 frames per second ÷ 5 frames per second = 6 frames. This means that the facial expression sampling unit extracts a frame from the original video stream every 6 frames. When the frame counter reaches an integer multiple of the sampling interval (for example, frame 1, frame 7, frame 13, etc.), the current frame is selected and added to the facial expression sampling frame sequence. Specifically, this fixed frequency setting was determined through empirical evaluation and optimization. For facial expression recognition tasks, a sampling frequency of 5 to 10 frames per second is considered to be a reasonable range, effectively capturing facial expressions while significantly reducing data volume. For example, in actual deployment, this fixed frequency can be set to 8 frames per second, depending on the specific interaction scenario in which the robot is used, such as daily conversations with the elderly, observing emotional fluctuations, and the available computing resources.

[0024] Next, considering that after the health care companion robot obtains the sampled facial expression frame sequence, these image frames have reduced redundancy, but still contain areas such as background and body that are irrelevant to facial expression analysis. In order to ensure that the subsequent facial expression spatial feature extraction can focus on the most core facial information, avoid the interference of background noise, and improve the efficiency and accuracy of processing, it is necessary to accurately identify and crop the face area from each sampling frame to provide a pure and focused input for the refined analysis of facial expressions, so that the subsequent feature extraction module can more effectively capture the subtle movements and expression changes of facial muscles, thereby laying the foundation for accurate emotion recognition. The face area detection unit 122 receives the facial expression sampling frame sequence from the facial expression sampling unit and performs face detection processing. Face detection uses a pre-trained deep learning model, such as a multi-task cascaded convolutional network model. The model consists of three cascaded convolutional neural networks: P-Net, R-Net and O-Net. P-Net is a lightweight convolutional neural network. Its primary task is to rapidly scan an image pyramid (i.e., scaled versions of an image at different scales) to generate a large number of rough face proposals and their corresponding confidence scores. These proposals are numerous and often overlap significantly. R-Net receives the P-Net output, normalizes their size, and then feeds them into a deeper convolutional neural network. R-Net's task is to further classify these rough proposals (determine whether they are faces) and perform bounding box regression (refining the position and size of the proposals), while filtering out most non-face areas, resulting in fewer, more accurate proposals. Finally, O-Net receives the R-Net refined proposals, normalizes them again, and feeds them into the deepest convolutional neural network. O-Net performs final face classification and bounding box regression, and additionally predicts five facial landmarks (such as the left and right eye centers, the nose tip, and the left and right corners of the mouth). The model is trained on a public dataset containing a large number of face images and their corresponding bounding box and landmark annotations. Its internal weights and bias parameters are continuously adjusted using a backpropagation algorithm and an optimizer. When a facial expression sample frame is input to the trained face detection model, it first generates preliminary candidate boxes through P-Net. These candidate boxes are then refined by R-Net and O-Net in turn, and finally output the bounding box coordinates of the face that may exist in the image (for example, the x, y coordinates of the upper left corner and the width and height) and the corresponding confidence score. For example, if a sample frame containing the face of an elderly person is input, the model may output a bounding box with coordinates of (100, 50, 200, 250), indicating that the face area starts at the (100, 50) pixel point of the image, has a width of 200 pixels, and a height of 250 pixels, with a confidence score of 0.98.In some cases, the model may detect multiple faces, for example, when other faces appear in the background or when the model generates multiple overlapping bounding boxes for the same face. To ensure that only the target elderly subject's face is detected, the face region detection unit uses the non-maximum suppression (NMS) algorithm to eliminate overlapping bounding boxes and selects the most appropriate face based on a pre-set strategy. For example, the face bounding box with the highest confidence score is selected; or, if a face has already been successfully detected in the previous frame, the bounding box closest to the face position in the previous frame is selected to maintain target tracking continuity. For example, a confidence threshold, such as 0.9, can be set; only bounding boxes with a confidence score above this threshold are considered. If multiple faces still meet the criteria, the face with the largest area is selected as the target face. Once the face bounding box of the target elderly subject is determined, the face region detection unit uses this coordinate information to accurately crop the corresponding face region image from the original sample frame. These cropped face region images together form the face region image sequence.

[0025] Finally, even after the facial region image sequence is extracted, issues such as color redundancy, inconsistent sizes, and uneven pixel distribution still exist. Directly inputting these into a deep model can lead to low efficiency, slow convergence, and decreased precision. To improve the accuracy and efficiency of subsequent expression recognition, the images need to be grayscaled, resized, and standardized. This unifies the data format and optimizes pixel distribution to ensure high quality and consistency of the model input, providing a reliable foundation for deep feature extraction. The facial region processing unit 123 first performs grayscale processing. A commonly used grayscale conversion method is based on a weighted average of the human eye's sensitivity to different colors. Specifically, for each pixel in the image, its original red (R), green (G), and blue (B) components are linearly combined according to preset weights to calculate a grayscale value. For example, the following formula can be used: Grayscale value = 0.299*R + 0.587*G + 0.114*B. This formula reflects the human eye's highest sensitivity to green light, followed by red, and lowest sensitivity to blue light. After grayscaling, each color image in the facial region image sequence is converted to a corresponding grayscale image, forming a grayscale facial region image sequence. Next, size normalization is performed. Each grayscale facial region image is scaled or enlarged to a preset uniform size. For example, the target size can be set to 96x96 pixels. During resizing, an interpolation algorithm is used to fill in or calculate the values ​​of new pixels to maintain image clarity and detail. Common interpolation algorithms include bilinear interpolation and bicubic interpolation, which provide smooth transitions based on the grayscale values ​​of surrounding pixels, avoiding noticeable jagged edges or distortion during image scaling. After size normalization, all facial region images have the same width and height, forming a size-normalized facial region image sequence. Finally, standardization is performed. After size normalization, the pixel values ​​of the image remain in the integer range of 0 to 255. To optimize the training performance and convergence speed of deep learning models, these pixel values ​​need to be normalized to a range more suitable for model learning. A commonly used normalization method is Z-score normalization. This method converts pixel values ​​into a distribution with a mean of 0 and a standard deviation of 1 by subtracting the mean of the pixel values ​​and dividing by their standard deviation. Specifically, for each size-normalized image, the average grayscale value (μ) and standard deviation (σ) of all its pixels are first calculated. These means and standard deviations are global statistics pre-calculated based on the large image dataset used to train facial expression recognition models to ensure consistency in standardization. Then, each pixel value (P) in the image is converted according to the formula P=(P-μ) / σ to obtain the standardized pixel value P'. After three consecutive processing steps of grayscale conversion, size normalization, and standardization, the facial region image sequence is finally converted into a facial expression frame sequence with a unified format and standard distribution.

[0026] Specifically, the facial expression spatial feature extraction module 130 is configured to extract facial expression spatial features from each facial expression frame in the facial expression frame sequence to obtain a sequence of facial expression spatial feature maps. It should be understood that after grayscale conversion, size normalization, and standardization of the facial region image, a series of facial expression frames with a unified format and standard distribution are obtained. However, these frames remain raw pixel data. Although preprocessed, the complex facial expression information contained therein, such as fine lines at the corners of the eyes, the curvature of the mouth corners, and the slight rise or fall of the eyebrows, cannot be directly identified and quantified simply through pixel values. In order to extract more abstract, discriminative, and sensitive deep features from this pixel data, thereby effectively distinguishing subtle emotional differences such as forced laughter and genuine joy in the elderly, a powerful feature learning mechanism is required. Therefore, extracting facial expression spatial features from each facial expression frame in the facial expression frame sequence to obtain a sequence of facial expression spatial feature maps is a key step in achieving accurate emotion recognition. It converts raw image data into high-dimensional, semantically rich feature representations, laying the foundation for subsequent emotion analysis.

[0027] In particular, in a specific embodiment, the facial expression spatial feature extraction module 130 is used to: input each facial expression frame in the facial expression frame sequence into a trained 2D convolutional neural network model to obtain the facial expression spatial feature map sequence. It is worth mentioning that the 2D convolutional neural network can automatically learn and extract multi-level and multi-scale spatial features from the original pixel data, without the need to manually design complex feature extractors, which greatly simplifies feature engineering. Its local receptive field and weight sharing mechanism make it robust to translation, scaling and rotation in the image, and can effectively capture subtle texture and shape changes in different facial regions (such as eyes, eyebrows, and mouth). In addition, by stacking multiple layers of convolution and pooling operations, the model can learn abstract features from low-level edges to high-level semantic concepts. These features are crucial for distinguishing subtle emotional differences among the elderly and provide more discriminative representations for subsequent emotion recognition.

[0028] In this embodiment, the facial expression spatial feature extraction module 130 is implemented as follows: a trained 2D convolutional neural network model employs a deep learning architecture, such as a modified VGG network or ResNet network, designed to automatically learn and extract multi-level visual features from images. This model is composed of a stack of multiple convolutional layers, activation layers, pooling layers, and possibly batch normalization layers. Specifically, the architecture of this 2D convolutional neural network model may include: an input layer that receives preprocessed facial expression frames; convolutional layers, each containing multiple learnable convolution kernels (or filters). These convolution kernels slide over the input image or feature map of the previous layer, performing convolution operations to detect local patterns such as edges, textures, and corners. As the network depth increases, the convolutional layers are able to learn increasingly abstract and complex facial features, such as eye shape, mouth contour, and eyebrow orientation. An activation layer applies a nonlinear activation function, such as a rectified linear unit (ReLU), after each convolutional layer. The ReLU function introduces nonlinearity, enabling the network to learn and represent more complex nonlinear relationships, thereby enhancing the model's expressive power. Pooling layers, such as max pooling or average pooling, are used to downsample feature maps, reducing their spatial size, thereby reducing computational effort and making the extracted features robust to small translations and deformations of the input image. Specifically, this 2D convolutional neural network model is trained on a large-scale facial expression dataset, containing millions of facial images with emotion labels (such as happiness, sadness, and anger). During training, the model continuously adjusts its internal convolution kernel weights and bias parameters through a backpropagation algorithm and optimizer. These trained weights and bias parameters enable the model to automatically extract effective spatial features from facial images.

[0029] In practice, when each facial expression frame in a facial expression frame sequence is input into this trained 2D convolutional neural network model, it passes through the model's various convolutional layers, activation function layers, and pooling layers in sequence. The model does not execute the final classification layer, but instead intercepts the feature map output by the model's second-to-last convolutional layer or the last pooling layer. This output is a multi-channel tensor. Each value in this tensor represents the activation intensity of a specific region in the input image along a specific feature dimension, and together they constitute the facial expression spatial feature map for that facial expression frame. By repeating this process for each facial expression frame in the sequence, a sequence of facial expression spatial feature maps containing the feature maps of all frames is ultimately obtained.

[0030] Specifically, the facial expression space saliency module 140 is used to input the facial expression space feature map sequence into the spatial attention module to obtain a facial expression space saliency feature vector sequence. Accordingly, after the facial expression space feature extraction module successfully extracts a high-dimensional facial expression space feature map sequence from the facial expression frame, although these feature maps contain rich spatial information, not all areas are equally important for emotion recognition. For example, when distinguishing between forced smiles and genuine joy in the elderly, micro-expressions around the eyes and subtle movements of the eyebrows may better reflect real emotions than macro-expressions of raised corners of the mouth. Traditional feature extraction methods often treat all spatial positions in the feature map equally, which may cause the model to be unable to effectively focus on the most discriminative areas when dealing with complex or contradictory expressions, thereby affecting the accuracy of emotion recognition. In order to enable the model to dynamically focus on areas in facial images that contribute more to emotion recognition, improve the ability to perceive subtle expressions, and effectively resolve the conflict between macro-expressions and micro-expressions, the present application inputs the facial expression spatial feature map sequence into the spatial attention module to obtain a facial expression spatial saliency feature vector sequence to highlight key information, suppress irrelevant or interfering information, and thus generate a more discriminative feature representation.

[0031] In particular, in a specific embodiment, Figure 4 FIG is a block diagram of a facial expression space saliency module in a health care companion robot based on multimodal facial feature fusion according to an embodiment of the present application. Figure 4 As shown, the facial expression space saliency module 140 includes: a facial expression space attention generation unit 141, used to input the facial expression space feature map into the spatial attention module to obtain a facial expression space attention weight tensor; a facial expression space attention weighting unit 142, used to calculate the element-by-element product between the facial expression space feature map and the facial expression space attention weight tensor to obtain a facial expression space saliency feature map; a facial expression space feature pooling unit 143, used to perform global mean pooling on the facial expression space saliency feature map to obtain the facial expression space saliency feature vector.

[0032] In this embodiment, the facial expression spatial saliency module 140 is implemented as follows: First, a facial expression spatial attention generation unit 141 is used. This unit inputs the input facial expression spatial feature map, for example, an HxWxC tensor, where H and W are the spatial dimensions and C is the number of channels, into a specially designed spatial attention module. This spatial attention module consists of a series of convolutional layers and activation functions, and its purpose is to learn and generate a facial expression spatial attention weight tensor with the same spatial dimensions as the input feature map (for example, a HxWx1 tensor). Specifically, the spatial attention module can first perform global average pooling and global maximum pooling on the input feature map in the channel dimension to obtain two HxWx1 feature maps, representing the average information and maximum information in the channel dimension, respectively. These two feature maps are then concatenated in the channel dimension to form an HxWx2 tensor. This concatenated tensor is then processed through a convolutional layer, for example, a 7x7 convolution kernel with an output channel of 1, and a sigmoid activation function. The sigmoid function compresses the output values ​​to a range between 0 and 1. These values ​​form the facial expression spatial attention weight tensor, where each value represents the importance of the corresponding spatial position. In particular, each parameter in this spatial attention module is trained through end-to-end learning.

[0033] Next is the facial expression spatial attention weighting unit 142. This unit broadcasts the facial expression spatial attention weight tensor in the channel dimension (if the attention weight tensor is single-channel, it is replicated C times to match the number of channels of the feature map), and then multiplies it element-wise with the original facial expression spatial feature map. This multiplication operation implements spatial weighting of the original feature map: the feature values ​​of areas with high attention weights are retained or amplified; the feature values ​​of areas with low attention weights are suppressed or reduced. For example, if the attention weight of the eye region is 0.9 and the attention weight of the background region is 0.1, the feature information of the eye region will be strengthened in the saliency feature map, while the feature information of the background region will be weakened. In this way, the facial expression spatial saliency feature map highlights the facial region features that are more important for emotion recognition.

[0034] Finally, there is the facial expression spatial feature pooling unit 143. Global mean pooling is a common method for converting a two-dimensional feature map into a one-dimensional feature vector. It is achieved by calculating the average value of all spatial positions of each channel in the saliency feature map. For example, if the size of the facial expression spatial saliency feature map is HxWxC, then global mean pooling will calculate the average value of each C channel, and finally obtain a 1x1xC vector, that is, a C-dimensional facial expression spatial saliency feature vector. This feature vector is a generalized representation of the entire saliency feature map. It aggregates all spatial information after attention weighting and compresses it into a fixed-length vector for subsequent processing by the temporal dynamic encoding module. By performing the above operation on each facial expression spatial feature map in the facial expression spatial feature map sequence, they are all converted into a compact feature vector rich in key spatial information, which together constitute a facial expression spatial saliency feature vector sequence.

[0035] Specifically, the facial expression hidden coding module 150 is used to perform temporal dynamic coding on the facial expression spatially significant feature vector sequence to obtain a facial expression hidden state coding vector sequence. It should be understood that human emotional expression is not an isolated static moment, but a dynamically evolving process. For example, sadness may be a gradually deepening emotion, while surprise is a short burst. Relying solely on the feature vector of a single frame to judge emotions will ignore the continuity, contextual information and dynamic change patterns of the expression in the temporal dimension, resulting in the inability to accurately capture the ups and downs, duration and subtle emotional changes of emotions. In particular, for complex emotions such as forced smiles that may exist in the elderly, their authenticity often needs to be judged by observing the dynamic process of the expression. In order to capture the dynamic change rules of facial expressions in the temporal dimension, understand the evolution process of emotions, and use historical information to assist in the judgment of current emotions, thereby achieving more accurate and context-aware real emotion recognition, the present application performs temporal dynamic coding on the facial expression spatially significant feature vector sequence to obtain a facial expression hidden state coding vector sequence.

[0036] In particular, in a specific embodiment, Figure 5 FIG is a block diagram of a facial expression hidden coding module in a health care companion robot based on multimodal facial feature fusion according to an embodiment of the present application. Figure 5 As shown, the facial expression hidden coding module 150 includes: a facial expression space feature calibration unit 151, which is used to perform time-series collaborative dynamic attention calibration on the facial expression space saliency feature vector sequence to obtain a facial expression space saliency calibration feature vector sequence; a hidden state coding unit 152, which is used to input the facial expression space saliency calibration feature vector sequence into a gated recurrent network to obtain the facial expression hidden state coding vector sequence.

[0037] The facial expression hidden encoding module 150 is implemented in the following manner in this embodiment: In particular, in the facial expression spatial saliency module, the facial expression spatial feature map sequence is converted into a facial expression spatial saliency feature vector sequence through spatial attention and global mean pooling. However, in the process of compressing the two-dimensional feature map into a one-dimensional vector, global mean pooling will inevitably smooth or even lose some key local spatial attention information, resulting in the final feature vector may not be able to fully and highly express the image semantics enhanced by the spatial attention module in the original feature map. The degree of this information loss may not be consistent at different time steps of the sequence, thereby introducing a time series imbalance. In order to compensate for this loss and ensure that the temporal dimension of the spatial attention information is retained with high fidelity during the feature vectorization process, it is necessary to establish a time series coordinated calibration mechanism to correct the generated feature vector sequence so that it is more consistent with the original feature map sequence in information expression.

[0038] Based on this, in a specific embodiment, the facial expression space feature calibration unit 151 is used to: first, calculate the difference standard deviation between each group of corresponding facial expression space feature graphs and facial expression space saliency feature vectors in the facial expression space feature graph sequence and the facial expression space saliency feature vector sequence to obtain a facial expression difference standard deviation sequence, that is: ;in, is each facial expression spatial feature map in the facial expression spatial feature map sequence, is the spatially significant feature vector of each facial expression in the sequence of spatially significant feature vectors of facial expression, Indicates the standard deviation. To take the absolute value, is the standard deviation of each facial expression difference in the facial expression difference standard deviation sequence. It should be understood that in the facial expression analysis scenario of a companion health care robot, the facial expression spatial feature map retains rich, spatially attention-enhanced detailed information from various facial regions (such as the corners of the eyes and mouth), while the facial expression spatial saliency feature vector is a summary representation of this information after global pooling. Calculating the standard deviation of each feature set and taking their difference is intended to quantify the difference in hierarchical saliency balance resulting from the dimensionality reduction process from the two-dimensional feature map to the one-dimensional feature vector. This difference standard deviation can be considered a metric reflecting the degree of information loss or distortion caused by the uneven distribution of attention in the spatial dimension during the global mean pooling operation. Performing this calculation for each frame in the sequence yields a sequence of facial expression difference standard deviations, which dynamically reveals the quality of information preservation at each time point throughout the evolution of the expression.

[0039] Next, the facial expression difference standard deviation sequence is reset and activated to obtain a facial expression activation threshold sequence, namely: ;in, and Represent the maximum value and mean value in the standard deviation sequence of facial expression differences, are the individual facial expression activation thresholds in the facial expression activation threshold sequence. Consequently, simply obtaining a sequence of facial expression activation thresholds is insufficient for direct correction, as their values ​​may be noisy or scale-invariant. To generate a more stable and globally context-aware correction factor, the sequence needs to be reactivated. This step combines the differential standard deviation at the current time step, the maximum differential standard deviation across the entire sequence, and the mean using a dynamic weighting formula. The goal is to maximize response-compatible alignment globally. Specifically, if a large differential standard deviation at a particular time point indicates significant information loss, the corresponding activation threshold will be closer to the maximum differential standard deviation in the sequence, generating a stronger correction signal; otherwise, it will be closer to the mean, resulting in a milder correction. This process generates a sequence of facial expression activation thresholds, each of which is weighted according to both the current frame and the global sequence dynamics.

[0040] Finally, based on the facial expression activation threshold sequence, the facial expression space saliency feature vector sequence is subjected to hard constraint correction to obtain the facial expression space saliency calibration feature vector sequence, namely: ;in, To calculate The reciprocal of It is the point product of position. is the calibrated saliency feature vector for each facial expression in the sequence of calibrated saliency feature vectors. That is, after obtaining a globally calibrated sequence of facial expression activation thresholds that accurately reflects the degree of information loss, the final step is to apply a correction. This step multiplies each saliency feature vector by the inverse of the corresponding activation threshold element-wise. This is a hard-constrained correction, which suppresses the values ​​of feature vectors that experience significant information loss during pooling (i.e., higher facial expression activation thresholds) while preserving or even amplifying the influence of feature vectors that retain more information (i.e., lower facial expression activation thresholds). This operation constrains the activation of saliency bias from the perspective of the temporal distribution of saliency balance differences, optimizing the correlation between the temporal distribution of saliency in the sequence of facial expression spatial feature maps and the sequence of saliency feature vectors. This establishes a collaborative attention mechanism between the two, ultimately resulting in a calibrated sequence of saliency feature vectors that more accurately reflects the original spatial attention information.

[0041] It's worth noting that the gated recurrent network (GRN) is a special recurrent neural network architecture that uses a gating mechanism to control the flow and memory of information within a sequence. This effectively addresses the vanishing or exploding gradient problems that traditional RNNs often encounter when processing long sequences, enabling them to better capture long-term dependencies. Therefore, the hidden state encoding unit 152 is implemented as follows: A common gated recurrent network architecture is the gated recurrent unit (GRU). The GRU model primarily consists of two gates: an update gate and a reset gate. The update gate determines how much of the hidden state information from the previous moment is retained and how much of the current input information is incorporated. The reset gate determines how the previous hidden state is combined with the current input. Through these gating mechanisms, the GRU can selectively remember or forget historical information, effectively learning temporal dependencies in sequential data. Specifically, this GRN model was trained on a large-scale dataset of dynamic facial expression videos. During the training process, the model continuously adjusts its internal weights and bias parameters through the back-propagation algorithm and optimizer. These trained weights and bias parameters enable the model to automatically extract temporal dynamic features from facial expression sequences.

[0042] In practice, when a sequence of spatially saliency-calibrated facial expression feature vectors (e.g., a sequence of T vectors, each with dimension D, or TxD) is input into this trained gated recurrent network, the network processes these vectors one by one, timestep by timestep. At each timestep t, the network receives the input feature vector for the current moment (i.e., the tth spatially saliency-calibrated facial expression feature vector in the sequence) and the hidden state for the previous moment. By calculating update gates and reset gates, the network generates a new hidden state. This new hidden state not only incorporates the facial expression features at the current moment, but also incorporates the facial expression information and temporal dynamic patterns from all previous moments. For example, if the sequence length is 10, the network starts with the first vector and iteratively calculates 10 hidden states. Ultimately, the network outputs a sequence of facial expression hidden state encoding vectors with the same length as the input sequence, where each vector encodes all relevant facial expression dynamic information from the beginning of the sequence to the current timestep.

[0043] Specifically, the emotion vector generation module 160 is used to input the facial expression hidden state encoding vector sequence into the temporal attention module to obtain a final emotion vector. It is understandable that in the entire facial expression hidden state encoding vector sequence, not all hidden states at all time steps are equally important to the final emotion judgment. For example, when judging whether an elderly person is truly sad, a micro-expression or eye contact at a certain moment may be more decisive than the expression at other moments, or in a scene of forced laughter, a brief flaw may be the key to revealing the true emotion. Simply averaging all hidden states or taking the last hidden state as the final emotion representation may dilute key information or ignore important historical context. In order to dynamically identify and focus on the time step in the sequence that contributes most to the final emotion judgment, thereby more accurately capturing subtle changes in emotions and key clues, the present application inputs the facial expression hidden state encoding vector sequence into the temporal attention module to obtain the final emotion vector. By assigning different weights to the hidden states at different time steps, the information at key time points is highlighted and irrelevant or interfering information is suppressed, thereby generating a more discriminative emotion representation.

[0044] In particular, in a specific embodiment, the emotion vector generation module 160 includes: a time weight vector generation unit 161, which is used to input the facial expression hidden state encoding vector sequence into the time attention module to obtain a time weight vector; a time weight vector weighting unit 162, which is used to calculate the weighted sum of the facial expression hidden state encoding vector sequence based on the time weight of each position in the time weight vector to obtain the final emotion vector.

[0045] In this embodiment, the emotion vector generation module 160 is implemented as follows: First, a temporal weight vector generation unit 161 is included. It should be understood that the temporal attention module is a mechanism that dynamically assigns a weight to each element in the sequence (here, each hidden state encoding vector) based on the context of the input sequence to indicate its importance to the final output. In particular, in one embodiment, the temporal weight vector generation unit 161 is configured to: The temporal attention module calculates the temporal weight of each facial expression hidden state encoding vector in the facial expression hidden state encoding vector sequence using the following formula: ;in, is the weight matrix, is the bias vector, is the facial expression hidden state encoding vector of each facial expression hidden state encoding vector sequence, is the time weight metric reference vector, is the vector transpose, is the exponential function value with the natural constant e as the base, yes Normalization function, yes The corresponding time weight. Specifically, for each vector in the sequence , which first undergoes a linear transformation Here is a learnable weight matrix, It is a learnable bias vector. These weights and bias parameters are automatically learned through back propagation and optimizer during model training. The goal is to enable the model to accurately identify the time step that is most important for sentiment judgment. Is a learnable time weight metric reference vector, which is a parameter automatically learned by back propagation and optimizer during model training. This vector can be regarded as a query vector, which represents the feature pattern that the model wants to focus on in the current task. By calculating After linear transformation The inner product between them can measure the similarity or correlation between each facial expression hidden state encoding vector and the time weight measurement reference vector, thereby obtaining a scalar value representing the importance score of the hidden state. is a normalization function that converts the importance scores of all time steps into a probability distribution, ensuring that all time weights The sum is 1. Thus, It represents the contribution ratio of the i-th facial expression hidden state encoding vector to the final emotion vector. In this way, the time weight vector generation unit generates a corresponding time weight for each hidden state encoding vector in the sequence, and these weights together constitute the time weight vector.

[0046] Next is the time weight vector weighting unit 162. Each facial expression hidden state encoding vector is scalar multiplied by its corresponding time weight, and then all weighted vectors are summed up, that is, the final emotion vector is This weighted summation effectively aggregates information from all hidden states in the sequence, but assigns higher weights to more important timesteps. For example, if a micro-expression at a certain timestep reveals true emotion, its corresponding hidden state vector will receive a higher weight, thus accounting for a larger proportion of the final emotion vector. This results in a more discriminative emotion representation that integrates all temporal information in the sequence and highlights the contributions of key timesteps.

[0047] Specifically, the instruction generation module 170 is configured to generate robot action instructions based on the final emotion vector. That is, after the healthcare companion robot generates a final emotion vector that accurately reflects the elderly user's true emotions through multimodal facial feature fusion, this high-dimensional numerical representation alone cannot directly drive the robot's behavior. The core value of the healthcare companion robot lies in its ability to provide appropriate, personalized, and empathetic companionship and services based on the user's true emotional state. If the robot merely recognizes emotions without translating these perceptions into concrete, meaningful interactive actions, its companionship effectiveness will be greatly reduced, and inappropriate responses may even exacerbate the elderly user's negative emotions. To enable truly intelligent and adaptive interaction, the robot must translate its deep understanding of the elderly user's emotions into practical, user-friendly robot actions, thereby providing timely and effective psychological comfort or positive guidance. Furthermore, robot action instructions are generated based on the final emotion vector. This step transforms abstract emotional perceptions into concrete, physical actions, enabling the robot to truly serve as an emotional supporter and life assistant for the elderly user.

[0048] In particular, in a specific embodiment, the instruction generation module 170 is used to: input the final emotion vector into a fully connected layer classifier to obtain an emotion recognition result; input the emotion recognition module into a decision module based on a behavior tree to obtain a response strategy; and generate the robot action instruction based on the response strategy.

[0049] In this embodiment, the instruction generation module 170 is implemented as follows: First, the final emotion vector is input into a fully connected layer classifier to obtain an emotion recognition result. The final emotion vector is a high-dimensional numerical representation that contains emotional information after complex processing and attention weighting. To convert it into human-understandable emotion categories, the vector is input into a pre-trained fully connected layer classifier. This classifier consists of one or more fully connected layers (also called dense layers). Each fully connected layer takes the output of the previous layer as input and processes it through a linear transformation and a nonlinear activation function such as ReLU. The final layer is a fully connected layer whose output dimension matches the number of predefined emotion categories, such as happiness, sadness, anger, surprise, calmness, and forced smile. It is followed by a softmax activation function. The softmax function converts the output value into a probability distribution, representing the probability that the final emotion vector belongs to each emotion category. The classifier is trained on a large-scale and diverse emotion annotation dataset, and its internal weights and bias parameters are adjusted using a backpropagation algorithm and an optimizer (such as the Adam optimizer). For example, if the probability distribution of the classifier output after the final emotion vector is input is: happy 0.1, sad 0.05, angry 0.02, surprised 0.03, calm 0.05, forced smile 0.75, then the emotion recognition result is forced smile because it has the highest probability.

[0050] Next, the emotion recognition results are input into a behavior tree-based decision module to determine a response strategy. Emotion recognition results, such as a forced smile, are fed into a behavior tree-based decision module. Those skilled in the art will recognize that a behavior tree is a hierarchical, modular control architecture widely used in robotics and game AI to manage complex behavioral logic. It consists of different types of nodes, including control flow nodes, such as sequence nodes (which execute child nodes sequentially until one fails or all succeed), selector nodes (which try child nodes sequentially until one succeeds or all fail), and parallel nodes (which execute all child nodes simultaneously). Condition nodes check whether specific conditions are met (for example, whether the user is alone or it is nighttime). Behavior nodes perform specific actions or tasks. The decision module pre-designs a series of response strategies for different emotions and situations. Upon receiving an emotion recognition result, the behavior tree is traversed starting from its root node. The root node is a selector node that attempts to activate its subordinate branches based on the current emotion recognition result. For example, if the emotion recognition result is a forced smile, the selector node will find and activate a branch specifically for handling forced smiles. This branch is a sequence node, which defines a series of subtasks or subbehaviors to be executed in sequence. Within this forced smile sequence branch, the first child node might be a condition node, such as checking whether the user is exhibiting a persistent forced smile. This condition node queries the robot's internal state or perception module to determine whether the forced smile emotion has persisted for a period of time or has reached a preset duration threshold. If the condition node evaluates to success (i.e., the user is exhibiting a persistent forced smile), the sequence node proceeds to its next child node. If it evaluates to failure, the current sequence node fails, and the behavior tree may backtrack and try other branches (although branches for specific emotions are designed to terminate directly or have fallback strategies if the condition is not met). Once the condition node succeeds, the sequence node activates subsequent behavior nodes in sequence. For example, the next behavior node might be to initiate an empathy conversation. When this behavior node is activated, it sends instructions to the robot's speech synthesis module to prepare for a pre-defined, empathetic conversation. Next, the sequence node activates a behavior node for adjusting the robot's posture. This node sends instructions to the robot's motion control module, causing the robot to make gestures that express concern, such as tilting its head slightly or leaning forward. Finally, the sequence node activates a behavior node for providing comfort, which might trigger the robot to speak soothing words or play soothing music. This specific path, formed from the root node to the final behavior node, and the high-level action plans represented by all the behavior nodes along this path, together constitute the response strategy for the current emotion.For example, for the emotion recognition result of a forced smile, if the condition is continuously met, the generated response strategy may be an ordered list of actions including initiating an empathetic dialogue, adjusting the robot's posture, and providing comfort.

[0051] Finally, based on the response strategy, the robot motion instructions are generated. A response strategy is a high-level action plan that needs to be translated into low-level action instructions that the robot can understand and execute. For example, a response strategy for the "forced smile" emotion might be a sequential list of actions, including initiating an empathic conversation, adjusting the robot's posture, and providing comfort. These high-level action plans need to be translated into low-level action instructions that the robot can understand and execute. The instruction generation module contains a predefined mapping mechanism or set of conversion rules. It iterates over each high-level action plan in the response strategy and parses it into a series of specific, executable low-level robot motion instructions. For example, when the instruction generation module receives the high-level action plan to initiate an empathic conversation, it generates a specific speech synthesis instruction, such as "speak("You look troubled. Would you like to talk to me?"), based on a preset conversation script or scenario template. This instruction contains the text to be synthesized and is sent to the robot's speech synthesis module for processing. Next, for the action plan to adjust the robot's posture, the instruction generation module parses it into a series of detailed motion control instructions. For example, to express concern, it might generate the move_head(pitch=-5) command, instructing the robot's head to tilt downward by 5 degrees, and the move_torso(lean_forward=0.1) command, instructing the robot's torso to lean forward by 0.1 meters. These parameter values ​​are pre-set based on the robot's kinematic characteristics and the posture requirements for expressing concern, ensuring the robot can perform movements in a natural and expressive manner. These motion commands are then sent to the robot's motion control module. The action plan to provide comfort might be converted into combined commands to achieve a multimodal comforting effect. For example, the command generation module might generate the speech synthesis command "speak("I'm here for you")" and the display module command "display_emoticon(sad_face)" to simultaneously display a sad or concerned face on the robot's screen, enhancing the richness of emotional expression. These commands are sent to the speech synthesis module and the display module, respectively. Together, these robot action commands constitute the final robot action command sequence. These instructions are then sent to the robot's execution layer, driving its various functional modules (such as speech synthesis module, motion control module, and display module) to work together, thereby achieving intelligent response to the emotions of elderly users.

[0052] In summary, the multimodal facial feature fusion-based healthcare companion robot 100, based on the embodiments of this application, is described. It aims to address the ambiguity and contradictions in emotional expressions of elderly individuals, as well as the inaccurate recognition issues of existing systems, as discussed in the prior art. This solution captures a video stream of facial expressions of elderly individuals and performs sophisticated preprocessing. It then utilizes a facial expression spatial feature extraction module to capture subtle features of various facial regions. Key to this is the facial expression spatial saliency module, which intelligently identifies and assigns higher weight to micro-expressions (such as subtle changes in the eyes and eyebrows), effectively distinguishing true emotions even when they conflict with macro-expressions of the mouth (such as a forced smile). Subsequently, the facial expression hidden encoding module captures the temporal dynamics of expressions. Combined with the temporal attention module, it further focuses on key moments and patterns of expression changes, thereby generating a more accurate final emotion vector. Finally, based on this emotion vector, the command generation module generates appropriate action commands for the robot, overcoming the traditional methods' reliance on single macro-expressions and neglect of conflicting signals, achieving accurate and unambiguous understanding and response to the elderly user's emotional state.

[0053] As described above, the health care companion robot 100 based on multimodal facial feature fusion according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with a health care companion algorithm based on multimodal facial feature fusion. In one possible implementation, the health care companion robot 100 based on multimodal facial feature fusion according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or hardware module. For example, the health care companion robot 100 based on multimodal facial feature fusion can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the health care companion robot 100 based on multimodal facial feature fusion can also be one of the many hardware modules of the wireless terminal.

[0054] Alternatively, in another example, the healthcare companion robot 100 based on multimodal facial feature fusion and the wireless terminal may also be separate devices, and the healthcare companion robot 100 based on multimodal facial feature fusion may be connected to the wireless terminal through a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0055] While various implementations of the present disclosure have been described above, the above description is intended to be illustrative and not exhaustive. The present disclosure is not limited to the disclosed implementations, and numerous modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations.

Claims

1. A health care companion robot based on multimodal facial feature fusion, characterized in that: include: The elderly subject's facial expression video stream acquisition module is used to obtain the target elderly subject's facial expression video stream captured by the health care companion robot's camera; A facial expression preprocessing module is used to preprocess the facial expression video stream to obtain a facial expression frame sequence; a facial expression spatial feature extraction module is used to extract facial expression spatial features from each facial expression frame in the facial expression frame sequence to obtain a facial expression spatial feature map sequence; a facial expression spatial saliency module is used to input the facial expression spatial feature map sequence into a spatial attention module to obtain a facial expression spatial saliency feature vector sequence; A facial expression hidden encoding module is used to perform temporal dynamic encoding on the facial expression spatial saliency feature vector sequence to obtain a facial expression hidden state encoding vector sequence; an emotion vector generation module is used to input the facial expression hidden state encoding vector sequence into a temporal attention module to obtain a final emotion vector; An instruction generation module is used to generate robot action instructions based on the final emotion vector.

2. The health care companion robot based on multimodal facial feature fusion according to claim 1, characterized in that: The facial expression preprocessing module includes: a facial expression sampling unit for sampling the facial expression video stream at a fixed frequency to obtain a facial expression sampling frame sequence; a face area detection unit for performing face detection on each facial expression sampling frame in the facial expression sampling frame sequence to obtain a face area image sequence; and a face area processing unit for grayscale, size normalization, and standardization on each face area image in the face area image sequence to obtain the facial expression frame sequence.

3. The health care companion robot based on multimodal facial feature fusion according to claim 2 is characterized in that: The facial expression spatial feature extraction module is used to input each facial expression frame in the facial expression frame sequence into a trained 2D convolutional neural network model to obtain the facial expression spatial feature map sequence.

4. The health care companion robot based on multimodal facial feature fusion according to claim 3 is characterized in that: The facial expression space saliency module includes: a facial expression space attention generation unit, which is used to input the facial expression space feature map into the spatial attention module to obtain a facial expression space attention weight tensor; a facial expression space attention weighting unit, which is used to calculate the element-by-element product between the facial expression space feature map and the facial expression space attention weight tensor to obtain a facial expression space saliency feature map; and a facial expression space feature pooling unit, which is used to perform global mean pooling on the facial expression space saliency feature map to obtain the facial expression space saliency feature vector.

5. The health care companion robot based on multimodal facial feature fusion according to claim 1, characterized in that: The facial expression hidden coding module includes: a facial expression space feature calibration unit, which is used to perform time-series collaborative dynamic attention calibration on the facial expression space saliency feature vector sequence to obtain a facial expression space saliency calibration feature vector sequence; and a hidden state coding unit, which is used to input the facial expression space saliency calibration feature vector sequence into a gated recurrent network to obtain the facial expression hidden state coding vector sequence.

6. The health care companion robot based on multimodal facial feature fusion according to claim 5 is characterized in that: The facial expression space feature calibration unit is used to calculate the difference standard deviation between each group of corresponding facial expression space feature graphs and facial expression space saliency feature vectors in the facial expression space feature graph sequence and the facial expression space saliency feature vector sequence to obtain a facial expression difference standard deviation sequence; The facial expression difference standard deviation sequence is reset and activated to obtain a facial expression activation threshold sequence; based on the facial expression activation threshold sequence, the facial expression space saliency feature vector sequence is hard-constrainedly corrected to obtain the facial expression space saliency calibration feature vector sequence.

7. The health care companion robot based on multimodal facial feature fusion according to claim 1, characterized in that: The emotion vector generation module includes: a time weight vector generation unit, which is used to input the facial expression hidden state encoding vector sequence into a time attention module to obtain a time weight vector; and a time weight vector weighting unit, which is used to calculate the weighted sum of the facial expression hidden state encoding vector sequence based on the time weight of each position in the time weight vector to obtain the final emotion vector.

8. The health care companion robot based on multimodal facial feature fusion according to claim 7, characterized in that: The temporal weight vector generating unit is configured to: the temporal attention module calculates the temporal weight of each facial expression hidden state encoding vector in the facial expression hidden state encoding vector sequence using the following formula, wherein: ;in, is the weight matrix, is the bias vector, is the facial expression hidden state encoding vector of each facial expression hidden state encoding vector sequence, is the time weight metric reference vector, is the vector transpose, is the exponential function value with the natural constant e as the base, yes Normalization function, yes The corresponding time weight.

9. The health care companion robot based on multimodal facial feature fusion according to claim 1, characterized in that: The instruction generation module is used to: input the final emotion vector into a fully connected layer classifier to obtain an emotion recognition result; input the emotion recognition module into a decision module based on a behavior tree to obtain a response strategy; Based on the response strategy, the robot action instruction is generated.

Images