Health care robot system, emotional state recognition method, equipment, medium and product
By integrating image acquisition and heart rate sensing devices in health care robots, combining three-dimensional convolutional image recognition and DS evidence theory, multimodal emotion recognition is achieved, solving the problem of low accuracy of emotional recognition of health care robots and improving service quality.
Patent Information
- Application Number
- CN202510616812.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-14
- Publication Date
- 2025-08-26
AI Technical Summary
The existing health care robot has a single modality of emotional recognition signals, which leads to inaccurate emotional recognition results and affects the quality of companionship.
The image acquisition device and the contactless heart rate sensing device are used to collect continuous image frames and heart rate data of the target object, combined with the three-dimensional convolutional image recognition model and DS evidence theory, and improve the accuracy of emotion recognition through multimodal information fusion.
It improves the accuracy of user emotional state analysis and improves the service quality of health care robots.
Smart Images
Figure CN120531393A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a health care robot system, an emotional state recognition method, equipment, medium and product. Background Art
[0002] Healthcare robots are products designed to monitor and interact with the emotions of the elderly and those with mental health issues. However, current healthcare robots use a single emotion recognition signal modality, resulting in inaccurate emotion recognition for those they accompany, impacting the quality of their companionship. Summary of the Invention
[0003] Embodiments of the present invention provide a health care robot system, an emotional state recognition method, a device, a medium, and a product, which can improve the accuracy of user emotional state analysis and enhance the service quality of the health care robot.
[0004] In a first aspect, an embodiment of the present invention provides a health care robot system, the system comprising:
[0005] A robot body, an image acquisition device and a physiological signal sensing device configured on the robot body, and a signal analysis device;
[0006] The image acquisition device is used to acquire continuous image frames containing the target object; the physiological signal sensing device is a non-contact heart rate sensing device, which is used to acquire the heart rate data of the target object;
[0007] The signal analysis device is used to identify the emotional characteristics of the target object based on continuous image frames, identify the heart rate change characteristics of the target object based on heart rate data, and determine the emotion category corresponding to the target object based on the emotion recognition characteristics and heart rate change characteristics.
[0008] In a second aspect, an embodiment of the present invention further provides an emotional state recognition method, the method comprising:
[0009] Acquire continuous image frames containing the target object and heart rate data of the target object;
[0010] Inputting continuous image frames into a pre-trained three-dimensional convolutional image recognition model to obtain the target object's emotional characteristics and a first confidence level that the emotional characteristics belong to each preset emotional category; and identifying the target object's heart rate variation characteristics based on the heart rate data, and determining a second confidence level that the heart rate variation characteristics belong to each preset emotional category;
[0011] Calculating conflict coefficients between the emotion recognition feature and the heart rate change feature corresponding to different preset emotion categories based on the first confidence level and the second confidence level;
[0012] The probability values of the preset emotion categories corresponding to the target objects are calculated respectively according to the conflict coefficients, and the emotion categories corresponding to the target objects are determined based on the probability values.
[0013] In a third aspect, an embodiment of the present invention further provides an emotional state recognition device, the device comprising:
[0014] An information acquisition module, configured to acquire continuous image frames containing a target object and heart rate data of the target object;
[0015] An information analysis module is configured to input continuous image frames into a pre-trained three-dimensional convolutional image recognition model to obtain emotional characteristics of the target object and a first confidence level that the emotional characteristics belong to each preset emotional category; and to identify heart rate variation characteristics of the target object based on the heart rate data and determine a second confidence level that the heart rate variation characteristics belong to each preset emotional category;
[0016] An information analysis result processing module, configured to calculate, based on the first confidence level and the second confidence level, conflict coefficients between the emotion recognition feature and the heart rate change feature corresponding to different preset emotion categories;
[0017] The information analysis result processing module is further used to calculate the probability values of the preset emotion categories corresponding to the target objects according to the conflict coefficients, and determine the emotion categories corresponding to the target objects based on the probability values.
[0018] In a fourth aspect, an embodiment of the present invention further provides a computer device, comprising:
[0019] one or more processors;
[0020] a memory for storing one or more programs;
[0021] When one or more programs are executed by one or more processors, the one or more processors implement the emotional state recognition method provided by any embodiment of the present invention.
[0022] In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the emotional state recognition method provided by any embodiment of the present invention.
[0023] In a sixth aspect, an embodiment of the present disclosure further provides a computer program product, including a computer program, which, when executed by a processor, implements the emotional state recognition method provided by any embodiment of the present invention.
[0024] The embodiments of the above invention have the following advantages or beneficial effects:
[0025] The embodiment of the present invention comprises a robot body, an image acquisition device and a physiological signal sensing device configured thereon, and a signal analysis device. The image acquisition device is used to acquire continuous image frames containing a target object. The physiological signal sensing device is a non-contact heart rate sensing device used to acquire the target object's heart rate data. The signal analysis device is used to identify the target object's emotional characteristics based on the continuous image frames, identify the target object's heart rate variation characteristics based on the heart rate data, and determine the target object's corresponding emotion category based on the emotion recognition characteristics and the heart rate variation characteristics. The technical solution of the embodiment of the present invention solves the problem of low emotion recognition accuracy of current health care robots, can improve the accuracy of user emotional state analysis, and enhance the service quality of health care robots. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 This is a structural diagram of a health care robot system provided by an embodiment of the present invention;
[0027] Figure 2 is a flow chart of an emotional state recognition method provided by an embodiment of the present invention;
[0028] Figure 3 1 is a schematic structural diagram of an emotional state recognition device provided by an embodiment of the present invention;
[0029] Figure 4 It is a structural diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0030] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.
[0031] Figure 1 This is a structural schematic diagram of a health care robot system provided in an embodiment of the present invention. This embodiment can be applied to scenarios where robots are used, especially situations where a health care robot is used to accompany and care for a target object.
[0032] like Figure 1 As shown, the emotional state recognition of the health care robot system in this embodiment includes:
[0033] A robot body, an image acquisition device and a physiological signal sensing device configured on the robot body, and a signal analysis device.
[0034] Healthcare robots can be intelligent devices used in settings such as homes and community-based elderly care facilities where care and companionship are needed. The robot itself can be configured in a humanoid, anthropomorphic, or other suitable shapes and structures, depending on the usage scenario. The target object is the service object of the healthcare robot system during operation, and is also the object of information collection by the system's image acquisition device and physiological signal sensing device.
[0035] Among them, in order to realize the emotional state recognition function of the health care robot, an image acquisition device and a physiological signal sensing device can be set on the robot body.
[0036] Specifically, the image acquisition device can be a camera or other camera, which is used to capture continuous image frames containing the target object. The number of image acquisition devices can be one or more. The setting position of the image acquisition device can be set in combination with the shape of the robot body, such as the forehead, eyes or other parts of the face of a humanoid robot, so that the target object is included in the field of view of the image acquisition device. Optimally, the target object can be within the central area of the field of view of the image acquisition device. The image acquisition device can be a camera set in at least two locations of the robot body: the eyes, forehead, palms and chest. The camera frame rate is greater than 30 frames per second, which is a high frame rate camera. Figure 1 The solid circle and the dotted circle are both image acquisition devices. The solid circle represents the current setting position of the image acquisition device in the robot body, and the dotted circle represents the optional setting position of the image acquisition device in the robot body.
[0037] The physiological signal sensing device is a non-contact heart rate sensing device used to collect heart rate data from a target subject. A non-contact heart rate sensing device is a device that can measure heart rate without direct contact with the human body. It utilizes photoplethysmography, radar sensing technology, and other methods to monitor heart rate. It offers advantages such as convenience, hygiene, and continuous monitoring, and is widely used in medical and health monitoring fields. For the target subject, the non-contact heart rate sensing device is non-invasive and comfortable, allowing the target subject to undergo heart rate monitoring in a natural and comfortable state. The non-contact heart rate sensing device has strong anti-interference capabilities and is suitable for dynamic scenarios without affecting the heart rate sensing results.
[0038] The continuous image frames containing the target object captured by the image acquisition device can be image frames collected continuously at a preset frame rate for a preset duration, or can be image frames selected from multiple consecutive image frames collected within a preset duration. The continuous image frames contain temporal information, which can be used to identify the dynamic changes in facial expressions during subsequent image recognition, reducing the probability of misidentification caused by a single image.
[0039] The signal analysis device is used to identify the emotional characteristics of the target object based on continuous image frames, identify the heart rate change characteristics of the target object based on heart rate data, and determine the emotion category corresponding to the target object based on the emotion recognition characteristics and heart rate change characteristics.
[0040] Specifically, when the signal analysis device is used to identify the emotional features of the target object based on continuous image frames, the continuous image frames can be input into a pre-trained three-dimensional convolution image recognition model to obtain the emotional features of the target object and the confidence that the emotional features belong to each preset emotion category. Among them, the three-dimensional convolution operation of the three-dimensional convolution image recognition model on the continuous image frames can be a convolution operation on the three-dimensional data using a three-dimensional convolution kernel. The three-dimensional convolution kernel not only slides in the width and height dimensions of the image, but also slides in the depth dimension, so that it can simultaneously capture the features of the data in space and time (or other dimensions). For example, when processing video data, the three-dimensional convolution kernel can simultaneously consider the pixel information of the video frame in space and the time information between frames to extract richer spatiotemporal features. The three-dimensional convolution image recognition model can also be provided with a self-attention mechanism, and the training samples of the three-dimensional convolution image recognition model include image sequences corresponding to different emotion categories.
[0041] The architecture of a 3D convolutional neural network typically consists of multiple components, including an input layer, convolutional layers, pooling layers, fully connected layers, and an output layer. The input layer receives continuous image frames, with image frame data represented as (number of frames × height × width × number of channels). Image frame data may require preprocessing before entering the network, such as normalization, cropping, and scaling, to improve model training and convergence speed. The convolutional layer is the core component of a 3D convolutional neural network and consists of multiple 3D convolution kernels. Each convolution kernel performs a convolution operation on the 3D data, extracting features in the depth, height, and width dimensions using a sliding window. The convolution operation generates a series of feature maps, one corresponding to each convolution kernel. These feature maps capture different aspects of the input data. As the number of network layers increases, the convolutional layers can gradually extract higher-level and more abstract features. To enhance the model's nonlinear representation capabilities, the convolutional layers are typically followed by an activation function, such as the ReLU (Rectified Linear Unit) function. Pooling layers follow the convolutional layers to downsample the data. Common pooling methods include max pooling and average pooling. Max pooling takes the maximum value within a sliding window as the pooling result, while average pooling takes the average value within the sliding window. Pooling can reduce the amount of data and the computational complexity of the model, while also preventing overfitting to a certain extent and preserving important data features. After feature extraction through multiple convolutional and pooling layers, the fully connected layer flattens the extracted feature map into a one-dimensional vector, which is then input into the fully connected layer. Neurons in the fully connected layer are connected to all neurons in the previous layer. Through learned weight parameters, the extracted features are mapped to a specific output space for prediction in classification, regression, or other tasks. A fully connected layer typically contains multiple neurons, and the number of neurons and connection weights can be adjusted to suit different task requirements. The output layer's form varies depending on the specific task. In classification tasks, the output layer typically uses the Softmax function to convert the output of the fully connected layer into a probability distribution for each category, representing the probability of the input data belonging to each category. In regression tasks, the output layer can directly output one or more continuous numerical values. In addition, structures such as residual connection layers and batch normalization layers can be added to the three-dimensional convolutional neural network to improve the performance and generalization ability of the model.
[0042] When the signal analysis device is used to identify the heart rate variation characteristics of a target subject based on heart rate data, it may analyze the sensing signals obtained by the physiological signal sensing device, use the sensing signals to draw a Poincaré scatter plot, and / or calculate the ratio of low-frequency signal components to high-frequency signal components as the heart rate variation characteristics to quantify sympathetic nerve activity. Sympathetic nerve activity varies in different types of emotions. Based on the correspondence between sympathetic nerve activity and different types of emotions, the emotion category corresponding to the heart rate variation characteristics and the corresponding probability / confidence can be analyzed.
[0043] The signal analysis device may determine the emotion category corresponding to the target object based on the emotion recognition features and the heart rate variability features. Alternatively, the signal analysis device may calculate a conflict coefficient between the emotion recognition features and the heart rate variability features corresponding to different preset emotion categories based on the confidence level of each preset emotion category corresponding to the emotion recognition features and the heart rate variability features; then, calculate the probability values of the preset emotion categories corresponding to the target object based on the conflict coefficients, and determine the emotion category corresponding to the target object based on the probability values. The conflict coefficient is a quantified value indicating whether the emotion categories indicated by the synchronously acquired continuous image frames and heart rate data are different emotion categories.
[0044] Specifically, the process of determining the emotion category corresponding to the target object based on emotion recognition features and heart rate change features can adopt the DS evidence theory. First, establish a result recognition framework (denoted as Θ) to determine the set of all possible results to be processed. In the emotion recognition scenario, it can be an emotion category such as {anger, sadness, happiness, surprise, fear, disgust, neutrality}. Emotion recognition features and heart rate change features are both evidence for the final emotion category recognition result. In subsequent calculations, the conflict coefficient is further calculated based on the basic probability distribution function of each emotion category corresponding to the pre-constructed evidence to measure the degree of conflict between different evidence. Then, evidence synthesis recognition is performed according to the set Dempster synthesis rule to obtain the probability values corresponding to each subset of Θ, and the preset emotion category with the largest probability value is determined as the emotion category corresponding to the target object.
[0045] Furthermore, the signal analysis device can also be used to trigger a preset emotion relief scheme based on the target object's corresponding emotion category. The preset emotion relief scheme includes an information output scheme and / or a physical action scheme by the target robot. For example, the information output can include playing music, playing text content, or providing encouraging words. The physical action scheme can include dancing, hugging, shaking hands, and other actions that can express or relieve emotions in certain emotional states.
[0046] In an optional embodiment, the physiological signal sensing device configured on the robot body is a millimeter wave radar sensor; the operating frequency of the millimeter wave radar is 24GHz or 60GHz, which is used to detect micro-motion signals caused by chest movement of the target object user.
[0047] Correspondingly, the signal analysis device is also used to determine the distance between the robot body and the target object based on the sensing signal from the millimeter radar wave sensor, and to determine the relative positional relationship between the target object and the robot body based on any image frame in the continuous image frames; when the distance and relative positional relationship meet preset distance conditions and preset relative position conditions, the sensing signal from the millimeter radar wave sensor is converted into the heart rate data of the target object. The relative positional relationship between the robot and the target object will affect the sensing result, and changes in the target object's posture will change the relative positional relationship between the robot and the target object. Therefore, the distance between the robot body and the target object can be determined using the sensing signal from the millimeter radar wave sensor, and the angle between the target object and the robot in the captured image frames can be identified through image recognition. The optimal signal acquisition angle is when the target object and the robot are face to face.
[0048] The process of identifying the angle between the target object and the robot in the captured image frames can involve preprocessing the raw image frames to remove noise and mitigate the effects of uneven lighting. Common preprocessing operations include filtering (such as Gaussian filtering to remove noise), grayscaling (converting color images to grayscale to simplify subsequent processing), and histogram equalization (enhancing image contrast). These preprocessing steps improve image quality and provide a better foundation for subsequent object detection and angle calculation. Object detection is then performed, using an object detection algorithm to identify the target object in the preprocessed image. Common object detection algorithms include deep learning-based algorithms, such as Faster R-CNN, the YOLO (You Only Look Once) series, and SSD (Single Shot MultiBox Detector). These algorithms are trained on large amounts of annotated data to learn the characteristic patterns of the target object and accurately detect its location in the image. The output of the object detection algorithm is typically a bounding box (Bounding Box), a rectangular box enclosing the target. Its coordinate information (such as the coordinates of the upper left and lower right corners) is used for subsequent angle calculation. Then, features in the image frame are identified, and characteristic information is extracted for the detected target object. Deep learning methods use the feature extraction layer of a convolutional neural network (CNN) to obtain target features. These features, which include information such as the target's shape and texture, are crucial for accurately identifying the target object's angle. Finally, angle calculation is performed based on the target object's position and posture in the image.
[0049] When the distance and relative position relationship meet the preset distance and relative position conditions, converting the sensing signal from the millimeter radar wave sensor into the target object's heart rate data can improve the validity of the heart rate data and further enhance the accuracy of emotion recognition. If the distance and relative position relationship do not meet the preset distance and relative position conditions, the relative position relationship between the robot and the target object can be adjusted to move the robot closer to or further away from the target object, or the robot can be rotated a certain angle to adjust the angle between the robot and the target object. This can make the heart rate signal collected by the millimeter radar wave sensor more accurate and reduce signal recognition errors. The above distance and angle can be determined through a comprehensive calculation of radar signals and / or image information.
[0050] The technical solution of this embodiment constitutes a health care robot system through a robot body, an image acquisition device and a physiological signal sensing device and a signal analysis device configured on the robot body; wherein, the image acquisition device is used to acquire continuous image frames containing the target object; the physiological signal sensing device is a non-contact heart rate sensing device, used to acquire the heart rate data of the target object; the signal analysis device is used to identify the emotional characteristics of the target object based on the continuous image frames, and identify the heart rate change characteristics of the target object based on the heart rate data, and determine the emotion category corresponding to the target object based on the emotion recognition characteristics and the heart rate change characteristics. The technical solution of the embodiment of the present invention solves the problem of low accuracy in emotion recognition of current health care robots, and can perform emotion analysis based on multimodal information, improve the accuracy of user emotional state analysis, and improve the service quality of health care robots. The technical solution of the embodiment of the present invention solves the problem of low accuracy in emotion recognition of current health care robots, and can improve the accuracy of user emotional state analysis, and improve the service quality of health care robots.
[0051] Figure 2 This is a flowchart of an emotional state recognition method provided by an embodiment of the present invention. This embodiment, which shares the same inventive concept as the healthcare robot system described above, further describes the process by which the healthcare robot implements emotional state recognition. This method can be performed by an emotional state recognition device, which can be implemented using software and / or hardware and integrated into a computer device with application development capabilities.
[0052] like Figure 2 As shown, the emotional state recognition method of this embodiment includes the following steps:
[0053] S110: Acquire continuous image frames containing a target object and heart rate data of the target object.
[0054] The continuous image frames of the target object are image data captured by the health care robot's image acquisition device. These continuous image frames can be image frames captured continuously at a preset frame rate for a preset duration, or they can be image frames selected from image frames captured over multiple consecutive preset durations. The continuous image frames contain temporal information, which can be used in subsequent image recognition to identify the dynamic changes in facial expressions, reducing the probability of misidentification caused by a single image.
[0055] The subject's heart rate data is collected using a non-contact heart rate sensor. This device is non-invasive and comfortable, allowing the subject to monitor their heart rate in a natural and comfortable state. It also offers strong interference resistance, making it suitable for dynamic scenarios without affecting the heart rate sensing results.
[0056] S120. Input the continuous image frames into a pre-trained three-dimensional convolutional image recognition model to obtain the emotional characteristics of the target object and the first confidence level that the emotional characteristics belong to each preset emotional category; and identify the heart rate change characteristics of the target object based on the heart rate data, and determine the second confidence level that the heart rate change characteristics belong to each preset emotional category.
[0057] The three-dimensional convolutional image recognition model may be a pre-trained neural network model capable of recognizing emotional features. The preset emotion categories corresponding to the recognition results may be the first confidence level corresponding to each of the seven categories: anger, sadness, happiness, surprise, fear, disgust, and neutral.
[0058] When identifying a target subject's heart rate variability characteristics based on heart rate data, the sensing signals acquired by the physiological signal sensing device can be analyzed, and a Poincaré scatter plot can be plotted using the sensing signals. The ratio of low-frequency signal components to high-frequency signal components can also be calculated as the heart rate variability characteristic to quantify sympathetic nerve activity. Sympathetic nerve activity varies depending on the emotion. Based on the correspondence between sympathetic nerve activity and different emotion types, the emotion category corresponding to the heart rate variability characteristic and the corresponding second confidence level can be analyzed.
[0059] S130. Calculate conflict coefficients between the emotion recognition feature and the heart rate change feature corresponding to different preset emotion categories based on the first confidence level and the second confidence level.
[0060] The conflict coefficient is a numerical value used in DS evidence theory to measure the degree of conflict between different pieces of evidence. The first and second confidence levels can be two pieces of evidence used to determine the target object's emotional category. The conflict coefficient can be further calculated based on a pre-established basic probability distribution function that assigns evidence to each emotional category to measure the degree of conflict between different pieces of evidence. Evidence synthesis and identification is then performed according to the pre-defined Dempster synthesis rule, resulting in probability values corresponding to each subset of Θ. The preset emotional category with the highest probability value is then determined as the emotional category corresponding to the target object.
[0061] S140 , calculating probability values of preset emotion categories corresponding to the target objects according to the conflict coefficients, and determining the emotion category corresponding to the target objects based on the probability values.
[0062] Assume that the confidence level of emotion category X corresponding to the emotion features of consecutive image frames is A, and the confidence level of emotion category X corresponding to the heart rate feature analysis results is B. Based on the DS evidence theory, calculate the combined confidence level C. If C ≥ a threshold (e.g., 0.8), it can be determined that the target object's emotion category is X, and the corresponding interaction strategy of the health care robot can be triggered (e.g., playing music, initiating a psychological counseling conversation).
[0063] The technical solution of this embodiment is to obtain continuous image frames containing a target object and the heart rate data of the target object; input the continuous image frames into a pre-trained three-dimensional convolutional image recognition model to obtain the emotional characteristics of the target object and the first confidence level that the emotional characteristics belong to each preset emotional category; and identify the heart rate change characteristics of the target object based on the heart rate data, and determine the second confidence level that the heart rate change characteristics belong to each preset emotional category; calculate the conflict coefficients of the emotion recognition characteristics and the heart rate change characteristics corresponding to different preset emotional categories based on the first confidence level and the second confidence level; calculate the probability values of the preset emotional categories corresponding to the target object based on the conflict coefficients, and determine the emotional category corresponding to the target object based on the probability values. The technical solution of the embodiment of the present invention solves the problem of low accuracy in emotion recognition of current health care robots, can improve the accuracy of user emotional state analysis, and enhance the service quality of health care robots.
[0064] Figure 3 This is a structural diagram of an emotional state recognition device provided in an embodiment of the present invention. This embodiment is applicable to situations where emotion recognition is performed based on multimodal information. The emotional state recognition device can be implemented by software and / or hardware and integrated into a computer terminal device with application development capabilities.
[0065] like Figure 3 The emotional state recognition device shown includes: an information acquisition module 210 , an information analysis module 220 and an information analysis result processing module 230 .
[0066] Among them, the information acquisition module 210 is used to obtain continuous image frames containing the target object and the heart rate data of the target object; the information analysis module 220 is used to input the continuous image frames into a pre-trained three-dimensional convolutional image recognition model to obtain the emotional characteristics of the target object and the first confidence that the emotional characteristics belong to each preset emotion category; and identify the heart rate change characteristics of the target object based on the heart rate data, and determine the second confidence that the heart rate change characteristics belong to each preset emotion category; the information analysis result processing module 230 is used to calculate the conflict coefficient of the emotion recognition characteristics and the heart rate change characteristics corresponding to different preset emotion categories based on the first confidence and the second confidence; the information analysis result processing module 230 is also used to calculate the probability value of the preset emotion category corresponding to the target object according to the conflict coefficient, and determine the emotion category corresponding to the target object based on the probability value.
[0067] The technical solution of this embodiment is to obtain continuous image frames containing a target object and the heart rate data of the target object; input the continuous image frames into a pre-trained three-dimensional convolutional image recognition model to obtain the emotional characteristics of the target object and the first confidence level that the emotional characteristics belong to each preset emotional category; and identify the heart rate change characteristics of the target object based on the heart rate data, and determine the second confidence level that the heart rate change characteristics belong to each preset emotional category; calculate the conflict coefficients of the emotion recognition characteristics and the heart rate change characteristics corresponding to different preset emotional categories based on the first confidence level and the second confidence level; calculate the probability values of the preset emotional categories corresponding to the target object based on the conflict coefficients, and determine the emotional category corresponding to the target object based on the probability values. The technical solution of the embodiment of the present invention solves the problem of low accuracy in emotion recognition of current health care robots, can improve the accuracy of user emotional state analysis, and enhance the service quality of health care robots.
[0068] In an optional implementation, the information analysis module 220 may also be used to:
[0069] The distance between the robot body and the target object is determined based on the sensing signal of the physiological signal sensing device, and the relative position relationship between the target object and the robot body is determined based on any image frame in the continuous image frames; when the distance and the relative position relationship meet the preset distance condition and the preset relative position condition, the information acquisition module 210 is used to convert the sensing signal of the millimeter radar wave sensor into the heart rate data of the target object.
[0070] In an optional implementation, the information analysis result processing module 230 may also be used to:
[0071] Trigger the corresponding preset emotion relief plan according to the target object's corresponding emotion category;
[0072] Among them, the preset emotion relief plan includes the information output plan and / or body movement plan of the target robot body.
[0073] The emotional state recognition device provided by the embodiment of the present invention can execute the emotional state recognition method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0074] Figure 4 A schematic structural diagram of a computer device provided in an embodiment of the present invention. Figure 4 A block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 4 The computer device 12 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present invention. The computer device 12 can be any terminal device with computing capabilities, such as an intelligent controller, a server, a mobile phone, or other terminal devices.
[0075] like Figure 4 As shown, computer device 12 is implemented as a general-purpose computing device. Components of computer device 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that connects various system components (including system memory 28 and processing unit 16).
[0076] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.
[0077] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.
[0078] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 4 Not shown, often called a "hard drive"). Although Figure 4Not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.
[0079] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.
[0080] The computer device 12 may also communicate with one or more external devices 14 (e.g., a keyboard, a pointing device, a display 24, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via the bus 18. It should be understood that although Figure 4 Not shown, other hardware and / or software modules may be used in conjunction with computer device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0081] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the emotional state recognition method provided by the embodiment of the present invention, which includes:
[0082] Acquire continuous image frames containing the target object and heart rate data of the target object;
[0083] Inputting continuous image frames into a pre-trained three-dimensional convolutional image recognition model to obtain the target object's emotional characteristics and a first confidence level that the emotional characteristics belong to each preset emotional category; and identifying the target object's heart rate variation characteristics based on the heart rate data, and determining a second confidence level that the heart rate variation characteristics belong to each preset emotional category;
[0084] Calculating conflict coefficients between the emotion recognition feature and the heart rate change feature corresponding to different preset emotion categories based on the first confidence level and the second confidence level;
[0085] The probability values of the preset emotion categories corresponding to the target objects are calculated respectively according to the conflict coefficients, and the emotion categories corresponding to the target objects are determined based on the probability values.
[0086] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method for identifying an emotional state as provided in any embodiment of the present invention is implemented. The method includes:
[0087] Acquire continuous image frames containing the target object and heart rate data of the target object;
[0088] Inputting continuous image frames into a pre-trained three-dimensional convolutional image recognition model to obtain the target object's emotional characteristics and a first confidence level that the emotional characteristics belong to each preset emotional category; and identifying the target object's heart rate variation characteristics based on the heart rate data, and determining a second confidence level that the heart rate variation characteristics belong to each preset emotional category;
[0089] Calculating conflict coefficients between the emotion recognition feature and the heart rate change feature corresponding to different preset emotion categories based on the first confidence level and the second confidence level;
[0090] The probability values of the preset emotion categories corresponding to the target objects are calculated respectively according to the conflict coefficients, and the emotion categories corresponding to the target objects are determined based on the probability values.
[0091] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.
[0092] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0093] Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0094] The computer program code for performing the operations of the present invention can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0095] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computing device. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computer device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
[0096] An embodiment of the present disclosure further provides a computer program product, including a computer program, which, when executed by a processor, implements the emotional state recognition method provided in any embodiment of the present disclosure.
[0097] The computer program product, during implementation, may be written in one or more programming languages, or a combination thereof, for performing the operations of the present disclosure and may include computer program code written in object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0098] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.
Claims
1. A health care robot system, characterized in that: include: A robot body, an image acquisition device and a physiological signal sensing device configured on the robot body, and a signal analysis device; The image acquisition device is used to acquire continuous image frames containing the target object; the physiological signal sensing device is a non-contact heart rate sensing device, which is used to acquire the heart rate data of the target object; The signal analysis device is used to identify the emotional characteristics of the target object based on the continuous image frames, identify the heart rate change characteristics of the target object based on the heart rate data, and determine the emotion category corresponding to the target object based on the emotion recognition characteristics and the heart rate change characteristics.
2. The system according to claim 1, wherein: The signal analysis device is used for: Inputting the continuous image frames into a pre-trained three-dimensional convolutional image recognition model to obtain the emotional features of the target object and the confidence that the emotional features belong to each preset emotional category; The three-dimensional convolutional image recognition model is provided with a self-attention mechanism, and the training samples of the three-dimensional convolutional image recognition model include image sequences corresponding to different emotion categories.
3. The system according to claim 2, characterized in that The signal analysis device is used for: Calculating, based on the confidence levels of the preset emotion categories corresponding to the emotion recognition feature and the heart rate change feature, conflict coefficients between the emotion recognition feature and the heart rate change feature corresponding to different preset emotion categories; The probability values of the preset emotion categories corresponding to the target objects are calculated respectively according to the conflict coefficients, and the emotion category corresponding to the target objects is determined based on the probability values.
4. The system according to claim 1, wherein: The signal analysis device is used to trigger a corresponding preset emotion relief scheme according to the emotion category corresponding to the target object; The preset emotion relief scheme includes an information output scheme and / or a body movement scheme of the target robot body.
5. The system according to claim 1, wherein: The image acquisition device includes: cameras arranged at at least two positions of the robot body, namely, the eyes, forehead, palms and chest, and the frame rate of the cameras is greater than 30 frames per second.
6. The system according to claim 1, wherein: The physiological signal sensing device is a millimeter radar wave sensor; Correspondingly, the signal analysis device is also used to determine the distance between the robot body and the target object based on the sensing signal of the millimeter radar wave sensor, and determine the relative position relationship between the target object and the robot body based on any image frame in the continuous image frames; when the distance and the relative position relationship meet the preset distance condition and the preset relative position condition, the sensing signal of the millimeter radar wave sensor is converted into the heart rate data of the target object.
7. A method for identifying an emotional state, characterized in that: include: Acquiring continuous image frames containing a target object and heart rate data of the target object; Inputting the continuous image frames into a pre-trained three-dimensional convolutional image recognition model to obtain the emotional characteristics of the target object and a first confidence level that the emotional characteristics belong to each preset emotional category; and identifying the heart rate variation characteristics of the target object based on the heart rate data, and determining a second confidence level that the heart rate variation characteristics belong to each preset emotional category; Calculating conflict coefficients between the emotion recognition feature and the heart rate change feature corresponding to different preset emotion categories based on the first confidence level and the second confidence level; The probability values of the preset emotion categories corresponding to the target objects are calculated respectively according to the conflict coefficients, and the emotion category corresponding to the target objects is determined based on the probability values.
8. A computer device, characterized in that: The computer device comprises: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the emotional state recognition method according to claim 7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the emotional state recognition method according to claim 7 is implemented.
10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the emotional state recognition method according to claim 7.