An old person health monitoring method and system based on expression emotion calculation

Abstract

The application discloses a kind of old person health monitoring method and system based on expression emotion calculation in the field of computer vision, method includes real-time collection old person's face image;The face image collected is input into the old person health monitoring model based on expression emotion calculation constructed, and the physiological health signal and the psychological health signal of old person based on face image are output.Based on the old person health monitoring model based on expression emotion calculation, the key expression area is positioned based on the proposed regional navigation multi-task training module, face expression and face posture are jointly trained, expression feature extraction is completed, spatiotemporal motion enhancement network is trained using the extracted expression feature, multi-pose and dynamic face expression recognition is completed, and the expression information obtained is converted into physiological health signal and psychological health signal.The application can identify the expression information of old person facial posture real-time change in natural scene, improve recognition accuracy, and further monitor the health signal of old person.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method and system for health monitoring of the elderly based on facial expression and emotion calculation. Background Technology

[0002] Facial expressions are the most direct and effective mode of emotion recognition, and they are widely used in robotics, healthcare, and human-computer interaction systems. With the increasing aging population, classifying and recognizing the facial expressions of the elderly to assist their daily lives is receiving increasing attention. However, current facial expression recognition methods still have some problems, especially in cases involving multiple poses and dynamic expressions, where recognition accuracy is relatively low. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a method and system for health monitoring of the elderly based on facial expression and emotion calculation, which can improve recognition accuracy in the case of multiple postures and dynamics of facial expressions.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] Firstly, a method for monitoring the health of the elderly based on facial expression and emotion computing is provided, including: real-time acquisition of facial images of the elderly; inputting the acquired facial images into a constructed health monitoring model for the elderly based on facial expression and emotion computing, and outputting physiological and psychological health signals of the elderly based on the facial images.

[0006] Furthermore, the elderly health monitoring model based on facial expression and emotion computing first locates key expression regions of facial images using the proposed regional navigation multi-task training module, and jointly trains facial expressions and facial poses to complete expression feature extraction; then, it uses the extracted expression features to train a spatiotemporal motion enhancement network to complete multi-pose and dynamic facial expression recognition; finally, it converts the obtained expression information into physiological health signals and psychological health signals.

[0007] Furthermore, based on the proposed region navigation multi-task training module, key expression regions of facial images are located, and facial expressions and facial poses are jointly trained to complete expression feature extraction, including: using a region navigation network to locate key facial expression regions, and introducing an attention mechanism to give high weight to features of regions rich in pose and expression information, thus highlighting the features of key regions.

[0008]

[0009] Where F represents the features of the prominent key region; α i f represents the learnable attention weight of the i-th element; iLet represent the facial expression region features of the i-th person; n represents the number of facial expression region features.

[0010] A multi-task learning method is used to jointly train facial expressions and facial poses to complete the extraction of facial expression features.

[0011] Furthermore, the region navigation network shares convolutional layers with the classification network for feature extraction and uses an anchoring mechanism to simultaneously predict multiple region proposals {R'1,R'2,...,R'}. A Each anchor point is related to the position, aspect ratio, and frame ratio of the sliding window; find the region with the richest expression information, and the information in the region with the richest expression information satisfies:

[0012] for any R1,R2∈A,if C(R1)>C(R2),I(R1)>I(R2) (6)

[0013] Where R1 and R2 are any two rectangular regions in A regions, I represents the amount of facial expression information in the rectangular region, and C represents the probability that the rectangular region is the correct facial expression;

[0014] To ensure the final result satisfies the above relationship, M regions are selected from A regions. For each region R... i ∈A M The regional navigation network will evaluate its information I, and then the regional R i The data is transmitted to a regional navigation network for learning and to calculate the region R. i The probability C of correctly classifying facial expressions; to optimize the region navigation network so that {I(R1), I(R2), ..., I(R... M {C(R1),C(R2),...,C(R)} and {C(R)} M )} have the same order.

[0015] Furthermore, the spatiotemporal motion enhancement network is an improved residual neural network obtained by considering the time scale and improving the residual neural network after transfer learning training;

[0016] The spatiotemporal motion enhancement network uses the appearance and motion information of historical frames to remove noise from the current frame, fill in the missing motion information in the current frame, and add facial expression motion information to the current frame.

[0017] Furthermore, the spatiotemporal motion enhancement network is based on the classic ResNet network, including one convolutional layer and eight residual blocks, each residual block having two convolutional layers; namely, convolutional layer 1, residual block 1, residual block 2, residual block 3, residual block 4, residual block 5, residual block 6, residual block 7, and residual block 8. Each of residual blocks 1, 2, 3, 4, 5, 6, 7, and 8 consists of two cascaded convolutional layers. The spatiotemporal motion enhancement network also includes pooling layer 1, pooling layer 2, and fully connected layer 1.

[0018] Furthermore, the spatiotemporal motion enhancement network performs a Dropout operation after the fully connected layer 1 to improve generalization ability.

[0019] Furthermore, convolutional layer 1 is used for the first layer of feature extraction of the expression image, using a 7×7 convolutional kernel. Residual edges are introduced in residual blocks 1, 2, 3, 4, 5, 6, 7, and 8 to improve training performance by increasing the number of layers while avoiding the gradient vanishing problem. These convolutional kernels are 3×3. Pooling layers are used to reduce the size of the parameter matrix, with pooling layer 1 using max pooling and pooling layer 2 using average pooling. Fully connected layers are used to map features to the sample label space. The spatiotemporal motion enhancement network uses the Leaky ReLU activation function.

[0020] Furthermore, according to medical and health judgment standards, after obtaining facial expression information, it is converted into physiological health signal ph and mental health signal ps.

[0021] Secondly, a health monitoring system for the elderly based on facial expression and emotion computing is provided, including: an image acquisition module for real-time acquisition of facial images of the elderly; and a monitoring module for inputting the acquired facial images into a health monitoring model for the elderly based on facial expression and emotion computing, and outputting physiological and psychological health signals of the elderly based on the facial images.

[0022] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0023] (1) This invention collects facial images of the elderly in real time, inputs the collected facial images into a health monitoring model for the elderly based on facial expression and emotion calculation, and outputs physiological health signals and psychological health signals of the elderly based on facial images. It can improve the recognition accuracy in the case of multiple postures and dynamic facial expressions.

[0024] (2) The regional navigation multi-task training module used in this invention is used to locate key facial expression areas, jointly train facial expressions and postures, and improve the ability to learn facial expression features and posture features.

[0025] (3) The spatiotemporal motion enhancement network used in this invention improves the generalization ability of the model and at the same time considers the spatiotemporal relationship between images to complete the dynamic information of static images.

[0026] (4) The present invention uses the Leaky ReLU activation function in the convolutional layer to improve the convergence speed; and performs Dropout operation after the fully connected layer to improve the generalization ability and reduce overfitting. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the main process of a health monitoring method for the elderly based on facial expression and emotion calculation provided in an embodiment of the present invention;

[0028] Figure 2 This is a structural diagram of the regional navigation multi-task training module in an embodiment of the present invention;

[0029] Figure 3 This is a diagram of the spatiotemporal motion enhancement network structure in an embodiment of the present invention;

[0030] Figure 4 This is an application scenario for real-time monitoring of the health of the elderly using the method described in the embodiments of the present invention. Detailed Implementation

[0031] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0032] Example 1:

[0033] like Figures 1-4 As shown, a method for monitoring the health of the elderly based on facial expression and emotion computing includes: real-time acquisition of facial images of the elderly; inputting the acquired facial images into a constructed elderly health monitoring model based on facial expression and emotion computing, and outputting physiological and psychological health signals of the elderly based on the facial images. The elderly health monitoring model based on facial expression and emotion computing first locates key expression regions of the facial images using a proposed regional navigation multi-task training module, jointly trains facial expressions and facial poses, and completes expression feature extraction; then, it uses the extracted expression features to train a spatiotemporal motion enhancement network to complete multi-pose and dynamic facial expression recognition; finally, it converts the obtained expression information into physiological and psychological health signals.

[0034] like Figure 1 As shown, firstly, a face sample dataset is constructed. Then, a region navigation multi-task training module is used to locate key expression regions. Face expressions and poses are jointly trained to extract expression features. Finally, the extracted expression features are used to train a spatiotemporal motion augmentation network to achieve multi-pose and dynamic face expression recognition. Specifically, the following steps are included:

[0035] S1, Construct an image dataset of sample faces.

[0036] We obtain facial expression image datasets from existing facial expression databases and perform dataset preprocessing, specifically including the following steps:

[0037] Obtain sample face datasets such as LFPW and AFW provided by the ibug website.

[0038] S2 proposes a regional navigation multi-task training module to locate key facial expression regions, jointly train facial expressions and facial poses, and complete facial expression feature extraction.

[0039] S21, utilize a regional navigation network to locate key regions of facial expressions, and introduce an attention mechanism to weight global and local features, highlighting the features of key regions. Specifically, this includes the following steps:

[0040] First, input a face image X, then use a navigation network to generate local expression regions: {R'1,R'2,...,R' A} and its information content {I'1,I'2,...,I' A};

[0041] Nonmaximum suppression is used to remove information redundancy and reset the information content:

[0042]

[0043] Among them, R′ t Representing the t-th rectangular region, I′ t Let A represent the amount of facial expression information in the t-th rectangular region, and let A represent the number of rectangular regions.

[0044] Select the M regions with the highest confidence levels: Learn to calculate its confidence level:

[0045] {C1,C2,...,C M}=C({R1,R2,...,R M}) (2)

[0046] Among them, R i C represents the i-th rectangular region. i This represents the probability that the i-th rectangular region contains the correct expression;

[0047] Feature extraction is performed on the entire emoji image and its key regions. The extracted features are then subjected to a Squeeze operation to encode them into a preliminary feature set. And it maintains the receptive field of the features unchanged. It can be represented as:

[0048]

[0049] Where s is the Squeeze operation, F i The features are extracted after passing through a shared convolutional layer, where H and W are the spatial dimensions of the feature F. i (i, j) represents the feature whose spatial dimension coordinates are (i, j);

[0050] Then, an FC layer is used to estimate the attention weights for each region feature, and the weights are passed to a Sigmoid function, setting the weights between 0 and 1. The weight for each feature can be defined as follows:

[0051] α M =σ(W F ×F' i (4)

[0052] Among them, W F Here are the parameters of the fully connected layer, and σ is the Sigmoid function. Indicates preliminary characteristics.

[0053] Finally, the attention weight is compared with the f of the first branch. i After weighting, the final aggregated feature F is as follows:

[0054]

[0055] Where α i f represents the learnable attention weight of the i-th element; i Let represent the facial expression region features of the i-th person; n represents the number of facial expression region features.

[0056] S22 utilizes a multi-task learning method to jointly train facial expression recognition and pose recognition.

[0057] By using a multi-task learning approach, we can jointly train two tasks: facial expression recognition and pose recognition, so that the features can learn more useful information from both tasks.

[0058] First, a regional navigation network is proposed, which uses an anchor point mechanism to simultaneously predict multiple regional suggestions {R'1,R'2,...,R'}. A Each anchor point is related to the position, aspect ratio, and frame ratio of the sliding window. Find the area with the richest expression information; the information in this area should satisfy the following:

[0059] for any R1,R2∈A,if C(R1)>C(R2),I(R1)>I(R2) (6)

[0060] Where R1 and R2 are any two rectangular regions in A regions, I represents the amount of facial expression information in the rectangular region, and C represents the probability that the rectangular region is the correct facial expression;

[0061] To ensure the final result satisfies the above relationship, M regions are selected from A regions. For each region R... i ∈A M The regional navigation network will evaluate its information I, and then the regional R i The data is transmitted to a regional navigation network for learning and to calculate the region R. i The probability C of correctly classifying facial expressions; to optimize the region navigation network so that {I(R1), I(R2), ..., I(R... M {C(R1),C(R2),...,C(R)} and {C(R)} M )} have the same order.

[0062] S3 utilizes extracted facial expression features to train a spatiotemporal motion enhancement network, enabling multi-pose and dynamic facial expression recognition. It considers the time scale to improve the residual neural network trained by transfer learning, using the appearance and motion information of historical frames to remove noise from the current frame and fill in the missing motion information in the current frame, thus increasing the facial expression motion information in the current frame.

[0063] In dynamic facial expressions, the motion region of a face is roughly the difference between the motion of the previous frame and the current frame. By analyzing the motion region, we can discover local motion details of the face and improve the quality of frame-level expression features. We use the feature difference between consecutive frames to model the motion region features.

[0064] The MS-Celeb-1M dataset is a large database containing 85,000 faces (3.8 million faces), primarily used for face verification. Since facial expressions are one of the attributes of a face, the MS-Celeb-1M dataset is more suitable for expression recognition tasks compared to datasets like ImageNet. Furthermore, to make the model more compatible with video facial expression recognition tasks, transfer learning was performed on the model using both the MS-Celeb-1M face recognition dataset and the FER2013 static face expression recognition dataset.

[0065] As described in S3, considering the time scale, this embodiment uses the original feature difference X between the upper and lower frames. i,k,t -X i,k,t-1 The feature difference X between the refined features of the previous frame and the features of the current frame i,k,t -S i,k,t-1 This will improve the residual neural network trained by transfer learning.

[0066] The spatiotemporal motion enhancement network is based on the classic ResNet network, consisting of one convolutional layer, eight residual blocks (each residual block has two convolutional layers), namely Convolutional Layer 1, Residual Block 1, Residual Block 2, Residual Block 3, Residual Block 4, Residual Block 5, Residual Block 6, Residual Block 7, Residual Block 8, Pooling Layer 1, Pooling Layer 2, and Fully Connected Layer 1. Residual Blocks 1, 2, 3, 4, 5, 6, 7, and 8 are each composed of two cascaded convolutional layers. The specific structure of the residual neural network after transfer learning training is as follows... Figure 2 As shown.

[0067] Convolutional and pooling layers extract and filter information. The convolutional layer performs the first feature extraction on the expression image, using a 7×7 kernel, a stride of 2, and padding of 3. Each of the residual blocks (1, 2, 3, 4, 5, 6, 7, and 8) includes two stacked convolutional layers and introduces residual edges. This allows for a relatively stable improvement in training performance by increasing the number of layers, while also avoiding the vanishing gradient problem. The concatenation of two 3×3 convolutional layers is equivalent to one 5×5 convolutional layer, but with far fewer parameters than a 5×5 convolutional layer, reducing the overall network training time. Pooling layer one uses a 3×3 kernel, while pooling layer two uses adaptive pooling, outputting a 512×1×1 value. Pooling layers reduce the size of the parameter matrix; pooling layer one uses max pooling, and pooling layer two uses average pooling. Fully connected layers map features to the sample label space.

[0068] Dropout is performed after the first fully connected layer to improve generalization. LeakyReLU is chosen as the activation function.

[0069]

[0070] Where, x i Let y represent the feature of the i-th region after convolution. i This represents the feature of the i-th region obtained after nonlinear mapping;

[0071] The LeakyReLU function converges faster than the traditional ReLU function.

[0072] S4, after obtaining facial expression information through neural network training, converts it into a physiological health signal (ph) and a psychological health signal (ps) according to medical health judgment standards. By setting health thresholds and comparing it with the standards, it is further divided into four health signals: positive physiological state (ph+), negative physiological state (ph-), positive psychological state (ps+), and negative psychological state (ps-). (Specific standards: The default health values ​​are set to ph+ and ps+. A comprehensive judgment is made on the facial expression information sampled within 6 seconds. If the elderly person's painful expressions account for 75% of the total expressions, it is determined that the elderly person may be experiencing physiological pain, and the physiological health signal is changed to ph-; if more than 80% of the facial expression signals sampled over fifteen consecutive days are sad, it is determined that the elderly person has been experiencing some depression during this period, and the psychological health signal is changed to ps-).

[0073] Example 2:

[0074] Based on the elderly health monitoring method based on facial expression and emotion calculation described in Embodiment 1, this embodiment provides an elderly health monitoring system based on facial expression and emotion calculation, including: an image acquisition module for real-time acquisition of facial images of the elderly; and a monitoring module for inputting the acquired facial images into a constructed elderly health monitoring model based on facial expression and emotion calculation, and outputting physiological and psychological health signals of the elderly based on the facial images.

[0075] Embodiments of this application may be provided as methods, systems, or computer program products. Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application may be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0076] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0078] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0079] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0080] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for health monitoring of the elderly based on facial expression and emotion calculation, characterized in that, include: Real-time acquisition of facial images of elderly people; The collected facial images are input into the constructed elderly health monitoring model based on facial expression and emotion computing, and the model outputs physiological and psychological health signals of the elderly based on facial images. The elderly health monitoring model based on facial expression and emotion computing first locates key expression regions in a face image using a proposed regional navigation multi-task training module, and then jointly trains facial expressions and facial poses to complete expression feature extraction. Next, it uses the extracted expression features to train a spatiotemporal motion enhancement network to complete multi-pose and dynamic facial expression recognition. Finally, it converts the obtained expression information into physiological health signals and psychological health signals. Specifically, based on the proposed regional navigation multi-task training module, key expression regions of facial images are located, and facial expressions and facial poses are jointly trained to complete expression feature extraction, including: Using a region navigation network to locate key facial expression regions, an attention mechanism is introduced to give high weight to features rich in pose and expression information, thus highlighting the features of key regions. （5） in, Indicates the characteristics of prominent key areas; This represents the i-th learnable attention weight; Represents the facial expression region features of the i-th person; Indicates the number of facial expression region features; A multi-task learning method is used to jointly train facial expressions and facial poses to complete the extraction of facial expression features.

2. The method for health monitoring of the elderly based on facial expression and emotion calculation according to claim 1, characterized in that, The region navigation network shares convolutional layers with the classification network for feature extraction and uses an anchoring mechanism to simultaneously predict multiple region proposals. Each anchor point is related to the position, aspect ratio, and frame ratio of the sliding window; find the region with the richest expression information, and the information in the region with the richest expression information satisfies: （6） in, for Any two rectangular regions within a given region, This represents the amount of facial expression information within a rectangular region. This represents the probability that the rectangular area represents the correct facial expression. To ensure the final result satisfies the above relationship, select from region A. Each region Regional navigation networks will evaluate their information. Then the area Transmitted to the regional navigation network for learning and calculation of the region The probability of correctly classifying facial expressions To optimize the regional navigation network and They have the same order.

3. The method for health monitoring of the elderly based on facial expression and emotion calculation according to claim 1, characterized in that, The spatiotemporal motion enhancement network is an improved residual neural network obtained by considering the time scale and improving the residual neural network after transfer learning training. The spatiotemporal motion enhancement network uses the appearance and motion information of historical frames to remove noise from the current frame, fill in the missing motion information in the current frame, and add facial expression motion information to the current frame.

4. The method for health monitoring of the elderly based on facial expression and emotion calculation according to claim 3, characterized in that, The spatiotemporal motion enhancement network is based on the classic ResNet network and includes one convolutional layer and eight residual blocks. Each residual block has two convolutional layers: convolutional layer 1, residual block 1, residual block 2, residual block 3, residual block 4, residual block 5, residual block 6, residual block 7, and residual block 8. Each of residual blocks 1, 2, 3, 4, 5, 6, 7, and 8 consists of two cascaded convolutional layers. The spatiotemporal motion enhancement network also includes pooling layer 1, pooling layer 2, and fully connected layer 1.

5. The method for health monitoring of the elderly based on facial expression and emotion calculation according to claim 4, characterized in that, The spatiotemporal motion enhancement network performs a Dropout operation after the fully connected layer 1 to improve generalization ability.

6. The method for health monitoring of the elderly based on facial expression and emotion calculation according to claim 4, characterized in that, Convolutional layer 1 is used for the first layer of feature extraction of the facial expression image, using a 7×7 convolutional kernel. Residual edges are introduced in residual blocks 1, 2, 3, 4, 5, 6, 7, and 8 to improve training performance by increasing the number of layers and avoid the gradient vanishing problem. These convolutional layers use a 3×3 kernel. Pooling layers are used to reduce the size of the parameter matrix. Pooling layer 1 uses max pooling, and pooling layer 2 uses average pooling. Fully connected layers are used to map features to the sample label space. The spatiotemporal motion enhancement network uses the Leaky ReLU activation function.

7. The method for health monitoring of the elderly based on facial expression and emotion calculation according to claim 1, characterized in that, According to medical and health assessment standards, after obtaining facial expression information, it is converted into physiological health signal ph and mental health signal ps.

8. A health monitoring system for the elderly based on facial expression and emotion calculation, characterized in that, include: The image acquisition module is used to acquire facial images of elderly people in real time. The monitoring module is used to input the collected facial images into the constructed elderly health monitoring model based on facial expression and emotion calculation, and output the physiological and psychological health signals of the elderly based on the facial images. The elderly health monitoring model based on facial expression and emotion computing first locates key expression regions in a face image using a proposed regional navigation multi-task training module, and then jointly trains facial expressions and facial poses to complete expression feature extraction. Next, it uses the extracted expression features to train a spatiotemporal motion enhancement network to complete multi-pose and dynamic facial expression recognition. Finally, it converts the obtained expression information into physiological health signals and psychological health signals. Specifically, based on the proposed regional navigation multi-task training module, key expression regions of facial images are located, and facial expressions and facial poses are jointly trained to complete expression feature extraction, including: Using a region navigation network to locate key facial expression regions, an attention mechanism is introduced to give high weight to features rich in pose and expression information, thus highlighting the features of key regions. （5） in, Indicates the characteristics of prominent key areas; This represents the i-th learnable attention weight; Represents the facial expression region features of the i-th person; Indicates the number of facial expression region features; A multi-task learning method is used to jointly train facial expressions and facial poses to complete the extraction of facial expression features.

Images