Real-time assessment method and system for the physical and mental state of the elderly based on facial expression recognition

Abstract

The present invention discloses a real-time assessment method and system for the physical and mental state of the elderly based on facial expression recognition. The method comprises: constructing a facial expression recognition network framework based on an attention module, the facial expression recognition network framework based on the attention module comprising an image sequence, a three-dimensional convolution Conv3D, a three-dimensional residual block 3D Residual Block, a fully connected layer FC, a Softmax classifier based on cross-entropy loss, and an efficient attention module EAM Block; constructing a training data set according to user needs; inputting the constructed training data set into a facial expression recognition model to train the facial expression recognition model; inputting a video of the elderly's facial expression to be processed into the trained facial expression recognition model to determine the elderly's emotion or physical and mental state. The present invention constructs an efficient attention module EAM for facial emotion recognition, models the information features of different channels and video frames, improves the network's attention to important information, and more accurately and efficiently recognizes the elderly's physical and mental state based on the elderly's facial expression video.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, image processing, emotion recognition, and video classification, and in particular to a real-time assessment method and system for the physical and mental state of the elderly based on facial expression recognition. Background Art

[0002] Facial expressions are one of the non-verbal ways people understand a person's emotions and mental states. The ability to identify physical and mental states through facial expressions is crucial for health monitoring. Traditionally, video-based emotion recognition is a type of video classification, categorizing emotions or the physical and mental states of the elderly into discrete categories, such as happiness, anger, sadness, and fear. In the past few years, most approaches have used either manual or learning-based feature extraction methods to identify these six basic physical and mental states.

[0003] Research has shown that facial expressions change with age, but the literature lacks systematic research on facial expression recognition methods for older adults. Over the next 20 years, the aging population will lead to profound societal changes, necessitating more innovative services for seniors. Many potential applications, such as robotics, communications, security, and healthcare, would benefit from technologies that automatically recognize facial expressions in the elderly. Different facial expressions can reflect emotions and, in turn, reveal the subject's psychological activities, enabling better analysis of the elderly's physical and mental state.

[0004] The rapid development of deep learning technology has led to the emergence of numerous deep learning-based facial expression recognition methods. Many deep learning-based facial expression recognition methods employ image datasets. However, images lack temporal context and are not as informative as videos, which contain contextual information. Consequently, deep learning-based classifiers that use video sequences as input are becoming increasingly important for facial emotion recognition.

[0005] One existing technique, "A Method for Facial Expression Recognition from Videos Based on Convolutional Neural Networks and Recurrent Neural Networks," combines a convolutional neural network (CNN) with a recurrent neural network (RNN) to perform facial expression recognition on video data. However, this method lacks a mechanism to implicitly account for prominent facial features. Specifically, the 300-dimensional feature vector extracted by the CNN is unable to represent the rich, detailed features of the image.

[0006] The second existing technology, "A Method for Facial Expression Recognition in Videos Based on Three-Dimensional Convolutional Neural Networks," utilizes a three-dimensional convolutional neural network (3D CNN) instead of the commonly used two-dimensional convolutional neural network to recognize facial emotions in videos, adding a temporal dimension to capture temporal information. However, the drawback of this method is that the feature representations learned by 3D CNN-based methods are still not discriminative for action recognition. For example, there is no significant attention mechanism to focus on extracting spatial regions or video frames related to the action to be recognized. Summary of the Invention

[0007] The present invention aims to overcome the shortcomings of existing methods by proposing a real-time method and system for assessing the physical and mental state of the elderly based on facial expression recognition. The primary problem addressed by this invention is that, because elderly people have more wrinkles and folds, their expressions are more difficult to interpret, making it difficult for simple 3D CNN methods to capture information about facial structure and expression changes. Specifically, the challenge is to better capture this information, generating more distinctive features that allow the 3D convolutional neural network to focus on important features.

[0008] In order to solve the above problems, the present invention proposes a real-time assessment method for the physical and mental state of the elderly based on facial expression recognition, the method comprising:

[0009] Building a facial expression recognition network framework based on the attention module, the facial expression recognition network framework based on the attention module includes an image sequence, a three-dimensional convolution Conv3D, a three-dimensional residual block 3D Residual Block, a fully connected layer FC, a Softmax classifier based on cross entropy loss Softmax, and an efficient attention module EAM Block;

[0010] Construct a training dataset based on user needs. After collecting a large number of videos of elderly people's facial expressions, label them with the correct labels of anger, disgust, fear, happiness, sadness, or surprise to construct a training dataset.

[0011] Input the constructed training data set into the facial expression recognition network based on the attention module to obtain estimated emotion labels, then use the constructed cross entropy loss function to calculate the loss between the estimated emotion labels and the correct emotion labels of the video, and then reversely update the parameters of the network model. Repeat this step multiple times to obtain a trained facial expression recognition model;

[0012] The video of the facial expression of the elderly to be processed is input into the trained facial expression recognition model to obtain an output emotion label, and the emotion or physical and mental state of the elderly is determined according to the label.

[0013] Preferably, the facial expression recognition network framework based on the attention module is constructed, and the facial expression recognition network framework based on the attention module includes an image sequence, a three-dimensional convolution Conv3D, a three-dimensional residual block 3D ResidualBlock, a fully connected layer FC, a Softmax classifier based on cross entropy loss Softmax, and an efficient attention module EAMBlock, specifically:

[0014] The input of the attention-based facial expression recognition network framework is an image sequence I∈R H×W×T×3 , that is, a video containing the facial expressions of the elderly, where T represents the number of frames in the video, 3 represents the number of channels in one frame of the video, and the output is the emotion label, which is set to six labels: anger, disgust, fear, happiness, sadness, and surprise;

[0015] The three-dimensional convolution operation is designed to extract shallow features. The shallow features are dynamic features, which are video features containing a time dimension. They are used to represent facial elements, muscle movements, and other information in facial expression videos. The extracted features are recorded as C out The number of output channels is 32, followed by three 3D residual blocks and three EAM attention modules. Finally, after the operation of the fully connected layer and the calculation of the Softmax layer, the emotion label recognized by the network is obtained.

[0016] The three-dimensional residual block structure is designed. The structure includes two convolutional layer operations. After each convolutional layer operation, batch normalization BatchNormal and ReLU nonlinear activation function are performed. The structure is set as the i-th (i=1, 2, 3) three-dimensional residual block structure, and its input is the feature V extracted by the previous layer EAM module or convolution. i-1 , the output features By introducing the EAM attention module, the residual network architecture improves image classification performance by increasing the network depth, thereby alleviating the problem of relatively small training datasets;

[0017] Design the efficient attention module EAM Block, which includes the channel efficient attention module EAM-S and the time efficient attention module EAM-T;

[0018] The fully connected layer and Softmax classifier are designed. After processing by the Softmax classifier, a six-dimensional vector is obtained. The six dimensions represent six different emotions. Each element of the vector is a probability value, indicating the probability that the elderly person is in the corresponding mood. The element with the largest probability is the elderly person's current main emotion.

[0019] Preferably, the design of the efficient attention module EAM Block includes a channel efficient attention module EAM-S and a time efficient attention module EAM-T, specifically:

[0020] The input of the EAM module is the output of the 3D residual module in the same layer set up The dimensions are H×W×T×C, and the output is V i In the EAM module, the EAM-S module is responsible for learning the attention weights M of different channels c ∈R 1×1×1×C , the EAM-T module is responsible for learning the attention weights M of different frames T ∈R 1×1×1×T , the calculation formula of the working process of the EAM module is as follows:

[0021]

[0022]

[0023]

[0024] in, Indicates element-wise multiplication, and Trans indicates the conversion of matrix dimensions, from H×W×T×C to H×W×C×T or from H×W×C×T to H×W×T×C.

[0025] The main task of the EAM-S module is to calculate the attention weights M of different channels c , separate the input features by channel where v c ∈R H×W×T , perform 3D global average pooling (3D-GAP) and 3D global max pooling (3D-GMP) operations on the input features to obtain a c ∈R 1×1×1×C and m c ∈R 1×1×1×C Two different channel descriptions, the calculation formula is as follows:

[0026]

[0027]

[0028] a=1,…,H

[0029] b=1,…,W

[0030] t=1,…,T

[0031] Where c∈[1,2,…,C], H, W and T represent the length, width and number of frames;

[0032] The results of the two pooling operations are combined by element-wise summation to obtain the final channel feature descriptor. After that, the final channel feature descriptor is subjected to a one-dimensional convolution operation. Finally, the vector after the convolution operation is put into the sigmoid function to obtain the attention weights M of different channels. c , the obtained channel attention weight M c and input features Calculate the channel-weighted feature V i ′, the specific calculation formula is as follows:

[0033]

[0034] in, Indicates element-wise multiplication, and Trans indicates the conversion of matrix dimensions, from H×W×T×C to H×W×C×T or from H×W×C×T to H×W×T×C.

[0035] The goal of the EAM-T module is to calculate the attention weights M of different frames T , after the Transpose operation before the EAM-T module, the feature V i The dimension of ′ is converted from H×W×T×C to H×W×C×T, and 3D-GAP and 3D-GMP operations are performed on the input features to obtain a t ∈R 1×1×1×T and m t ∈R 1×1×1×T Two different frame descriptions, the calculation formula is as follows:

[0036]

[0037] m t =3DGMP(V′ i )=max{V′ i (a,b,k)},

[0038] a=1,…,H

[0039] b=1,…,W

[0040] K=1,…,C

[0041] Where t∈[1,2,…,T], H, W and C represent the length, width and number of channels;

[0042] The results of the two pooling operations are combined by element-wise summation to obtain the final frame feature descriptor. After that, the final frame feature descriptor is subjected to a one-dimensional convolution operation. Finally, the vector after the convolution operation is put into the sigmoid function to obtain the attention weights M of different frames. T , the obtained frame attention weight M T And the input feature V′ t Calculate the weighted feature V by frame i ″, the calculation formula is as follows:

[0043]

[0044] in, Indicates multiplication by corresponding elements, Trans indicates the conversion operation of matrix dimension, from H×W×T×C dimension to H×W×C×T or from H×W×C×T to H×W×T×C,

[0045] A Transpose operation is performed after the EAM-T module to transform the feature V i The dimension of ″ is converted from H×W×C×T to H×W×T×C, and V i ″ and input features Add up to get the output feature V of the EAM module i .

[0046] Accordingly, the present invention also provides a real-time assessment system for the physical and mental state of the elderly based on facial expression recognition, comprising:

[0047] A facial expression recognition network framework unit based on an attention module is used to build a facial expression recognition network framework based on an attention module. The facial expression recognition network framework based on the attention module includes an image sequence, a three-dimensional convolution Conv3D, a three-dimensional residual block 3D Residual Block, a fully connected layer FC, a Softmax classifier based on cross entropy loss Softmax, and an efficient attention module EAM Block;

[0048] The training dataset construction unit is used to construct a training dataset based on user needs. After collecting a large number of videos of elderly people's facial expressions, they are labeled with the correct labels of anger, disgust, fear, happiness, sadness, or surprise to construct a training dataset.

[0049] A facial expression recognition model training unit is used to input the constructed training data set into the facial expression recognition network based on the attention module to obtain an estimated emotion label, then use the constructed cross entropy loss function to calculate the loss between the estimated emotion label and the correct emotion label of the video, and then reversely update the parameters of the network model, and repeat this step multiple times to obtain a trained facial expression recognition model;

[0050] The real-time assessment unit for the physical and mental state of the elderly is used to input the video of the facial expression of the elderly to be processed into the trained facial expression recognition model, obtain the output emotion label, and determine the emotion or physical and mental state of the elderly according to the label.

[0051] The implementation of the present invention has the following beneficial effects:

[0052] The present invention proposes a solution for emotion recognition in facial expression videos using a three-dimensional convolutional neural network based on a time and channel attention mechanism. The solution constructs an efficient attention module EAM for facial emotion recognition. The EAM module includes a channel EAM-S module and an EAM-T module, which can learn the weights of different channels and different frames, and then assign more weights to important features, so that the neural network focuses more attention on important channel information and important video frames; the information features of channels and different video frames are modeled by the EAM module, which improves the network's attention to important information, and more accurately and efficiently recognizes the physical and mental state of the elderly based on their facial expression videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is an overall flow chart of a method for real-time assessment of the physical and mental state of the elderly based on facial expression recognition according to an embodiment of the present invention;

[0054] Figure 2 1 is a diagram of a facial expression recognition network structure based on an attention module according to an embodiment of the present invention;

[0055] Figure 3 3D residual block structure diagram according to an embodiment of the present invention;

[0056] Figure 4 is a structural diagram of an efficient attention module according to an embodiment of the present invention;

[0057] Figure 5 It is a structural diagram of a real-time assessment system for the physical and mental state of the elderly based on facial expression recognition according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0059] Figure 1 FIG. 1 is an overall flow chart of a method for real-time assessment of the physical and mental state of the elderly based on facial expression recognition according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0060] S1, building a facial expression recognition network framework based on the attention module, the facial expression recognition network framework based on the attention module includes an image sequence, a three-dimensional convolution Conv3D, a three-dimensional residual block 3D Residual Block, a fully connected layer FC, a Softmax classifier based on cross entropy loss Softmax, and an efficient attention module EAM Block;

[0061] S2, constructs a training dataset based on user needs. After collecting a large number of videos of elderly people's facial expressions, they are labeled with the correct labels of anger, disgust, fear, happiness, sadness, or surprise to construct the training dataset.

[0062] S3, inputting the constructed training data set into the facial expression recognition network based on the attention module to obtain an estimated emotion label, then using the constructed cross entropy loss function to calculate the loss between the estimated emotion label and the correct emotion label of the video, and then reversely updating the parameters of the network model, repeating this step multiple times to obtain a trained facial expression recognition model;

[0063] S4, inputting the video of the facial expression of the elderly to be processed into the trained facial expression recognition model to obtain an output emotion label, and determining the elderly's emotion or physical and mental state according to the label.

[0064] Step S1, such as Figure 2 As shown, the details are as follows:

[0065] S1-1, the input of the facial expression recognition network framework based on the attention module is the image sequence I∈R H ×W×T×3 , that is, a video containing the facial expressions of the elderly, where T represents the number of frames in the video, 3 represents the number of channels in one frame of the video, and the output is the emotion label, which is set to six labels: anger, disgust, fear, happiness, sadness, and surprise;

[0066] S1-2, design a three-dimensional convolution operation to extract shallow features. The shallow features are dynamic features, which are video features containing a time dimension. They are used to represent facial elements, muscle movements, and other information in facial expression videos. The extracted features are recorded as C out The number of output channels is 32, followed by three 3D residual blocks and three EAM attention modules. Finally, after the operation of the fully connected layer and the calculation of the Softmax layer, the emotion label recognized by the network is obtained.

[0067] S1-3, such as Figure 3 As shown in the figure, a three-dimensional residual block structure is designed, which includes two convolutional layer operations. After each convolutional layer operation, batch normalization BatchNormal and ReLU nonlinear activation function are performed. Let this structure be the i-th (i=1, 2, 3) three-dimensional residual block structure, and its input is the feature V extracted by the previous layer EAM module or convolution. i-1 , the output features By introducing the EAM attention module, the residual network architecture improves image classification performance by increasing the network depth, thereby alleviating the problem of relatively small training datasets.

[0068] S1-4, such as Figure 4 As shown, the efficient attention module EAM Block is designed, and the efficient attention module includes a channel efficient attention module EAM-S and a time efficient attention module EAM-T;

[0069] S1-4-1, the input of the EAM module is the output of the 3D residual module in the same layer set up The dimensions are H×W×T×C, and the output is V i In the EAM module, the EAM-S module is responsible for learning the attention weights M of different channels c ∈R 1×1×1×C , the EAM-T module is responsible for learning the attention weights M of different frames T ∈R 1×1×1×T , the calculation formula of the working process of the EAM module is as follows:

[0070]

[0071]

[0072]

[0073] in, Indicates element-wise multiplication, and Trans indicates the conversion of matrix dimensions, from H×W×T×C to H×W×C×T or from H×W×C×T to H×W×T×C.

[0074] S1-4-2, the main task of the EAM-S module is to calculate the attention weights M of different channels c , separate the input features by channel where v c ∈R H×W×T , perform 3D-GAP and 3D-GMP operations on the input features to obtain a c ∈R 1×1×1×C and m c ∈R 1×1×1×C Two different channel descriptions, the calculation formula is as follows:

[0075]

[0076]

[0077] a=1,…,H

[0078] b=1,…,W

[0079] t=1,…,T

[0080] Where c∈[1,2,…,C], H, W and T represent the length, width and number of frames;

[0081] The results of the two pooling operations are combined by element-wise summation to obtain the final channel feature descriptor. After that, the final channel feature descriptor is subjected to a one-dimensional convolution operation. Finally, the vector after the convolution operation is put into the sigmoid function to obtain the attention weights M of different channels. c , the obtained channel attention weight M c and input features Calculate the channel weighted feature V′ i , the specific calculation formula is as follows:

[0082]

[0083] in, Indicates element-wise multiplication, and Trans indicates the conversion of matrix dimensions, from H×W×T×C to H×W×C×T or from H×W×C×T to H×W×T×C.

[0084] S1-4-3, the goal of the EAM-T module is to calculate the attention weights M of different frames T , after the Transpose operation before the EAM-T module, the feature V′i The dimension is converted from H×W×T×C to H×W×C×T, and 3D-GAP and 3D-GMP operations are performed on the input features to obtain a t ∈R 1×1×1×T and m t ∈R 1×1×1×T Two different frame descriptions, the calculation formula is as follows:

[0085]

[0086] m t =3DGMP(V′ i )=max{V′ i (a,b,k)},

[0087] a=1,…,H

[0088] b=1,…,W

[0089] k=1,…,C

[0090] Where t∈[1,2,…,T], H, W and C represent the length, width and number of channels;

[0091] The results of the two pooling operations are combined by element-wise summation to obtain the final frame feature descriptor. After that, the final frame feature descriptor is subjected to a one-dimensional convolution operation. Finally, the vector after the convolution operation is put into the sigmoid function to obtain the attention weights M of different frames. T , the obtained frame attention weight M T And the input feature V′ i Calculate the weighted feature V″ by frame i , the calculation formula is as follows:

[0092]

[0093] in, Indicates multiplication by corresponding elements, Trans indicates the conversion operation of matrix dimension, from H×W×T×C dimension to H×W×C×T or from H×W×C×T to H×W×T×C,

[0094] A Transpose operation is performed after the EAM-T module to transform the feature V″ i The dimensions are converted from H×W×C×T to H×W×T×C, and V″ i With the input features Add up to get the output feature V of the EAM module i .

[0095] S1-5, design the fully connected layer and Softmax classifier. After processing by the Softmax classifier, a six-dimensional vector is obtained. The six dimensions represent six different emotions. Each element of the vector is a probability value, indicating the probability of the elderly person being in the corresponding mood. The element with the highest probability is the elderly person's current main emotion.

[0096] Step S2 is as follows:

[0097] S2-1, construct a training data set according to user needs. After collecting a large number of videos of elderly people's facial expressions, label them with correct labels of anger, disgust, fear, happiness, sadness or surprise, that is, construct a training data set.

[0098] Step S3 is as follows:

[0099] S3-1, input the constructed training data set into the facial expression recognition network based on the attention module to obtain the estimated emotion label, and then use the constructed cross entropy loss function to calculate the loss between the estimated emotion label and the correct emotion label of the video, and then reversely update the parameters of the network model, and repeat this step multiple times to obtain a trained facial expression recognition model.

[0100] Step S4 is as follows:

[0101] S4-1, inputting the video of the facial expression of the elderly to be processed into the trained facial expression recognition model to obtain an output emotion label, and determining the emotion or physical and mental state of the elderly according to the label.

[0102] Accordingly, the present invention also provides a real-time assessment system for the physical and mental state of the elderly based on facial expression recognition, such as Figure 5 Shown, including:

[0103] The facial expression recognition network framework unit 1 based on the attention module is used to build a facial expression recognition network framework based on the attention module. The facial expression recognition network framework based on the attention module includes an image sequence, a three-dimensional convolution Conv3D, a three-dimensional residual block 3D Residual Block, a fully connected layer FC, a Softmax classifier based on cross entropy loss Softmax, and an efficient attention module EAM Block.

[0104] Specifically, the input of the facial expression recognition network framework based on the attention module is an image sequence I∈R H ×W×T×3, that is, a video containing the facial expressions of the elderly, where T represents the number of frames of the video, 3 represents the number of channels in one frame of the video is 3, and the output is the emotion label, which is set to six labels: anger, disgust, fear, happiness, sadness, and surprise; a three-dimensional convolution operation is designed to extract shallow features. The shallow features are dynamic features, which are video features containing the time dimension. They are used to represent facial elements, muscle movements, and other information in the facial expression video. The extracted features are recorded as C out The number of output channels is 32, followed by three three-dimensional residual blocks and three EAM attention modules. Finally, after the operation of the fully connected layer and the calculation of the Softmax layer, the emotion label recognized by the network is obtained. The three-dimensional residual block structure is designed, which includes two convolutional layer operations. Each convolutional layer operation is followed by batch normalization BatchNormal and ReLU nonlinear activation function. The structure is set as the i-th (i=1, 2, 3) three-dimensional residual block structure, and its input is the feature V extracted by the previous EAM module or convolution. i-1 , the output features The EAM attention module is input, and the residual network architecture improves the image classification performance by increasing the network depth, thereby alleviating the problem of a relatively small training data set; the efficient attention module EAM Block is designed, and the efficient attention module includes the channel efficient attention module EAM-S and the time efficient attention module EAM-T; the fully connected layer and the Softmax classifier are designed, and after processing by the Softmax classifier, a six-dimensional vector is obtained. The six dimensions represent six different emotions, and each element of the vector is a probability value, indicating the probability that the elderly person is in the corresponding mood. The element with the largest probability is the elderly person's current main emotion.

[0105] Specifically, the efficient attention module EAM Block is designed, and the efficient attention module includes a channel efficient attention module EAM-S and a time efficient attention module EAM-T, including: the input of the EAM module is the output of the three-dimensional residual module of the same layer set up The dimensions are H×W×T×C, and the output is V i In the EAM module, the EAM-S module is responsible for learning the attention weights M of different channels c ∈R 1×1×1×C , the EAM-T module is responsible for learning the attention weights M of different frames T ∈R 1 ×1×1×T The main task of the EAM-S module is to calculate the attention weights M of different channels. c , separate the input features by channel where vc ∈R H×W×T , perform 3D-GAP and 3D-GMP operations on the input features to obtain a c ∈R 1×1×1×C and m c ∈R 1×1×1×C Two different channel descriptions, the results of the two pooling operations are combined by element-wise summation to obtain the final channel feature descriptor. After that, the final channel feature descriptor is subjected to a one-dimensional convolution operation. Finally, the vector after the convolution operation is put into the sigmoid function to obtain the attention weights M of different channels. c , the obtained channel attention weight M c and input features Calculate the channel weighted feature V′ i ; The goal of the EAM-T module is to calculate the attention weights M of different frames T , after the Transpose operation before the EAM-T module, the feature V′ i The dimension is converted from H×W×T×C to H×W×C×T, and 3D-GAP and 3D-GMP operations are performed on the input features to obtain a t ∈R 1×1×1×T and m t ∈R 1×1×1×T Two different frame descriptions, the results of the two pooling operations are combined by element-wise summation to obtain the final frame feature descriptor. After that, the final frame feature descriptor is subjected to a one-dimensional convolution operation. Finally, the vector after the convolution operation is put into the sigmoid function to obtain the attention weights M of different frames. T , the obtained frame attention weight M T And the input feature V′ i Calculate the weighted feature V″ by frame i ; Perform a Transpose operation after the EAM-T module to transform the feature V″ i The dimensions are converted from H×W×C×T to H×W×T×C, and V″ i With the input features Add up to get the output feature V of the EAM module i .

[0106] The training data set construction unit 2 is used to construct a training data set according to the user's needs. After collecting a large number of videos of elderly people's facial expressions, they are labeled with correct labels of anger, disgust, fear, happiness, sadness or surprise to construct a training data set.

[0107] The facial expression recognition model training unit 3 is used to input the constructed training data set into the facial expression recognition network based on the attention module to obtain an estimated emotion label, and then use the constructed cross entropy loss function to calculate the loss between the estimated emotion label and the correct emotion label of the video, and then reversely update the parameters of the network model. Repeat this step multiple times to obtain a trained facial expression recognition model.

[0108] The real-time assessment unit 4 for the elderly's physical and mental state is used to input the video of the elderly's facial expression to be processed into the trained facial expression recognition model, obtain the output emotion label, and determine the elderly's emotion or physical and mental state according to the label.

[0109] Therefore, the present invention constructs an efficient attention module EAM for facial emotion recognition, wherein the EAM module includes a channel EAM-S module and an EAM-T module, which can learn the weights of different channels and different frames, and then assign more weights to important features, so that the neural network focuses more attention on important channel information and important video frames; the information features of channels and different video frames are modeled by the EAM module, which improves the network's attention to important information, and more accurately and efficiently identifies the physical and mental state of the elderly based on the elderly's facial expression video.

[0110] The above is a detailed introduction to the real-time assessment method and system for the physical and mental state of the elderly based on facial expression recognition provided by the embodiments of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A real-time assessment method for the physical and mental state of the elderly based on facial expression recognition, characterized in that: The method comprises: Building a facial expression recognition network framework based on the attention module, the facial expression recognition network framework based on the attention module includes an image sequence, a three-dimensional convolution Conv3D, a three-dimensional residual block 3D Residual Block, a fully connected layer FC, a Softmax classifier based on cross entropy loss Softmax, and an efficient attention module EAM Block; Construct a training dataset based on user needs. After collecting a large number of videos of elderly people's facial expressions, label them with the correct labels of anger, disgust, fear, happiness, sadness, or surprise to construct a training dataset. Input the constructed training data set into the facial expression recognition network based on the attention module to obtain estimated emotion labels, then use the constructed cross entropy loss function to calculate the loss between the estimated emotion labels and the correct emotion labels of the video, and then reversely update the parameters of the network model. Repeat this step multiple times to obtain a trained facial expression recognition model; Inputting the video of the elderly person's facial expression to be processed into the trained facial expression recognition model to obtain an output emotion label, and determining the elderly person's emotion or physical and mental state based on the label; The facial expression recognition network framework based on the attention module is constructed, and the facial expression recognition network framework based on the attention module includes an image sequence, a three-dimensional convolution Conv3D, a three-dimensional residual block 3D Residual Block, a fully connected layer FC, a Softmax classifier based on cross entropy loss Softmax, and an efficient attention module EAM Block, specifically: The input of the facial expression recognition network framework based on the attention module is the image sequence I∈R H×W×T×3 , that is, a video containing the facial expressions of the elderly, where T represents the number of frames in the video, 3 represents the number of channels in one frame of the video, and the output is the emotion label, which is set to six labels: anger, disgust, fear, happiness, sadness, and surprise; The three-dimensional convolution operation is designed to extract shallow features. The shallow features are dynamic features, which are video features containing a time dimension. They are used to represent facial elements and muscle movement information in facial expression videos. The extracted features are recorded as C out The number of output channels is 32, followed by three 3D residual blocks and three EAM attention modules. Finally, after the operation of the fully connected layer and the calculation of the Softmax layer, the emotion label recognized by the network is obtained. The three-dimensional residual block structure is designed. The structure includes two convolutional layer operations. After each convolutional layer operation, batch normalization BatchNormal and ReLU nonlinear activation function are performed. The structure is set as the i-th (i=1, 2, 3) three-dimensional residual block structure, and its input is the feature V extracted by the previous layer EAM module or convolution. i-1 , the output feature V i R By introducing the EAM attention module, the residual network architecture improves image classification performance by increasing the network depth, thereby alleviating the problem of relatively small training datasets; Design the efficient attention module EAM Block, which includes the channel efficient attention module EAM-S and the time efficient attention module EAM-T; The fully connected layer and Softmax classifier are designed. After processing by the Softmax classifier, a six-dimensional vector is obtained. The six dimensions represent six different emotions. Each element of the vector is a probability value, indicating the probability that the elderly person is in the corresponding mood. The element with the largest probability is the elderly person's current main emotion.

2. The method for real-time assessment of the physical and mental state of the elderly based on facial expression recognition according to claim 1, characterized in that: The design of the efficient attention module EAM Block includes a channel efficient attention module EAM-S and a time efficient attention module EAM-T, specifically: The input of the EAM module is the output V of the three-dimensional residual module in the same layer i R , let V i R The dimensions are H×W×T×C, and the output is V i In the EAM module, the EAM-S module is responsible for learning the attention weights M of different channels c ∈R 1×1×1×C , the EAM-T module is responsible for learning the attention weights M of different frames T ∈R 1×1×1×T , the calculation formula of the working process of the EAM module is as follows: V i =V i ″+V i R , in, Indicates element-wise multiplication, and Trans indicates the conversion of matrix dimensions, from H×W×T×C to H×W×C×T or from H×W×C×T to H×W×T×C. The main task of the EAM-S module is to calculate the attention weights M of different channels c , separate the input features by channel V i R =[v1,v2,…,v c ], where v c ∈R H×W×T , perform three-dimensional global average pooling 3D-GAP and three-dimensional global maximum pooling 3D-GMP operations on the input features to obtain a c ∈R 1×1×1×C and m c ∈R 1×1×1×C Two different channel descriptions, the calculation formula is as follows: m c =3DGMP(V i R )=max{V i R (a,b,t)}, a=1,…,H b=1,…,W t=1,…,T Where c∈[1,2,…,C], H, W and T represent the length, width and number of frames; The results of the two pooling operations are combined by element-wise summation to obtain the final channel feature descriptor. After that, the final channel feature descriptor is subjected to a one-dimensional convolution operation. Finally, the vector after the convolution operation is put into the sigmoid function to obtain the attention weights M of different channels. c , the obtained channel attention weight M c And the input feature V i R Calculate the channel-weighted feature V i ′, the specific calculation formula is as follows: in, Indicates element-wise multiplication, and Trans indicates the conversion of matrix dimensions, from H×W×T×C to H×H×C×T or from H×W×C×T to H×W×T×C. The goal of the EAM-T module is to calculate the attention weights M of different frames T , after the Transpose operation before the EAM-T module, the feature V i The dimension of ′ is converted from H×W×T×C to H×W×C×T, and 3D-GAP and 3D-GMP operations are performed on the input features to obtain a t ∈R 1×1×1×T and m t ∈R 1×1×1×T Two different frame descriptions, the calculation formula is as follows: m t =3DGMP(V i ′)=max{V i ′(a,b,k)}, a=1,…,H b=1,…,W k=1,…,C Where t∈[1,2,…,T], H, W and C represent the length, width and number of channels; The results of the two pooling operations are combined by element-wise summation to obtain the final frame feature descriptor. After that, the final frame feature descriptor is subjected to a one-dimensional convolution operation. Finally, the vector after the convolution operation is put into the sigmoid function to obtain the attention weights M of different frames. T , the obtained frame attention weight M T And the input feature V i 'Calculate the weighted feature V by frame i ″, the calculation formula is as follows: in, Indicates multiplication by corresponding elements, Trans indicates the conversion operation of matrix dimension, from H×W×T×C dimension to H×W×C×T or from H×W×C×T to H×W×T×C, A Transpose operation is performed after the EAM-T module to transform the feature V i The dimension of ″ is converted from H×W×C×T to H×W×T×C, and V i ″ and the input feature V i R Add up to get the output feature V of the EAM module i .

3. A real-time assessment system for the physical and mental state of the elderly based on facial expression recognition, characterized by: The system comprises: A facial expression recognition network framework unit based on an attention module is used to build a facial expression recognition network framework based on an attention module. The facial expression recognition network framework based on the attention module includes an image sequence, a three-dimensional convolution Conv3D, a three-dimensional residual block 3D ResidualBlock, a fully connected layer FC, a Softmax classifier based on cross entropy loss, and an efficient attention module EAM Block; The training dataset construction unit is used to construct a training dataset based on user needs. After collecting a large number of videos of elderly people's facial expressions, they are labeled with the correct labels of anger, disgust, fear, happiness, sadness, or surprise to construct a training dataset. A facial expression recognition model training unit is used to input the constructed training data set into the facial expression recognition network based on the attention module to obtain an estimated emotion label, then use the constructed cross entropy loss function to calculate the loss between the estimated emotion label and the correct emotion label of the video, and then reversely update the parameters of the network model, and repeat this step multiple times to obtain a trained facial expression recognition model; A real-time assessment unit for the physical and mental state of the elderly is used to input a video of the elderly's facial expression to be processed into the trained facial expression recognition model, obtain an output emotion label, and determine the elderly's emotion or physical and mental state based on the label; The facial expression recognition network framework unit based on the attention module requires that the input of the facial expression recognition network framework based on the attention module is an image sequence I∈R H×W×T×3 , that is, a video containing facial expressions of the elderly, where T represents the number of frames of the video, 3 represents the number of channels in one frame of the video is 3, and the output is the emotion label, which is set to six labels: anger, disgust, fear, happiness, sadness, and surprise; the three-dimensional convolution operation is designed to extract shallow features, which are dynamic features and are video features containing a time dimension. They are used to represent facial elements and muscle movement information in facial expression videos, and the extracted features are recorded as C out Indicates the number of output channels, which is 32. It then includes three three-dimensional residual blocks and three EAM attention modules. Finally, after the operation of the fully connected layer and the calculation of the Softmax layer, the emotion label recognized by the network is obtained. The three-dimensional residual block structure is designed. The structure includes two convolutional layer operations. Each convolutional layer operation is followed by batch normalization BatchNormal and ReLU nonlinear activation function. The structure is set as the i-th (i=1, 2, 3) three-dimensional residual block structure, and its input is the feature V extracted by the previous EAM module or convolution. i-1 , the output feature V i R The EAM attention module is input, and the residual network architecture improves the image classification performance by increasing the network depth, thereby alleviating the problem of a relatively small training data set; the efficient attention module EAMBlock is designed, and the efficient attention module includes the channel efficient attention module EAM-S and the time efficient attention module EAM-T; the fully connected layer and the Softmax classifier are designed, and after processing by the Softmax classifier, a six-dimensional vector is obtained. The six dimensions represent six different emotions, and each element of the vector is a probability value, indicating the probability that the elderly person is in the corresponding mood. The element with the largest probability is the elderly person's current main emotion.

4. The real-time assessment system for the physical and mental state of the elderly based on facial expression recognition as claimed in claim 3, characterized in that: The facial expression recognition network framework unit based on the attention module requires that the input of the EAM module is the output V of the three-dimensional residual module of the same layer i R , let V i R The dimensions are H×W×T×C, and the output is V i In the EAM module, the EAM-S module is responsible for learning the attention weights M of different channels c ∈R 1×1×1×C , the EAM-T module is responsible for learning the attention weights M of different frames T ∈R 1 ×1×1×T The main task of the EAM-S module is to calculate the attention weights M of different channels. c , separate the input features by channel V i R =[v1,v2,…,v c ], where v c ∈R H×W×T , perform 3D-GAP and 3D-GMP operations on the input features to obtain a c ∈R 1×1×1×C and m c ∈R 1×1×1×C Two different channel descriptions, the results of the two pooling operations are combined by element-wise summation to obtain the final channel feature descriptor. After that, the final channel feature descriptor is subjected to a one-dimensional convolution operation. Finally, the vector after the convolution operation is put into the sigmoid function to obtain the attention weights M of different channels. c , the obtained channel attention weight M c And the input feature V i R Calculate the channel-weighted feature V i '; The goal of the EAM-T module is to calculate the attention weights M of different frames T , after the Transpose operation before the EAM-T module, the feature V i The dimension of ′ is converted from H×W×T×C to H×W×C×T, and 3D-GAP and 3D-GMP operations are performed on the input features to obtain a t ∈R 1×1×1×T and m t ∈R 1×1×1×T Two different frame descriptions, the results of the two pooling operations are combined by element-wise summation to obtain the final frame feature descriptor. After that, the final frame feature descriptor is subjected to a one-dimensional convolution operation. Finally, the vector after the convolution operation is put into the sigmoid function to obtain the attention weights M of different frames. T , the obtained frame attention weight M T And the input feature V i 'Calculate the weighted feature V by frame i ″, a Transpose operation is performed after the EAM-T module to transform the feature V i The dimension of ″ is converted from H×W×C×T to H×W×T×C, and V i ″ and the input feature V i R Add up to get the output feature V of the EAM module i .

5. The real-time assessment system for the physical and mental state of the elderly based on facial expression recognition as claimed in claim 3, characterized in that: The training data set construction unit needs to construct a training data set according to user needs. After collecting a large number of videos of elderly people's facial expressions, they are labeled with correct labels of anger, disgust, fear, happiness, sadness or surprise to construct a training data set.

6. The real-time assessment system for the physical and mental state of the elderly based on facial expression recognition as claimed in claim 3, characterized in that: The facial expression recognition model training unit needs to input the constructed video data into the network to obtain the estimated emotion label, and then use the constructed cross entropy loss function to calculate the loss between the estimated emotion label and the correct emotion label of the video, and then reversely update the parameters of the network model. Repeat this process multiple times to obtain a trained facial expression recognition model.

7. The real-time assessment system for the physical and mental state of the elderly based on facial expression recognition as claimed in claim 3, characterized in that: The real-time assessment unit for the elderly's physical and mental state needs to input the video of the elderly's facial expression into a trained network to obtain an output emotion label, and determine the elderly's emotion or physical and mental state based on the label.

Images