Old people expression recognition method, device and equipment and storage medium

Abstract

The invention relates to the technical field of image recognition, and discloses an old people expression recognition method and device, equipment and a storage medium. The method comprises the steps of firstly constructing an old people expression data set, then extracting face key region features through a space attention mechanism, extracting face local dynamic features by using a time attention mechanism, finally performing feature fusion on multi-level features, and completing old people expression recognition based on the fused features. The old people expression recognition model constructed by the invention can obtain the static features at the current moment and the dynamic features in the time domain based on the feature fusion of cross attention, and obtains more and more detailed face information, thereby achieving the more accurate recognition of the facial expressions of the old people.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method, device, equipment and storage medium for recognizing facial expressions of the elderly. Background Art

[0002] In the fields of health monitoring, auxiliary medical care, and smart home monitoring, the elderly are the main service group, and these applications are helpful to improve the quality of life of the elderly. Among them, facial expressions are one of the ways for humans to express their own state. Expression recognition technology helps the intelligent system to autonomously identify the state of people and respond accordingly during human-computer interaction. Due to the special characteristics of the elderly, such as the many textures and loose muscles on their faces, it is necessary to study expression recognition technology for the elderly. On the other hand, considering the main application scenarios of elderly monitoring, the facial postures of the elderly in the field of view of the monitoring camera are diverse, which may be frontal or sideways, and it is necessary to build an expression recognition system suitable for different facial postures.

[0003] Most of the data sets used by existing expression recognition models contain complete frontal expression samples of young people. These models are mainly suitable for frontal expression recognition of the general population. Moreover, since the facial texture and muscle structure of the elderly will become loose and wrinkled as they age, the intensity of facial movements will also weaken when expressing emotions. Therefore, existing expression recognition models have large deviations when recognizing the expressions of the elderly, and the generalization of expression recognition of side faces is poor in some special scenarios.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a method, device, equipment and storage medium for facial expression recognition for the elderly, aiming to solve the problem that the existing facial expression recognition model has large deviations when recognizing the facial expressions of the elderly, and the generalization of facial expression recognition is poor in some special scenarios.

[0006] The first aspect of the present invention provides a method for recognizing facial expressions of the elderly, comprising: expanding and labeling acquired facial expression images of the elderly to obtain a facial expression data set of the elderly, wherein the facial expression data set of the elderly includes a plurality of sequence samples consisting of a plurality of consecutive images, each sequence sample corresponds to a different facial expression category label, and the facial expression category labels include pain, fear, sadness, anger, disgust, happiness and neutrality; dividing the facial expression data set of the elderly into a training set and a test set according to a predetermined ratio, wherein both the training set and the test set include sequence samples corresponding to seven facial expression category labels, namely pain, fear, sadness, anger, disgust, happiness and neutrality; constructing an initial facial expression recognition model of the elderly based on multi-level feature fusion; training the initial facial expression recognition model of the elderly through the training set, adjusting the parameters of the initial facial expression recognition model of the elderly, and obtaining a trained facial expression recognition model of the elderly after being tested through the test set; using the facial expression images of the elderly at the current time t and the previous time tT in consecutive T+1 frames as the input of the trained facial expression recognition model of the elderly, and outputting the facial expression recognition result of the elderly at the current time t, where T is an integer greater than or equal to 2.

[0007] Optionally, in a first implementation method of the first aspect of the present invention, the acquired facial expression images of the elderly are expanded and labeled to obtain an elderly facial expression dataset, including the steps of: collecting sample video data by photographing each individual in a group of elderly people in a variety of facial postures, wherein the facial postures include front face, side face, half side face, upper side face and lower side face; converting the sample video data into sequence samples consisting of multiple consecutive frames of pictures according to the number of frames; performing data enhancement processing on the sequence samples to obtain expanded sequence samples; performing labeling processing on the expanded sequence samples to obtain an elderly facial expression dataset labeled with expression category labels, wherein the expression category labels include pain, fear, sadness, anger, disgust, happiness and neutrality.

[0008] Optionally, in a second implementation manner of the first aspect of the present invention, in the step of performing data enhancement processing on the sequence samples, the data enhancement processing includes one or more of illumination change processing, noise addition processing, color change processing and mirror transformation processing.

[0009] Optionally, in a third implementation of the first aspect of the present invention, an initial elderly expression recognition model is constructed based on multi-level feature fusion, including the steps of: constructing a facial key area feature extraction module based on spatial attention, the facial key area feature extraction module including a convolutional neural network module for extracting a facial key area map in an image, a residual network module for extracting a global face feature map in an image, and a spatial attention module that takes the facial key area map and the global face feature map as input and outputs facial key area features in the image; constructing a time domain dynamic feature extraction module based on temporal attention, the time domain dynamic feature extraction module including a time attention module and a recurrent neural network RNN, the time attention module is used to generate an attention weight matrix of facial key area features in the time domain and weight the facial key area features; the recurrent neural network RNN ​​is used to receive the weighted facial key area features and output local facial dynamic features; and constructing a feature fusion module and a classifier after the facial key area feature extraction module and the time domain dynamic feature extraction module to obtain the initial elderly expression recognition model.

[0010] Optionally, in a fourth implementation of the first aspect of the present invention, the temporal attention module includes a maximum pooling layer, an average pooling layer, a shared fully connected layer, and a summed activation layer.

[0011] Optionally, in a fifth implementation of the first aspect of the present invention, the feature fusion module takes the facial key area features and the facial local dynamic features as input, and uses the spatiotemporal fusion feature MF t As output, the facial key area feature is represented as F t , after linear transformation, we get the corresponding feature K F , Q F 、V F The local dynamic feature of the face is represented as Dt, and the corresponding feature K is obtained through linear transformation D , Q D 、V D ; The spatiotemporal fusion feature MF t =II aμ (CM DF +CM Fd ),in, Where d is K F or K D Dimension.

[0012] Optionally, in a sixth implementation of the first aspect of the present invention, the initial elderly expression recognition model is trained using the training set, the parameters of the initial elderly expression recognition model are adjusted, and after being tested using the test set, a trained elderly expression recognition model is obtained, including the steps of: assigning initial values ​​to all parameters in the initial elderly expression recognition model using a random initialization method; randomly selecting sequence samples from the training set and inputting them into the initial elderly expression recognition model for training, calculating the prediction results through forward propagation, and then calculating the loss value according to the FocalLoss function; updating the parameters of the network using the back propagation algorithm so that the loss value gradually decreases; repeating the above training steps until a predetermined number of training rounds is reached to obtain a trained initial elderly expression recognition model; using the test set to evaluate the trained initial elderly expression recognition model, adjusting the hyperparameters according to the evaluation results, and obtaining a trained elderly expression recognition model.

[0013] The second aspect of the present invention provides an elderly facial expression recognition device, comprising: an image processing module, used to expand and label the acquired elderly facial expression images to obtain an elderly facial expression data set, wherein the elderly facial expression data set includes a plurality of sequence samples consisting of a plurality of consecutive images, each sequence sample corresponds to a different facial expression category label, and the facial expression category labels include pain, fear, sadness, anger, disgust, happiness and neutrality; a data division module, used to divide the elderly facial expression data set into a training set and a test set according to a predetermined ratio, and the training set and the test set both include pain, fear, sadness, anger, disgust, happiness and neutrality. The present invention relates to a method for generating a facial expression recognition model for the elderly using the training set, a facial expression recognition model for the elderly using the training set, and a facial expression recognition model for the elderly using the test set. The method comprises the following steps: a) determining a facial expression recognition model for the elderly using the training set, and b) determining a facial expression recognition model for the elderly using the training set, and c) determining a facial expression recognition model for the elderly using the test set; d) determining a facial expression recognition model for the elderly using the training set, and c ...

[0014] The third aspect of the present invention provides an expression recognition device for the elderly, comprising: a memory and at least one processor, wherein the memory stores computer-readable instructions, and the memory and the at least one processor are interconnected via lines; the at least one processor calls the computer-readable instructions in the memory so that the expression recognition device for the elderly performs the various steps of the expression recognition method for the elderly as described above.

[0015] The fourth aspect of the present invention provides a computer-readable storage medium, which stores computer-readable instructions. When the computer-readable storage medium is run on a computer, it enables the computer to execute the various steps of the above-mentioned method for recognizing facial expressions of the elderly.

[0016] Beneficial effects: The present invention provides a method for facial expression recognition of the elderly. First, an elderly facial expression data set is constructed. Then, the key facial area features are extracted through a spatial attention mechanism. The local dynamic features of the face are extracted using a temporal attention mechanism. Finally, the multi-level features are fused, and the facial expression recognition of the elderly is completed based on the fused features. Since the generation of facial expressions is a gradual process, the information related to facial expressions is not only reflected in the facial features at a certain moment, but can also be expressed through some subtle dynamic features of the facial state in a short period of time. The intensity of facial movements of the elderly when expressing emotions may not be too large. Therefore, obtaining the static features at the current moment and the dynamic features in the time domain can make up for the shortcomings of a single type of features and obtain more information. The elderly facial expression recognition model constructed by the present invention can obtain more and more detailed facial information based on the feature fusion of cross-attention, thereby achieving more accurate classification of the facial expressions of the elderly. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a method for recognizing facial expressions of the elderly provided in an embodiment of the present invention.

[0018] Figure 2 This is the structural diagram of the facial key area feature extraction module based on spatial attention.

[0019] Figure 3 This is the structural diagram of the temporal attention-based temporal dynamic feature extraction module.

[0020] Figure 4 This is a structural diagram of the initial elderly expression recognition model of the present invention.

[0021] Figure 5 This is the structural diagram of the feature fusion module based on cross attention.

[0022] Figure 6 This is a schematic diagram of the structure of the elderly facial expression recognition device provided by the present invention.

[0023] Figure 7 This is a schematic diagram of the structure of the facial expression recognition device for the elderly provided by the present invention. DETAILED DESCRIPTION

[0024] Embodiments of the present invention provide a method, apparatus, device and storage medium for facial expression recognition for the elderly. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] See also Figure 1 , Figure 1 A flow chart of a method for recognizing facial expressions of the elderly provided by the present invention, as shown in the figure, comprises the following steps:

[0026] S10, expanding and labeling the acquired elderly facial expression images to obtain an elderly facial expression dataset, wherein the elderly facial expression dataset includes a plurality of sequence samples consisting of a plurality of consecutive images, each sequence sample corresponds to a different facial expression category label, and the facial expression category labels include pain, fear, sadness, anger, disgust, happiness, and neutrality;

[0027] Specifically, sample video data is collected by recording a variety of facial gestures for each individual in the elderly group, and the facial gestures include front face, side face, half side face, upper side face and lower side face. In this embodiment, the elderly group covers men and women aged 60-100 years old, and when recording, it is ensured that each individual presents seven expression categories of pain, fear, sadness, anger, disgust, happiness and neutrality, so as to improve the diversity and representativeness of the data set; the front face in this embodiment refers to the angle facing the camera, the side face is a 45-degree side turn, and the half side face is a slight side turn, such as a 20-degree side turn, but not limited to this; the upper side face and the lower side face are tilted up or down at a certain angle, such as 20 degrees up or down, respectively, but not limited to this. When shooting, use a high-definition camera, ensure uniform light, simple background, reduce external interference, record a variety of facial gestures for each elderly person, and keep each gesture for a few seconds for subsequent frame extraction; then check the video clarity, natural expression and gesture accuracy, and eliminate unqualified videos.

[0028] After obtaining the sample video data, the sample video data is converted into a sequence sample consisting of multiple consecutive frames of pictures according to the number of frames. For example, all frames of the same video are arranged in chronological order using video processing software or programming tools (such as OpenCV) to form a continuous image sequence; then the sequence samples are subjected to data enhancement processing to obtain extended sequence samples, and the data enhancement processing includes one or more of illumination change processing, noise addition processing, color change processing, and mirror transformation processing, but is not limited thereto. The cameraman or the elderly themselves perform labeling processing on the extended sequence samples during shooting to obtain an elderly expression data set labeled with expression category labels, and the expression category labels include pain, fear, sadness, anger, disgust, happiness, and neutrality.

[0029] This embodiment ensures the diversity and accuracy of the elderly expression dataset through strict screening and quality control, providing a solid foundation for the subsequent training of the elderly expression recognition model; data enhancement processing can simulate image changes in a variety of actual scenarios, so that the model can learn more feature changes during the training process, thereby improving its generalization ability in complex environments; manual annotation ensures the accuracy and consistency of labels, which helps to improve the recognition accuracy of the model; high-quality elderly expression datasets can be used for research and application in multiple fields such as automatic recognition of elderly expressions, sentiment analysis, human-computer interaction, health monitoring, etc., to promote the development and innovation of related technologies.

[0030] S20, dividing the elderly facial expression dataset into a training set and a test set according to a predetermined ratio, wherein the training set and the test set both include sequence samples corresponding to seven facial expression category labels: pain, fear, sadness, anger, disgust, happiness and neutrality;

[0031] In this embodiment, in order to ensure the randomness and fairness of the division results, the entire elderly expression data set needs to be randomly shuffled first, which can be achieved through random functions or data processing tools in programming languages; then, according to the determined division ratio, the shuffled elderly expression data set is divided into a training set and a test set, and the common ratios are 70% training set + 30% test set or 80% training set + 20% test set; when dividing, ensure that both the training set and the test set contain sequence samples of all seven expression category labels to maintain the diversity and representativeness of the data set; store the divided training set and test set in different files or databases for subsequent use. By dividing the data set into a training set and a test set, this embodiment can ensure that the model will not be exposed to the data in the test set during the training process, thereby avoiding the occurrence of overfitting, which helps to improve the generalization ability and accuracy of the model; the test set is a data set independent of the training set, which can be used to evaluate the performance of the model. By comparing the performance of the model on the test set with the performance on the training set, the generalization ability and accuracy of the model can be objectively evaluated.

[0032] S30, constructing an initial elderly expression recognition model based on multi-level feature fusion;

[0033] This embodiment first constructs Figure 2 The face key area feature extraction module based on spatial attention shown in the figure comprises a convolutional neural network module for extracting a face key area map in an image, a residual network module for extracting a face global feature map in an image, and a spatial attention module that takes the face key area map and the face global feature map as input and outputs the face key area features in the image. Specifically, Figure 2 As shown in the figure, for the image at the current time t, the convolutional neural network module is used to extract the face key area map in the image; the residual network module is used to extract the global feature map of the face in the image; the spatial attention module is constructed based on the transformer's multi-head attention mechanism to explore the connection between the key areas of the face, and the face key area map and the global feature map of the face are used as inputs of the attention module to obtain the face key area features in the image at the current time t. As an example, the face key area map extracted by the convolutional neural network module is represented as I t , after linear transformation, it corresponds to Q I , corresponding to the multi-head attention The global face feature map extracted by the residual network module is represented as G t , after linear transformation, it corresponds to K G and V G , corresponding to the multi-head attention The facial key area feature map and facial key area features obtained by the transformer-based multi-head attention module are respectively represented by F t and F i Indicates that, F t =MA(α t , G t ), α t =concat(α t1 , ..., α tn ), (i=1,...,n),d i for The dimension of , softmax represents the softmax function calculation, concat represents vector concatenation, and MA represents the weighted calculation using multi-head attention weights. In this embodiment, the facial key area feature map is a two-dimensional matrix for visualizing the attention weights, and the facial key area feature is a one-dimensional vector (or matrix) for representing the weighted integrated information of the input sequence; both are the output results of the multi-head attention module, but have different uses and representations. The facial key area feature map can be regarded as an intermediate representation of the facial key area feature in the calculation process, which helps the model visualize the distribution of attention weights; and the facial key area feature is the key information finally output by the multi-head attention module.

[0034] Then this embodiment constructs Figure 3 The temporal dynamic feature extraction module based on temporal attention is shown, and the temporal dynamic feature extraction module includes a temporal attention module and a recurrent neural network RNN. The temporal attention module is used to generate an attention weight matrix of facial key area features in the temporal domain and weight the facial key area features; the recurrent neural network RNN ​​is used to receive the weighted facial key area features and output facial local dynamic features; the temporal attention module includes a maximum pooling layer, an average pooling layer, a shared fully connected layer, and a summed activation layer. Specifically, as Figure 3 As shown, the key facial region features F based on multiple frames of images (i.e., sequence samples) i , using the temporal attention module and the recurrent neural network RNN, focusing on the subtle changes in the key areas of the sequence samples over time, and extracting the local dynamic features of the face. As an example, in this embodiment, the facial key area feature set of the sequence samples consisting of {tT,...,t-1,t} for a total of T+1 moments is F = {F i |i=t,t-1,...,tT}, the output of the facial key area feature set after the average pooling layer is The output of the facial key area feature set after the maximum pooling layer is Said and The attention weight matrix obtained in the time domain after the shared fully connected layer MLP and the summed activation layer Use this attention weight to i After weighting, it is used as the input of the recurrent neural network RNN ​​to obtain the local dynamic features D of the face t .

[0035] After the facial key area feature extraction module and the time domain dynamic feature extraction module, a feature fusion module and a classifier are constructed to obtain the following: Figure 4 The initial elderly facial expression recognition model shown in Figure 1 is shown in Figure 2. Figure 4 and Figure 5 As shown in the figure, the feature fusion module takes the key facial area features and the local facial dynamic features as input, and uses the spatiotemporal fusion feature MF t As output, the facial key area feature is represented as F t , after linear transformation, we get the corresponding feature K F , Q F 、V F The local dynamic feature of the face is represented as Dt, and the corresponding feature K is obtained through linear transformation D , Q D 、V D ; The spatiotemporal fusion feature MF t =H up (CM DF +CM Fd ),in, Where d is K F or K D dimension. In this embodiment, since the generation of expression is a gradual process, expression-related information is not only reflected in the facial features at a certain moment, but can also be expressed through some subtle dynamic features of the facial state in a short period of time; and the intensity of facial movements of the elderly when expressing emotions may not be too large. Therefore, obtaining the static features of the current moment and the dynamic features in the time domain can make up for the shortcomings of a single type of feature and obtain more information. The feature fusion module based on cross-attention in this embodiment can establish connections and integrate information between different types of features. Finally, the spatiotemporal fusion feature MF t After the classifier, the expression recognition results of the elderly are obtained.

[0036] S40, training the initial elderly facial expression recognition model through the training set, adjusting the parameters of the initial elderly facial expression recognition model, and obtaining a trained elderly facial expression recognition model after testing with the test set;

[0037] In this embodiment, first ensure that the training set and the test set have been divided according to a predetermined ratio and both contain sequence samples corresponding to the seven expression category labels of pain, fear, sadness, anger, disgust, happiness and neutrality; then determine the hyperparameters in the training process, such as learning rate, batch size, number of training rounds (epochs), and select a suitable loss function (such as cross entropy loss) and optimization algorithm (such as stochastic gradient descent SGD); then input the training set into the initial elderly expression recognition model, perform forward propagation to calculate the loss value, perform back propagation according to the loss value, update the parameters of the model, repeat the forward propagation and back propagation process until the predetermined number of training rounds or loss value convergence is reached; finally, use the test set to test the trained model, evaluate the generalization ability and accuracy of the model, and according to the test results, further optimize and fine-tune the model to obtain a trained elderly expression recognition model. This embodiment can obtain a trained elderly expression recognition model with superior performance and strong generalization ability through reasonable training steps and parameter adjustment.

[0038] S50, using the continuous T+1 frames of elderly facial expression images at the current time t and the previous time tT as the input of the trained elderly facial expression recognition model, and outputting the elderly facial expression recognition result at the current time t, where T is an integer greater than or equal to 2.

[0039] In this embodiment, T+1 frames of elderly facial expression images are arranged in chronological order to form an input sequence, which will be used as the input of the trained elderly facial expression recognition model. The model outputs the facial expression recognition result of the elderly at the current time t, which is usually a probability distribution, indicating the possibility of each expression category, and the expression category with the highest probability is selected as the final recognition result. Since expressions are dynamic and cannot be fully expressed by static images, this embodiment uses continuous frames as input, and the model can use temporal context information to improve the accuracy of expression recognition in the current frame, which helps to reduce misrecognition caused by noise or blur in a single frame image, and can also capture the dynamic process of expression changes, so as to more accurately identify the current expression state.

[0040] The above describes the elderly expression recognition method in the embodiment of the present invention. The following describes the elderly expression recognition device in the embodiment of the present invention. Figure 6 , an embodiment of the elderly facial expression recognition device in the embodiment of the present invention includes:

[0041] The image processing module 10 is used to expand and label the acquired elderly facial expression images to obtain an elderly facial expression data set, wherein the elderly facial expression data set includes a plurality of sequence samples consisting of a plurality of consecutive images, each sequence sample corresponds to a different facial expression category label, and the facial expression category labels include pain, fear, sadness, anger, disgust, happiness and neutrality;

[0042] A data partitioning module 20 is used to divide the elderly facial expression data set into a training set and a test set according to a predetermined ratio, wherein the training set and the test set both include sequence samples corresponding to seven facial expression category labels, namely, pain, fear, sadness, anger, disgust, happiness and neutrality;

[0043] A model building module 30, for building an initial elderly expression recognition model based on multi-level feature fusion;

[0044] A model training module 40 is used to train the initial elderly expression recognition model through the training set, adjust the parameters of the initial elderly expression recognition model, and obtain a trained elderly expression recognition model after testing with the test set;

[0045] The recognition module 50 is used to use the continuous T+1 frames of elderly facial expression images at the current time t and the previous time tT as the input of the trained elderly facial expression recognition model, and output the elderly facial expression recognition result at the current time t, where T is an integer greater than or equal to 2.

[0046] Based on the same idea as the method in the above-mentioned embodiment, the device provided in the present application can implement the method in the above-mentioned embodiment. For the convenience of explanation, the structural diagram of the device embodiment only shows the parts related to the embodiment of the present application. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer modules than shown in the figure, or a combination of certain modules, or different module arrangements.

[0047] Figure 6 The expression recognition device for the elderly in the embodiment of the present invention is described in detail from the perspective of modular functional entities, and the expression recognition device for the elderly in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0048] Figure 71 is a schematic diagram of the structure of an expression recognition device for the elderly provided by an embodiment of the present invention. The expression recognition device 100 for the elderly may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 11 (for example, one or more processors) and a memory 12, and one or more storage media 13 (for example, one or more mass storage devices) storing application programs 133 or data 132. Among them, the memory 12 and the storage medium 13 can be short-term storage or permanent storage. The program stored in the storage medium 13 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the expression recognition device 100 for the elderly. Furthermore, the processor 11 can be configured to communicate with the storage medium 13 to execute a series of instruction operations in the storage medium 13 on the expression recognition device 100 for the elderly.

[0049] The elderly facial expression recognition device 100 may also include one or more power supplies 14, one or more wired or wireless network interfaces 15, one or more input and output interfaces 16, and / or one or more operating systems 131, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 7 The device structure shown does not constitute a limitation on the elderly expression recognition device 100, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0050] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the method for recognizing facial expressions of the elderly.

[0051] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0052] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0053] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recognizing facial expressions of the elderly, characterized in that: Includes steps: The acquired facial expression images of the elderly are expanded and labeled to obtain an facial expression dataset of the elderly, wherein the facial expression dataset of the elderly includes a plurality of sequence samples consisting of a plurality of consecutive images, each sequence sample corresponds to a different facial expression category label, and the facial expression category labels include pain, fear, sadness, anger, disgust, happiness and neutrality; Dividing the elderly facial expression dataset into a training set and a test set according to a predetermined ratio, wherein the training set and the test set both include sequence samples corresponding to seven facial expression category labels: pain, fear, sadness, anger, disgust, happiness and neutrality; Construct an initial elderly facial expression recognition model based on multi-level feature fusion; The initial elderly expression recognition model is trained by the training set, the parameters of the initial elderly expression recognition model are adjusted, and after being tested by the test set, a trained elderly expression recognition model is obtained; The continuous T+1 frames of elderly facial expression images at the current time t and the previous time tT are used as the input of the trained elderly facial expression recognition model, and the elderly facial expression recognition result at the current time t is output, where T is an integer greater than or equal to 2.

2. The method for recognizing facial expressions of the elderly according to claim 1, characterized in that: The acquired facial expression images of the elderly are expanded and labeled to obtain a facial expression dataset of the elderly, including the following steps: Collecting sample video data by photographing each individual in a group of elderly people in a variety of facial postures, wherein the facial postures include frontal face, side face, half side face, upper side face and lower side face; Convert the sample video data into a sequence sample consisting of multiple consecutive frames of pictures according to the number of frames; Performing data enhancement processing on the sequence samples to obtain expanded sequence samples; The expanded sequence samples are labeled to obtain an expression dataset of the elderly marked with expression category labels, wherein the expression category labels include pain, fear, sadness, anger, disgust, happiness and neutrality.

3. The method for recognizing facial expressions of the elderly according to claim 1, characterized in that: In the step of performing data enhancement processing on the sequence samples, the data enhancement processing includes one or more of illumination change processing, noise addition processing, color change processing and mirror transformation processing.

4. The method for recognizing facial expressions of the elderly according to claim 1, characterized in that: The initial elderly expression recognition model is constructed based on multi-level feature fusion, including the following steps: Constructing a facial key area feature extraction module based on spatial attention, the facial key area feature extraction module comprising a convolutional neural network module for extracting a facial key area map in an image, a residual network module for extracting a global face feature map in an image, and a spatial attention module that takes the facial key area map and the global face feature map as input and outputs facial key area features in the image; Constructing a temporal dynamic feature extraction module based on temporal attention, wherein the temporal dynamic feature extraction module includes a temporal attention module and a recurrent neural network (RNN), wherein the temporal attention module is used to generate an attention weight matrix of facial key area features in the temporal domain and weight the facial key area features; The recurrent neural network RNN ​​is used to receive weighted facial key area features and output facial local dynamic features; A feature fusion module and a classifier are constructed after the facial key area feature extraction module and the time domain dynamic feature extraction module to obtain the initial elderly expression recognition model.

5. The method for recognizing facial expressions of the elderly according to claim 4, characterized in that: The temporal attention module includes a maximum pooling layer, an average pooling layer, a shared fully connected layer, and a summed activation layer.

6. The method for recognizing facial expressions of the elderly according to claim 4, characterized in that: The feature fusion module takes the key facial area features and the local facial dynamic features as input, and uses the spatiotemporal fusion feature MF t As output, the facial key area feature is represented as F t , after linear transformation, we get the corresponding feature K F , Q F 、V F The local dynamic feature of the face is represented as Dt, and the corresponding feature K is obtained through linear transformation D , Q D 、V D ; The spatiotemporal fusion feature MF t =H up (CM DF +CM Fd ),in, Where d is K F or K D Dimension.

7. The method for recognizing facial expressions of the elderly according to claim 1, characterized in that: The initial elderly expression recognition model is trained by the training set, the parameters of the initial elderly expression recognition model are adjusted, and after being tested by the test set, a trained elderly expression recognition model is obtained, comprising the steps of: Using a random initialization method to assign initial values ​​to all parameters in the initial elderly expression recognition model; Randomly select sequence samples from the training set and input them into the initial elderly expression recognition model for training, calculate the prediction results through forward propagation, and then calculate the loss value according to the FocalLoss function; Use the back propagation algorithm to update the parameters of the network so that the loss value gradually decreases; Repeat the above training steps until a predetermined number of training rounds is reached to obtain a trained initial elderly expression recognition model; The trained initial elderly facial expression recognition model is evaluated using the test set, and the hyperparameters are adjusted according to the evaluation results to obtain the trained elderly facial expression recognition model.

8. An expression recognition device for the elderly, characterized in that: include: An image processing module is used to expand and label the acquired elderly facial expression images to obtain an elderly facial expression data set, wherein the elderly facial expression data set includes a plurality of sequence samples consisting of a plurality of consecutive images, each sequence sample corresponds to a different facial expression category label, and the facial expression category labels include pain, fear, sadness, anger, disgust, happiness and neutrality; A data partitioning module is used to divide the elderly facial expression data set into a training set and a test set according to a predetermined ratio, wherein the training set and the test set both include sequence samples corresponding to seven facial expression category labels, namely, pain, fear, sadness, anger, disgust, happiness and neutrality; Model building module, used to build an initial elderly expression recognition model based on multi-level feature fusion; A model training module, used for training the initial elderly expression recognition model through the training set, adjusting the parameters of the initial elderly expression recognition model, and obtaining a trained elderly expression recognition model after testing with the test set; The recognition module is used to use the continuous T+1 frames of elderly facial expression images at the current time t and the previous time tT as the input of the trained elderly facial expression recognition model, and output the elderly facial expression recognition result at the current time t, where T is an integer greater than or equal to 2.

9. An expression recognition device for the elderly, characterized in that: comprising a memory and at least one processor, wherein the memory has computer-readable instructions stored therein; The at least one processor calls the computer-readable instructions in the memory to execute the various steps of the method for recognizing facial expressions of the elderly as described in any one of claims 1-7.

10. A computer-readable storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the various steps of the method for recognizing facial expressions of the elderly as described in any one of claims 1-7 are implemented.

Images