An old person expression recognition method, device and equipment and storage medium

Abstract

The application relates to the technical field of image recognition, and discloses an old person expression recognition method, device and equipment and a storage medium. The method comprises the following steps: firstly, an old person expression data set is constructed; then, a spatial attention mechanism is used to extract face key region features, and a time attention mechanism is used to extract face local dynamic features; finally, multi-level features are fused, and old person expression recognition is completed based on the fused features. The old person expression recognition model constructed by the application can fuse features based on cross attention, obtain static features at the current moment and dynamic features in the time domain, obtain more and more detailed face information, and thus more accurate old person face expression recognition can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and particularly relates to an old person expression recognition method, device, equipment and storage medium. BACKGROUND

[0002] In the fields of health monitoring, auxiliary medical treatment, intelligent home monitoring, etc., old people are the main service group, and these applications are helpful to improve the life quality of old people. Among them, expression is one of the ways for human beings to show their state, and expression recognition technology is helpful to the intelligent system in the process of human-computer interaction to autonomously recognize the state of people and make corresponding responses. Due to the particularity of the old people's face, such as more texture and muscle relaxation, it is necessary to study the expression recognition technology for old people. On the other hand, considering the main application scenarios of old people monitoring, the facial posture of the old people in the monitoring camera field of view is diverse, which can be front or side, and it is necessary to construct an expression recognition system suitable for different facial postures.

[0003] Most of the existing expression recognition models use data sets containing young people's complete front expression samples, and these models are mainly suitable for front facial expression recognition of ordinary people. Moreover, the facial texture and muscle structure of old people will become relaxed and wrinkles will increase with age, and the intensity of facial movements when expressing emotions will also weaken, so the existing expression recognition model has a large deviation when recognizing the expressions of old people, and the generalization of side facial expression recognition in some special scenarios is poor.

[0004] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0005] In order to overcome the defects of the prior art, the purpose of the present application is to provide an old person expression recognition method, device, equipment and storage medium, which aims to solve the problem that the existing expression recognition model has a large deviation when recognizing the expressions of old people, and the generalization of side facial expression recognition in some special scenarios is poor.

[0006] The first aspect of the present application provides an expression recognition method for the elderly, comprising: performing expansion and labeling processing on the acquired expression image of the elderly to obtain an expression dataset of the elderly, the expression dataset of the elderly comprising a plurality of sequence samples composed of continuous multiple pictures, each sequence sample corresponding to a different expression category label, and the expression category label comprising pain, fear, sadness, anger, disgust, happiness and neutrality; dividing the expression dataset of the elderly into a training set and a test set according to a predetermined proportion, the training set and the test set each comprising sequence samples corresponding to the seven expression category labels of pain, fear, sadness, anger, disgust, happiness and neutrality; constructing an initial expression recognition model for the elderly based on multi-level feature fusion; training the initial expression recognition model for the elderly through the training set, adjusting the parameters of the initial expression recognition model for the elderly, and obtaining a trained expression recognition model for the elderly after testing by the test set; taking the continuous T+1 frames of expression images of the elderly at the current t time and the previous t-T time as the input of the trained expression recognition model for the elderly, and outputting the expression recognition result of the face of the elderly at the current t time, T being an integer greater than or equal to 2.

[0007] Optionally, in the first implementation manner of the first aspect of the present application, the expansion and labeling processing on the acquired expression image of the elderly to obtain an expression dataset of the elderly comprises the steps of: collecting sample video data by taking pictures of multiple facial poses for each individual in the elderly group, the facial poses comprising front face, side face, half side face, upper side face and lower side face; converting the sample video data into sequence samples composed of continuous multiple pictures according to the number of frames; performing data enhancement processing on the sequence samples to obtain expanded sequence samples; and performing labeling processing on the expanded sequence samples to obtain an expression dataset of the elderly with expression category labels, the expression category labels comprising pain, fear, sadness, anger, disgust, happiness and neutrality.

[0008] Optionally, in the second implementation manner of the first aspect of the present application, in the step of performing data enhancement processing on the sequence samples, the data enhancement processing comprises one or more of illumination change processing, noise addition processing, color change processing and mirror transformation processing.

[0009] Optionally, in the third implementation manner of the first aspect of the present application, the initial old people expression recognition model is constructed based on multi-level feature fusion, comprising the steps of: constructing a face key region feature extraction module based on spatial attention, wherein the face key region feature extraction module comprises a convolutional neural network module for extracting a face key region graph in an image, a residual network module for extracting a face global feature graph in the image, and a spatial attention module taking the face key region graph and the face global feature graph as inputs and outputting face key region features in the image; constructing a time domain dynamic feature extraction module based on time attention, wherein the time domain dynamic feature extraction module comprises a time attention module and a recurrent neural network (RNN), the time attention module is used to generate an attention weight matrix of the face key region features in the time domain and weight the face key region features, and the recurrent neural network (RNN) is used to receive the weighted face key region features and output face local dynamic features; and constructing a feature fusion module and a classifier after the face key region feature extraction module and the time domain dynamic feature extraction module to obtain the initial old people expression recognition model.

[0010] Optionally, in the fourth implementation manner of the first aspect of the present application, the time attention module comprises a max-pooling layer, an average-pooling layer, a shared fully connected layer, and a summation activation layer.

[0011] Optionally, in the fifth implementation manner of the first aspect of the present application, the feature fusion module takes the face key region features and the face local dynamic features as inputs, and outputs a spatio-temporal fusion feature MF t , wherein the face key region features are represented as F t , and corresponding features K F , Q F , and V F are obtained through linear transformation; the face local dynamic features are represented as Dt, and corresponding features K D , Q D , and V D are obtained through linear transformation; and the spatio-temporal fusion feature MF t = II aμ (CM DF +CM Fd ), wherein, wherein d is the dimension of K F or K D .

[0012] Optionally, in the sixth implementation form of the first aspect of the present application, the training set is used to train the initial elderly expression recognition model, the parameters of the initial elderly expression recognition model are adjusted, and after being tested by the test set, a trained elderly expression recognition model is obtained, including the steps of: using a random initialization method to assign initial values to all parameters in the initial elderly expression recognition model; randomly selecting a sequence sample from the training set and inputting it into the initial elderly expression recognition model for training, calculating a prediction result by forward propagation, and then calculating a loss value according to a FocalLoss function; using a back propagation algorithm to update the parameters of the network so that the loss value gradually decreases; repeating the above training steps until a predetermined training round is reached, and obtaining a trained initial elderly expression recognition model; using the test set to evaluate the trained initial elderly expression recognition model, adjusting the hyperparameters according to the evaluation result, and obtaining the trained elderly expression recognition model.

[0013] The second aspect of the present application provides an elderly expression recognition device, comprising: an image processing module, configured to perform expansion and labeling processing on acquired elderly expression images to obtain an elderly expression dataset, the elderly expression dataset comprising a plurality of sequence samples each consisting of a plurality of consecutive pictures, each sequence sample corresponding to a different expression category label, the expression category label including pain, fear, sadness, anger, disgust, happiness and neutrality; a data division module, configured to divide the elderly expression dataset into a training set and a test set according to a predetermined proportion, the training set and the test set each comprising sequence samples corresponding to the seven expression category labels of pain, fear, sadness, anger, disgust, happiness and neutrality; a model construction module, configured to construct an initial elderly expression recognition model based on multi-level feature fusion; a model training module, configured to train the initial elderly expression recognition model through the training set, adjust the parameters of the initial elderly expression recognition model, and obtain a trained elderly expression recognition model after being tested by the test set; and a recognition module, configured to input consecutive T+1 frames of elderly expression images at the current t time and before the t-T time into the trained elderly expression recognition model as input, and output an elderly facial expression recognition result at the current t time, T being an integer greater than or equal to 2.

[0014] The third aspect of the present application provides an elderly expression recognition device, comprising: a memory and at least one processor, the memory storing computer readable instructions, and the memory and the at least one processor being interconnected by a circuit; the at least one processor invoking the computer readable instructions in the memory to enable the elderly expression recognition device to perform each step of the above-mentioned elderly expression recognition method.

[0015] The fourth aspect of the present application provides a computer readable storage medium, wherein computer readable instructions are stored in the computer readable storage medium, and when the computer readable instructions are run on a computer, the computer is caused to perform each step of the method for recognizing expressions of the elderly as described above.

[0016] Beneficial effects: the present application provides a method for recognizing expressions of the elderly, first constructs an expression dataset of the elderly, then extracts facial key region features through a spatial attention mechanism, extracts facial local dynamic features using a temporal attention mechanism, finally performs feature fusion on multi-level features, and completes expression recognition of the elderly based on the fused features. Since the generation of an expression is a gradual process, expression-related information is not only reflected in facial features at a certain moment, but also can be expressed through some subtle dynamic features of facial states within a short period of time; and the facial action intensity of the elderly when expressing emotions may not be too large, therefore, the static features at the current moment and the dynamic features in the time domain are obtained, which can make up for the deficiencies of a single type of feature and obtain more information; the expression recognition model of the elderly constructed by the present application can obtain more detailed facial information based on cross-attention feature fusion, so as to realize more accurate classification of facial expressions of the elderly. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 The flowchart of the method for recognizing expressions of the elderly provided by the embodiment of the present application.

[0018] Figure 2 The structure diagram of the facial key region feature extraction module based on spatial attention.

[0019] Figure 3 The structure diagram of the time domain dynamic feature extraction module based on temporal attention.

[0020] Figure 4 The structure diagram of the initial expression recognition model of the elderly.

[0021] Figure 5 The structure diagram of the feature fusion module based on cross-attention.

[0022] Figure 6 The structure diagram of the expression recognition device of the elderly provided by the present application.

[0023] Figure 7 The structure diagram of the expression recognition device of the elderly provided by the present application. DETAILED DESCRIPTION

[0024] The embodiments of the present application provide an old person expression recognition method, device and equipment and a storage medium. The terms "first", "second", "third", "fourth" and the like (if any) in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" or "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or equipment including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or equipment.

[0025] Please refer to Figure 1 , Figure 1 A flow chart of an old person expression recognition method provided by the present application is shown in the figure, which includes the following steps:

[0026] S10, the acquired old person expression image is expanded and labeled to obtain an old person expression data set, the old person expression data set includes a plurality of sequence samples composed of continuous multiple pictures, each sequence sample corresponds to a different expression category label, and the expression category label includes pain, fear, sadness, anger, disgust, happiness and neutrality.

[0027] Specifically, in the old person group, sample video data is collected by photographing each individual in multiple facial poses, including front face, side face, half side face, upper side face and lower side face. In this embodiment, the old person group covers males and females aged 60-100 years old, and each individual is ensured to show the seven expression categories of pain, fear, sadness, anger, disgust, happiness and neutrality during photographing, so as to improve the diversity and representativeness of the data set. In this embodiment, the front face refers to the angle of facing the lens, the side face is 45 degrees of side turning, the half side face is slight side turning, for example, 20 degrees of side turning, and the like. The upper side face and the lower side face are respectively inclined upward or downward by a certain angle, for example, 20 degrees upward or downward, and the like. During shooting, a high-definition camera is used, and uniform light and simple background are ensured to reduce external interference. Each old person is recorded in multiple facial poses, each pose is kept for a few seconds for subsequent frame extraction, and then the video clarity, expression naturalness and pose accuracy are checked, and unqualified videos are removed.

[0028] After obtaining the sample video data, the sample video data is converted into a sequence sample composed of continuous multiple frames according to the frame number, for example, all frames of the same video are arranged in chronological order to form a continuous image sequence by using video processing software or programming tools such as OpenCV; then the sequence sample is subjected to data enhancement processing to obtain an expanded sequence sample, the data enhancement processing including one or more of illumination change processing, noise addition processing, color change processing and mirror transformation processing, but not limited thereto. The expanded sequence sample is labeled by the cameraman or the elderly themselves during shooting to obtain an elderly expression dataset labeled with expression category labels, the expression category labels including pain, fear, sadness, anger, disgust, happiness and neutrality.

[0029] The embodiment ensures the diversity and accuracy of the elderly expression dataset through strict screening and quality control, and provides a solid foundation for subsequent training of the elderly expression recognition model; the data enhancement processing can simulate image changes in various actual scenarios, so that the model learns more feature changes during the training process, thereby improving its generalization ability in complex environments; manual labeling ensures the accuracy and consistency of the labels, which helps to improve the recognition accuracy of the model; the high-quality elderly expression dataset can be used for research and application in multiple fields such as automatic recognition of elderly expressions, sentiment analysis, human-computer interaction, health monitoring, etc., and promotes the development and innovation of related technologies.

[0030] S20, dividing the elderly expression dataset into a training set and a test set according to a predetermined proportion, the training set and the test set each including sequence samples corresponding to seven expression category labels of pain, fear, sadness, anger, disgust, happiness and neutrality;

[0031] In the embodiment, in order to ensure the randomness and fairness of the division result, the entire elderly expression dataset needs to be shuffled first, which can be realized by using a random function in a programming language or a data processing tool; then the shuffled elderly expression dataset is divided into a training set and a test set according to a determined division proportion, common proportions being 70% training set + 30% test set or 80% training set + 20% test set; during the division, it is ensured that the training set and the test set each contain sequence samples of all seven expression category labels, so as to maintain the diversity and representativeness of the dataset; the divided training set and test set are respectively stored in different files or databases for subsequent use. By dividing the dataset into a training set and a test set, the embodiment can ensure that the model does not contact the data in the test set during the training process, thereby avoiding the occurrence of overfitting, which helps to improve the generalization ability and accuracy of the model; the test set is a dataset independent of the training set, which can be used to evaluate the performance of the model, and by comparing the performance of the model on the test set with the performance on the training set, the generalization ability and accuracy of the model can be objectively evaluated.

[0032] S30. Construct an initial facial expression recognition model for the elderly based on multi-level feature fusion;

[0033] This embodiment first constructs as follows: Figure 2 The facial key region feature extraction module based on spatial attention shown includes a convolutional neural network module for extracting facial key region maps from an image, a residual network module for extracting global facial feature maps from an image, and a spatial attention module that takes the facial key region maps and global facial feature maps as input and outputs facial key region features from the image. Specifically, as shown... Figure 2 As shown, for the image at time t, a convolutional neural network module is used to extract the facial key region map; a residual network module is used to extract the global facial feature map; and a spatial attention module is constructed based on the multi-head attention mechanism of the transformer to explore the connections between various key facial regions. The facial key region map and the global facial feature map are used as inputs to the attention module to obtain the facial key region features in the image at time t. For example, the facial key region map extracted by the convolutional neural network module is represented as I. t After a linear transformation, Q corresponds to... I Corresponding to the attention of the bulls The global face feature map extracted by the residual network module is represented as G. t After a linear transformation, we have K. G and V G Corresponding to the attention of the bulls The facial key region feature map and facial key region features obtained after the transformer-based multi-head attention module are respectively labeled with F... t and F i It indicates that, among them, F t =MA(α) t G t ), α t =concat(α) t1 , ..., α tn ), (i = 1, ..., n), d i for softmax represents the softmax function calculation, concat represents vector concatenation, and MA represents weighted calculation using multi-head attention weights. In this embodiment, the face key region feature map is a two-dimensional matrix used to visualize the attention weights, while the face key region feature is a one-dimensional vector (or matrix) used to represent the weighted integrated information of the input sequence; both are output results of the multi-head attention module, but their purposes and representations are different. The face key region feature map can be regarded as an intermediate representation of the face key region feature in the calculation process, which helps the model visualize the distribution of attention weights. The face key region feature is the key information finally output by the multi-head attention module.

[0034] Then, the embodiment constructs a time-domain dynamic feature extraction module based on time attention as shown in Figure 3 The time-domain dynamic feature extraction module includes a time attention module and a recurrent neural network (RNN). The time attention module is used to generate an attention weight matrix of the face key region feature in the time domain and to weight the face key region feature. The recurrent neural network (RNN) is used to receive the weighted face key region feature and output a face local dynamic feature. The time attention module includes a max-pooling layer, an average-pooling layer, a shared fully connected layer, and a summation activation layer. Specifically, as shown in Figure 3 The face key region feature F i based on multiple frames of images (i.e., sequence samples) is input into the time attention module and the recurrent neural network (RNN) to focus on the subtle change features of each key region over time in the sequence sample, and to extract a face local dynamic feature. As an example, in this embodiment, the face key region feature set of the sequence sample composed of T+1 time points {t-T,..., t-1, t} is F = {F i |i = t, t-1,..., t-T}, the output obtained by the average-pooling layer is The output obtained by the max-pooling layer is The and are input into the shared fully connected layer MLP and the summation activation layer to obtain an attention weight matrix W Each F i is weighted using the attention weight, and then input into the recurrent neural network (RNN) as an input to obtain a face local dynamic feature D t .

[0035] After the face key region feature extraction module and the time-domain dynamic feature extraction module, a feature fusion module and a classifier are constructed to obtain an initial elderly expression recognition model as shown in Figure 4 As shown in FIG. 8, the initial elderly expression recognition model includes a face key region feature extraction module, a time-domain dynamic feature extraction module, a feature fusion module, and a classifier. The face key region feature extraction module is used to extract a face key region feature FFigure 4 and Figure 5 As shown in the figure, the feature fusion module takes face key region features and face local dynamic features as input, and outputs spatio-temporal fusion features MF t , which are represented as F t , and corresponding features K F , Q F , V F obtained through linear transformation; the face local dynamic features are represented as Dt, and corresponding features K D , Q D , V D obtained through linear transformation; the spatio-temporal fusion features MF t = H up (CM DF + CM Fd ), wherein, wherein d is the dimension of K F or K D . In this embodiment, since the generation of an expression is a gradual process, expression-related information is not only embodied in the face features at a certain moment, but also can be expressed through some subtle dynamic features of the face state in a short period of time; and the face movement intensity of the elderly when expressing emotions can not be too large, therefore, the static features at the current moment and the dynamic features in the time domain are obtained to make up for the deficiency of a single type of feature and obtain more information; the feature fusion module based on cross attention can realize the connection and integration of information between different types of features. Finally, the spatio-temporal fusion features MF t are input into a classifier to obtain the expression recognition result of the elderly.

[0036] S40, training the initial expression recognition model of the elderly through the training set, adjusting the parameters of the initial expression recognition model of the elderly, and obtaining a trained expression recognition model of the elderly after testing by the test set;

[0037] In the embodiment, first, it is ensured that the training set and the test set have been divided according to a predetermined proportion, and both contain sequence samples corresponding to the seven expression category labels of pain, fear, sadness, anger, disgust, happiness and neutral; then the hyperparameters in the training process are determined, such as the learning rate, the batch size, the number of epochs, and a suitable loss function (such as cross-entropy loss) and optimization algorithm (such as stochastic gradient descent, SGD) are selected; then the training set is input into the initial elderly expression recognition model, the loss value is calculated by forward propagation, the model parameters are updated according to the loss value by back propagation, and the process of forward propagation and back propagation is repeated until the predetermined number of epochs or the loss value converges; finally, the trained model is tested using the test set to evaluate the generalization ability and accuracy of the model, and the model is further optimized and fine-tuned according to the test results to obtain the trained elderly expression recognition model. Through reasonable training steps and parameter adjustment, the trained elderly expression recognition model with superior performance and strong generalization ability can be obtained.

[0038] S50, the current t time and the previous t-T time of the continuous T+1 frames of elderly expression images are taken as the input of the trained elderly expression recognition model, and the recognition result of the current t time of the elderly facial expression is output, and T is an integer greater than or equal to 2.

[0039] In the embodiment, T+1 frames of elderly expression images are arranged in time sequence to form an input sequence, which will be used as the input of the trained elderly expression recognition model. The model outputs the recognition result of the current t time of the elderly facial expression, which is usually a probability distribution representing the possibility of each expression category. The expression category with the highest probability is selected as the final recognition result. Since the expression is dynamic, not just static images can fully express, the embodiment uses consecutive frames as input, and the model can use temporal context information to improve the accuracy of expression recognition of the current frame. This helps to reduce misidentification caused by noise or blur in single-frame images, and also captures the dynamic process of expression changes, thereby more accurately identifying the current expression state.

[0040] The elderly expression recognition method in the embodiment of the application is described above, and the elderly expression recognition device in the embodiment of the application is described below. Please refer to Figure 6 An embodiment of the elderly expression recognition device in the embodiment of the application includes:

[0041] An image processing module 10 is configured to perform expansion and labeling processing on the acquired facial expression images of the elderly to obtain an elderly facial expression dataset, the elderly facial expression dataset including a plurality of sequence samples each composed of a plurality of continuous images, each sequence sample corresponding to a different expression category label, and the expression category label including pain, fear, sadness, anger, disgust, happiness and neutrality.

[0042] A data division module 20 is configured to divide the elderly facial expression dataset into a training set and a test set according to a predetermined proportion, and the training set and the test set each include sequence samples corresponding to the seven expression category labels of pain, fear, sadness, anger, disgust, happiness and neutrality.

[0043] A model construction module 30 is configured to construct an initial elderly facial expression recognition model based on multi-level feature fusion.

[0044] A model training module 40 is configured to train the initial elderly facial expression recognition model through the training set, adjust parameters of the initial elderly facial expression recognition model, and obtain a trained elderly facial expression recognition model after testing by the test set.

[0045] An identification module 50 is configured to take the continuous T+1 frames of facial expression images of the elderly at the current t time and the previous t-T time as input of the trained elderly facial expression recognition model, and output the facial expression recognition result of the elderly at the current t time, T being an integer greater than or equal to 2.

[0046] Based on the same idea as the method in the above embodiment, the device provided by the present application can implement the method of the above embodiment. For the convenience of description, only the part related to the embodiment of the present application is shown in the structure diagram of the device embodiment, and the person skilled in the art can understand that the structure diagram does not constitute a limitation to the device, and the device can include more or less modules than the diagram, or combine certain modules, or arrange different modules.

[0047] Figure 6 The elderly facial expression recognition device in the embodiment of the present application is described in detail from the perspective of modular functional entities, and the elderly facial expression recognition device in the embodiment of the present application is described in detail from the perspective of hardware processing.

[0048] Figure 7Fig. 1 is a structural schematic diagram of an embodiment of an expression recognition device for the elderly provided by the present application. The expression recognition device for the elderly 100 can have great differences due to different configurations or performances, and can include one or more central processing units (CPUs) 11 (for example, one or more processors) and a memory 12, one or more storage media 13 (for example, one or more mass storage devices) storing application programs 133 or data 132. The memory 12 and the storage media 13 can be temporary storage or persistent storage. The programs stored in the storage media 13 can include one or more modules (not shown in the figure), each of which can include a series of instruction operations in the expression recognition device for the elderly 100. Furthermore, the processor 11 can be configured to communicate with the storage media 13 to execute the series of instruction operations in the storage media 13 on the expression recognition device for the elderly 100.

[0049] The expression recognition device for the elderly 100 can also include one or more power supplies 14, one or more wired or wireless network interfaces 15, one or more input / output interfaces 16, and / or one or more operating systems 131, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art can understand that the expression recognition device for the elderly 100 can include more or fewer components than those shown, or some components can be combined, or different components can be arranged. Figure 7 The device structure shown does not constitute a limitation on the expression recognition device for the elderly 100, and can include more or fewer components than those shown, or some components can be combined, or different components can be arranged.

[0050] The present application also provides a computer readable storage medium, which can be a non-volatile computer readable storage medium or a volatile computer readable storage medium. The computer readable storage medium stores instructions, which, when executed on a computer, cause the computer to perform the steps of the expression recognition method for the elderly.

[0051] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system or device, unit can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0052] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the entire or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0053] The above-described embodiments are merely used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some of the technical features; and these modifications or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of recognizing an expression of an elderly person, characterized by, The method comprises the steps of: performing expansion and labeling processing on the obtained old person expression images to obtain an old person expression dataset, the old person expression dataset comprising a plurality of sequence samples each composed of a plurality of continuous images, each sequence sample corresponding to a different expression category label, the expression category label comprising pain, fear, sadness, anger, disgust, happiness and neutrality; dividing the old person expression dataset into a training set and a test set according to a predetermined proportion, the training set and the test set each comprising sequence samples corresponding to the seven expression category labels of pain, fear, sadness, anger, disgust, happiness and neutrality; constructing a face key region feature extraction module based on spatial attention, the face key region feature extraction module comprising a convolutional neural network module for extracting a face key region map in an image, a residual network module for extracting a global feature map of a human face in the image, and a spatial attention module taking the face key region map and the global feature map of the human face as input and outputting a face key region feature in the image; constructing a time domain dynamic feature extraction module based on time attention, the time domain dynamic feature extraction module comprising a time attention module and a recurrent neural network (RNN), the time attention module being configured to generate an attention weight matrix of the face key region feature in the time domain and to weight the face key region feature; the recurrent neural network (RNN) being configured to receive the weighted face key region feature and output a face local dynamic feature; constructing a feature fusion module and a classifier after the face key region feature extraction module and the time domain dynamic feature extraction module to obtain an initial old person expression recognition model; training the initial old person expression recognition model through the training set, adjusting parameters of the initial old person expression recognition model, and obtaining a trained old person expression recognition model after testing through the test set; inputting consecutive T+1 frames of old person expression images at a current time t and a previous time t-T as input of the trained old person expression recognition model, and outputting an old person face expression recognition result at the current time t, T being an integer greater than or equal to 2.

2. The method of claim 1, wherein, The method comprises the steps of: collecting sample video data by taking pictures of each individual in a plurality of facial poses in an old person group, the facial poses comprising a front face, a side face, a half side face, an upper side face and a lower side face; converting the sample video data into sequence samples composed of a plurality of continuous frames according to the number of frames; performing data enhancement processing on the sequence samples to obtain expanded sequence samples; performing labeling processing on the expanded sequence samples to obtain an old person expression dataset labeled with expression category labels, the expression category labels comprising pain, fear, sadness, anger, disgust, happiness and neutrality. 3.The method of claim 1, wherein, In the step of performing data enhancement processing on the sequence samples, the data enhancement processing comprises one or more of illumination change processing, noise addition processing, color change processing and mirror transformation processing. 4.The method of claim 1, wherein, The time attention module comprises a max pooling layer, an average pooling layer, a shared fully connected layer, and a summation activation layer. 5.The method of claim 1, wherein, The feature fusion module takes face key region features and face local dynamic features as inputs to obtain spatio-temporal fusion features MF t as outputs, wherein the face key region features are represented as F t , and corresponding features K F , Q F , and V F are obtained through linear transformation; the face local dynamic features are represented as Dt, and corresponding features K D , Q D , and V D are obtained through linear transformation; and the spatio-temporal fusion features are represented as MF , wherein , , and d is the dimension of K F or K D . 6.The method of claim 1, wherein, The initial elderly expression recognition model is trained through the training set, parameters of the initial elderly expression recognition model are adjusted, and after being tested through the test set, a trained elderly expression recognition model is obtained, including the following steps: Initial values are assigned to all parameters in the initial elderly expression recognition model using a random initialization method; Sequence samples are randomly selected from the training set and input into the initial elderly expression recognition model for training, a prediction result is calculated through forward propagation, and then a loss value is calculated according to a Focal Loss function; The parameters of the network are updated using a back propagation algorithm, so that the loss value gradually decreases; The above training steps are repeated until a predetermined training round is reached, and a trained initial elderly expression recognition model is obtained; The trained initial elderly expression recognition model is evaluated using a test set, and hyperparameters are adjusted according to the evaluation result to obtain a trained elderly expression recognition model.

7. An elderly person expression recognition device, characterized by, It comprises: An image processing module is configured to expand and label the acquired elderly expression images to obtain an elderly expression dataset, wherein the elderly expression dataset comprises a plurality of sequence samples each composed of a plurality of continuous images, and each sequence sample corresponds to a different expression category label, and the expression category label comprises pain, fear, sadness, anger, disgust, happiness, and neutrality. A data division module is configured to divide the elderly expression dataset into a training set and a test set according to a predetermined proportion, and the training set and the test set each comprise sequence samples corresponding to the seven expression category labels of pain, fear, sadness, anger, disgust, happiness, and neutrality. A model construction module is configured to construct a face key region feature extraction module based on spatial attention, wherein the face key region feature extraction module comprises a convolutional neural network module configured to extract a face key region map from an image, a residual network module configured to extract a global face feature map from the image, and a spatial attention module configured to take the face key region map and the global face feature map as inputs and output face key region features in the image; a time domain dynamic feature extraction module based on time attention, wherein the time domain dynamic feature extraction module comprises a time attention module configured to generate an attention weight matrix of the face key region features in the time domain and weight the face key region features, and a recurrent neural network (RNN) configured to receive the weighted face key region features and output local face dynamic features. A feature fusion module and a classifier are constructed after the face key region feature extraction module and the time domain dynamic feature extraction module to obtain an initial elderly expression recognition model. A model training module is configured to train the initial elderly expression recognition model through the training set, adjust parameters of the initial elderly expression recognition model, and obtain a trained elderly expression recognition model after being tested through the test set. The recognition module is configured to take the T+1 continuous frames of the old person expression images at the current time t and the previous time t-T as inputs of a trained old person expression recognition model, and output an old person facial expression recognition result at the current time t, where T is an integer greater than or equal to 2.

8. An elderly person expression recognition device characterized by comprising: comprising a memory and at least one processor, the memory having computer readable instructions stored therein; The at least one processor invokes the computer readable instructions in the memory to perform the steps of the old person expression recognition method according to any one of claims 1-6.

9. A computer-readable storage medium having stored thereon computer-readable instructions, wherein, The computer readable instructions, when executed by the processor, implement the steps of the old person expression recognition method according to any one of claims 1-6.

Images