A real-time prediction method and system for elderly people's emotions based on body posture recognition

Abstract

The present invention discloses a real-time prediction method and system for the emotions of the elderly based on body posture recognition. First, the body posture of the video is extracted using a body posture neural network, and the body posture is input into a high-dimensional feature neural network to extract the high-dimensional features of the video. Then, the emotional features of the video are calculated, and the high-dimensional features of the video are combined with the emotional features to obtain the combined features of the video. Finally, the combined features of the video are input into an emotion classifier to predict the emotions of the characters in the video. The present invention recognizes the emotions of the characters by extracting the body posture of the characters in the video data. It does not require the facial expressions of the characters to be clearly captured. Therefore, the present invention has lower requirements for video acquisition equipment and better universality. In addition, the present invention also considers the temporal nature of the video data when performing character emotion recognition, applies the motion characteristics of the characters to the character emotion recognition process, and improves the accuracy of character emotion recognition through body posture.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for real-time prediction of elderly people's emotions based on posture recognition. Background Art

[0002] With the continuous development of the internet and multimedia technologies, and the widespread adoption of digital devices and high-capacity storage devices, massive amounts of image and video data are generated daily. As the acquisition of image and video information becomes easier, how to utilize this multimedia data has become a worthy topic. Inferring the emotions of people in videos based on video data is another application of multimedia information.

[0003] Emotions play a crucial role in our daily lives. Our perception of others' emotions helps us understand their behavior and decide how to interact with them. Because emotions play such a crucial role, many applications, such as treatment and rehabilitation, therapy, gaming and entertainment, security and law enforcement, shopping, and human-computer interaction, will benefit from emotion recognition. Furthermore, with the increasing number of elderly people in my country, elderly care has received widespread attention. Understanding the emotional state of the elderly in real time will facilitate the development of appropriate management measures. However, manually monitoring the mental well-being of the elderly in real time is unrealistic, and emotion recognition can effectively address this issue.

[0004] In recent years, the development of deep learning has led to the emergence of emotion recognition technologies. These technologies primarily input images containing people into a trained neural network model and output an emotion category. However, due to the limited or insufficient character features considered, inaccurate emotion predictions are prone to occur.

[0005] One of the current existing technologies is the patented "Method and device for recognizing human emotions in videos, computer equipment, and medium." This technology includes: 1. Obtaining an image containing a human face from the video to be recognized, N frames of images extracted from the video to be recognized at predetermined time intervals, dividing the video to be recognized into M segments and randomly sampling at least one frame from each segment to obtain at least two frames of images, and sound spectrograms corresponding to the above images; 2. Extracting facial emotion feature vectors from the image containing the human face, extracting frame emotion feature vectors from the N frames of images, extracting video emotion feature vectors from at least two frames of images, and extracting sound emotion feature vectors from the sound spectrogram; 3. Performing feature fusion on the above four feature vectors to obtain a multi-modal information feature vector; 4. Calling a pre-trained human emotion recognition model to recognize the multi-modal information feature vector to obtain a recognition result. The disadvantage of this technology is that it requires a complete facial image in the video and correct audio information, which is difficult to guarantee in specific practical applications.

[0006] The second existing technology is the patented "Micro-expression Fitting Method and System Based on Multiple Feature Points." This technology includes: 1. Obtaining a reference image and an emotional image, where the reference image refers to an image without a stimulus, and the emotional image refers to an image with a predetermined stimulus; 2. Inputting the reference image and the emotional image into a facial detection neural network for facial detection to obtain a reference facial image and an emotional facial image, respectively; 3. Inputting the reference facial image and the emotional facial image into a facial feature point labeling network for facial feature point extraction to obtain reference feature points and emotional feature points, respectively; 4. Calculating facial micro-feature vectors based on the positional information of the reference feature points and the positional information of the emotional feature points, and outputting facial micro-feature vectors that meet a predetermined threshold as facial motion units; 5. Fitting the facial motion units using a recursive neural network model to obtain predicted emotional features. A drawback of this technology is that emotion recognition requires a facial image of the corresponding person when they are calm, and the image used for recognition also requires a complete face, which is difficult to meet in practical applications.

[0007] The third existing technology is the patented "Speech Emotion Recognition Method, Apparatus, Computer Equipment and Storage Medium". This technology includes: 1. Obtaining the customer's speech data to be identified; 2. Extracting features from the customer's speech data to obtain a speech feature set; 3. Combining and splicing the speech features in the speech feature set to obtain a speech feature vector corresponding to the customer's speech data; 4. Recognizing the speech feature vector using a trained speech emotion recognition model to obtain the speech emotion category corresponding to the customer's speech data. The disadvantage of this technology is that speech information is required to recognize character emotions. However, there are many clips in video data that do not contain character speech information, so it cannot be well applied to video and image data. Summary of the Invention

[0008] The present invention aims to overcome the shortcomings of existing methods by proposing a real-time emotion prediction method and system for the elderly based on body posture recognition. The main problems addressed by this invention are how to predict a person's current emotion by extracting their body posture features from video data, even in the absence of facial image or voice information, and how to make more accurate emotion predictions by using the person's motion characteristics in the video.

[0009] In order to solve the above problems, the present invention proposes a real-time prediction method for elderly people's emotions based on body posture recognition, which includes:

[0010] Processing the Ewalk dataset includes filtering the data in the dataset and converting it into the same size for training purposes, thus forming a video dataset that includes video data and corresponding human posture and emotion labels.

[0011] Training a TP-Net network using the video data and the corresponding human postures in the video dataset to form a human posture neural network for extracting human postures in video frames;

[0012] Using the human body postures in the video dataset to train an LSTM network to form a high-dimensional feature neural network for extracting high-dimensional features of human body postures, and then using the human body postures in the video dataset to input the trained network to obtain high-dimensional features of each video in the video dataset;

[0013] Calculating the posture features and motion features of each frame in each video using the human postures in the video dataset, then taking the vector obtained by concatenating the posture features and motion features as the emotion features corresponding to each frame, and taking the average of the emotion features corresponding to all frames in any video as the emotion feature of the video, thereby forming the emotion feature of each video in the video dataset;

[0014] The high-dimensional features of each video are concatenated with the emotional features of each video to obtain the combined features of each video, which together with the emotional labels corresponding to the video data in the video dataset form a training set for training the classifier to form an emotional classifier;

[0015] The video to be processed is input, the human posture of the video to be processed is extracted using the human posture neural network, the human posture is input into the high-dimensional feature neural network to extract the high-dimensional features of the video to be processed, and then the emotional features of the video to be processed are calculated, and the high-dimensional features of the video to be processed are combined with the emotional features to obtain the combined features of the video to be processed, and finally the combined features of the video to be processed are input into the emotion classifier to predict the emotions of the characters in the video to be processed.

[0016] Preferably, the human body posture corresponding to the video data is represented by 16 three-dimensional coordinates, each coordinate represents the position of an important part of the human body in three-dimensional space, that is, a posture is a 48-dimensional vector; the 16 important parts are the head, neck, left shoulder, right shoulder, left hand joint, right hand joint, left hand, right hand, the middle of the upper body, the bottom center of the upper body, the connection between the left leg and the upper body, the connection between the right leg and the upper body, the left leg joint, the right leg joint, the left foot, and the right foot.

[0017] Preferably, there are four emotion labels corresponding to the video data, namely happy, sad, angry and calm.

[0018] Preferably, the high-dimensional feature neural network is specifically:

[0019] The high-dimensional feature neural network is trained using an encoding-decoding structure, with one LSTM network as the encoder and another LSTM network as the decoder.

[0020] The input of the encoder is the human posture extracted from the video clip, that is, a 48-dimensional vector, and the output is the high-dimensional feature of the human posture; the input of the decoder is the high-dimensional feature output of the encoder, and the output of the decoder is a 48-dimensional vector, that is, the human posture;

[0021] The optimization of the encoder and decoder is to calculate the L2 norm loss between the 48-dimensional vector output by the decoder and the 48-dimensional vector input to the encoder, and then perform feedback propagation to optimize the encoder and decoder; suppose the input of the encoder is a vector P, where the i-th component is represented by p i The decoder output is a vector Q, whose i-th component is represented by q i Indicates that the L2 norm calculation formula between them is as follows:

[0022]

[0023] Preferably, the method of calculating the posture features of each frame in each video segment by using the human postures in the video dataset is as follows:

[0024] There are 12 posture features in total, which are divided into four categories: volume, angle, distance and area. The volume refers to the volume of the outer bounding box of the human body posture in three-dimensional space. The angles include the angle formed by the two shoulders at the neck, the angle formed by the neck and the left shoulder at the right shoulder, the angle formed by the neck and the right shoulder at the left shoulder, and the angle formed by the head and the center of the upper body at the neck. The distance includes the distance between the center of the bottom of the upper body and the right hand, left hand, right foot, left foot, and the distance between the two feet. The area includes the area of ​​the triangle formed by the hands and the neck, and the area of ​​the triangle formed by the center of the bottom of the upper body and the two feet.

[0025] The calculations of volume, angle, distance and area are all operations between three-dimensional coordinates.

[0026] Preferably, the method of calculating the motion features of each frame in each video segment by using the human body posture in the video dataset is as follows:

[0027] There are 10 motion features in total, which are divided into two categories: movement speed and movement acceleration. The movement speed includes the movement speed of the five parts of the body: hands, feet, and head. The movement acceleration also includes the movement acceleration of the five parts of the body: hands, feet, and head. The movement speed is obtained by calculating the transformation of the coordinates of the human body posture corresponding to two frames of images in the video, while the movement acceleration is calculated by the change in movement speed.

[0028] Accordingly, the present invention also provides a real-time prediction system for elderly people's emotions based on body posture recognition, comprising:

[0029] A training data construction unit is used to process the Ewalk dataset, including filtering the data in the dataset and processing it into the same size for the convenience of training, thereby forming a video dataset including video data and human posture and emotion labels corresponding to the video data;

[0030] A neural network and classifier training unit is used to train a TP-Net network using the video data and the corresponding human postures in the video data set to form a human posture neural network for extracting human postures in video frames; train an LSTM network using the human postures in the video data set to form a high-dimensional feature neural network for extracting high-dimensional features of human postures, and then input the human postures in the video data set into the trained network to obtain high-dimensional features of each video in the video data set; calculate the posture features and motion features of each frame in each video using the human postures in the video data set, and then use the vector obtained by splicing the posture features and motion features as the emotional features corresponding to each frame, and take the average of the emotional features corresponding to all frames in any video as the emotional features of the video, thereby forming the emotional features of each video in the video data set; splice the high-dimensional features of each video with the emotional features of each video to obtain the combined features of each video, and form a training set together with the emotional labels corresponding to the video data in the video data set to train the classifier to form an emotional classifier;

[0031] The emotion recognition unit is used to input the video to be processed, use the human posture neural network to extract the human posture of the video to be processed, input the human posture into the high-dimensional feature neural network to extract the high-dimensional features of the video to be processed, then calculate the emotional features of the video to be processed, and combine the high-dimensional features of the video to be processed with the emotional features to obtain the combined features of the video to be processed, and finally input the combined features of the video to be processed into the emotion classifier to predict the emotions of the characters in the video to be processed.

[0032] The implementation of the present invention has the following beneficial effects:

[0033] The present invention recognizes human emotions by extracting the human posture of the human in video data. It has low requirements on the quality of video data and does not require the human facial expression to be clearly captured. It only needs to capture the human posture. Therefore, the present invention has lower requirements on video acquisition equipment and has better universality. In addition, the present invention also considers the temporal sequence in the video data when performing human emotion recognition, and applies the human movement characteristics to the human emotion recognition process, thereby improving the accuracy of human emotion recognition through posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is an overall flow chart of a method for real-time prediction of elderly people's emotions based on posture recognition according to an embodiment of the present invention;

[0035] Figure 2 This is a structural diagram of a real-time prediction system for elderly people's emotions based on body posture recognition according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0037] Figure 1 This is a general flow chart of a method for real-time prediction of elderly people's emotions based on body posture recognition according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0038] S1, processing the Ewalk dataset, including filtering the data in the dataset and processing it into the same size for training purposes, to form a video dataset including video data and human posture and emotion labels corresponding to the video data;

[0039] S2, training a TP-Net network using the video data and the corresponding human postures in the video dataset to form a human posture neural network for extracting human postures in video frames;

[0040] S3, using the human body postures in the video dataset to train an LSTM network to form a high-dimensional feature neural network for extracting high-dimensional features of human body postures, and then using the human body postures in the video dataset to input the trained network to obtain high-dimensional features of each video in the video dataset;

[0041] S4, using the human body postures in the video dataset to calculate the posture features and motion features of each frame in each video segment, then concatenating the posture features and motion features to obtain a vector as the emotion feature corresponding to each frame, and taking the average of the emotion features corresponding to all frames in any video segment as the emotion feature of the video segment, thereby forming the emotion feature of each video segment in the video dataset;

[0042] S5, concatenating the high-dimensional features of each video with the emotional features of each video to obtain a combined feature of each video, and combining the combined features with the emotional labels corresponding to the video data in the video dataset to form a training set for training a classifier to form an emotional classifier;

[0043] S6, input the video to be processed, use the human posture neural network to extract the human posture of the video to be processed, input the human posture into the high-dimensional feature neural network to extract the high-dimensional features of the video to be processed, then calculate the emotional features of the video to be processed, and combine the high-dimensional features of the video to be processed with the emotional features to obtain the combined features of the video to be processed, and finally input the combined features of the video to be processed into the emotion classifier to predict the emotions of the characters in the video to be processed.

[0044] Step S1 is as follows:

[0045] The dataset uses the publicly available Ewalk (Emotion Walk) dataset, which contains video clips, human poses extracted from each frame, and the corresponding emotion labels for the characters in the video clips. Human poses are represented by 16 three-dimensional coordinates, each representing the position of a key body part in three-dimensional space. In other words, a pose is a 48-dimensional vector. The 16 key parts are the head, neck, left shoulder, right shoulder, left hand joint, right hand joint, left hand, right hand, middle of the upper body, bottom center of the upper body, left leg and upper body connection, right leg and upper body connection, left leg joint, right leg joint, left foot, and right foot. There are four emotion labels: happy, sad, angry, and calm.

[0046] Step S2 is as follows:

[0047] The human pose neural network is implemented using the TP-Net (Temporal PoseNet) network model. This neural network model receives a video as input and estimates a human pose for each frame in the video. The training data used for this neural network model is the video data in S1 and the human poses corresponding to the video data.

[0048] Step S3 is as follows:

[0049] The high-dimensional feature neural network is trained using an encoding-decoding structure, with an LSTM (Long Short-Term Memory) network as the encoder and another LSTM network as the decoder.

[0050] The input of the encoder is the human posture extracted from the video clip, that is, a 48-dimensional vector, and the output is the high-dimensional feature of the human posture; the input of the decoder is the high-dimensional feature output of the encoder, and the output of the decoder is a 48-dimensional vector, that is, the human posture;

[0051] The optimization of the encoder and decoder is to calculate the L2 norm loss between the 48-dimensional vector output by the decoder and the 48-dimensional vector input to the encoder, and then perform feedback propagation to optimize the encoder and decoder; suppose the input of the encoder is a vector P, where the i-th component is represented by p i The decoder output is a vector Q, whose i-th component is represented by q i Indicates that the L2 norm calculation formula between them is as follows:

[0052]

[0053] Step S4 is as follows:

[0054] Emotional features consist of two components: posture and movement. Considering posture is crucial, as posture can, to some extent, reveal a person's current emotional state. Furthermore, emotions can be expressed in movement, such as walking faster when angry. Therefore, considering movement is also crucial.

[0055] S4-1, in this embodiment, there are a total of 12 posture features. They can be divided into four categories: volume, angle, distance and area. The volume refers to the volume of the outer bounding box of the human body posture in three-dimensional space. The angles include the angle formed by the two shoulders at the neck, the angle formed by the neck and the left shoulder at the right shoulder, the angle formed by the neck and the right shoulder at the left shoulder, and the angle formed by the head and the center of the upper body at the neck. The distance includes the distance between the center of the bottom of the upper body and the right hand, left hand, right foot, left foot, and the distance between the two feet. The area includes the area of ​​the triangle formed by the hands and the neck, and the area of ​​the triangle formed by the bottom center of the upper body and the two feet. Note that the above-mentioned names such as the neck and right shoulder are all expressed in three-dimensional coordinates, so the above-mentioned calculations are only operations between three-dimensional coordinates.

[0056] For volume, this method takes the outermost points in the six directions of the human body posture (a total of six points in the positive and negative directions of the three coordinate axes xyz), a total of six points, and makes a plane parallel to the plane formed by the corresponding coordinate axes (for example, the point farthest in the positive and negative directions of the x-axis is made on a plane parallel to the plane formed by the y-axis and z-axis) to form a rectangular parallelepiped as the outer bounding box, and its volume is the required volume. For angles, such as the angle formed by the neck and the left shoulder at the right shoulder, in three-dimensional space, the right shoulder forms a three-dimensional vector with the neck and the left shoulder respectively, assuming they are a and b, a i and b iRepresent the corresponding components respectively, then the angle formed by the neck and left shoulder at the right shoulder is calculated as follows:

[0057]

[0058] For distance, it refers to the distance between two parts of the human body posture. These two parts are represented by three-dimensional coordinates. Suppose they are points A and B. A i and B i Represent the corresponding components respectively, then the distance calculation formula between the two parts is as follows:

[0059]

[0060] For area, it refers to the area of ​​a triangle formed by three points in the human body posture. These three points can form two vectors with one point as the starting point and the other two points as the end points. Assuming they are represented by a and b respectively, the calculation formula for the area is:

[0061]

[0062] S4-2, since this method is based on video for emotion recognition, it will have continuity in input, so the accuracy of emotion recognition can be enhanced by calculating motion features. In this embodiment, there are a total of 10 motion features, which are divided into two categories: movement speed and movement acceleration. The movement speed includes the movement speed of the five parts of the hands, feet and head, and the acceleration is also the movement acceleration of the five parts of the hands, feet and head. The movement speed is obtained by calculating the transformation of the coordinates in the human body posture corresponding to the two frames of images in the video, and the movement acceleration is calculated by the change in the movement speed. Assume that t represents the time difference between the two frames, P i Represents the three-dimensional coordinates of a certain part in the i-th frame, Represents the corresponding component, then the calculation formula for the moving speed of the i-th frame is as follows:

[0063]

[0064] The calculation formula for the i-th frame movement acceleration is as follows:

[0065]

[0066] In addition, the movement speed of the first frame is recorded as 0, and the movement acceleration of the first and second frames is recorded as 0.

[0067] S4-3, finally the posture features and motion features are concatenated to obtain a 22-dimensional vector as the emotion feature.

[0068] Step S5 is as follows:

[0069] The classifier is implemented using a two-layer fully connected network model. The high-dimensional features of each video obtained in S3 and the emotional features of each video obtained in S4 are combined to create a composite feature. The composite feature and the corresponding emotion label of each video serve as training data for the classifier. The classifier outputs four probability values ​​whose sum is 1, and the one with the highest probability is taken as the emotion represented by the current input feature.

[0070] Accordingly, the present invention also provides a real-time prediction system for elderly people's emotions based on body posture recognition, such as Figure 2 As shown, including:

[0071] A training data construction unit 1 is configured to process the Ewalk dataset, including filtering the data in the dataset and processing the data into the same size for the convenience of training, thereby forming a video dataset including video data and human posture and emotion labels corresponding to the video data;

[0072] The neural network and classifier training unit 2 is used to train the TP-Net network using the video data and the corresponding human postures in the video data set to form a human posture neural network for extracting human postures in video frames; train the LSTM network using the human postures in the video data set to form a high-dimensional feature neural network for extracting high-dimensional features of human postures, and then input the human postures in the video data set into the trained network to obtain high-dimensional features of each video in the video data set; calculate the posture features and motion features of each frame in each video using the human postures in the video data set, and then use the vector obtained by splicing the posture features and motion features as the emotional features corresponding to each frame, and take the average of the emotional features corresponding to all frames in any video as the emotional features of the video, thereby forming the emotional features of each video in the video data set; splice the high-dimensional features of each video with the emotional features of each video to obtain the combined features of each video, and form a training set together with the emotional labels corresponding to the video data in the video data set to train the classifier to form an emotional classifier;

[0073] The emotion recognition unit 3 is used to input the video to be processed, use the human posture neural network to extract the human posture of the video to be processed, input the human posture into the high-dimensional feature neural network to extract the high-dimensional features of the video to be processed, then calculate the emotional features of the video to be processed, and combine the high-dimensional features of the video to be processed with the emotional features to obtain the combined features of the video to be processed, and finally input the combined features of the video to be processed into the emotion classifier to predict the emotions of the characters in the video to be processed.

[0074] Therefore, the present invention recognizes the emotions of characters by extracting the body postures of characters in video data. It has low requirements for the quality of video data and does not require the clear capture of the facial expressions of characters. It only needs to capture the body posture of characters. Therefore, the present invention has lower requirements for video acquisition equipment and has better universality. In addition, the present invention also considers the temporal sequence in the video data when performing character emotion recognition, and applies the character's motion characteristics to the character's emotion recognition process, thereby improving the accuracy of character emotion recognition through body posture.

[0075] The above is a detailed introduction to a method and system for real-time prediction of emotions of the elderly based on body posture recognition provided by an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A real-time prediction method for elderly people's emotions based on body posture recognition, characterized in that: The method comprises: Processing the Ewalk dataset includes filtering the data in the dataset and converting it into the same size for training purposes, thus forming a video dataset that includes video data and corresponding human posture and emotion labels. Training a TP-Net network using the video data and the corresponding human postures in the video dataset to form a human posture neural network for extracting human postures in video frames; Using the human body postures in the video dataset to train an LSTM network to form a high-dimensional feature neural network for extracting high-dimensional features of human body postures, and then using the human body postures in the video dataset to input the trained network to obtain high-dimensional features of each video in the video dataset; Calculating the posture features and motion features of each frame in each video using the human postures in the video dataset, then taking the vector obtained by concatenating the posture features and motion features as the emotion features corresponding to each frame, and taking the average of the emotion features corresponding to all frames in any video as the emotion feature of the video, thereby forming the emotion feature of each video in the video dataset; The high-dimensional features of each video are concatenated with the emotional features of each video to obtain the combined features of each video, which together with the emotional labels corresponding to the video data in the video dataset form a training set for training the classifier to form an emotional classifier; Input a video to be processed, extract human postures from the video to be processed using the human posture neural network, input the human postures into the high-dimensional feature neural network to extract high-dimensional features of the video to be processed, then calculate emotional features of the video to be processed, combine the high-dimensional features of the video to be processed with the emotional features to obtain combined features of the video to be processed, and finally input the combined features of the video to be processed into the emotion classifier to predict the emotions of the characters in the video to be processed; The method of calculating the posture features of each frame in each video segment by using the human posture in the video dataset is as follows: There are 12 posture features in total, which are divided into four categories: volume, angle, distance and area. The volume refers to the volume of the outer bounding box of the human body posture in three-dimensional space. The angles include the angle formed by the two shoulders at the neck, the angle formed by the neck and the left shoulder at the right shoulder, the angle formed by the neck and the right shoulder at the left shoulder, and the angle formed by the head and the center of the upper body at the neck. The distances include the distances between the center of the bottom of the upper body and the right hand, left hand, right foot, left foot, and the distance between the two feet. The area includes the area of ​​the triangle formed by the hands and the neck, and the area of ​​the triangle formed by the center of the bottom of the upper body and the two feet. The calculations of the volume, angle, distance and area are all operations between three-dimensional coordinates. The method of calculating the motion features of each frame in each video segment by using the human body posture in the video dataset is as follows: There are 10 motion features in total, which are divided into two categories: movement speed and movement acceleration. The movement speed includes the movement speed of the five parts of the body: hands, feet, and head. The movement acceleration also includes the movement acceleration of the five parts of the body: hands, feet, and head. The movement speed is obtained by calculating the transformation of the coordinates of the human body posture corresponding to two frames of images in the video, while the movement acceleration is calculated by the change in movement speed.

2. The method for real-time prediction of elderly people's emotions based on body posture recognition according to claim 1, characterized in that: The human body posture corresponding to the video data is represented by 16 three-dimensional coordinates, each coordinate represents the position of an important part of the human body in three-dimensional space, that is, a posture is a 48-dimensional vector; the 16 important parts are the head, neck, left shoulder, right shoulder, left hand joint, right hand joint, left hand, right hand, the middle of the upper body, the bottom center of the upper body, the connection between the left leg and the upper body, the connection between the right leg and the upper body, the left leg joint, the right leg joint, the left foot, and the right foot.

3. The method for real-time prediction of elderly people's emotions based on body posture recognition according to claim 1, characterized in that: There are four emotion labels corresponding to the video data, namely happy, sad, angry and calm.

4. The method for real-time prediction of elderly people's emotions based on body posture recognition according to claim 1, characterized in that: The high-dimensional feature neural network is specifically: The high-dimensional feature neural network is trained using an encoding-decoding structure, with one LSTM network as the encoder and another LSTM network as the decoder. The input of the encoder is the human posture extracted from the video clip, that is, a 48-dimensional vector, and the output is the high-dimensional feature of the human posture; The input of the decoder is the high-dimensional features output by the encoder, and the output of the decoder is a 48-dimensional vector, i.e., the human body posture; The optimization of the encoder and decoder is to calculate the L2 norm loss between the 48-dimensional vector output by the decoder and the 48-dimensional vector input to the encoder, and then perform feedback propagation to optimize the encoder and decoder; suppose the input of the encoder is a vector P, where the i-th component is represented by p i The decoder output is a vector Q, whose i-th component is represented by q i Indicates that the L2 norm calculation formula between them is as follows:

5. A real-time prediction system for elderly people's emotions based on body posture recognition, characterized by: The system comprises: A training data construction unit is used to process the Ewalk dataset, including filtering the data in the dataset and processing it into the same size for the convenience of training, thereby forming a video dataset including video data and human posture and emotion labels corresponding to the video data; A neural network and classifier training unit is used to train a TP-Net network using the video data and the corresponding human postures in the video data set to form a human posture neural network for extracting human postures in video frames; train an LSTM network using the human postures in the video data set to form a high-dimensional feature neural network for extracting high-dimensional features of human postures, and then input the human postures in the video data set into the trained network to obtain high-dimensional features of each video in the video data set; calculate the posture features and motion features of each frame in each video using the human postures in the video data set, and then use the vector obtained by splicing the posture features and motion features as the emotional features corresponding to each frame, and take the average of the emotional features corresponding to all frames in any video as the emotional features of the video, thereby forming the emotional features of each video in the video data set; splice the high-dimensional features of each video with the emotional features of each video to obtain the combined features of each video, and form a training set together with the emotional labels corresponding to the video data in the video data set to train the classifier to form an emotional classifier; An emotion recognition unit is configured to input a video to be processed, extract human postures from the video to be processed using the human posture neural network, input the human postures into the high-dimensional feature neural network to extract high-dimensional features of the video to be processed, calculate emotional features of the video to be processed, combine the high-dimensional features of the video to be processed with the emotional features to obtain combined features of the video to be processed, and finally input the combined features of the video to be processed into the emotion classifier to predict the emotions of the characters in the video to be processed; The neural network and classifier training unit calculates the posture features of each frame in each video using the human posture in the video dataset, specifically: There are 12 posture features in total, which are divided into four categories: volume, angle, distance and area. The volume refers to the volume of the outer bounding box of the human body posture in three-dimensional space. The angles include the angle formed by the two shoulders at the neck, the angle formed by the neck and the left shoulder at the right shoulder, the angle formed by the neck and the right shoulder at the left shoulder, and the angle formed by the head and the center of the upper body at the neck. The distances include the distances between the center of the bottom of the upper body and the right hand, left hand, right foot, left foot, and the distance between the two feet. The area includes the area of ​​the triangle formed by the hands and the neck, and the area of ​​the triangle formed by the center of the bottom of the upper body and the two feet. The calculations of the volume, angle, distance and area are all operations between three-dimensional coordinates. The neural network and classifier training unit calculates the motion features of each frame in each video using the human body posture in the video dataset, specifically: There are 10 motion features in total, which are divided into two categories: movement speed and movement acceleration. The movement speed includes the movement speed of the five parts of the body: hands, feet, and head. The movement acceleration also includes the movement acceleration of the five parts of the body: hands, feet, and head. The movement speed is obtained by calculating the transformation of the coordinates of the human body posture corresponding to two frames of images in the video, while the movement acceleration is calculated by the change in movement speed.

Images