An emotion recognition method based on character gait video under angle selection

Abstract

The application discloses an emotion recognition method based on a human gait video under angle selection, which first converts a human normal walking video into a black-and-white contour picture sequence and a skeleton sequence. First, a picture set with a selected frame number length is extracted and input into a contour feature extraction network to extract frame-level features and sequence-level features of the human gait in the video. Meanwhile, the skeleton sequence is input into a skeleton feature extraction network for feature extraction. The obtained emotional features of the two modalities are fused and input into an inference network. At the same time, the picture sequence is input into an angle classifier as input information to determine angle information, and the model parameters in the inference network are selected and adjusted according to the angle information. Finally, the fused features are classified and logically output, so that the final result is obtained. The application fully utilizes various modal information of the gait, and effectively improves the accuracy and robustness of gait emotion recognition under different angles.

Description

Technical Field

[0001] This invention belongs to the field of gait emotion recognition in computer vision, specifically relating to an emotion recognition method based on human gait videos under angle selection. Background Technology

[0002] Gait emotion recognition, as a cutting-edge technology in the field of gait recognition applications, represents the intersection of computer vision and emotion recognition. The goal of this task is to identify the emotions of pedestrians during normal walking activities; that is, to identify the emotions of a pedestrian while walking normally, given a video clip of them walking.

[0003] In the field of gait recognition, human contour sequences and skeleton model data are relatively common gait sequence modalities, and have achieved good results in gait recognition. For example, GaitSet treats gait sequences as a set of independent frames that are not affected by the frame arrangement order, attempts to learn identity information from each input set, and can integrate frames collected under various walking conditions. It can achieve an accuracy of 82% with only 7 contours, which is close to the best performance achieved with more than 25 contours.

[0004] Previous studies on gait emotion recognition have used gait contours as input to study gait emotions. Most research primarily employs computer vision or sensors to simulate the 3D skeleton of the subject, identifying and classifying emotions based on gait features such as arm swing amplitude and walking speed. While gait recognition methods based on human contour maps are highly valued in the field of gait recognition, they lack sufficient experimental support in gait emotion recognition. Furthermore, when using human skeleton models generated from pose estimation in gait recognition, the performance of gait recognition based solely on skeleton models is not as high as that of contour-based methods. On the other hand, gait contour-based gait recognition is easily affected by changes in perspective, while skeleton-based gait recognition models have better robustness in this regard. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides an emotion recognition method based on human gait videos under angle selection. The method first converts a video of a person walking normally into a sequence of black-and-white contour images and a skeleton sequence. A set of images with a selected frame length is extracted and input into a contour feature extraction network to extract frame-level and sequence-level features of the person's gait in the video. Simultaneously, the skeleton sequence is input into a skeleton feature extraction network for feature extraction, and the obtained emotional features from the two modalities are fused and input into an inference network. Meanwhile, the image sequence is used as input to an angle classifier to determine angle information. Based on the angle information, the model parameters in the inference network are selected and adjusted. Finally, the fused features are classified logically and output to obtain the final result. This invention fully utilizes various modal information of gait, effectively improving the accuracy and robustness of gait emotion recognition under different angles.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0007] This invention is an emotion recognition method based on human gait videos under angle selection, the method comprising the following steps:

[0008] Step 1): Input a video of a person walking;

[0009] Step 2): The Robust Video Matting algorithm is used on the walking video from Step 1 to detect moving figures in the video. The figures are marked in white, and the rest in black, thus obtaining a binary gait contour sequence. Finally, the size of all contour sequence images is adjusted to 64×64 pixels to obtain the final gait contour sequence. VideoPose3D is then used to extract the skeleton joint information from the walking video from Step 1, obtaining the skeleton joint coordinate information, which is used as input to the skeleton feature network.

[0010] Step 3): Input the gait contour sequence obtained in Step 2) into the AG angle classifier to obtain the angle recognition result of the gait contour sequence. First, for each gait contour sequence, one sample frame is extracted every 5 frames. In the AG angle classifier, a series of CNN convolutional neural networks are used to extract single-frame features. Finally, the angle determination result of a single frame is obtained through a fully connected layer. The angle with the most frequent angle recognition results in all sample frames is selected as the angle determination information of the current gait contour sequence. The angle determination result of the gait contour sequence is obtained by analogy.

[0011] Step 4): Through angle-specific training, obtain the optimal model parameters of the GESS backbone network at different angles. Input the human gait contour sequence obtained in Step 2) and identified as 45°-90° in Step 3) into the GESS backbone network with the optimal model parameters at different angles, extract frame-level features and sequence-level features, obtain fused features, and further obtain the emotion inference results.

[0012] Step 5): If the angle judgment result in Step 3) is 0°-44° for a walking video, input the coordinate information of the skeleton joints obtained in Step 2) into the STEP backbone network to extract spatiotemporal features and further obtain the emotion inference result.

[0013] Step 6): Based on the results of Step 4) and Step 5), the final output is the emotion recognition result of the input walking video.

[0014] Furthermore, step 2) includes the following steps:

[0015] Step 21): Input the video of the person walking normally into the Robust Video Matting network to obtain a binary image sequence. Compress and crop the binary image sequence to obtain a 64×64 pixel two-dimensional contour sequence. Then, filter out the black frame images by thresholding and readjust the sequence number.

[0016] Step 22): Input the video of a person walking normally into the VideoPose3D network. First, simulate the keypoint information of the person in 2D through object detection. Then, further calculate and upgrade the 2D keypoint information to 3D keypoint information. The keypoint information is sorted and adjusted in the following order: Root, Spine, neck, head, Left shoulder, Left elbow, Left hand, Right shoulder, Right elbow, Right hand, Lefthip, Left knee, Left foot, Right hip, Right knee, Right foot. These 15 nodes are sorted from 0 to 14 to obtain a .csv file recording the time sequence information, which is used as the input to the skeleton network.

[0017] Furthermore, step 3) includes the following steps:

[0018] Step 31): Based on the length k of the input human gait contour sequence, uniformly select from the sequence. From frame images, obtain a set of sampled contour sequences;

[0019] Step 32): The set of sampled contour sequences obtained in step 31) is sequentially input into the trained AG angle classifier for angle classification;

[0020] The angle classifier consists of three CNN convolutional layers, three pooling layers, a fully connected layer, and a softmax layer. First, the original 3D contour features of size 3×64×64 are convolved into a 3D feature vector of size 64×8×8 through the three CNN convolutional layers and the pooling layers. Then, the dimensions are compressed through the fully connected layer and normalized through the softmax layer to obtain a one-dimensional vector, which represents the predicted probability of each angle in the current frame. The angle with the highest probability is selected as the output result.

[0021] Step 33): Statistically analyze the classification results of the sampling contour sequences obtained in Step 32), count the number of classification results, and take the result with the highest number as the final sequence angle output result.

[0022] Step 34): Select the angle with the highest number of single-frame angle predictions in the sampled contour sequence as the final output result of this sequence.

[0023] Furthermore, step 4) includes the following steps:

[0024] Step 41): During training, the Cross Entropy loss function is selected as the loss function for emotion classification, where l i y represents the actual target distribution. i The matching distribution representing sentiment prediction;

[0025]

[0026] Step 42): The process of extracting frame-level features of the character is as follows:

[0027] X={x j |j=1,2…,N}

[0028]

[0029] Where X represents a 64×64 sequence of human gait contours, x j This represents a single-frame contour image from a sequence of human gait contours, where N is the length of each sequence segment. Each sequence segment undergoes multiple frame-level feature extractions via a multi-layered CNN convolutional neural network, with Conv representing a single convolutional operation. Where T... F The set of results extracted from each CNN convolutional layer. This represents the frame-level features extracted by the j-th layer of the CNN network.

[0030] Step 43): The process of extracting character sequence-level features is as follows:

[0031]

[0032] Here, `cat` represents the link along the first dimension of the features, and `max` represents max pooling of the frame-level features along the dimensions of the links, meaning that for the current sequence, the most prominent features are collected for each frame based on its sequence number. `Conv` represents a convolutional operation in one stage, which can learn appropriate weights and adjust the weights of the values ​​extracted by the statistical function. This represents the direct addition of vectors;

[0033] Step 44): The fusion of frame-level and sequence-level features is as follows:

[0034]

[0035] The contour map sequence x is processed by frame-level and sequence-level feature extraction modules to generate frame-level features T. F and sequence-level features T G The two emotional features extracted from each sequence are equally divided using the third dimension height as the scale. Each feature is then divided into different groups, and horizontal pooling is performed on each group. The pooling results are then merged into the final output feature f.

[0036] In frame-level feature T F and sequence-level features T G In the grouping process, where H is the height of the human gait contour sequence image, it is first based on different scales. Frame-level features T F and sequence-level features T G The original data is reshaped by splitting it into 's' subgroups, each with a third dimension of size 'size'. Then, average pooling and max pooling are performed separately for each subgroup, and the sum of the two results is taken. This preserves the spatial location features of the contour sequences at different scales and receptive fields.

[0037] Step 45): Using a one-dimensional convolutional layer and a fully connected layer, obtain the final output four-dimensional tensor Y = {y1, y2, y3, y4} as the four emotion prediction probabilities of the final output.

[0038] Furthermore, step 5) includes the following steps:

[0039] Step 51): Input the coordinate information of the character skeleton joints obtained in Step 2;

[0040] Step 52): Perform feature engineering on the coordinate information of the human skeleton joints to further extract the features of the human skeleton joint coordinate information, such as the angles of the left and right shoulders and neck, the angle between the head and spine, the distance between the left and right hands and the base of the neck, the area of ​​the triangle formed by the left and right hands and neck, and the area of ​​the triangle formed by the left and right feet and the base of the neck. These are used as further input information for the skeleton feature extraction network. The specific input information is as follows:

[0041]

[0042] Step 53) Obtain the final output four-dimensional tensor Y = {y1, y2, y3, y4} through a two-dimensional convolutional layer and a fully connected layer as the four emotion prediction probabilities of the final output.

[0043] Furthermore, step 6) includes the following steps:

[0044] Step 6): Based on the emotion output Y = {y1, y2, y3, y4} obtained in Step 4) or Step 5), use it as the four emotion prediction probabilities for the final output; obtain the emotion prediction probabilities through the Softmax activation function, and select the highest probability Softmax(y i ) as the prediction result;

[0045]

[0046] The beneficial effects of this invention are:

[0047] This invention first acquires a video of a person walking normally, extracts gait contour information from the video, and further uses a GESS network to obtain frame-level and sequence-level features for fusion. Simultaneously, a STEP network extracts emotional features from the gait skeleton. The gait contour features and gait skeleton features are then fused. An AG angle classifier is used to determine the viewpoint of the frame sequence, thereby selecting appropriate emotion recognition model parameters based on the determined viewpoint. This achieves more accurate gait emotion recognition from different angles.

[0048] Specifically:

[0049] (1) A frame-level angle classifier is proposed to infer the viewpoint of gait contours. It also extends the classification from frame-level angles to sequence-level angles, and eliminates the influence of different viewpoints on gait contour emotion recognition through an angle selection strategy.

[0050] (2) This invention extracts gait contours using the Robust Video Matting algorithm, which serves as input for gait contour emotion. It also extracts gait joint information using the VideoPose3D algorithm, which serves as input for gait skeleton joint emotion.

[0051] (3) This invention proposes a gait recognition model, GESS, to learn the emotional features of gait contours, extract frame-level features and sequence-level features, and fuse them. At the same time, the STEP network extracts the emotional features of the gait skeleton and achieves angle selection fusion with the gait contour features. Attached Figure Description

[0052] Figure 1 This is a comprehensive flowchart of the gait emotion recognition method of the present invention.

[0053] Figure 2 This is a schematic diagram of human gait contour sequence extraction.

[0054] Figure 3 This is a schematic diagram of extracting the coordinates of the joints of a human skeleton.

[0055] Figure 4 This is a diagram of the frame-level angle classifier architecture of the present invention. Detailed Implementation

[0056] The embodiments of the present invention will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the invention. That is, in some embodiments of the invention, these practical details are not essential.

[0057] like Figure 1-4 As shown, this invention is an emotion recognition method based on human gait videos under angle selection, comprising the following steps:

[0058] Step 1: Input a video of a person walking;

[0059] Step 2: The Robust Video Matting algorithm is used to detect moving figures in the walking video from Step 1. The figures are marked in white, and the rest in black, resulting in a binary gait contour sequence. Finally, the size of all contour sequences is adjusted to 64×64 pixels to obtain the final gait contour sequence. VideoPose3D is then used to extract the skeleton joint information from the walking video from Step 1, obtaining the skeleton joint coordinates as input to the skeleton feature network.

[0060] Step 21: Input the walking video of the person into the Robust Video Matting network to obtain a binary image sequence. Compress and crop the binary image sequence to obtain a 64×64 pixel two-dimensional contour sequence. Then, filter out the black frame images by thresholding and readjust the sequence number.

[0061] Step 22: Input the walking video into the VideoPose3D network. First, simulate the key point information of the person in two dimensions through object detection. Then, upgrade the key point information of the person in two dimensions to key point information of the person in three dimensions through calculation. The key point information is sorted and adjusted in the following order: Root, Spine, neck, head, Left shoulder, Left elbow, Left hand, Right shoulder, Right elbow, Right hand, Left hip, Left knee, Left foot, Right hip, Right knee, Right foot. Sort these 15 nodes from 0 to 14 to obtain a .csv file that records the time sequence information as input to the skeleton feature network.

[0062] Step 3: Input the gait contour sequence obtained in Step 2 into the AG angle classifier to obtain the angle recognition result of the gait contour sequence. First, for each gait contour sequence, one sample frame is extracted every 5 frames. In the AG angle classifier, a series of CNN convolutional neural networks are used to extract single-frame features. Finally, the angle judgment result of a single frame is obtained through a fully connected layer. The angle with the most frequent angle recognition results in all sample frames is selected as the angle judgment information of the current gait contour sequence. The angle judgment result of the gait contour sequence is obtained by analogy.

[0063] Step 31: Based on the length k of the input human gait contour sequence, uniformly select from the sequence. From frame images, obtain a set of sampled contour sequences;

[0064] Step 32: The set of sampled contour sequences obtained in Step 31 is sequentially input into the trained AG angle classifier for angle classification;

[0065] The angle classifier consists of three CNN convolutional layers, three pooling layers, a fully connected layer, and a softmax layer. First, the original 3D contour features of size 3×64×64 are convolved into a 3D feature vector of size 64×8×8 through the three CNN convolutional layers and the pooling layers. Then, the dimensions are compressed through the fully connected layer and normalized through the softmax layer to obtain a one-dimensional vector, which represents the predicted probability of each angle in the current frame. The angle with the highest probability is selected as the output result.

[0066] Step 33: Statistically analyze the classification results of the sampled contour sequences obtained in Step 3-2, count the number of classification results, and take the result with the highest number as the final sequence angle output result;

[0067] Step 34: Select the angle with the highest number of single-frame angle predictions in the sampled contour sequence as the final output result of this sequence.

[0068] Step 4: Through angle-separated training, obtain the optimal model parameters of the GESS backbone network at different angles. Input the human gait contour sequence obtained in Step 2 and identified as 45°-90° in Step 3 into the GESS backbone network with the optimal model parameters at different angles, extract frame-level features and sequence-level features, obtain fused features, and further obtain the emotion inference results.

[0069] Step 41: During training, the Cross Entropy loss function is selected as the loss function for emotion classification, where l i y represents the actual target distribution. i The matching distribution representing sentiment prediction;

[0070]

[0071] Step 42: The process of extracting frame-level features of the character is as follows:

[0072] X={x j |j=1,2…,N}

[0073]

[0074] Where X represents a 64×64 sequence of human gait contours, x j This represents a single-frame contour image from a sequence of human gait contours, where N is the length of each sequence segment. Each sequence segment undergoes multiple frame-level feature extractions via a multi-layered CNN convolutional neural network, with Conv representing a single convolutional operation. Where T... F The set of results extracted from each CNN convolutional layer. This represents the frame-level features extracted by the j-th layer of the CNN network.

[0075] Step 43: The process of extracting character sequence-level features is as follows:

[0076]

[0077] Here, `cat` represents the link along the first dimension of the features, and `max` represents max pooling of the frame-level features along the dimensions of the links, meaning that for the current sequence, the most prominent features are collected for each frame based on its sequence number. `Conv` represents a convolutional operation in one stage, which can learn appropriate weights and adjust the weights of the values ​​extracted by the statistical function. This represents the direct addition of vectors;

[0078] Step 44: The fusion of frame-level and sequence-level features is as follows:

[0079]

[0080] The contour image sequence X, after passing through frame-level and sequence-level feature extraction modules, generates frame-level features T. F and sequence-level features T G The two emotional features extracted from each sequence are equally divided using the third dimension height as the scale. Each feature is then divided into different groups, and horizontal pooling is performed on each group. The pooling results are then merged into the final output feature f.

[0081] In frame-level feature T F and sequence-level features T G In the grouping process, where H is the height of the human gait contour sequence image, it is first based on different scales. Frame-level features T F and sequence-level features T G The original data is reshaped by splitting it into 's' subgroups, each with a third dimension of size 'size'. Then, average pooling and max pooling are performed separately for each subgroup, and the sum of the two results is taken. This preserves the spatial location features of the contour sequences at different scales and receptive fields.

[0082] Step 45: Obtain the final output four-dimensional tensor Y = {y1, y2, y3, y4} through a one-dimensional convolutional layer and a fully connected layer as the four emotion prediction probabilities of the final output.

[0083] Step 5: If the angle judgment result in Step 3 is 0°-44° for the walking video, input the coordinate information of the human skeleton joints obtained in Step 2 into the STEP backbone network to extract spatiotemporal features and further obtain the emotion inference result.

[0084] Step 51: Input the coordinate information of the character skeleton joints obtained in Step 2;

[0085] Step 52: Perform feature engineering on the coordinate information of the human skeleton joints to further extract the features of the human skeleton joint coordinate information, such as the angles of the left and right shoulders and neck, the angle between the head and spine, the distance between the left and right hands and the base of the neck, the area of ​​the triangle formed by the left and right hands and neck, and the area of ​​the triangle formed by the left and right feet and the base of the neck. These are used as further input information for the skeleton feature extraction network. The specific input information is as follows:

[0086]

[0087]

[0088] Step 53: Obtain the final output four-dimensional tensor Y = {y1, y2, y3, y4} through a two-dimensional convolutional layer and a fully connected layer as the four emotion prediction probabilities of the final output.

[0089] Step 6: Based on the emotion output Y = {y1, y2, y3, y4} obtained in Step 4 or Step 5, use it as the four emotion prediction probabilities for the final output; obtain the emotion prediction probabilities through the Softmax activation function, and select the highest probability Softmax(y i ) as the prediction result;

[0090]

[0091] The goal of this invention is to fully utilize human contour information and skeletal joint information from videos to obtain gait emotion features and angular information, and to eliminate redundant influences from angles by appropriately selecting model parameters. Finally, the integrated fused emotion features are input into the corresponding emotion classification module, thereby achieving accurate recognition of gait emotions. This method is designed to improve the accuracy and reliability of gait emotion recognition and has significant academic and applied value in the field of emotion recognition.

[0092] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. An emotion recognition method based on human gait videos under angle selection, characterized in that, The method includes the following steps: Step 1: Input a video of a person walking; Step 2: Use the Robust Video Matting algorithm to detect moving figures in the walking video from Step 1. Mark the figures in white and the rest in black to obtain a binary gait contour sequence. Finally, adjust the size of all contour sequence images to 64×64 pixels to obtain the final gait contour sequence. Use VideoPose3D to extract the skeleton joint information from the walking video from Step 1 to obtain the skeleton joint coordinate information, which is used as the input to the skeleton feature network. Step 3: Input the gait contour sequence obtained in Step 2 into the AG angle classifier to obtain the angle recognition result of the gait contour sequence. First, for each gait contour sequence, extract one sample frame every 5 frames. In the AG angle classifier, extract single-frame features through a series of CNN convolutional neural networks. Finally, obtain the single-frame angle determination result through a fully connected layer. Select the angle with the most angle recognition results in all sample frames as the angle determination information of the current gait contour sequence. And so on to obtain the angle determination result of the gait contour sequence. Step 4: Through angle-separated training, obtain the optimal model parameters of the GESS backbone network at different angles. Input the human gait contour sequence obtained in Step 2 and identified as 45°-90° in Step 3 into the GESS backbone network with the optimal model parameters at different angles, extract frame-level features and sequence-level features, obtain fused features, and obtain the emotion inference result. Step 5: If the angle judgment result in Step 3 is 0°-44° for the walking video, input the coordinate information of the human skeleton joints obtained in Step 2 into the STEP backbone network to extract spatiotemporal features and obtain the emotion inference result. Step 6: Based on the results of Step 4 and Step 5, the final output is the emotion recognition result of the input walking video.

2. The emotion recognition method based on human gait video under angle selection according to claim 1, characterized in that, Step 2 specifically includes the following steps: Step 2-1: Input the walking video of the person into the Robust Video Matting network to obtain a binary image sequence. Compress and crop the binary image sequence to obtain a 64×64 pixel two-dimensional contour sequence. Then, filter out the black frame images by thresholding and readjust the sequence number. Step 2-2: Input the walking video into the VideoPose3D network. First, simulate the key point information of the person in 2D through object detection. Then, upgrade the 2D key point information of the person into 3D key point information through calculation. The key point information is sorted and adjusted in the following order: Root, Spine, neck, head, Left shoulder, Left elbow, Left hand, Right shoulder, Right elbow, Right hand, Left hip, Left knee, Left foot, Right hip, Right knee, Right foot. Sort these 15 nodes from 0 to 14 to obtain a .csv file that records the time sequence information as input to the skeleton feature network.

3. The emotion recognition method based on human gait video under angle selection according to claim 2, characterized in that, Step 3 specifically includes the following steps: Step 3-1: Based on the length k of the input human gait contour sequence, uniformly select from the sequence. From frame images, obtain a set of sampled contour sequences; The sampled contour sequence sets obtained in steps 3-2 and 3-1 are sequentially input into the trained AG angle classifier for angle classification. The AG angle classifier consists of three CNN convolutional layers, three pooling layers, a fully connected layer, and a softmax layer. First, the original 3D contour features of size 3×64×64 are convolved into a 3D feature vector of size 64×8×8 through the three CNN convolutional layers and the pooling layers. Then, the dimensions are compressed through the fully connected layer and normalized through the softmax layer to obtain a one-dimensional vector, which represents the predicted probability of each angle in the current frame. The angle with the highest probability is selected as the output result. Step 3-3: Statistically analyze the classification results of the sampled contour sequences obtained in Step 3-2, count the number of classification results, and take the result with the highest number as the final sequence angle output result. Steps 3-4: Select the angle with the highest number of single-frame angle predictions in the sampled contour sequence as the final output result of this sequence.

4. The emotion recognition method based on human gait video under angle selection according to claim 3, characterized in that, Step 4 specifically includes the following steps: Step 4-1: During training, the Cross Entropy loss function is selected as the loss function for emotion classification, where l i y represents the actual target distribution. i The matching distribution representing sentiment prediction; Step 4-2, the process of extracting frame-level features of the person is as follows: X={x j |j=1,2…,N} Where X represents a 64×64 sequence of human gait contours, x j This represents a single-frame contour image from a sequence of human gait contours, where N is the length of each sequence segment. Each sequence segment will undergo multiple layers of a CNN convolutional neural network for multiple frame-level feature extractions, with Conv representing a convolutional operation at one stage. Where T... F The set of results extracted from each CNN convolutional layer. This represents the frame-level features extracted by the j-th layer of the CNN network. Step 4-3, the process of extracting character sequence-level features is as follows: Where `cat` represents the connection along the first dimension of the features, `max` represents max pooling of the frame-level features along the connection dimension, that is, for the current sequence, the most prominent features are collected for each frame based on the sequence number; `Conv` represents a stage of convolution operation, which learns appropriate weights and adjusts the weights of the values ​​extracted by the statistical function. This represents the direct addition of vectors; Step 4-4: The fusion of frame-level and sequence-level features is as follows: The contour image sequence X, after passing through frame-level and sequence-level feature extraction modules, generates frame-level features T. F and sequence-level features T G The two emotional features extracted from each sequence are equally divided using the third dimension height as the scale. Each feature is divided into different groups, and horizontal pooling is performed on each group. The pooling results are then merged into the final output feature f. In frame-level feature T F and sequence-level features T G In the grouping process, where H is the height of the human gait contour sequence image, it is first based on different scales. Frame-level features T F and sequence-level features T G The original data is reshaped by splitting it into s groups, with the third dimension of each group being size. Then, average pooling and max pooling are performed separately for each group, and the sum of the two results is taken. This preserves the different spatial location features of the contour sequences under different scales and receptive fields. Steps 4-5: Using a one-dimensional convolutional layer and a fully connected layer, obtain the final output four-dimensional tensor Y = {y1, y2, y3, y4} as the four emotion prediction probabilities of the final output.

5. The emotion recognition method based on human gait video under angle selection according to claim 4, characterized in that, Step 5 specifically includes the following steps: Step 5-1: Input the coordinate information of the character skeleton joints obtained in Step 2; Step 5-2: Perform feature engineering on the coordinate information of the character skeleton joints, including the angle information, distance information, area information of the triangles formed, velocity of the nodes and inter-frame transformation features of the nodes as further input information for the skeleton feature extraction network; Step 5-5: Using a one-dimensional convolutional layer and a fully connected layer, obtain the final output four-dimensional tensor Y = {y1, y2, y3, y4} as the four emotion prediction probabilities of the final output.

6. The emotion recognition method based on human gait video under angle selection according to claim 5, characterized in that, Step 6 specifically includes the following steps: Based on the emotion output Y = {y1, y2, y3, y4} obtained in step 4 or 5, four emotion prediction probabilities are used as the final output; the emotion prediction probabilities are obtained through the Softmax activation function, and the highest probability Softmax(y) is selected. i ) as the prediction result;

Images