Emotion recognition method based on character gait video under angle selection

Abstract

The invention discloses an emotion recognition method based on a character gait video under angle selection. The method comprises the following steps: firstly, converting a character normal walking video into a black and white outline picture sequence and a skeleton sequence; the method comprises the following steps: firstly, extracting a picture set with a selected frame number length, and inputting the picture set into a contour feature extraction network to extract frame-level features and sequence-level features of a character gait in a video; and inputting the skeleton sequence into a skeleton feature extraction network for feature extraction. And fusing the obtained emotional characteristics of the two modes and inputting the fused emotional characteristics into an inference network. Meanwhile, the picture sequence is used as input information to be transmitted into an angle classifier to judge angle information, and model parameters in the reasoning network are selected and adjusted according to the angle information. And finally, performing classification logic output on the fused features to obtain a final result. According to the method, various modal information of gaits is fully utilized, and the accuracy and robustness of gait emotion recognition at different angles are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of gait emotion recognition in computer vision, and specifically relates to an emotion recognition method based on a person's gait video under angle selection. Background Art

[0002] Gait emotion recognition, as a cutting-edge technology in the field of gait recognition applications, represents the intersection of computer vision and emotion recognition. The purpose of this task is to recognize the emotions of pedestrians during normal walking activities, that is, to identify the emotions of pedestrians during normal walking activities by inputting a video of people walking normally.

[0003] In the field of gait recognition, human contour sequences and skeleton model data, as more common gait sequence modalities, have achieved good results in gait recognition. For example, GaitSet regards gait sequences as a set of independent frames that are not affected by the order of frame arrangement, attempts to learn identity information from each input set, and can integrate frames collected under various walking conditions. It can achieve an accuracy of 82% with only 7 contours, and is close to the best performance achieved using more than 25 contours.

[0004] Previous gait emotion recognition has used gait contours as input to study gait emotions. Most studies mainly use computer vision or sensors to simulate the 3D skeleton of the subject, and identify and classify the subject's emotions through gait features such as arm swing amplitude and walking speed. Although the gait recognition research method based on human body contour map is highly valued in the field of gait recognition, there is still no sufficient experimental support in the field of gait emotion recognition. In addition, the human skeleton model generated by posture estimation is used in gait recognition. Compared with the contour-based method, the result of gait recognition based entirely on the skeleton model is not as high as that based on the contour. On the other hand, the gait recognition method based on gait contour is easily affected by changes in perspective, while the gait recognition model based on the skeleton has better robustness in this regard. Summary of the invention

[0005] In order to solve the above problems, the present invention provides an emotion recognition method based on a person's gait video under angle selection. The method first converts a normal walking video of a person into a black and white outline picture sequence and a skeleton sequence, extracts a picture set of a selected frame length, and inputs it into a contour feature extraction network to extract the frame-level features and sequence-level features of the person's gait in the video. At the same time, the skeleton sequence is input into the skeleton feature extraction network for feature extraction, and the obtained emotional features of the two modalities are fused and input into the inference network. At the same time, the picture sequence is passed as input information into the angle classifier to discriminate the angle information, and the model parameters in the inference network are selected and adjusted according to the angle information, and finally the fused features are classified and logically output to obtain the final result. The present invention makes full use of various modal information of gait, and effectively improves the accuracy and robustness of gait emotion recognition at different angles.

[0006] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0007] The present invention is an emotion recognition method based on a person's gait video under angle selection, and the method comprises the following steps:

[0008] Step 1): Input a video of a person walking;

[0009] Step 2): Use the Robust Video Matting algorithm to detect the moving person in the video of the person walking in step 1, mark the person part as white and the rest as black, and then obtain a binary person gait contour sequence. Finally, adjust the size of all contour sequence images to 64×64 pixels to finally obtain a person gait contour sequence. Use VideoPose3D to extract the person skeleton joint point information from the person walking video in step 1, and obtain the coordinate information of the person skeleton joint point as the input of the skeleton feature network;

[0010] Step 3): Input the character gait contour sequence obtained in step 2) into the AG angle classifier to obtain the angle recognition result of the character gait contour sequence. First, extract a sample frame every 5 frames for each gait contour sequence, extract single-frame features through a series of CNN convolutional neural networks in the AG angle classifier, and finally obtain the single-frame angle judgment result through full connection, and select the most angle in the angle recognition results of all sample frames as the current character gait contour sequence angle judgment information. In this way, the angle judgment result of the character gait contour sequence is obtained;

[0011] Step 4): Obtain the optimal model parameters of the GESS backbone network at different angles through angle-by-angle training, input the character gait profile sequence obtained in step 2) and recognized as 45°-90° in step 3) into the GESS backbone network with the optimal model parameters at different angles, extract frame-level features and sequence-level features, obtain fusion features, and further obtain emotion inference results;

[0012] Step 5): If the angle judgment result in step 3) is a video of a person walking at 0°-44°, the coordinate information of the person's skeleton joint points obtained in step 2) is input into the STEP backbone network to extract spatiotemporal features, and further obtain the emotion inference result;

[0013] Step 6): According to the results of step 4) and step 5), the emotion recognition result of the input person walking video is finally output.

[0014] Further, step 2) comprises the following steps:

[0015] Step 21): The normal walking video of the person is passed into the Robust Video Matting network to obtain a binary image sequence, which is compressed and cropped to obtain a two-dimensional contour sequence of 64×64 pixels. The black frame images are filtered out through a threshold and the sequence numbers are readjusted.

[0016] Step 22): The normal walking video of the person is passed into the VideoPose3D network. First, the two-dimensional key point information of the person is simulated through target detection. The two-dimensional character joint point information is upgraded to three-dimensional character joint point information through further calculation. The joint point information is sorted and adjusted in the following order: Root, Spine, neck, head, Left shoulder, Left elbow, Left hand, Right shoulder, Right elbow, Right hand, Left hip, Left knee, Left foot, Right hip, Right knee, Right foot. These 15 nodes are sorted from 0 to 14, and the .csv file recording the timing information is obtained as the input of the skeleton network.

[0017] Further, step 3) comprises the following steps:

[0018] Step 31): According to the input character gait contour sequence length k, evenly select Frame picture, obtain a set of sampled contour sequences;

[0019] Step 32): The sampled contour sequence set obtained in step 31) is sequentially input into the trained AG angle classifier for angle classification;

[0020] The structure of the angle classifier consists of 3 layers of CNN convolutional layers, 3 layers of Pooling layers, a fully connected layer, and a Softmax layer. First, the 3×64×64 3D original contour features are convolved into a 64×8×8 3D feature vector through 3 layers of CNN convolutional layers and Pooling layers, and then the dimensions are compressed through the fully connected layer and normalized through the Softmax layer. Finally, a one-dimensional vector is obtained, which represents the predicted probability of each angle of the current frame. The angle with the highest probability is selected as the output result.

[0021] Step 33): Count the classification results of the sampled contour sequence obtained in step 32), count the classification results, and take the result with the highest number of times as the final sequence angle output result;

[0022] Step 34): Select the angle with the highest number of single-frame angle predictions in the sampled contour sequence as the final output result of this sequence.

[0023] Further, step 4) comprises the following steps:

[0024] Step 41): During the training process, the cross entropy loss function is selected as the emotion classification loss function, where l i represents the actual target distribution, y i represents the matching distribution of sentiment predictions;

[0025]

[0026] Step 42): The character frame-level feature extraction process is as follows:

[0027] X={x j |j=1,2…,N}

[0028]

[0029] Where X is a 64×64 character gait contour sequence, j is a single-frame contour image in the character gait contour sequence, N is the length of each sequence; for each sequence, multiple layers of CNN convolutional neural network will be used to perform multiple frame-level feature extractions, and Conv represents a stage of convolution operation. F The result set extracted for each layer of CNN convolution, Represents the frame-level features extracted by the j-th layer CNN network;

[0030] Step 43): The process of extracting character sequence-level features is as follows:

[0031]

[0032] Among them, cat represents the link on the first dimension in the feature, and max represents the maximum pooling of frame-level features along the connected dimension, that is, for the current sequence, the most prominent features of each frame are collected according to the sequence number. Conv represents a stage of convolution operation, which can realize the learning of appropriate weights and adjust the weights of the values ​​extracted by the statistical function. Represents direct addition of vectors;

[0033] Step 44): The fusion of frame-level and sequence-level features is as follows:

[0034]

[0035] The contour sequence x generates frame-level features T through the frame-level and sequence-level feature extraction modules F and sequence-level features T G The two emotion features extracted from each sequence are equally divided based on the height of the third dimension, and each feature is divided into different groups. The groups are horizontally pooled, and the pooling results are merged into the final output feature f.

[0036] In the frame-level feature T F and sequence-level features T G In the grouping process, H is the height of the character gait profile sequence diagram. First, according to different scales The frame-level features T F and sequence-level features T G Reshape the original data into s groups, each with a third dimension of size, and then perform average pooling and maximum pooling respectively, taking the sum of the two results. This preserves the different spatial position features of contour sequences at different scales and different receptive fields.

[0037] Step 45): Through the one-dimensional convolution layer and the fully connected layer, the final output four-dimensional tensor Y = {y1, y2, y3, y4} is obtained as the final output of the four emotion prediction probabilities.

[0038] Further, step 5) comprises the following steps:

[0039] Step 51): input the coordinate information of the character skeleton joint points obtained in step 2;

[0040] Step 52): Perform feature engineering on the coordinate information of the character skeleton joint points, and further extract the features of the coordinate information of the character skeleton joint points, such as the angle between the left and right shoulders and the neck, the angle between the head and the spine, the distance between the left and right hands and the roots, the area of ​​the triangle formed by the left and right hands and the neck, the area of ​​the triangle formed by the left and right feet and the roots, etc. as further input information of the skeleton feature extraction network. The specific input information is as follows:

[0041]

[0042] Step 53) Through the two-dimensional convolution layer and the fully connected layer, the final output four-dimensional tensor Y = {y1, y2, y3, y4} is obtained as the final output of the four emotion prediction probabilities.

[0043] Further, step 6) comprises the following steps:

[0044] Step 6): According to the emotion output result Y = {y1, y2, y3, y4} obtained in step 4) or step 5), the four emotion prediction probabilities are used as the final output; the emotion prediction probability is obtained through the Softmax activation function, and the maximum probability Softmax (y i ) as the prediction result;

[0045]

[0046] The beneficial effects of the present invention are:

[0047] The present invention first obtains a video of a person walking normally, extracts gait profile information from the video, and further uses the GESS network to obtain frame-level features and sequence-level features for fusion, while the STEP network extracts the emotional features of the gait skeleton. The gait profile features and gait skeleton features are fused; the AG angle classifier is used to judge the frame sequence perspective, and the corresponding emotion recognition model parameters are selected according to the judged perspective. This achieves more accurate gait emotion recognition of people at different angles.

[0048] Specifically:

[0049] (1) A frame-level angle classifier is proposed to infer the perspective of the gait profile. The frame-level angle classification is extended to the sequence-level angle classification, and the influence of different perspectives on the emotion recognition of the gait profile is eliminated through the angle selection strategy.

[0050] (2) The present invention extracts the gait contour through the Robust Video Matting algorithm as the input of the gait contour emotion. The gait joint point information is extracted through the VideoPose3D algorithm as the input of the gait skeleton joint point emotion.

[0051] (3) The present invention proposes a gait recognition model GESS to learn the emotional features of gait contour, extract frame-level features and sequence-level features and fuse them. At the same time, the STEP network extracts the emotional features of the gait skeleton and implements angle selection fusion with the gait contour features. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a comprehensive flow chart of gait emotion recognition of the present invention.

[0053] Figure 2 This is a schematic diagram of character gait contour sequence extraction.

[0054] Figure 3 This is a schematic diagram of extracting the coordinates of the character skeleton joint points.

[0055] Figure 4 This is the architecture diagram of the frame-level angle classifier of the present invention. DETAILED DESCRIPTION

[0056] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are not necessary.

[0057] like Figure 1-4 As shown, the present invention is an emotion recognition method based on a person's gait video under angle selection, comprising the following steps:

[0058] Step 1: Input a video of a person walking;

[0059] Step 2: Use the Robust Video Matting algorithm to detect the moving person in the video of the person walking in step 1, mark the person part as white and the rest as black, and then obtain the binary person gait contour sequence. Finally, adjust the size of all contour sequence images to 64×64 pixels to obtain the person gait contour sequence. Use VideoPose3D to extract the person skeleton joint point information from the person walking video in step 1, and obtain the coordinate information of the person skeleton joint point as the input of the skeleton feature network;

[0060] Step 21: The walking video of the person is passed into the Robust Video Matting network to obtain a binary image sequence, which is compressed and cropped to obtain a 64×64 pixel two-dimensional contour sequence, and the black frame images are filtered out by a threshold, and the sequence number is readjusted;

[0061] Step 22: Pass the walking video of the person into the VideoPose3D network. First, simulate the two-dimensional character key point information through target detection, upgrade the two-dimensional character joint point information to three-dimensional character joint point information through calculation, and sort and adjust the joint point information in the following order: Root, Spine, neck, head, Left shoulder, Left elbow, Left hand, Right shoulder, Right elbow, Right hand, Left hip, Left knee, Left foot, Right hip, Right knee, Right foot; sort these 15 nodes from 0 to 14, and obtain a .csv file recording the timing information as the input of the skeleton feature network.

[0062] Step 3: Input the character gait contour sequence obtained in step 2 into the AG angle classifier to obtain the angle recognition result of the character gait contour sequence. First, extract a sample frame every 5 frames for each gait contour sequence, extract single-frame features through a series of CNN convolutional neural networks in the AG angle classifier, and finally obtain the single-frame angle judgment result through full connection, and select the most angle in the angle recognition results of all sample frames as the current character gait contour sequence angle judgment information. In this way, the angle judgment result of the character gait contour sequence is obtained;

[0063] Step 31: According to the input character gait contour sequence length k, evenly select Frame picture, obtain a set of sampled contour sequences;

[0064] Step 32: The sampled contour sequence set obtained in step 31 is sequentially input into the trained AG angle classifier for angle classification;

[0065] The structure of the angle classifier consists of 3 layers of CNN convolutional layers, 3 layers of Pooling layers, a fully connected layer, and a Softmax layer. First, the 3×64×64 3D original contour features are convolved into a 64×8×8 3D feature vector through 3 layers of CNN convolutional layers and Pooling layers, and then the dimensions are compressed through the fully connected layer and normalized through the Softmax layer. Finally, a one-dimensional vector is obtained, which represents the predicted probability of each angle of the current frame. The angle with the highest probability is selected as the output result.

[0066] Step 33: Count the classification results of the sampled contour sequence obtained in step 3-2, count the classification results, and take the result with the highest number of times as the final sequence angle output result;

[0067] Step 34: Select the angle with the highest number of single-frame angle predictions in the sampled contour sequence as the final output result of this sequence.

[0068] Step 4: Obtain the optimal model parameters of the GESS backbone network at different angles through angle-by-angle training, input the character gait contour sequence obtained in step 2 and recognized as 45°-90° in step 3 into the GESS backbone network with the optimal model parameters at different angles, extract frame-level features and sequence-level features, obtain fusion features, and further obtain emotion inference results;

[0069] Step 41: During the training process, the cross entropy loss function is selected as the emotion classification loss function, where l i represents the actual target distribution, y i represents the matching distribution of sentiment predictions;

[0070]

[0071] Step 42: The character frame-level feature extraction process is as follows:

[0072] X={x j |j=1,2…,N}

[0073]

[0074] Where X is a 64×64 character gait contour sequence, j is a single-frame contour image in the character gait contour sequence, N is the length of each sequence; for each sequence, multiple layers of CNN convolutional neural network will be used to perform multiple frame-level feature extractions, and Conv represents a stage of convolution operation. F The result set extracted for each layer of CNN convolution, Represents the frame-level features extracted by the j-th layer CNN network;

[0075] Step 43: The process of extracting character sequence-level features is as follows:

[0076]

[0077] Among them, cat represents the link on the first dimension in the feature, and max represents the maximum pooling of frame-level features along the connected dimension, that is, for the current sequence, the most prominent features of each frame are collected according to the sequence number. Conv represents a stage of convolution operation, which can realize the learning of appropriate weights and adjust the weights of the values ​​extracted by the statistical function. Represents direct addition of vectors;

[0078] Step 44: The fusion of frame-level and sequence-level features is as follows:

[0079]

[0080] The contour sequence X generates frame-level features T through the frame-level and sequence-level feature extraction modules. F and sequence-level features T G The two emotion features extracted from each sequence are equally divided based on the height of the third dimension, and each feature is divided into different groups. The groups are horizontally pooled, and the pooling results are merged into the final output feature f.

[0081] In the frame-level feature T F and sequence-level features T G In the grouping process, H is the height of the character gait profile sequence diagram, first according to different scales The frame-level features T F and sequence-level features T G Reshape the original data into s groups, each with a third dimension of size, and then perform average pooling and maximum pooling respectively, taking the sum of the two results. This preserves the different spatial position features of contour sequences at different scales and different receptive fields.

[0082] Step 45: Through the one-dimensional convolution layer and the fully connected layer, the final output four-dimensional tensor Y = {y1, y2, y3, y4} is obtained as the final output of the four emotion prediction probabilities.

[0083] Step 5: If the angle judgment result in step 3 is a video of a person walking at 0°-44°, the coordinate information of the skeleton joint points of the person obtained in step 2 is input into the STEP backbone network to extract spatiotemporal features, and further obtain the emotion inference result;

[0084] Step 51: input the coordinate information of the character skeleton joint points obtained in step 2;

[0085] Step 52: Perform feature engineering on the coordinate information of the character skeleton joint points, and further extract the features of the coordinate information of the character skeleton joint points, such as the angle between the left and right shoulders and the neck, the angle between the head and the spine, the distance between the left and right hands and the roots, the area of ​​the triangle formed by the left and right hands and the neck, the area of ​​the triangle formed by the left and right feet and the roots, etc. as further input information of the skeleton feature extraction network. The specific input information is as follows:

[0086]

[0087]

[0088] Step 53: Through the two-dimensional convolution layer and the fully connected layer, the final output four-dimensional tensor Y = {y1, y2, y3, y4} is obtained as the final output of the four emotion prediction probabilities.

[0089] Step 6: According to the emotion output result Y={y1,y2,y3,y4} obtained in step 4 or step 5, the four emotion prediction probabilities are used as the final output; the emotion prediction probability is obtained through the Softmax activation function, and the maximum probability Softmax(y i ) as the prediction result;

[0090]

[0091] The goal of this invention is to make full use of the human body contour information and human skeleton joint point information in the video to obtain gait emotional characteristics and angle information, and to eliminate the redundant influence of angle by reasonably selecting model parameters. Finally, the integrated fusion emotional features will be input into the corresponding emotion classification module to achieve accurate recognition of gait emotions. This method is designed to improve the accuracy and reliability of gait emotion recognition, and has important academic and application value in the field of emotion recognition.

[0092] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. An emotion recognition method based on a person's gait video under angle selection, characterized in that: The method comprises the following steps: Step 1: Input a video of a person walking; Step 2: Use the Robust Video Matting algorithm to detect the moving person in the walking video in step 1, mark the person part as white and the rest as black, and then obtain a binary person gait contour sequence. Finally, adjust the size of all contour sequence images to 64×64 pixels to obtain a person gait contour sequence. Use VideoPose3D to extract the person skeleton joint point information in the walking video in step 1, and obtain the person skeleton joint point coordinate information as the input of the skeleton feature network. Step 3: Input the character gait contour sequence obtained in step 2 into the AG angle classifier to obtain the angle recognition result of the character gait contour sequence; firstly, extract a sample frame every 5 frames for each gait contour sequence, extract single-frame features through a series of CNN convolutional neural networks in the AG angle classifier, and finally obtain the single-frame angle judgment result through full connection, and select the most angle in the angle recognition results of all sample frames as the current character gait contour sequence angle judgment information; and obtain the angle judgment result of the character gait contour sequence by analogy; Step 4: Obtain the optimal model parameters of the GESS backbone network at different angles through angle-by-angle training, input the character gait contour sequence obtained in step 2 and recognized as 45°-90° in step 3 into the GESS backbone network with the optimal model parameters at different angles, extract frame-level features and sequence-level features, obtain fusion features, and obtain emotion inference results; Step 5: If the angle judgment result in step 3 is a video of a person walking at 0°-44°, the coordinate information of the person's skeleton joint points obtained in step 2 is input into the STEP backbone network to extract spatiotemporal features and obtain the emotion inference result; Step 6: Based on the results of step 4 and step 5, the emotion recognition result of the input person walking video is finally output.

2. The method for emotion recognition based on a person's gait video under angle selection according to claim 1, characterized in that: The step 2 specifically includes the following steps: Step 2-1: Pass the walking video of the person into the Robust Video Matting network to obtain a binary image sequence, compress and crop the binary image sequence to obtain a 64×64 pixel two-dimensional contour sequence, filter out the black frame images through the threshold, and readjust the sequence number; Step 2-2: Pass the walking video of the person into the VideoPose3D network. First, simulate the two-dimensional character key point information through target detection, upgrade the two-dimensional character joint point information to three-dimensional character joint point information through calculation, and sort and adjust the joint point information in the following order: Root, Spine, neck, head, Left shoulder, Left elbow, Left hand, Right shoulder, Right elbow, Right hand, Left hip, Left knee, Left foot, Right hip, Right knee, Right foot; sort these 15 nodes from 0 to 14, and obtain a .csv file recording the timing information as the input of the skeleton feature network.

3. The method for emotion recognition based on a person's gait video under angle selection according to claim 2 is characterized in that: The step 3 specifically comprises the following steps: Step 3-1: According to the input character gait contour sequence length k, evenly select Frame picture, obtain a set of sampled contour sequences; The sampled contour sequence sets obtained in step 3-2 and step 3-1 are sequentially input into the trained AG angle classifier for angle classification; The structure of the AG angle classifier consists of 3 layers of CNN convolutional layers, 3 layers of Pooling layers, a fully connected layer, and a Softmax layer. First, the 3×64×64 3D original contour features are convolved into a 64×8×8 3D feature vector through 3 layers of CNN convolutional layers and Pooling layers, and then the dimensions are compressed through the fully connected layer and normalized through the Softmax layer. Finally, a one-dimensional vector is obtained, which represents the predicted probability of each angle of the current frame, and the angle with the highest probability is selected as the output result. Step 3-3, counting the classification results of the sampled contour sequence obtained in step 3-2, counting the classification results, and taking the result with the highest number of times as the final sequence angle output result; Step 3-4: Select the angle with the highest number of single-frame angle predictions in the sampled contour sequence as the final output result of this sequence.

4. The method for emotion recognition based on a person's gait video under an angle selection according to claim 3 is characterized in that: The step 4 specifically comprises the following steps: Step 4-1: During the training process, the cross entropy loss function is selected as the emotion classification loss function, where l i represents the actual target distribution, y i represents the matching distribution of sentiment predictions; Step 4-2: The character frame-level feature extraction process is as follows: X={x j |j=1,2…,N} Where X is a 64×64 character gait contour sequence, j is a single-frame contour image in the character gait contour sequence, N is the length of each sequence; for each sequence, multiple layers of CNN convolutional neural network will be used to perform multiple frame-level feature extractions, and Conv represents a stage of convolution operation; where T F The result set extracted for each layer of CNN convolution, Represents the frame-level features extracted by the j-th layer CNN network; Step 4-3: The process of extracting character sequence-level features is as follows: Among them, cat represents the link on the first dimension in the feature, max represents the maximum pooling of frame-level features along the connected dimension, that is, for the current sequence, the most prominent features of each frame are collected according to the sequence number; Conv represents a stage of convolution operation, which realizes the learning of appropriate weights and adjusts the weights of the values ​​extracted by the statistical function. Represents direct addition of vectors; Step 4-4: The fusion of frame-level and sequence-level features is as follows: The contour sequence X generates frame-level features T through the frame-level and sequence-level feature extraction modules. F and sequence-level features T G ; The two emotion features extracted from each sequence are equally divided based on the height of the third dimension, each feature is divided into different groups, the groups are horizontally pooled, and the pooling results are merged into the final output feature f; In the frame-level feature T F and sequence-level features T G In the grouping process, H is the height of the character gait profile sequence diagram, first according to different scales The frame-level features T F and sequence-level features T G Reshape the original data into s groups, with the size of the third dimension of each group being size. Then, average pooling and maximum pooling are performed each time and the sum of the two results is taken. This preserves the different spatial position features of contour sequences at different scales and different receptive fields. Step 4-5: Through the one-dimensional convolution layer and the fully connected layer, the final output four-dimensional tensor Y = {y1, y2, y3, y4} is obtained as the final output of the four emotion prediction probabilities.

5. The method for emotion recognition based on a person's gait video under angle selection according to claim 4 is characterized in that: The step 5 specifically comprises the following steps: Step 5-1, input the coordinate information of the character skeleton joint points obtained in step 2; Step 5-2, feature engineering is performed on the coordinate information of the character skeleton joint points, including the angle information, distance information, formed triangle area information, node speed and inter-frame transformation features between different nodes as further input information of the skeleton feature extraction network; Step 5-5, through the one-dimensional convolution layer and the fully connected layer, obtain the final output four-dimensional tensor Y = {y1, y2, y3, y4} as the final output of the four emotion prediction probabilities.

6. The method for emotion recognition based on a person's gait video under angle selection according to claim 5, characterized in that: The step 6 specifically comprises the following steps: According to the emotion output result Y={y1,y2,y3,y4} obtained in step 4 or step 5, the four emotion prediction probabilities are used as the final output; the emotion prediction probability is obtained through the Softmax activation function, and the maximum probability Softmax(y i ) as the prediction result;

Images