A human body limb language emotion recognition method, device, equipment and medium

Abstract

The present application provides a human body limb language emotion recognition method, device, equipment and medium, belongs to human body limb recognition technical field, method includes: collecting video image, extracting the skeleton key point of video image, obtaining key point coordinate and confidence value. Extract the human body posture key point of preprocessed video image, form key point data in each frame, integrate the key point data of continuous frame into continuous frame key point sequence; the local feature of the input continuous frame key point sequence is extracted by using one-dimensional convolution network, then the local feature is modeled to time sequence emotion recognition model, accesses full connection layer and classification layer to realize emotion classification; the time sequence emotion recognition model is deployed. The present application realizes the video acquisition, image and data processing, model construction training to optimization deployment and frame rate adjustment process, can recognize the emotion expressed by human body limb in various scenes, and can be deployed on the equipment.

Description

Technical Field

[0001] This invention belongs to the field of human body recognition technology, specifically relating to a method, device, equipment, and medium for recognizing human body language emotions. Background Technology

[0002] In the field of intelligent human-computer interaction and social behavior analysis, nonverbal emotion recognition has become one of the key technologies. Currently, while facial expressions and voice tone modalities are widely studied, their recognition accuracy often declines in practical applications due to issues such as occlusion and limited data acquisition conditions. In contrast, body language (including posture, movements, and variations) has advantages such as being difficult to fake and having strong cross-cultural generalization, and is gradually becoming a new hotspot in emotion recognition research.

[0003] Spatiotemporal analysis based on 3D skeletal keypoints can extract emotion-related features from the overall human movement trajectory. For example, the STEP model uses a spatial-temporal graph convolutional network to extract features from walking sequences, enabling the classification of emotions such as "happy," "sad," "angry," and "neutral." Other studies combine RGB-D cameras with motion capture technology to achieve more precise kinematic data analysis.

[0004] However, existing methods lack the accuracy for extracting key points of human pose and cannot obtain the coordinates and confidence values ​​of key points of various parts of the human body; they lack multi-frame motion analysis and key point standardization processing, making it difficult to effectively capture pose information and eliminate the influence of individual differences; in multi-human scenarios, their ability to track individual key point sequences is weak and cannot meet the application needs of complex scenarios.

[0005] Related technologies using a certain neural network structure have difficulty simultaneously and effectively extracting local features of body movements and performing long-term sequence modeling, failing to fully explore the correlation between body movements and emotions, resulting in low accuracy in emotion recognition; during the training process, due to data imbalance and data augmentation, the model performs poorly in recognizing minority emotions, has insufficient generalization ability, and is poorly adaptable in practical applications. Summary of the Invention

[0006] This invention provides a method for human body language emotion recognition. The method realizes the process from video acquisition, image and data processing, model building and training to optimization deployment and frame rate adjustment. It can recognize the emotions expressed by human body language in various scenarios and can be deployed on devices.

[0007] The methods include: Step S101: Acquire video images and preprocess the video images; Step S102: Extract human pose key points from the preprocessed video image, perform human region detection, locate the coordinates and confidence values ​​of key points within the preset area, and form key point data in each frame. Integrate the key point data of consecutive frames into a consecutive frame key point sequence. Step S103: Construct a one-dimensional convolutional network and a long short-term memory network. Use the one-dimensional convolutional network to extract local features from the input continuous frame key point sequence. Then input the local features into the long short-term memory network to model the temporal emotion recognition model. Finally, connect the fully connected layer and the softmax classification layer to achieve emotion classification. Step S104: Train the temporal emotion recognition model using the sequence of key points in consecutive frames as training data, and introduce data augmentation strategies to improve the model's generalization ability. Step S105: Based on pruning methods and quantization tools, perform redundancy removal and weight quantization on the temporal emotion recognition model, and then deploy it.

[0008] It should be further explained that step S102 specifically includes: For the preprocessed single-frame grayscale image, edge detection and color histogram analysis are used to determine candidate regions in the image that may contain human bodies; Based on the heatmap prediction method, a probability heatmap of each pixel in the image belonging to human key points is output through a pre-trained network structure; non-maximum suppression is performed on the pixel with the highest probability in the heatmap to obtain an initial key point candidate set; combined with the human skeleton topological structure constraint, the candidate points are geometrically corrected to remove abnormal points that do not conform to the physiological structure, and the coordinates and confidence values ​​of 33 two-dimensional key points are determined. Based on keypoint coordinate data from at least three consecutive frames, the displacement difference of the same keypoint between adjacent frames is calculated to obtain the motion velocity of the keypoint; the angle change of adjacent joints is calculated using trigonometric function formulas, and the included angle is calculated using vector dot product formulas; the horizontal / vertical distance change between the shoulder and hip is extracted to form a sequence of posture information containing velocity, angle, and displacement. A tracking method based on detection box matching is adopted to generate a bounding box containing 33 key points for each detected subject; by comparing the positional overlap of bounding boxes between adjacent frames and the similarity of key point coordinates, the tracking association between subjects is established.

[0009] It should be further noted that step S102 also includes: The key point data of multiple consecutive frames are initially screened for quality, and frames with a confidence level higher than 0.6 and whose displacement of the same key point between adjacent frames does not exceed the normal range of human movement are retained to form an effective calculation frame sequence; For the same human keypoint in an effective frame sequence, by comparing its coordinate position in adjacent frames and combining it with the human motion continuity constraint, a unique correspondence between keypoints in consecutive frames is established. For the key points of the matched consecutive frames, the displacement difference between adjacent frames is calculated; based on the displacement difference and time interval, the motion speed is calculated; for the shoulder, elbow, and wrist, a vector is constructed using the coordinates of the three points, and the joint angle change is calculated using the formula of the angle between the vectors to reflect the bending or extension trend of the limb. The obtained velocity, angle changes, and relative displacement parameters are combined with the shoulder height and torso tilt at key points and arranged in chronological order to form a multi-dimensional posture information sequence.

[0010] It should be further noted that step S102 also includes: For the preprocessed single-frame grayscale image, a lightweight detector based on bounding boxes identifies human body regions, generates bounding boxes for each subject, and assigns a unique identifier to each subject to form the initial multi-subject detection results. In two consecutive frames, the positional overlap of the bounding boxes of each subject in the current frame and the bounding boxes of each subject in the previous frame is calculated. At the same time, the offset of the subject center point coordinates and the bounding box center is combined with the key point coordinates to establish the initial inter-frame subject association relationship, determine the subject in the previous frame corresponding to each subject in the current frame, and complete the initial matching of cross-frame subjects. When the overlap between the bounding box of a subject in the current frame and the bounding boxes of all subjects in the previous frame is less than a threshold, the subject is determined to be occluded or temporarily lost. Using the historical key point trajectories of the previous and earlier frames, the displacement direction is calculated by linear interpolation or based on the position changes of the previous 3 frames to generate the virtual key point coordinates of the current frame.

[0011] It should be further explained that step S103 specifically includes: Based on the feature dimension and time step of the input keypoint sequence, set the kernel size, number of kernels and stride of the one-dimensional convolutional network; Input a sequence of key points from consecutive frames into a 1D-CNN, and perform one-dimensional convolution operations on the key point coordinates on the time axis through a sliding window. During the convolution process, the ReLU activation function is used to enhance the nonlinear feature extraction capability, capture the small changes in joint positions between adjacent frames, and output local feature maps; The local feature sequence output by the 1D-CNN is input into a two-layer LSTM network. The first LSTM unit extracts short-term temporal dependencies and outputs a hidden state sequence. The second LSTM unit takes the hidden state of the first layer as input, captures long-term temporal patterns, and finally outputs a temporal feature vector. The temporal feature vector output from the two-layer LSTM is input into the fully connected layer, and the output of the fully connected layer is mapped to the probability distribution of each emotion category, thus outputting the final emotion recognition result.

[0012] It should be further noted that step S103 also includes: The number of nodes in the fully connected layer is set according to the emotion category; at the same time, the redundancy coefficient of the number of nodes is adjusted by combining the time dimension and feature dimension of the local feature map of the key point sequence. After the linear transformation of the fully connected layer, a nonlinear activation function is introduced to enhance the nonlinear combination capability of features through piecewise linearity; ReLU is used instead of traditional ReLU to address the sparsity of emotion features. At the output of the fully connected layer, an L2 regularization constraint is introduced to suppress the amplification of noise features by penalizing large weight parameters; at the same time, the output features are normalized. Before inputting the feature vector output from the fully connected layer into the softmax classification layer, the smoothness of the softmax layer is adjusted by the temperature parameter: when there is class imbalance in the training data, the temperature parameter is increased to reduce the probability peak of the high-confidence class. Finally, the probability distribution of each emotion class is output, and the classification layer parameters are optimized by backpropagation through the cross-entropy loss function.

[0013] It should be further explained that step S104 specifically includes: During training, basic weights are set based on the initial sample distribution; based on the 20%-80% training cycle as the middle of the training, the weights are adaptively adjusted by statistically analyzing the class error rate of the validation set in real time. Based on the 80th round of training as the later stage, the weight adjustment range is reduced to stabilize the model's ability to distinguish the majority class. The final loss function is:

[0014] Where w_c(t) is the weight of category c in round t, y_c is the true label, and ŷ_c is the predicted probability; The method employs phased training: The first stage uses keypoint sequences with a confidence level > 0.8 to train the basic feature extraction capability; The second stage introduces a confidence level of 0.5-0.8 and incorporates local feature perturbations to perform training; The third stage uses the full sample and adds cross-scenario interference; After each stage, the intermediate model is saved, and the optimal parameters of each stage are finally merged to form the final training model. Real-time monitoring of training loss, validation set accuracy, and class recall: When the validation set accuracy does not improve for three consecutive rounds, an early stopping mechanism is triggered to stop training and save the current best model.

[0015] This application also provides a human body language emotion recognition system, the system comprising: The video preprocessing module is used to acquire video images and perform preprocessing on them. The key point extraction and sequence construction module is used to extract human pose key points from preprocessed video images, perform human region detection, locate the coordinates and confidence values ​​of key points within a preset area, and form key point data in each frame. The key point data of consecutive frames are then integrated into a consecutive frame key point sequence. The feature extraction and emotion modeling module is used to construct a one-dimensional convolutional network and a long short-term memory network. The one-dimensional convolutional network is used to extract local features from the input continuous frame key point sequence, and then the local features are input into the long short-term memory network to model the temporal emotion recognition model. Finally, the fully connected layer and the softmax classification layer are connected to achieve emotion classification. The model training and optimization module is used to train the temporal emotion recognition model using a sequence of key points in consecutive frames as training data, while introducing data augmentation strategies to improve the model's generalization ability. The model compression and deployment module uses pruning methods and quantization tools to remove redundancy and quantize weights in the temporal sentiment recognition model, and then deploys it.

[0016] According to another embodiment of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the human body language emotion recognition method.

[0017] According to another embodiment of this application, a storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the human body language emotion recognition method.

[0018] As can be seen from the above technical solutions, the present invention has the following advantages: The human body language emotion recognition method involved in this application uses a USB interface camera and OpenCV library to acquire video images; the video frames are scaled to a uniform size and converted into grayscale images to reduce the amount of data computation. Based on the MediaPipePose framework, 33 two-dimensional key points are extracted with high precision to ensure data accuracy; a multi-frame motion analysis strategy is used to extract posture information, enhancing the semantic expression capability of features; the relative coordinates of key points are standardized to eliminate the influence of individual differences; and a target tracking algorithm in multi-human scenes ensures accurate tracking of individual key point sequences even in complex scenes.

[0019] A hybrid neural network architecture combining 1D-CNN and LSTM is employed. 1D-CNN effectively extracts subtle local motion features from keypoint sequences, while LSTM performs temporal modeling of actions. The combination fully explores the temporal and spatial correlation between body movements and emotions, achieving an effective mapping from body movements to emotions and improving the accuracy and reliability of emotion recognition. A weighted cross-entropy loss function addresses the data imbalance problem, preventing poor performance in recognizing minority emotions. Data augmentation strategies increase the diversity of training data, enabling the model to adapt to more complex scenarios and improving its generalization ability. Structured pruning removes redundant parts of the network, weight quantization reduces parameter precision, and multiple rounds of processing achieve model lightweighting, significantly reducing model size and improving inference speed. Attached Figure Description

[0020] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 Flowchart of a method for recognizing emotions through human body language; Figure 2 A schematic diagram of skeletal points recorded by the MediaPipePose algorithm. Figure 3 Flowchart of an embodiment of a method for recognizing human body language emotions; Figure 4 A schematic diagram of a human body language emotion recognition system; Figure 5 This is a schematic diagram of an electronic device. Detailed Implementation

[0022] The human body language emotion recognition method involved in this application uses a camera as an input device, combines the MediaPipePose model to extract 2D skeletal key points, and uses OpenCV to acquire and preprocess the video stream, automatically extracting image frames and performing scaling, grayscale conversion and noise suppression operations to obtain clear human movement features.

[0023] This invention constructs a deep learning model that integrates a 1D convolutional neural network (1D-CNN) and a long short-term memory network (LSTM) to model the temporal features of human actions and achieve emotion classification. After model training, lightweight optimization is performed using pruning and quantization tools in the PyTorch framework, significantly reducing the model's computational complexity and improving operational efficiency. Furthermore, the system incorporates a built-in frame rate adjustment mechanism that automatically adjusts the video frame rate based on processing performance, enabling real-time operation on low-cost devices. The final results are visualized through a web front-end interface, allowing users to view the emotion judgment results and historical emotion trends in real time.

[0024] The following describes in detail the human body language emotion recognition method involved in this application. Specific details, such as particular system structures and techniques, are presented for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details.

[0025] It should be understood that, when used in this specification, the term "comprising" indicates the presence of the described feature, integral, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0026] The terms "one embodiment" or "some embodiments" used in this application mean that one or more embodiments of this application include the specific features, structures, or characteristics described in that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this application do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Please see Figure 1 The diagram shows a flowchart of a method for recognizing human body language emotions in a specific embodiment. The method includes: Step S101: Acquire video images and preprocess the video images.

[0029] In some embodiments, the camera acquires video images. After system startup, the camera device is initialized and a continuous stream of video frames is captured by integrating the OpenCV open-source computer vision library. To balance recognition performance and operational efficiency, the system defaults to a video frame rate of 15 frames per second, and supports frame rate adjustment based on the operating environment. The frame rate can be flexibly switched within the range of 10–25 FPS to adapt to different computing resources and application scenarios, such as portable devices or edge terminals.

[0030] Before entering the processing flow, each video frame is first scaled to a uniform size (640×480 pixels) to ensure consistency of model input and reduce unnecessary computation. The image is then converted to grayscale to reduce the complexity of image processing while preserving the main spatial structural features, facilitating subsequent human pose recognition.

[0031] To further improve the accuracy and robustness of keypoint extraction, this embodiment employs Gaussian filtering or median filtering to denoise the image, reducing interference caused by lighting fluctuations, camera noise, or background disturbances. Furthermore, the system features automatic brightness and contrast adjustment, which enhances image clarity under different lighting conditions, improves the accuracy of skeletal point extraction, and enhances the accuracy and stability of subsequent skeletal keypoint extraction operations.

[0032] Step S102: Extract human pose key points from the preprocessed video image, perform human region detection, locate the coordinates and confidence values ​​of key points within the preset area, and form key point data in each frame. Integrate the key point data of consecutive frames into a consecutive frame key point sequence. For multi-human scenes, introduce a target tracking algorithm to maintain the consistency of the key point sequence.

[0033] In some embodiments, such as Figure 2 As shown, the MediaPipePose framework is used to first detect human body regions, and then perform key point localization within the detected human body regions to obtain the coordinates (x, y) and visibility confidence values ​​of 33 two-dimensional key points of the main parts such as the head, torso, and limbs, and output these data in a structured manner.

[0034] This embodiment uses a multi-frame motion analysis strategy to extract pose information such as velocity, angle changes, and relative displacement by utilizing the positional differences of key points between consecutive frames. By standardizing the relative coordinates between key points, the influence of individual height and camera position on pose distribution is eliminated.

[0035] It should be noted that the multi-frame motion analysis strategy uses Kalman filtering, SORT, and DeepSORT target tracking algorithms to associate key points in consecutive frames to form a trajectory, thereby achieving the tracking of the same target in a time series.

[0036] The key point data in the time series is analyzed to calculate the key point displacement, velocity, acceleration and other features between adjacent frames. The displacement is obtained by calculating the difference between the key point coordinates of adjacent frames by the difference method, and the velocity is obtained by dividing by the time interval. The velocity sequence is then differentiald and smoothed to obtain the acceleration.

[0037] Features such as the relative position and angle changes between key points are extracted, like calculating joint angles in human posture analysis. Displacement, velocity, and acceleration features are fused to form a comprehensive feature vector, which is then concatenated after normalization of different features. Based on the extracted motion features, a classification algorithm is used to train and classify different motion patterns, constructing a classification model. The extracted motion features serve as the model input, and the corresponding motion pattern labels serve as the output. The trained model can then classify and predict new motion data.

[0038] Optionally, in multi-person scenarios, the SORT target tracking algorithm maintains the consistency of keypoint sequences for each individual, ensuring data accuracy. Pose information extraction and standardization enhance the semantic expressive power of features, making keypoint data more reflective of real limb movements. Multi-person detection and tracking capabilities allow the system to be applied to complex scenarios, improving the applicability and practicality of the solution.

[0039] Step S103: Construct a one-dimensional convolutional network and a long short-term memory network. Use the one-dimensional convolutional network to extract local features from the input continuous frame key point sequence. Then input the local features into the long short-term memory network to model the temporal emotion recognition model. Finally, connect the fully connected layer and the softmax classification layer to achieve emotion classification.

[0040] In some embodiments, a hybrid neural network structure combining a 1D-CNN one-dimensional convolutional network and an LSTM long short-term memory network is constructed.

[0041] The 1D-CNN module extracts local features from the input keypoint sequence, extracting subtle action variations over a short period from continuous keypoint changes. After extracting multi-scale local features, the data enters a stacked LSTM network for temporal modeling. The LSTM units memorize long-term action trajectories, recognizing action trends driven by emotions. Finally, the model connects to a fully connected layer and a softmax classification layer to classify emotions such as happiness, tension, anxiety, anger, and indifference into multiple categories.

[0042] 1D-CNN uses convolutional kernels to slide along the time axis to capture local action features in keypoint sequences, enhancing the model's robustness to local perturbations; LSTM networks, with their memory units, can process and remember long-term action change information, mining action evolution patterns over time; fully connected layers integrate features, and softmax classification layers output the probabilities of different emotion categories based on the features, achieving emotion classification.

[0043] In this way, the combination of 1D-CNN and LSTM enables the model to capture both the subtle local changes in body movements and the long-term evolution trend of movements over time, effectively mining emotional cues in body movements and improving the accuracy and reliability of emotion recognition.

[0044] Step S104: Train the temporal emotion recognition model using the sequence of key points in consecutive frames as training data, and introduce data augmentation strategies to improve the model's generalization ability.

[0045] In some embodiments, a sequence of keypoints from consecutive frames is input as training data into a pre-constructed temporal emotion recognition model. During training, a weighted cross-entropy loss function is used to address the data imbalance problem, assigning higher weights to emotion categories with smaller sample sizes, thus making the model focus more on these categories. Simultaneously, data augmentation strategies, such as keypoint perturbation and time-scale stretching, are introduced to expand the diversity of the training data and simulate body movement data under different scenarios and individual differences.

[0046] This embodiment addresses the problem of data imbalance, preventing model bias due to differences in sample size and improving the ability to identify a few emotion categories. Data augmentation strategies increase the diversity of training data, enabling the model to adapt to more complex and varied real-world scenarios and enhancing its versatility and stability.

[0047] Step S105: Based on pruning methods and quantization tools, perform redundancy removal and weight quantization on the temporal emotion recognition model, and then deploy it.

[0048] In some embodiments, the model is built using the PyTorch framework, and its pruning and quantization toolchain is used to compress the model structure. First, a structured pruning method is used to remove redundant channels, convolutional kernels, or neurons from the network; the pruning ratio is adjusted according to the sensitivity of different layers. After pruning, post-training quantization is used to convert the model parameters from 32-bit floating-point numbers to 8-bit integers. Through multiple rounds of pruning and quantization, the original model size is compressed to 20%-30% of its original size, improving inference speed by more than 3 times, while maintaining the model's recognition accuracy fluctuation within 1%.

[0049] In some specific embodiments, redundant regions are identified by analyzing the importance of parameters in each layer of the temporal emotion recognition model; the L2 norm of the weight gradient of each layer is calculated to evaluate the importance between layers, and low-contribution layers are identified by combining the activation value variance; based on the evaluation results, measures are formulated such as prioritizing the pruning of fully connected layer channels, LSTM hidden units or 1D-CNN convolutional kernels with gradient norms less than a threshold to ensure that the temporal emotion recognition model maintains the integrity of the key feature transmission path after pruning.

[0050] This embodiment removes redundant channels, convolutional kernels, or neurons according to a pruning strategy. For 1D-CNN layers, low-importance channels (such as the bottom 20% of channels in terms of gradient norm) are removed after sorting by channel importance, and the size of subsequent convolutional kernels is adjusted to match the new input dimension. For LSTM layers, low-contribution hidden units (such as units with activation variance below a threshold) are removed, and the hidden state dimension is maintained through linear projection. For fully connected layers, redundant nodes (such as connections highly correlated with other nodes) are pruned, while core discriminative connections are retained. After pruning, the network structure is reconstructed to ensure that the input and output dimensions of each layer match.

[0051] For example, if the gradient norm of a 1D-CNN convolutional kernel is extremely small, it means that its contribution to feature extraction is negligible and can be safely removed; if the variance of the activation value of a hidden unit in an LSTM is close to 0, it means that its output hardly changes with the input and is a redundant unit.

[0052] Pruning ensures network functionality by preserving core feature paths. For example, when pruning low-gradient channels in a 1D-CNN, channels sensitive to changes in local joint angles are prioritized; when pruning hidden units in an LSTM, units crucial to long-term temporal emotional development processes are retained to avoid compromising the network's temporal modeling capabilities.

[0053] Weighted quantization reduces storage and computational overhead by lowering numerical precision. 8-bit integer quantization maps 32-bit floating-point numbers to integers in the range [-128, 127], achieved through scaling factors and zeros; a calibration dataset is used to determine the actual quantization range for each layer, ensuring minimal quantization error.

[0054] Finally, lightweight model validation ensures practicality. For example, on the test set, the original model achieved an accuracy of 92%, while the compressed model achieved an accuracy of 91.5%, with a fluctuation of 0.5% ≤ 1%. The original model had a single-frame processing time of 80ms, while the compressed model had a processing time of 25ms, meeting the real-time requirements of embedded devices.

[0055] This significantly reduces the size of the temporal emotion recognition model, lowers the demand for hardware resources, and allows the model to be deployed on resource-constrained platforms such as embedded devices and mobile terminals, enabling rapid emotion recognition responses and meeting the needs of real-time applications.

[0056] In one embodiment of the present invention, based on step S102, the following will provide a possible embodiment and its specific implementation will be described in a non-limiting manner. Step S102 specifically includes: Step S1021: For the preprocessed single-frame grayscale image, determine the candidate regions in the image that may contain human bodies through edge detection and color histogram analysis; exclude obvious non-human regions in the image, such as solid color backgrounds, animal outlines or low-contrast noise regions, to narrow the search range for subsequent key point detection and improve detection efficiency.

[0057] It should be noted that, based on edge detection algorithms and color histogram statistics, i.e. skin color region screening, the image is converted from the RGB color space to the YCrCb space, and the histogram distribution of the Cr component is statistically analyzed; a skin color Cr range is set, and the percentage of Cr pixels in the image within this range is calculated. If the percentage exceeds the threshold, it is determined that the region may contain a human body.

[0058] Step S1022: A deep learning-based heatmap prediction method outputs a probability heatmap of each pixel in the image belonging to a human keypoint through a pre-trained network structure; non-maximum suppression is applied to the pixel with the highest probability in the heatmap to obtain an initial keypoint candidate set; combined with human skeletal topological constraints, such as the relative positional relationship between the shoulder and elbow, the relative positional relationship between the leg and the body, and the relative positional relationship between the head and the neck, geometric correction is performed on the candidate points to remove abnormal points that do not conform to the physiological structure, and finally the coordinates (x, y) and confidence values ​​(visibility) of 33 two-dimensional keypoints are determined.

[0059] In this embodiment, the coordinates of the shoulder keypoint can be set as (Sx, Sy), the elbow as (Elx, Ely), and the wrist as (Wx, Wy). Then the vector from the shoulder to the elbow is (Elx-Sx, Ely-Sy), and the vector from the elbow to the wrist is (Wx-Elx, Wy-Ely). According to the human physiological structure, the length L1 from the shoulder to the elbow should be approximately 1.5 times the length L2 from the elbow to the wrist. If L2 > L1 × 1.8, then the candidate point of the wrist is determined to be a false keypoint and is eliminated.

[0060] Step S1023: Based on key point coordinate data of at least 3 consecutive frames, calculate the displacement difference of the same key point between adjacent frames to obtain the motion velocity of the key point; calculate the angle change of adjacent joints using trigonometric function formulas, such as the elbow angle which is a vector composed of the coordinates of the shoulder, elbow and wrist, and calculate the angle using the vector dot product formula; extract the horizontal / vertical distance change between the shoulder and hip to form a sequence of posture information containing velocity, angle and displacement.

[0061] For step S1023, the following steps are also involved: Step S10231: Perform initial quality screening on key point data of multiple consecutive frames, and exclude invalid data based on key point confidence and position stability; retain frames with confidence higher than 0.6 and displacement of the same key point between adjacent frames not exceeding the normal range of human movement, such as single frame displacement ≤ 0.3 meters, to form a valid calculation frame sequence.

[0062] Step S10232: For the same human keypoint in the valid frame sequence, compare its coordinate positions in adjacent frames, taking into account human motion continuity constraints, such as joint movement speed not exceeding physiological limits. Establish a unique correspondence between keypoints in consecutive frames.

[0063] Step S10233: For the key points of the matched consecutive frames, calculate the displacement difference (Δx=x) between adjacent frames. t ₊1-x t Δy=y t ₊1-y t Based on the displacement difference and time interval, Δt=1 / frame rate, the motion speed is calculated (v=√(Δx²+Δy²) / Δt); for adjacent joints, such as the shoulder, elbow, and wrist, a vector is constructed using the coordinates of three points, and the joint angle change is calculated using the formula for the angle between vectors, reflecting the bending or extension trend of the limb.

[0064] Step S10234: Combine the calculated parameters such as velocity, angle change, and relative displacement with the shoulder height and torso tilt of key points, and arrange them in chronological order to form a multi-dimensional posture information sequence; smooth out abnormal fluctuations, eliminate noise interference, and ensure the interpretability and stability of the sequence.

[0065] This embodiment establishes key point correspondence based on the continuity constraint of human movement, which conforms to the actual movement law of the human body and ensures the rationality and accuracy of key point matching; it obtains information such as the speed and angle changes of key points, which can accurately reflect the state and trend of limb movement; it combines parameters with static features to form a multi-dimensional information sequence and performs smoothing processing, so that the final posture information sequence improves the effectiveness and reliability of the entire emotion recognition method.

[0066] Step S1024: For cases where there are multiple subjects in the same frame, a tracking method based on detection box matching is adopted to generate a bounding box containing 33 key points for each detected subject; by comparing the positional overlap of the bounding boxes between adjacent frames and the similarity of the key point coordinates, the tracking association between subjects is established; for tracking interruptions caused by occlusion or brief departure from the frame, the lost sequence data is recovered by linear interpolation of the historical key point trajectory to maintain the temporal continuity of the key point sequence of a single subject.

[0067] As can be seen, for multi-subject scenes, a detection-bound box matching-based tracking method is adopted to generate bounding boxes containing 33 key points for each subject. Tracking associations are established by comparing the overlap of bounding box positions in adjacent frames with the similarity of key point coordinates. For tracking interruptions caused by occlusion, lost data is recovered through linear interpolation of historical key point trajectories, maintaining the temporal continuity of the key point sequence for a single subject. The entire process, from narrowing the detection range and accurately locating key points to extracting information and achieving multi-subject tracking, realizes the accurate extraction and processing of key points of the human skeleton.

[0068] In this way, by using multi-frame data to calculate speed, angle, and displacement to form a sequence of posture information, the details of human movement changes can be effectively captured, enhancing the semantic expression capability of features. For multiple subjects, detection box matching tracking and interpolation recovery data are used to ensure accurate tracking of key point sequences of a single individual even in complex scenes, achieving accurate tracking of emotional changes of a single individual, and improving the applicability and reliability of the entire method in various scenarios.

[0069] Step S1024 of this embodiment further includes the following steps: Step S10241: For the preprocessed single-frame grayscale image, a lightweight detector based on bounding boxes identifies all possible human body regions, generates a bounding box for each subject, and the bounding box contains the coordinates of the upper left corner, the coordinates of the lower right corner, and width and height information. A unique identifier is assigned to each subject to form the initial multi-subject detection result and clarify the independent human targets existing in the current frame.

[0070] Step S10242: In two consecutive frames, calculate the positional overlap between the bounding boxes of each subject in the current frame and the bounding boxes of each subject in the previous frame. At the same time, combine the subject center point coordinates of the key point coordinates with the offset of the bounding box center to establish the initial inter-frame subject association relationship, determine the subject in the previous frame corresponding to each subject in the current frame, and complete the preliminary matching of cross-frame subjects.

[0071] Step S10243: When the overlap between the bounding box of a subject in the current frame and the bounding boxes of all subjects in the previous frame is lower than a threshold, or the average confidence of the subject's key points is lower than a set value, the average confidence of all 33 key points is used to determine that the subject is occluded or temporarily lost; using the historical key point trajectories of the previous and earlier frames, the displacement direction is calculated by linear interpolation or based on the position changes of the previous 3 frames to generate virtual key point coordinates for the current frame, maintaining the continuity of the subject's key point sequence and avoiding sequence interruption due to temporary occlusion.

[0072] Step S10244: For the established subject association relationship, verify the rationality of the association by comparing the joint angle distribution of key points in the current frame with that in the previous frame, such as the shoulder-elbow-wrist angle, hip-knee-ankle angle, and torso tilt angle.

[0073] If the key point angular distributions of different subjects are highly similar, leading to mismatches, then the nearest reasonable subject will be rematched, such as selecting the unmatched subject with the closest angular distribution, to correct the erroneous tracking association and ensure the continuity and physiological rationality of the key point sequence of a single subject in the time dimension.

[0074] It can be seen that the core objective of step S1024 is to maintain the temporal consistency of the key point sequence of a single subject in a multi-human scene through multi-dimensional information fusion. The principle is based on the physiological regularity of human movement and the complementarity of multi-frame information.

[0075] In other words, the initial multi-subject detection uses a human detection model to clearly identify individual human targets in the current frame. This step directly locates the human body region through bounding box detection, avoiding subject confusion caused by false detections of key points. For example, when there are two overlapping human bodies in the image, the bounding box can intuitively distinguish their physical boundaries, providing a spatial basis for subsequent association.

[0076] Secondly, the initial cross-frame association combines bounding box overlap and keypoint spatial distribution to improve matching accuracy. In this step, bounding box overlap reflects the positional continuity of the subject, while the offset between the keypoint center and the bounding box center reflects the pose consistency of the subject. Combining the two can effectively eliminate mismatches between different subjects. For example, if the bounding box center of subject A in the previous frame is (x1, y1), and the bounding box center of candidate subject B in the current frame is (x2, y2), if IoU > 0.6 and the offset < 5 pixels, then they are determined to be the same subject.

[0077] Furthermore, sequence restoration for occluded or missing subjects utilizes historical trajectories and motion direction predictions to address sequence interruptions caused by brief occlusions. When a subject is occluded, the confidence level of keypoint detection decreases and the overlap of bounding boxes decreases. At this point, the system retrieves the keypoint coordinates from the previous three frames, calculates the displacement direction of the subject, and generates virtual keypoints.

[0078] For example, if a subject is occluded and only 2 key points are detected in the current frame with a confidence level of 0.4, but the displacement direction of the previous 3 frames is 0.1 meters to the right per frame, then the position of the key point in the current frame is predicted to be 0.1 meters to the right of the position in the previous frame, thus maintaining the continuity of the sequence.

[0079] Finally, the consistency verification process ensures the rationality of the association through physiological characteristic constraints. Human joint angles have physiological ranges; optionally, for example, the elbow angle is typically between 90° and 180°.

[0080] If the keypoint angle distribution of a subject suddenly changes to 0°-30°, it is considered an association error. For example, if the keypoint angle distribution of subject A is normal, but the angle distribution of candidate subject B is abnormal, other more reasonable unassociated subjects will be re-matched to avoid subsequent feature extraction errors due to misassociations.

[0081] In one embodiment of the present invention, based on step S103, the following will provide a possible embodiment and its specific implementation will be described in a non-limiting manner. For example... Figure 3 As shown, step S103 specifically includes: Step S1031: Based on the feature dimension and time step of the input keypoint sequence, set the kernel size, number of kernels and stride of the one-dimensional convolutional network.

[0082] The number of hidden units and layers of the LSTM Long Short-Term Memory network are set synchronously to form a hierarchical structure of 1D, CNN, LSTM, and fully connected layers, ensuring that the network input and output dimensions match the temporal and spatial features of the key point sequence.

[0083] Step S1032: Input the sequence of key points in consecutive frames into the 1D-CNN, and perform one-dimensional convolution operation on the key point coordinates on the time axis through a sliding window; During the convolution process, the ReLU activation function is used to enhance the non-linear feature extraction capability, capture the small changes in joint position between adjacent frames, such as elbow tremors, wrist shifts, and changes in the horizontal distance between the shoulder and hip, and output local feature maps.

[0084] Step S1033: Input the local feature sequence output by the 1D-CNN into a two-layer LSTM network. The first LSTM unit extracts short-term temporal dependencies, such as action trends within 0.5 seconds, and outputs a hidden state sequence. The second LSTM unit takes the hidden states of the first layer as input and captures long-term temporal patterns, such as the emotional development process within 3 seconds, such as the gradual change from calm to tension, and finally outputs a temporal feature vector.

[0085] Step S1034: Input the temporal feature vector output by the dual-layer LSTM into the fully connected layer, and enhance the discriminative power of the features through linear transformation and nonlinear activation; finally, connect to the softmax classification layer, map the output of the fully connected layer to the probability distribution of each emotion category, and output the final emotion recognition result.

[0086] In this embodiment, the input keypoint sequence has a time step size of T and a spatial dimension of 66. Each keypoint contains x / y coordinates. The 1D-CNN convolutional kernel size is set to cover the keypoint changes in three adjacent frames, with 16 kernels to extract 16 different local feature patterns. The stride is 1 frame. The LSTM hidden units are set to 64. The first layer extracts short-term dependencies, and the second layer integrates long-term trends to match the time scale of emotion development.

[0087] 1D-CNN filters the input sequence through one-dimensional convolution operations to extract local features. For example, if it detects that the x-coordinate of the elbow increases continuously within 3 frames and the y-coordinate fluctuates by less than 0.01 meters, it is determined to be a micro-movement of the arm extending forward, corresponding to local features of tension or anger.

[0088] A two-layer LSTM processes local feature sequences through input, forget, and output gates. The first LSTM layer receives local features from the 1D-CNN output and outputs short-term hidden states. The second LSTM layer takes the first layer's hidden states as input, integrates long-term temporal information, and outputs a final temporal feature vector. For example, if an arm slowly extends forward for the first two seconds and then the extension speed increases for the next second, the LSTM captures a long-term pattern of gradually escalating emotions.

[0089] The fully connected layer maps 64-dimensional temporal features to the emotion category space and enhances the non-linear discriminative ability of the features through the Sigmoid activation function; the softmax layer converts 10-dimensional features into 5 probabilities (e.g., happy 0.1, tense 0.7, anxious 0.1, angry 0.05, indifferent 0.05) and outputs the most likely emotion category.

[0090] Furthermore, step S1034 of this embodiment specifically includes: Step S10341: Based on the emotion category, such as 5 categories: happy, nervous, anxious, angry, and indifferent, set the number of nodes in the fully connected layer; at the same time, combine the time dimension and feature dimension of the local feature map of the key point sequence, adjust the redundancy coefficient of the number of nodes to ensure that the fully connected layer can fully extract the discriminative information of high-dimensional features.

[0091] Step S10342: After the linear transformation of the fully connected layer, a nonlinear activation function is introduced to enhance the nonlinear combination ability of features through piecewise linearity; to address the sparsity of emotion features, ReLU with leakage is used instead of traditional ReLU to retain more potential effective features.

[0092] Step S10343: At the output of the fully connected layer, an L2 regularization constraint is introduced to suppress the excessive amplification of noise features by penalizing large weight parameters; at the same time, the output features are normalized to eliminate the influence of feature scale differences on classification, ensuring that the features of different emotion categories are more concentrated in space and improving the clarity of classification boundaries.

[0093] Step S10344: Before inputting the feature vector output by the fully connected layer into the softmax classification layer, adjust the smoothness of the softmax by using the temperature parameter: when there is class imbalance in the training data, increase the temperature parameter to reduce the probability peak of the high-confidence class and alleviate the class dominance problem; finally, output the probability distribution of each emotion class, and optimize the classification layer parameters by backpropagation through the cross-entropy loss function.

[0094] It can be seen that by setting the number of nodes and redundancy coefficients of the fully connected layers according to the emotion category and feature dimension, the network structure is adapted to the emotion recognition task, improving the targeting of feature extraction. The ReLU activation function is used to effectively deal with the sparsity of emotion features and enhance the network's ability to capture and express emotion features. L2 regularization and normalization processes suppress noise interference, eliminate feature scale differences, and improve the stability and accuracy of the classification model. The smoothness of softmax is adjusted by the temperature parameter, which effectively solves the problem of class imbalance in training data. Combined with the cross-entropy loss function to optimize parameters, the accuracy and reliability of emotion classification are improved.

[0095] In one embodiment of the present invention, based on step S104, the following will provide a possible embodiment and its specific implementation will be described in a non-limiting manner. Step S104 specifically includes: Step S1041: For the training data of continuous frame keypoint sequences, design a multi-dimensional data augmentation method: perform random time offset in the time dimension to simulate the continuity of action at different time starting points; add Gaussian noise to the keypoint coordinates in the spatial dimension to simulate camera shake or detection error; in the sample dimension, randomly replace low-confidence keypoints to supplement sparse samples; at the same time, perform random occlusion simulation on the keypoint sequences of multi-human scenes to improve the robustness of the model to incomplete data.

[0096] Step S1042: In the early stage of training, set basic weights based on the initial sample distribution; during the 20%-80% training epochs, adaptively adjust the weights by statistically analyzing the class error rate of the validation set in real time; after 80% of the training epochs, reduce the magnitude of weight adjustments to stabilize the model's ability to distinguish the majority class. The final loss function is:

[0097] Where w_c(t) is the weight of category c in round t, y_c is the true label, and ŷ_c is the predicted probability.

[0098] Step S1043: Improve the model's ability to capture complex emotions through phased training: In the first phase, keypoint sequences with confidence > 0.8 are used to train the basic feature extraction capability; The second stage introduces a confidence level of 0.5-0.8 and adds local feature perturbations, such as joint angle changes of ±10°, to enhance the model's sensitivity to micro-movements. The third stage uses the full set of samples and adds cross-scene interference to improve the model's generalization ability; after each stage, the intermediate model is saved, and finally the optimal parameters of each stage are merged to form the final training model.

[0099] Step S1044: Real-time monitoring of training loss, validation set accuracy, and class recall: When the validation set accuracy does not improve for three consecutive rounds, trigger the early stopping mechanism, stop training, and save the current best model; adjust the learning rate according to the rate of change of training loss; for classes with low recall, increase their sample proportion in subsequent rounds to specifically improve the discrimination ability of weak links.

[0100] Step S104 in this embodiment enables the model to learn more correspondences between body movements and emotions in different states, thereby improving the model's generalization ability; adjusting the weights of the loss function can effectively address the problem of imbalanced training data, prevent the model from being biased towards the majority class, and improve the ability to recognize minority class emotions; the phased training strategy guides the training step by step according to the different ability requirements of the model, from basic feature extraction to micro-motion capture and then to complex scene adaptation, progressively improving the model's ability to capture complex emotions; real-time monitoring and adjustment of the training strategy according to indicators can promptly identify and optimize training problems, trigger an early stopping mechanism to prevent overfitting, adjust the learning rate to accelerate model convergence, and ultimately improve the overall accuracy of the model.

[0101] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0102] The following are embodiments of the human body language emotion recognition system provided in this disclosure. This system and the human body language emotion recognition methods in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the human body language emotion recognition system, please refer to the embodiments of the above human body language emotion recognition methods.

[0103] like Figure 4 As shown, the system includes: The video preprocessing module is used to acquire video images and preprocess them.

[0104] The key point extraction and sequence construction module is used to extract human pose key points from preprocessed video images, perform human region detection, locate the coordinates and confidence values ​​of key points within a preset area, and form key point data in each frame. The key point data of consecutive frames are then integrated into a consecutive frame key point sequence.

[0105] The feature extraction and emotion modeling module is used to construct a one-dimensional convolutional network and a long short-term memory network. The one-dimensional convolutional network is used to extract local features from the input continuous frame key point sequence. The local features are then input into the long short-term memory network to model the temporal emotion recognition model. Finally, the fully connected layer and softmax classification layer are connected to achieve emotion classification.

[0106] The model training and optimization module is used to train the temporal emotion recognition model using a sequence of key points in consecutive frames as training data, while introducing data augmentation strategies to improve the model's generalization ability.

[0107] The model compression and deployment module uses pruning methods and quantization tools to remove redundancy and quantize weights in the temporal sentiment recognition model, and then deploys it.

[0108] like Figure 5 As shown, this application also provides an electronic device, including a display module 103, a memory 102, a processor 101, and a computer program stored in the memory and executable on the processor 101. When the processor 101 executes the program, it implements the steps of a human body language emotion recognition method.

[0109] In embodiments of the present invention, electronic devices include, but are not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments described and / or claimed herein.

[0110] In this embodiment, processor 101 may be implemented using at least one of an application-specific integrated circuit, a programmable logic device, a field-programmable gate array, a processor, a controller, a microcontroller, a microprocessor, or an electronic unit designed to perform the functions described herein. In some cases, such an implementation may be implemented within a controller. For software implementation, implementations such as processes or functions may be implemented with separate software modules that allow the performance of at least one function or operation. Software code may be implemented by a software application (or program) written in any suitable programming language, and the software code may be stored in memory and executed by the controller.

[0111] The display module 103 is used to display information input by the user or information provided to the user. The display module 103 may include a display panel, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like.

[0112] The memory 102 can be used to store software programs and various data. The memory 102 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0113] This application also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the human body language emotion recognition method.

[0114] The storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0115] In a storage medium, a readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0116] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for recognizing emotions through human body language, characterized in that, The methods include: Step S101: Acquire video images and preprocess the video images; Step S102: Extract human pose key points from the preprocessed video image, perform human region detection, locate the coordinates and confidence values ​​of key points within the preset area, and form key point data in each frame. Integrate the key point data of consecutive frames into a consecutive frame key point sequence. Step S102 specifically includes: For the preprocessed single-frame grayscale image, edge detection and color histogram analysis are used to determine candidate regions in the image that may contain human bodies; Based on the heatmap prediction method, a probability heatmap of each pixel in the image belonging to human keypoints is output through a pre-trained network structure; non-maximum suppression is performed on the pixel with the highest probability in the heatmap to obtain an initial keypoint candidate set; combined with the human skeleton topological structure constraint, the candidate points are geometrically corrected to determine the coordinates and confidence values ​​of 33 two-dimensional keypoints. Based on keypoint coordinate data from at least three consecutive frames, the displacement difference of the same keypoint between adjacent frames is calculated to obtain the motion velocity of the keypoint; the angle change of adjacent joints is calculated using trigonometric function formulas, and the included angle is calculated using vector dot product formulas; the horizontal / vertical distance change between the shoulder and hip is extracted to form a sequence of posture information containing velocity, angle, and displacement. A tracking method based on detection box matching is adopted to generate a bounding box containing 33 key points for each detected subject; by comparing the positional overlap of bounding boxes between adjacent frames and the similarity of key point coordinates, the tracking association between subjects is established. Step S103: Construct a one-dimensional convolutional network and a long short-term memory network. Use the one-dimensional convolutional network to extract local features from the input continuous frame key point sequence. Then input the local features into the long short-term memory network to model the temporal emotion recognition model. Finally, connect the fully connected layer and the softmax classification layer to achieve emotion classification. Step S104: Train the temporal emotion recognition model using the sequence of key points in consecutive frames as training data, and introduce data augmentation strategies to improve the model's generalization ability. Step S105: Based on pruning methods and quantization tools, perform redundancy removal and weight quantization on the temporal emotion recognition model, and then deploy it.

2. The method for recognizing human body language emotions according to claim 1, characterized in that, Step S102 also includes: The key point data of multiple consecutive frames are initially screened for quality, and frames with a confidence level higher than 0.6 and whose displacement of the same key point between adjacent frames does not exceed the normal range of human movement are retained to form an effective calculation frame sequence; For the same human keypoint in an effective frame sequence, by comparing its coordinate position in adjacent frames and combining it with the human motion continuity constraint, a unique correspondence between keypoints in consecutive frames is established. For the key points of the matched consecutive frames, the displacement difference between adjacent frames is calculated; based on the displacement difference and time interval, the motion speed is calculated; for the shoulder, elbow, and wrist, a vector is constructed using the coordinates of the three points, and the joint angle change is calculated using the formula of the angle between the vectors to reflect the bending or extension trend of the limb. The obtained velocity, angle changes, and relative displacement parameters are combined with the shoulder height and torso tilt at key points and arranged in chronological order to form a multi-dimensional posture information sequence.

3. The method for recognizing human body language emotions according to claim 1, characterized in that, Step S102 also includes: For the preprocessed single-frame grayscale image, a lightweight detector based on bounding boxes identifies human body regions, generates bounding boxes for each subject, and assigns a unique identifier to each subject to form the initial multi-subject detection results. In two consecutive frames, the positional overlap of the bounding boxes of each subject in the current frame and the bounding boxes of each subject in the previous frame is calculated. At the same time, the offset of the subject center point coordinates and the bounding box center is combined with the key point coordinates to establish the initial inter-frame subject association relationship, determine the subject in the previous frame corresponding to each subject in the current frame, and complete the initial matching of cross-frame subjects. When the overlap between the bounding box of a subject in the current frame and the bounding boxes of all subjects in the previous frame is less than a threshold, the subject is determined to be occluded or temporarily lost. Using the historical key point trajectories of the previous and earlier frames, the displacement direction is calculated by linear interpolation or based on the position changes of the previous 3 frames to generate the virtual key point coordinates of the current frame.

4. The method for recognizing human body language emotions according to claim 1, characterized in that, Step S103 specifically includes: Based on the feature dimension and time step of the input keypoint sequence, set the kernel size, number of kernels and stride of the one-dimensional convolutional network; Input a sequence of key points from consecutive frames into a 1D-CNN, and perform one-dimensional convolution operations on the key point coordinates on the time axis through a sliding window. During the convolution process, the ReLU activation function is used to enhance the nonlinear feature extraction capability, capture the small changes in joint positions between adjacent frames, and output local feature maps; The local feature sequence output by the 1D-CNN is input into a two-layer LSTM network. The first LSTM unit extracts short-term temporal dependencies and outputs a hidden state sequence. The second LSTM unit takes the hidden state of the first layer as input, captures long-term temporal patterns, and finally outputs a temporal feature vector. The temporal feature vector output from the two-layer LSTM is input into the fully connected layer, and the output of the fully connected layer is mapped to the probability distribution of each emotion category, thus outputting the final emotion recognition result.

5. The method for recognizing human body language emotions according to claim 4, characterized in that, Step S103 also includes: The number of nodes in the fully connected layer is set according to the emotion category; at the same time, the redundancy coefficient of the number of nodes is adjusted by combining the time dimension and feature dimension of the local feature map of the key point sequence. After the linear transformation of the fully connected layer, a nonlinear activation function is introduced to enhance the nonlinear combination capability of features through piecewise linearity; ReLU is used instead of traditional ReLU to address the sparsity of emotion features. At the output of the fully connected layer, an L2 regularization constraint is introduced to suppress the amplification of noise features by penalizing large weight parameters; at the same time, the output features are normalized. Before inputting the feature vector output from the fully connected layer into the softmax classification layer, the smoothness of the softmax layer is adjusted by the temperature parameter: when there is class imbalance in the training data, the temperature parameter is increased to reduce the probability peak of the high-confidence class. Finally, the probability distribution of each emotion class is output, and the classification layer parameters are optimized by backpropagation through the cross-entropy loss function.

6. The method for recognizing human body language emotions according to claim 1, characterized in that, Step S104 specifically includes: During training, basic weights are set based on the initial sample distribution; based on the 20%-80% training cycle as the middle of the training, the weights are adaptively adjusted by statistically analyzing the class error rate of the validation set in real time. Based on the 80th round of training as the later stage, the weight adjustment range is reduced to stabilize the model's ability to distinguish the majority class. The final loss function is: Where w_c(t) is the weight of category c in round t, and y_c is the true label. _c represents the predicted probability; The method employs phased training: The first stage uses keypoint sequences with a confidence level > 0.8 to train the basic feature extraction capability; The second stage introduces a confidence level of 0.5-0.8 and incorporates local feature perturbations to perform training; The third stage uses the full sample and adds cross-scenario interference; After each stage, the intermediate model is saved, and the optimal parameters of each stage are finally merged to form the final training model. Real-time monitoring of training loss, validation set accuracy, and class recall: When the validation set accuracy does not improve for three consecutive rounds, an early stopping mechanism is triggered to stop training and save the current best model.

7. A human body language emotion recognition system, characterized in that, The system is used to implement the human body language emotion recognition method as described in any one of claims 1 to 6; The system includes: The video preprocessing module is used to acquire video images and perform preprocessing on them. The key point extraction and sequence construction module is used to extract human pose key points from preprocessed video images, perform human region detection, locate the coordinates and confidence values ​​of key points within a preset area, and form key point data in each frame. The key point data of consecutive frames are then integrated into a consecutive frame key point sequence. The feature extraction and emotion modeling module is used to construct a one-dimensional convolutional network and a long short-term memory network. The one-dimensional convolutional network is used to extract local features from the input continuous frame key point sequence, and then the local features are input into the long short-term memory network to model the temporal emotion recognition model. Finally, the fully connected layer and the softmax classification layer are connected to achieve emotion classification. The model training and optimization module is used to train the temporal emotion recognition model using a sequence of key points in consecutive frames as training data, while introducing data augmentation strategies to improve the model's generalization ability. The model compression and deployment module uses pruning methods and quantization tools to remove redundancy and quantize weights in the temporal sentiment recognition model, and then deploys it.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the human body language emotion recognition method as described in any one of claims 1 to 6.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the human body language emotion recognition method as described in any one of claims 1 to 6.

Images