Gait data-based emotion classification method and device, equipment and storage medium

Abstract

The invention discloses an emotion classification method and device based on gait data, equipment and a storage medium, relates to the technical field of emotion recognition, and discloses an emotion classification method based on the gait data, and the method comprises the steps: obtaining human gait video data, extracting skeleton three-dimensional coordinate data according to the human gait video data, and storing the skeleton three-dimensional coordinate data in a database; performing Gaussian filtering processing on the skeleton three-dimensional coordinate data to obtain skeleton joint point data; performing kinetic energy and potential energy calculation on the bone joint point data to obtain a joint energy mean value; performing rigid body representation reconstruction on the joint energy mean value, and constructing a reconstructed skeleton structure; performing normalization processing on a kinetic energy matrix and a potential energy matrix in the reconstructed skeleton structure to obtain a normalized matrix, and performing fast Fourier transform on the normalized matrix to obtain frequency domain features; and performing feature selection and dimension reduction processing according to the frequency domain features to obtain emotion classification data. The technical effect of accurately classifying and recognizing emotions is achieved.

Description

Technical Field

[0001] This application relates to the field of emotion recognition technology, and in particular to emotion classification methods, devices, equipment and storage media based on gait data. Background Technology

[0002] In recent years, some scholars have found that an individual's gait behavior can reflect their psychological state to some extent. People with poor psychological state, such as patients with depression, often exhibit characteristics such as slower walking speed, shorter stride, forward leaning posture, and poor coordination between the upper and lower body. Therefore, emotions can be identified through external characteristics.

[0003] In the field of emotion recognition technology based on external features, existing solutions attempt to collect human skeletal data and assess an individual's emotional state by statistically analyzing gait characteristics or the periodic trends of joint changes. Traditional solutions mainly rely on surface information such as stride length, cadence, and limb swing amplitude, using machine learning models for classification, and have initially achieved the identification of high-risk groups for depression. However, these methods generally ignore dynamic features with more physiological interpretability, such as movement amplitude, rhythmic changes, and kinetic energy distribution. Furthermore, the actual collected skeletal data often contains noise and redundant information, failing to effectively filter out invalid or low-relevance limb areas, resulting in low model training efficiency, susceptibility to perturbations, and inaccurate emotion recognition.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a method, device, equipment and storage medium for emotion classification based on gait data, which aims to solve the technical problem that the energy change characteristics of different joints in gait cannot be accurately extracted and expressed, resulting in insufficient accuracy in the acquisition of emotion-related behavioral data.

[0006] To achieve the above objectives, this application proposes a sentiment classification method based on gait data, the method comprising:

[0007] Acquire human gait video data, extract skeletal three-dimensional coordinate data based on the human gait video data, and perform Gaussian filtering on the skeletal three-dimensional coordinate data to obtain skeletal joint data;

[0008] The kinetic and potential energy of the skeletal joint data are calculated to obtain the average joint energy.

[0009] The average joint energy is reconstructed using rigid body representation to build a reconstructed skeleton structure;

[0010] The kinetic energy matrix and potential energy matrix in the reconstructed skeleton structure are normalized to obtain a normalized matrix, and the normalized matrix is ​​subjected to a fast Fourier transform to obtain frequency domain features.

[0011] Based on the frequency domain features, feature selection and dimensionality reduction are performed to obtain emotion classification data.

[0012] In one embodiment, the step of calculating the kinetic and potential energy of the skeletal joint data to obtain the average joint energy includes:

[0013] Based on the skeletal joint data, the ratio of the Euclidean displacement of the skeleton points to the sampling time interval is calculated in consecutive frames to obtain the velocity value, and the joint kinetic energy component is constructed based on the velocity value.

[0014] In the skeletal joint data, gait neutral posture data is selected as the reference posture, and the relative potential energy value of the joint is calculated based on the reference posture and joint coordinates. The gait neutral posture data is frame image data that includes human upright movement, arm hanging down movement, and legs naturally straightening movement.

[0015] The average joint energy is calculated based on the joint kinetic energy components and the joint relative potential energy value.

[0016] In one embodiment, the step of reconstructing the joint energy mean using a rigid body representation to build the reconstructed skeleton structure includes:

[0017] The energy average of the joints is visualized to obtain energy visualization data.

[0018] High-energy retention processing is performed on the energy visualization data to obtain retained joint point data;

[0019] Based on the preserved joint data, a rigid body representation reconstruction is performed to obtain the reconstructed skeleton structure.

[0020] In one embodiment, the step of reconstructing the rigid body representation based on the retained joint data to obtain the reconstructed skeleton structure includes:

[0021] Based on the preserved joint point data, the target three-dimensional coordinate set is averaged to obtain the three-dimensional spatial position of the trunk joints. The target three-dimensional coordinate set includes the three-dimensional coordinates of the base of the spine, the middle of the spine, the upper end of the spine, the left hip joint, and the right hip joint.

[0022] In consecutive frames, the three-dimensional spatial position of the trunk joint is subtracted from the position of the limb joints to obtain the centered joint position;

[0023] Rigid body representation reconstruction is performed based on the centralized joint positions to obtain a reconstructed skeleton structure containing multiple key joints, including target active parts such as the upper limb, lower limb, and shoulder.

[0024] In one embodiment, the step of normalizing the kinetic energy matrix and potential energy matrix in the reconstructed skeleton structure to obtain a normalized matrix, and then performing a fast Fourier transform on the normalized matrix to obtain frequency domain features includes:

[0025] The joint kinetic energy matrix and joint potential energy matrix in the reconstructed skeleton structure are subjected to minimum normalization and maximum normalization operations column by column to obtain the kinetic energy normalization matrix and the potential energy normalization matrix.

[0026] Based on the kinetic energy normalization matrix and the potential energy normalization matrix, a fast Fourier transform is performed on each column according to the data segment length and energy to obtain the kinetic energy frequency characteristics and the potential energy frequency characteristics.

[0027] The frequency domain features are obtained by splicing the kinetic energy frequency features and the potential energy frequency features along the joint dimension.

[0028] In one embodiment, the step of performing feature selection and dimensionality reduction based on the frequency domain features to obtain emotion classification data includes:

[0029] Calculate the Pearson correlation coefficient between the frequency domain features and the preset emotion labels, and select a target number of features as candidate features based on the absolute value of the Pearson correlation coefficient.

[0030] Principal component analysis was performed on the candidate features to obtain dimensionality-reduced features;

[0031] The dimensionality reduction features are used as emotion feature vectors, and emotion classification data is obtained based on these emotion feature vectors.

[0032] In one embodiment, the steps of acquiring human gait video data, extracting skeletal three-dimensional coordinate data from the human gait video data, and performing Gaussian filtering on the skeletal three-dimensional coordinate data to obtain skeletal joint point data include:

[0033] Human gait video data is acquired, the three-dimensional positions of human joints in the human gait video data are identified, and initial three-dimensional coordinate data of the skeleton is obtained. The human gait video data is the video of the subject walking towards the camera.

[0034] The initial three-dimensional coordinate data of the skeleton are processed by Gaussian weighted average in the X-axis, Y-axis and Z-axis directions according to a sliding window of preset length to obtain the skeleton joint data.

[0035] Furthermore, to achieve the above objectives, this application also proposes an emotion classification device based on gait data, the emotion classification device based on gait data comprising:

[0036] The data acquisition module is used to acquire human gait video data, extract skeletal three-dimensional coordinate data based on the human gait video data, and perform Gaussian filtering on the skeletal three-dimensional coordinate data to obtain skeletal joint point data.

[0037] An energy calculation module is used to calculate the kinetic and potential energy of the skeletal joint data to obtain the average joint energy.

[0038] The rigid body reconstruction module is used to reconstruct the joint energy mean value using a rigid body representation and to construct a reconstructed skeleton structure.

[0039] The normalization processing module is used to normalize the kinetic energy matrix and potential energy matrix in the reconstructed skeleton structure to obtain a normalized matrix, and to perform a fast Fourier transform on the normalized matrix to obtain frequency domain features.

[0040] The emotion classification module is used to perform feature selection and dimensionality reduction based on the frequency domain features to obtain emotion classification data.

[0041] In addition, to achieve the above objectives, this application also proposes an emotion classification device based on gait data, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the emotion classification method based on gait data as described above.

[0042] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the emotion classification method based on gait data as described above.

[0043] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the emotion classification method based on gait data as described above.

[0044] One or more technical solutions proposed in this application have at least the following technical effects:

[0045] This technology employs a series of complex and innovative techniques, including extracting 3D skeletal coordinate data from human gait video data and performing Gaussian filtering; calculating the kinetic and potential energy of skeletal joints to obtain the mean joint energy; reconstructing the skeleton structure using rigid body representation of the mean joint energy; normalizing the kinetic and potential energy matrices in the skeleton structure and performing Fast Fourier Transform to obtain frequency domain features; and performing feature selection and dimensionality reduction based on the frequency domain features. This approach solves the key problem of accurately and effectively extracting features suitable for emotion classification from human gait data, a problem present in existing technologies. Compared to existing technologies, it achieves more accurate emotion classification and recognition, improving the accuracy and reliability of emotion recognition, and providing more effective technical support and methodological pathways for the application of emotion recognition technology in related fields. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating an embodiment of the emotion classification method based on gait data provided in this application.

[0049] Figure 2 This is a schematic diagram of a gait-neutral posture provided in Embodiment 1 of the emotion classification method based on gait data in this application;

[0050] Figure 3 This is an energy visualization diagram provided for Embodiment 1 of the emotion classification method based on gait data in this application;

[0051] Figure 4 This is a flowchart illustrating Embodiment 2 of the emotion classification method based on gait data provided in this application;

[0052] Figure 5 A simplified flowchart illustrating the emotion classification method based on gait data provided in Embodiment 2 of this application;

[0053] Figure 6 This is a schematic diagram of the module structure of the emotion classification device based on gait data according to an embodiment of this application;

[0054] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the emotion classification method based on gait data in the embodiments of this application.

[0055] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0056] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0057] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0058] The main solution of this application embodiment is as follows: acquire human gait video data, extract skeletal three-dimensional coordinate data from the human gait video data, and perform Gaussian filtering on the skeletal three-dimensional coordinate data to obtain skeletal joint data; calculate the kinetic and potential energy of the skeletal joint data to obtain the average joint energy; reconstruct the average joint energy using rigid body representation to construct a reconstructed skeleton structure; normalize the kinetic and potential energy matrices in the reconstructed skeleton structure to obtain a normalized matrix, and perform a fast Fourier transform on the normalized matrix to obtain frequency domain features; perform feature selection and dimensionality reduction processing based on the frequency domain features to obtain emotion classification data.

[0059] In this embodiment, for ease of description, the following description will focus on an emotion classification device based on gait data.

[0060] Because existing technologies cannot accurately extract and represent the energy change characteristics of different joints in gait, resulting in insufficient accuracy in acquiring emotion-related behavioral data, this application provides a solution. This solution employs a series of complex and innovative techniques, including extracting 3D skeletal coordinate data from human gait video data and performing Gaussian filtering; calculating the kinetic and potential energy of skeletal joint points to obtain the mean joint energy; reconstructing the skeleton structure using rigid body representation of the mean joint energy; normalizing the kinetic and potential energy matrices in the skeleton structure and performing Fast Fourier Transform to obtain frequency domain features; and performing feature selection and dimensionality reduction based on the frequency domain features. This addresses the key problem in existing technologies of accurately and effectively extracting features suitable for emotion classification from human gait. Compared to existing technologies, this solution achieves more accurate emotion classification and recognition, improving the accuracy and reliability of emotion recognition, and providing more effective technical support and methodological pathways for the application of emotion recognition technology in related fields.

[0061] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, such as an emotion classification device based on gait data. The following description uses an emotion classification device based on gait data as an example to illustrate this embodiment and the subsequent embodiments.

[0062] Based on this, embodiments of this application provide an emotion classification method based on gait data, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the emotion classification method based on gait data in this application.

[0063] In this embodiment, the emotion classification method based on gait data includes steps S10 to S50:

[0064] Step S10: Acquire human gait video data, extract skeletal three-dimensional coordinate data based on the human gait video data, and perform Gaussian filtering on the skeletal three-dimensional coordinate data to obtain skeletal joint data.

[0065] It should be noted that human gait video data refers to video data recorded by camera devices such as Kinect that captures the process of a human walking. It includes dynamic information such as limb movements and posture changes when a person is walking, such as recording continuous video frames of a person walking indoors.

[0066] Additionally, skeletal 3D coordinate data is the positional information of various joints in the human body in 3D space extracted from human gait video data.

[0067] Additionally, Gaussian filtering is a signal processing method used to smooth data and reduce noise. It involves using a filter constructed with a Gaussian function to perform convolution operations on the 3D coordinate data of the skeleton to reduce the impact of high-frequency noise (such as camera shake or joint recognition errors).

[0068] Furthermore, the skeletal joint data is smoothed 3D coordinate data of joints obtained after Gaussian filtering, which eliminates noise interference in the original data and more accurately reflects the real position of human joints during walking, such as the stable coordinate values ​​of the elbow joint in each frame after filtering.

[0069] Understandably, video data of the human walking process, i.e., human gait video data, is acquired through camera equipment. Then, the coordinate information of each joint point in three-dimensional space is extracted from this video data to obtain the skeletal three-dimensional coordinate data. Since the raw extracted skeletal three-dimensional coordinate data may contain noise, a Gaussian weighted average is used to reduce the noise effect. The smoothed joint point coordinate data obtained after filtering is the skeletal joint point data.

[0070] Step S20: Calculate the kinetic and potential energy of the bone joint data to obtain the average joint energy.

[0071] It should be noted that kinetic energy represents a physical quantity reflecting the intensity of joint movement, calculated based on the displacement velocity of adjacent frames in the skeletal joint data. The mass can be simplified to a constant, calculated solely by the ratio of the inter-frame displacement of the joint to the sampling time interval (velocity), meaning kinetic energy is proportional to the square of the velocity.

[0072] Additionally, potential energy is a physical quantity that reflects the amplitude of vertical fluctuation of a joint relative to the gait neutral posture. The gait neutral posture, or GNP, is defined as a frame image of a person standing upright with arms hanging down and legs naturally extended, and is calculated by the difference between the Y-axis coordinate of the joint and the Y-axis coordinate of the corresponding point in the GNP.

[0073] Furthermore, the joint energy mean is the value obtained by averaging the kinetic and potential energy of each joint across all frames in the entire gait segment. It represents the average kinetic and potential energy of the joint during walking and is used to reflect the overall energy level of the joint.

[0074] Understandably, based on skeletal joint data, the kinetic and potential energy of each joint is calculated separately. The kinetic and potential energy of each joint across all frames of the gait segment are summed, and then divided by the number of frames to obtain the average kinetic and potential energy of each joint, which is the average joint energy.

[0075] In one feasible implementation, step S20 may include steps S21 to S23:

[0076] Step S21: Based on the skeletal joint data, calculate the ratio of the Euclidean displacement of the skeleton points to the sampling time interval in consecutive frames to obtain the velocity value, and construct the joint kinetic energy component based on the velocity value.

[0077] It should be noted that consecutive frames refer to video frames arranged sequentially in time within the skeletal joint data, such as the i-th frame and the (i-1)-th frame, the (i+1)-th frame, etc., which are adjacent to each other and reflect the continuous movement process of human gait.

[0078] Additionally, the Euclidean displacement of a skeleton point refers to the geometric distance between the differences in the three-dimensional coordinates of the same skeleton point in two consecutive frames. It is calculated by squaring the coordinate differences on the X, Y, and Z axes respectively, summing the results, and then taking the square root. The Euclidean displacement of a skeleton point is calculated as follows:

[0079]

[0080] In the formula, This is a Euclidean displacement. Let be the coordinates of the skeleton point on the m-th axis in the i-th frame.

[0081] Understandably, kinetic energy is calculated using the following formula:

[0082]

[0083] In the formula, m represents mass and v represents velocity. Since the original data only provides spatial coordinates and cannot obtain mass information, this embodiment simplifies the mass to a constant when calculating the kinetic energy of the skeleton, focusing only on the change in velocity.

[0084] Additionally, the sampling time interval refers to the time difference between two adjacent frames of skeletal joint data, i.e., the time interval between two frames of video captured by the camera device; the velocity value is the displacement of the skeleton point per unit time, obtained by removing the Euclidean position of the skeleton point by the sampling time interval, reflecting the speed of the skeleton point's movement.

[0085] Specifically, speed The Euclidean distance between the j-th skeleton point in the i-th frame and the previous frame. Dividing by the sampling time interval Δf, the velocity value is calculated as follows:

[0086]

[0087] In the formula, Δf represents the Euclidean displacement, and Δf represents the sampling time interval.

[0088] Furthermore, the joint kinetic energy component is a component that reflects the energy of joint movement, calculated based on velocity values.

[0089] The simplified kinetic energy calculation formula can be obtained from the velocity calculation formula and the Euclidean displacement calculation formula, as follows:

[0090]

[0091] Understandable Let be the coordinates of the skeleton point on the m-th axis in the i-th frame, and Δf be the sampling time interval.

[0092] Step S22: Select gait neutral posture data as reference posture from the skeletal joint data, and calculate the relative potential energy value of the joint based on the reference posture and joint coordinates. The gait neutral posture data is frame image data that includes human upright movement, arm hanging down movement and legs naturally straightening movement.

[0093] It should be noted that gait neutral posture data refers to the coordinate data of frame images in the skeletal joint data that include the human body in an upright position (torso straight), with arms hanging naturally (arms close to the sides of the body), and legs naturally straight (legs together and knees not bent). In this embodiment, this posture is used as the reference benchmark for potential energy calculation.

[0094] Reference Figure 2 , Figure 2 This is a schematic diagram of a gait-neutral posture from the first embodiment of the emotion classification method based on gait data in this application.

[0095] like Figure 2 As shown in the figure, red and cyan represent two different sets of skeletal data. The red skeletal data represents the skeletal information of a human movement; these skeletal points are connected by red lines to form a three-dimensional human model. The cyan skeletal data, also connected by lines, shows the skeletal information of another set of human movements. The coordinate axes in the figure are labeled x, y, and z, representing three directions in three-dimensional space. The red and cyan skeletal data differ in position and shape in three-dimensional space, possibly representing different movements or postures. Each skeletal point in the figure is located at a specific three-dimensional coordinate position; by connecting these points, the changes in human posture in space can be observed. The GNP (General Neutral Pose) in the figure is defined as a frame image of a human standing upright with arms hanging down and legs naturally extended, as shown in the red image in the figure. For each data segment, a frame conforming to the GNP standard is selected as a reference, which helps to provide a standard posture benchmark when analyzing and comparing different movements.

[0096] Additionally, the reference pose, i.e., the human pose corresponding to the gait neutral pose data, serves as the zero point for calculating potential energy and is used to measure the relative positional changes of joints in other frames.

[0097] Furthermore, joint coordinates refer to the X, Y, and Z axis coordinates of each joint in the skeletal joint data in three-dimensional space, such as the specific coordinate values ​​of the hip joint in a certain frame. Potential energy is calculated as follows:

[0098]

[0099] In the formula, F i Let GNP be the Y-axis coordinate of the j-th joint in the i-th frame, and GNP be the coordinates of the corresponding point in the neutral posture. Potential energy characteristics reflect the degree of vertical fluctuation in gait and are closely related to factors such as stride length and postural adjustment.

[0100] Step S23: Calculate the average joint energy based on the joint kinetic energy components and the joint relative potential energy value.

[0101] It should be noted that the average joint energy value refers to the average value obtained by summing the joint kinetic energy components and the joint relative potential energy values ​​of the same joint in all frames of the entire gait segment, and then dividing by the total number of frames. It reflects the average energy level of the joint during the gait process.

[0102] Understandably, the mean joint energy is calculated using the following formula:

[0103]

[0104] In the formula, E k,i and E p,i These are the joint kinetic energy components of joint i, i.e., the total kinetic energy and the relative potential energy of the joint, i.e., the total potential energy.

[0105] Step S30: Reconstruct the joint energy mean value using rigid body representation to build a reconstructed skeleton structure;

[0106] It should be noted that rigid body representation reconstruction is based on the average joint energy, screening high-energy key joints, merging low-energy redundant joints, and treating multiple joints in areas such as the torso as a whole to reconstruct a simplified skeleton.

[0107] Furthermore, the reconstructed skeleton structure is a simplified skeleton structure composed of key joints obtained after reconstruction using rigid body representation.

[0108] Understandably, based on the average joint energy, the energy distribution of each joint is analyzed; higher energy indicates a greater contribution of that joint to gait characteristics. Ultimately, 12 key joints are retained, and their positions relative to the trunk joints are recalculated to construct a streamlined reconstructed skeletal structure.

[0109] In one feasible implementation, step S30 may include steps S31 to S33:

[0110] Step S31: Visualize the average joint energy to obtain energy visualization data;

[0111] It should be noted that energy visualization data refers to data that presents the average joint energy in the form of an image, using color depth to intuitively reflect the energy level of different joints. In this embodiment, the darker the color of the bones in various parts of the human body, the greater the range of motion (energy) of the joints in that part. For example, joints with higher energy, such as the head and torso, are darker in the image, while joints with lower energy, such as the fingers and thumbs, are lighter in the image.

[0112] Reference Figure 3 , Figure 3 This is an energy visualization diagram of the first embodiment of the emotion classification method based on gait data in this application.

[0113] like Figure 3 As shown in the figure, there are 24 marked joints, each identified by a number and a name. The joints are numbered from 0 to 23 and named as follows: Pelvis (SpineBase), SpineMid, Neck, Head, Left ShoulderLeft, Left ElbowLeft, Left WristLeft, Left HandLeft, Right ShoulderRight, Right ElbowRight, Right WristRight, Right HandRight, Left HipLeft, Left KneeLeft, Left AnkleLeft, Left FootLeft, Right HipRight, Right KneeRight, Right AnkleRight, Right FootRight, SpineShoulder, Left HandTipLeft, Left ThumbLeft, Right HandTipRight, and Right ThumbRight. These joints are connected by lines to form a complete human skeletal structure. It also displays the range of motion of the joints, represented by a color gradient from light blue (minimum range of motion) to dark blue (max range of motion). This representation helps in understanding the mobility and range of each joint.

[0114] Step S32: Perform high-energy retention processing on the energy visualization data to obtain retained joint point data;

[0115] It should be noted that high-energy retention processing refers to the process of retaining key joints with higher energy and merging or removing redundant joints with lower energy based on the color intensity (energy level) of the joints in the energy visualization data. In this embodiment, multiple joints in the torso region are merged, and low-energy joints in the hand and foot areas are compressed (e.g., the average energy of the left wrist, left hand, left fingertip, and left thumb is taken) to retain the core joints.

[0116] Furthermore, the retained joint data consists of the three-dimensional coordinate data of high-energy key joints selected after high-energy retention processing. These joints contribute more to gait characteristics, such as trunk joints, shoulders, elbows, hips, knees, and ankles.

[0117] Understandably, based on energy visualization data, joints with darker colors (higher energy) and lighter colors (lower energy) are identified. Redundant joints with lower energy are then merged: for example, the coordinates of the left wrist with the left hand, left fingertip, and left thumb are averaged, and the coordinates of the left ankle with the left foot are averaged; the same applies to the right hand and right foot. At the same time, multiple high-energy joints in the torso region are treated as a whole, and their merged coordinate data is retained.

[0118] Step S33: Reconstruct the rigid body representation based on the retained joint data to obtain the reconstructed skeleton structure.

[0119] Understandably, based on the retained joint data, the joints in the trunk region are merged. For example, the retained joint data contains 17 joints. By merging 5 trunk joints, trunk joints are obtained. Then, the positions of the upper and lower limb joints are adjusted based on the trunk joints, and finally a reconstructed skeleton composed of 12 key joints is formed. The position of each joint is determined relative to the trunk joints.

[0120] In one feasible implementation, step S33 may include steps S331 to S333:

[0121] Step S331: Based on the retained joint point data, the target three-dimensional coordinate set is averaged to obtain the three-dimensional spatial position of the trunk joints. The target three-dimensional coordinate set includes the three-dimensional coordinates of the bottom of the spine, the middle of the spine, the upper end of the spine, the left hip joint, and the right hip joint.

[0122] It should be noted that the target 3D coordinate set refers to the set of 3D coordinates of five key parts related to the torso from the retained joint data, specifically including the coordinates of the base of the spine, the middle of the spine, the upper part of the spine, the left hip joint, and the right hip joint in 3D space (X, Y, Z axes). These parts together constitute the core structure of the human torso, and their positions can reflect the overall posture of the torso.

[0123] Additionally, the averaging process involves calculating the arithmetic mean of the three-dimensional coordinates of the five parts within the target's three-dimensional coordinate set. Specifically, this involves summing the X-axis coordinates, summing the Y-axis coordinates, and summing the Z-axis coordinates of the five parts and taking the average, resulting in a composite three-dimensional coordinate system.

[0124] Furthermore, the three-dimensional spatial position of the trunk joints is a three-dimensional coordinate value obtained by averaging the target three-dimensional coordinate set. It represents the position of the center of mass of the human trunk and is a reference point for measuring the overall movement of the trunk.

[0125] Understandably, from the preserved joint point data, the three-dimensional coordinates of five locations—the base of the spine, the middle of the spine, the upper part of the spine, the left hip joint, and the right hip joint—are extracted to form a target three-dimensional coordinate set. The X-axis, Y-axis, and Z-axis coordinates of each of these five locations are summed, and then the sum of each axis is divided by 5 (i.e., the number of locations) to obtain the average coordinate value of the three axes. These three average coordinate values ​​together constitute the position of the trunk joints in three-dimensional space, i.e., the three-dimensional spatial position of the trunk joints.

[0126] Step S332: In consecutive frames, subtract the three-dimensional spatial position of the trunk joint from the position of the limb joints to obtain the centered joint position.

[0127] It should be noted that the limb joint positions refer to the coordinate values ​​in three-dimensional space of joints belonging to the upper limbs (such as shoulders, elbows, and wrists) and lower limbs (such as hips, knees, and ankles) in the retained joint data, such as the positions of the left elbow and right knee in each frame.

[0128] Furthermore, the centralized processing of joint positions represents the relative coordinate values ​​obtained by subtracting the three-dimensional spatial positions of the trunk joints from the positions of the limb joints, reflecting the positional relationship of the limb joints relative to the trunk.

[0129] Understandably, for the three-dimensional spatial position of the trunk joints, the position coordinates of the limb joints (such as the shoulder, elbow, wrist, knee, and ankle) are extracted from each consecutive frame of skeletal data. The centered joint position of the limb joint in the current frame is obtained by subtracting the X-axis coordinate of the trunk joint from the X-axis coordinate, the Y-axis coordinate of the trunk joint from the Y-axis coordinate, and the Z-axis coordinate of the trunk joint from the Z-axis coordinate.

[0130] Step S333: Reconstruct rigid body representation based on the centralized processing joint position to obtain a reconstructed skeleton structure containing multiple key joints, including target active parts of the upper limb, lower limb, and shoulder.

[0131] Understandably, based on the centralized processing of joint positions, joints in target active areas such as the upper limbs (e.g., elbows, wrists), lower limbs (e.g., knees, ankles), and shoulders (e.g., left and right shoulders) are selected. These joints are then used as key nodes, and a simplified reconstructed skeleton structure is built according to their relative positions in three-dimensional space, ensuring that each key node reflects the motion characteristics of the corresponding part (e.g., the degree of flexion of the elbow, the range of flexion and extension of the knee).

[0132] Step S40: Normalize the kinetic energy matrix and potential energy matrix in the reconstructed skeleton structure to obtain a normalized matrix, and perform a fast Fourier transform on the normalized matrix to obtain frequency domain features.

[0133] It should be noted that the kinetic energy matrix represents a matrix composed of the kinetic energy values ​​of the 12 key joints in the reconstructed skeleton structure across all frames of the gait segment. It has dimensions of 12 x 64, where 12 represents the number of joints and 64 represents the number of frames. Each row corresponds to one joint, and each column corresponds to the kinetic energy value of one frame. The potential energy matrix is ​​similar to the kinetic energy matrix, also a 12 x 64 matrix composed of the potential energy values ​​of the 12 key joints across all frames of the gait segment. Each row corresponds to one joint, and each column corresponds to the potential energy value of one frame.

[0134] In addition, normalization is a method that maps the values ​​of the kinetic energy matrix and potential energy matrix to a unified range. It uses min-max normalization to map the values ​​to between -1 and 1, eliminating the magnitude difference in energy values ​​at different joints.

[0135] Furthermore, the normalized matrix is ​​the kinetic energy matrix and potential energy matrix after normalization. All values ​​in the matrix are within a uniform range. For example, each element in the kinetic energy normalized matrix takes a value from -1 to 1.

[0136] In addition, the Fast Fourier Transform (FFT) is a mathematical transformation method that converts time-domain signals into frequency-domain signals, revealing the periodic variation patterns of signals, such as the periodic movement of gait.

[0137] Furthermore, the frequency domain features are characteristics reflecting the change of energy with frequency extracted from the normalized matrix through the fast Fourier transform. After transforming each column of the normalized matrix, 32 amplitude coefficients are obtained, each coefficient corresponding to the energy intensity of a frequency component.

[0138] Understandably, for the reconstructed skeleton structure, namely the 12 key joints, the kinetic and potential energy of each joint in all frames of the gait segment are recalculated, forming a 12x64 kinetic energy matrix and a 12x64 potential energy matrix, where 64 represents the number of frames in the gait segment. These two matrices are then subjected to min-max normalization, mapping the values ​​in each matrix proportionally to the range of -1 to 1, resulting in a normalized matrix. This eliminates the magnitude differences in energy values ​​between different joints. A Fast Fourier Transform is applied to each column of the normalized matrix, corresponding to the energy values ​​of all joints in each frame, to calculate 32 amplitude coefficients. These coefficients reflect the frequency domain characteristics of energy variation with frequency.

[0139] In one feasible implementation, step S40 may include steps S41 to S43:

[0140] Step S41: Perform minimum normalization and maximum normalization operations on the joint kinetic energy matrix and joint potential energy matrix in the reconstructed skeleton structure column by column to obtain the kinetic energy normalization matrix and the potential energy normalization matrix.

[0141] It should be noted that minimum normalization and maximum normalization are processing methods that map data to a specific range. By calculating the difference between each element in the matrix and the minimum value of that column, and then dividing by the difference between the maximum value and the minimum value of that column, the values ​​are uniformly mapped to the range of 0 to 1. Then, they are further adjusted to the range of -1 to 1, eliminating the difference in magnitude between different columns of data.

[0142] Furthermore, the kinetic energy normalization matrix is ​​obtained by performing minimum normalization and maximum normalization operations on the joint kinetic energy matrix. All elements in the matrix are within a uniform range, such as between -1 and 1, which facilitates the comparison of kinetic energy characteristics of different joints.

[0143] Furthermore, the potential energy normalization matrix is ​​obtained by performing the same normalization operation on the joint potential energy matrix, and the elements are also within the same range, ensuring that the potential energy characteristics and kinetic energy characteristics are analyzed on the same scale.

[0144] Understandably, for the reconstructed skeleton structure, the kinetic energy matrix and potential energy matrix of 12 key joints are obtained. The two matrices are processed column-by-column. First, the minimum and maximum values ​​in each column are found. The difference between each element in that column and the minimum value is calculated. Then, this difference is divided by the difference between the maximum and minimum values ​​to obtain values ​​mapped to the range of 0 to 1. After adjustment, the value range is converted to -1 to 1, completing the minimum and maximum normalization operations. The resulting kinetic energy matrix is ​​the kinetic energy normalized matrix, and the potential energy matrix is ​​the potential energy normalized matrix.

[0145] Step S42: Based on the kinetic energy normalization matrix and the potential energy normalization matrix, perform a fast Fourier transform on each column according to the data segment length and energy to obtain the kinetic energy frequency characteristics and the potential energy frequency characteristics.

[0146] It should be noted that the data segment length refers to the number of data points contained in each column of the kinetic energy normalization matrix and the potential energy normalization matrix, corresponding to the number of frames of the gait segment.

[0147] Furthermore, the kinetic energy frequency feature is a feature obtained by performing a fast Fourier transform on each column of the kinetic energy normalization matrix. It reflects the law of kinetic energy change with frequency. Each feature corresponds to the energy intensity of a frequency component, such as the fluctuation amplitude of kinetic energy corresponding to a certain frequency.

[0148] In addition, the potential energy frequency characteristic is a feature obtained by performing the same transformation on each column of the potential energy normalization matrix. It reflects the law of potential energy change with frequency, such as the change of potential energy at different frequencies.

[0149] Understandably, the Fast Fourier Transform (FFT) is calculated using the following formula:

[0150]

[0151] In the formula, JE k Defined as the extracted feature, i.e., kinetic energy frequency feature or potential energy frequency feature, where N is the length of the data segment, and E... m Let be the energy of the m-th frame.

[0152] Step S43: The frequency domain features are obtained by splicing the kinetic energy frequency features and the potential energy frequency features according to the joint dimension.

[0153] It should be noted that the joint dimension refers to the classification dimension of the 12 key joints in the reconstructed skeleton structure. Each joint corresponds to a set of kinetic energy frequency features and potential energy frequency features.

[0154] Understandably, for each of the 12 key joints in the reconstructed skeleton structure, the corresponding kinetic energy frequency features (e.g., 32 features) and potential energy frequency features (e.g., 32 features) are extracted. The kinetic energy frequency features and potential energy frequency features of the same joint are then merged sequentially to form the comprehensive frequency features of that joint (e.g., 64 features). The comprehensive frequency features of all 12 joints are then summarized to obtain the frequency domain features containing information about all joint dimensions.

[0155] Step S50: Perform feature selection and dimensionality reduction based on the frequency domain features to obtain emotion classification data.

[0156] It should be noted that feature selection is the process of filtering out the features most relevant to the identification of depressive mood from frequency domain features. In this embodiment, Pearson correlation analysis can be used to select the 20 features with the largest absolute values ​​of correlation coefficients in each column. The larger the absolute value of the correlation coefficient, the stronger the correlation between the feature and the depressive state.

[0157] Another method is dimensionality reduction, which uses principal component analysis to reduce the dimensionality of features. This reduces data complexity while preserving key feature information, such as converting a high-dimensional feature matrix into a low-dimensional principal component matrix.

[0158] Furthermore, the emotion classification data, after feature selection and dimensionality reduction, is input into the classification model to obtain the result data used to distinguish the individual's risk of depression, that is, the identification result of high or low risk of depression.

[0159] Understandably, feature selection is based on frequency domain characteristics. Pearson correlation analysis is used to calculate the correlation coefficient between each frequency domain feature and the depressive state. The 20 features with the largest absolute values ​​of correlation coefficients in each column are selected to form a 12x20 dimensional feature matrix, thus reducing interference from irrelevant features. This feature matrix is ​​then subjected to dimensionality reduction. Principal component analysis is used to extract principal components, reducing the dimensionality while retaining most of the feature information; for example, reducing the 12x20 dimensional matrix to a lower-dimensional principal component matrix. The dimensionality-reduced features are then input into a classification model, such as a support vector machine or random forest. Through model training and prediction, results are obtained that distinguish individual depression risk, i.e., emotion classification data.

[0160] In one feasible implementation, step S50 may include steps S51 to S53:

[0161] Step S51: Calculate the Pearson correlation coefficient between the frequency domain features and the preset emotion labels, and select a target number of features as candidate features based on the absolute value of the Pearson correlation coefficient.

[0162] It should be noted that the preset emotion label refers to the emotion category label that is pre-labeled for gait data, which is used to indicate whether the individual corresponding to the gait has a risk of depression, such as "high risk of depression" or "low risk of depression", and is usually determined based on depression scale scores or professional assessment results.

[0163] Additionally, the Pearson correlation coefficient is a statistic that measures the degree of linear correlation between two variables, ranging from -1 to 1. The closer the absolute value of the coefficient is to 1, the stronger the correlation between the two variables; the closer it is to 0, the weaker the correlation. For example, a correlation coefficient of 0.8 between a certain frequency domain feature and the label "high risk of depression" indicates that this feature is highly correlated with the risk of depression.

[0164] Furthermore, the target number is the number of most representative features selected from the frequency domain features, specifically the 20 features with the largest absolute value of the correlation coefficient in each column, which have the strongest correlation with the sentiment label.

[0165] Furthermore, candidate features are frequency domain features selected after Pearson correlation coefficient analysis, which are the core features for subsequent emotion classification. For example, 20 features highly correlated with depression risk were selected from all frequency domain features.

[0166] Understandably, based on the frequency domain features and pre-labeled preset emotion tags (such as "high risk of depression" and "low risk of depression"), the Pearson correlation coefficient between each frequency domain feature and the emotion tag is calculated. The absolute values ​​of the correlation coefficients of all frequency domain features are taken, and they are sorted from largest to smallest. The top 20 features (the target number) are selected as candidate features. These features have the strongest linear correlation with the emotion tags and best reflect the differences in depressive gait.

[0167] Step S52: Perform principal component analysis on the candidate features to obtain dimensionality-reduced features;

[0168] It should be noted that principal component analysis is a statistical method used to reduce the dimensionality of data. It transforms multiple related candidate features into a few unrelated composite features (principal components) through linear transformation. These principal components can retain most of the information of the original features. For example, transforming 20 candidate features into 5 principal components can still reflect the main patterns of the original features.

[0169] Furthermore, dimensionality reduction features are principal component features obtained after principal component analysis. They have a lower dimension than candidate features. For example, a 12×20 candidate feature matrix can be reduced to a 12×5 principal component matrix, which simplifies the data structure but retains the core information.

[0170] Understandably, for candidate features (such as a 12×20 feature matrix), principal component analysis is applied to convert high-dimensional candidate features into low-dimensional principal components by calculating the covariance matrix of the features, extracting eigenvalues ​​and eigenvectors. The top principal components with high cumulative contribution rates (e.g., exceeding 85%) are selected as dimensionality-reduced features. These principal components can comprehensively reflect the main information of the original candidate features while eliminating correlations between features.

[0171] Step S53: Use the dimensionality reduction features as the emotion feature vector, and obtain emotion classification data based on the emotion feature vector.

[0172] It should be noted that the emotion feature vector refers to the vector form composed of dimensionality-reduced features. Each vector corresponds to the core emotion feature of a gait segment. For example, a vector composed of 5 principal components comprehensively reflects the depression-related gait features of that segment.

[0173] Understandably, the dimensionality-reduced features are converted into emotion feature vectors (e.g., one feature vector for each gait segment). These emotion feature vectors are then input into a pre-trained classification model (Support Vector Machine, Random Forest, etc.). The model learns the correspondence between the feature vectors and preset emotion labels, performs classification prediction on the input feature vectors, and outputs the emotion classification data corresponding to that gait segment.

[0174] This embodiment provides an emotion classification method based on gait data. By acquiring human gait video data, extracting 3D skeletal coordinate data, and performing Gaussian filtering on the data to obtain smoothed joint point data, the kinetic and potential energy of the joints are calculated to obtain the average joint energy. Then, the simplified skeleton structure is reconstructed using rigid body representation. The kinetic and potential energy matrices are normalized and subjected to Fast Fourier Transform to extract frequency domain features. Finally, emotion classification data is obtained through feature selection and dimensionality reduction. This method solves the technical problem of accurately extracting and utilizing the emotional information contained in gait in traditional gait analysis, and achieves the beneficial effect of improving the accuracy and efficiency of emotion recognition.

[0175] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 Step S10 of the emotion classification method based on gait data includes steps S11 to S12:

[0176] Step S11: Acquire human gait video data, identify the three-dimensional position of human joints in the human gait video data, and obtain initial three-dimensional coordinate data of bones. The human gait video data is the video of the subject walking towards the camera.

[0177] It should be noted that human gait video data refers to video data recorded by camera equipment that shows the subject walking, and only includes the video content of the phase when the subject is "walking towards the camera", excluding the part away from the camera, to ensure the consistency of gait direction.

[0178] Furthermore, the three-dimensional position of human joints refers to the specific position of each human joint (such as head, shoulder, hip, knee, ankle, etc.) in three-dimensional space (X, Y, Z axes) identified from human gait video data, reflecting the spatial coordinate information of each joint in the video frame.

[0179] Furthermore, the initial skeletal 3D coordinate data is the unprocessed 3D position data of each joint of the human body extracted directly from human gait video data, including the X, Y, and Z axis coordinate values ​​of each joint in different video frames.

[0180] Understandably, video footage of the subject walking is captured using a camera, with only the portion of the video showing the subject walking towards the camera retained, serving as the human gait video data. Skeletal recognition technology (such as Kinect's skeleton tracking function) is then used to identify key joints (such as those in the head, spine, shoulders, and limbs) from this video data, and the position of each joint in three-dimensional space (X, Y, Z axes) is determined to obtain initial skeletal three-dimensional coordinate data.

[0181] Step S12: According to the preset length of the sliding window, the initial three-dimensional coordinate data of the skeleton is processed by Gaussian weighted average in the X-axis, Y-axis and Z-axis directions to obtain the skeleton joint data.

[0182] It should be noted that the preset length sliding window refers to a fixed-length window used for data smoothing. In this embodiment, it can be a window with a length of 5. This window will slide along the time axis (video frame order) to cover 5 consecutive frames of data, such as consecutive frame combinations like frames 1-5, frames 2-6, etc.

[0183] Another method is Gaussian weighted average processing, which is a smoothing method based on Gaussian function. It assigns different weights to the initial 3D coordinate data of the skeleton within the sliding window (the center frame of the window has the highest weight, which gradually decreases towards both ends), and then calculates the weighted average value to reduce noise interference. For example, the average value of the X-axis coordinates of 5 frames within the window is calculated according to Gaussian weights.

[0184] It is understandable that the X-axis, Y-axis, and Z-axis directions refer to the three coordinate axes in three-dimensional space.

[0185] Furthermore, the skeletal joint data: smoothed joint 3D coordinate data obtained after Gaussian weighted averaging eliminates noise in the initial data (such as device recognition errors and minor limb tremors), and more accurately reflects the true movement trajectory of the joint.

[0186] Understandably, based on the initial 3D skeletal coordinate data, a sliding window of preset length (length 5) is used to process the coordinate data in the X, Y, and Z axes respectively. For each axis's coordinate data, the sliding window moves frame by frame along the time axis, assigning weights to the coordinate values ​​of the 5 frames within the window using a Gaussian function (with the center frame having the largest weight), and calculating the weighted average as the smoothed coordinate of the window's center frame. This process is repeated until all frames' three-axis coordinates have been processed, and the final smoothed coordinate data is the skeletal joint data.

[0187] This embodiment provides an emotion classification method based on gait data. By acquiring gait video data of a subject walking towards a camera, the method identifies and extracts the three-dimensional position information of human joints to obtain initial three-dimensional skeletal coordinate data. Then, it uses a Gaussian weighted average processing technique with a sliding window of preset length in the X, Y, and Z axes. This solves the problem of inaccurate joint position data caused by device recognition errors and slight limb tremors in traditional gait analysis, and achieves the beneficial effect of improving the accuracy of joint position data and more realistically reflecting the trajectory of human joint movement.

[0188] For example, to help understand the implementation process of the emotion classification method based on gait data obtained in this embodiment combined with the above embodiment one, please refer to... Figure 5 , Figure 5 A simplified flowchart of a gait data-based emotion classification method is provided, specifically:

[0189] Preprocessing involves resampling and denoising the input data; rigid body representation utilizes joint energy and clinical gait features to construct a rigid body model of the human body; feature extraction first encodes the potential energy and kinetic energy, then obtains frequency features through fast Fourier transform; feature selection and classification first selects features, then uses the processed features to perform classification, completing the conversion from gait data to classification results.

[0190] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the emotion classification method based on gait data in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0191] This application also provides an emotion classification device based on gait data; please refer to [reference needed]. Figure 6 The emotion classification device based on gait data includes:

[0192] The data acquisition module 10 is used to acquire human gait video data, extract three-dimensional coordinate data of bones based on the human gait video data, and perform Gaussian filtering on the three-dimensional coordinate data of bones to obtain bone joint data.

[0193] The energy calculation module 20 is used to calculate the kinetic and potential energy of the bone joint data to obtain the average joint energy.

[0194] Rigid body reconstruction module 30 is used to reconstruct the joint energy mean value using rigid body representation and construct a reconstructed skeleton structure.

[0195] The normalization processing module 40 is used to normalize the kinetic energy matrix and potential energy matrix in the reconstructed skeleton structure to obtain a normalized matrix, and to perform a fast Fourier transform on the normalized matrix to obtain frequency domain features.

[0196] The emotion classification module 50 is used to perform feature selection and dimensionality reduction based on the frequency domain features to obtain emotion classification data.

[0197] The emotion classification device based on gait data provided in this application employs the emotion classification method based on gait data in the above embodiments, which can solve the technical problem of insufficient accuracy in obtaining emotion-related behavioral data due to the inability to accurately extract and express the energy change characteristics of different joints in gait. Compared with the prior art, the beneficial effects of the emotion classification device based on gait data provided in this application are the same as those of the emotion classification method based on gait data provided in the above embodiments, and other technical features in the emotion classification device based on gait data are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0198] In one embodiment, the energy calculation module 20 is further configured to: calculate the ratio of the Euclidean displacement of the skeleton points to the sampling time interval in consecutive frames based on the skeletal joint data to obtain a velocity value; construct a joint kinetic energy component based on the velocity value; select gait neutral posture data as a reference posture from the skeletal joint data; and calculate a joint relative potential energy value based on the reference posture and joint coordinates, wherein the gait neutral posture data is frame image data including human upright movement, arm hanging down movement, and natural leg straightening movement; and calculate the joint energy mean based on the joint kinetic energy component and the joint relative potential energy value.

[0199] In one embodiment, the rigid body reconstruction module 30 is further configured to visualize the average joint energy to obtain energy visualization data; perform high-energy retention processing on the energy visualization data to obtain retained joint data; and perform rigid body representation reconstruction on the retained joint data to obtain a reconstructed skeleton structure.

[0200] In one embodiment, the rigid body reconstruction module 30 is further configured to average the target three-dimensional coordinate set according to the retained joint point data to obtain the three-dimensional spatial position of the trunk joints. The target three-dimensional coordinate set includes the three-dimensional coordinates of the base of the spine, the middle of the spine, the upper end of the spine, the left hip joint, and the right hip joint. In consecutive frames, the three-dimensional spatial position of the trunk joints is subtracted from the position of the limb joints to obtain the centered joint position. Rigid body representation reconstruction is performed based on the centered joint position to obtain a reconstructed skeleton structure containing multiple key joint points. The multiple key joint points include the target active parts of the upper limb, lower limb, and shoulder.

[0201] In one embodiment, the normalization processing module 40 is further configured to perform minimum normalization and maximum normalization operations on the joint kinetic energy matrix and joint potential energy matrix in the reconstructed skeleton structure column by column to obtain kinetic energy normalized matrix and potential energy normalized matrix; based on the kinetic energy normalized matrix and the potential energy normalized matrix, perform fast Fourier transform on each column according to the data segment length and energy to obtain kinetic energy frequency features and potential energy frequency features; and concatenate the kinetic energy frequency features and the potential energy frequency features according to the joint dimension to obtain frequency domain features.

[0202] In one embodiment, the emotion classification module 50 is further configured to calculate the Pearson correlation coefficient between the frequency domain features and the preset emotion labels, and select a target number of features as candidate features based on the absolute value of the Pearson correlation coefficient; perform principal component analysis on the candidate features to obtain dimensionality-reduced features; use the dimensionality-reduced features as emotion feature vectors, and obtain emotion classification data based on the emotion feature vectors.

[0203] In one embodiment, the data acquisition module 10 is further configured to acquire human gait video data, identify the three-dimensional positions of human joints in the human gait video data, and obtain initial skeletal three-dimensional coordinate data, wherein the human gait video data is a video of the subject walking towards the camera; and perform Gaussian weighted average processing on the initial skeletal three-dimensional coordinate data in the X-axis, Y-axis and Z-axis directions respectively according to a sliding window of preset length to obtain skeletal joint point data.

[0204] This application provides an emotion classification device based on gait data. The emotion classification device based on gait data includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the emotion classification method based on gait data in the above embodiment 1.

[0205] The following is for reference. Figure 7 The diagram illustrates a structural schematic suitable for implementing the emotion classification device based on gait data in the embodiments of this application. The emotion classification device based on gait data in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7The emotion classification device based on gait data shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0206] like Figure 7 As shown, the emotion classification device based on gait data may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the emotion classification device based on gait data. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the gait data-based emotion classification device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a gait data-based emotion classification device with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems can be implemented alternatively.

[0207] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0208] The emotion classification device based on gait data provided in this application, employing the emotion classification method based on gait data in the above embodiments, can solve the technical problem of insufficient accuracy in acquiring emotion-related behavioral data due to the inability to accurately extract and express the energy change characteristics of different joints in gait. Compared with the prior art, the beneficial effects of the emotion classification device based on gait data provided in this application are the same as those of the emotion classification method based on gait data provided in the above embodiments, and other technical features in this emotion classification device based on gait data are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0209] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0210] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0211] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the emotion classification method based on gait data in the above embodiments.

[0212] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, RAM (Random Access Memory), ROM (Read Only Memory), Erasable Programmable Read Only Memory (EPROM), optical fiber, CD-ROM (CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0213] The aforementioned computer-readable storage medium may be included in a gait-based emotion classification device; or it may exist independently and not incorporated into a gait-based emotion classification device.

[0214] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a gait data-based emotion classification device, cause the gait data-based emotion classification device to: acquire human gait video data; extract skeletal three-dimensional coordinate data from the human gait video data; perform Gaussian filtering on the skeletal three-dimensional coordinate data to obtain skeletal joint data; calculate the kinetic and potential energy of the skeletal joint data to obtain the average joint energy; reconstruct the average joint energy using rigid body representation to construct a reconstructed skeleton structure; normalize the kinetic and potential energy matrices in the reconstructed skeleton structure to obtain a normalized matrix; perform a fast Fourier transform on the normalized matrix to obtain frequency domain features; and perform feature selection and dimensionality reduction based on the frequency domain features to obtain emotion classification data.

[0215] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LAN (Local Area Network) or WAN (Wide Area Network)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0216] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0217] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0218] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described emotion classification method based on gait data. This solves the technical problem of insufficient accuracy in acquiring emotion-related behavioral data due to the inability to accurately extract and express the energy change characteristics of different joints in gait. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the emotion classification method based on gait data provided in the above embodiments, and will not be repeated here.

[0219] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the emotion classification method based on gait data as described above.

[0220] The computer program product provided in this application can solve the technical problem of insufficient accuracy in acquiring emotion-related behavioral data due to the inability to accurately extract and express the energy change characteristics of different joints in gait. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the emotion classification method based on gait data provided in the above embodiments, and will not be repeated here.

[0221] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A sentiment classification method based on gait data, characterized in that, The method includes: Acquire human gait video data, extract skeletal three-dimensional coordinate data based on the human gait video data, and perform Gaussian filtering on the skeletal three-dimensional coordinate data to obtain skeletal joint data; The kinetic and potential energy of the skeletal joint data are calculated to obtain the average joint energy. The average joint energy is reconstructed using rigid body representation to build a reconstructed skeleton structure; The kinetic energy matrix and potential energy matrix in the reconstructed skeleton structure are normalized to obtain a normalized matrix, and the normalized matrix is ​​subjected to a fast Fourier transform to obtain frequency domain features. Based on the frequency domain features, feature selection and dimensionality reduction are performed to obtain emotion classification data.

2. The method as described in claim 1, characterized in that, The step of calculating the kinetic and potential energy of the skeletal joint data to obtain the average joint energy includes: Based on the skeletal joint data, the ratio of the Euclidean displacement of the skeleton points to the sampling time interval is calculated in consecutive frames to obtain the velocity value, and the joint kinetic energy component is constructed based on the velocity value. In the skeletal joint data, gait neutral posture data is selected as the reference posture, and the relative potential energy value of the joint is calculated based on the reference posture and joint coordinates. The gait neutral posture data is frame image data that includes human upright movement, arm hanging down movement, and legs naturally straightening movement. The average joint energy is calculated based on the joint kinetic energy components and the joint relative potential energy value.

3. The method as described in claim 1, characterized in that, The step of reconstructing the skeleton structure by performing rigid body representation reconstruction on the mean joint energy values ​​includes: The energy average of the joints is visualized to obtain energy visualization data. High-energy retention processing is performed on the energy visualization data to obtain retained joint point data; Based on the preserved joint data, a rigid body representation reconstruction is performed to obtain the reconstructed skeleton structure.

4. The method as described in claim 3, characterized in that, The step of reconstructing the rigid body representation based on the retained joint data to obtain the reconstructed skeleton structure includes: Based on the preserved joint point data, the target three-dimensional coordinate set is averaged to obtain the three-dimensional spatial position of the trunk joints. The target three-dimensional coordinate set includes the three-dimensional coordinates of the base of the spine, the middle of the spine, the upper end of the spine, the left hip joint, and the right hip joint. In consecutive frames, the three-dimensional spatial position of the trunk joint is subtracted from the position of the limb joints to obtain the centered joint position; Rigid body representation reconstruction is performed based on the centralized joint positions to obtain a reconstructed skeleton structure containing multiple key joints, including target active parts such as the upper limb, lower limb, and shoulder.

5. The method as described in claim 1, characterized in that, The steps of normalizing the kinetic energy matrix and potential energy matrix in the reconstructed skeleton structure to obtain a normalized matrix, and then performing a fast Fourier transform on the normalized matrix to obtain frequency domain features include: The joint kinetic energy matrix and joint potential energy matrix in the reconstructed skeleton structure are subjected to minimum normalization and maximum normalization operations column by column to obtain the kinetic energy normalization matrix and the potential energy normalization matrix. Based on the kinetic energy normalization matrix and the potential energy normalization matrix, a fast Fourier transform is performed on each column according to the data segment length and energy to obtain the kinetic energy frequency characteristics and the potential energy frequency characteristics. Frequency domain features are obtained by splicing the kinetic energy frequency features and the potential energy frequency features along the joint dimension.

6. The method as described in claim 1, characterized in that, The steps of performing feature selection and dimensionality reduction based on the frequency domain features to obtain emotion classification data include: Calculate the Pearson correlation coefficient between the frequency domain features and the preset emotion labels, and select a target number of features as candidate features based on the absolute value of the Pearson correlation coefficient. Principal component analysis was performed on the candidate features to obtain dimensionality-reduced features; The dimensionality reduction features are used as emotion feature vectors, and emotion classification data is obtained based on these emotion feature vectors.

7. The method as described in claim 1, characterized in that, The steps of acquiring human gait video data, extracting skeletal 3D coordinate data from the human gait video data, and performing Gaussian filtering on the skeletal 3D coordinate data to obtain skeletal joint point data include: Human gait video data is acquired, the three-dimensional positions of human joints in the human gait video data are identified, and initial three-dimensional coordinate data of the skeleton is obtained. The human gait video data is the video of the subject walking towards the camera. The initial three-dimensional coordinate data of the skeleton are processed by Gaussian weighted average in the X-axis, Y-axis and Z-axis directions according to a sliding window of preset length to obtain the skeleton joint data.

8. An emotion classification device based on gait data, characterized in that, The device includes: The data acquisition module is used to acquire human gait video data, extract skeletal three-dimensional coordinate data based on the human gait video data, and perform Gaussian filtering on the skeletal three-dimensional coordinate data to obtain skeletal joint point data. The energy calculation module is used to calculate the kinetic and potential energy of the bone joint data to obtain the average joint energy. The rigid body reconstruction module is used to reconstruct the joint energy mean value using a rigid body representation and to build a reconstructed skeleton structure. The normalization processing module is used to normalize the kinetic energy matrix and potential energy matrix in the reconstructed skeleton structure to obtain a normalized matrix, and to perform a fast Fourier transform on the normalized matrix to obtain frequency domain features. The emotion classification module is used to perform feature selection and dimensionality reduction based on the frequency domain features to obtain emotion classification data.

9. An emotion classification device based on gait data, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the emotion classification method based on gait data as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the emotion classification method based on gait data as described in any one of claims 1 to 7.

Images