Human body posture estimation method and system
By dividing the human body into multiple parts and using the MLP network model for processing step by step, the problem of action confusion and dynamic modeling of sparse inertial sensors in human body pose estimation is solved, and higher accuracy and real-time pose estimation are achieved.
Patent Information
- Application Number
- CN202510713019.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-30
- Publication Date
- 2025-08-26
- Estimated Expiration
- 2045-05-30
AI Technical Summary
The sparse inertial sensor configuration faces problems such as insufficient number of sensors, easy signal interference, difficulty in dynamic modeling and confusion in movement in human posture estimation, especially in complex movements, it is difficult to accurately distinguish different postures.
The human body is divided into multiple parts, and a deep learning network model is constructed, and the acceleration and joint position information are processed step by step. The MLP network model is used to estimate joint position and posture, and the joint layout is calculated in real time based on the speed information to reduce motion blur.
It improves the accuracy and real-time nature of human posture estimation, enhances the model's perception and response efficiency of local features, and improves the stability and generalization ability of the system.
Smart Images
Figure CN120234676B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a human body posture estimation method and system. Background Art
[0002] With the rapid development of artificial intelligence, the Internet of Things, and wearable devices, human motion analysis technology has become a core pillar in fields such as human-computer interaction and healthcare. Inertial measurement unit (IMU) technology has gradually become the market's preferred technology due to its low cost, high accuracy, and strong privacy. Although human pose estimation systems based on dense IMUs have been industrialized, large numbers of IMUs (typically 17) are difficult to wear and hinder the wearer's movements. Therefore, wearing a sparse set of IMUs is clearly more conducive to human movement. However, sparse inertial sensor configurations (typically requiring only 6-10 sensors) still face significant challenges in practical applications. First, the number of sensors is far less than the number of joints in the human body (typically 20-30), resulting in incomplete motion information. Sensor signals are also susceptible to electromagnetic interference, temperature drift, and loose wear. Furthermore, due to the strong nonlinearity of human motion (such as joint coupling during running and rapid posture changes in dancing), traditional models struggle to capture the dynamic correlations between spatiotemporal features, making dynamic modeling difficult.
[0003] When faced with complex movements (such as switching between standing and sitting), the lack of distinct changes in sensor signals can easily lead to motion confusion, making it difficult to accurately distinguish between different postures. Previous solutions have often achieved some success by analyzing historical motion records to distinguish these similar postures. However, this approach overlooks a key issue: the distribution of postures in datasets for different actions (such as cycling, reading, or driving) varies significantly. If the model relies solely on learning overall posture, it may perform poorly in inferring certain specific actions because the local motion characteristics of certain body structural units may be repeated in other actions. Summary of the Invention
[0004] The purpose of the present invention is to provide a method and system for estimating human posture in order to reduce motion blur and confusion and improve the accuracy of distinguishing different postures.
[0005] In order to achieve the above object, the technical solution adopted by the present invention is:
[0006] A method for estimating a human body posture comprises the following steps:
[0007] S1. Divide the human body into multiple parts and obtain motion data, velocity data in the initial state, joint starting position information set, and initial posture data of different parts of the human body; the motion data includes acceleration and sensor orientation collected by the motion sensor;
[0008] S2. Using motion data of different parts of the human body, velocity data in the initial state, joint starting position information set, and initial posture data as inputs of a deep learning network model to train the deep learning network model;
[0009] The deep learning network model includes a first MLP network model, a second MLP network model and a third MLP network model, the first MLP network model, the second MLP network model and the third MLP network model are connected in sequence, and the first MLP network model is connected to the third MLP network model;
[0010] The motion data are input into the first MLP network model, the second MLP network model and the third MLP network model respectively. The velocity data in the initial state is input into the first MLP network model, the joint starting position information set is input into the second MLP network model, and the initial posture data is input into the third MLP network model.
[0011] An objective function is constructed through the loss function of the first MLP network model, the loss function of the second MLP network model, and the loss function of the third MLP network model. The deep learning network model is trained by iterative optimization of the objective function to obtain posture estimation models of different parts of the human body.
[0012] The present invention provides an effective solution to the problem of motion blur by jointly estimating joint position information and motion speed. It uses acceleration data to calculate motion speed characteristics and capture the changing details of the motion as much as possible. The position of the human joints is deduced in real time based on the speed information to form an accurate joint layout. Combining the results of the first two steps, the posture information is output to complete the motion reconstruction. This step-by-step processing method fully utilizes the clues of acceleration and joint position, reduces motion blur, and maintains the advantage of fast response.
[0013] The present invention divides the human body into multiple parts for posture estimation, which correspond to the characteristics of these human body parts respectively. In this way, the model can better describe the relationship and characteristics of the joints within the part according to the characteristics of each part, better consider the autonomous feature information of different local body parts, and accurate local body posture estimation can ensure correct human body movements; by learning the spatial relationship between local body parts, the present invention effectively reduces the interference caused by weak dependence between long-distance joints, and improves the accuracy of human body movement estimation.
[0014] The present invention combines human body region decoupling and step-by-step processing to fully explore some of the movement characteristics of the human body structure, integrate multimodal features, and more accurately learn the joint dependencies of various parts. It uses the spatial characteristics of these parts to improve the calculation accuracy of speed and joint position, thereby achieving more stable human posture recognition in diverse action scenarios, improving the real-time and accuracy of human motion estimation, and improving the stability and generalization ability of the overall system.
[0015] Furthermore, the first MLP network model, the second MLP network model, and the third MLP network model each include a first input layer, a second input layer, a cosine discrete transform layer, a first fusion layer, a first fully connected layer, a first normalization layer, a second fusion layer, an inverse cosine discrete transform layer, a third fusion layer, a second fully connected layer, a second normalization layer, a fourth fusion layer, a fifth fusion layer, a third fully connected layer, and a third normalization layer;
[0016] The first input layer, the cosine discrete transform layer, the first fusion layer, the first fully connected layer, the first normalization layer, the second fusion layer and the inverse cosine discrete transform layer are connected in sequence, the second input layer is connected to the first fusion layer, and the cosine discrete transform layer is connected to the second fusion layer;
[0017] The first input layer, the third fusion layer, the second fully connected layer, the second normalized layer, and the fourth fusion layer are connected in sequence. The second input layer is connected to the third fusion layer, and the first input layer is connected to the fourth fusion layer.
[0018] The inverse cosine discrete transform layer and the fourth fusion layer are connected to the fifth fusion layer, and the fifth fusion layer, the third fully connected layer and the third normalization layer are connected in sequence.
[0019] The proposed MLP network model utilizes a dual-channel architecture, fully integrating the local and global characteristics of joint velocity and position information. The cosine discrete transform (CDT) path constructs long-range dependencies within time series through CDT, enhancing global dynamic modeling capabilities. The other path focuses on preserving micro-motion changes at the joint level, improving the model's perception and response efficiency to local dynamic features. The overall architecture achieves lower latency and higher real-time performance while maintaining high accuracy, improving computational efficiency and generalization capabilities.
[0020] Furthermore, the different parts of the human body include an upper part, a middle part, and a lower part.
[0021] Furthermore, the motion data of the upper part, the velocity data in the initial state, the joint initial position information set, and the initial posture data are input into the posture estimation model of the upper part to obtain the posture of the upper part;
[0022] Input the motion data of the middle part, the velocity data in the starting state, the joint starting position information set, and the initial posture data into the posture estimation model of the middle part to obtain the posture of the middle part;
[0023] Input the motion data of the lower part, the velocity data in the initial state, the joint starting position information set, and the initial posture data into the posture estimation model of the lower part to obtain the posture of the lower part;
[0024] The upper part posture, the middle part posture and the lower part posture are combined to obtain the human body posture.
[0025] Furthermore, the loss function L1 of the first MLP network model is expressed as follows:
[0026] ;
[0027] in, represents positive kinematics, Indicates the time interval, Representing different parts of the human body The true attitude of the moment, The motion data of different parts of the human body, is the speed data of different parts of the human body at the initial state, represents the output of the first MLP network model, is the 2-norm.
[0028] Furthermore, the loss function L2 of the second MLP network model is expressed as follows:
[0029] ;
[0030] in, , is the output of the first MLP network model, represents the output of the second MLP network model, It is the starting position information set of joints of different parts of the human body.
[0031] Furthermore, the expression of the loss function L3 of the third MLP network model is as follows:
[0032] ;
[0033] in, Indicates the true posture of different parts of the human body, , is the output of the second MLP network model, Represents the output of the third MLP network model, It is the initial posture data of different parts of the human body.
[0034] Furthermore, the objective function L is expressed as follows:
[0035] ;
[0036] Among them, L1 is the loss function of the first MLP network model, L2 is the loss function of the second MLP network model, and L3 is the loss function of the third MLP network model.
[0037] Based on the same inventive concept, the present invention also provides a human body posture estimation system, comprising:
[0038] A data acquisition module, used to obtain sensor data or image data related to human body movement;
[0039] A data preprocessing module, used for performing format conversion, denoising and normalization processing on the sensor data or image data;
[0040] A posture estimation module, configured to perform a human posture estimation method based on preprocessed sensor data or image data;
[0041] The output module is used to output the estimated human posture results.
[0042] Based on the same inventive concept, the present invention further provides an electronic device, comprising:
[0043] one or more processors;
[0044] A memory having one or more programs stored thereon, which, when executed by the one or more processors, enables the one or more processors to implement the steps of the human body posture estimation method.
[0045] Based on the same inventive concept, the present invention also provides a computer-readable storage medium storing a computer program, which implements the steps of the human body posture estimation method when executed by a processor.
[0046] Compared with the prior art, the present invention has the following beneficial effects:
[0047] The present invention divides the human body into multiple parts for posture estimation, which correspond to the characteristics of these human body parts respectively. In this way, the model can better describe the relationship and characteristics of the joints within the part according to the characteristics of each part, and better consider the autonomous feature information of different local body parts. Accurate local body posture estimation can ensure correct human body movements and improve the accuracy and applicability of human body movement reconstruction.
[0048] The present invention constructs posture estimation models for different parts of the human body respectively, uses motion data to calculate the velocity characteristics in the initial state, and captures the changing details of the movement as much as possible; the position of the human joints is inferred in real time based on the velocity information to form an accurate joint layout; combined with the results of the first two steps, the posture information is output to complete the movement reconstruction; this step-by-step processing method fully utilizes the clues of acceleration and joint position, reduces motion blur, and maintains the advantage of fast response.
[0049] The present invention combines human body region decoupling and step-by-step processing to fully explore some of the movement characteristics of the human body structure, integrate multimodal features, and more accurately learn the joint dependencies of various parts. It uses the spatial characteristics of these parts to improve the calculation accuracy of speed and joint position, thereby achieving more stable human posture recognition in diverse action scenarios, improving the real-time and accuracy of human motion estimation, and improving the stability and generalization ability of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 Schematic diagram of a human body posture estimation model of the present invention;
[0051] Figure 2 Schematic diagram of the MLP model of the present invention;
[0052] Figure 3 This is a schematic diagram of the human body joints;
[0053] Figure 4 Schematic diagram comparing model visualization actions and real actions according to an embodiment of the present invention. DETAILED DESCRIPTION
[0054] The present invention will be described in detail below with reference to the following embodiments. It should be noted that the embodiments and features of the embodiments may be combined unless they conflict. For ease of description, the words "upper," "lower," "left," and "right" appearing below merely indicate the directions of upper, lower, left, and right relative to the accompanying drawings and do not limit the structure.
[0055] Example
[0056] 1. Global direction alignment
[0057] Six inertial sensors (IMUs) are installed on the pelvis, left and right calves, left and right forearms, and head. The pelvis is called the root joint, and the other joints are called leaf joints. The rotation information (sensor orientation) and acceleration measurement values of each IMU are aligned to the same reference frame and normalized to obtain the concatenated input vector: , aligns the global orientation of the IMU with the global rotation matrix of the skeleton, where is the acceleration, is the rotation matrix (sensor orientation), expanded column-wise. Indicates that the length of the time series is P, LA, RA, H, LL, RL represent pelvis, left arm, right arm, head, left leg, and right leg, respectively.
[0058] 2. Body Part Division
[0059] Traditional sparse IMU-based human pose estimation methods typically rely on six sensors to capture full-body motion signals. However, when faced with complex movements (such as switching between standing and sitting), the lack of distinct sensor signal changes can easily lead to motion confusion, making it difficult to accurately distinguish between different postures. Previous solutions have often achieved some success by analyzing historical motion records to distinguish these similar postures. However, this approach overlooks a key issue: the distribution of postures varies significantly across different actions in the dataset (such as cycling, reading, or driving). Relying solely on overall posture learning, the model may perform poorly in inferring certain specific actions, as the local motion characteristics of certain body structural units may be repeated in other actions.
[0060] To overcome these issues, this embodiment proposes an improved human pose estimation method that significantly improves the accuracy and applicability of motion reconstruction by dividing the human body into three independent parts. Specifically, the human body is divided into the upper region (UR), the torso region (TR), and the lower region (LR), and the motion characteristics of each part are analyzed separately.
[0061] The core of this method is to construct a joint association graph (such as Figure 3 ). For all the joints in the actions of the dataset DIP-IMU and Xsens Dataset (Xsens Dataset is composed of AnDy, Emokine, Virginia Natural Motion, UNIPD, CIP) Figure 3 The sensor orientation information (shown in the horizontal and vertical coordinates of the left figure) is analyzed. The Pearson correlation coefficient between the sensor orientation information is used to calculate the correlation matrix, and then the different joints are classified according to their correlation. The original human joints are then divided into parts according to the classification information (i.e. Figure 3 different dotted boxes in the left figure).
[0062] Figure 3The results show that the joint motion characteristics within the same body part are closely connected, while the connection between different parts is weak. Based on this finding, a regional decoupling processing strategy was designed, which uses the data collected by 6 IMU sensors and groups them according to body parts, with each group of data corresponding to the motion information of one body part.
[0063] The input vector X is divided into parts corresponding to the input vector X, which can be expressed as:
[0064] (1)
[0065] in, Indicates the acceleration and sensor orientation data of the upper part, Represents the acceleration and sensor orientation data of the middle part, Represents the acceleration and sensor orientation data of the lower part.
[0066] 3. Determine the modeling structure
[0067] A deep learning network model is constructed. By dividing the human body into different parts, the acceleration and orientation data for each part are fed into the deep learning network for training. This results in a posture estimation model for each part, corresponding to the characteristics of the three body parts. This allows the model to better describe the relationships and characteristics of the joints within each part based on the characteristics of each part, and better considers the autonomous characteristic information of different local body parts. Accurate local body posture estimation is crucial for ensuring accurate human motion estimation.
[0068] When using sparse IMU sensors to determine human posture, a major challenge is extracting sufficient motion details from the diverse data while ensuring simple and fast processing. Common IMU data includes information such as sensor orientation and joint positions. Previous methods relied solely on sensor orientation to infer motion, but this can be confusing in certain situations (such as sitting or standing) because the signals for these movements are not significantly different. In reality, IMU acceleration data and joint position information contain richer motion cues, effectively reducing this confusion. However, current technologies fail to fully exploit this information for the following reasons: joint positions require complex calculations, which are time-consuming and cannot be processed in real time; acceleration data only reflects transient changes at the beginning of a movement and cannot fully capture the state of ongoing motion; and acceleration signals are subject to significant noise interference, making them more difficult to process than orientation data.
[0069] To address the above shortcomings, this embodiment proposes a simple and efficient human posture estimation method, called "Simple Inertial Poser" (SiIP), such as Figure 1This method uses a network model based on MLP (Multi-layer Perceptron) to process multiple information step by step, ultimately achieving accurate posture estimation.
[0070] Specifically, the workflow of this method is divided into three steps (e.g. Figure 1 ):
[0071] Velocity extraction: Utilizes the acceleration data provided by the IMU sensor to calculate the motion velocity characteristics and capture the changing details of the movement as much as possible;
[0072] Joint positioning: Calculate the position of human joints in real time based on speed information to form an accurate joint layout;
[0073] Posture generation: Combining the results of the first two steps, output posture information (i.e., the rotation angle of each joint) to complete motion reconstruction.
[0074] The MLP network is used as the basic framework to avoid complex calculations. Compared with traditional methods, this step-by-step processing method fully utilizes the cues of acceleration and joint position, reducing motion blur while maintaining the advantage of fast response.
[0075] (1) Model process first level: velocity extraction
[0076] Since IMU acceleration data can only capture the changes at the beginning of the movement and cannot reflect the continuous movement state, this method first uses a velocity estimation module (V_MLP, the first MLP network model, such as Figure 2 ) analyzes acceleration information to generate joint motion velocities. The velocity estimation module takes as input calibrated IMU data (including acceleration and sensor orientation) and a velocity reference value from the initial movement, and outputs an estimated velocity result. This approach uses velocity information to compensate for the limitations of acceleration and ensure movement continuity.
[0077] The input of the velocity estimation module is the calibrated IMU measurement value, that is, , is the time series length of the action dataset, and each frame of data contains acceleration information and orientation information , the specific composition is shown in formula (2):
[0078] (2)
[0079] in, is the acceleration of the pelvis and the sensor orientation data, is the acceleration and sensor orientation data of the left arm, is the acceleration and sensor orientation data of the right arm, is the head acceleration and sensor orientation data, is the acceleration and sensor orientation data of the left leg, The acceleration and sensor orientation data for the right leg.
[0080] The first-level model uses a multi-layer perceptron to build the model V_MLP, and the output is the speed estimate , the first-level model output is expressed as , this method converts the velocity data in the initial state contained in the first frame of an action sequence As an additional input, the specific composition is shown in formulas (3) and (4):
[0081] (3)
[0082] (4)
[0083] (5)
[0084] in, Indicates the true speed of the human body, The superscripts indicate different parts of the human body or different positions of different parts. is the true speed of the upper part, is the true speed of the middle part, is the true speed of the lower part, is the true velocity of the pelvis, is the true speed of the left arm, is the true speed of the right arm, is the true velocity of the head, is the true speed of the left leg, is the true speed of the right leg.
[0085] Take the first frame of the human body's true speed as the speed data in the initial state , The superscripts indicate different parts of the human body or different locations of different parts. is the speed data in the initial state of the above part, is the speed data at the starting state of the middle part, This is the speed data in the initial state of the following part, is the velocity data of the pelvis at the initial state, is the speed data of the left arm at the starting state, is the speed data of the right arm at the initial state, is the speed data of the head in the initial state, is the speed data of the left leg at the starting state, This is the speed data of the right leg in the starting state.
[0086] represents positive kinematics, express The true attitude of the moment, Indicates a time interval.
[0087] Therefore, the first level is as shown in formula (6):
[0088] (6)
[0089] in, is the output of the first MLP network model, The acceleration and orientation data of different parts of the human body, for or or , when estimating the above posture, ; When estimating the posture of the middle part, ; When estimating the posture in the following part, ; is the speed data of different parts of the human body at the initial state, for or or , when estimating the above posture, ; When estimating the posture of the middle part, ; When estimating the posture in the following part, .
[0090] The loss function of the first level As shown in formula (7):
[0091] (7)
[0092] in, is the 2-norm, Representing different parts of the human body The true attitude of the moment, for or or , For the upper part The true attitude of the moment, For the middle part The true attitude of the moment, For the following part The true attitude of the moment.
[0093] For public data sets, the true speed, posture, joint position, etc. are derived from the public data sets; for self-made data sets, the IMU is equipped with a motion capture system, and the motion capture system calculates the true speed, posture, joint position, etc. when collecting and acquiring IMU data.
[0094] (2) Second level of model process: joint positioning
[0095] After obtaining the speed information, the second step is to use the position estimation module (J_MLP, the second MLP network model, such as Figure 2 ) calculates the specific position of each joint. The dead reckoning module combines the velocity results from the first step, IMU data, and the initial joint position reference, and outputs the estimated joint positions. This step-by-step design quickly generates accurate joint layouts.
[0096] Take the joint starting position information set contained in the first frame of an action sequence, and its specific composition is shown in formulas (8) and (9):
[0097] (8)
[0098] (9)
[0099] (10)
[0100] in, Indicates the real joint position of the human body, The superscripts indicate different parts of the human body or different locations of different parts. is the actual joint position of the upper part, is the real joint position of the middle part, is the actual joint position of the lower part, is the true joint position of the pelvis, is the true joint position of the left arm, is the true joint position of the right arm, is the actual joint position of the head, is the true joint position of the left leg, is the true joint position of the right leg.
[0101] Take the first frame of the real joint position of the human body as the joint starting position information set , The superscripts indicate different parts of the human body or different locations of different parts. is the joint starting position information set in the upper part, is the starting position information set of the joints in the middle part, For the joint starting position information set in the following part, is the starting position information set of the pelvic joints, is the starting position information set of the left arm joint, is the starting position information set of the right arm joint, is the starting position information set of the head joints, is the starting position information set of the left leg joints, is the starting position information set of the right leg joints.
[0102] The output of the second-level model J_MLP is expressed as , for joint positions The second level is shown in formula (11):
[0103] (11)
[0104] in, , is the starting position information set of joints of different parts of the human body, for or or ; When estimating the posture of the above part, ; When estimating the posture of the middle part, ; When estimating the posture in the following part, .
[0105] The loss function of the second level It can be expressed as:
[0106] (12)
[0107] (3) Model process level 3: pose generation
[0108] The third step is to generate the pose through the pose generation module (P_MLP, the third MLP network model, such as Figure 2 ) integrates the results of the previous two steps (velocity and joint position) with IMU data to generate the pose (i.e., joint rotation angles). The pose generation module also inputs an initial pose reference value to ensure stable results. Finally, the poses of different body parts are combined to output a complete description of the human body's movements.
[0109] Take the initial posture information contained in the first frame of an action sequence, and the specific composition is shown in formulas (13) and (14):
[0110] (13)
[0111] (14)
[0112] in, Indicates the real posture of the human body. The superscripts represent different parts of the human body or different positions of different parts. LS, LUA, RS, RUA, N, LUL, RUL, L5, L3, T12, and T8 represent left shoulder, left upper arm, right shoulder, right upper arm, neck, left thigh, right thigh, spine L5, spine L3, spine T12, and spine T8 respectively. For the true posture of the upper part, The true posture of the middle part, For the true posture of the lower part, For the true posture of the left shoulder, This is the true posture of the left upper arm. For the true posture of the right shoulder, This is the true posture of the right upper arm. For the true posture of the neck, This is the true posture of the left thigh. This is the true posture of the right thigh. The true posture of the spine L5, The true posture of the spine L3, The true posture of the spine T12, This is the true posture of spine T8.
[0113] Take the first frame of the human body's true posture as the initial posture information , The superscripts indicate different parts of the human body or different locations of different parts. is the initial posture data of the above part, is the initial posture data of the middle part, is the initial posture data of the following part, is the initial posture data of the left shoulder, is the initial posture data of the left upper arm, is the initial posture data of the right shoulder, is the initial posture data of the right upper arm, is the initial posture data of the neck, is the initial posture data of the left thigh, is the initial posture data of the right thigh, is the initial posture data of the spine L5, is the initial posture data of the spine L3, is the initial posture data of the spine T12, is the initial posture data of spine T8.
[0114] The output of the third-level model P_MLP is expressed as , in order to estimate the joint velocity from the IMU acceleration and orientation and the estimated joint positions Estimated pose , the specific process is shown in formula (15):
[0115] (15)
[0116] in, , is the initial posture data of different parts of the human body, for or or ; When estimating the posture of the above part, ; When estimating the posture of the middle part, ; When estimating the posture in the following part, .
[0117] Finally, according to the estimated human pose Real human posture , loss function It can be expressed as
[0118] (16)
[0119] in, Indicates the true posture of different parts of the human body, for or or .
[0120] Finally, the objective function used to train the entire model is shown in formula (17):
[0121] (17)
[0122] The velocity estimation module, position estimation module, and posture generation module are all MLP_Block embedding models for dual-channel feature processing. The MLP_Block embedding model for dual-channel feature processing includes a first input layer, a second input layer, a cosine discrete transform layer, a first fusion layer, a first fully connected layer, a first normalization layer, a second fusion layer, an inverse cosine discrete transform layer, a third fusion layer, a second fully connected layer, a second normalization layer, a fourth fusion layer, a fifth fusion layer, a third fully connected layer, and a third normalization layer.
[0123] The first input layer, the cosine discrete transform layer, the first fusion layer, the first fully connected layer, the first normalization layer, the second fusion layer, and the inverse cosine discrete transform layer are connected in sequence. The second input layer is connected to the first fusion layer; the cosine discrete transform layer is connected to the second fusion layer.
[0124] The first input layer, the third fusion layer, the second fully connected layer, the second normalized layer, and the fourth fusion layer are connected in sequence. The second input layer is connected to the third fusion layer; the first input layer is connected to the fourth fusion layer.
[0125] The inverse cosine discrete transform layer and the fourth fusion layer are both connected to the fifth fusion layer, and the fifth fusion layer, the third fully connected layer and the third normalization layer are connected in sequence.
[0126] The first input layer inputs motion data, and the second input layer inputs velocity data in the starting state or a joint starting position information set or initial posture data.
[0127] The input to the MLP_Block embedding model, which uses dual-channel feature processing, consists of two parts: the main network input and the initial auxiliary information for each stage. The MLP_Block process divides the input features into two paths: one path first performs a discrete cosine transform (DCT), then inputs into a fully connected layer (FC) and layer normalization (LN) module, and then restores the temporal information through an inverse discrete cosine transform (IDCT). The other path skips frequency domain processing and directly performs linearization and normalization operations. Finally, the output features of the two paths are concatenated and fused, and then output through the FC and LN layers again to form the estimated result for this stage. This dual-channel architecture design fully integrates the local and global characteristics of joint velocity and position information. The DCT path effectively establishes long-range dependencies in the time series, enhancing global dynamic modeling capabilities. The original path, on the other hand, preserves the detailed variations in joint motion, ensuring the model's efficient response to local features. The overall structure achieves lower latency and higher real-time performance while maintaining high accuracy, demonstrating excellent computational efficiency and generalization capabilities, and is an effective technical path for real-time human motion estimation tasks.
[0128] This embodiment provides an innovative method for human pose estimation that significantly improves the accuracy and reliability of estimation by processing motion information in a step-by-step manner. The method consists of two main steps: first, estimating the movement speed of the joints, and then determining the specific positions of the joints. This step-by-step strategy can capture the unique movement characteristics of the upper and lower parts respectively, and use the spatial characteristics of these parts to improve the calculation accuracy of speed and joint position, thereby achieving more stable pose recognition in a variety of action scenarios.
[0129] Frames are selected from DIP-IMU and Xsens Dataset (including AnDy, CIP, and Virginia) to visualize the action. The model visualization action and the real action are compared as follows: Figure 4As shown in the figure, it can be seen that because the model adopts the body region decoupling strategy and the step-by-step strategy, it can better capture the local motion characteristics of the body and the temporal and spatial information related to the joints during human movements. Therefore, it still has stable and good performance in some postures that are prone to ambiguity, such as sitting, standing, and body crossing.
[0130] The following metrics are used to evaluate the performance of this method: Joint Position Error, which measures the average Euclidean distance error (in centimeters) of all estimated joints when the hip joint is aligned. Performance is tested on the Xsens Dataset (AnDy, UNIPD, CIP, Virginia) and the DIP-IMU dataset.
[0131] 1. Experimental results of decoupling schemes for different regions of the human body
[0132] The experimental results of different human body region decoupling schemes are shown in Table 1. 0-part indicates no human body region decoupling and the combination of this pose estimation model. 3-part (Ours) indicates the combination of three human body parts (upper part, middle part, lower part) and this pose estimation model. 5-part indicates the combination of five human body parts (left arm, right arm, middle part, left leg, right leg) and this pose estimation model.
[0133] Experimental results show that without body region decoupling, the model cannot capture the local temporal and spatial information of human movements, while too much region decoupling will cause the model to be unable to establish global information, thereby reducing performance.
[0134] 2. Experimental results of different pose estimation models
[0135] Table 2 shows the experimental results of different pose estimation models. "with / o Vel" represents a model combination that divides the human body into three parts and does not include a velocity extraction stage; "with / o Joint" represents a model combination that divides the human body into three parts and does not include a joint localization stage; and "with / o Joint & Vel" represents a model combination that divides the human body into three parts and does not include both a velocity extraction stage and a joint localization stage. The experimental results show that using both joint velocity and joint position information can better identify human motions.
[0136] 3. Experimental results of different MLP_Block feature processing methods
[0137] Table 3 shows the experimental results for different MLP_Block feature processing methods. "w / o DCT" indicates that the MLP model does not include a DCT (Discrete Cosine Transform) branch, and "only DCT" indicates that only the DCT branch is retained. The experimental results show that the MLP model with dual-channel feature processing performs better.
[0138] This human pose recognition method is based on the idea of local body segmentation. It divides the human body into three parts: upper, middle, and lower. These parts are then modeled separately to more accurately learn the inter-joint dependencies within each part. This approach not only effectively reduces interference caused by weak inter-joint dependencies across long distances, but also improves the accuracy of motion estimation.
[0139] To simplify the complex structure of traditional methods, a multi-layer perceptron (MLP) was used as the basic framework to design a step-by-step regression mechanism to jointly estimate joint position information and motion velocity, providing an effective solution to the dynamic fuzzy problem. This strategy reduces model complexity while integrating multimodal features, improving the stability and generalization capabilities of the overall system, thereby achieving higher-quality motion estimation.
[0140] The feature modeling strategy based on human body part division effectively enhances the representation ability between local parts; the MLP-driven step-by-step framework achieves a good balance between accuracy and computational efficiency.
[0141] This method fully exploits some characteristics of the human body structure and its spatiotemporal dynamic properties, improving the real-time and accuracy of motion estimation.
[0142] Another embodiment of the present invention provides a human body posture estimation system, comprising:
[0143] A data acquisition module, used to obtain sensor data or image data related to human body movement;
[0144] A data preprocessing module, used for performing format conversion, denoising and normalization processing on the sensor data or image data;
[0145] A posture estimation module, configured to perform a human posture estimation method based on preprocessed sensor data or image data;
[0146] The output module is used to output the estimated human posture results.
[0147] Another embodiment of the present invention provides an electronic device, including:
[0148] one or more processors;
[0149] A memory stores one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the steps of the human body posture estimation method.
[0150] In some implementations, the memory may be a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage.
[0151] In other implementations, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors, which are not limited herein.
[0152] Another embodiment of the present invention provides a computer-readable storage medium storing a computer program, which implements the steps of the human body posture estimation method when executed by a processor.
[0153] The contents illustrated in the above embodiments should be understood as these embodiments are only used to more clearly illustrate the present invention, and are not used to limit the scope of the present invention. After reading the present invention, various equivalent modifications of the present invention by those skilled in the art shall fall within the scope defined by the claims attached to this application.
Claims
1. A method for estimating a human body posture, characterized in that: The following steps are involved: S1. Divide the human body into multiple parts and obtain motion data, velocity data in the initial state, joint starting position information set, and initial posture data of different parts of the human body; the motion data includes acceleration and sensor orientation collected by the motion sensor; S2. Using motion data of different parts of the human body, velocity data in the initial state, joint starting position information set, and initial posture data as inputs of a deep learning network model to train the deep learning network model; The deep learning network model includes a first MLP network model, a second MLP network model and a third MLP network model, the first MLP network model, the second MLP network model and the third MLP network model are connected in sequence, and the first MLP network model is connected to the third MLP network model; The motion data are input into the first MLP network model, the second MLP network model and the third MLP network model respectively. The velocity data in the initial state is input into the first MLP network model, the joint starting position information set is input into the second MLP network model, and the initial posture data is input into the third MLP network model. Constructing an objective function through the loss function of the first MLP network model, the loss function of the second MLP network model, and the loss function of the third MLP network model, and training the deep learning network model through iterative optimization of the objective function to obtain posture estimation models of different parts of the human body; The first MLP network model, the second MLP network model, and the third MLP network model each include a first input layer, a second input layer, a cosine discrete transform layer, a first fusion layer, a first fully connected layer, a first normalization layer, a second fusion layer, an inverse cosine discrete transform layer, a third fusion layer, a second fully connected layer, a second normalization layer, a fourth fusion layer, a fifth fusion layer, a third fully connected layer, and a third normalization layer; The first input layer, the cosine discrete transform layer, the first fusion layer, the first fully connected layer, the first normalization layer, the second fusion layer and the inverse cosine discrete transform layer are connected in sequence, the second input layer is connected to the first fusion layer, and the cosine discrete transform layer is connected to the second fusion layer; The first input layer, the third fusion layer, the second fully connected layer, the second normalized layer, and the fourth fusion layer are connected in sequence. The second input layer is connected to the third fusion layer, and the first input layer is connected to the fourth fusion layer. The inverse cosine discrete transform layer and the fourth fusion layer are connected to the fifth fusion layer, and the fifth fusion layer, the third fully connected layer and the third normalization layer are connected in sequence.
2. The method for estimating human body posture according to claim 1, wherein The different parts of the human body include an upper part, a middle part, and a lower part.
3. The method for estimating human body posture according to claim 2, wherein: Input the motion data of the upper part, the velocity data in the initial state, the joint starting position information set, and the initial posture data into the posture estimation model of the upper part to obtain the posture of the upper part; Input the motion data of the middle part, the velocity data in the starting state, the joint starting position information set, and the initial posture data into the posture estimation model of the middle part to obtain the posture of the middle part; Input the motion data of the lower part, the velocity data in the initial state, the joint starting position information set, and the initial posture data into the posture estimation model of the lower part to obtain the posture of the lower part; The upper part posture, the middle part posture and the lower part posture are combined to obtain the human body posture.
4. The method for estimating human body posture according to claim 1, wherein: The expression of the loss function L1 of the first MLP network model is as follows: ; in, represents positive kinematics, Indicates the time interval, Representing different parts of the human body The true attitude of the moment, The motion data of different parts of the human body, is the speed data of different parts of the human body at the initial state, represents the output of the first MLP network model, is the 2-norm.
5. The method for estimating human body posture according to claim 1, wherein: The expression of the loss function L2 of the second MLP network model is as follows: ; in, , is the output of the first MLP network model, represents the output of the second MLP network model, It is the starting position information set of joints of different parts of the human body.
6. The method for estimating human body posture according to claim 1, wherein: The expression of the loss function L3 of the third MLP network model is as follows: ; in, Indicates the true posture of different parts of the human body, , is the output of the second MLP network model, Represents the output of the third MLP network model, It is the initial posture data of different parts of the human body.
7. The method for estimating human body posture according to claim 1, wherein: The expression of the objective function L is as follows: ; Among them, L1 is the loss function of the first MLP network model, L2 is the loss function of the second MLP network model, and L3 is the loss function of the third MLP network model.
8. A human body posture estimation system, characterized in that: include: A data acquisition module, used to obtain sensor data or image data related to human body movement; A data preprocessing module, used for performing format conversion, denoising and normalization processing on the sensor data or image data; a posture estimation module, configured to execute the human posture estimation method according to any one of claims 1 to 7 based on preprocessed sensor data or image data; The output module is used to output the estimated human posture results.
9. An electronic device, characterized in that: include: one or more processors; A memory having one or more programs stored thereon, which, when executed by the one or more processors, enables the one or more processors to implement the steps of the method according to any one of claims 1 to 7.