A human motion tracking method and system based on regional saliency analysis

Abstract

The application provides a human motion tracking method and system based on regional saliency analysis in the technical field of motion tracking, and the method comprises the following steps: step S10, creating a gating network model and a plurality of expert models based on a convolutional neural network, and training each gating network model and expert model; step S20, acquiring human posture data and monitoring data of a flexible sensor; step S30, inputting the human posture data into the gating network model to obtain weight coefficients of each expert model; step S40, inputting the monitoring data into each expert model to obtain posture prediction results, obtaining the human posture of a current frame based on the weight coefficients and the posture prediction results, and then completing human motion tracking. The application has the advantage that the accuracy of human motion tracking is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of motion tracking technology, and in particular to a human motion tracking method and system based on regional saliency analysis. Background Technology

[0002] Motion tracking, also known as motion capture, involves using sensors (devices) to track and record the movement of a human body or other moving objects. Motion tracking devices are categorized into optical motion capture devices based on optical cameras and inertial motion capture devices based on inertial measurement units (IMUs). In recent years, with the development of flexible sensors, motion tracking can also be performed using flexible sensors. Flexible sensors are generally used to detect stretching and deformation, and have advantages such as biocompatibility, high tensile strength, and lightweight design.

[0003] Existing methods for human motion tracking using flexible sensors generally fall into two categories: one directly uses full-body sensor data to predict the overall posture of the human body, such as Deep Full-Body Motion Network for a Soft Wearable Motion Sensing Suit; the other uses sensors only for specific body parts to predict the joint posture of those parts, such as Soft Smart Garments for Lower Limb Joint Position Analysis. However, the former is susceptible to noise in the dataset, leading to overfitting, while the latter fails to learn the implicit relationships between different joint angles in the human body. Furthermore, flexible sensors often experience positional shifts and deformation / aging, resulting in significant noise in the measured data, posing a major challenge to data analysis and utilization.

[0004] Therefore, how to provide a human motion tracking method and system based on regional saliency analysis to improve the accuracy of human motion tracking has become an urgent technical problem to be solved. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a human motion tracking method and system based on regional saliency analysis, so as to improve the accuracy of human motion tracking.

[0006] In a first aspect, the present invention provides a human motion tracking method based on regional saliency analysis, comprising the following steps:

[0007] Step S10: Create a gated network model and several expert models based on a convolutional neural network, and train each of the gated network models and expert models;

[0008] Step S20: Acquire human posture data and monitoring data from flexible sensors;

[0009] Step S30: Input the human posture data into the gating network model to obtain the weight coefficients of each expert model;

[0010] Step S40: Input the monitoring data into each expert model to obtain the posture prediction result. Based on the weight coefficient and the posture prediction result, obtain the human posture of the current frame, and then complete the human motion tracking.

[0011] Furthermore, in step S10, the gated network model is used to output the weight coefficients of each expert model based on human posture data, and includes three fully connected layers.

[0012] The expert model is used to output attitude prediction results based on the monitoring data of the flexible sensor. It includes one BN layer, four LSTM layers, and one fully connected layer. The fully connected layer is used as the input layer, the MSELoss function is used as the loss function, and the ReLU function is used as the activation function.

[0013] Further, step S20 specifically includes:

[0014] Human posture data is acquired through optical motion capture devices, and human monitoring data is acquired through flexible sensors;

[0015] The human posture data is recorded as y (t) y (t) It is a 3×M matrix; the monitoring data record is x (t) x (t) It is a vector of length S;

[0016] Where t represents the time step; S represents the number of flexible sensors; and M represents the number of joints in the human body that need to be predicted.

[0017] Further, step S30 specifically includes:

[0018] The human pose data y from the previous frame (t-1) Input the gated network model and obtain the weight coefficients w(K) of each expert model; where K represents the expert model number.

[0019] Further, step S40 specifically includes:

[0020] The monitoring data from the previous frame is input into each expert model to obtain the pose prediction results. The pose prediction results are then weighted and averaged based on the weight coefficients to obtain the human pose of the current frame, thereby completing human motion tracking.

[0021] Secondly, the present invention provides a human motion tracking system based on regional saliency analysis, comprising the following modules:

[0022] The model creation module is used to create a gated network model and several expert models based on a convolutional neural network, and to train each of the gated network models and expert models.

[0023] The human body data acquisition module is used to acquire human posture data and monitoring data from flexible sensors;

[0024] The weight coefficient calculation module is used to input the human posture data into the gated network model and obtain the weight coefficients of each expert model.

[0025] The motion tracking module is used to input the monitoring data into each expert model to obtain the posture prediction result, and obtain the human posture of the current frame based on the weight coefficients and the posture prediction result, thereby completing human motion tracking.

[0026] Furthermore, in the model creation module, the gated network model is used to output the weight coefficients of each expert model based on human posture data, and includes three fully connected layers;

[0027] The expert model is used to output attitude prediction results based on the monitoring data of the flexible sensor. It includes one BN layer, four LSTM layers, and one fully connected layer. The fully connected layer is used as the input layer, the MSELoss function is used as the loss function, and the ReLU function is used as the activation function.

[0028] Furthermore, the human body data acquisition module specifically comprises:

[0029] Human posture data is acquired through optical motion capture devices, and human monitoring data is acquired through flexible sensors;

[0030] The human posture data is recorded as y (t) y (t) It is a 3×M matrix; the monitoring data record is x (t) x (t) It is a vector of length S;

[0031] Where t represents the time step; S represents the number of flexible sensors; and M represents the number of joints in the human body that need to be predicted.

[0032] Furthermore, the weight coefficient calculation module specifically comprises:

[0033] The human pose data y from the previous frame (t-1) Input the gated network model and obtain the weight coefficients w(K) of each expert model; where K represents the expert model number.

[0034] Furthermore, the motion tracking module specifically comprises:

[0035] The monitoring data from the previous frame is input into each expert model to obtain the pose prediction results. The pose prediction results are then weighted and averaged based on the weight coefficients to obtain the human pose of the current frame, thereby completing human motion tracking.

[0036] The advantages of this invention are:

[0037] By creating and training a gated network model, the gated network model calculates the weight coefficients of each expert model based on human posture data. That is, it analyzes the changes in the prediction weights of different flexible sensors on joints under different movements based on regional saliency analysis to reduce the impact of noise. By setting multiple expert models to predict the monitoring data of flexible sensors, the implicit correlation between different joint angles of the human body can be learned, which ultimately greatly improves the accuracy of human motion tracking. Attached Figure Description

[0038] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0039] Figure 1 This is a flowchart of a human motion tracking method based on regional saliency analysis according to the present invention.

[0040] Figure 2 This is a schematic diagram of the structure of a human motion tracking system based on regional saliency analysis according to the present invention.

[0041] Figure 3 This is a flowchart illustrating the present invention.

[0042] Figure 4 This is a schematic diagram of the human motion tracking of the present invention. Detailed Implementation

[0043] The technical solution in this application embodiment has the following general idea: the weight coefficients of each expert model are calculated based on human posture data through a gating network model, that is, the changes in the prediction weights of different flexible sensors on joints under different actions are analyzed based on regional saliency to reduce the influence of noise; by setting multiple expert models, the implicit correlation between different joint angles of the human body is learned, thereby improving the accuracy of human motion tracking.

[0044] Please refer to Figures 1 to 4 As shown, a preferred embodiment of the human motion tracking method based on regional saliency analysis of the present invention includes the following steps:

[0045] Step S10: Create a gated network model and several expert models based on a convolutional neural network, and train each of the gated network models and expert models; the prediction network is composed of several expert models.

[0046] Step S20: Acquire human posture data and monitoring data from flexible sensors;

[0047] Step S30: Input the human posture data into the gating network model to obtain the weight coefficients of each expert model;

[0048] Step S40: Input the monitoring data into each expert model to obtain the posture prediction result. Based on the weight coefficient and the posture prediction result, obtain the human posture of the current frame, and then complete the human motion tracking.

[0049] In step S10, the gated network model is used to output the weight coefficients of each expert model based on human posture data, and includes three fully connected layers.

[0050] When the human body is in different motion states, the weight coefficients of the influence of different flexible sensors on the corresponding joints are different. Therefore, the purpose of the gating network model is to dynamically change the weights of each expert model by providing different weight coefficients when the human body is in different motion states.

[0051] The expert model is used to output attitude prediction results based on the monitoring data of the flexible sensor. It includes one BN layer, four LSTM layers, and one fully connected layer. The fully connected layer is used as the input layer, the MSELoss function is used as the loss function, and the ReLU function is used as the activation function.

[0052] Step S20 specifically involves:

[0053] Human posture data is acquired through optical motion capture devices, and human monitoring data is acquired through flexible sensors;

[0054] The human posture data is recorded as y (t) y (t) It is a 3×M matrix; the monitoring data record is x (t) x (t) It is a vector of length S;

[0055] Where t represents the time step; S represents the number of flexible sensors; and M represents the number of joints in the human body that need to be predicted. The three degrees of freedom of each joint are the three Euler angles of rotation relative to the parent node.

[0056] Step S30 specifically involves:

[0057] The human pose data y from the previous frame (t-1)Input the gating network model and obtain the weight coefficients w(K) of each expert model; where K represents the expert model number. The number of expert models can be dynamically adjusted according to the number of movements to be covered.

[0058] Step S40 specifically involves:

[0059] The monitoring data from the previous frame is input into each expert model to obtain the pose prediction results. The pose prediction results are then weighted and averaged based on the weight coefficients to obtain the human pose of the current frame, thereby completing human motion tracking.

[0060] A preferred embodiment of the human motion tracking system based on regional saliency analysis of the present invention includes the following modules:

[0061] The model creation module is used to create a gating network model and several expert models based on a convolutional neural network, train each of the gating network models and expert models, and form a prediction network composed of the several expert models.

[0062] The human body data acquisition module is used to acquire human posture data and monitoring data from flexible sensors;

[0063] The weight coefficient calculation module is used to input the human posture data into the gated network model and obtain the weight coefficients of each expert model.

[0064] The motion tracking module is used to input the monitoring data into each expert model to obtain the posture prediction result, and obtain the human posture of the current frame based on the weight coefficients and the posture prediction result, thereby completing human motion tracking.

[0065] In the model creation module, the gated network model is used to output the weight coefficients of each expert model based on human posture data, and includes three fully connected layers.

[0066] When the human body is in different motion states, the weight coefficients of the influence of different flexible sensors on the corresponding joints are different. Therefore, the purpose of the gating network model is to dynamically change the weights of each expert model by providing different weight coefficients when the human body is in different motion states.

[0067] The expert model is used to output attitude prediction results based on the monitoring data of the flexible sensor. It includes one BN layer, four LSTM layers, and one fully connected layer. The fully connected layer is used as the input layer, the MSELoss function is used as the loss function, and the ReLU function is used as the activation function.

[0068] The human body data acquisition module is specifically as follows:

[0069] Human posture data is acquired through optical motion capture devices, and human monitoring data is acquired through flexible sensors;

[0070] The human posture data is recorded as y (t) y (t) It is a 3×M matrix; the monitoring data record is x (t) x (t) It is a vector of length S;

[0071] Where t represents the time step; S represents the number of flexible sensors; and M represents the number of joints in the human body that need to be predicted. The three degrees of freedom of each joint are the three Euler angles of rotation relative to the parent node.

[0072] The weighting coefficient calculation module is specifically as follows:

[0073] The human pose data y from the previous frame (t-1) Input the gating network model and obtain the weight coefficients w(K) of each expert model; where K represents the expert model number. The number of expert models can be dynamically adjusted according to the number of movements to be covered.

[0074] The motion tracking module is specifically:

[0075] The monitoring data from the previous frame is input into each expert model to obtain the pose prediction results. The pose prediction results are then weighted and averaged based on the weight coefficients to obtain the human pose of the current frame, thereby completing human motion tracking.

[0076] An embodiment of the human motion tracking invention is as follows:

[0077] The system uses 24 flexible sensors, strategically placed on the left and right shoulders, armpits, front and back of the left and right waists, elbows, abdomen, back, thighs, buttocks, knees, and groin, with three sensors on each shoulder. The system predicts 13 joints, including the left and right thighs, lower legs, three vertebrae, shoulders, upper arms, and forearms. Data is sampled 20 times per second. The gated network model consists of three fully connected layers, and there are eight expert models. Each expert model receives 10 input time intervals (data from the past 0.5 seconds). Each expert model contains one batch normalization (BN) layer, four LSTM layers, and one fully connected layer as its input. The MSELoss function is used as the loss function, and ReLU as the activation function. The average error per degree of freedom is approximately 10 degrees.

[0078] Prediction examples Figure 4 As shown, the left side shows the human posture predicted by the present invention, and the right side shows the actual human posture obtained by the optical motion capture device.

[0079] In summary, the advantages of this invention are as follows:

[0080] By creating and training a gated network model, the gated network model calculates the weight coefficients of each expert model based on human posture data. That is, it analyzes the changes in the prediction weights of different flexible sensors on joints under different movements based on regional saliency analysis to reduce the impact of noise. By setting multiple expert models to predict the monitoring data of flexible sensors, the implicit correlation between different joint angles of the human body can be learned, which ultimately greatly improves the accuracy of human motion tracking.

[0081] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A human motion tracking method based on regional saliency analysis, characterized in that: Includes the following steps: Step S10: Create a gated network model and several expert models based on a convolutional neural network, and train each of the gated network models and expert models; Step S20: Acquire human posture data and monitoring data from flexible sensors; Step S30: Input the human posture data into the gating network model and obtain the weight coefficients of each expert model; Step S40: Input the monitoring data into each expert model to obtain the pose prediction result, and obtain the human pose of the current frame based on the weight coefficient and the pose prediction result, thereby completing human motion tracking; In step S10, the gated network model is used to output the weight coefficients of each expert model based on human posture data, and includes three fully connected layers. The expert model is used to output attitude prediction results based on the monitoring data of the flexible sensor. It includes one BN layer, four LSTM layers, and one fully connected layer. The fully connected layer is used as the input layer, the MSELoss function is used as the loss function, and the ReLU function is used as the activation function. Step S20 specifically involves: Human posture data is acquired through optical motion capture devices, and human monitoring data is acquired through flexible sensors; The human posture data is recorded as follows: , for The matrix; the monitoring data records are , It is a length of ; in, Indicates the time step; Indicates the number of flexible sensors; This indicates the need to predict the number of joints in the human body; Step S30 specifically involves: The human pose data from the previous frame Input the gated network model and obtain the weight coefficients of each expert model. ;in, This indicates the number of the expert model.

2. The human motion tracking method based on regional saliency analysis as described in claim 1, characterized in that: Step S40 specifically involves: The monitoring data from the previous frame is input into each expert model to obtain the pose prediction results. The pose prediction results are then weighted and averaged based on the weight coefficients to obtain the human pose of the current frame, thereby completing human motion tracking.

3. A human motion tracking system based on regional saliency analysis, characterized in that: Includes the following modules: The model creation module is used to create a gated network model and several expert models based on a convolutional neural network, and to train each of the gated network models and expert models. The human body data acquisition module is used to acquire human posture data and monitoring data from flexible sensors; The weight coefficient calculation module is used to input the human posture data into the gated network model and obtain the weight coefficients of each expert model. The motion tracking module is used to input the monitoring data into each expert model to obtain the posture prediction result, and obtain the human posture of the current frame based on the weight coefficients and the posture prediction result, thereby completing human motion tracking. In the model creation module, the gated network model is used to output the weight coefficients of each expert model based on human posture data, and includes three fully connected layers. The expert model is used to output attitude prediction results based on the monitoring data of the flexible sensor. It includes one BN layer, four LSTM layers, and one fully connected layer. The fully connected layer is used as the input layer, the MSELoss function is used as the loss function, and the ReLU function is used as the activation function. The human body data acquisition module is specifically as follows: Human posture data is acquired through optical motion capture devices, and human monitoring data is acquired through flexible sensors; The human posture data is recorded as follows: , for The matrix; the monitoring data records are , It is a length of ; in, Indicates the time step; Indicates the number of flexible sensors; This indicates the need to predict the number of joints in the human body; The weighting coefficient calculation module is specifically as follows: The human pose data from the previous frame Input the gated network model and obtain the weight coefficients of each expert model. ;in, This indicates the number of the expert model.

4. The human motion tracking system based on regional saliency analysis as described in claim 3, characterized in that: The motion tracking module is specifically: The monitoring data from the previous frame is input into each expert model to obtain the pose prediction results. The pose prediction results are then weighted and averaged based on the weight coefficients to obtain the human pose of the current frame, thereby completing human motion tracking.

Images