Human body motion extraction method and related device based on flexible wearable device

Abstract

The present invention belongs to the technical field of human motion extraction and discloses a method and related apparatus for extracting whole-body human motion based on a flexible wearable device. The method comprises the following steps: obtaining sensor capacitance output data of a test subject in a preset scenario; using a trained motion tracking model to make predictions based on the obtained sensor capacitance output data to obtain three-dimensional data predictions of joint points; constructing a human kinematic model based on the obtained three-dimensional data predictions of joint points, and extracting kinematic parameters and gait spatiotemporal parameters based on the human kinematic model. The technical solution disclosed by the present invention can effectively solve the technical problems of existing linear fitting methods, such as low accuracy in extracting joint angles using flexible sensors and difficulty in applying to whole-body motion tracking.

Description

Technical Field

[0001] The present invention belongs to the technical field of human motion extraction, and in particular relates to a human body whole-body motion extraction method based on a flexible wearable device and a related device. Background Art

[0002] The data obtained by extracting gait parameters can provide important support for various scientific research and commercial fields such as gait analysis technology, robotic humanoid technology, biomechanics, and clinical rehabilitation.

[0003] Currently, the main methods for gait acquisition include motion capture systems, inertial units, flexible sensors, and pressure sensors. It is worth noting that flexible sensors can linearly sense the angles of human joints. Due to their elasticity and wearability, they can be sewn onto specific clothing as wearable devices to track specific body parts. Furthermore, they are relatively inexpensive and have no application restrictions. They do not require strict experimental conditions and can be worn in daily life, providing more realistic gait data. Therefore, flexible sensors are currently an important means of tracking human movement.

[0004] Flexible sensors often exhibit high nonlinearity and hysteresis during their response, and their noise shielding capabilities are weak, resulting in low data acquisition accuracy. Specifically, in practical applications, linear fitting methods are often used to convert sensor capacitance output into joint angles. However, due to the presence of sensor hysteresis and nonlinearity, the sensor capacitance output and joint angle are not strongly linearly correlated, resulting in low accuracy in extracting joint angles using linear fitting methods. Furthermore, for complex joints, multiple flexible sensors need to be placed at the joints. Currently, there is no unified standard for sensor placement, and placement is highly subjective and susceptible to environmental interference, further reducing the accuracy of linear fitting methods for extracting joint angles from complex joints. Furthermore, the joint radius of different joints in the human body is inconsistent, requiring measurement and calibration of all joints to obtain the linear coefficient for each joint. This cumbersome operation makes existing linear fitting methods difficult to use for whole-body motion tracking and extraction. Summary of the Invention

[0005] The present invention aims to provide a method and related apparatus for extracting whole-body human motion based on a flexible wearable device to address one or more of the aforementioned technical issues. The technical solution disclosed in this invention effectively addresses the low accuracy of joint angle extraction using flexible sensors in existing linear fitting methods and their difficulty in applying to the whole body.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for extracting whole-body motion of a human body based on a flexible wearable device, comprising the following steps:

[0008] Based on the dynamic capture system and the tester wearing the flexible wearable device, the sensor capacitance output data of the tester in the preset scenario is obtained;

[0009] Based on the obtained sensor capacitance output data, the trained motion tracking model is used to make predictions and obtain the predicted three-dimensional data values ​​of the joints.

[0010] Constructing a human kinematics model based on the obtained three-dimensional data prediction values ​​of the joint points, and extracting kinematic parameters and gait spatiotemporal parameters based on the human kinematics model;

[0011] in,

[0012] The motion tracking model includes: a sequence encoding network and a feature decoding network; wherein, the sequence encoding network includes a two-layer biLstm network and a fully connected network set between the two layers of biLstm networks, and the feature decoding network includes a seven-layer fully connected network; the sequence encoding network is used to input sensor capacitance output data and extract feature vectors, and the feature decoding network is used to input the feature vectors and map them to the three-dimensional data of the joint points; the feature vectors are spliced ​​by the hidden state of the second layer of biLstm network and the sensor capacitance output data at the current moment.

[0013] A further improvement of the present invention is that the training step of the trained motion tracking model includes:

[0014] Based on the dynamic capture system and the sample tester wearing the flexible wearable device, the three-dimensional data samples of the sample tester's body surface marker points and the sensor capacitance output data samples in the preset scene are collected;

[0015] The sensor capacitance output data samples are smoothed preprocessed using an adaptive smoothing method to obtain preprocessed sensor capacitance output data samples; based on the established human kinematics model, the three-dimensional data samples of the human body surface marker points are converted into three-dimensional joint point data of the human joint points using the joint center estimation method and adaptively smoothed to obtain preprocessed three-dimensional joint point data samples; the preprocessed sensor capacitance output data samples and the preprocessed three-dimensional joint point data samples are interpolated using the nearest neighbor interpolation method respectively to construct a training set;

[0016] Based on the constructed training set, the PSO optimization algorithm is used to update and optimize the model parameters of the motion tracking model to achieve the preset convergence conditions and obtain a trained motion tracking model; wherein, during the update optimization, the motion tracking model is used to map the sensor capacitance output data samples in the training set to the corresponding joint point three-dimensional data samples in the training set.

[0017] A further improvement of the present invention is that, in the step of updating and optimizing the model parameters of the motion tracking model based on the constructed training set and achieving a preset convergence condition to obtain a trained motion tracking model,

[0018] The updated and optimized model parameters include the number of hidden layers of the first layer biLstm network in the sequence encoding network, the number of hidden layers of the second layer biLstm network, the number of hidden layers of the fully connected network, and the number of hidden layers of the first six layers of the fully connected network in the feature decoding network, the number of model training rounds, and the initial training speed of the model.

[0019] A further improvement of the present invention is that the step of obtaining sensor capacitance output data of the tester in a preset scenario based on the motion capture system and the tester wearing the flexible wearable device includes:

[0020] Based on the dynamic capture system and the tester wearing the flexible wearable device, the initial sensor capacitance output data of the tester in a preset scenario is obtained;

[0021] The initial sensor capacitance output data is smoothed and preprocessed using an adaptive smoothing method to obtain the sensor capacitance output data of the tester in a preset scenario.

[0022] A further improvement of the present invention is that the human kinematic model is a human joint rigidity model with 14 body segments and 28 degrees of freedom constructed with 13 joint points as the core, including a head, a chest segment, an abdomen segment, two upper arm segments, two lower arm segments, a pelvis segment, two thigh segments, two calf segments and two foot segments.

[0023] A further improvement of the present invention is that the preset scenes include one or more of walking, running, upper limb stretching and lower limb leg lifting at different speeds.

[0024] A further improvement of the present invention is that in the step of extracting and obtaining kinematic parameters and gait spatiotemporal parameters based on the human kinematic model,

[0025] The kinematic parameters include one or both of the joint angle parameters and the joint motion trajectory;

[0026] The spatiotemporal parameters of gait include one or more of stride period, stance phase time proportion, stride length, stride frequency, stride width and relative symmetry index.

[0027] In a second aspect, the present invention provides a human body motion extraction system based on a flexible wearable device, comprising:

[0028] A data acquisition unit is used to acquire sensor capacitance output data of the tester in a preset scenario based on the dynamic capture system and the tester wearing the flexible wearable device;

[0029] A prediction value acquisition unit is used to make predictions based on the acquired sensor capacitance output data using a trained motion tracking model to obtain a predicted value of the three-dimensional data of the joint point;

[0030] a parameter extraction unit, configured to construct a human kinematics model based on the obtained three-dimensional data prediction values ​​of the joint points, and extract kinematic parameters and gait spatiotemporal parameters based on the human kinematics model;

[0031] in,

[0032] The motion tracking model includes: a sequence encoding network and a feature decoding network; wherein, the sequence encoding network includes a two-layer biLstm network and a fully connected network set between the two layers of biLstm networks, and the feature decoding network includes a seven-layer fully connected network; the sequence encoding network is used to input sensor capacitance output data and extract feature vectors, and the feature decoding network is used to input the feature vectors and map them to the three-dimensional data of the joint points; the feature vectors are spliced ​​by the hidden state of the second layer of biLstm network and the sensor capacitance output data at the current moment.

[0033] In a third aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for extracting whole-body motion of a human body based on a flexible wearable device as described in any one of the first aspects of the present invention is implemented.

[0034] In a fourth aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for extracting human body whole-body motion based on a flexible wearable device as described in any one of the first aspects of the present invention is implemented.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] In the method for extracting whole-body motion of a human body based on a flexible wearable device disclosed in the present invention, prediction is performed based on a motion tracking model established and trained using deep learning, and low-precision sensor capacitance output data of a flexible wearable device that can perform whole-body measurements is mapped to high-precision three-dimensional data of joint points to obtain predicted values ​​of the three-dimensional data of the joint points; a human kinematic model is established based on the predicted values ​​of the three-dimensional data of the joint points, and high-precision kinematic parameters and gait spatiotemporal parameters can be extracted based on the constructed human kinematic model, thereby reducing the difficulty of whole-body motion tracking extraction and effectively solving the technical problems of low accuracy in extracting joint angles using flexible sensors in existing linear fitting methods and difficulty in applying them to whole-body motion tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 This is a flow chart of a method for extracting whole-body motion of a human body based on a flexible wearable device in an embodiment of the present invention.

[0039] Figure 2 3 is a schematic diagram of the principle of a method for extracting human body motion based on a flexible wearable device in an embodiment of the present invention.

[0040] Figure 3 Schematic diagram of gait data collection in an embodiment of the present invention.

[0041] Figure 4 Schematic diagram of the position distribution of marker points on the human body in an embodiment of the present invention.

[0042] Figure 5 FIG. 4 is a schematic diagram of the specific position distribution of flexible sensors in a flexible wearable device in an embodiment of the present invention.

[0043] Figure 6 Schematic diagram of a sequence coding network structure in an embodiment of the present invention.

[0044] Figure 7 Schematic diagram of the biLstm unit structure and LSTM cell in an embodiment of the present invention.

[0045] Figure 8 Schematic diagram of a feature decoding network structure in an embodiment of the present invention.

[0046] Figure 9This is a schematic diagram of extracting five important joint angles of the upper and lower limbs in an embodiment of the present invention.

[0047] Figure 10 2 is a schematic diagram of separating gait cycles and extracting kinematic parameters based on joint angle characteristics in an embodiment of the present invention.

[0048] Figure 11 4 is a schematic diagram of a human body motion extraction system based on a flexible wearable device in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments and technical solutions are only part of the embodiments of the present invention, not all of the embodiments.

[0050] All other embodiments obtained by persons of ordinary skill in the art based on the technical solutions disclosed in the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0051] See also Figure 1 and Figure 2 In an embodiment of the present invention, a method for extracting whole-body motion of a human body based on a flexible wearable device is provided, comprising the following steps:

[0052] Step 1: Based on a motion capture system and a tester wearing a flexible wearable device, obtain sensor capacitance output data of the tester in a preset scenario;

[0053] Step 2: Based on the sensor capacitance output data obtained in step 1, the trained motion tracking model is used to perform predictions to obtain the predicted three-dimensional data values ​​of the joints.

[0054] Step 3: constructing a human kinematic model based on the three-dimensional data of the joints obtained in step 2, and extracting kinematic parameters and gait spatiotemporal parameters based on the human kinematic model;

[0055] in,

[0056] The motion tracking model includes: a sequence encoding network and a feature decoding network; wherein, the sequence encoding network includes a two-layer biLstm network (bidirectional long short-term memory network) and a fully connected network set between the two layers of biLstm networks, and the feature decoding network includes a seven-layer fully connected network; the sequence encoding network is used to input sensor capacitance output data and extract feature vectors, and the feature decoding network is used to input the feature vectors and map them to the three-dimensional data of the joint points; the feature vectors are spliced ​​by the hidden state of the second layer of biLstm network and the sensor capacitance output data at the current moment.

[0057] The technical solution disclosed in the embodiment of the present invention innovatively uses deep learning technology to build an advanced motion tracking model. The core advantage of this model is that it can effectively map the low-precision sensor capacitance output data collected by flexible wearable devices to high-precision three-dimensional data of joint points through complex algorithm processing; this conversion process greatly improves the accuracy and application value of the data, allowing data that was originally limited by sensor accuracy to be utilized in a higher dimension. Furthermore, after the model processes and predicts the data, a human kinematic model can be established; the established human kinematic model can not only provide high-precision kinematic parameters, such as joint angles, motion trajectories, etc., but also analyze and extract key information such as gait spatiotemporal parameters, such as walking speed, stride length, and gait cycle. This information is extremely valuable for understanding the human movement mechanism, evaluating sports performance, and conducting rehabilitation training. What is particularly important is that the technical solution of the embodiment of the present invention significantly reduces the difficulty of whole-body motion tracking and extraction; explanatory speaking, the existing traditional linear fitting methods often have low accuracy when extracting joint angles and are difficult to be effectively applied to whole-body motion tracking; in contrast, the deep learning model of the embodiment of the present invention effectively solves this technical problem through its powerful nonlinear mapping capability, bringing breakthrough progress to the field of whole-body motion tracking, not only improving the accuracy and reliability of motion data analysis, but also laying a solid foundation for the widespread application of flexible wearable devices in sports science, medical rehabilitation, and human-computer interaction.

[0058] In one embodiment of the present invention, the motion tracking model optimizes model parameters by using a PSO optimization algorithm (Particle Swarm Optimization algorithm).

[0059] See also Figures 2 to 5 In a specific embodiment of the present invention, the training acquisition step of the trained motion tracking model includes:

[0060] S1. Under the dynamic capture system, the tester wears a flexible wearable device and synchronously collects the three-dimensional data of the human body surface marker points and the 24-channel sensor capacitance output data in preset scenes such as walking, running, upper limb extension, and lower limb leg lifting at different speeds on the treadmill; for example, using Figure 3 The data collection process shown in the figure is as follows: the tester wears a flexible wearable device with 36 markers attached to corresponding positions on the body. Under the motion capture system, the tester completes 24 sets of walking motions at 1.0m / s, 1.5m / s, and 2.0m / s, and running motions at 2.0m / s, 3.0m / s, and 4.0m / s on a treadmill. Figure 4 Shows the distribution of marker points throughout the body. Figure 5 The specific location distribution of sensors in the flexible wearable device is shown. For further explanation, the flexible sensor used is the ESSA sensor, the data acquisition system has an acquisition frequency of 20 Hz, the data of the flexible wearable device is collected through the serial port, and the three-dimensional data of the dynamic capture system markers is collected using QTM software.

[0061] S2. Establish a human kinematic model with 13 joints as the core. Explanatoryally, a 14-segment, 28-degree-of-freedom human joint rigid model is constructed starting from the 13 joints. The model is simplified by the head, chest and abdomen segments, two upper and lower arm segments, a pelvic segment, and two thigh, calf, and foot segments.

[0062] S3. Preprocess the two types of data collected in step S1 by smoothing and interpolating the data to obtain preprocessed training set data. Specifically, for example, the sensor capacitance output data is preprocessed by smoothing the data using an adaptive smoothing method. The three-dimensional data of the human body surface markers is converted into three-dimensional joint data of 13 human joints by estimating the human joint centers and adaptively smoothed. Both types of data are interpolated using a nearest neighbor interpolation method to reduce or eliminate the problem of asynchronous acquisition time.

[0063] S4. Build a motion tracking model, which consists of a sequence encoding network and a feature decoding network, and use the PSO optimization algorithm to optimize the model parameters; it is used to realize the mapping of the 24-channel sensor capacitance output data of the flexible device to 13 high-precision three-dimensional data of joint points; explanatory, the sequence encoding network adopts a two-layer biLstm network, and a fully connected network is set between the two layers of biLstm; the feature decoding network consists of a seven-layer fully connected network; in addition, when optimizing the model parameters of the motion tracking model by the PSO optimization algorithm, specifically eleven parameters of the model are optimized, namely: the number of hidden layers of the first biLstm layer of the sequence encoding network, the number of hidden layers of the second biLstm layer, the number of hidden layers of the fully connected layer between the two biLstm layers, the number of hidden layers of the first six fully connected layers of the feature decoding network, the number of model training rounds, and the initial training speed of the model;

[0064] S5. Use the training set data obtained in step S3 to train and test the motion tracking model. During the testing process, the predicted data output by the model is used to construct a human kinematic model. The human kinematic model is used to extract kinematic parameters and gait spatiotemporal parameters. When the training set is input into the motion tracking model, its data flow is as follows: 24 sensor capacitance output data is input into the sequence encoding network to extract feature vectors. The feature vectors are composed of the second-layer BiLSTM hidden state and the sensor capacitance output at the current moment. The feature vectors are input into the feature decoding network and ultimately mapped to high-precision three-dimensional data of the joint points. Further explaining, during the testing process, the sensor capacitance output data is input into the trained motion tracking model. The output model predictions are represented as three-dimensional data of 13 joint points, which are used to subsequently construct the human kinematic model. Kinematic parameters that can be extracted from the human kinematic model include stride period, stance phase time ratio, stride length, cadence, stride width, relative symmetry index, etc. Gait spatiotemporal parameters include joint angle parameters of the 13 joints and joint motion trajectory.

[0065] The embodiment of the present invention specifically discloses a method for accurately extracting human whole-body motion based on a flexible wearable device, which can be applied to the field of robotics; the embodiment of the present invention proposes a motion tracking model based on deep learning, which is composed of a sequence encoding network and a feature decoding network, and uses a PSO algorithm to optimize the model parameters. It can map 24-channel whole-body low-precision sensor data of the flexible wearable device to high-precision joint point data, and further establish a human kinematic model based on the model output. High-precision kinematic parameters and gait spatiotemporal parameters can be extracted from the predicted human kinematic model, solving the problems of low extraction accuracy and difficulty in whole-body motion tracking in the prior art.

[0066] In the solution provided by the embodiment of the present invention, data preprocessing is performed on the two types of collected raw data. Among them, sensor data is easily affected by environmental interference and mechanical stress, which may cause abnormal values ​​in the capacitance output. Therefore, the present invention uses an adaptive smoothing method to smooth the preprocessed data. The three-dimensional data of the human body surface marker points is converted into three-dimensional data of 13 joint points in the human body through estimation of the human joint center and adaptively smoothed. The two types of data are interpolated using the nearest neighbor interpolation method to reduce or eliminate the problem of acquisition asynchrony.

[0067] For example, after data processing, the sensor output capacitance data x at each time step t can be obtained. (t) Expressed as:

[0068]

[0069] Where, Indicates the output capacitance data of the 24th sensor;

[0070] The 3D data y of 13 joint points of the dynamic capture system at each time step t (t) Expands to:

[0071]

[0072] Where, Represents the 3D data of the 39th joint point;

[0073] The sensor time series data from (tn) to (t) is selected to establish a mapping relationship with the three-dimensional data of the joint point at time (t), and the input and output of the model are x (t-n:t) with y (t) :

[0074] x (t-n:t) =[x (t-n) ,x (t-n+1) ,x (t-n+2) ,…,x (t) ];

[0075] Where n is the number of time intervals.

[0076] Specifically, a total of 24 data collections were conducted to construct the data sets required for model training and testing.

[0077] In a specific embodiment of the present invention, a motion tracking model is constructed by a deep learning algorithm to achieve the goal of converting low-precision x (t -n:t) Regression mapping high precision y (t) To achieve this goal, it is necessary to select a suitable time series processing network to extract x (t-n:t)The multi-scale features are then mapped to y through a regression network (t) In the technical solution disclosed in the embodiment of the present invention, the motion tracking model consists of a sequence encoding network and a feature decoding network. Figure 2 The overall structure of the motion tracking model and the data transmission flow are shown.

[0078] During the training process, input time series data x (t-n:t) After being processed by the sequence encoding network, the feature vector r(t) is extracted and input into the feature decoding network to be mapped to the high-precision y (t) In order to achieve the best effect of the network, the PSO algorithm is used to optimize the network layer parameters and training parameters. (t -n:t) Input motion tracking model, and output high-precision prediction data after network mapping pass A human kinematics model is further established for use in parameter extraction and analysis applications.

[0079] Explanatory principle: during the data collection process, sensors are susceptible to environmental interference and mechanical stress, and the output electricity is prone to abnormal values, missing values, and noise; at the same time, when the tester collects data, there are no environmental restrictions, and the environment and other influences are uncontrollable, so the data collection accuracy is very low. This requires the sequence encoding network to select a suitable sub-network to fully extract features from low-precision data. Finally, the present invention selects the biLstm network to build a sequence encoding network. Its "gate" structure can effectively avoid ladder explosion and gradient disappearance, and has the characteristics of long-term memory. The biLstm consists of two independent LSTMs, which process the input sequence from the forward and backward directions, and then splice the hidden states in the two directions. The bidirectional structure improves the model's ability to model time series data, and at the same time extracts features more comprehensively and accurately. Figure 6 Its specific structure is shown.

[0080] In a specific embodiment of the present invention, the sequence encoding network is composed of a double-layer biLstm, which absorbs the features of the first layer and transforms the hidden state h of the second layer biLstm(t) into (t) With sensor data x (t) Spliced ​​into a feature vector r(t)∈R S , as the output of the sequence processing network. Figure 7 Take forward LSTM as an example to analyze its data processing method, use σ to represent the sigmoid activation function, and tanh to represent the hyperbolic tangent function. represents the addition operation of elements, Represents the multiplication operation of elements. Before the LSTM unit is processed, it is necessary to obtain the sensor input x at the current time (t) (t) ∈R 24 , cell state at time (t-1) and hidden state k is the dimension of hidden state and cell state.

[0081] The LSTM unit controls the cell state through three gates:

[0082] First, the forget gate uses the forget gate unit to measure the information of the hidden state and data input. Selectively forget the information of the cell state memory at the previous moment. The formula is as follows:

[0083]

[0084] Where W∈R k×s and b∈R k are the cycle weight and bias respectively.

[0085] The input gate can obtain the cell state through the input time series data and the hidden state of the previous moment Through the input gate unit Select valid information, and then add new memory to the cell state through calculation to update the cell state and obtain the cell state at that moment The formula is as follows:

[0086]

[0087]

[0088]

[0089] The output gate determines the hidden state h at that moment by the new cell state output (t) , the formula is as follows:

[0090]

[0091]

[0092] The hidden states of the two LSTM units in the forward and reverse directions are concatenated to form the hidden state of the current biLstm unit. The formula is as follows:

[0093]

[0094] In the embodiment of the present invention, the feature decoding network is composed of seven fully connected layers. The feature decoding network needs to regress the feature vector r(t) obtained after the time encoding network processing to the three-dimensional data y of the human joint point.(t) Because the input-output relationship of the feature decoding network is nonlinear and has a relatively complex mapping relationship, the fully connected layer can approximate any continuous function by increasing the number of hidden layers and units, which can meet the functional requirements of the feature decoding network. At the same time, it is simple and easy to operate, making it the first choice for constructing feature mapping networks. Figure 8 The structure of the feature mapping network is shown, which consists of seven fully connected layers, used to establish the mapping relationship between the feature vector and the three-dimensional coordinates of the joint center, which can realize the y (t) Each layer is modeled according to the following formula:

[0095] f (i) (r i ;w i ,b i )=r i T w i +b i

[0096] Among them, r i is the input vector of layer i, w i 、b i are weights and biases respectively;

[0097] Except for the last layer output which does not require an activation function, the rest use the linear rectifier unit RELU as the activation function. The formula is as follows:

[0098] ReLU=max{0,r}

[0099] Therefore, the input of the next layer can be determined as:

[0100] r i+1 =max{0,r i T w i +b i}

[0101] During the training process, the output of the last layer is set to y (t) ∈R 39 , used to train the model's weight parameters. During the test, the last layer of the model will output the predicted value Track the 3D coordinates of 13 human joints to build a human kinematic model.

[0102] In the embodiment of the present invention, considering the characteristics of the PSO optimization algorithm, such as multi-dimensional optimization, low sensitivity to initial parameters, and stable convergence to the optimal solution, it is used to optimize the parameters of the motion tracking model to give full play to the network effect. Specifically, the optimization parameters selected are the number of hidden layers in each network layer, the number of training rounds, and the initial training speed. That is, the dimension of PSO optimization is 11, and the goal of PSO optimization is set to find a set of parameter values ​​so that the model predicts the output value y (t) and the true value The error value is minimal.

[0103] PSO uses an initial swarm of random particles and then iterates to find the optimal solution. During each iteration, it continuously optimizes by tracking the optimal position of the particle swarm. Table 1 defines the basic parameter values ​​for PSO optimization. The values ​​are selected based on existing research experience.

[0104] Table 1. PSO basic parameter values

[0105]

[0106] Suppose there are 15 particles in a region of dimension 11, where the position x of the i-th particle is i With speed v i for:

[0107] x i ={x i1 ,x i2 ,x i3 ,…,x i11}

[0108] v i ={v i1 ,v i2 ,v i3 ,…,v i11}

[0109] In each iteration, the particle is updated by tracking two optimal solutions. The first one is the optimal solution found by the particle itself, that is, the individual extreme value p i ={p i1 ,p i2 ,p i3 ,…,p i11}, the other is the optimal solution found by the entire population so far, that is, the global optimal solution p g ={p gbest1 ,p gbest2 ,p gbest3 ,…,p gbest11}, set the optimization function so that when the parameters of the gait tracking network obtain the global optimal solution, the root mean square error value of the model training obtains the minimum value.

[0110] At the kth iteration, the speed and position of the i-th particle are updated as follows:

[0111]

[0112]

[0113] Where r1 and r2 are uniform random numbers in the range [0,1].

[0114] The training data set is used for model training. After model training, the model parameter values ​​after PSO optimization can be obtained, as shown in Table 2.

[0115] Table 2. PSO optimized network parameter values

[0116]

[0117] See also Figure 9 and Figure 10 During the test, the sensor data is input into the network to obtain the predicted output value At the same time, the dynamic capture system obtains high-precision reference data (t) . According to the predicted output value Can establish human kinematics model and extract gait parameters, Figure 9 A method for extracting five important joint angles of the upper and lower limbs is provided; Figure 10 A method is provided to separate gait cycles based on joint angle characteristics and further extract kinematic parameters.

[0118] In a further preferred embodiment, the present invention formulates three types of standards to evaluate the effectiveness of model training, including:

[0119] 1) Evaluate y (t) and gaps, including:

[0120] The root mean square error (RMSE), mean absolute error (MAE), correlation coefficient (r), and relative error (RE) are four indicators that reflect the degree of deviation and direct gap between the true value and the predicted value. The relative error is introduced and the data magnitude is taken into consideration. The calculation method is:

[0121]

[0122] Among them, q is the predicted output y (t) dimension.

[0123] 2) Evaluate the gap between the prediction and the reference joint point, including:

[0124] based on with y (t) Calculate the predicted joint point at time t With reference joint The Euclidean distance difference of the relative distance RD relative to the tester's height (d = 1615mm) is calculated as follows:

[0125]

[0126] Where i∈(1,13).

[0127] 3) Evaluate the accuracy of the model in extracting gait parameters, including:

[0128] The ability of the model to extract kinematic parameters is evaluated using relative error. For joint angle parameters, the extraction method of the present invention is compared with the existing linear fitting method. The goal of linear regression is to find the coefficient W∈R by linear fitting. j×1 The tracked joint angle θ is consistent with the sensor capacitance output X j×1 The predicted value has the minimum residual value XW, and its formula is:

[0129]

[0130] Among them, get W∈R j×1 The dimension j and X j×1 Data composition.

[0131] The following are device embodiments of the present invention, which can be used to perform the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0132] See also Figure 11 In an embodiment of the present invention, a human body motion extraction system based on a flexible wearable device is provided, comprising:

[0133] A data acquisition unit is used to acquire sensor capacitance output data of the tester in a preset scenario based on the dynamic capture system and the tester wearing the flexible wearable device;

[0134] A prediction value acquisition unit is used to make predictions based on the acquired sensor capacitance output data using a trained motion tracking model to obtain a predicted value of the three-dimensional data of the joint point;

[0135] a parameter extraction unit, configured to construct a human kinematics model based on the obtained three-dimensional data prediction values ​​of the joint points, and extract kinematic parameters and gait spatiotemporal parameters based on the human kinematics model;

[0136] in,

[0137] The motion tracking model includes: a sequence encoding network and a feature decoding network; wherein, the sequence encoding network includes a two-layer biLstm network and a fully connected network set between the two layers of biLstm networks, and the feature decoding network includes a seven-layer fully connected network; the sequence encoding network is used to input sensor capacitance output data and extract feature vectors, and the feature decoding network is used to input the feature vectors and map them to the three-dimensional data of the joint points; the feature vectors are spliced ​​by the hidden state of the second layer of biLstm network and the sensor capacitance output data at the current moment.

[0138] In one embodiment of the present invention, a computer device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used to perform the operation of the human body whole-body motion extraction method based on a flexible wearable device.

[0139] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory). The computer-readable storage medium is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM (Random Access Memory) memory or a non-volatile memory (non-volatile memory), such as at least one disk storage. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the method for extracting human body motion based on a flexible wearable device in the above embodiment.

[0140] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, optical storage, etc.) that contain computer-usable program code.

[0141] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0142] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A method for extracting human body motion based on flexible wearable devices, characterized in that: The following steps are involved: Based on the dynamic capture system and the tester wearing the flexible wearable device, the sensor capacitance output data of the tester in the preset scenario is obtained; Based on the obtained sensor capacitance output data, the trained motion tracking model is used to make predictions and obtain the predicted three-dimensional data values ​​of the joints. Constructing a human kinematics model based on the obtained three-dimensional data prediction values ​​of the joint points, and extracting kinematic parameters and gait spatiotemporal parameters based on the human kinematics model; in, The motion tracking model includes: a sequence encoding network and a feature decoding network; wherein, the sequence encoding network includes a two-layer biLstm network and a fully connected network set between the two-layer biLstm network, and the feature decoding network includes a seven-layer fully connected network; the sequence encoding network is used to input sensor capacitance output data and extract feature vectors, and the feature decoding network is used to input the feature vectors and map them to the three-dimensional data of the joint points; the feature vectors are spliced ​​by the hidden state of the second-layer biLstm network and the sensor capacitance output data at the current moment.

2. A method for extracting human body motion based on a flexible wearable device according to claim 1, characterized in that: The training steps of the trained motion tracking model include: Based on the dynamic capture system and the sample tester wearing the flexible wearable device, the three-dimensional data samples of the sample tester's body surface marker points and the sensor capacitance output data samples in the preset scene are collected; The sensor capacitance output data samples are smoothed preprocessed using an adaptive smoothing method to obtain preprocessed sensor capacitance output data samples; based on the established human kinematics model, the three-dimensional data samples of the human body surface marker points are converted into three-dimensional joint point data of the human joint points using the joint center estimation method and adaptively smoothed to obtain preprocessed three-dimensional joint point data samples; the preprocessed sensor capacitance output data samples and the preprocessed three-dimensional joint point data samples are interpolated using the nearest neighbor interpolation method respectively to construct a training set; Based on the constructed training set, the PSO optimization algorithm is used to update and optimize the model parameters of the motion tracking model to achieve the preset convergence conditions and obtain a trained motion tracking model; wherein, during the update optimization, the motion tracking model is used to map the sensor capacitance output data samples in the training set to the corresponding joint point three-dimensional data samples in the training set.

3. The method for extracting human body motion based on a flexible wearable device according to claim 2, characterized in that: In the step of updating and optimizing the model parameters of the motion tracking model using the PSO optimization algorithm based on the constructed training set to achieve a preset convergence condition and obtain a trained motion tracking model, The updated and optimized model parameters include the number of hidden layers of the first layer biLstm network in the sequence encoding network, the number of hidden layers of the second layer biLstm network, the number of hidden layers of the fully connected network, and the number of hidden layers of the first six layers of the fully connected network in the feature decoding network, the number of model training rounds, and the initial training speed of the model.

4. The method for extracting human body motion based on a flexible wearable device according to claim 1, characterized in that: The step of obtaining sensor capacitance output data of the tester in a preset scenario based on the dynamic capture system and the tester wearing the flexible wearable device includes: Based on the dynamic capture system and the tester wearing the flexible wearable device, the initial sensor capacitance output data of the tester in a preset scenario is obtained; The initial sensor capacitance output data is smoothed and preprocessed using an adaptive smoothing method to obtain the sensor capacitance output data of the tester in a preset scenario.

5. The method for extracting human body motion based on a flexible wearable device according to claim 1, characterized in that: The human kinematic model is a human joint rigid model with 14 body segments and 28 degrees of freedom constructed with 13 joint points as the core, including a head, a chest segment, an abdomen segment, two upper arm segments, two lower arm segments, a pelvis segment, two thigh segments, two calf segments and two foot segments.

6. The method for extracting human body motion based on a flexible wearable device according to claim 1, characterized in that: The preset scenes include one or more of walking at different speeds, running, upper limb stretching, and lower limb leg lifting.

7. The method for extracting human body motion based on a flexible wearable device according to claim 1, characterized in that: In the step of extracting and obtaining kinematic parameters and gait spatiotemporal parameters based on the human kinematic model, The kinematic parameters include one or both of the joint angle parameters and the joint motion trajectory; The spatiotemporal parameters of gait include one or more of stride period, stance phase time proportion, stride length, stride frequency, stride width and relative symmetry index.

8. A human body motion extraction system based on flexible wearable devices, characterized in that: include: A data acquisition unit is used to acquire sensor capacitance output data of the tester in a preset scenario based on the dynamic capture system and the tester wearing the flexible wearable device; A prediction value acquisition unit is used to make predictions based on the acquired sensor capacitance output data using a trained motion tracking model to obtain a predicted value of the three-dimensional data of the joint point; a parameter extraction unit, configured to construct a human kinematics model based on the obtained three-dimensional data prediction values ​​of the joint points, and extract kinematic parameters and gait spatiotemporal parameters based on the human kinematics model; in, The motion tracking model includes: a sequence encoding network and a feature decoding network; wherein, the sequence encoding network includes a two-layer biLstm network and a fully connected network set between the two-layer biLstm network, and the feature decoding network includes a seven-layer fully connected network; the sequence encoding network is used to input sensor capacitance output data and extract feature vectors, and the feature decoding network is used to input the feature vectors and map them to the three-dimensional data of the joint points; the feature vectors are spliced ​​by the hidden state of the second-layer biLstm network and the sensor capacitance output data at the current moment.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for extracting human body whole-body motion based on a flexible wearable device according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for extracting human body whole-body motion based on a flexible wearable device according to any one of claims 1 to 7 is implemented.

Images