Human body whole body motion extraction method based on flexible wearable device and related device

Abstract

The invention belongs to the technical field of human body motion extraction, and discloses a human body whole body motion extraction method based on flexible wearable equipment and a related device. The human body whole body motion extraction method based on the flexible wearable device comprises the following steps: acquiring sensor capacitance output data of a tester in a preset scene; based on the acquired sensor capacitance output data, utilizing the trained motion tracking model to carry out prediction to obtain a joint point three-dimensional data prediction value; a human body kinematics model is constructed and obtained based on the obtained joint point three-dimensional data predicted value, and kinematics parameters and gait space-time parameters are extracted and obtained based on the human body kinematics model. The technical scheme disclosed by the invention can effectively solve the technical problems that the precision of extracting the joint angle by using a flexible sensor in the existing linear fitting method is low and the existing linear fitting method is difficult to be applied to whole-body motion tracking.

Description

Technical Field

[0001] The present invention belongs to the technical field of human body motion extraction, and in particular relates to a human body whole body motion extraction method based on a flexible wearable device and a related device. Background Art

[0002] The data obtained by extracting gait parameters can provide important support for various scientific research and commercial fields such as gait analysis technology, robotic humanoid technology, biomechanics, and clinical rehabilitation.

[0003] At present, the main methods of gait collection include dynamic capture systems, inertial units, flexible sensors, pressure sensors, etc. It is worth noting that flexible sensors can linearly sense the angles of human joints, and because of their elasticity and wearability, they can be sewn on specific clothes as wearable devices to track specific parts; at the same time, their prices are relatively low and there are no restrictions on application scenarios. There is no need to set strict experimental conditions, and they can also be worn in daily life, which can provide more realistic gait collection data; therefore, flexible sensors are currently an important means of tracking human movement.

[0004] Flexible sensors often have high nonlinearity and hysteresis in the response process, and their ability to shield noise is also weak, resulting in low data collection accuracy. Explanatory, in actual applications, linear fitting methods are often used to convert sensor capacitance output into joint angles. However, due to the existence of sensor hysteresis nonlinearity, the sensor capacitance output and joint angles are not strongly linearly correlated. At this time, the accuracy of extracting joint angles by linear fitting methods is low. In addition, for complex joints, multiple flexible sensors need to be arranged at the joints. Currently, there is no unified standard for the sensor arrangement position, and the arrangement position is highly subjective and easily affected by environmental interference, which further reduces the accuracy of extracting joint angles of complex joints by linear fitting methods. In addition, as far as the joints of the human body are concerned, the joint radii of different joints are inconsistent, and it is necessary to measure and calibrate the joints of the whole body to obtain the linear coefficient of each joint. The operation is cumbersome, and the existing linear fitting method is difficult to extract whole-body motion tracking. Summary of the invention

[0005] The purpose of the present invention is to provide a method and related device for extracting human body motion based on a flexible wearable device to solve one or more of the above-mentioned technical problems. The technical solution disclosed in the present invention can effectively solve the technical problems that the existing linear fitting method uses a flexible sensor to extract joint angles with low accuracy and is difficult to apply to the whole body.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for extracting human body motion based on a flexible wearable device, comprising the following steps:

[0008] Based on the dynamic capture system and the tester wearing the flexible wearable device, the sensor capacitance output data of the tester in the preset scenario is obtained;

[0009] Based on the obtained sensor capacitance output data, the trained motion tracking model is used for prediction to obtain the three-dimensional data prediction value of the joint point;

[0010] Constructing a human kinematics model based on the obtained three-dimensional data prediction values ​​of the joint points, and extracting kinematic parameters and gait spatiotemporal parameters based on the human kinematics model;

[0011] in,

[0012] The motion tracking model includes: a sequence encoding network and a feature decoding network; wherein the sequence encoding network includes two layers of biLstm networks and a fully connected network arranged between the two layers of biLstm networks, and the feature decoding network includes a seven-layer fully connected network; the sequence encoding network is used to input sensor capacitance output data and extract feature vectors, and the feature decoding network is used to input the feature vectors and map them to the three-dimensional data of the joint points; the feature vector is spliced ​​by the hidden state of the second layer of biLstm network and the sensor capacitance output data at the current moment.

[0013] A further improvement of the present invention is that the training step of the trained motion tracking model comprises:

[0014] Based on the dynamic capture system and the sample tester wearing the flexible wearable device, the three-dimensional data samples of the human body surface marker points and the sensor capacitance output data samples of the sample tester in the preset scene are collected;

[0015] The sensor capacitance output data samples are smoothed preprocessed by an adaptive smoothing method to obtain preprocessed sensor capacitance output data samples; based on the established human kinematics model, the three-dimensional data samples of the human body surface marker points are converted into three-dimensional data of the joint points of the human body joints by using the joint center estimation method and adaptively smoothed to obtain preprocessed three-dimensional data samples of the joint points; the preprocessed sensor capacitance output data samples and the preprocessed three-dimensional data samples of the joint points are interpolated by the nearest neighbor interpolation method respectively to construct a training set;

[0016] Based on the constructed training set, the PSO optimization algorithm is used to update and optimize the model parameters of the motion tracking model to achieve the preset convergence conditions and obtain a trained motion tracking model; wherein, during the update optimization, the motion tracking model is used to map the sensor capacitance output data samples in the training set to the corresponding joint point three-dimensional data samples in the training set.

[0017] A further improvement of the present invention is that, in the step of updating and optimizing the model parameters of the motion tracking model based on the constructed training set using the PSO optimization algorithm to achieve a preset convergence condition and obtain a trained motion tracking model,

[0018] The updated and optimized model parameters include the number of hidden layers of the first layer biLstm network in the sequence encoding network, the number of hidden layers of the second layer biLstm network, the number of hidden layers of the fully connected network, and the number of hidden layers of the first six layers of the fully connected network in the feature decoding network, the number of model training rounds, and the initial training speed of the model.

[0019] A further improvement of the present invention is that the step of obtaining the sensor capacitance output data of the tester in a preset scenario based on the dynamic capture system and the tester wearing the flexible wearable device includes:

[0020] Based on the dynamic capture system and the tester wearing the flexible wearable device, the initial sensor capacitance output data of the tester in a preset scenario is obtained;

[0021] The initial sensor capacitance output data is smoothed and preprocessed using an adaptive smoothing method to obtain the sensor capacitance output data of the tester in a preset scenario.

[0022] A further improvement of the present invention is that the human kinematic model is a human joint rigid model with 14 body segments and 28 degrees of freedom constructed with 13 joint points as the core, including a head, a chest segment, an abdomen segment, two upper arm segments, two lower arm segments, a pelvis segment, two thigh segments, two calf segments and two foot segments.

[0023] A further improvement of the present invention is that the preset scenes include one or more of walking, running, upper limb stretching and lower limb leg raising at different speeds.

[0024] A further improvement of the present invention is that in the step of extracting and obtaining kinematic parameters and gait spatiotemporal parameters based on the human kinematic model,

[0025] The kinematic parameters include one or both of the joint angle parameters and the joint motion trajectory;

[0026] The spatiotemporal parameters of gait include one or more of stride period, stance phase time proportion, stride length, stride frequency, step width and relative symmetry index.

[0027] In a second aspect, the present invention provides a human body motion extraction system based on a flexible wearable device, comprising:

[0028] A data acquisition unit, used to acquire sensor capacitance output data of the tester in a preset scenario based on a dynamic capture system and a tester wearing the flexible wearable device;

[0029] A prediction value acquisition unit, used to make predictions based on the acquired sensor capacitance output data using a trained motion tracking model to obtain a prediction value of the three-dimensional data of the joint point;

[0030] A parameter extraction unit, used to construct a human kinematics model based on the obtained three-dimensional data prediction values ​​of the joint points, and extract kinematic parameters and gait spatiotemporal parameters based on the human kinematics model;

[0031] in,

[0032] The motion tracking model includes: a sequence encoding network and a feature decoding network; wherein the sequence encoding network includes two layers of biLstm networks and a fully connected network arranged between the two layers of biLstm networks, and the feature decoding network includes a seven-layer fully connected network; the sequence encoding network is used to input sensor capacitance output data and extract feature vectors, and the feature decoding network is used to input the feature vectors and map them to the three-dimensional data of the joint points; the feature vector is spliced ​​by the hidden state of the second layer of biLstm network and the sensor capacitance output data at the current moment.

[0033] In a third aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for extracting whole-body motion of a human body based on a flexible wearable device as described in any one of the first aspect of the present invention is implemented.

[0034] In a fourth aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for extracting human body motion based on a flexible wearable device as described in any one of the first aspects of the present invention is implemented.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] In the method for extracting whole-body motion of a human body based on a flexible wearable device disclosed in the present invention, prediction is performed based on a motion tracking model established and trained using deep learning, and low-precision sensor capacitance output data of a flexible wearable device that can perform whole-body measurement is mapped to high-precision three-dimensional data of joint points to obtain predicted values ​​of the three-dimensional data of the joint points; a human kinematic model is established through the predicted values ​​of the three-dimensional data of the joint points, and high-precision kinematic parameters and gait spatiotemporal parameters can be extracted based on the constructed human kinematic model, thereby reducing the difficulty of whole-body motion tracking extraction, and effectively solving the technical problems of low accuracy in extracting joint angles using flexible sensors in existing linear fitting methods and difficulty in applying to whole-body motion tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 It is a flow chart of a method for extracting whole-body motion of a human body based on a flexible wearable device in an embodiment of the present invention.

[0039] Figure 2 Schematic diagram of the principle of a method for extracting human body motion based on a flexible wearable device in an embodiment of the present invention.

[0040] Figure 3 It is a schematic diagram of gait data collection in an embodiment of the present invention.

[0041] Figure 4 Schematic diagram of the position distribution of marker points on the human body in an embodiment of the present invention.

[0042] Figure 5 Schematic diagram of specific position distribution of flexible sensors in a flexible wearable device in an embodiment of the present invention.

[0043] Figure 6 It is a schematic diagram of a sequence coding network structure in an embodiment of the present invention.

[0044] Figure 7 Schematic diagram of biLstm unit structure and LSTM cell in an embodiment of the present invention.

[0045] Figure 8 It is a schematic diagram of a feature decoding network structure in an embodiment of the present invention.

[0046] Fig. 9It is a schematic diagram of extracting five important joint angles of upper and lower limbs in an embodiment of the present invention.

[0047] Fig.10 It is a schematic diagram of separating gait cycles and extracting kinematic parameters based on joint angle characteristics in an embodiment of the present invention.

[0048] Fig.11 4 is a schematic diagram of a human body motion extraction system based on a flexible wearable device in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments and technical solutions are only part of the embodiments of the present invention, not all of the embodiments.

[0050] All other embodiments obtained by those of ordinary skill in the art without creative work based on the technical solutions disclosed in the embodiments of the present invention belong to the scope of protection of the present invention. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0051] See also Figure 1 and Figure 2 In an embodiment of the present invention, a method for extracting human body motion based on a flexible wearable device is provided, comprising the following steps:

[0052] Step 1, based on a dynamic capture system and a tester wearing a flexible wearable device, obtaining sensor capacitance output data of the tester in a preset scenario;

[0053] Step 2: Based on the sensor capacitance output data obtained in step 1, a trained motion tracking model is used to make predictions to obtain the three-dimensional data prediction value of the joint point;

[0054] Step 3, constructing a human kinematics model based on the three-dimensional data of the joint points obtained in step 2, and extracting kinematic parameters and gait spatiotemporal parameters based on the human kinematics model;

[0055] in,

[0056] The motion tracking model includes: a sequence encoding network and a feature decoding network; wherein the sequence encoding network includes two layers of biLstm networks (bidirectional long short-term memory network, Bidirectional Long Short-Term Memory) and a fully connected network set between the two layers of biLstm networks, and the feature decoding network includes seven layers of fully connected networks; the sequence encoding network is used to input sensor capacitance output data and extract feature vectors, and the feature decoding network is used to input the feature vectors and map them to the three-dimensional data of the joint points; the feature vector is spliced ​​by the hidden state of the second layer of biLstm network and the sensor capacitance output data at the current moment.

[0057] The technical solution disclosed in the embodiment of the present invention innovatively uses deep learning technology to build an advanced motion tracking model. The core advantage of this model is that it can effectively map the low-precision sensor capacitance output data collected by the flexible wearable device to high-precision three-dimensional data of the joint points through complex algorithm processing; this conversion process greatly improves the accuracy and application value of the data, allowing the data originally limited by the accuracy of the sensor to be used in a higher dimension. Furthermore, after the model processes and predicts the data, a human kinematic model can be established; the established human kinematic model can not only provide high-precision kinematic parameters, such as joint angles, motion trajectories, etc., but also analyze and extract the spatiotemporal parameters of gait, such as walking speed, step length, and gait cycle and other key information, which are of great value in understanding the human movement mechanism, evaluating sports performance, and conducting rehabilitation training. What is particularly important is that the technical solution of the embodiment of the present invention significantly reduces the difficulty of whole-body motion tracking and extraction; explainably, the existing traditional linear fitting methods often have low accuracy when extracting joint angles and are difficult to be effectively applied to whole-body motion tracking; in contrast, the deep learning model of the embodiment of the present invention effectively solves this technical problem through its powerful nonlinear mapping capability, bringing breakthrough progress to the field of whole-body motion tracking, which not only improves the accuracy and reliability of motion data analysis, but also lays a solid foundation for the widespread application of flexible wearable devices in sports science, medical rehabilitation, and human-computer interaction.

[0058] In one embodiment of the present invention, the motion tracking model optimizes model parameters by using a PSO optimization algorithm (Particle Swarm Optimization).

[0059] See also Figures 2 to 5 In a specific embodiment of the present invention, the training acquisition step of the trained motion tracking model includes:

[0060] S1. The tester wears a flexible wearable device under the dynamic capture system, and synchronously collects the three-dimensional data of the human body surface marker points and the 24-channel sensor capacitance output data in preset scenes such as walking, running, upper limb extension, and lower limb leg lifting at different speeds on the treadmill; illustratively, using the following Figure 3 In the data collection process shown, the tester wears a flexible wearable device, affixes 36 markers to the corresponding positions of the body, and completes 24 sets of walking motions at 1.0m / s, 1.5m / s, and 2.0m / s, and running motion data collection at 2.0m / s, 3.0m / s, and 4.0m / s on a treadmill under a dynamic capture system; Figure 4 The distribution of marker points on the whole body is shown. Figure 5 The specific location distribution of sensors of the flexible wearable device is shown; further illustrative example, the flexible sensor used is the ESSA sensor, the acquisition frequency of the data acquisition system is 20hz, the data of the flexible wearable device is collected through the serial port, and the three-dimensional data of the dynamic capture system marker point is collected using QTM software;

[0061] S2. Establish a human kinematic model with 13 joints as the core; explanatory, construct a human joint rigid model with 14 body segments and 28 degrees of freedom starting from the 13 joints. The model is simplified by the head, chest and abdomen segments, two upper and lower arm segments, a pelvis segment and two thighs, calves and feet segments;

[0062] S3, preprocessing the two types of data collected in step S1 by data smoothing, data interpolation, etc., to obtain preprocessed training set data; specifically, for example, the sensor capacitance output data is preprocessed by smoothing the data using an adaptive smoothing method; the three-dimensional data of the human body surface marker points is converted into three-dimensional data of the joint points of 13 human joints through human joint center estimation and adaptive smoothing is performed; the two types of data are interpolated using the nearest neighbor interpolation method to reduce or eliminate the problem of asynchronous acquisition time;

[0063] S4. Build a motion tracking model, which consists of a sequence encoding network and a feature decoding network, and use the PSO optimization algorithm to optimize the model parameters; it is used to realize the mapping of the 24-channel sensor capacitance output data of the flexible device to 13 high-precision three-dimensional data of joint points; explanatory, the sequence encoding network adopts a two-layer biLstm network, and a fully connected network is set between the two layers of biLstm; the feature decoding network consists of a seven-layer fully connected network; in addition, when the model parameters of the motion tracking model are optimized by the PSO optimization algorithm, specifically eleven parameters of the model are optimized, namely: the number of hidden layers of the first biLstm layer of the sequence encoding network, the number of hidden layers of the second biLstm layer, the number of hidden layers of the fully connected layer between the two biLstm layers, the number of hidden layers of the first six fully connected layers of the feature decoding network, the number of model training rounds, and the initial training speed of the model;

[0064] S5, using the training set data obtained in step S3 to train and test the motion tracking model; wherein, during the test, the predicted data output by the model is used to construct a human kinematics model, and the human kinematics model is used to extract kinematic parameters and gait spatiotemporal parameters; when the training set is input into the motion tracking model, its data flow is: 24-channel sensor capacitor output data is input into the sequence encoding network to extract feature vectors, and the feature vectors are spliced ​​by the second-layer bilstm hidden state and the sensor capacitor output at the current moment, and the feature vectors are input into the feature decoding network, and finally mapped to high-precision joint point three-dimensional data. Further explanatory, during the test, the sensor capacitor output data is input into the motion tracking model after training, and the output model prediction value is represented as three-dimensional data of 13 joint points, which is used for the subsequent construction of the human kinematics model. The kinematic parameters that can be extracted from the human kinematics model include: stride period, stance phase time ratio, stride length, step frequency, step width, relative symmetry index, etc.; gait spatiotemporal parameters include: joint angle parameters of 13 joints, joint motion trajectory, etc.

[0065] The embodiment of the present invention specifically discloses a method for accurately extracting human whole-body motion based on a flexible wearable device, which can be applied to the field of robotics; the embodiment of the present invention proposes a motion tracking model based on deep learning, which is composed of a sequence encoding network and a feature decoding network, and uses a PSO algorithm to optimize the model parameters, and can map 24-channel whole-body low-precision sensor data of the flexible wearable device to high-precision joint point data, and further establish a human kinematic model based on the model output. High-precision kinematic parameters and gait spatiotemporal parameters can be extracted from the predicted human kinematic model, which solves the problems of low extraction accuracy and difficulty in whole-body motion tracking in the prior art.

[0066] In the scheme provided by the implementation case of the present invention, data preprocessing is performed on the two types of collected raw data; among them, the sensor data is easily affected by environmental interference and mechanical stress, etc., which may cause abnormal values ​​in the capacitor output. Therefore, the present invention adopts an adaptive smoothing method to smooth the data preprocessing; the three-dimensional data of the human body surface marker points is converted into three-dimensional data of 13 joint points in the human body through the estimation of the human body joint center and adaptively smoothed; the two types of data are interpolated by the nearest neighbor interpolation method to reduce or eliminate the problem of asynchronous acquisition;

[0067] For example, after data processing, the sensor output capacitance data x at each time step t can be obtained. (t) It is expressed as:

[0068]

[0069] In the formula, Indicates the output capacitance data of the 24th sensor;

[0070] The 3D data y of the 13 joints of the dynamic capture system at each time step t (t) Expands to:

[0071]

[0072] In the formula, Indicates the three-dimensional data of the 39th joint point;

[0073] The sensor time series data from (tn) to (t) is selected to establish a mapping relationship with the three-dimensional data of the joint point at time (t). The input and output of the model are x (t-n:t) With y (t) :

[0074] x (t-n:t) =[x (t-n) ,x (t-n+1) ,x (t-n+2) ,…,x (t) ];

[0075] Where n is the number of time intervals.

[0076] Specifically and exemplarily, a total of 24 data collections were conducted to construct the data sets required for model training and testing.

[0077] In a specific embodiment of the present invention, a motion tracking model is constructed by a deep learning algorithm to realize the conversion of low-precision x (t -n:t) Regression mapping high precision y (t) To achieve this goal, it is necessary to select a suitable time series processing network to extract x (t-n:t)The multi-scale features are then mapped to y through a regression network (t) In the technical solution disclosed in the embodiment of the present invention, the motion tracking model is composed of a sequence encoding network and a feature decoding network. Figure 2 The overall structure of the motion tracking model and the data transfer flow are shown.

[0078] During the training process, input time series data x (t-n:t) After being processed by the sequence encoding network, the feature vector r(t) is extracted, and r(t) is input into the feature decoding network and mapped to the high-precision y (t) In order to achieve the best network effect, the PSO algorithm is used to optimize the network layer parameters and training parameters. (t -n:t) Input the motion tracking model and output the high-precision prediction data after network mapping pass A human kinematics model is further established for use in parameter extraction and analysis applications.

[0079] Explanatory principle: during the data collection process, the sensor is susceptible to environmental interference and mechanical stress, and the output electricity is prone to abnormal values, missing values ​​and noise; at the same time, when the tester collects data, there are no restrictions on environmental requirements, and the impact of the environment is uncontrollable, and the data collection accuracy is very low. This requires the sequence encoding network to select a suitable sub-network to fully extract features from low-precision data. Finally, the present invention selects the biLstm network to build a sequence encoding network. Its "gate" structure can effectively avoid ladder explosion and gradient disappearance, and has the characteristics of long-term memory. The biLstm consists of two independent LSTMs, which process the input sequence from the forward and backward directions, and then splice the hidden states in the two directions. The bidirectional structure improves the model's ability to model time series data, and at the same time extracts features more comprehensively and accurately. Figure 6 Its specific structure is shown.

[0080] In a specific embodiment of the present invention, the sequence encoding network is composed of a double-layer biLstm, which absorbs the features of the first layer and converts the hidden state h of the second layer biLstm(t) into (t) With sensor data x (t) Concatenate into a feature vector r(t)∈R S , as the output of the sequence processing network. Figure 7 Take the forward LSTM as an example to analyze its data processing method, use σ to represent the sigmoid activation function, and tanh to represent the hyperbolic tangent function. represents the addition operation of elements, Represents the multiplication operation of elements. Before the LSTM unit is processed, it is necessary to obtain the sensor input x at the current time (t) (t) ∈R 24 , the cell state at time (t-1) and hidden state k is the dimension of hidden state and cell state.

[0081] The LSTM unit controls the cell state through three gates:

[0082] First, the forget gate uses the forget gate unit by weighing the information of the hidden state and the data input. Selectively forget the information of the cell state memory at the previous moment. The formula is as follows:

[0083]

[0084] Where W∈R k×s and b∈R k are the cycle weight and bias respectively.

[0085] The input gate can obtain the cell state through the input time series data and the hidden state of the previous moment. Through the input gate unit Select valid information, and then add new memory to the cell state through calculation to update the cell state and obtain the cell state at that moment The formula is as follows:

[0086]

[0087]

[0088]

[0089] The output gate determines the hidden state h at that moment by the new cell state output (t) , the formula is as follows:

[0090]

[0091]

[0092] The hidden states of the two LSTM units output in the forward and reverse directions are concatenated into the hidden state of the current biLstm unit. The formula is as follows:

[0093]

[0094] In the embodiment of the present invention, the feature decoding network is composed of seven fully connected layers. The feature decoding network needs to regress the feature vector r(t) obtained after the time encoding network is processed to the three-dimensional data y of the human joint point.(t) Because the input-output relationship of the feature decoding network is nonlinear and has a more complex mapping relationship, the fully connected layer can approximate any continuous function by increasing the number of hidden units and units, which can meet the functional requirements of the feature decoding network. At the same time, it is simple and easy to operate, and is the first choice for building a feature mapping network. Figure 8 The structure of the feature mapping network is shown, which consists of seven fully connected layers, which are used to establish the mapping relationship between the feature vector and the three-dimensional coordinates of the joint center, and can realize the y (t) Each layer is modeled according to the following formula:

[0095] f (i) (r i ;w i ,b i )=r i T w i +b i

[0096] Among them, r i is the input vector of the i-th layer, w i , b i are weight and bias respectively;

[0097] Except for the last layer output which does not require an activation function, the rest use the linear rectifier unit RELU as the activation function. The formula is as follows:

[0098] ReLU=max{0,r}

[0099] Therefore, the input of the next layer can be determined as:

[0100] r i+1 =max{0,r i T w i +b i}

[0101] During the training process, the output of the last layer is set to y (t) ∈R 39 , used to train the model's weight parameters. During the test, the last layer of the model will output the predicted value Track the 3D coordinates of 13 joints of the human body to build a human kinematic model.

[0102] In the embodiment of the present invention, considering that the PSO optimization algorithm has the characteristics of multi-dimensional optimization, low sensitivity to initial parameters, and stable convergence to the optimal solution, it is used to optimize the parameters of the motion tracking model to give full play to the network effect. Specifically, the selected optimization parameters are the number of hidden layers in each network layer, the number of training rounds, and the initial training speed, that is, the dimension of PSO optimization is 11, and the goal of PSO optimization is set to find a set of parameter values ​​that make the model predict the output value y (t) With the true value The error value is minimal.

[0103] PSO obtains the optimal solution by continuously iterating through an initial group of random particles, and continuously optimizes by tracking the optimal position of the particle group during each iteration. Table 1 determines the basic parameter values ​​of PSO optimization, and the value selection method is based on existing research experience.

[0104] Table 1. PSO basic parameter values

[0105]

[0106] Suppose there are 15 particles in a region of dimension 11, and the position x of the i-th particle is i With speed v i for:

[0107] x i ={x i1 ,x i2 ,x i3 ,…,x i11}

[0108] v i = {v i1 ,v i2 ,v i3 ,…,v i11}

[0109] In each iteration, the particle is updated by tracking two optimal solutions. The first is the optimal solution found by the particle itself, that is, the individual extreme value p i ={p i1 ,p i2 ,p i3 ,…,p i11}, the other is the optimal solution found by the entire population so far, that is, the global optimal solution p g ={p gbest1 ,p gbest2 ,p gbest3 ,…,p gbest11}, set the optimization function so that when the parameters of the gait tracking network obtain the global optimal solution, the root mean square error value of the model training obtains the minimum value.

[0110] At the kth iteration, the speed and position of the i-th particle are updated as follows:

[0111]

[0112]

[0113] Where r1 and r2 are uniform random numbers in the range [0,1].

[0114] The training data set is used for model training. After model training, the model parameter values ​​after PSO optimization can be obtained, as shown in Table 2.

[0115] Table 2. PSO optimized network parameter values

[0116]

[0117] See also Fig. 9 and Fig.10 During the test, the sensor data is input into the network to obtain the predicted output value At the same time, the dynamic capture system obtains high-precision reference data (t) . According to the predicted output value Can establish human kinematics model and extract gait parameters, Fig. 9 A method for extracting five important joint angles of the upper and lower limbs is provided; Fig.10 A method is provided to separate gait cycles according to joint angle characteristics and further extract kinematic parameters.

[0118] In a further preferred solution, the embodiment of the present invention formulates three types of standards to evaluate the effect of model training, including:

[0119] 1) Evaluate y (t) and The gaps include:

[0120] Root mean square error RMSE, mean absolute error MAE, relationship coefficient r and relative error RE, these four indicators reflect the degree of deviation and direct gap between the true value and the predicted value. The relative error is introduced and the data magnitude is taken into consideration. The calculation method is:

[0121]

[0122] Among them, q is the predicted output y (t) Dimension.

[0123] 2) Evaluate the gap between the prediction and the reference joint point, including:

[0124] based on With y (t) Calculate the predicted joint point at time t With reference joint point The Euclidean distance difference is the relative distance RD of the tester's height (d = 1615mm). The calculation formula for the i-th joint is:

[0125]

[0126] Among them, i∈(1,13).

[0127] 3) Evaluate the accuracy of the model in extracting gait parameters, including:

[0128] The ability of the model to extract kinematic parameters is evaluated using relative error. For joint angle parameters, the extraction method of the present invention is compared with the existing linear fitting method. The goal of linear regression is to find the coefficient W∈R by linear fitting. j×1 The tracked joint angle θ is consistent with the sensor capacitance output X j×1 The predicted value has the minimum residual value XW, and its formula is:

[0129]

[0130] Among them, get W∈R j×1 The dimension j and X j×1 Data composition.

[0131] The following are device embodiments of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0132] See also Fig.11 In an embodiment of the present invention, a human body motion extraction system based on a flexible wearable device is provided, comprising:

[0133] A data acquisition unit, used to acquire sensor capacitance output data of the tester in a preset scenario based on a dynamic capture system and a tester wearing the flexible wearable device;

[0134] A prediction value acquisition unit, used to make predictions based on the acquired sensor capacitance output data using a trained motion tracking model to obtain a prediction value of the three-dimensional data of the joint point;

[0135] A parameter extraction unit, used to construct a human kinematics model based on the obtained three-dimensional data prediction values ​​of the joint points, and extract kinematic parameters and gait spatiotemporal parameters based on the human kinematics model;

[0136] in,

[0137] The motion tracking model includes: a sequence encoding network and a feature decoding network; wherein the sequence encoding network includes two layers of biLstm networks and a fully connected network arranged between the two layers of biLstm networks, and the feature decoding network includes a seven-layer fully connected network; the sequence encoding network is used to input sensor capacitance output data and extract feature vectors, and the feature decoding network is used to input the feature vectors and map them to the three-dimensional data of the joint points; the feature vector is spliced ​​by the hidden state of the second layer of biLstm network and the sensor capacitance output data at the current moment.

[0138] In one embodiment of the present invention, a computer device is provided, the computer device comprising a processor and a memory, the memory being used to store a computer program, the computer program comprising program instructions, and the processor being used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in a computer storage medium to implement a corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used to perform the operation of a human body motion extraction method based on a flexible wearable device.

[0139] In one embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM (Random Access Memory) memory, or a non-volatile memory (non-volatile memory), such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the method for extracting human body motion based on a flexible wearable device in the above embodiment.

[0140] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, optical storage, etc.) that contain computer-usable program code.

[0141] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0142] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for extracting human body motion based on a flexible wearable device, characterized in that: The following steps are involved: Based on the dynamic capture system and the tester wearing the flexible wearable device, the sensor capacitance output data of the tester in the preset scenario is obtained; Based on the obtained sensor capacitance output data, the trained motion tracking model is used for prediction to obtain the three-dimensional data prediction value of the joint point; Constructing a human kinematics model based on the obtained three-dimensional data prediction values ​​of the joint points, and extracting kinematic parameters and gait spatiotemporal parameters based on the human kinematics model; in, The motion tracking model includes: a sequence encoding network and a feature decoding network; wherein the sequence encoding network includes two layers of biLstm networks and a fully connected network arranged between the two layers of biLstm networks, and the feature decoding network includes a seven-layer fully connected network; the sequence encoding network is used to input sensor capacitance output data and extract feature vectors, and the feature decoding network is used to input the feature vectors and map them to the three-dimensional data of the joint points; the feature vector is spliced ​​by the hidden state of the second layer of biLstm network and the sensor capacitance output data at the current moment.

2. According to claim 1, a method for extracting human body motion based on a flexible wearable device is characterized in that: The training steps of the trained motion tracking model include: Based on the dynamic capture system and the sample tester wearing the flexible wearable device, the three-dimensional data samples of the human body surface marker points and the sensor capacitance output data samples of the sample tester in the preset scene are collected; The sensor capacitance output data samples are smoothed preprocessed by an adaptive smoothing method to obtain preprocessed sensor capacitance output data samples; based on the established human kinematics model, the three-dimensional data samples of the human body surface marker points are converted into three-dimensional data of the joint points of the human body joints by using the joint center estimation method and adaptively smoothed to obtain preprocessed three-dimensional data samples of the joint points; the preprocessed sensor capacitance output data samples and the preprocessed three-dimensional data samples of the joint points are interpolated by the nearest neighbor interpolation method respectively to construct a training set; Based on the constructed training set, the PSO optimization algorithm is used to update and optimize the model parameters of the motion tracking model to achieve the preset convergence conditions and obtain a trained motion tracking model; wherein, during the update optimization, the motion tracking model is used to map the sensor capacitance output data samples in the training set to the corresponding joint point three-dimensional data samples in the training set.

3. According to claim 2, a method for extracting human body motion based on a flexible wearable device is characterized in that: In the step of updating and optimizing the model parameters of the motion tracking model based on the constructed training set using the PSO optimization algorithm to achieve a preset convergence condition and obtain a trained motion tracking model, The updated and optimized model parameters include the number of hidden layers of the first layer biLstm network in the sequence encoding network, the number of hidden layers of the second layer biLstm network, the number of hidden layers of the fully connected network, and the number of hidden layers of the first six layers of the fully connected network in the feature decoding network, the number of model training rounds, and the initial training speed of the model.

4. The method for extracting human body motion based on a flexible wearable device according to claim 1, characterized in that: The step of obtaining sensor capacitance output data of the tester in a preset scenario based on the dynamic capture system and the tester wearing the flexible wearable device includes: Based on the dynamic capture system and the tester wearing the flexible wearable device, the initial sensor capacitance output data of the tester in a preset scenario is obtained; The initial sensor capacitance output data is smoothed and preprocessed using an adaptive smoothing method to obtain the sensor capacitance output data of the tester in a preset scenario.

5. The method for extracting human body motion based on a flexible wearable device according to claim 1, characterized in that: The human kinematic model is a human joint rigid model with 14 body segments and 28 degrees of freedom built around 13 joint points, including a head, a chest segment, an abdomen segment, two upper arm segments, two lower arm segments, a pelvis segment, two thigh segments, two calf segments and two foot segments.

6. The method for extracting human body motion based on a flexible wearable device according to claim 1, characterized in that: The preset scenes include one or more of walking, running, upper limb stretching and lower limb leg raising at different speeds.

7. The method for extracting human body motion based on a flexible wearable device according to claim 1, characterized in that: In the step of extracting and obtaining kinematic parameters and gait spatiotemporal parameters based on the human kinematics model, The kinematic parameters include one or both of the joint angle parameters and the joint motion trajectory; The spatiotemporal parameters of gait include one or more of stride period, stance phase time proportion, stride length, stride frequency, step width and relative symmetry index.

8. A human body motion extraction system based on flexible wearable devices, characterized in that: include: A data acquisition unit, used to acquire sensor capacitance output data of the tester in a preset scenario based on a dynamic capture system and a tester wearing the flexible wearable device; A prediction value acquisition unit, used to make predictions based on the acquired sensor capacitance output data using a trained motion tracking model to obtain a prediction value of the three-dimensional data of the joint point; A parameter extraction unit, used to construct a human kinematics model based on the obtained three-dimensional data prediction values ​​of the joint points, and extract kinematic parameters and gait spatiotemporal parameters based on the human kinematics model; in, The motion tracking model includes: a sequence encoding network and a feature decoding network; wherein the sequence encoding network includes two layers of biLstm networks and a fully connected network arranged between the two layers of biLstm networks, and the feature decoding network includes a seven-layer fully connected network; the sequence encoding network is used to input sensor capacitance output data and extract feature vectors, and the feature decoding network is used to input the feature vectors and map them to the three-dimensional data of the joint points; the feature vector is spliced ​​by the hidden state of the second layer of biLstm network and the sensor capacitance output data at the current moment.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for extracting human body whole body motion based on a flexible wearable device as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for extracting human body motion based on a flexible wearable device as described in any one of claims 1 to 7 is implemented.

Images