Knee joint flexion and extension angle estimation method based on flexible sensor

Abstract

The invention relates to a knee joint flexion and extension angle estimation method based on a flexible sensor. The method comprises the following steps: acquiring a voltage signal of a knee part when a target person moves; the voltage signals are input into a neural network model, the knee flexion and extension angle estimated value of the target person is output, the neural network model is obtained based on training of a training set, the training set comprises knee voltage signals of a testee and the corresponding knee flexion and extension angle, and the knee flexion and extension angle estimated value of the target person is obtained. The neural network model is constructed based on a long short-term memory network introducing an attention mechanism. According to the invention, accurate estimation of knee joint flexion and extension angles of different individuals is realized.

Description

Technical Field

[0001] The present invention relates to the technical fields of intelligent wearables and signal processing, and particularly relates to a method for estimating the knee flexion and extension angle based on a flexible sensor. Background Art

[0002] Severe head impacts caused by accidents such as falling from a height, violent strikes, and traffic accidents may lead to serious injuries such as cerebral contusion and intracranial hematoma. The damage may affect the areas of the brain that control movement and coordination, causing cerebral palsy symptoms such as limb movement disorders and balance disorders. Cerebral palsy patients have limited limb activities, stiff and uncoordinated movements, may have slow walking, abnormal gaits, and in severe cases, they may even be unable to walk.

[0003] Patients with severe knee osteoarthritis, rheumatoid arthritis, traumatic arthritis and other diseases can undergo knee replacement surgery to replace the severely damaged knee joint with an artificial joint, relieve knee pain, restore joint function, and improve the quality of life.

[0004] The above-mentioned patients can continuously recover their walking and movement abilities through lower limb rehabilitation training. During the lower limb rehabilitation training process, the knee flexion and extension angle is a very crucial indicator, which can measure the leg movement ability and the knee flexion and extension ability. Due to the small number of professional rehabilitation trainers and the large number of patients in need of rehabilitation training, an uneven supply-demand relationship has been formed. In addition, there is also a lack of convenient equipment and technologies for estimating the knee flexion and extension angle of patients, so that patients can timely understand their own recovery situation. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for estimating the knee flexion and extension angle based on a flexible sensor, so as to accurately estimate the knee flexion and extension angles of different individuals.

[0006] To achieve the above purpose, the present invention provides the following solution:

[0007] A method for estimating the knee flexion and extension angle based on a flexible sensor, comprising:

[0008] Collecting the voltage signal at the knee of the target person during movement;

[0009] Inputting the voltage signal into a neural network model, and outputting an estimated value of the knee flexion and extension angle of the target person, wherein the neural network model is obtained by training based on a training set, the training set includes the voltage signal at the knee of the tester and the corresponding knee flexion and extension angle, and the neural network model is constructed based on a long short-term memory network introducing an attention mechanism.

[0010] Optionally, obtaining the training set includes:

[0011] An intelligent knee brace with a flexible strain sensor is used to obtain the voltage signal of the tester's knee.

[0012] Optical markers are fixed at the hip joint, knee joint and ankle joint of the tester. The position information of the optical markers is measured by an action capture system and the flexion and extension angle of the knee joint is calculated.

[0013] The voltage signal of the tester's knee and the flexion and extension angle of the knee joint are processed to establish a database to obtain the training set.

[0014] Optionally, the processing of the voltage signal of the tester's knee and the flexion and extension angle of the knee joint includes:

[0015] Adopt inertial filtering and mean filtering methods to remove the high-frequency noise in the voltage signal of the tester's knee to obtain the filtered voltage signal.

[0016] Align the time information of the filtered voltage signal and the flexion and extension angle of the tester's knee joint to construct the database.

[0017] Optionally, the removal of high-frequency noise in the voltage signal of the tester's knee by inertial filtering includes:

[0018]

[0019] The method for removing high-frequency noise in the voltage signal of the tester's knee by mean filtering includes:

[0020]

[0021] where x represents a physical quantity, representing the voltage values V1, V2 or V3 on each signal acquisition channel; x k is the measured value of the physical quantity x at time k; is the filtered value of the physical quantity x at time k; α and m are the relevant parameters of inertial filtering and mean filtering respectively.

[0022] Optionally, the alignment of the time information of the filtered voltage signal and the flexion and extension angle of the tester's knee joint includes:

[0023] Extract the extreme value points of the voltage signal and the corresponding time information on each voltage signal acquisition channel, and obtain the average value of the corresponding time information of the voltage signal extreme value points as the voltage extreme value time;

[0024] Extract the extreme values of the flexion and extension angle of the knee joint calculated by the action capture system and the corresponding angle extreme value time;

[0025] Take the difference between the voltage extreme value time and the angle extreme value time and calculate the average value to obtain the relative time difference.

[0026] Add the time information of the knee joint flexion and extension angle calculated by the motion capture system to the relative time difference to complete the alignment of the time information.

[0027] Optionally, after completing the alignment of the time information, it includes:

[0028] Retain the continuous voltage and angle information after the first extreme moment and before the last extreme moment;

[0029] Calculate the angle information corresponding to each voltage sampling moment through the spline interpolation method based on the retained voltage and angle information, and establish the database.

[0030] Optionally, training the neural network model based on the training set includes:

[0031] Take the root mean square error between the output of the neural network model and the label as the optimization target, and use the Adam optimizer for optimization to train the neural network model on the training set.

[0032] Optionally, the neural network model includes an input layer, a long short-term memory layer, an attention mechanism layer, a fully connected layer, and an output layer connected in sequence.

[0033] The beneficial effects of the present invention are as follows: 1. In the process of lower limb rehabilitation training for patients, due to the lack of convenient detection equipment and special detection sites, it is difficult to conveniently, quickly, and accurately understand the lower limb rehabilitation situation of themselves. By fixing the detachable flexible strain sensor on the knee pad, it is stretched to different lengths as the patient's knee joint moves, showing different electrical characteristics. By collecting, processing, and analyzing relevant electrical signals through the supporting circuit, the stretching degree of the sensor can be judged, and then the bending degree of the patient's knee joint can be estimated.

[0034] 2. The present invention can estimate the knee joint flexion and extension angle in real time only by using the flexible strain sensor fixed on the knee pad and a simple supporting circuit, without the assistance of professionals and without building a detection platform in a specific site. It solves the problems existing in existing solutions such as the optical motion capture system requiring a fixed and strict detection site and the inertial motion capture system requiring complex detection equipment.

[0035] 3. The present invention uses a variety of filtering methods to remove the noise interference contained in the original electrical signal; uses methods such as timestamp alignment method and spline interpolation method to solve problems such as inconsistent acquisition frequencies and misaligned acquisition times between different data acquisition systems; uses the detection results of a high-precision dynamic capture system as the gold standard to provide high-quality data for training an artificial neural network model; divides the training set, validation set, and test set, trains using neural network models with a variety of different structures, comprehensively considers factors such as generalization ability and estimation error, and selects and applies the optimal model. Description of the Drawings

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0037] Figure 1 It is a flowchart of a method for estimating knee joint flexion and extension angles based on a flexible sensor according to an embodiment of the present invention;

[0038] Figure 2 It is a physical diagram of an intelligent knee brace fixed with a flexible strain sensor according to an embodiment of the present invention;

[0039] Figure 3 It is a schematic diagram of the intelligent knee brace system collecting three-channel voltage signals according to an embodiment of the present invention, where (a) is the three-channel voltage signals before filtering, and (b) is the schematic diagram of the three-channel voltage signals after filtering and the voltage extreme value time;

[0040] Figure 4 It is a schematic diagram of the motion capture system analyzing the knee joint angle signal to extract the angle extreme value time according to an embodiment of the present invention;

[0041] Figure 5 It is a schematic diagram of the result of time information alignment between multiple systems according to an embodiment of the present invention;

[0042] Figure 6 It is a schematic diagram of a neural network model based on a long short-term memory network and an attention mechanism according to an embodiment of the present invention. Detailed Embodiments

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0044] To make the above objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] This embodiment provides a method for estimating the flexion and extension angle of the knee joint based on a flexible sensor, which is characterized by including:

[0046] Collect the voltage signal at the knee of the target person during movement;

[0047] Input the voltage signal into a neural network model to output an estimated value of the knee flexion and extension angle of the target person. Among them, the neural network model is obtained by training based on a training set, and the training set includes the voltage signal at the knee of the tester and the corresponding knee joint flexion and extension angle. The neural network model is constructed based on a long short-term memory network with an attention mechanism introduced.

[0048] Furthermore, obtaining the training set includes:

[0049] Obtain the voltage signal at the knee of the tester through an intelligent knee brace fixed with a flexible strain sensor;

[0050] Fix optical markers at the hip joint, knee joint, and ankle joint of the tester, and measure the position information of the optical markers through a motion capture system and calculate the knee joint flexion and extension angle;

[0051] Process the voltage signal at the knee of the tester and the knee joint flexion and extension angle, and establish a database to obtain the training set.

[0052] Specifically, as Figure 2 shown. The intelligent knee brace mainly uses a detachable and fixed flexible strain sensor as the sensor, and is equipped with a hardware circuit to assist in data acquisition, processing, transmission, etc.

[0053] The process of collecting data includes:

[0054] Step 1-1: Wear an intelligent knee brace fixed with a flexible strain sensor on the knee of the tester.

[0055] Step 1-2: Fix an optical marker at each of the hip joint, knee joint, and ankle joint of the tester for measurement by the optical motion capture system.

[0056] Step 1-3: After simultaneously turning on the intelligent knee brace fixed with a flexible strain sensor and the optical motion capture system for data collection, the tester performs periodic lower limb activities within the effective capture range of the optical motion capture system lens, causing the knee joint to bend and stretch to different degrees. In particular, the tester can slowly perform continuous squatting and standing movements in place or repeatedly bend the knees backward, and try to move the knee joint as widely as possible to collect more effective data.

[0057] Further, the processing of the voltage signal and the knee joint flexion / extension angle of the tester includes:

[0058] Adopting inertial filtering and mean filtering methods to remove high-frequency noise in the voltage signal of the tester's knee, and obtaining the filtered voltage signal;

[0059] Aligning the time information of the filtered voltage signal and the knee joint flexion / extension angle of the tester, and constructing a database.

[0060] Furthermore, adopting inertial filtering to remove high-frequency noise in the voltage signal of the tester's knee includes:

[0061]

[0062] Adopting the mean filtering method to remove high-frequency noise in the voltage signal of the tester's knee includes:

[0063]

[0064] where x represents a physical quantity, representing the voltage values V1, V2 or V3 on each signal acquisition channel; x k is the measured value of the physical quantity x at time k; is the filtered value of the physical quantity x at time k; α and m are the relevant parameters of inertial filtering and mean filtering respectively.

[0065] Furthermore, aligning the time information of the filtered voltage signal and the knee joint flexion / extension angle of the tester includes:

[0066] Extracting the extreme value points and corresponding time information of the voltage signal on each voltage signal acquisition channel, and obtaining the average value of the corresponding time information of the voltage signal extreme value points as the voltage extreme value time;

[0067] Extracting the extreme values of the knee joint flexion / extension angle calculated by the motion capture system and the corresponding angle extreme value times;

[0068] Calculating the difference between the voltage extreme value time and the angle extreme value time and taking the mean value to obtain the relative time difference;

[0069] Adding the relative time difference to the time information of the knee joint flexion / extension angle calculated by the motion capture system to complete the time information alignment.

[0070] Furthermore, after completing the time information alignment, it includes:

[0071] Retaining the continuous voltage and angle information after the first extreme value moment and before the last extreme value moment;

[0072] Calculate the angle information corresponding to each voltage sampling moment through the spline interpolation method based on the reserved voltage and angle information, and establish a database.

[0073] Further, training the neural network model based on the training set includes:

[0074] Taking the root mean square error between the output of the neural network model and the label as the optimization objective, select the Adam optimizer, and train the neural network model on the training set to obtain the mapping relationship between the knee voltage signal and the knee joint flexion and extension angle.

[0075] Further, the neural network model includes an input layer, a long short-term memory layer, an attention mechanism layer, a fully connected layer, and an output layer connected in sequence.

[0076] Specifically, an input layer, a long short-term memory layer, an attention mechanism layer, three fully connected layers, and an output layer are organically combined to form a complete artificial neural network model. Among them, the size of the input layer is 3×m, the long short-term memory layer has 100 hidden states, the three fully connected layers have 100, 50, and 25 neurons in sequence, and the size of the output layer is 1×1.

[0077] The method of this embodiment will be further described below with reference to the accompanying drawings:

[0078] A method for estimating the knee joint flexion and extension angle based on a flexible sensor, the process is as Figure 1 shown, and specifically includes the following steps:

[0079] Step 1: Wear the intelligent knee brace with the flexible strain sensor fixed on the knee of the tester, and at the same time fix an optical marker at each of the hip joint, knee joint, and ankle joint of the tester for measurement by the optical motion capture system. The tester performs periodic lower limb activities within the effective capture range of the optical motion capture system lens, so that the knee joint bends and stretches to different degrees. At the same time, use the intelligent knee brace with the flexible strain sensor and the optical motion capture system to collect data as the original sampling data for establishing the database. The intelligent knee brace collects the voltage signal output by the flexible sensor after passing through the measurement circuit, and the motion capture system measures the position information of the marker and calculates the corresponding angle information, that is, the angle of knee joint flexion and extension.

[0080] In particular, the tester can slowly perform continuous squatting and standing up movements in place or repeatedly bend the knees backward, and try to move the knee joint as widely as possible to collect more effective data.

[0081] Step 2: Use the inertial filtering and mean filtering methods to initially remove the high-frequency noise in the original data. The processing process is shown in formulas (1) and (2):

[0082]

[0083] Among them, x represents a physical quantity, which represents the voltage values V1, V2, or V3 on each signal acquisition channel; x k is the measured value of the physical quantity x at time k; is the filtered value of the physical quantity x at time k; α and m are the relevant parameters of inertial filtering and mean filtering respectively.

[0084] Noting that the tester is performing a periodic reciprocating motion, the collected data also has a periodic change pattern. The extreme value points of the voltage signals on each channel are extracted respectively and the corresponding time information is recorded, and then the average value is calculated as the voltage extreme value time at this sampling moment, and the effect is as Figure 3 (a)-(b) shown. Similarly, the extreme value situation of the angle information parsed by the motion capture system and the corresponding angle extreme value time are extracted, and the effect is as Figure 4 shown.

[0085] The difference between the matched voltage extreme value time and the angle extreme value time is calculated and its average value is obtained, and then the relatively accurate time difference between the two systems can be obtained. Adding the relative time difference between the systems to the time information corresponding to the angle information of the motion capture system can align the time information between the two systems, and the effect is as Figure 5 shown. After the time information is aligned, only the high-quality continuous voltage and angle information after the first extreme value moment (after time alignment, it can be considered that the extreme value moments of voltage and angle are the same.) and before the last extreme value moment are retained. The voltage and angle information corresponding to the remaining time information will not be used as valid information for building the database.

[0086] Since the sampling frequencies of different systems are not exactly the same and the sampling moments are not exactly the same either, the spline interpolation method is used to process the retained voltage and angle information to calculate the angle information corresponding to each voltage sampling moment, satisfying the high-order continuity and relatively high accuracy of the interpolation curve, and establishing a database based on this.

[0087] Step 3. For the sampling moments T1, T2, …, T n corresponding three-channel voltage information V i,1 , V i,2 , …, V i,n and the angle information α1, α2, …, α n , an appropriate data length m is selected as the length of each sample data to form the following data correspondence as shown in Table 1.

[0088] Table 1

[0089]

[0090] Among them, V i,jdenotes the voltage value collected by the \(i\)-th channel at the \(j\)-th sampling moment \(T\) (\(i\in\{1,2,3\}, 1\leq j\leq n\)); \(\alpha\) j the voltage value collected by the \(i\)-th channel (\(i\in\{1,2,3\}, 1\leq j\leq n\)); \(\alpha\) j denotes at the \(j\)-th sampling moment \(T\) j the corresponding angle value (\(1\leq j\leq n\)).

[0091] In particular, by choosing \(m = 50\), relatively ideal results can be obtained.

[0092] For the dataset composed of the above \(n - m + 1\) sample pairs, randomly select 80% of them as the training set, 10% as the validation set, and 10% as the test set. The training set is used to train the model, that is, to determine the parameters such as the weights and biases of the model. Usually, we call these parameters learning parameters. The validation set is used for model selection, for the selection of hyperparameters such as the number of network layers, the number of network nodes, the number of iterations, and the learning rate. The test set evaluates the final model after training, and neither participates in the process of learning parameters nor in the process of selecting hyperparameters.

[0093] Step 4, construct a neural network model as Figure 6 shown, including an input layer, a long short-term memory layer, an attention mechanism layer, three fully connected layers, and an output layer. The input layer is the entrance of the neural network, responsible for receiving the original data; the long short-term memory layer is specifically used to process sequence data. By introducing a gating mechanism (input gate, forget gate, and output gate) to control the flow and storage of information, it can effectively solve the problem of gradient disappearance or gradient explosion that may occur in traditional recurrent neural networks; the attention mechanism layer determines the importance of each element to the current output by calculating the attention weights of each element in the input sequence, enabling the model to automatically focus on the important parts of the input sequence when processing the input data; each neuron in the fully connected layer is connected to all neurons in the previous layer. Through a series of linear transformations and non-linear activation functions, the input features are mapped to a new feature space to meet different task requirements; the output layer is the last layer of the neural network, which can output the final prediction result based on the feature extraction and transformation of the previous layers. Among them, the size of the input layer is \(3\times m\), and the size of the output layer is \(1\times1\).

[0094] In particular, by choosing the long short-term memory layer to have 100 hidden states, and the three fully connected layers to have 100, 50, and 25 neurons in sequence, relatively ideal results can be obtained.

[0095] Step 5, according to the training set, validation set, and test set divided in Step 3, using the root mean square error between the output of the neural network model constructed in Step 4 and the label as the optimization objective, select the Adam optimizer, train the neural network on the training set, use the validation set to adjust and optimize the hyperparameters of the neural network, and finally evaluate the model effect with the test set.

[0096] Step 6: Based on the evaluation of the effects of different hyperparameter models in Step 5, comprehensively consider factors such as generalization performance, estimation error, and computational efficiency to determine the finally selected model. Write its specific parameters into the single-chip microcomputer supporting the intelligent knee pad, and the knee flexion and extension angles can be estimated in real time during actual use, reflecting the range of motion of the knee joint and providing feedback on the knee joint's movement ability.

[0097] The embodiments described above are only descriptions of the preferred embodiments of the present invention, and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for estimating knee flexion and extension angle based on flexible sensor, characterized in that: include: Collect voltage signals from the knees of target persons when they are exercising; The voltage signal is input into a neural network model, and an estimated value of the knee flexion and extension angle of the target person is output, wherein the neural network model is obtained based on training of a training set, and the training set includes the tester's knee voltage signal and the corresponding knee flexion and extension angle, and the neural network model is constructed based on a long short-term memory network that introduces an attention mechanism.

2. The method for estimating knee flexion and extension angle based on flexible sensor according to claim 1, characterized in that: Acquiring the training set includes: The voltage signal of the tester's knee is obtained by fixing a smart knee brace with a flexible strain sensor; Fixing optical markers at the hip joint, knee joint and ankle joint of the tester, measuring the position information of the optical markers through a motion capture system and calculating the flexion and extension angle of the knee joint; The voltage signal of the tester's knee and the knee flexion and extension angle are processed to establish a database to obtain the training set.

3. The method for estimating knee flexion and extension angle based on flexible sensor according to claim 2, characterized in that: Processing the voltage signal of the tester's knee and the knee flexion and extension angle includes: Using inertial filtering and mean filtering methods to remove high-frequency noise in the voltage signal of the tester's knee to obtain a filtered voltage signal; The filtered voltage signal and the tester's knee flexion and extension angle are aligned with time information to construct the database.

4. The method for estimating knee flexion and extension angle based on flexible sensor according to claim 3, characterized in that: Using inertial filtering to remove high-frequency noise in the voltage signal of the tester's knee includes: The method of removing high-frequency noise in the voltage signal of the tester's knee by using a mean filtering method comprises: Wherein, x represents a physical quantity, which represents the voltage value V1, V2 or V3 on each signal acquisition channel; x k is the measured value of the physical quantity x at time k; is the filtered value of the physical quantity x at time k; α and m are the relevant parameters of inertial filtering and mean filtering respectively.

5. The method for estimating knee flexion and extension angle based on flexible sensor according to claim 3, characterized in that: Aligning the filtered voltage signal with the tester's knee flexion and extension angle in time includes: Extracting the voltage signal extreme value points and corresponding time information on each voltage signal acquisition channel, and obtaining the average value of the corresponding time information of the voltage signal extreme value points as the voltage extreme value time; Extracting the extreme value of the knee flexion and extension angle calculated by the motion capture system and the corresponding extreme angle time; Subtract the voltage extreme value time from the angle extreme value time and calculate the average to obtain a relative time difference; The time information of the knee joint flexion and extension angle calculated by the motion capture system is added to the relative time difference to complete the alignment of the time information.

6. The method for estimating knee flexion and extension angle based on flexible sensor according to claim 5, characterized in that: After the time information alignment is completed, the following steps are included: The continuous voltage and angle information after the first extreme value moment and before the last extreme value moment is retained; The angle information corresponding to each voltage sampling moment is calculated by using a spline interpolation method according to the retained voltage and angle information, and the database is established.

7. The method for estimating knee flexion and extension angle based on flexible sensor according to claim 1, characterized in that: Training a neural network model based on a training set includes: The root mean square error between the output of the neural network model and the label is used as the optimization target, and the Adam optimizer is used for optimization, and the neural network model is trained on the training set.

8. The method for estimating knee flexion and extension angle based on flexible sensor according to claim 1, characterized in that: The neural network model includes an input layer, a long short-term memory layer, an attention mechanism layer, a fully connected layer and an output layer which are connected in sequence.

Images