Joint angle prediction method and system, medium and electronic equipment
The prediction of joint angles through the timing neural network model is solved, and the accuracy and portability of joint angle monitoring in the prior art is achieved, and stable and accurate joint angle prediction in instantaneous motion is achieved.
Patent Information
- Application Number
- CN202411776522.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-05
- Publication Date
- 2025-03-11
AI Technical Summary
When monitoring the angle of the human joints, the optical detection is accurate but the portability is poor, the inertial detection is good but the accuracy is poor, and the data is prone to fluctuations in instantaneous motion, resulting in inaccurate data.
The timing neural network model is adopted to obtain muscle deformation data and joint angle data during the user's joint flexion and stretching process, and form a training data set, train the timing neural network model, and then predict joint angle data.
It realizes accurate prediction of joint angles in instantaneous motion, avoids cumulative errors in inertial detection, improves data stability and accuracy, and supports fast prediction, suitable for fast or complex motion scenarios.
Smart Images
Figure CN119587012A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of human motion monitoring, and in particular, relates to a joint angle prediction method, system, medium and electronic equipment. Background Art
[0002] At present, human motion monitoring involves various fields such as human-computer interaction and health care. Human motion monitoring usually uses optical detection or inertial detection to monitor the bending angles of joints of body parts such as fingers, arms or legs.
[0003] Optical detection has high accuracy, but usually relies on a multi-camera system and has poor portability. In order to accurately capture changes in joint angles, multiple cameras need to be deployed to monitor various angles of the joints.
[0004] The inertial sensors used in inertial detection are portable but have poor accuracy. Inertial sensors include accelerometers and angular velocity sensors (i.e. gyroscopes), which calculate the size of the joint angle by calculating the linear acceleration or angular velocity. Since the accelerometer and angular velocity sensor accumulate measurement errors over time, the accuracy of the calculated joint angle decreases, and the measurement errors must be corrected frequently.
[0005] In addition, the data monitored by optical detection or inertial detection is prone to fluctuation during the user's instantaneous movement, and cannot accurately reflect the continuity of joint angle changes, thereby reducing the reliability of the measured data. Summary of the invention
[0006] One of the purposes of the present application is to provide a joint angle prediction method, system, medium and electronic device, which can obtain accurate joint angle data including instantaneous motion through prediction.
[0007] To achieve the above objectives and other related objectives, the present application provides a joint angle prediction method in a first aspect, comprising the following steps:
[0008] Acquisition step: Acquiring muscle deformation data S(t) and joint angle data y(t) of different users during joint flexion and extension, wherein the muscle deformation data S(t) and the joint angle data y(t) are respectively a set of data related to time t, and the muscle deformation data S(t) and the joint angle data y(t) are obtained synchronously;
[0009] Training step: The muscle deformation data S(t) and the joint angle data y(t) constitute a training data set, and the training data set is input into the time series neural network model for training to obtain a trained time series neural network model;
[0010] Prediction step: Collect muscle deformation data S within a predetermined time step T during the joint flexion and extension process of the user j to be predicted j (t), the muscle deformation data S j (t) Input the trained time series neural network model to obtain the joint angle data y within the predetermined time step T j (t).
[0011] A second aspect of the present application provides a joint angle prediction system, comprising:
[0012] An acquisition module; used to acquire muscle deformation data S(t) and joint angle data y(t) of different users during joint flexion and extension, wherein the muscle deformation data s(t) and the joint angle data y(t) are respectively a set of data related to time t, and the muscle deformation data S(t) and the joint angle data y(t) are acquired synchronously;
[0013] A training module; used for forming a training data set from the muscle deformation data S(t) and the joint angle data y(t), inputting the training data set into a time series neural network model for training, and obtaining a trained time series neural network model;
[0014] Prediction module: used to collect muscle deformation data S within a predetermined time step T during the joint flexion and extension process of the user j to be predicted j (t), the muscle deformation data S j (t) Input the trained time series neural network model to obtain the joint angle data y within the predetermined time step T j (t).
[0015] The third aspect of the present application provides a computer-readable storage medium, comprising: the computer-readable storage medium stores a computer program, and when the computer program runs on a processor, the processor executes the joint angle prediction method as described above.
[0016] A fourth aspect of the present application provides an electronic device, comprising: at least one memory for storing a program;
[0017] At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the above-mentioned joint angle prediction method.
[0018] The present application adopts the joint angle prediction method, system, medium and electronic device of the present application, which has at least the following beneficial effects:
[0019] 1. Accurately predict joint angles. The joint angle prediction method collects muscle deformation data S of the user j to be predicted. j(t), the muscle deformation data S j (t) Input the trained time series neural network model to obtain the joint angle data y within the time step T j (t), that is, it can predict the user’s joint angle in real time and thus accurately reflect the movement state of the joint.
[0020] 2. It can solve the problem mentioned in the background technology that the data monitored by optical detection or inertial detection during the user's instantaneous movement is prone to fluctuations, resulting in inaccurate data. In the prediction step, there is no need to rely on inertial data for calculation, which avoids the influence of cumulative errors and ensures the stability and accuracy of the prediction of the user's joint angle in any instantaneous movement.
[0021] 3. Able to quickly predict joint angles. This application uses a time-series neural network model to quickly predict joint angles. For example, joint angle predictions can be made 60 times per second, ensuring that even in fast or complex motion scenes, the system can provide instant angle feedback to meet the user's needs for fast response.
[0022] 4. The joint angle prediction method is based on flexible sensors, and the flexible sensors can be integrated into joint sleeves, which is convenient for users to carry and improves its applicability.
[0023] 5. The joint angle prediction method is to collect the user's muscle deformation data S j (t), so the output prediction result can make personalized joint angle prediction for the collected user. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0025] Figure 1 A flowchart of an acquisition step and a training step according to an embodiment of the present application;
[0026] Figure 2 A flowchart of the prediction steps according to an embodiment of the present application;
[0027] Figure 3 is a schematic structural diagram of a flexible sensor according to an embodiment of the present application;
[0028] Illustration Description:
[0029] 1 data interface; 2 flexible sensor; 21 stretchable conductor; 22 connection end DETAILED DESCRIPTION
[0030] The following describes the implementation methods of the present application through specific specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in the present application can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.
[0031] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0032] Reference Figure 1 and Figure 2 The first aspect of the present application provides a joint angle prediction method, comprising the following steps:
[0033] Acquisition step: Acquiring muscle deformation data S(t) and joint angle data y(t) of different users during joint flexion and extension, wherein the muscle deformation data S(t) and the joint angle data y(t) are respectively a set of data related to time t, and the muscle deformation data S(t) and the joint angle data y(t) are obtained synchronously;
[0034] Training steps: The muscle deformation data S(t) and the joint angle data y(t) form a training data set, and the training data set is input into the time series neural network model for training to obtain the trained time series neural network model; the time series neural network model refers to a deep learning model specifically used to process time series data, including recurrent neural networks RNNs and convolutional neural networks CNNs. The training data set used for training is time series data, including muscle deformation data S(t) and joint angle data y(t) after data preprocessing. The trained time series neural network model can obtain the relationship between muscle deformation and joint angle.
[0035] Prediction step: Collect muscle deformation data S within a predetermined time step T during the joint flexion and extension process of the user j to be predicted j (t), the muscle deformation data S j (t) Input the trained time series neural network model to obtain the joint angle data y within the predetermined time step T j (t).
[0036] In some embodiments of the present application, the muscle deformation data S(t) and / or the muscle deformation data S j (t) Includes muscle deformation data at maximum joint angle and muscle deformation data at minimum joint angle.
[0037] Furthermore, muscle deformation data at maximum joint angle and muscle deformation data at minimum joint angle can be obtained through motion calibration. Motion calibration aims to collect key data of the user's joint motion range and provide an accurate reference benchmark for subsequent angle prediction. The following is a specific introduction using knee joint motion calibration as an example:
[0038] The knee joint motion calibration includes squat leg raise calibration and standing calibration. Squat leg raise calibration means that the user performs a complete squat leg raise action, recording the data of the knee joint in the most flexed and extended states. This motion calibration can obtain the maximum flexion angle of the knee joint, providing a reference for the upper limit of the user's joint angle in the prediction step. Standing calibration means that the user remains standing. This motion calibration is used to determine the minimum flexion angle of the knee joint, providing a reference for the lower limit of the user's joint angle in the prediction step.
[0039] In some embodiments of the present application, the training step also includes data preprocessing of the muscle deformation data S(t) and the joint angle data y(t) collected by the camera or inertial sensor. Through data preprocessing, the availability of the data to be trained can be improved, which is conducive to the training of the temporal neural network model.
[0040] Data preprocessing includes denoising, smoothing and normalization. Denoising means that since the data collected by the sensor may be affected by environmental noise or individual differences of users during actual movement, the original data is first filtered to remove invalid data points. Smoothing means that in order to eliminate the fluctuation of data in instantaneous movement, a smoothing algorithm is used to smooth the sensor signal to ensure the continuity and stability of the joint angle data, thereby improving the reliability of the prediction. Normalization means normalizing the joint angle data collected by the sensor to ensure that all data fluctuates within a uniform range and avoid errors caused by differences in user body shape, environment, etc. The muscle deformation data S(t) is normalized to obtain the normalized muscle deformation data S norm (t). For example, normalization processing can be performed based on the muscle deformation data at the maximum joint angle and the muscle deformation data at the minimum joint angle obtained during previous calibration. Obtaining the muscle deformation data at the maximum joint angle and the muscle deformation data at the minimum joint angle through action calibration helps to quantify the muscle deformation characteristics, thereby providing a better basis for normalization processing in data preprocessing.
[0041] In some embodiments of the present application, in the acquisition step, the muscle deformation data S(t) can be collected by using a flexible sensor, and the joint angle data y(t) can be collected by optical detection or inertial detection.
[0042] The flexible sensor can be a flexible pressure sensor, a flexible resistive strain sensor or a flexible force strain sensor. The flexible sensor is attached to the surface of the muscle to collect muscle deformation data during muscle contraction or extension, and transmits the collected muscle deformation data to an external device through a data interface. Figure 3 The example shown details how the flex sensor works.
[0043] In one example, referring to Figure 3 , the data interface 1 is used to obtain the muscle deformation data collected by the flexible sensor 2 and transmit the collected muscle deformation data to an external device. The flexible sensor 2 includes a stretchable conductor 21 and a connection end 22. The stretchable conductor 21 is used to deform as the muscle groups around the joints move. The flexible sensor 2 can collect the muscle deformation data according to the deformation of the stretchable conductor 21. The flexible sensor 2 is connected to the data interface 1 through the connection end 22, and is used to transmit the collected muscle deformation data to the data interface 1. The material of the stretchable conductor 21 can be liquid metal, for example, the liquid metal is gallium-indium alloy, gallium-tin alloy, gallium-indium-tin alloy or gallium-zinc alloy. The advantage of using such liquid metal materials is that they have good electrical conductivity, stretching durability and biosafety.
[0044] In some embodiments of the present application, the flexible sensor is integrated into the joint sleeve. The flexible sensor is integrated into the joint sleeve to maintain the stability of the position of the flexible sensor during the flexion and extension of the joint. The joint sleeve includes knee pads, wrist pads, ankle pads, finger joint sleeves, elbow joint sleeves, etc. The joint sleeve can be integrated with a visual interface and storage function, so that the user can obtain real-time motion feedback, view and analyze historical motion data, and provide guidance for motion optimization or rehabilitation treatment.
[0045] Furthermore, in some embodiments of the present application, there may be multiple flexible sensors arranged at different positions of muscle groups near joints to capture muscle deformation. Multiple flexible sensors can increase the accuracy of judging the flexion and extension movements of the joints.
[0046] When there are multiple flexible sensors, the collected multiple muscle deformation data S(t) can directly form a time series feature vector X(t), or can be preprocessed to obtain multiple normalized muscle deformation data S 1，norm (t),…,S n，norm (t), where n is the number of flexible sensors, and then multiple normalized muscle deformation data S 1，norm (t),…,Sn，norm (t) constitutes the time series feature vector X(t) = [S 1，norm (t),…,S n，norm (t)]. The time series feature vector X(t) can be combined with the joint angle data y(t) to form a training data set for training the time series neural network model.
[0047] In some embodiments of the present application, a single camera or a multi-camera configuration can be used to collect joint angle data y(t). In a multi-camera configuration, multiple cameras can form a motion capture system, which can accurately capture joint angle changes. It should be noted that in the acquisition step, the muscle deformation data S(t) and the joint angle data y(t) are acquired synchronously, which is crucial for the accuracy of the training and prediction results of the timing neural network model. In some other embodiments, an inertial sensor can also be used to collect joint angle data y(t).
[0048] In some embodiments of the present application, in the training step, the training process of the time series neural network model can be optimized to improve the accuracy of the training results. For example, the back propagation through time (BPTT) algorithm can be used to optimize the time series neural network model.
[0049] In some embodiments of the present application, the prediction step also includes the muscle deformation data S j (t) Perform data preprocessing, which includes denoising, smoothing and normalization, etc., which can improve the availability of collected data and facilitate input into the trained time series neural network model to obtain more accurate joint angle data. Similar to the training step, key data of the user's joint motion range can be collected through action calibration. Obtaining muscle deformation data at the maximum joint angle and muscle deformation data at the minimum joint angle through action calibration helps to quantify muscle deformation characteristics, thereby providing a better basis for normalization processing in data preprocessing.
[0050] Furthermore, in some embodiments of the present application, a flexible sensor may be used to collect muscle deformation data S j (t) There may be multiple flexible sensors, which are arranged at different positions of the muscle groups near the joints to capture muscle deformation. Multiple flexible sensors can increase the accuracy of judging the flexion and extension movements of the joints.
[0051] When there are multiple flexible sensors, the collected multiple muscle deformation data S j (t) can directly form the cumulative time series information, or it can be preprocessed to obtain multiple normalized muscle deformation data S j1,norm (t),…,S jn,norm(t), where n is the number of flexible sensors, and then multiple normalized muscle deformation data S j1,norm (t),…,S jn,norm (t) constitutes the accumulated time series information, the accumulated time series information includes the accumulated time series feature vector X(t) = [S j1，norm (t),…,S jn，norm (t)], the accumulated time series information is input into the trained time series neural network model to obtain the joint angle data y within the predetermined time step T j (t).
[0052] In the prediction step, the predetermined time step T can be from the acquisition start time t1 to the acquisition current time t N The predetermined time step T may be 10s, 20s or other suitable time lengths. The joint angle prediction method of the present application can predict the first moment t1 and the Nth moment t N The joint angle data y between j (t), the predicted joint angles can be used to analyze the motion state of the predicted user j during the acquisition process, so as to carry out posture correction, rehabilitation training and other activities.
[0053] The joint angle prediction method of the present application has the following beneficial effects:
[0054] 1. Able to accurately predict joint angles. The joint angle prediction method collects muscle deformation data S of the user y to be predicted. j (t), the muscle deformation data S j (t) Input the trained time series neural network model to obtain the joint angle data y within the predetermined time step T j (t), that is, it can predict the user’s joint angle in real time and thus accurately reflect the movement state of the joint.
[0055] 2. It can solve the problem mentioned in the background technology that the data monitored by optical detection or inertial detection during the user's instantaneous movement is prone to fluctuations, resulting in inaccurate data. In the prediction step, there is no need to rely on inertial data for calculation, which avoids the influence of cumulative errors and ensures the stability and accuracy of the prediction of the user's joint angle in any instantaneous movement.
[0056] 3. Able to quickly predict joint angles. This application uses a time-series neural network model to quickly predict joint angles. For example, joint angle predictions can be made 60 times per second, ensuring that even in fast or complex motion scenes, the system can provide instant angle feedback to meet the user's needs for fast response.
[0057] 4. The joint angle prediction method is based on flexible sensors, and the flexible sensors can be integrated into joint sleeves, which is convenient for users to carry and improves its applicability.
[0058] 5. The joint angle prediction method is to collect the user's muscle deformation data S j (t), so the output prediction result can make personalized joint angle prediction for the collected user.
[0059] The second aspect of the present application also provides a joint angle prediction system, comprising:
[0060] An acquisition module; used to acquire muscle deformation data S(t) and joint angle data y(t) of different users during joint flexion and extension, wherein the muscle deformation data S(t) and the joint angle data y(t) are respectively a set of data related to time t, and the muscle deformation data S(t) and the joint angle data y(t) are acquired synchronously;
[0061] Training module: used to form a training data set with muscle deformation data S(t) and joint angle data y(t), input the training data set into the time series neural network model for training, and obtain the trained time series neural network model;
[0062] Prediction module: used to collect muscle deformation data S within a predetermined time step T during the joint flexion and extension process of the user j to be predicted j (t), the muscle deformation data S j (t) Input the trained time series neural network model to obtain the joint angle data y within the predetermined time step T j (t).
[0063] In some embodiments of the present application, the joint angle prediction system includes a data processing module for identifying possible abnormal motion states based on the joint angles and changing trends, helping users prevent potential joint injuries. Especially for athletes and rehabilitation patients, the prediction system can be used as an effective health monitoring tool to provide technical support for safe exercise and health management. In addition, the system can also display the real-time change curve of the joint angle through smart devices (such as mobile phones, smart watches or display screens) to help users intuitively understand the joint status during their own movements. In addition to real-time feedback, the system can also store joint angle data in the cloud or local storage device for users to review and analyze in the future. These data can be used to analyze the correctness of the movement, identify potential joint injury risks, or be used for guidance in sports rehabilitation.
[0064] It should be understood that the specific process of each module executing the above corresponding steps has been described in detail in the above method embodiment, and for the sake of brevity, it will not be repeated here.
[0065] It should also be understood that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.
[0066] The third aspect of the present application provides a computer-readable storage medium, on which a program of a joint angle prediction method is stored, so that the joint angle prediction method provided in the first aspect of the present application is executed. As for the aforementioned computer-readable storage medium, those skilled in the art can understand that the embodiments for realizing the functions of the aforementioned system and each unit can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, it executes embodiments including the functions of the aforementioned system and each unit. The aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM or optical disk.
[0067] The fourth aspect of the present application provides an electronic device, comprising: at least one memory for storing programs; at least one processor for executing the programs stored in the memory. The electronic device may be a server. The electronic device comprises a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device comprises a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps of the joint angle prediction method of the present application are implemented.
[0068] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present application. These improvements and substitutions should also be regarded as the scope of protection of the present application.
Claims
1. A joint angle prediction method, characterized in that: The following steps are involved: Get steps; Obtain muscle deformation data S(t) and joint angle data y(t) of different users during joint flexion and extension, wherein the muscle deformation data S(t) and the joint angle data y(t) are respectively a set of data related to time t, and the muscle deformation data S(t) and the joint angle data y(t) are obtained synchronously; Training step: The muscle deformation data S(t) and the joint angle data y(t) constitute a training data set, and the training data set is input into the time series neural network model for training to obtain a trained time series neural network model; Prediction step: Collect muscle deformation data S within a predetermined time step T during the joint flexion and extension process of the user j to be predicted j (t), the muscle deformation data S j (t) Input the trained time series neural network model to obtain the joint angle data y within the predetermined time step T j (t).
2. The joint angle prediction method according to claim 1, characterized in that: The muscle deformation data S(t) and / or the muscle deformation data S j (t) Includes muscle deformation data at maximum joint angle and muscle deformation data at minimum joint angle.
3. The joint angle prediction method according to claim 2, characterized in that: The muscle deformation data at the maximum joint angle and the muscle deformation data at the minimum joint angle are obtained through action calibration.
4. The joint angle prediction method according to claim 1, characterized in that: The training step also includes data preprocessing of the muscle deformation data S(t) and the joint angle data y(t).
5. The joint angle prediction method according to claim 1, characterized in that: In the prediction step, the muscle deformation data S is also included j (t) Perform data preprocessing.
6. The joint angle prediction method according to claim 1, characterized in that: The muscle deformation data S(t) collected by the flexible sensor is obtained.
7. The joint angle prediction method according to claim 1, characterized in that: Obtain the muscle deformation data S collected by the flexible sensor j (t).
8. The joint angle prediction method according to claim 6 or 7, characterized in that: The flexible sensor includes a stretchable conductor, and the material of the stretchable conductor includes liquid metal.
9. The joint angle prediction method according to claim 6 or 7, characterized in that: The flexible sensor is integrated into the joint sleeve.
10. The joint angle prediction method according to claim 9, characterized in that: The joint protectors include knee protectors, wrist protectors, ankle protectors, finger joint protectors and elbow joint protectors.
11. The joint angle prediction method according to claim 1, characterized in that: The joint angle data y(t) collected by optical detection or inertial detection is acquired.
12. A joint angle prediction system, characterized in that: include: Get module; Used to obtain muscle deformation data S(t) and joint angle data y(t) of different users during joint flexion and extension, wherein the muscle deformation data S(t) and the joint angle data y(t) are respectively a set of data related to time t, and the muscle deformation data S(t) and the joint angle data y(t) are obtained synchronously; A training module; used for forming a training data set from the muscle deformation data S(t) and the joint angle data y(t), inputting the training data set into a time series neural network model for training, and obtaining a trained time series neural network model; Prediction module: used to collect muscle deformation data S within a predetermined time step T during the joint flexion and extension process of the user j to be predicted j (t), the muscle deformation data S j (t) Input the trained time series neural network model to obtain the joint angle data y within the predetermined time step T j (t).
13. The joint angle prediction system according to claim 12, characterized in that: It also includes a data processing module for identifying possible abnormal motion states based on joint angles and change trends.
14. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores a computer program, and when the computer program is executed on a processor, the processor is enabled to execute the joint angle prediction method according to any one of claims 1 to 11.
15. An electronic device, characterized in that: include: at least one memory for storing a program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the joint angle prediction method as described in any one of claims 1 to 11.