Human body motion signal labeling method, device and computing equipment

Abstract

The present application provides a method, device and computing equipment for labeling human motion signals, which relates to the technical field of human behavior analysis, wherein the method includes: obtaining an inertial measurement unit IMU signal and a target motion signal of a target joint, and labeling the target motion signal according to the IMU signal, obtaining a labeling result of the target motion signal, wherein the labeling result includes a corresponding relationship between the target motion signal and the joint torque, and the joint torque is determined based on the IMU signal. The technical solution provided by the present application, because the change of the joint torque is not affected by individual differences or action differences, therefore, the target motion signal is labeled with the joint torque, which can improve the accuracy of the labeling result; moreover, any test subject can be selected, and the test subject can also be allowed to perform any action of any duration, so it has a wide range of applications; in addition, there is no need for repeated testing and manual labeling, so the labeling efficiency can be improved.

Description

Technical Field

[0001] The present application relates to human behavior analysis technology, and in particular to a method, device and computing equipment for labeling human motion signals, and belongs to the technical field of human behavior analysis signal processing. Background Art

[0002] With the development of microelectronics technology, the development of human behavior analysis technology has been driven. Human behavior analysis technology is currently widely used in medical monitoring, auxiliary health treatment, humanoid robots and motion prediction.

[0003] Human behavior analysis technology is a technology that identifies human behavior, converts abstract behavior into specific data, and then analyzes the data. The key to this technology is the process of converting behavior into data. Current human behavior analysis technology often uses motion signals as the data to be analyzed. For example, the motion signal is the electromyographic signal generated when the muscle moves. In order to obtain high-quality electromyographic signals, it is necessary to annotate the electromyographic signals and determine the corresponding actions and start and end points of the electromyographic signals.

[0004] The existing EMG signal labeling work mainly relies on personal experience to judge the start and end of each EMG signal. However, the completion of an action requires the participation of multiple muscles, and the force of different muscles varies greatly. Therefore, there will be a large error in judging the start and end of each EMG signal based solely on personal experience. Many tests are required to accurately find the start and end times corresponding to the action. This leads to the problems of inefficiency, time-consuming and labor-intensive existing data labeling work. Summary of the invention

[0005] In view of this, the present application provides a method, apparatus and computing device for labeling human motion signals, so as to improve the efficiency of data labeling work.

[0006] In order to achieve the above-mentioned purpose, in a first aspect, an embodiment of the present application provides a method for labeling a human motion signal, comprising:

[0007] Obtain the inertial measurement unit (IMU) signal of the target joint and the target motion signal;

[0008] The target motion signal is labeled according to the IMU signal to obtain a labeling result of the target motion signal. The labeling result includes a corresponding relationship between the target motion signal and the joint torque, and the joint torque is determined based on the IMU signal.

[0009] Optionally, the target motion signal is labeled according to the IMU signal to obtain a labeling result of the target motion signal, including:

[0010] The IMU signal and the target motion signal are input into a pre-trained motion signal labeling model to obtain the labeling result of the target motion signal. The motion signal labeling model is used to determine the joint torque according to the IMU signal, and label the target motion signal of the target joint based on the joint torque.

[0011] Optionally, the motion signal labeling model is trained based on a training sample set, the training sample set includes multiple training samples, each training sample includes a sample IMU signal, a sample plantar pressure signal, a sample target motion signal and a labeling result of the sample target motion signal, the labeling result of the sample target motion signal includes the correspondence between the sample target motion signal and the joint torque, and the joint torque corresponding to the sample target motion signal is determined based on the sample IMU signal and the sample plantar pressure signal.

[0012] Optionally, the target motion signal is labeled according to the IMU signal to obtain a labeling result of the target motion signal, including:

[0013] Determine the joint torque of the target joint according to the IMU signal and the corresponding plantar pressure signal;

[0014] Annotate the target motion signal of the target joint according to the joint torque.

[0015] Optionally, determining the joint torque of the target joint according to the IMU signal and the plantar pressure signal includes:

[0016] The joint angle of the target joint is determined according to a predetermined attitude rotation matrix and an IMU signal. The attitude rotation matrix is ​​a conversion matrix between an inertial coordinate system and a joint coordinate system. The joint coordinate system is a coordinate system established based on the target joint. The joint angle is a motion angle of the target joint in the joint coordinate system.

[0017] Determine the joint angular velocity and joint angular acceleration of the target joint according to the joint angle;

[0018] The joint torque of the target joint is determined based on the inverse dynamics model according to the plantar pressure signal, joint angle, joint angular velocity and joint angular acceleration.

[0019] Optionally, the target motion signal is an electromyography signal or an acceleration signal.

[0020] In a second aspect, an embodiment of the present application provides a human motion signal labeling device, including:

[0021] An acquisition module is used to acquire an inertial measurement unit (IMU) signal of a target joint and a target motion signal;

[0022] The labeling module is used to label the target motion signal according to the IMU signal to obtain the labeling result of the target motion signal, wherein the labeling result includes the corresponding relationship between the target motion signal and the joint torque, and the joint torque is determined based on the IMU signal.

[0023] Optionally, the annotation module is specifically used to:

[0024] The IMU signal and the target motion signal are input into a pre-trained motion signal labeling model to obtain the labeling result of the target motion signal. The motion signal labeling model is used to determine the joint torque according to the IMU signal, and label the target motion signal of the target joint based on the joint torque.

[0025] Optionally, the motion signal labeling model is trained based on a training sample set, the training sample set includes multiple training samples, each training sample includes a sample IMU signal, a sample plantar pressure signal, a sample target motion signal and a labeling result of the sample target motion signal, the labeling result of the sample target motion signal includes the correspondence between the sample target motion signal and the joint torque, and the joint torque corresponding to the sample target motion signal is determined based on the sample IMU signal and the sample plantar pressure signal.

[0026] Optionally, the annotation module is specifically used to:

[0027] Determine the joint torque of the target joint according to the IMU signal and the corresponding plantar pressure signal;

[0028] Annotate the target motion signal of the target joint according to the joint torque.

[0029] Optionally, the annotation module is specifically used to:

[0030] The joint angle of the target joint is determined according to a predetermined attitude rotation matrix and an IMU signal. The attitude rotation matrix is ​​a conversion matrix between an inertial coordinate system and a joint coordinate system. The joint coordinate system is a coordinate system established based on the target joint. The joint angle is a motion angle of the target joint in the joint coordinate system.

[0031] Determine the joint angular velocity and joint angular acceleration of the target joint according to the joint angle;

[0032] The joint torque of the target joint is determined based on the inverse dynamics model according to the plantar pressure signal, joint angle, joint angular velocity and joint angular acceleration.

[0033] Optionally, the target motion signal is an electromyography signal or an acceleration signal.

[0034] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, the memory being used to store a computer program; the processor being used to execute the method described in the first aspect or any one of the implementations of the first aspect when calling the computer program.

[0035] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in the first aspect or any one of the embodiments of the first aspect is implemented.

[0036] The marking method of human motion signal provided by the embodiment of the present application can obtain the inertial measurement unit IMU signal and the target motion signal of the target joint, and mark the target motion signal according to the IMU signal, and obtain the marking result of the target motion signal, and the marking result includes the corresponding relationship between the target motion signal and the joint torque, and the joint torque is determined based on the IMU signal. Since the change of the joint torque is not affected by individual differences or action differences, therefore, the accuracy of the marking result can be improved by marking the target motion signal with the joint torque; moreover, any testee can be selected, and any action of any duration can be allowed to be performed by the testee, so that the scope of application is wide; in addition, there is no need for repeated testing and manual marking, so that the marking efficiency can be improved. In addition, when the target motion signal is an electromyographic signal, when marking, the electromyographic signal of the target joint can be marked according to the joint torque based on the predetermined advance marking amount, so that the accuracy of the marking result can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of a flow chart of a method for labeling a human motion signal provided in an embodiment of the present application;

[0038] Figure 2 A schematic diagram of a method for determining joint torque provided in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of myoelectric signal annotation provided in an embodiment of the present application;

[0040] Figure 4 A schematic diagram for comparing the advance marking quantities provided in the embodiments of the present application;

[0041] Figure 5 A schematic diagram of the structure of a human motion signal labeling device provided in an embodiment of the present application;

[0042] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] At present, the annotation of motion signals is mainly based on personal experience to determine the start and end of each motion signal. However, the completion of an action requires the participation of multiple muscles, and the force of different muscles varies greatly. In addition, the time when muscles in different parts generate motion signals is also different. Therefore, multiple tests are required to accurately find the start and end time corresponding to the action. On the other hand, since different test subjects have different habits when doing actions, this will also affect the force of the muscles. Therefore, in order to accurately annotate the motion signal, only one person is often used as the test subject, and the actions during the collection are all standard actions. In order to reduce the differences between actions, the test subject cannot do the action for a long time, so there are great limitations.

[0044] In order to solve the above problems, the embodiment of the present application provides a method for labeling human motion signals. The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0045] The human motion signal labeling method provided in the embodiment of the present application can be applied to electronic devices such as laptops, tablet computers or mobile terminals, and the embodiment of the present application does not impose any restrictions on the specific type of electronic devices. The electromyographic signals referred to in the embodiment of the present application are all surface electromyographic signals.

[0046] Figure 1 A schematic diagram of a flow chart of a method for labeling a human motion signal provided in an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0047] S110, obtaining an IMU signal of a target joint and a target motion signal.

[0048] The electronic device can obtain IMU signals and target motion signals from various sensors placed on the subject, or the user can input the IMU signals and target motion signals of the target joints into the electronic device after collecting them through sensors. In this embodiment, the IMU signals and target motion signals obtained by the electronic device from the sensors are used as an example for illustrative description. Among them, the target motion signal may include the electromyographic signal of the target joint, and may also include the acceleration signal of the target joint. The target joint may include the hip joint, knee joint and / or ankle joint of the legs of the subject.

[0049] Specifically, before obtaining the IMU signal and target motion signal of the target joint, various sensors for testing can be placed on the body of the test subject, which may include an IMU sensor for obtaining IMU signals, a pressure sensor for obtaining plantar pressure signals, an acceleration sensor for obtaining acceleration signals, and an electromyography sensor for obtaining electromyography signals.

[0050] For example, the electromyographic sensor can be fixed on the surface of the rectus femoris, vastus lateralis, biceps femoris, gastrocnemius and tibialis anterior of the lower limbs of the testee to obtain the electromyographic signals of the muscles of the lower limbs; the IMU sensor can also be placed on the thigh, calf and foot of the testee to obtain the angle signal of the hip joint movement, the angle signal of the knee joint movement and the angle signal of the ankle joint movement of the testee's lower limbs; the pressure sensor can also be placed on the sole of the foot of the testee to obtain the plantar pressure information of the testee. After the electromyographic sensor, pressure sensor and IMU sensor are placed on the lower limbs of the testee as required, the testee can perform corresponding actions, such as walking, running, jumping, etc. At this time, the electronic device can obtain a large number of electromyographic signals of the rectus femoris, vastus lateralis, biceps femoris, gastrocnemius and tibialis anterior, angle signals of the thigh, calf and foot, and plantar pressure signals.

[0051] S120, labeling the target motion signal according to the IMU signal to obtain a labeling result of the target motion signal.

[0052] During the labeling process, the electronic device can determine the joint torque of the test subject based on the IMU signal, and then label the target motion signal based on the joint torque, that is, to establish a corresponding relationship between the target motion signal and the joint torque. The joint torque represents the force condition of the target joint. Since the positional relationship between the joints will change when the target joint moves, and the joint torque will also change when the positional relationship between the joints changes, the joint torque has a corresponding relationship with the motion signal of the target joint. Furthermore, since the change of joint torque is not affected by individual differences or movement differences, the use of joint torque to label motion signals can improve the accuracy of the labeling results; moreover, any test subject can be selected, and the test subject can be allowed to perform any action for any duration, so it has a wide range of applications.

[0053] Specifically, the electronic device can first determine the joint torque according to the inverse dynamics model, and then label the target motion signal according to the joint torque; it can also first determine the joint torque according to a neural network model, and then label the target motion signal according to another neural network model; it can also directly use a neural network model to label the target motion signal according to the IMU signal to obtain the labeling result of the target motion signal. The specific implementation process is as follows:

[0054] The first one is to determine the joint torque of the target joint based on the inverse dynamics model, and then label the target motion signal according to the joint torque.

[0055] The first step is to determine the joint torque of the target joint based on the IMU signal and the corresponding plantar pressure signal.

[0056] Figure 2 A schematic flow chart of a method for determining joint torque provided in an embodiment of the present application is shown in FIG. Figure 2As shown, the method comprises the following steps:

[0057] S121. Determine the joint angle of the target joint according to the predetermined posture rotation matrix and the IMU signal.

[0058] As mentioned above, the IMU signal is acquired by the IMU sensor, and the motion angle of the target joint in the inertial coordinate system can be obtained according to the signal; the joint angle of the target joint is the motion angle of the target joint in the joint coordinate system. Before determining the joint angle of the target joint, the joint coordinate system can be established in advance with the target joint as the reference, and the attitude rotation matrix can be determined according to the relative position relationship between the IMU sensor and the target joint. Among them, the attitude rotation matrix is ​​the conversion matrix between the inertial coordinate system and the joint coordinate system.

[0059] For example, after placing all kinds of sensors, the tester can use the hip joint, knee joint and ankle joint as the reference to establish the joint coordinate system of the hip joint, the joint coordinate system of the knee joint and the joint coordinate system of the ankle joint respectively, and determine the posture rotation matrix corresponding to each target joint according to the relative position relationship between the sensor and each target joint.

[0060] After completing the above preparations, the electronic device can determine the joint angle of the target joint based on the determined posture rotation matrix and the IMU signal.

[0061] S122. Determine the joint angular velocity and joint angular acceleration of the target joint according to the joint angle.

[0062] The electronic device can perform first-order algebraic difference operations and second-order algebraic difference operations on the joint angles of the target joints to obtain the joint angular velocity and joint angular acceleration of the target joints.

[0063] S123. Determine the joint torque of the target joint based on an inverse dynamics model according to the plantar pressure signal, joint angle, joint angular velocity and joint angular acceleration.

[0064] The joints of the human body can be regarded as rigid bodies, and the bones can be regarded as rods, so the lower limbs of the subject can be regarded as a multi-link structure. Therefore, the electronic device can use the Lagrange method or the Newton-Euler method to establish an inverse dynamics model. In the embodiment of the present application, the Newton-Euler method is used as an example to establish an inverse dynamics model.

[0065] The inverse dynamics model can be expressed as follows:

[0066]

[0067] Where q represents the joint angle of the target joint, represents the joint angular velocity of the target joint, represents the joint angular acceleration of the target joint, A(q) represents the inertia matrix of the target joint, represents the offset torque, which includes centripetal force and Coriolis force, external force and external torque, muscle arm matrix and gravity torque, F represents the plantar pressure signal, and T represents the joint torque of the target joint.

[0068] The electronic device can determine the joint torque of the target joint based on the above-mentioned inverse dynamics model according to the plantar pressure signal, the joint angle, the joint angular velocity and the joint angular acceleration.

[0069] The second step is to label the target motion signal of the target joint according to the joint torque.

[0070] Since joint torque is the expression of the force between joints, the joint torque will also change with the change of the action when the subject is doing the action. In this way, the motion signal of the corresponding target joint can be annotated with the joint torque, and the starting and ending points of the motion signal of the corresponding action can be determined based on the change of the joint torque.

[0071] Take electromyographic signal as an example. Figure 3 The electromyographic signal annotation schematic diagram provided in the embodiment of the present application is as follows: Figure 3 As shown, Figure 3 (a) is the curve of the change of the joint torque of the knee joint. Figure 3 (b) is the curve of the change of the electromyographic signal of the knee joint. Figure 3 As shown in (b) in the figure, when the subject changes from a standing posture to a squatting posture, the electromyographic signal generated by the muscles near the knee joint will change; at the same time, the force applied to the knee joint of the subject when standing is inconsistent with that applied to the knee joint when squatting. Figure 3 As shown in (a) in the figure, when the subject changes from a standing posture to a squatting posture, the joint torque of the knee joint will also change. That is, the joint torque can reflect the changes in human body movements, and there is a correlation between it and the electromyographic signal. The corresponding electromyographic signal can be annotated with the joint torque. Figure 3 As shown, point A is the starting change point of the joint torque, point B is the ending change point of the joint torque, the electromyographic signal at the corresponding moment of point A is point C, and the electromyographic signal at the corresponding moment of point B is point D, that is, the joint torque of point A can be used to mark the electromyographic signal of point C, and the joint torque of point B can be used to mark the electromyographic signal of point D, wherein point A and point B are the starting and ending points of the joint torque corresponding to the above action, and correspondingly, point C and point D are the starting and ending points of the electromyographic signal corresponding to the above action.

[0072] According to research, myoelectric signals are generated during the contraction and relaxation of muscle movement, and the generation of myoelectric signals is about 30-150 milliseconds earlier than the muscle force. In order to improve the accuracy of the labeling results, in an embodiment of the present application, the electronic device can label the myoelectric signals of the target joint based on a predetermined advance labeling amount and joint torque. Among them, the advance labeling amount is the generation time difference between the joint torque and the myoelectric signal.

[0073] Specifically, the joint torque is determined based on the IMU signal, so the generation time difference between the joint torque and the electromyographic signal can be determined based on the generation time difference between the IMU signal and the electromyographic signal. Furthermore, there is a time difference in the generation time between the IMU signal and the electromyographic signal, and correspondingly, there is also a corresponding time difference in the end time between the IMU signal and the electromyographic signal. Therefore, the electronic device can also determine the advance annotation amount based on the end time between the IMU signal and the electromyographic signal corresponding to the target action.

[0074] Specifically, during the test, the subject can stop for a while after each action and then proceed to the next action. Figure 4 A schematic diagram for comparing the advance marking amounts provided in the embodiments of the present application is shown in FIG. Figure 4 As shown, Figure 4 (a) is the change curve of the IMU signal of the knee joint. Figure 4 (b) in FIG. 1 is the curve of the change of the electromyographic signal of the knee joint. The subject can first step with the left leg, then stop for 1 second, then step with the right leg, and then stop for 1 second. Since the electromyographic signal collected by the electromyographic sensor will end first after the action stops, and then the IMU signal collected by the IMU sensor will also end, so point E is the end point of the IMU signal when the subject steps with the left leg, point F is the end point of the IMU signal when the subject steps with the right leg, point G is the end point of the electromyographic signal when the subject steps with the left leg, and point H is the end point of the electromyographic signal when the subject steps with the right leg. In this way, the end time difference between the electromyographic signal and the IMU signal can be determined according to points E and G, or according to points F and H, and then the marked lead time corresponding to the leg step action can be determined. In addition, in order to obtain the marked lead time more accurately, the subject can also be asked to repeat the same action multiple times, and then take the average of multiple marked lead times as the marked lead time, which will not be repeated here.

[0075] After the advance annotation amount is determined, the electronic device can annotate the electromyographic signal of the target joint according to the advance annotation amount and the joint torque.

[0076] Specifically, the electronic device can determine the starting and ending points of the electromyographic signal according to the pre-marked amount. Figure 3Point A is the starting change point of the joint torque, point B is the ending change point of the joint torque, and the advance marking amount predetermined by the electronic device is 50 milliseconds. The starting point of the electromyographic signal is point C' and the end point is point D'.

[0077] The second method determines the joint torque of the target joint based on a neural network model, and then labels the target motion signal according to another neural network model.

[0078] The first step is to determine the joint torque of the target joint based on the joint torque prediction model.

[0079] In this embodiment, the joint torque prediction model can also be pre-trained according to the training samples of the joint torque prediction model. When determining the joint torque, the electronic device can input the IMU signal into the pre-trained joint torque prediction model to obtain the joint torque corresponding to the target joint.

[0080] Specifically, the training samples of the joint torque prediction model may include sample IMU signals and sample joint torques determined according to the sample IMU signals, wherein the sample joint torques may be obtained based on a method of an inverse dynamics model.

[0081] The training process of the joint torque prediction model may include: the electronic device may extract the signal features of the sample IMU signal in each training sample, and then train the neural network model to be trained according to the signal features of the sample IMU signal and the sample joint torque in each training sample to obtain the joint torque prediction model. Among them, the signal features of the sample IMU signal can be extracted by Fourier transform method or wavelet transform method; the neural network model can be a deep feedforward neural network, a recursive neural network, a deep convolutional network, etc., which is not particularly limited in this embodiment.

[0082] It should be noted that when the joint torque is determined using a pre-trained joint torque prediction model, the electronic device may no longer acquire the plantar pressure signal of the test subject in step S110.

[0083] In an embodiment of the present application, by adopting a joint torque prediction model, the electronic device can directly obtain the joint torque based on the IMU signal, eliminating the steps of processing the IMU signal and the solution process based on the inverse dynamics model, making the entire target motion signal labeling work more efficient.

[0084] The second step is to label the target motion signal of the target joint according to the labeling model.

[0085] For greater convenience and speed, in this embodiment, a labeling model may be trained first, and then the joint torque and target motion signal may be input into the trained labeling model to obtain a labeled target motion signal.

[0086] Specifically, the labeling model can be trained based on a training sample set, which may include multiple training samples. Each training sample may include a sample target motion signal, a sample joint torque, and a labeling result of the sample target motion signal. The labeling result of the sample target motion signal can be determined based on the sample target motion signal, the sample joint torque, and a predetermined advance labeling amount.

[0087] The training process of the annotation model may include: the electronic device may extract the signal features of the sample target motion signal in each training sample, and then train the neural network model to be trained according to the signal features of the sample target motion signal in each training sample, the sample joint torque and the annotation results of the sample target motion signal to obtain the annotation model. The signal features of the sample target motion signal may be extracted by Fourier transform or wavelet transform; the neural network model may be a deep feedforward neural network, a recursive neural network, a deep convolutional network, etc., which is not particularly limited in this embodiment.

[0088] Correspondingly, when labeling, the electronic device can first extract the signal features of the target motion signal, and then input the signal features of the joint torque and the target motion signal into the pre-trained labeling model to obtain the labeled target motion signal. By adopting the labeling model, the electronic device can directly obtain the labeled target motion signal based on the joint torque and the target motion signal, without the need to obtain the advance labeling amount while ensuring the accuracy of the labeling result, making the entire target motion signal labeling work more efficient.

[0089] The third method is to label the target motion signal according to the IMU signal based on a neural network model to obtain the labeling result of the target motion signal.

[0090] In an embodiment of the present application, the motion signal labeling model can also be pre-trained based on the training samples of the motion signal labeling model. Then, the electronic device can directly input the IMU signal and the target motion signal into the pre-trained motion signal labeling model to obtain the labeling result of the target motion signal, wherein the motion signal labeling model is used to determine the joint torque based on the IMU signal, and label the target motion signal of the target joint based on the joint torque.

[0091] Specifically, the motion signal labeling model can be trained based on a training sample set, and the training sample set may include multiple training samples. Each training sample may include a sample IMU signal, a sample plantar pressure signal, a sample target motion signal, and a labeling result of the sample target motion signal, wherein the labeling result of the sample target motion signal includes the correspondence between the sample target motion signal and the joint torque, and the joint torque corresponding to the sample target motion signal is determined based on the sample IMU signal and the sample plantar pressure signal. The specific determination method of the labeling result of the sample target motion signal can refer to the above two schemes.

[0092] The training process of the motion signal annotation model may include: the electronic device may extract the signal features of the sample IMU signal and the sample target motion signal in each training sample, and then train the neural network model to be trained according to the signal features of the sample IMU signal, the signal features of the sample target motion signal and the annotation results of the sample target motion signal in each training sample to obtain the motion signal annotation model. Among them, the signal features of the sample IMU signal and the sample target motion signal can be extracted by Fourier transform method or wavelet transform method; the neural network model can be a deep feedforward neural network, a recursive neural network, a deep convolutional network, etc., which is not particularly limited in this embodiment.

[0093] Correspondingly, when labeling, the electronic device can first extract the signal features of the IMU signal and the target motion signal, and then input the signal features of the IMU signal and the target motion signal into the pre-trained motion signal labeling model to obtain the labeling result of the target motion signal. By adopting the motion signal labeling model, the electronic device can directly obtain the labeled target motion signal based on the IMU signal and the target motion signal, while ensuring the accuracy of the labeling result without determining the joint torque in advance, making the entire target motion signal labeling work more efficient.

[0094] The human motion signal labeling method provided by the present application can obtain the IMU signal and the target motion signal of the target joint, and label the target motion signal according to the IMU signal to obtain the labeling result of the target motion signal, and the labeling result includes the corresponding relationship between the target motion signal and the joint torque. Since the change of the joint torque is not affected by individual differences or action differences, the accuracy of the labeling result can be improved by labeling the target motion signal with the joint torque; moreover, any test subject can be selected, and the test subject can be allowed to perform any action of any duration, so the application range is wide; in addition, there is no need for repeated testing and manual labeling, so the labeling efficiency can be improved; and based on the correlation between the joint torque and various motion signals of the human body, the labeling of multiple human motion signals can be realized.

[0095] In addition, when the target motion signal is an electromyographic signal, during labeling, the electromyographic signal of the target joint can be labeled according to the joint torque based on the predetermined advance labeling amount, which can further improve the accuracy of the labeling result.

[0096] Based on the same inventive concept, as an implementation of the above method, an embodiment of the present application provides a human motion signal labeling device. The device embodiment corresponds to the above method embodiment. For ease of reading, the device embodiment will no longer repeat the details of the above method embodiment one by one, but it should be clear that the device in this embodiment can correspond to all the contents of the above method embodiment.

[0097] Figure 5 A schematic diagram of a human motion signal labeling device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the device provided in this embodiment includes:

[0098] An acquisition module 110 is used to acquire an inertial measurement unit IMU signal of a target joint and a target motion signal;

[0099] The labeling module 120 is used to label the target motion signal according to the IMU signal to obtain the labeling result of the target motion signal, wherein the labeling result includes the corresponding relationship between the target motion signal and the joint torque, and the joint torque is determined based on the IMU signal.

[0100] Optionally, the marking module 120 is specifically used for:

[0101] The IMU signal and the target motion signal are input into a pre-trained motion signal labeling model to obtain the labeling result of the target motion signal. The motion signal labeling model is used to determine the joint torque according to the IMU signal, and label the target motion signal of the target joint based on the joint torque.

[0102] Optionally, the motion signal labeling model is trained based on a training sample set, the training sample set includes multiple training samples, each training sample includes a sample IMU signal, a sample plantar pressure signal, a sample target motion signal and a labeling result of the sample target motion signal, the labeling result of the sample target motion signal includes the correspondence between the sample target motion signal and the joint torque, and the joint torque corresponding to the sample target motion signal is determined based on the sample IMU signal and the sample plantar pressure signal.

[0103] Optionally, the marking module 120 is specifically used for:

[0104] Determine the joint torque of the target joint according to the IMU signal and the corresponding plantar pressure signal;

[0105] Annotate the target motion signal of the target joint according to the joint torque.

[0106] Optionally, the marking module 120 is specifically used for:

[0107] The joint angle of the target joint is determined according to a predetermined attitude rotation matrix and an IMU signal. The attitude rotation matrix is ​​a conversion matrix between an inertial coordinate system and a joint coordinate system. The joint coordinate system is a coordinate system established based on the target joint. The joint angle is a motion angle of the target joint in the joint coordinate system.

[0108] Determine the joint angular velocity and joint angular acceleration of the target joint according to the joint angle;

[0109] The joint torque of the target joint is determined based on the inverse dynamics model according to the plantar pressure signal, joint angle, joint angular velocity and joint angular acceleration.

[0110] Optionally, the target motion signal is an electromyography signal or an acceleration signal.

[0111] The device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0112] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0113] Based on the same inventive concept, an embodiment of the present application also provides an electronic device. Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the electronic device provided in this embodiment includes: a memory 21 and a processor 20, the memory 21 is used to store a computer program 23; the processor 20 is used to execute the method described in the above method embodiment when calling the computer program 23.

[0114] The electronic device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0115] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method described in the above method embodiment is implemented.

[0116] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, USB flash drive, mobile hard disk, disk or optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0117] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0118] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0119] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0120] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0121] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0122] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0123] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0124] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for labeling human motion signals, characterized in that: include: Obtain the inertial measurement unit (IMU) signal of the target joint and the target motion signal; Determine the joint torque of the target joint according to the IMU signal and the corresponding plantar pressure signal, and annotate the target motion signal of the target joint according to the joint torque to obtain an annotation result of the target motion signal; Alternatively, the IMU signal and the target motion signal are input into a pre-trained motion signal annotation model to obtain an annotation result of the target motion signal, wherein the motion signal annotation model is used to determine the joint torque according to the IMU signal, and annotate the target motion signal of the target joint based on the joint torque; The annotation result includes the start and end points corresponding to the target action in the target motion signal, and there is a corresponding time difference between the corresponding moments of the start and end points and the corresponding moments of the start and end change points of the joint torque.

2. The method according to claim 1, characterized in that: The motion signal annotation model is obtained through training based on a training sample set, wherein the training sample set includes multiple training samples, each training sample includes a sample IMU signal, a sample plantar pressure signal, a sample target motion signal and an annotation result of the sample target motion signal, the annotation result of the sample target motion signal includes a correspondence between the sample target motion signal and the joint torque, and the joint torque corresponding to the sample target motion signal is determined based on the sample IMU signal and the sample plantar pressure signal.

3. The method according to claim 1, characterized in that The determining the joint torque of the target joint according to the IMU signal and the corresponding plantar pressure signal comprises: Determine the joint angle of the target joint according to a predetermined attitude rotation matrix and the IMU signal, wherein the attitude rotation matrix is ​​a conversion matrix between an inertial coordinate system and a joint coordinate system, the joint coordinate system is a coordinate system established with the target joint as a reference, and the joint angle is a motion angle of the target joint in the joint coordinate system; Determining a joint angular velocity and a joint angular acceleration of the target joint according to the joint angle; The joint torque of the target joint is determined based on an inverse dynamics model according to the plantar pressure signal, the joint angle, the joint angular velocity and the joint angular acceleration.

4. The method according to any one of claims 1 to 3, characterized in that: The target motion signal is an electromyographic signal or an acceleration signal.

5. A human motion signal labeling device, characterized in that: include: An acquisition module is used to acquire an inertial measurement unit (IMU) signal of a target joint and a target motion signal; A labeling module, used to determine the joint torque of the target joint according to the IMU signal and the corresponding plantar pressure signal, label the target motion signal of the target joint according to the joint torque, and obtain the labeling result of the target motion signal; or, input the IMU signal and the target motion signal into a pre-trained motion signal labeling model to obtain the labeling result of the target motion signal, the motion signal labeling model is used to determine the joint torque according to the IMU signal, and label the target motion signal of the target joint based on the joint torque; The annotation result includes the start and end points corresponding to the target action in the target motion signal, and there is a corresponding time difference between the corresponding moments of the start and end points and the corresponding moments of the start and end change points of the joint torque.

6. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the method according to any one of claims 1 to 4 when calling the computer program.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

8. A chip system, characterized in that: The chip system includes a processor, which is coupled to a memory, and the processor executes a computer program stored in the memory to implement the method according to any one of claims 1 to 4.

Images