Joint angle estimation method and device based on visual and surface electromyogram signals

Abstract

The invention discloses a joint angle estimation method and device based on visual and surface electromyogram signals. The method comprises the following steps: capturing an action video stream; capturing an electromyographic signal flow; extracting primary three-channel visual features according to the action video stream; capturing electromyographic characteristics according to the electromyographic signal flow; performing oversampling on the primary three-channel visual features so as to obtain secondary three-channel visual features with the same length as the myoelectricity features; inputting the secondary three-channel visual features and the myoelectricity features into a trained LSTM model for prediction, wherein the trained LSTM model is used for mapping internal parameters to three joint angles of a shoulder, an elbow and a wrist; and obtaining the output of the trained LSTM model as predicted values of the angles of the three joints of the shoulder, the elbow and the wrist. According to the method, the influence of multi-modal sensing data on continuous arm joint angle estimation is researched, and the precision and stability of joint angle continuous estimation based on surface muscle electric signals can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to a method and device for estimating joint angles based on vision and surface electromyography signals. Background Art

[0002] At present, when using deep learning models to continuously estimate human joint angles, the deep learning models are usually trained only through the characteristics of surface muscle electrical signals, so as to continuously predict human joint angles. In human-computer interaction systems, continuous estimation methods play a vital role in achieving natural and intuitive interactions, and this method has more advantages than pattern recognition methods. Surface electromyography signals are the most common basis for continuous estimation of joint angles. However, surface electromyography signals have disadvantages such as low signal-to-noise ratio and insensitivity to small arm movements, which restricts the improvement of the accuracy of joint angle estimation applications. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present invention studies the influence of multimodal perception data on continuous arm joint angle estimation, and proposes a method and device combining monocular vision features to improve the accuracy and stability of continuous joint angle estimation based on surface muscle electrical signals.

[0004] The technical means adopted by the present invention are as follows:

[0005] A multimodal human joint angle continuous estimation method based on monocular vision and surface electromyography signals comprises the following steps:

[0006] Capture the action video stream through the camera; capture the electromyography signal stream through the electromyography sensor;

[0007] Extract primary three-channel visual features based on the action video stream; capture electromyographic features based on the electromyographic signal stream;

[0008] Supersampling the primary three-channel visual features, wherein the supersampling frequency is determined according to the length of the electromyographic features, so as to obtain secondary three-channel visual features of the same length as the electromyographic features;

[0009] Inputting the secondary three-channel visual features and electromyographic features into the trained LSTM model for prediction, wherein the trained LSTM model is used to map internal parameters to three joint angles of shoulder, elbow and wrist;

[0010] The output of the trained LSTM model is obtained as the predicted values ​​of the three joint angles of shoulder, elbow and wrist.

[0011] Furthermore, primary three-channel visual features are extracted based on the action video stream, including:

[0012] Acquire a frame image set from the action video stream;

[0013] The plane coordinates of the shoulder, elbow, and wrist in each frame of the image are obtained through the posture estimation model;

[0014] The plane angles of the shoulder, elbow and wrist in each frame of image are calculated according to the plane coordinates of the shoulder, elbow and wrist in each frame of image;

[0015] The plane angles of the shoulder, elbow, and wrist are merged according to the timestamp to obtain a primary three-channel visual feature.

[0016] Furthermore, the electromyographic features are captured according to the electromyographic signal stream, including:

[0017] Capture the EMG signal stream through a four-channel EMG sensor;

[0018] Aligning the electromyographic signal stream with the action video stream according to the timestamp;

[0019] The feature extraction of electromyographic signal is carried out by RMS method to obtain four-channel electromyographic features.

[0020] Furthermore, the primary three-channel visual features are super-sampled, including:

[0021] According to the length of the EMG feature, the visual feature is super-sampled using linear interpolation to obtain a secondary three-channel visual feature with the same length as the EMG feature.

[0022] Furthermore, the secondary three-channel visual features and electromyographic features are input into the trained LSTM model for prediction, including:

[0023] The secondary three-channel visual features and electromyographic features are input into the trained LSTM model through a sliding window. The LSTM model is set to three layers, each layer contains 64 hidden units, and the last fully connected layer with an input of 64 and an output of 3 maps the internal parameters of the network to the three joint angles of the shoulder, elbow, and wrist.

[0024] Furthermore, the size of the sliding window is 100 ms, and the sliding length is 100 ms.

[0025] The present invention also discloses a multi-modal human joint angle continuous estimation device based on monocular vision and surface electromyography signals, comprising:

[0026] The acquisition unit is used to capture the action video stream through the camera and capture the electromyographic signal stream through the electromyographic sensor;

[0027] A feature extraction unit, which is used to extract primary three-channel visual features according to the action video stream; and capture electromyographic features according to the electromyographic signal stream;

[0028] A supersampling unit, which is used to supersample the primary three-channel visual features, wherein the supersampling frequency is determined according to the length of the electromyographic features, so as to obtain a secondary three-channel visual feature of the same length as the electromyographic features;

[0029] A prediction unit, which is used to input the secondary three-channel visual features and electromyographic features into a trained LSTM model for prediction, and the trained LSTM model is used to map internal parameters to three joint angles of shoulder, elbow and wrist;

[0030] The output unit is used to obtain the output of the trained LSTM model as the predicted values ​​of the three joint angles of the shoulder, elbow and wrist.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] The present invention proposes a method and device for continuous estimation of multimodal human joint angles based on monocular vision and surface electromyography signals. First, the features of each modality are extracted and aligned, and then the features of each modality are input into a deep learning model through a sliding window. Finally, the model outputs a continuous estimation value of the joint angle. Compared with the existing single-modal continuous estimation method, the present invention can improve the accuracy and stability of the model's continuous estimation of joint angles. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0034] Figure 1 This is a flow chart of a multimodal human joint angle continuous estimation method based on monocular vision and surface electromyography signals of the present invention.

[0035] Figure 2 The present invention is a schematic structural diagram of a multimodal human joint angle continuous estimation device based on monocular vision and surface electromyography signals. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] like Figure 1 As shown, the present invention provides a multimodal human joint angle continuous estimation method based on monocular vision and surface electromyography signals, comprising the following steps:

[0039] S1: Capture the action video stream through the camera; capture the electromyography signal stream through the electromyography sensor.

[0040] As a preferred implementation manner of the present invention, the input data in this embodiment are respectively a video stream from a monocular camera and an electromyographic signal stream captured by a four-channel surface electromyographic sensor.

[0041] S2: Extract primary three-channel visual features based on the action video stream; capture electromyographic features based on the electromyographic signal stream.

[0042] As a preferred implementation mode of the present invention, in this embodiment, primary three-channel visual features are extracted according to the action video stream, including:

[0043] The coordinates of the human body key points in each frame of the video stream are extracted through the posture estimation model.

[0044] The plane angle between the right shoulder and the right arm is calculated by the image coordinates of the left shoulder, right shoulder, and right elbow, recorded as the right shoulder angle. The angle between the right forearm and the right back arm is calculated by the image coordinates of the right shoulder, right elbow, and right wrist, recorded as the right elbow angle. The angle between the right hand and the right forearm is calculated by the image coordinates of the right elbow, right wrist, and right hand, recorded as the right wrist angle. The specific calculation method is as follows:

[0045] Calculation of radians: Use math.atan2 function to calculate the angle between two vectors, namely (cy-by,cx-bx) and (ay-by,ax-bx). x represents the horizontal coordinate and y represents the vertical coordinate. atan2 function returns the angle in radians. Then, calculate radians by the difference between these two angles.

[0046] Angle calculation: Convert radians to degrees and take the absolute value. This conversion can be achieved using the math.degrees function in Python.

[0047] The above right shoulder angle, right elbow angle and right wrist angle are all achieved by first calculating radians and then calculating angle. The calculation process is described below by taking the calculation of the right shoulder angle as an example.

[0048] First, assume that the image coordinates of the left shoulder, right shoulder, and right elbow are (ax, ay), (bx, by), and (cx, cy), respectively. Then, two vectors are obtained based on the coordinates of the left shoulder, right shoulder, and right elbow: vector (cy-by, cx-bx) and vector (ay-by, ax-bx), respectively. The angle between the two vectors is calculated using the math.atan2 function. Then, the radians are converted to degrees using the math.degrees function in Python.

[0049] As a preferred implementation mode of the present invention, in this embodiment, capturing myoelectric features according to the myoelectric signal stream includes:

[0050] The EMG signal stream is captured by a four-channel EMG sensor.

[0051] The electromyographic signal stream is aligned with the action video stream according to the timestamp.

[0052] The feature extraction of electromyographic signal is carried out by RMS method to obtain four-channel electromyographic features.

[0053] S3: Super-sampling the primary three-channel visual features, wherein the super-sampling frequency is determined according to the length of the electromyographic features, so as to obtain secondary three-channel visual features of the same length as the electromyographic features.

[0054] As a preferred implementation mode of the present invention, in this embodiment, each angle feature of the video stream is super-sampled by linear interpolation method according to the electromyography rms feature length and timestamp to generate a three-channel visual angle feature with the same length as the electromyography rms feature.

[0055] S4: Inputting the secondary three-channel visual features and electromyographic features into the trained LSTM model for prediction, wherein the trained LSTM model is used to map internal parameters to three joint angles of shoulder, elbow and wrist.

[0056] As a preferred embodiment of the present invention, in this embodiment, the electromyographic rms features and visual features are normalized and input into the trained LSTM multi-feature upper limb joint angle prediction model through a sliding window, including:

[0057] The secondary three-channel visual features and electromyographic features are input into the trained LSTM model through a sliding window. The LSTM model is set to three layers, each layer contains 64 hidden units, and the last fully connected layer with 64 inputs and 3 outputs maps the internal parameters of the network to the three joint angles of the shoulder, elbow, and wrist. Furthermore, the size of the sliding window is 100ms, and the sliding length is 100ms.

[0058] The normalized angle prediction values ​​of the shoulder, elbow, and wrist joints predicted in each window are restored to the actual angle values, which are the final continuously estimated shoulder, elbow, and wrist joint angle values, and the algorithm ends.

[0059] S5: Output the predicted values ​​of the shoulder, elbow, and wrist joint angles.

[0060] The present invention also discloses a multi-modal human joint angle continuous estimation device based on monocular vision and surface electromyography signals, comprising:

[0061] The acquisition unit is used to capture the action video stream through the camera and capture the electromyographic signal stream through the electromyographic sensor;

[0062] A feature extraction unit, which is used to extract primary three-channel visual features according to the action video stream; and capture electromyographic features according to the electromyographic signal stream;

[0063] A supersampling unit, which is used to supersample the primary three-channel visual features, wherein the supersampling frequency is determined according to the length of the electromyographic features, so as to obtain a secondary three-channel visual feature of the same length as the electromyographic features;

[0064] A prediction unit, which is used to input the secondary three-channel visual features and electromyographic features into a trained LSTM model for prediction, and the trained LSTM model is used to map internal parameters to three joint angles of shoulder, elbow and wrist;

[0065] The output unit is used to obtain the output of the trained LSTM model as the predicted values ​​of the three joint angles of the shoulder, elbow and wrist.

[0066] For the multimodal human joint angle continuous estimation device based on monocular vision and surface electromyography signals of the present invention, since it corresponds to the multimodal human joint angle continuous estimation method based on monocular vision and surface electromyography signals in the above embodiment, the description is relatively simple. For relevant similarities, please refer to the description of the multimodal human joint angle continuous estimation method based on monocular vision and surface electromyography signals in the above embodiment, which will not be described in detail here.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal human joint angle continuous estimation method based on monocular vision and surface electromyography signals, characterized in that: The following steps are involved: Capture the action video stream through the camera; capture the electromyography signal stream through the electromyography sensor; Extracting primary three-channel visual features from action video streams; Capturing myoelectric features according to the myoelectric signal stream; Supersampling the primary three-channel visual features, wherein the supersampling frequency is determined according to the length of the electromyographic features, so as to obtain secondary three-channel visual features of the same length as the electromyographic features; Inputting the secondary three-channel visual features and electromyographic features into the trained LSTM model for prediction, wherein the trained LSTM model is used to map internal parameters to three joint angles of shoulder, elbow and wrist; The output of the trained LSTM model is obtained as the predicted values ​​of the three joint angles of shoulder, elbow and wrist.

2. The method for continuous estimation of multimodal human joint angles based on monocular vision and surface electromyography signals according to claim 1, characterized in that: Extract primary three-channel visual features from the action video stream, including: Acquire a frame image set from the action video stream; The plane coordinates of the shoulder, elbow, and wrist in each frame of the image are obtained through the posture estimation model; The plane angles of the shoulder, elbow and wrist in each frame of image are calculated according to the plane coordinates of the shoulder, elbow and wrist in each frame of image; The plane angles of the shoulder, elbow, and wrist are merged according to the timestamp to obtain a primary three-channel visual feature.

3. The method for continuous estimation of multimodal human joint angles based on monocular vision and surface electromyography signals according to claim 1, characterized in that: Capture EMG features based on EMG signal stream, including: Capture the EMG signal stream through a four-channel EMG sensor; Aligning the electromyographic signal stream with the action video stream according to the timestamp; The feature extraction of electromyographic signal is carried out by RMS method to obtain four-channel electromyographic features.

4. The method for continuous estimation of multimodal human joint angles based on monocular vision and surface electromyography signals according to claim 1, characterized in that: Supersampling of primary three-channel visual features, including: According to the length of the EMG feature, the visual feature is super-sampled using linear interpolation to obtain a secondary three-channel visual feature with the same length as the EMG feature.

5. The method for continuous estimation of multimodal human joint angles based on monocular vision and surface electromyography signals according to claim 1, characterized in that: The secondary three-channel visual features and electromyographic features are input into the trained LSTM model for prediction, including: The secondary three-channel visual features and electromyographic features are input into the trained LSTM model through a sliding window. The LSTM model is set to three layers, each layer contains 64 hidden units, and the last fully connected layer with an input of 64 and an output of 3 maps the internal parameters of the network to the three joint angles of the shoulder, elbow, and wrist.

6. The method for continuous estimation of multimodal human joint angles based on monocular vision and surface electromyography signals according to claim 5, characterized in that: The size of the sliding window is 100 ms, and the sliding length is 100 ms.

7. A multimodal human joint angle continuous estimation device based on monocular vision and surface electromyography signals, characterized in that: include: The acquisition unit is used to capture the action video stream through the camera and capture the electromyographic signal stream through the electromyographic sensor; A feature extraction unit, which is used to extract primary three-channel visual features according to the action video stream; and capture electromyographic features according to the electromyographic signal stream; A supersampling unit, which is used to supersample the primary three-channel visual features, wherein the supersampling frequency is determined according to the length of the electromyographic features, so as to obtain a secondary three-channel visual feature of the same length as the electromyographic features; A prediction unit, which is used to input the secondary three-channel visual features and electromyographic features into a trained LSTM model for prediction, and the trained LSTM model is used to map internal parameters to three joint angles of shoulder, elbow and wrist; The output unit is used to obtain the output of the trained LSTM model as the predicted values ​​of the three joint angles of the shoulder, elbow and wrist.

Images