Climbing worker lower limb joint motion prediction method and system based on multi-task learning, terminal equipment and storage medium

Abstract

The invention discloses a climbing worker lower limb joint motion prediction method and system based on multi-task learning, terminal equipment and a storage medium, and the method comprises the steps: adding a multi-task learning module between a convolutional neural network and a long-short-term memory network of a joint prediction model, and carrying out the secondary feature extraction of the convolutional features through the multi-task learning module; more accurate data support is provided for subsequent joint motion angle prediction, the joint prediction model can perform joint motion angle prediction on a plurality of joints by using a multi-task learning module, the accuracy of a joint motion angle prediction result is improved, and the joint motion angle prediction accuracy is improved. The problem that in the prior art, a model can only predict the motion angle of a single lower limb joint and cannot comprehensively predict the motion angle of the whole lower limb joint of a human body, mutual influence between the joints is ignored, and then the prediction result of the motion angle of the lower limb joint is low in accuracy is solved.

Description

Technical Field

[0001] The present invention relates to the field of lower limb joint motion prediction, and particularly to a method, system, terminal device and storage medium for predicting the lower limb joint motion of climbing workers based on multi-task learning. Background Art

[0002] Electric power energy has become one of the indispensable energies in people's daily lives. Power outages in urban life and factory equipment will cause great inconvenience to people's production and life, and also bring economic losses. To ensure the reliability of the power supply system, power enterprises generally need to dispatch a large number of climbing workers to install, inspect, maintain, etc. the equipment. However, the working positions of climbing workers are relatively high and the working environment is relatively harsh, facing operation safety problems, and workers engaged in climbing operations for a long time may suffer from health problems such as musculoskeletal diseases and joint injuries, resulting in a further increase in the existing operation risks. Therefore, accurately predicting the lower limb joint motion of climbing workers can better analyze and prevent action patterns that may lead to accidents, and guide the design of climbing assistive exoskeletons, thereby reducing the safety risks during the operation process.

[0003] Currently, the prior art usually uses a model combining a convolutional neural network and a long short-term memory network to predict the joint motion angle of lower limb movement electromyography signal data. However, this method can only perform motion prediction for a single lower limb joint and cannot perform comprehensive motion prediction for the overall lower limb joints of the human body. For example, in a method for continuously predicting the lower limb knee joint angle based on surface electromyography signals with the patent application number 202410476207.X, this patent predicts the motion angle of the lower limb knee joint according to the electromyography signal data of multiple muscles in the lower limb. Therefore, the prior art cannot perform comprehensive motion angle prediction for the overall lower limb joints of the human body because the model can only perform motion angle prediction for a single lower limb joint, ignoring the mutual influence between joints, and thus resulting in a low accuracy of the prediction result of the lower limb joint motion angle. Summary of the Invention

[0004] The present invention provides a method, system, terminal device and storage medium for predicting the lower limb joint motion of climbing workers based on multi-task learning, which can solve the problem that the prior art cannot perform comprehensive motion angle prediction for the overall lower limb joints of the human body because the model can only perform motion angle prediction for a single lower limb joint, ignoring the mutual influence between joints, and thus resulting in a low accuracy of the prediction result of the lower limb joint motion angle.

[0005] To solve the above technical problems, an embodiment of the present invention provides a method for predicting the lower limb joint motion of climbing workers based on multi-task learning, including:

[0006] Obtain the electromyography signals of the muscle groups that control the movement of the lower limb joints of the climbers to be measured; wherein, the lower limb joints include the hip joint, the knee joint, and the ankle joint;

[0007] Extract the time-domain features of the electromyography signals of the muscle groups of the hip joint, the knee joint, and the ankle joint respectively to obtain the time-domain features of the muscle groups of the hip joint, the knee joint, and the ankle joint;

[0008] Input the extracted time-series features into the trained joint prediction model, so that the joint prediction model performs the first feature extraction on the time-domain features of the muscle groups of the hip joint, the knee joint, and the ankle joint through the built-in CNN module to obtain the convolutional features of the muscle groups of the hip joint, the knee joint, and the ankle joint; perform the second feature extraction on the convolutional features of the muscle groups of the hip joint, the knee joint, and the ankle joint through the built-in multi-task learning module to obtain the depth features of the muscle groups of the hip joint, the knee joint, and the ankle joint; based on the built-in LSTM module, predict the joint movement angles according to the depth features of the muscle groups of the hip joint, the knee joint, and the ankle joint to obtain the prediction results of the movement angles of the lower limb joints.

[0009] Further, the extracting the time-domain features of the electromyography signals of the muscle groups of the hip joint, the knee joint, and the ankle joint respectively to obtain the time-domain features of the muscle groups of the hip joint, the knee joint, and the ankle joint includes:

[0010] Calculate the time-domain features of the electromyography signals of the muscle groups of the hip joint, the knee joint, and the ankle joint respectively to obtain the time-domain features of the muscle groups of the hip joint, the knee joint, and the ankle joint; wherein, the time-domain feature calculation includes mean absolute value calculation, root mean square calculation, variance calculation, zero-crossing number calculation, waveform length calculation, and slope sign change calculation.

[0011] Further, the calculation formula of the mean absolute value is:

[0012]

[0013] wherein, MAV is the mean absolute value of the electromyography signal, which is used to describe the average value of the absolute value of the electromyography signal amplitude and represents the muscle contraction level; N is the total number of electromyography data points in each segment of the electromyography signal; x i is the i-th electromyography data in each segment of the electromyography signal;

[0014] The calculation formula of the root mean square is:

[0015]

[0016] wherein, RMS is the root mean square of the electromyography signal, which is used to describe the square root of the mean value of the electromyography signal amplitude and evaluate the muscle contraction intensity and fatigue degree; N is the total number of electromyography data points in each segment of the electromyography signal; xi The \(i\)-th EMG data in each segment of EMG signal;

[0017] The calculation formula for the variance is:

[0018]

[0019] where Var is the variance of the EMG signal, which is used to describe the dispersion degree of the EMG signal and represents the intensity of the EMG signal; N is the total number of EMG data points in each segment of the EMG signal; \(x\) i The \(i\)-th EMG data in each segment of EMG signal;

[0020] The calculation formula for the number of zero crossings is:

[0021]

[0022] where ZC is the number of zero crossings of the EMG signal, which is used for the number of times the EMG signal waveform crosses the horizontal reference line; N is the total number of EMG data points in each segment of the EMG signal; \(x\) i The \(i\)-th EMG data in each segment of EMG signal; \(x\) i-1 The \((i - 1)\)-th EMG data in each segment of EMG signal; sgn(\(x\)) is a function for judging whether the EMG signal waveform crosses the horizontal reference line;

[0023] The calculation formula for the waveform length is:

[0024]

[0025] where WL is the waveform length of the EMG signal, which is used to describe the signal fluctuation during muscle activity and can measure the signal complexity; N is the total number of EMG data points in each segment of the EMG signal; \(x\) i The \(i\)-th EMG data in each segment of EMG signal; \(x\) i-1 The \((i - 1)\)-th EMG data in each segment of EMG signal;

[0026] The calculation formula for the slope sign change is:

[0027]

[0028] where SCC is the slope sign change of the EMG signal, which is used to describe the frequency characteristics of the EMG signal; N is the total number of EMG data points in each segment of the EMG signal; \(x\) i The \(i\)-th EMG data in each segment of EMG signal; \(x\) i-1 The \((i - 1)\)-th EMG data in each segment of EMG signal; \(x\) i+1 The \((i + 1)\)-th EMG data in each segment of EMG signal; f(\(x\)) is a judgment function for slope sign change; T s is the threshold of the EMG signal, which can be obtained by T sis represented by μ + Jσ, where μ and σ respectively represent the mean and standard deviation of the EMG data within each segment of the EMG signal, and J is a preset constant.

[0029] Further, the first feature extraction is performed on the time-domain features of the movement muscle groups of the hip joint, knee joint, and ankle joint through the built-in CNN module to obtain the convolutional features of the movement muscle groups of the hip joint, knee joint, and ankle joint, including:

[0030] The first convolutional feature extraction and the first non-linear transformation activation are performed on the time-domain features of the movement muscle groups of the hip joint, knee joint, and ankle joint through the first convolutional layer and the first ReLU activation function layer of the CNN module to obtain the first convolutional features of the movement muscle groups of the hip joint, knee joint, and ankle joint after the first non-linear transformation activation;

[0031] The second convolutional feature extraction and the second non-linear transformation activation are performed on the first convolutional features of the movement muscle groups of the hip joint, knee joint, and ankle joint after the first non-linear transformation activation through the second convolutional layer and the second ReLU activation function layer of the CNN module to obtain the second convolutional features of the movement muscle groups of the hip joint, knee joint, and ankle joint after the second non-linear transformation activation;

[0032] The feature integration is performed on the second convolutional features of the movement muscle groups of the hip joint, knee joint, and ankle joint after the second non-linear transformation activation through the fully connected layer of the CNN module to obtain the convolutional features of the movement muscle groups of the hip joint, knee joint, and ankle joint.

[0033] Further, the second feature extraction is performed on the convolutional features of the movement muscle groups of the hip joint, knee joint, and ankle joint through the built-in multi-task learning module to obtain the deep features of the movement muscle groups of the hip joint, knee joint, and ankle joint, including:

[0034] The convolutional features of the movement muscle groups of the hip joint, knee joint, and ankle joint output by the CNN module are obtained through the feature sharing layer of the multi-task learning module, and the convolutional features of the movement muscle groups of the hip joint, knee joint, and ankle joint are respectively input into the expert network layer and the gating layer of the multi-task learning module; wherein, the expert network unit is composed of a plurality of expert networks; the gating layer is composed of a plurality of gating units;

[0035] The deep feature extraction is performed on the convolutional features of the movement muscle groups of the hip joint, knee joint, and ankle joint through each expert network in the expert network layer to obtain a plurality of expert network deep features;

[0036] According to the convolutional features of the movement muscle groups of the hip joint, knee joint, and ankle joint, the weight coefficients of each expert network are respectively calculated through each gating unit in the gating layer to obtain a plurality of expert network weight coefficients;

[0037] For each expert network, the deep features of the expert network are multiplied by the corresponding expert network weight coefficients through the output layer of the multi-task learning module, and classified according to the joint type to obtain the deep features of the hip, knee, and ankle joint movement muscle groups.

[0038] Furthermore, the calculation formula for the weight coefficient of the expert network is:

[0039]

[0040] Where, is the weight coefficient of the i-th expert network at time t in the k-th task; k is the number of tasks, representing the number of joint types in the lower limb joint movement angle prediction task, k = 1, 2, …, K; is the transformation matrix of the gating unit corresponding to the i-th expert network at time t in the k-th task; x t ′ is the convolution feature of the hip, knee, and ankle joint movement muscle groups.

[0041] Furthermore, the built-in LSTM module includes: several joint movement prediction sub-modules; among them, the joint movement prediction sub-modules include a hip joint movement prediction sub-module, a knee joint movement prediction sub-module, and an ankle joint movement prediction sub-module; each joint movement prediction sub-module includes several LSTM units; each LSTM unit includes a forget gate, an input gate, a candidate value, a cell state update, an output gate, and a hidden state output.

[0042] Based on the above method item embodiments, the present invention correspondingly provides system item embodiments;

[0043] An embodiment of the present invention provides a lower limb joint movement prediction system for climbing workers based on multi-task learning, including: a data acquisition module, a time domain feature extraction module, and a joint movement angle prediction module;

[0044] The data acquisition module is used to acquire the electromyogram signals of the muscle groups of the lower limb joints for the climbing worker to be tested to control the movement of the lower limb joints; among them, the lower limb joints include the hip joint, the knee joint, and the ankle joint;

[0045] The time domain feature extraction module is used to respectively extract the time domain features of the electromyogram signals of the muscle groups of the hip joint, the knee joint, and the ankle joint to obtain the time domain features of the muscle groups of the hip joint, the knee joint, and the ankle joint;

[0046] The joint motion angle prediction module is used to input the extracted time-series features into the trained joint prediction model, so that the joint prediction model performs the first feature extraction on the time-domain features of the hip joint, knee joint, and ankle joint movement muscle groups through the built-in CNN module to obtain the convolutional features of the hip joint, knee joint, and ankle joint movement muscle groups; perform the second feature extraction on the convolutional features of the hip joint, knee joint, and ankle joint movement muscle groups through the built-in multi-task learning module to obtain the depth features of the hip joint, knee joint, and ankle joint movement muscle groups; based on the built-in LSTM module, predict the joint motion angle according to the depth features of the hip joint, knee joint, and ankle joint movement muscle groups to obtain the motion angle prediction result of the lower limb joints.

[0047] Based on the above method item embodiments, the present invention correspondingly provides a terminal device item embodiment, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the method for predicting the lower limb joint movement of climbing workers based on multi-task learning as described in the present invention.

[0048] Based on the above method item embodiments, the present invention correspondingly provides a computer-readable storage medium item embodiment, including: a stored computer program. When the computer program runs, it controls the device where the computer-readable storage medium is located to execute the method for predicting the lower limb joint movement of climbing workers based on multi-task learning as described in the present invention.

[0049] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0050] The present invention first extracts the time-domain features of the electromyographic signals of the movement muscle groups of the hip joint, knee joint, and ankle joint to obtain the time-domain features of the movement muscle groups of each joint. Then, the convolutional features of the movement muscle groups of each joint are extracted by using the CNN neural network of the trained joint prediction model for the time-domain features of the movement muscle groups of each joint. Next, the deep features of the movement muscle groups of each joint are extracted by using the multi-task learning module of the trained joint prediction model for the convolutional features, and finally, the movement angles of each joint are predicted by using the LSTM module of the trained joint prediction model based on the deep features of the movement muscle groups of each joint to obtain the movement angles of each joint of the lower limb. That is, a multi-task learning module is added between the convolutional neural network and the long short-term memory network of the joint prediction model, and the multi-task learning module is used to perform secondary feature extraction on the convolutional features to provide more accurate data support for the subsequent prediction of the joint movement angles. Moreover, by using the multi-task learning module, the joint prediction model can predict the joint movement angles of multiple joints, improving the accuracy of the prediction results of the joint movement angles, and solving the problem that the existing technology cannot perform comprehensive prediction of the movement angles of the overall joints of the human lower limb because the model can only predict the movement angles of a single lower limb joint, ignoring the mutual influence between joints, and thus resulting in low accuracy of the prediction results of the lower limb joint movement angles. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 : is a flowchart of the steps of the method for predicting the movement of the lower limb joints of a climbing operator based on multi-task learning provided by an embodiment of the present invention;

[0052] Figure 2 : is a model framework diagram of the joint prediction model provided by an embodiment of the present invention;

[0053] Figure 3 : is a structural module diagram of the system for predicting the movement of the lower limb joints of a climbing operator based on multi-task learning provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0055] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features.

[0056] Embodiment 1:

[0057] Refer to Figure 1 : The flowchart of the steps of the method for predicting the lower limb joint movement of climbing workers based on multi-task learning provided by the embodiment of the present invention; Refer to Figure 2 : The model framework diagram of the joint prediction model provided by the embodiment of the present invention; The method at least includes the following steps:

[0058] Step S1: Obtain the electromyogram signals of the muscle groups that control the lower limb joint movement of the climbing worker to be measured; wherein, the lower limb joints include the hip joint, the knee joint, and the ankle joint;

[0059] In this embodiment, through an electromyogram signal acquisition device, obtain the electromyogram signals of the muscle groups that control the lower limb joint movement of the climbing worker to be measured; wherein, the lower limb joints include the hip joint, the knee joint, and the ankle joint; the muscle groups that control the hip joint include the gluteus maximus and the rectus femoris; the muscle groups that control the knee joint include the rectus femoris, the vastus medialis, the vastus lateralis, the gastrocnemius, and the biceps femoris; the muscle groups that control the ankle joint include the tibialis anterior, the gastrocnemius, and the soleus; the electromyogram signal acquisition device includes surface electrode patches, conductive glue, electrode wires, a signal amplifier, and a data acquisition system.

[0060] In this embodiment, after obtaining the electromyogram signals of the muscle groups that control the lower limb joint movement of the climbing worker to be measured, it further includes:

[0061] Perform data preprocessing on the obtained electromyogram signals of the muscle groups that control the lower limb joint movement of the climbing worker to be measured; wherein, the data preprocessing includes signal denoising, signal downsampling, and signal partitioning;

[0062] Exemplarily, the method of signal denoising is to first use notch filtering to remove the 50Hz power frequency interference from the original sEMG signal. The notch filter can remove the notch frequency, making the amplitude-frequency response at that point 0, and having no effect on other frequencies; then, filter the signal through a sixth-order Butterworth band-pass filter with a low cut-off frequency of 20Hz and a high cut-off frequency of 500Hz. The low cut-off frequency can eliminate the baseline drift caused by climbing movement or the sweat of the subject, remove the DC offset, and the high cut-off frequency can remove high-frequency random noise, etc., to prevent sampling signal overlap;

[0063] Exemplarily, the operation of signal partitioning is to first perform mean normalization on the electromyogram signals of the muscle groups that control the lower limb joint movement after signal denoising and signal downsampling, and then divide the normalized electromyogram signals into equal-length segments with non-overlapping signal activity windows to obtain several segments of electromyogram signals of the muscle groups that control the lower limb joint movement; wherein, the calculation formula of the mean normalization is X normis the EMG signal after mean normalization, X is the current value of the EMG signal, X b is the baseline of the EMG signal (0 after denoising), X mean is the mean value of the EMG signal; the sliding step of the non-overlapping signal activity window can be selected as 7.5 ms; the length of the sliding window can be set as 20 ms.

[0064] Step S2: Respectively extract the time-domain features of the EMG signals of the hip, knee, and ankle movement muscle groups to obtain the time-domain features of the hip, knee, and ankle movement muscle groups;

[0065] In this embodiment, the step of respectively extracting the time-domain features of the EMG signals of the hip, knee, and ankle movement muscle groups to obtain the time-domain features of the hip, knee, and ankle movement muscle groups includes:

[0066] Respectively calculate the time-domain features of the EMG signals of the hip, knee, and ankle movement muscle groups to obtain the time-domain features of the hip, knee, and ankle movement muscle groups; among them, the time-domain feature calculation includes mean absolute value calculation, root mean square calculation, variance calculation, zero-crossing number calculation, waveform length calculation, and slope sign change calculation.

[0067] In this embodiment, the calculation formula of the mean absolute value is:

[0068]

[0069] where MAV is the mean absolute value of the EMG signal, used to describe the average value of the absolute value of the EMG signal amplitude, representing the muscle contraction level; N is the total number of EMG data points in each segment of the EMG signal; x i is the i-th EMG data in each segment of the EMG signal;

[0070] The calculation formula of the root mean square is:

[0071]

[0072] where RMS is the root mean square of the EMG signal, used to describe the square root of the mean value of the EMG signal amplitude, evaluating the muscle contraction intensity and fatigue degree; N is the total number of EMG data points in each segment of the EMG signal; x i is the i-th EMG data in each segment of the EMG signal;

[0073] The calculation formula of the variance is:

[0074]

[0075] Among them, Var is the variance of the electromyogram signal, which is used to describe the degree of dispersion of the electromyogram signal and represents the intensity of the electromyogram signal; N is the total number of electromyogram data points in each segment of the electromyogram signal; x i is the i-th electromyogram data in each segment of the electromyogram signal;

[0076] The calculation formula for the number of zero crossings is as follows:

[0077]

[0078] Among them, ZC is the number of zero crossings of the electromyogram signal, which is used for the number of times the electromyogram signal waveform crosses the horizontal reference line; N is the total number of electromyogram data points in each segment of the electromyogram signal; x i is the i-th electromyogram data in each segment of the electromyogram signal; x i-1 is the (i - 1)-th electromyogram data in each segment of the electromyogram signal; sgn(x) is a function for judging whether the electromyogram signal waveform crosses the horizontal reference line;

[0079] The calculation formula for the waveform length is as follows:

[0080]

[0081] Among them, WL is the waveform length of the electromyogram signal, which is used to describe the signal fluctuation during muscle activity and can measure the signal complexity; N is the total number of electromyogram data points in each segment of the electromyogram signal; x i is the i-th electromyogram data in each segment of the electromyogram signal; x i-1 is the (i - 1)-th electromyogram data in each segment of the electromyogram signal;

[0082] The calculation formula for the slope sign change is as follows:

[0083]

[0084] Among them, SCC is the slope sign change of the electromyogram signal, which is used to describe the frequency characteristics of the electromyogram signal; N is the total number of electromyogram data points in each segment of the electromyogram signal; x i is the i-th electromyogram data in each segment of the electromyogram signal; x i-1 is the (i - 1)-th electromyogram data in each segment of the electromyogram signal; x i+1 is the (i + 1)-th electromyogram data in each segment of the electromyogram signal; f(x) is a judgment function for the slope sign change; T s is the threshold of the electromyogram signal, which can be represented by T s = μ + Jσ, where μ and σ respectively represent the mean and standard deviation of the electromyogram data in each segment of the electromyogram signal, and J is a preset constant.

[0085] Step S3: Input the extracted temporal features into the trained joint prediction model, so that the joint prediction model performs the first feature extraction on the temporal features of the hip joint, knee joint, and ankle joint's movement muscle groups through the in-built CNN module to obtain the convolution features of the hip joint, knee joint, and ankle joint's movement muscle groups; perform the second feature extraction on the convolution features of the hip joint, knee joint, and ankle joint's movement muscle groups through the in-built multi-task learning module to obtain the depth features of the hip joint, knee joint, and ankle joint's movement muscle groups; based on the in-built LSTM module, predict the joint movement angles according to the depth features of the hip joint, knee joint, and ankle joint's movement muscle groups to obtain the predicted results of the lower limb joint movement angles.

[0086] In this embodiment, the process of performing the first feature extraction on the temporal features of the hip joint, knee joint, and ankle joint's movement muscle groups through the in-built CNN module to obtain the convolution features of the hip joint, knee joint, and ankle joint's movement muscle groups includes:

[0087] Perform the first convolution feature extraction and the first non-linear transformation activation on the temporal features of the hip joint, knee joint, and ankle joint's movement muscle groups through the first convolution layer and the first ReLU activation function layer of the CNN module to obtain the first convolution features of the hip joint, knee joint, and ankle joint's movement muscle groups after the first non-linear transformation activation;

[0088] Perform the second convolution feature extraction and the second non-linear transformation activation on the first convolution features of the hip joint, knee joint, and ankle joint's movement muscle groups after the first non-linear transformation activation through the second convolution layer and the second ReLU activation function layer of the CNN module to obtain the second convolution features of the hip joint, knee joint, and ankle joint's movement muscle groups after the second non-linear transformation activation;

[0089] Perform feature integration on the second convolution features of the hip joint, knee joint, and ankle joint's movement muscle groups after the second non-linear transformation activation through the fully connected layer of the CNN module to obtain the convolution features of the hip joint, knee joint, and ankle joint's movement muscle groups.

[0090] In this embodiment, the convolution calculation formula of the convolution layer in the CNN module is:

[0091] F l =Re L U(W l *F l-1 +b l );

[0092] Wherein, the F l represents the convolution features of the l-th convolution layer; the F l-1 represents the convolution features of the (l - 1)-th convolution layer for inputting into the l-th convolution layer; W l and b lrespectively represent the weights and bias terms of the l-th convolutional layer.

[0093] In this embodiment, the multi-task learning module internally provided is used to perform a second feature extraction on the convolutional features of the hip, knee, and ankle movement muscle groups to obtain the depth features of the hip, knee, and ankle movement muscle groups, including:

[0094] Obtain the convolutional features of the hip, knee, and ankle movement muscle groups output by the CNN module through the feature sharing layer of the multi-task learning module, and respectively input the convolutional features of the hip, knee, and ankle movement muscle groups into the expert network layer and the gating layer of the multi-task learning module; wherein, the expert network unit is composed of several expert networks; the gating layer is composed of several gating units;

[0095] Perform depth feature extraction on the convolutional features of the hip, knee, and ankle movement muscle groups through each expert network in the expert network layer to obtain several expert network depth features;

[0096] According to the convolutional features of the hip, knee, and ankle movement muscle groups, calculate the weight coefficients of each expert network through each gating unit in the gating layer to obtain several expert network weight coefficients;

[0097] For each expert network, multiply the expert network depth feature by the corresponding expert network weight coefficient through the output layer of the multi-task learning module, and classify according to the joint type to obtain the depth features of the hip, knee, and ankle movement muscle groups.

[0098] In this embodiment, the multi-task learning module is an MMoE model;

[0099] In this embodiment, the calculation formula for the weight coefficient of the expert network is:

[0100]

[0101] wherein, is the weight coefficient of the i-th expert network at the t-th moment in the k-th task; k is the number of tasks, representing the number of joint types of the lower limb joint movement angle prediction task, k = 1, 2,... K; is the transformation matrix of the gating unit corresponding to the i-th expert network at the t-th moment in the k-th task; x t ′ is the convolutional feature of the hip, knee, and ankle movement muscle groups.

[0102] In this embodiment, the calculation formula for multiplying the expert network depth feature by the corresponding expert network weight coefficient is:

[0103]

[0104] where y k，t is the depth feature of the movement muscle group at time t; k is the number of tasks, representing the number of joint types of the lower limb joint movement angle prediction task, k = 1, 2, …, K; is the weight coefficient of the i-th expert network in the k-th task at time t; f i (x′ t ) is the depth feature of the expert network output by the i-th expert network at time t; x′ t is the convolution feature of the movement muscle groups of the hip joint, knee joint and ankle joint.

[0105] In this embodiment, the internal LSTM module includes: a plurality of joint movement prediction sub-modules; among them, the joint movement prediction sub-module includes a hip joint movement prediction sub-module, a knee joint movement prediction sub-module and an ankle joint movement prediction sub-module; each joint movement prediction sub-module includes a plurality of LSTM units; each LSTM unit includes a forget gate, an input gate, a candidate value, a cell state update, an output gate and a hidden state output.

[0106] In this embodiment, the expression of the forget gate is: f t = sigmoid(W fx y t + W fh h t-1 + b f ); where y t is the input at the current time t; W fx and W fh are the first weight and the second weight of the forget gate respectively; h t-1 is the hidden state at the previous time; b f is the bias term of the forget gate;

[0107] The expression of the input gate is: i t = sigmoid(W ix y t + W ih h t-1 + b i ); where y t is the input at the current time t; W ix and W ih are the first weight and the second weight of the input gate respectively; h t-1 is the hidden state at the previous time; b i is the bias term of the input gate;

[0108] The expression of the candidate value is: g t = tanh(W gx y t + Wgh h t-1 +b g );where, y t is the input at the current moment t; W gx and W gh are the first weight and the second weight of the candidate value respectively; h t-1 is the hidden state at the previous moment; b g is the bias term of the candidate value;

[0109] The expression for cell state update is: c t = f t * c t-1 + i t * g t ; where, y t is the input at the current moment t; f t is the output of the forget gate; c t-1 is the cell state at the previous moment; i t is the output of the input gate; g t is the candidate value; c t is the cell state at the current moment;

[0110] The expression for the output gate is: o t = sigmoid(W ox y t + W oh h t-1 + b o );where, y t is the input at the current moment t; W ox and W oh are the first weight and the second weight of the output gate respectively; h t-1 is the hidden state at the previous moment; b o is the bias term of the output gate;

[0111] The expression for hidden state output is: h t = o t * tanh(c t );where, o t is the output of the output gate; c t is the cell state at the current moment t; h t is the hidden state at the current moment t.

[0112] In this embodiment, the model training of the joint prediction model includes:

[0113] Collect the electromyographic signals of the motor muscle groups that control the lower limb joint movements and the corresponding actual movement angles of the lower limb joints when several subjects with different heights and weights climb while carrying different loads; wherein, the lower limb joints include the hip joint, the knee joint and the ankle joint;

[0114] For the EMG signals of the muscle groups controlling the lower limb joint movements during climbing of several subjects with different heights and weights under different load-bearing conditions, time-domain feature extraction is respectively performed to obtain the time-domain features of the muscle groups of the hip joint, knee joint, and ankle joint;

[0115] Input the extracted time-series features into the joint prediction model to be trained, so that the joint prediction model performs the first feature extraction on the time-domain features of the muscle groups of the hip joint, knee joint, and ankle joint through the built-in CNN module to obtain the convolutional features of the muscle groups of the hip joint, knee joint, and ankle joint; perform the second feature extraction on the convolutional features of the muscle groups of the hip joint, knee joint, and ankle joint through the built-in multi-task learning module to obtain the depth features of the muscle groups of the hip joint, knee joint, and ankle joint; based on the built-in LSTM module, predict the joint movement angles according to the depth features of the muscle groups of the hip joint, knee joint, and ankle joint to obtain the prediction results of the movement angles of the lower limb joints;

[0116] According to the prediction results of the movement angles of the lower limb joints and the actual movement angles of the lower limb joints, calculate the root mean square error and the mean absolute error respectively, use the root mean square error and the mean absolute error as the evaluation indicators of the model, and optimize the parameters of the joint prediction model according to the evaluation indicators until the evaluation indicators meet the preset judgment conditions to obtain the trained joint prediction model.

[0117] In this embodiment, the calculation formula of the root mean square error is:

[0118]

[0119] where y RMSE is the root mean square error; X act (i) is the predicted value of the movement angle of the lower limb joint at time i; X pred (i) is the actual movement angle of the lower limb joint; m is the number of samples of the prediction results;

[0120] The calculation formula of the mean absolute error is:

[0121]

[0122] where y MAE is the mean absolute error; X act (i) is the predicted value of the movement angle of the lower limb joint at time i; X pred (i) is the actual movement angle of the lower limb joint; m is the number of samples of the prediction results.

[0123] In this embodiment, the parameters of the joint prediction model can be optimized by using the Adam optimizer for parameter optimization.

[0124] In this embodiment, the preset judgment condition is that the root mean square error is less than 3.00 and the mean absolute error is less than 3.20.

[0125] Embodiment 2:

[0126] Refer to Figure 3 : which is the structural module diagram of the lower limb joint motion prediction system for climbing operators based on multi-task learning provided by the embodiment of the present invention; the system at least includes the following modules: a data acquisition module, a time-domain feature extraction module, and a joint motion angle prediction module;

[0127] The data acquisition module is used to acquire the electromyogram signals of the muscle groups that control the lower limb joint motion of the climbing operator to be measured; wherein, the lower limb joints include the hip joint, the knee joint, and the ankle joint;

[0128] The time-domain feature extraction module is used to respectively extract the time-domain features of the electromyogram signals of the muscle groups of the hip joint, the knee joint, and the ankle joint, and obtain the time-domain features of the muscle groups of the hip joint, the knee joint, and the ankle joint;

[0129] The joint motion angle prediction module is used to input the extracted time-series features into the trained joint prediction model, so that the joint prediction model performs the first feature extraction on the time-domain features of the muscle groups of the hip joint, the knee joint, and the ankle joint through the built-in CNN module, and obtains the convolutional features of the muscle groups of the hip joint, the knee joint, and the ankle joint; perform the second feature extraction on the convolutional features of the muscle groups of the hip joint, the knee joint, and the ankle joint through the built-in multi-task learning module, and obtain the depth features of the muscle groups of the hip joint, the knee joint, and the ankle joint; based on the built-in LSTM module, perform joint motion angle prediction according to the depth features of the muscle groups of the hip joint, the knee joint, and the ankle joint, and obtain the motion angle prediction result of the lower limb joint.

[0130] Based on the above method item embodiment, another embodiment is provided;

[0131] A terminal device provided by another embodiment of the present invention includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the method for predicting the lower limb joint motion of a climbing operator based on multi-task learning described in any one of the above method item embodiments of the present invention.

[0132] Exemplarily, the computer program may be divided into one or more modules / units. The one or more modules / units are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in the terminal device for predicting the lower limb joint movement of climbing workers based on multi-task learning.

[0133] The terminal device for predicting the lower limb joint movement of climbing workers based on multi-task learning may be a computing device such as a desktop computer, a notebook, a palm computer, or a cloud server. The terminal device for predicting the lower limb joint movement of climbing workers based on multi-task learning may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that, for example, the terminal device for predicting the lower limb joint movement of climbing workers based on multi-task learning may further include input / output devices, network access devices, a bus, etc.

[0134] The so-called processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The processor is the control center of the terminal device for predicting the lower limb joint movement of climbing workers based on multi-task learning, and connects various parts of the entire terminal device for predicting the lower limb joint movement of climbing workers based on multi-task learning through various interfaces and lines.

[0135] The memory can be used to store the computer programs and / or modules. By running or executing the computer programs and / or modules stored in the memory, and invoking the data stored in the memory, the processor implements various functions of the terminal device for predicting the lower limb joint movements of climbing workers based on multi-task learning. The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the mobile phone (such as audio data, phone book, etc.). In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one magnetic disk storage device, flash device, or other volatile solid-state storage devices.

[0136] Based on the above method item embodiments, another embodiment is provided;

[0137] A storage medium provided by another embodiment of the present invention includes a stored computer program, wherein when the computer program runs, it controls the device where the storage medium is located to execute the method for predicting the lower limb joint movements of climbing workers based on multi-task learning according to any one of the above method item embodiments of the present invention.

[0138] Among them, the above storage medium is a computer-readable storage medium. The modules / units integrated in the system / terminal device for predicting the lower limb joint movements of climbing workers based on multi-task learning, if implemented in the form of software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above embodiment methods of the present invention, it can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0139] In the specific embodiments described above, the objectives, technical solutions and beneficial effects of the present invention have been further described in detail. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the protection scope of the present invention. It is particularly pointed out that for those skilled in the art, any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for predicting the lower limb joint movement of climbing workers based on multi-task learning, characterized in that Including: Obtaining the electromyography signals of the muscle groups that control the movement of the lower limb joints of the climber to be measured; wherein, the lower limb joints include the hip joint, the knee joint and the ankle joint; Respectively extracting the time-domain features of the electromyography signals of the muscle groups of the hip joint, the knee joint and the ankle joint to obtain the time-domain features of the muscle groups of the hip joint, the knee joint and the ankle joint; Inputting the extracted time-series features into the trained joint prediction model, so that the joint prediction model performs the first feature extraction on the time-domain features of the muscle groups of the hip joint, the knee joint and the ankle joint through the built-in CNN module to obtain the convolutional features of the muscle groups of the hip joint, the knee joint and the ankle joint; performing the second feature extraction on the convolutional features of the muscle groups of the hip joint, the knee joint and the ankle joint through the built-in multi-task learning module to obtain the depth features of the muscle groups of the hip joint, the knee joint and the ankle joint; based on the built-in LSTM module, predicting the joint movement angles according to the depth features of the muscle groups of the hip joint, the knee joint and the ankle joint to obtain the prediction results of the movement angles of the lower limb joints.

2. The method for predicting the lower limb joint movement of climbing workers based on multi-task learning according to claim 1, characterized in that The step of respectively extracting the time-domain features of the electromyography signals of the muscle groups of the hip joint, the knee joint and the ankle joint to obtain the time-domain features of the muscle groups of the hip joint, the knee joint and the ankle joint includes: Respectively calculating the time-domain features of the electromyography signals of the muscle groups of the hip joint, the knee joint and the ankle joint to obtain the time-domain features of the muscle groups of the hip joint, the knee joint and the ankle joint; wherein, the time-domain feature calculation includes mean absolute value calculation, root mean square calculation, variance calculation, zero-crossing number calculation, waveform length calculation and slope sign change calculation.

3. The lower limb joint motion prediction method for climbing operators based on multi-task learning according to claim 2, characterized in that The calculation formula of the mean absolute value is: Among them, MAV is the mean absolute value of the EMG signal, which is used to describe the average value of the absolute value of the EMG signal amplitude and represents the muscle contraction level; N is the total number of EMG data points in each segment of the EMG signal; x i is the i-th EMG data in each segment of the EMG signal; The calculation formula of the root mean square is: Among them, RMS is the root mean square of the EMG signal, which is used to describe the square root of the mean value of the EMG signal amplitude and evaluate the muscle contraction intensity and fatigue degree; N is the total number of EMG data points in each segment of the EMG signal; x i is the i-th EMG data in each segment of the EMG signal; The calculation formula of the variance is: Among them, Var is the variance of the EMG signal, which is used to describe the degree of dispersion of the EMG signal and represents the intensity of the EMG signal; N is the total number of EMG data points in each segment of the EMG signal; x i is the i-th EMG data in each segment of the EMG signal; The calculation formula of the zero-crossing number is: Among them, ZC is the number of zero-crossing points of the EMG signal, which is used for the number of times the EMG signal waveform crosses the horizontal reference line; N is the total number of EMG data points in each segment of the EMG signal; x i is the i-th EMG data point in each segment of the EMG signal; x i-1 is the (i - 1)-th EMG data point in each segment of the EMG signal; sgn(x) is a function for judging whether the EMG signal waveform crosses the horizontal reference line; The calculation formula of the waveform length is: Among them, WL is the waveform length of the electromyogram signal, which is used to describe the signal fluctuation during muscle activity and can measure the signal complexity; N is the total number of electromyogram data points in each segment of the electromyogram signal; x i is the i-th electromyogram data in each segment of the electromyogram signal; x i-1 is the (i - 1)-th electromyogram data in each segment of the electromyogram signal; The calculation formula of the slope sign change is: Among them, SCC is the slope signature change of the electromyogram signal, which is used to describe the frequency characteristics of the electromyogram signal; N is the total number of electromyogram data points in each segment of the electromyogram signal; x i is the i-th electromyogram data in each segment of the electromyogram signal; x i-1 is the (i - 1)-th electromyogram data in each segment of the electromyogram signal; x i+1 is the (i + 1)-th electromyogram data in each segment of the electromyogram signal; f(x) is the judgment function of the slope signature change; T s is the threshold of the electromyogram signal, which can be expressed as T s = μ + Jσ, where μ and σ respectively represent the mean and standard deviation of the electromyogram data within each segment of the electromyogram signal, and J is a preset constant.

4. The method for predicting the lower limb joint movement of climbing operators based on multi-task learning according to claim 3, wherein The step of performing the first feature extraction on the time-domain features of the muscle groups of the hip joint, the knee joint and the ankle joint through the built-in CNN module to obtain the convolutional features of the muscle groups of the hip joint, the knee joint and the ankle joint includes: Performing the first convolutional feature extraction and the first non-linear transformation activation on the time-domain features of the muscle groups of the hip joint, the knee joint and the ankle joint through the first convolutional layer and the first ReLU activation function layer of the CNN module to obtain the first convolutional features of the muscle groups of the hip joint, the knee joint and the ankle joint after the first non-linear transformation activation; Performing the second convolutional feature extraction and the second non-linear transformation activation on the first convolutional features of the muscle groups of the hip joint, the knee joint and the ankle joint after the first non-linear transformation activation through the second convolutional layer and the second ReLU activation function layer of the CNN module to obtain the second convolutional features of the muscle groups of the hip joint, the knee joint and the ankle joint after the second non-linear transformation activation; Performing feature integration on the second convolutional features of the muscle groups of the hip joint, the knee joint and the ankle joint after the second non-linear transformation activation through the fully connected layer of the CNN module to obtain the convolutional features of the muscle groups of the hip joint, the knee joint and the ankle joint.

5. The method for predicting the lower limb joint movement of climbing operators based on multi-task learning according to claim 4, wherein Performing a second feature extraction on the convolutional features of the hip, knee, and ankle movement muscle groups through the built-in multi-task learning module to obtain the depth features of the hip, knee, and ankle movement muscle groups, including: Obtaining the convolutional features of the hip, knee, and ankle movement muscle groups output by the CNN module through the feature sharing layer of the multi-task learning module, and respectively inputting the convolutional features of the hip, knee, and ankle movement muscle groups into the expert network layer and the gating layer of the multi-task learning module; wherein, the expert network unit consists of a number of expert networks; the gating layer consists of a number of gating units; Performing depth feature extraction on the convolutional features of the hip, knee, and ankle movement muscle groups through each expert network in the expert network layer to obtain a number of expert network depth features; According to the convolutional features of the hip, knee, and ankle movement muscle groups, calculating the weight coefficients of each expert network through each gating unit in the gating layer to obtain a number of expert network weight coefficients; For each expert network, multiplying the expert network depth feature by the corresponding expert network weight coefficient through the output layer of the multi-task learning module and classifying according to the joint type to obtain the depth features of the hip, knee, and ankle movement muscle groups.

6. The method for predicting the lower limb joint movement of climbing operators based on multi-task learning according to claim 5, wherein, The calculation formula for the weight coefficient of the expert network is: wherein, is the weight coefficient of the i-th expert network in the k-th task at time t; k is the number of tasks, representing the number of joint types of the lower limb joint motion angle prediction task, k = 1, 2, … K; is the transformation matrix of the gating unit corresponding to the i-th expert network in the k-th task at time t; x t ′ are the convolution features of the hip, knee, and ankle joint movement muscle groups.

7. The method for predicting the lower limb joint movement of climbing operators based on multi-task learning according to claim 6, wherein The built-in LSTM module includes: a number of joint movement prediction sub-modules; wherein, the joint movement prediction sub-module includes a hip movement prediction sub-module, a knee movement prediction sub-module, and an ankle movement prediction sub-module; each joint movement prediction sub-module includes a number of LSTM units; each LSTM unit includes a forget gate, an input gate, a candidate value, a cell state update, an output gate, and a hidden state output.

8. A lower limb joint motion prediction system for climbing operators based on multi-task learning, characterized in that, Including: A data acquisition module, a time-domain feature extraction module, and a joint movement angle prediction module; The data acquisition module is used to acquire the electromyographic signals of the muscle groups of the hip, knee, and ankle joints for controlling the movement of the lower limbs of the climbing operator to be measured; wherein, the lower limb joints include the hip joint, the knee joint, and the ankle joint; The time-domain feature extraction module is used to perform time-domain feature extraction on the electromyographic signals of the hip, knee, and ankle movement muscle groups respectively to obtain the time-domain features of the hip, knee, and ankle movement muscle groups; The joint movement angle prediction module is used to input the extracted time-series features into the trained joint prediction model, so that the joint prediction model performs a first feature extraction on the time-domain features of the hip, knee, and ankle movement muscle groups through the built-in CNN module to obtain the convolutional features of the hip, knee, and ankle movement muscle groups; performing a second feature extraction on the convolutional features of the hip, knee, and ankle movement muscle groups through the built-in multi-task learning module to obtain the depth features of the hip, knee, and ankle movement muscle groups; based on the built-in LSTM module, predicting the joint movement angle according to the depth features of the hip, knee, and ankle movement muscle groups to obtain the movement angle prediction result of the lower limb joints.

9. A terminal device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the method for predicting the lower limb joint movement of climbing operators based on multi-task learning according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium includes a stored computer program, wherein when the computer program runs, it controls the device where the storage medium is located to execute the method for predicting the lower limb joint movement of climbing operators based on multi-task learning according to any one of claims 1 to 7.

Images