Active human-machine collaboration method based on multimodal information of human body

Abstract

The present invention discloses an active human-robot collaboration method based on multimodal human information. The method comprises the following steps: collecting signals from collaborative humans and robots; inputting joint angle signals into a constructed predictive neural network for processing to obtain a predicted motion trajectory; inputting electromyographic signals into a constructed estimation neural network for processing to obtain a three-dimensional arm force; inputting the three-dimensional arm force into a refined perception interface of the human collaborative state to obtain a collaborative comfort index and a quantitative stability index; weighting the collaborative comfort index and the quantitative stability index to obtain an optimization target, using an exploratory iterative algorithm to obtain the optimal robot impedance parameters, setting the robot's impedance model based on the impedance parameters to obtain a desired trajectory; and using the PD position control method to drive the robot to execute the corrected predicted motion trajectory, enabling the robot to actively follow the human's movements. The present invention improves collaborative efficiency and solves problems such as poor collaborative comfort and stability and time-consuming and labor-intensive collaboration.

Description

Technical Field

[0001] The present invention relates to human-machine collaboration technology, and in particular to an active human-machine collaboration method based on multimodal information of the human body. Background Art

[0002] The biggest challenge in achieving human-robot collaboration in shared environments and joint tasks lies in achieving proactive and synchronized collaboration in unstructured, unknown collaborative scenarios, enabling robots to actively and flexibly collaborate with humans and reduce the workload on humans. Current approaches to human-robot collaboration typically focus on the robot's ability to follow, achieving only passive human-robot collaboration.

[0003] The path to achieving active human-robot collaboration lies in accurately predicting human motion trajectories. Past research has included estimating discrete human motion intentions using minimum jerk models and hidden Markov models; measuring interaction forces with force sensors and developing force-based motion estimation methods; and developing motion estimation observers that consider the dynamic characteristics of collaborative objects to achieve three-dimensional motion prediction. However, these methods typically only achieve single-step / short-term predictions. Single-step / short-term motion prediction places more stringent demands on the real-time performance of controllers, making its application to real-time collaborative systems challenging. Currently, deep learning algorithms are commonly used to construct multi-step prediction models, including deep learning architectures combining convolutional layers with long short-term memory networks, models combining musculoskeletal models with long short-term memory networks, and models combining residual neural networks with bidirectional long short-term memory networks. All of these studies employed recurrent neural networks to analyze the temporal correlation of motion trajectories. However, human movement habits and regularity result in spatial continuity in motion trajectories. These studies ignored this spatial continuity and instead directly exploited the powerful fitting capabilities of deep learning simulations to simultaneously address both temporal correlation and spatial continuity in predicted trajectories. For real-time collaboration, this crude use of deep learning models to establish multi-step prediction methods will inevitably lead to increased model complexity and a decrease in the model's real-time calculation speed.

[0004] Furthermore, existing human-robot collaboration methods lack a refined interface for sensing the human's collaborative state, preventing the robot from adjusting its trajectory based on the human's collaborative state. Consequently, these control methods typically rely on one-way feedback, preventing the robot from adapting to the human's work rhythm. This one-way feedback approach struggles to ensure comfortable and stable collaboration. Summary of the Invention

[0005] The present invention aims to overcome the shortcomings of the above-mentioned existing technologies and provides a method for active human-machine collaboration based on multimodal human body information. This method overcomes the problems of passive human-machine collaboration, such as low efficiency, poor comfort and stability, and time-consuming and labor-intensive collaboration.

[0006] The object of the present invention is achieved through the following technical solution: This active human-machine collaboration method based on multimodal information of the human body comprises the following steps:

[0007] S1. Before the human-robot collaboration process, collect the joint angle signals related to the human arm movement, the electromyographic signals at multiple positions of the human arm, and the end position signals and force signals of the robot, and create a data set;

[0008] S2. Construct a prediction neural network and train it using the joint angle signals and end position signals in the dataset;

[0009] Construct an estimation neural network and train it using the electromyographic signals and force signals in the dataset;

[0010] During human-machine collaboration, human joint angle signals and myoelectric signals are collected in real time and input into the trained prediction neural network and estimation neural network respectively to obtain the predicted motion trajectory and three-dimensional arm force of the arm.

[0011] S4. Build a refined perception interface for human collaboration:

[0012]

[0013]

[0014] Where M represents the number of steps of the predicted trajectory; represents the arm strength output of the person in step i; is the actual running trajectory of the robot in step i, i is a natural number, W h represents the output work of humans, Δ represents the effective stroke of the robot, represents the quantitative indicator of collaboration comfort, σ E It represents a quantitative indicator of stationarity;

[0015] S5. Input the three-dimensional arm force into the refined perception interface of human collaboration status to obtain collaborative comfort index and stability quantitative index;

[0016] S6. Weighting the collaborative comfort index and the stability quantitative index to obtain the optimization target, using an exploratory iterative algorithm to determine the optimal robot impedance parameter, and then setting the robot's impedance model based on this impedance parameter to achieve the desired trajectory of the arm's motion prediction according to the interactive external force;

[0017] S7. Use the PD position control method to drive the robot to execute the corrected motion prediction trajectory, and finally enable the robot to actively follow human motion.

[0018] Preferably, the prediction neural network includes a time correlation analysis module and a space continuity analysis module connected in sequence;

[0019] The temporal correlation analysis module adopts a sequence-to-sequence model structure and integrates a multi-layer long short-term memory network to capture the temporal dependency between motion trajectories;

[0020] The spatial continuity analysis module uses a graph structure to establish the correlation between predicted trajectories to analyze the spatial correlation between motion trajectories.

[0021] Preferably, the estimation neural network is composed of a parallel long short-term memory network.

[0022] Preferably, the quantitative index of collaboration comfort is the average effective output force of humans during the collaboration process, that is, the ratio of the human output work to the effective stroke of the robot;

[0023] Among them, the expression of human output work is:

[0024]

[0025] Where M represents the number of steps of the predicted trajectory; represents the arm strength output of the person in step i; is the actual running trajectory of the robot in step i, where i is a natural number;

[0026] The expression of the robot's effective travel is:

[0027]

[0028] Then the expression of the quantitative index of collaborative comfort is:

[0029]

[0030] Preferably, the stability quantitative index is the degree of fluctuation of human arm strength during the collaboration process, and the expression of this stability quantitative index is:

[0031]

[0032] Preferably, in step S4, the process of obtaining the optimization target is as follows:

[0033]

[0034] Where R represents the optimization target, and λ1 and λ2 represent weight coefficients.

[0035] Preferably, in step S6, the iterative formula of the exploration iterative algorithm is:

[0036]

[0037] Among them, ε n represents the update step size, and ε n =c / n 1 / 2 , a represents the result of random sampling from a uniform distribution in the interval [0,1], For R about The gradient value, P n represents the exploration probability, R represents the optimization target, n represents the number of iterations, Indicates the impedance parameters that need to be updated iteratively. express The nth iteration value of .

[0038] Preferably, the impedance model is expressed as:

[0039]

[0040] and Both represent the impedance parameters of the robot, f r and They represent the external interaction force and force change on the end of the robot, ΔX r Indicates the motion correction amount of the robot.

[0041] Preferably, the control law of the PD position control method is:

[0042]

[0043] Among them, K p and K d Both represent the parameters of PD position control, X d and Denote the desired trajectory and velocity, respectively, X r and Represent the actual position and speed of the robot, F rb Indicates the control torque.

[0044] Preferably, the control law of the PD position control method further includes the following steps:

[0045] Based on the motion prediction trajectory corrected in step S6, the feedforward torque compensation term is calculated in combination with the robot dynamics equation, and the feedforward compensation amount F rf for:

[0046]

[0047] Among them, M r (X r ), G r(X r ) represents the robot dynamic parameters, are the velocity and acceleration of the desired trajectory;

[0048] Then, based on The results, combined with The calculated torque of the machine is:

[0049]

[0050] The present invention has the following advantages over the prior art:

[0051] Compared with the existing human-machine collaboration methods, the present invention adopts a modular design concept to design a prediction neural network that takes into account the temporal and spatial characteristics to address the problems of time correlation and spatial continuity in the predicted trajectory, thereby achieving accurate prediction of human motion trajectories, reducing the complexity of the model, and improving the real-time performance of human-machine collaboration. A parallel long short-term memory network is used to achieve synchronous estimation of three-dimensional arm force; a refined perception interface for human collaboration status is established to achieve two-way feedback of the human-machine collaboration system; an exploratory iterative learning algorithm is used to achieve online learning of the robot's optimal impedance parameters; an impedance model of the robot is established to achieve correction of the predicted trajectory; and a PD position control method is used to achieve accurate tracking of the motion trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 1 is a flow chart of the active human-machine collaboration method based on multimodal information of the human body according to the present invention;

[0053] Figure 2 It is a schematic diagram of the platform facilities used when performing data collection in the method of the present invention. DETAILED DESCRIPTION

[0054] The present invention will be further described below with reference to the accompanying drawings and examples.

[0055] like Figure 1 As shown, the active human-machine collaboration method based on multimodal information of the human body includes the following steps:

[0056] S1, such as Figure 2 As shown in the figure, before the human-robot collaboration process, joint angle signals related to human arm movement, myoelectric signals at multiple positions of the human arm, and end position signals and force signals of the robot are collected and generated into a data set. The specific process is as follows:

[0057] IMU sensors and goniometers are attached to the upper arm and elbow joints to collect joint angle signals. Myoelectric sensors are attached to the active muscles of the human body to collect myoelectric signals. The activated muscles include the pectoralis major, infraspinatus, anterior deltoid, posterior deltoid, biceps, and triceps. The encoder values ​​of the motors in each joint of the robot can be used to calculate the end position signal of the robot. A six-dimensional force sensor is installed at the end of the robot to collect force signals.

[0058] A human drags the robot to execute a random trajectory within the workspace. Joint angle signals and myoelectric signals are wirelessly transmitted to the robot's real-time control system. Position and force signals are amplified by Beckhoff modules and transmitted to the robot's real-time control system via the EtherCAT protocol. In addition to the above methods, the number of channels for joint position, muscle position, and force signals can be flexibly set.

[0059] The collected joint angle signals, electromyographic signals, position signals, and force signals are converted and analyzed in an industrial computer, where subsequent processing is also performed. The collected position and force signals of the robot's end-user are smoothed using a moving average with a window length of 9. The smoothed position and force signals are used to train the prediction and estimation neural networks, respectively.

[0060] S2. Construct a prediction neural network and train it using joint angle signals and end position signals. Then, the joint angle signals collected in real time during the human-machine collaboration process are input into the trained prediction neural network for processing to obtain the predicted motion trajectory of the arm.

[0061] Specifically, a predictive neural network is constructed using an end-to-end training approach to predict the three-dimensional arm motion. The robot's end-position signal can be considered the actual arm motion trajectory. The predicted trajectory is subtracted from the actual arm motion trajectory to obtain the prediction error. The predictive neural network is trained using error backpropagation, with minimizing the prediction error as the optimization objective.

[0062] Among them, the prediction neural network includes a temporal correlation analysis module and a spatial continuity analysis module connected in sequence; the temporal correlation analysis module adopts a sequence-to-sequence model structure and integrates a multi-layer long short-term memory network to capture the temporal dependency between motion trajectories; the spatial continuity analysis module uses a graph structure to establish the correlation between predicted trajectories to analyze the spatial correlation between motion trajectories.

[0063] Specifically, the prediction neural network uses the time correlation module and the spatial continuity analysis module to establish the mapping relationship between the joint angle signal and the motion trajectory. The process includes:

[0064] S21, constructing a time correlation analysis module to convert the collected joint angle information into a motion prediction trajectory with a temporal dependency relationship;

[0065] The temporal correlation analysis module adopts a sequence-to-sequence model structure, which includes an encoder and a decoder. The specific construction is:

[0066] The collected joint angle signals are segmented and input into the encoder. The encoder integrates a multi-layer long short-term memory network to capture the strong temporal dependencies between joint angle signals and converts them into high-dimensional encoding vectors. The encoding process of the encoder can be expressed as:

[0067]

[0068] in, is the segmented joint angle signal, H represents the converted high-dimensional encoding vector, is the encoder function, l represents the number of layers of the LSTM network integrated in the encoder, d represents the hidden unit dimension of the LSTM network, Represents a long short-term memory network layer.

[0069] The decoder takes the high-dimensional encoding vector output by the encoder as input and outputs the predicted trajectory of the arm movement with a temporal relationship. The decoder integrates a long short-term memory network with the same number of layers as the encoder. Its decoding process can be expressed as:

[0070]

[0071] in, is the arm motion trajectory predicted by the decoder, M represents the predicted number of steps, Represents a decoder function.

[0072] S22, inputting the predicted arm motion trajectory obtained in step S21 into the spatial continuity analysis module, and outputting the predicted arm motion trajectory taking into account both temporal and spatial characteristics;

[0073] The spatial continuity analysis module uses graph structures to establish spatial relationships between predicted trajectories and uses graph attention networks to achieve trajectory smoothing. The specific construction is:

[0074] The adjacency matrix of the established graph structure is,

[0075] This is formula (3);

[0076] The specific process of achieving smoothing in the graph attention network is as follows:

[0077] This is formula (4);

[0078] Among them, || represents the tensor glue operation, e ij is the correlation feature, f(e ij ) represents the correlation coefficient, is a nonlinear activation function, ∧ represents a tensor filtering operation, Represents the various positions of the predicted trajectory.

[0079] Construct an estimation neural network, and use electromyographic signals and force signals to train the estimation neural network. Then, the electromyographic signals collected in real time during the human-machine collaboration process are input into the trained estimation neural network for processing to obtain the estimated three-dimensional arm strength. Specifically, the estimation neural network is composed of a parallel long short-term memory network, and the force signal of the robot is regarded as the real strength of the arm. The estimated error of the arm strength is obtained by subtracting the estimated strength from the real strength of the arm. The estimation neural network takes minimizing the estimated error of the arm strength as the optimization goal and is trained by error back propagation. Establish a mapping relationship between multi-channel electromyographic signals and arm strength in each direction. The process can be expressed as follows:

[0080] This is formula (5);

[0081] Among them, EMG1, EMG2,…, EMG n is the multi-channel electromyographic signal collected, F x 、F y 、F z is the estimated three-dimensional arm strength.

[0082] S3. Build a refined perception interface for human collaboration:

[0083]

[0084]

[0085] Where M represents the number of steps of the predicted trajectory; represents the arm strength output of the person in step i; is the actual running trajectory of the robot in step i, i is a natural number, W h represents the output work of humans, Δ represents the effective stroke of the robot, represents the quantitative indicator of collaboration comfort, σ E It represents a quantitative indicator of stationarity;

[0086] S4. Input the three-dimensional arm force into the refined perception interface of human collaboration status to obtain collaborative comfort index and stability quantitative index;

[0087] The quantitative index of collaborative comfort is the average effective output force of humans during the collaborative process, that is, the ratio of human output work to the effective stroke of the robot;

[0088] Among them, during the collaboration process, the expression of human output work in the three-dimensional motion direction is:

[0089] This is formula (6);

[0090] Where M represents the number of steps of the predicted trajectory; represents the arm strength output of the person in step i; is the actual running trajectory of the robot in step i, where i is a natural number;

[0091] During the collaboration process, the effective stroke of the robot must be consistent with the direction of the arm's output force. The expression for the effective stroke of the robot is:

[0092] This is formula (7);

[0093] Then, combining equations (6) and (7), the expression of the collaborative comfort quantitative index is:

[0094] This is formula (8).

[0095] The stability quantitative index is the degree of fluctuation of human arm force during the collaboration process. The expression of this stability quantitative index is:

[0096] This is formula (9).

[0097] S5. Weighting the collaborative comfort index and the stability quantitative index to obtain the optimization target, using an exploratory iterative algorithm to determine the optimal robot impedance parameter, and then setting the robot's impedance model based on this impedance parameter to achieve the desired trajectory of the arm's motion, which is corrected according to the interactive external force.

[0098] In step S5, the process of obtaining the optimization target is as follows:

[0099]

[0100] This is formula (10);

[0101] Where R represents the optimization target, and λ1 and λ2 represent weight coefficients.

[0102] Robot impedance parameters It is related to the collaborative comfort index and the collaborative stability index, and affects the optimization target. The goal of the exploration iterative learning algorithm is to minimize the optimization target by adjusting the robot impedance parameters. The iterative formula of the exploration iterative algorithm is:

[0103] This is formula (11);

[0104] Among them, ε n represents the update step size, and ε n =c / n 1 / 2 , a represents the result of random sampling from a uniform distribution in the interval [0,1], For R about The gradient value of It can be approximated by the difference formula with range constraints, and its mathematical expression is:

[0105] This is formula (12); μ represents the variable representing the range constraint function;

[0106] P n represents the exploration probability, P n About the amount of information | R n -R n-1 | and the number of iteration rounds show an exponential decay trend, and the mathematical expression is:

[0107] This is formula (13); where γ represents the temperature attenuation coefficient, E is the number of iteration rounds, and T0 represents the initial temperature value.

[0108] R represents the optimization target, n represents the number of iterations, Indicates the impedance parameters that need to be updated iteratively. express The nth iteration value of .

[0109] Set the convergence criterion of the exploration iterative learning algorithm, and then iterate according to formula (11) until the convergence criterion is met, specifically:

[0110] This is formula (14);

[0111] When the changes in the impedance parameters for four consecutive steps are all less than the set threshold α, it is approximately considered that the optimal impedance parameters have been learned, and the iteration is stopped at this time; otherwise, the iteration continues.

[0112] Based on the above optimal impedance parameters, the optimal impedance model of the robot is set. The expression of this impedance model is:

[0113] This is formula (15);

[0114] in, and Both represent the impedance parameters of the robot, f r and They represent the external interaction force and force change on the end of the robot, ΔX rRepresents the motion correction of the robot. By testing (15), the predicted trajectory of Equation (2) is corrected to obtain the expected trajectory X with the optimal cooperative comfort indicator and cooperative smoothness index. d .

[0115] S6. Use the PD position control method to drive the robot to execute the corrected motion prediction trajectory, and finally enable the robot to actively follow human motion.

[0116] The control law of the PD position control method is:

[0117] This is formula (16);

[0118] Among them, K p and K d Both represent the parameters of PD position control, X d and Denote the desired trajectory and velocity, respectively, X r and Represent the actual position and speed of the robot, F rb Represents the control torque. The robot drives the robot to move according to the calculation result of formula (16), thereby completing human-machine collaboration.

[0119] The control law of the PD position control method further includes the following steps:

[0120] Based on the motion prediction trajectory corrected in step S5, the feedforward torque compensation term is calculated in combination with the robot dynamics equation, and the feedforward compensation amount F rf for:

[0121] This is formula (17);

[0122] Among them, M r (X r ), G r (X r ) represents the robot dynamic parameters, are the velocity and acceleration of the desired trajectory;

[0123] Then, based on the result of formula (17) and combined with the calculated torque of formula (16), the control torque of the machine is obtained as:

[0124]

[0125] This is formula (18).

[0126] According to the result of formula (18), the robot is driven to move along the desired trajectory, thereby achieving more accurate active human-robot collaboration.

[0127] The above specific implementation manner is a preferred embodiment of the present invention and does not limit the present invention. Any other changes or other equivalent replacement methods that do not deviate from the technical solution of the present invention are included in the protection scope of the present invention.

Claims

1. An active human-machine collaboration method based on multimodal information of the human body, characterized in that: The following steps are involved: S1. Before the human-robot collaboration process, collect joint angle signals related to human arm movement, myoelectric signals at multiple positions of the human arm, and end position signals and force signals of the robot, and create a data set; S2. Construct a prediction neural network and train it using the joint angle signals and end position signals in the dataset; Construct an estimation neural network and use the electromyographic signals and force signals in the dataset to train the estimation neural network; During human-machine collaboration, human joint angle signals and myoelectric signals are collected in real time and input into the trained prediction neural network and estimation neural network respectively to obtain the predicted motion trajectory and three-dimensional arm force of the arm. S3. Build a refined perception interface for human collaboration: Where M represents the number of steps of the predicted trajectory; represents the arm strength output of the person in step i; is the actual running trajectory of the robot in step i, i is a natural number, W h represents the output work of humans, Δ represents the effective stroke of the robot, represents the quantitative indicator of collaboration comfort, σ E It represents a quantitative indicator of stationarity; S4. Input the three-dimensional arm force into the refined perception interface of human collaboration status to obtain collaborative comfort index and stability quantitative index; S5. Weighting the collaborative comfort index and the stability quantitative index to obtain the optimization target, using an exploratory iterative algorithm to determine the optimal robot impedance parameter, and then setting the robot's impedance model based on this impedance parameter to achieve the desired trajectory of the arm's motion, which is corrected according to the interactive external force. S6. Use the PD position control method to drive the robot to execute the corrected motion prediction trajectory, and finally enable the robot to actively follow human motion.

2. The active human-machine collaboration method based on multimodal human information according to claim 1, characterized in that: The prediction neural network includes a time correlation analysis module and a space continuity analysis module connected in sequence; The temporal correlation analysis module adopts a sequence-to-sequence model structure and integrates a multi-layer long short-term memory network to capture the temporal dependency between motion trajectories; The spatial continuity analysis module uses a graph structure to establish the correlation between predicted trajectories to analyze the spatial correlation between motion trajectories.

3. The active human-machine collaboration method based on multimodal human information according to claim 1, characterized in that: The estimation neural network is composed of parallel long and short-term memory networks.

4. The active human-machine collaboration method based on multimodal human information according to claim 1, characterized in that: The quantitative index of collaborative comfort is the average effective output force of humans during the collaborative process, that is, the ratio of human output work to the effective stroke of the robot; Among them, the expression of human output work is: Where M represents the number of steps of the predicted trajectory; represents the arm strength output of the person in step i; is the actual running trajectory of the robot in step i, where i is a natural number; The expression of the robot's effective travel is: Then the expression of the quantitative index of collaborative comfort is:

5. The active human-machine collaboration method based on multimodal human information according to claim 1, characterized in that: The stability quantitative index is the degree of fluctuation of human arm force during the collaboration process. The expression of this stability quantitative index is:

6. The active human-machine collaboration method based on multimodal human information according to claim 1, characterized in that: In step S5, the process of obtaining the optimization target is as follows: Where R represents the optimization target, and λ1 and λ2 represent weight coefficients.

7. The active human-machine collaboration method based on multimodal human information according to claim 6, characterized in that: In step S5, the iterative formula of the exploration iterative algorithm is: Among them, ε n represents the update step size, and ε n =c / n 1 / 2 , a represents the result of random sampling from a uniform distribution in the interval [0,1], For R about The gradient value, P n represents the exploration probability, R represents the optimization target, Indicates the impedance parameter that needs to be updated iteratively, n indicates the number of iterations, express The nth iteration value of .

8. The active human-machine collaboration method based on multimodal human information according to claim 1, characterized in that: The impedance model is expressed as: and Both represent the impedance parameters of the robot, f r and They represent the external interaction force and force change on the end of the robot, ΔX r Indicates the motion correction amount of the robot.

9. The active human-machine collaboration method based on multimodal human information according to claim 1, characterized in that: The control law of the PD position control method is: Among them, K p and K d Both represent the parameters of PD position control, X d and Denote the desired trajectory and speed respectively, X r and Represent the actual position and speed of the robot, F rb Indicates the control torque.

10. The active human-machine collaboration method based on multimodal human information according to claim 9, characterized in that: The control law of the PD position control method further includes the following steps: Based on the motion prediction trajectory corrected in step S5, the feedforward torque compensation term is calculated in combination with the robot dynamics equation, and the feedforward compensation amount F rf for: Among them, M r (X r ), G r (X r ) represents the robot dynamic parameters, are the velocity and acceleration of the desired trajectory; Then, based on The results, combined with The calculated torque of the machine is:

Images