Mechanical arm control method based on video motion capture and interference prediction technology

Abstract

The invention discloses a mechanical arm control method based on a video motion capture and interference prediction technology. The method comprises the following steps that 1, a human body motion video is collected through a single camera; step 2, extracting 3D skeleton joint point data by using a Mediape large model to obtain a target action sequence; 3, real-time motion parameters are issued through the ROS to control the mechanical arm; step 4, capturing an actual action sequence by using a binocular camera, and making a difference between the actual action sequence and the target action sequence; step 5, step 4, combining GPIO with MPC model prediction, and estimating interference caused by visual transmission; and 5, carrying out feedback suppression on the predicted time delay interference. The invention provides a controller design method for reducing time delay interference of a mechanical arm for solving the problem that the mechanical arm with disturbance and time delay tracks human body actions in real time. The method has certain control performance, and can effectively reduce the adverse effect of time delay interference on system control.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and robot control, and in particular to a robotic arm control method based on video motion capture and interference prediction technology. Background Art

[0002] In recent years, with the rise and development of machine learning and the demands of the robotics industry, machine learning technology has been increasingly applied to robotics. However, video motion capture is mostly applied in software, while robotics technology that integrates video motion capture with hardware has been relatively slow to develop. However, in the industrial application of robotics, the integration of software and hardware is an inevitable trend. Therefore, this invention utilizes the MediaPipe large model to capture human motion, obtain 3D skeleton points of human motion, and use this skeleton point information to control a robotic arm. This allows real-time control of the robotic arm using video motion capture, achieving a comprehensive integration of software and hardware.

[0003] Since the visual servo system will inevitably produce a time lag effect due to the sensor, in the traditional video motion capture technology of controlling the robotic arm, most control algorithms deal with interference by treating the interference as a constant and eliminating it through feedback, or obtaining the interference at the current moment through an observer. However, due to the existence of time lag, in most cases only the interference at the kd moment can be obtained. Therefore, the GPIO interference prediction technology is designed to estimate the interference at the k moment by estimating the various order derivatives of the interference. The estimated interference is then used in the MPC model prediction to improve the accuracy and stability of the MPC model prediction. Summary of the Invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a robotic arm control method based on video motion capture and interference prediction technology to compensate for the time lag interference problem existing in the visual servo system.

[0005] Technical solution: In order to achieve the above-mentioned purpose, the present invention proposes a robotic arm control method based on video motion capture and interference prediction technology, which includes the following steps:

[0006] Step S1: using a monocular camera to collect RGB video streams of human body operation movements in real time, using a 30fps frame rate for dynamic capture;

[0007] Step S2: Use the MediaPipe large model to extract the coordinates of 33 3D joint points of the human body in each frame of the video, form a human motion coordinate sequence R, and convert it into a robotic arm coordinate sequence J that is adapted to the robotic arm structure. k ;

[0008] Step S3, transmitting the human motion coordinate sequence R data extracted by MediaPipe to the robotic arm system through the ROS robot operating system;

[0009] Step S4, driving the robotic arm to move to the coordinate position corresponding to the first frame;

[0010] Step S5: Use binocular cameras to capture the robot arm's motion process, and use the Mediapipe large model to generate the motion sequence K k ;

[0011] Step S6, calculate the captured action sequence K k and human action sequence J k Difference E k , generate the control sequence U 1,k ;

[0012] Step S7, using the robot's own structural features and action sequence J k , establish a mathematical model of the manipulator control method including time delay interference, and generate a control sequence U to eliminate the interference 2,k ;

[0013] Step S8, using U 1,k and U 2,k Generate the final control sequence U k , transmitted to the robotic arm system to drive the robotic arm;

[0014] In step S9, the binocular camera is used again to continuously capture the robot arm's motion effects, and steps S2 to S8 are repeated.

[0015] Furthermore, the steps of step S2 are specifically as follows:

[0016] Step S2.1: Decompose the RGB video stream captured by the monocular camera frame by frame at 30fps. The timestamp is:

[0017]

[0018] Where s is the time unit in seconds, and N is the total number of frames after video decomposition;

[0019] Step S2.2: The BlazePose algorithm based on the MediaPipe large model extracts the coordinates of 33 3D joint points of the human body in each frame image and constructs the human skeleton motion sequence R = {R k |k=1,2,…,N}, where The first dimension is the index of the human joint point defined by BlazePose; the second dimension is the coordinate component (x, y, z), in meters (m), where z is the pseudo depth value based on monocular depth estimation;

[0020] Step S2.3: Establish the human-robot joint mapping relationship through kinematic chain analysis, and convert the skeleton motion sequence R into the robot joint space coordinate sequence Where n is the degree of freedom of the robot arm.

[0021] Furthermore, the steps of step S3 are as follows:

[0022] Step S3.1: Encode the coordinate sequence J into a JointTrajectory message type through the ROS middleware architecture and transmit it to the robotic arm control system using the UDP protocol;

[0023] Step S3.2: deploy ROS nodes on the robot control computer to receive the coordinate sequence J in the standard message format. k and cached.

[0024] Furthermore, the steps of step S4 are specifically as follows:

[0025] Step S4.1, assume that the robot arm is a 2-DOF planar configuration, the length from the base to the first joint is L1, the length from the first joint to the second joint is L2, and the first frame coordinate J is analyzed. k =(x,y) has the following parameters:

[0026] Step S4.1.1, calculate the distance from the end of the robot arm to the base

[0027] Step S4.1.2, calculate the second joint angle according to the cosine theorem,

[0028] Step S4.1.3, calculate the first joint angle,

[0029] Step S4.1.4, drive the joint motor to θ target =[θ1,θ2] T .

[0030] Furthermore, the steps of step S5 are specifically as follows:

[0031] Step S5.1: The binocular camera captures the robot arm's motion process, performs data preprocessing on the captured motion video, and extracts each frame;

[0032] Step S5.2: Use the Mediapipe large model to migrate the models of each frame and extract the coordinates of each joint point at different time points;

[0033] Step S5.3: Encode the coordinates of each joint at different time points to form the actual action sequence of the robot arm n is the degree of freedom of the robot arm.

[0034] Furthermore, the steps of step S6 are specifically as follows:

[0035] Step S6.1, define the dynamic error function Ek =J k -K k , which is the robot arm joint space tracking error vector of the kth frame, k is the frame number, k = 1, 2, ..., N, and the error dynamics model including the delay factor τ is constructed:

[0036]

[0037] in, is the discretized state matrix of the robotic arm system, is the discrete control input matrix, is the external disturbance vector, represents the control input vector of the k-τth discrete time step, τ is the delay step number, which represents the delay step number from the generation of the control instruction to the actual action on the robot arm; m is the control input dimension of the robot arm;

[0038] Step S6.2, design an adaptive robust controller:

[0039]

[0040] in, The weight used to adjust the current tracking error; is the integral gain matrix, which is used to eliminate the steady-state error; is the differential gain matrix, p is the output dimension of the disturbance observer; is the disturbance estimation vector output by the time-delay disturbance observer, satisfying in, is the interference prediction value of the kth frame, d represents the system measurement delay lag of d steps, which is an integer.

[0041] Furthermore, the steps of step S7 are specifically as follows:

[0042] Step S7.1: For the visual servo system considering time delay, the discrete time state space equation of the system is constructed as follows:

[0043]

[0044] in, is the state vector of the robotic arm system, including joint positions and velocities, n is the degree of freedom, is the input vector of the robot arm when controlling the kth step, It is the time-free output of the robot arm joint coordinates at time k. is the measurement output of the robot arm joint measurement with a d-step delay, is the interference input matrix;

[0045] Step S7.2, get the current state

[0046]

[0047] Step S7.3, design the time-delay disturbance observer

[0048]

[0049] is the observer gain matrix; is the observer state vector, which is used to estimate the interference at time k and the 1st to dth order difference of the interference, Z d is the observer state vector at time d;

[0050]

[0051] I is the identity matrix;

[0052] Step S7.4, estimate the interference antecedent difference:

[0053]

[0054] in,

[0055] Step S7.5, calculate the estimated state at time k:

[0056]

[0057] Step S7.6, construct a robust predictive control law based on time-delay compensation:

[0058]

[0059] in, is the control input influence matrix; is the state tracking error weight matrix;

[0060] Input penalty matrix for control; is the reference state trajectory; is the terminal cost weight matrix; is the reference state vector after time delay compensation, is the interference impact matrix;

[0061] is the interference estimation residual.

[0062] Furthermore, the steps of step S8 are specifically as follows:

[0063] Step S8.1: Design dynamic mixing weight α∈(0,1):

[0064]

[0065] Generate the final control instruction U k =αU 1,k +(1-α)U 2,k ;

[0066] Step S8.2: Using the generated control sequence U k Control the robotic arm;

[0067] Step S9: Repeat steps S2-S8.

[0068] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0069] (1) The present invention aims at a video motion capture control robotic arm system under the influence of time-delay interference and external bounded disturbance. It adopts the MediaPipe large model to capture the joint points of human body motion posture, uses ROS to publish topics to control the robotic arm in real time to imitate human posture, and adopts MPC model predictive control combined with GPIO interference prediction algorithm to improve the tracking control performance and robustness of the control system.

[0070] (2) The robotic arm control system and method based on the combination of MediaPipe video motion capture and MPC model prediction and interference prediction technology can promote the integrated development of the field of machine learning and the field of robot control, realize the combination of software and hardware, and the combination of MPC model prediction and interference prediction technology can effectively reduce the adverse effects of time delay interference on the control effect in the industrial visual servo system, and can help the robotic arm system tracking error converge and stabilize faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 This is a flowchart of the implementation of a robotic arm control method based on video motion capture and interference prediction technology described in the present invention.

[0072] Figure 2 This is the MPC model predictive control anti-interference model described in the present invention. DETAILED DESCRIPTION

[0073] The present invention will be further explained below with reference to the accompanying drawings:

[0074] like Figure 1 、 Figure 2 As shown, the present invention proposes a robotic arm control method based on video motion capture and interference prediction technology, which includes the following steps:

[0075] Step S1: using a monocular camera to collect RGB video streams of human body operation movements in real time, using a 30fps frame rate for dynamic capture;

[0076] Step S2: Use the MediaPipe large model to extract the coordinates of 33 3D joint points of the human body in each frame of the video, form a human motion coordinate sequence R, and convert it into a robotic arm coordinate sequence J that is adapted to the robotic arm structure. k ;

[0077] Step S3, transmitting the human motion coordinate sequence R data extracted by MediaPipe to the robotic arm system through the ROS robot operating system;

[0078] Step S4, driving the robotic arm to move to the coordinate position corresponding to the first frame;

[0079] Step S5: Use binocular cameras to capture the robot arm's motion process, and use the Mediapipe large model to generate the motion sequence K k ;

[0080] Step S6, calculate the captured action sequence K k and human action sequence J k Difference E k , generate the control sequence U 1,k ;

[0081] Step S7, using the robot's own structural characteristics and action sequence J k , establish a mathematical model of the manipulator control method including time delay interference, and generate a control sequence U to eliminate the interference 2,k ;

[0082] Step S8, using U 1,k and U 2,k Generate the final control sequence U k , transmitted to the robotic arm system to drive the robotic arm;

[0083] Step S9: Use the binocular camera again to continuously capture the robot arm's motion effects, and repeat steps S2 to S8.

[0084] Furthermore, the steps of step S2 are specifically as follows:

[0085] Step S2.1: Decompose the RGB video stream captured by the monocular camera frame by frame at 30fps. The timestamp is:

[0086]

[0087] Where s is the time unit in seconds, and N is the total number of frames after video decomposition;

[0088] Step S2.2: The BlazePose algorithm based on the MediaPipe large model extracts the coordinates of 33 3D joint points of the human body in each frame image and constructs the human skeleton motion sequence R = {R k |k=1,2,…,N}, where The first dimension is the index of the human joint point defined by BlazePose; the second dimension is the coordinate component (x, y, z), in meters (m), where z is the pseudo depth value based on monocular depth estimation;

[0089] Step S2.3: Establish the human-robot joint mapping relationship through kinematic chain analysis, and convert the skeleton motion sequence R into the robot joint space coordinate sequence Where n is the degree of freedom of the robot arm.

[0090] Furthermore, the steps of step S3 are as follows:

[0091] Step S3.1: Encode the coordinate sequence J into a JointTrajectory message type through the ROS middleware architecture and transmit it to the robotic arm control system using the UDP protocol;

[0092] Step S3.2: deploy ROS nodes on the robot control computer to receive the coordinate sequence J in the standard message format. k and cached.

[0093] Furthermore, the steps of step S4 are specifically as follows:

[0094] Step S4.1, assume that the robot arm is a 2-DOF planar configuration, the length from the base to the first joint is L1, the length from the first joint to the second joint is L2, and the first frame coordinate J is analyzed. k =(x,y) has the following parameters:

[0095] Step S4.1.1, calculate the distance from the end of the robot arm to the base

[0096] Step S4.1.2, calculate the second joint angle according to the cosine theorem,

[0097] Step S4.1.3, calculate the first joint angle,

[0098] Step S4.1.4, drive the joint motor to θ target =[θ1,θ2] T .

[0099] Furthermore, the steps of step S5 are specifically as follows:

[0100] Step S5.1: The binocular camera captures the robot arm's motion process, performs data preprocessing on the captured motion video, and extracts each frame;

[0101] Step S5.2: Use the Mediapipe large model to migrate the models of each frame and extract the coordinates of each joint point at different time points;

[0102] Step S5.3: Encode the coordinates of each joint at different time points to form the actual action sequence of the robot arm n is the degree of freedom of the robot arm.

[0103] Furthermore, the steps of step S6 are specifically as follows:

[0104] Step S6.1, define the dynamic error function E k =J k -K k , which is the robot arm joint space tracking error vector of the kth frame, k is the frame number, k = 1, 2, ..., N, and the error dynamics model including the delay factor τ is constructed:

[0105]

[0106] in, is the discretized state matrix of the robotic arm system, is the discrete control input matrix, is the external disturbance vector, represents the control input vector of the k-τth discrete time step, τ is the delay step number, which represents the delay step number from the generation of the control instruction to the actual action on the robot arm; m is the control input dimension of the robot arm;

[0107] Step S6.2, design an adaptive robust controller:

[0108]

[0109] in, The weight used to adjust the current tracking error; is the integral gain matrix, which is used to eliminate the steady-state error; is the differential gain matrix, p is the output dimension of the disturbance observer; is the disturbance estimation vector output by the time-delay disturbance observer, satisfying in, is the interference prediction value of the kth frame, d represents the system measurement delay lag of d steps, which is an integer.

[0110] Furthermore, the steps of step S7 are specifically as follows:

[0111] Step S7.1: For the visual servo system considering time delay, the discrete time state space equation of the system is constructed as follows:

[0112]

[0113] in, is the state vector of the robotic arm system, including joint positions and velocities, n is the degree of freedom, is the input vector of the robot arm when controlling the kth step, It is the time-free output of the robot arm joint coordinates at time k. is the measurement output of the robot arm joint measurement with a d-step delay, is the interference input matrix;

[0114] Step S7.2, get the current state

[0115]

[0116] Step S7.3, design the time-delay disturbance observer

[0117]

[0118] is the observer gain matrix; is the observer state vector, which is used to estimate the interference at time k and the 1st to dth order difference of the interference, Z d is the observer state vector at time d;

[0119]

[0120] I is the identity matrix;

[0121] Step S7.4, estimate the interference antecedent difference:

[0122]

[0123] in,

[0124] Step S7.5, calculate the estimated state at time k:

[0125]

[0126] Step S7.6, construct a robust predictive control law based on time-delay compensation:

[0127]

[0128] in, is the control input influence matrix; is the state tracking error weight matrix; Input penalty matrix for control; is the reference state trajectory; is the terminal cost weight matrix; is the reference state vector after time delay compensation, is the interference impact matrix; is the interference estimation residual.

[0129] Furthermore, the steps of step S8 are specifically as follows:

[0130] Step S8.1: Design dynamic mixing weight α∈(0,1):

[0131]

[0132] Generate the final control instruction U k =αU 1,k +(1-α)U 2,k ;

[0133] Step S8.2: Using the generated control sequence U k Control the robotic arm;

[0134] Step S9: Repeat steps S2-S8.

Claims

1. A robotic arm control method based on video motion capture and interference prediction technology, characterized in that: The method comprises the following steps: Step S1: using a monocular camera to collect RGB video streams of human body operation movements in real time, using a 30fps frame rate for dynamic capture; Step S2: Use the MediaPipe large model to extract the coordinates of 33 3D joint points of the human body in each frame of the video, form a human motion coordinate sequence R, and convert it into a robotic arm coordinate sequence J that is adapted to the robotic arm structure. k ; Step S3, transmitting the human motion coordinate sequence R data extracted by MediaPipe to the robotic arm system through the ROS robot operating system; Step S4, driving the robotic arm to move to the coordinate position corresponding to the first frame; Step S5: Use binocular cameras to capture the robot arm's motion process, and use the Mediapipe large model to generate the motion sequence K k ; Step S6, calculate the captured action sequence K k and human action sequence J k Difference E k , generate the control sequence U 1,k ; Step S7, using the robot's own structural features and action sequence J k , establish a mathematical model of the manipulator control method including time delay interference, and generate a control sequence U to eliminate the interference 2,k ; Step S8, using U 1,k and U 2,k Generate the final control sequence U k , transmitted to the robotic arm system to drive the robotic arm; In step S9, the binocular camera is used again to continuously capture the robot arm's motion effects, and steps S2 to S8 are repeated.

2. The robotic arm control method based on video motion capture and interference prediction technology according to claim 1, characterized in that: The steps of step S2 are as follows: Step S2.1: Decompose the RGB video stream captured by the monocular camera frame by frame at 30fps. The timestamp is: t k =k·Δt, Where s is the time unit in seconds, and N is the total number of frames after video decomposition; Step S2.2: The BlazePose algorithm based on the MediaPipe large model extracts the coordinates of 33 3D joint points of the human body in each frame image and constructs the human skeleton motion sequence R = {R k |k=1,2,…,N}, where The first dimension is the index of the human joint point defined by BlazePose; the second dimension is the coordinate component (x, y, z), in meters (m), where z is the pseudo depth value based on monocular depth estimation; Step S2.3: Establish the human-robot joint mapping relationship through kinematic chain analysis, and convert the skeleton motion sequence R into the robot joint space coordinate sequence Where n is the degree of freedom of the robot arm.

3. The robotic arm control method based on video motion capture and interference prediction technology according to claim 2, characterized in that: The steps of step S3 are as follows: Step S3.1: Encode the coordinate sequence J into a JointTrajectory message type through the ROS middleware architecture and transmit it to the robotic arm control system using the UDP protocol; Step S3.2: deploy ROS nodes on the robot control computer to receive the coordinate sequence J in the standard message format. k and cached.

4. The method for controlling a robotic arm based on video motion capture and interference prediction technology according to claim 3, characterized in that: The steps of step S4 are as follows: Step S4.1, assume that the robot arm is a 2-DOF planar configuration, the length from the base to the first joint is L1, the length from the first joint to the second joint is L2, and the first frame coordinate J is analyzed. k =(x,y) has the following parameters: Step S4.1.1, calculate the distance from the end of the robot arm to the base Step S4.1.2, calculate the second joint angle according to the cosine theorem, Step S4.1.3, calculate the first joint angle, Step S4.1.4, drive the joint motor to θ target =[θ1,θ2] T .

5. The method for controlling a robotic arm based on video motion capture and interference prediction technology according to claim 4, characterized in that: The steps of step S5 are as follows: Step S5.1: The binocular camera captures the robot arm's motion process, performs data preprocessing on the captured motion video, and extracts each frame; Step S5.2: Use the Mediapipe large model to migrate the models of each frame and extract the coordinates of each joint point at different time points; Step S5.3: Encode the coordinates of each joint at different time points to form the actual action sequence of the robot arm n is the degree of freedom of the robot arm.

6. The method for controlling a robotic arm based on video motion capture and interference prediction technology according to claim 5, characterized in that: The steps of step S6 are as follows: Step S6.1, define the dynamic error function E k =J k -K k , which is the robot arm joint space tracking error vector of the kth frame, k is the frame number, k = 1, 2, ..., N, and the error dynamics model including the delay factor τ is constructed: in, is the discretized state matrix of the robotic arm system, is the discrete control input matrix, is the external disturbance vector, represents the control input vector of the k-τth discrete time step, τ is the delay step number, which represents the delay step number from the generation of the control instruction to the actual action on the robot arm; m is the control input dimension of the robot arm; Step S6.2, design an adaptive robust controller: in, The weight used to adjust the current tracking error; is the integral gain matrix, which is used to eliminate the steady-state error; is the differential gain matrix, p is the output dimension of the disturbance observer; is the disturbance estimation vector output by the time-delay disturbance observer, satisfying in, is the interference prediction value of the kth frame, d represents the system measurement delay lag of d steps, which is an integer.

7. The method for controlling a robotic arm based on video motion capture and interference prediction technology according to claim 6, characterized in that: The steps of step S7 are as follows: Step S7.1: For the visual servo system considering time delay, the discrete time state space equation of the system is constructed as follows: in, is the state vector of the robotic arm system, including joint positions and velocities, n is the degree of freedom, is the input vector of the robot arm when controlling the kth step, It is the time-free output of the robot arm joint coordinates at time k. is the measurement output of the robot arm joint measurement with a d-step delay, is the interference input matrix; Step S7.2, get the current state Step S7.3, design the time-delay disturbance observer is the observer gain matrix; is the observer state vector, which is used to estimate the interference at time k and the 1st to dth order difference of the interference, Z d is the observer state vector at time d; I is the identity matrix; Step S7.4, estimate the interference antecedent difference: in, Step S7.5, calculate the estimated state at time k: Step S7.6, construct a robust predictive control law based on time-delay compensation: in, is the control input influence matrix; is the state tracking error weight matrix; Input penalty matrix for control; is the reference state trajectory; is the terminal cost weight matrix; is the reference state vector after time delay compensation, is the interference impact matrix; is the interference estimation residual.

8. The method for controlling a robotic arm based on video motion capture and interference prediction technology according to claim 7, characterized in that: The steps of step S8 are as follows: Step S8.1: Design dynamic mixing weight α∈(0,1): Generate the final control instruction U k =αU 1,k +(1-α)U 2,k ; Step S8.2: Using the generated control sequence U k Control the robotic arm; Step S9: Repeat steps S2-S8.

Images