A human-machine collaborative motion control method for robotic arms based on machine vision

Abstract

The present invention relates to the field of human-computer interaction, and in particular to a method for human-computer collaborative motion control of a robot arm based on machine vision. In-depth research has been conducted on the efficiency of human-computer collaboration and the motion control method of the robot arm, and a set of human-computer collaborative solutions based on machine vision has been proposed. For the two parts of motion recognition and robot arm motion control, a motion recognition system and a robot arm motion control system were built respectively, and finally the two parts were combined into an industrial robot arm motion control system based on machine vision, and a UI display interface was designed using Tkinter. It meets the operator's task requirements for adjusting the position of the end of the robot arm, adjusting the posture of the end of the robot arm, grabbing and putting down objects, etc., realizes the operator's contactless and remote-controlled robot arm control, solves the problem of high delay in existing human-computer interaction control, and has great application value in industrial sites.

Description

Technical Field

[0001] The present invention belongs to the field of human-computer interaction, and in particular relates to a human-computer collaborative motion control method for a robotic arm based on machine vision. Background Art

[0002] With the continuous development of industrial automation and intelligent manufacturing, the application of robotic arms in industrial production is becoming increasingly widespread. The development of robotic arms capable of collaborating with humans and possessing intelligent perception and autonomous decision-making capabilities has become a hot topic of research. Machine vision technology, as an important perception tool, can capture images or videos through devices such as cameras and analyze them using image processing algorithms to extract useful information. In the field of human-computer interaction, machine vision can be used to monitor and recognize human movements in real time, providing a basis for the motion control of robotic arms. Human motion recognition technology uses machine vision to obtain the position coordinates of each human joint based on information identified in videos or images. It primarily uses computer vision methods to identify and predict human posture. Natural human-computer interaction modes, such as voice, video, and touch, are user-centric. From a recognition perspective, a robot's posture recognition capabilities are crucial to its ability to perform tasks.

[0003] Current technical issues and challenges include: Traditional robotic arms typically rely on pre-programmed trajectories for operation, lacking flexibility and adaptability, and unable to cope with complex and changing production environments. Human movements are highly diverse and complex, with significant differences in movement habits and postures between individuals. Even the same individual may perform the same movement differently at different times. Furthermore, factors such as lighting variations, background clutter, and occlusions can affect the accuracy of motion recognition. Traditional human-machine interaction methods include keyboards, remote controls, and joystick controls, but these are no longer sufficient to meet increasingly complex operational requirements. Robotic arm motion control requires high real-time performance to ensure timely response to operator commands. However, complex deep learning models, particularly LSTM models, typically require significant computing resources, which can lead to processing delays and affect the system's real-time performance. Improving motion recognition and processing speed while maintaining recognition accuracy is crucial. Seamlessly integrating the motion recognition module with the robotic arm control module to achieve stable and efficient communication and coordinated control also involves multiple technical challenges. Consequently, existing human-machine collaborative motion control suffers from high latency. Summary of the Invention

[0004] The purpose of the present invention is to solve the problems of unreasonable human-computer interaction, low efficiency, high communication delay, etc. in the existing technology, and proposes a human-computer collaborative motion control method for a robotic arm based on machine vision.

[0005] 1. Construct a human-machine collaborative motion control model for a robotic arm based on machine vision; train and obtain a trained human-machine collaborative motion control model for a robotic arm based on machine vision;

[0006] 2. Collect real-time action images of operators;

[0007] 3. Input the collected real-time action images of the operator into the trained machine vision-based human-machine collaborative motion control model of the robot arm to judge the real-time action of the operator and output the corresponding robot arm control instructions according to the action;

[0008] 4. Inputting the robot arm control command into the robot arm to realize the motion control of the robot arm;

[0009] In the first step, a human-machine collaborative motion control model for a robotic arm based on machine vision is constructed; and a trained human-machine collaborative motion control model for a robotic arm based on machine vision is obtained through training. The specific process is as follows:

[0010] S1: Collect videos of standard human movements as sample data of standard human movements, construct a standard human movement template based on the sample data of standard human movements, and then construct motion control instructions for the robotic arm based on the standard human movement template;

[0011] S2: Construct a human-machine collaborative motion control model for a robotic arm based on machine vision;

[0012] S3: Use the images of standard human movements as the standard movement training set, and train the machine vision-based human-machine collaborative motion control model of the robotic arm according to the standard movement training set to obtain a trained machine vision-based human-machine collaborative motion control model of the robotic arm.

[0013] The beneficial effects of the present invention are:

[0014] In-depth research has been conducted on the efficiency of human-robot collaboration and the motion control of robotic arms, leading to the development of a human-robot collaboration solution based on machine vision. A motion recognition system and a robotic arm motion control system were developed for each component, respectively. Ultimately, these two components were combined into a machine vision-based industrial robotic arm motion control system, complete with a user interface designed using Tkinter. This system meets the operator's needs for tasks such as adjusting the end-arm position and posture, and grasping and releasing objects. It enables contactless and remote robotic arm control, while overcoming the high latency issues associated with existing technologies and possessing significant application value in industrial settings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is the workflow diagram of the UR5 robotic arm motion control system based on machine vision in the present invention;

[0016] Figure 2 Figure 2a is a schematic diagram of the Hands model processing skeleton points in the MediaPipe pose estimation algorithm;

[0017] Figure 2 Figure 2b is a schematic diagram of the Pose model processing skeleton points of the MediaPipe pose estimation algorithm;

[0018] Figure 3 3a is the image processing result of the Hands model of the MediaPipe pose estimation algorithm in the present invention;

[0019] Figure 3 3b is the Pose model image processing result of the MediaPipe pose estimation algorithm in the present invention;

[0020] Figure 3 3c is the result of comprehensive image processing by the MediaPipe pose estimation algorithm in the present invention;

[0021] Figure 4 This is the LSTM network model structure diagram in the present invention;

[0022] Figure 5 5a is the Accuracy curve of the LSTM model in the present invention;

[0023] Figure 5 5b is the Loss curve of the LSTM model in the present invention;

[0024] Figure 6 This is a flowchart of the real-time human motion recognition system of the present invention;

[0025] Figure 7 This is a flow chart of the communication motion control between the UR5 robotic arm and the command issuing end in the present invention;

[0026] Figure 8 This is a structural block diagram of the motion control system of the present invention;

[0027] Figure 9 This is the UI display interface of the motion control system in the present invention. DETAILED DESCRIPTION

[0028] Specific implementation method 1: Combination Figure 1 The present invention is described, comprising:

[0029] 1. Construct a human-machine collaborative motion control model for a robotic arm based on machine vision; train and obtain a trained human-machine collaborative motion control model for a robotic arm based on machine vision;

[0030] 2. Collect real-time action images of operators;

[0031] 3. Input the collected real-time action images of the operator into the trained machine vision-based human-machine collaborative motion control model of the robot arm to judge the real-time action of the operator and output the corresponding robot arm control instructions according to the action;

[0032] 4. Inputting the robot arm control command into the robot arm to realize the motion control of the robot arm;

[0033] In the first step, a human-machine collaborative motion control model for a robotic arm based on machine vision is constructed; and a trained human-machine collaborative motion control model for a robotic arm based on machine vision is obtained through training. The specific process is as follows:

[0034] S1: Collect videos of standard human movements as sample data of standard human movements, construct a standard human movement template based on the sample data of standard human movements, and then construct motion control instructions for the robotic arm based on the standard human movement template;

[0035] S2: Construct a human-machine collaborative motion control model for a robotic arm based on machine vision;

[0036] S3: Use the images of standard human movements as the standard movement training set, and train the machine vision-based human-machine collaborative motion control model of the robotic arm according to the standard movement training set to obtain a trained machine vision-based human-machine collaborative motion control model of the robotic arm.

[0037] To address issues such as the relatively incomplete image feature information extracted using image recognition technology, the need to improve the accuracy of motion recognition methods using the human skeleton, and the inability of machines to quickly understand human intentions in human-machine collaboration, this solution has developed a machine vision-based industrial robotic arm motion control system for a single operator based on the UR5 robotic arm experimental environment. When the operator moves according to pre-set movements, the system camera can immediately capture the human body movements and match them in the motion recognition module. If the movement at this time can be determined to be meaningful, the robotic arm control code corresponding to the movement is transmitted to the UR5 robotic arm control system through the communication module, allowing the robotic arm to complete the corresponding movement. Through a series of operations, any operation of the robotic arm can be completed at a distance from the robotic arm, including adjusting the position of the robotic arm end, adjusting the posture of the robotic arm end, grabbing and placing objects, etc., realizing contactless and remote control of the operator.

[0038] Specific embodiment 2: The difference between this embodiment and specific embodiment 1 is that in S1, a video of standard human movements is collected as sample data of standard human movements, a standard human movement template is constructed based on the sample data of standard human movements, and then a motion control instruction of the robotic arm is constructed based on the standard human movement template; the specific process is as follows:

[0039] S1.1. Collect videos of standard human movements as sample data of standard human movements;

[0040] S1.2. Construct the original dataset based on the sample data of standard human movements;

[0041] S1.3. Preprocess the original data set to obtain the preprocessed images of standard actions;

[0042] S1.4, processing the pre-processed image obtained in S1.3 to obtain human skeleton point data;

[0043] S1.5. Construct a human standard motion template based on the human skeleton point data obtained in S1.4; and use the human standard motion template as a standard motion training set;

[0044] S1.6. Based on the standard motion template obtained in S1.5, construct motion control instructions for the robot arm's motion.

[0045] Other steps and parameters are the same as those in the first embodiment.

[0046] Specific embodiment three: This embodiment differs from the specific embodiment one in that:

[0047] The sample data of standard human movements in S1.1 are collected. The specific process is: using a Kinect industrial-grade 2k depth camera to collect standard static action images and standard dynamic action videos of human movements as sample data of standard human movements;

[0048] The original data set in S1.2 includes a standard static action original data set and a standard dynamic action original data set; the original data set is constructed based on the sample data of standard human body movements; the specific process is:

[0049] S1.2.1. Use the standard static action images collected in S1.1 as the original data set of standard static actions;

[0050] The standard static action images include: standard static gesture images and standard static body action images;

[0051] S1.2.2. Use the standard dynamic action video collected in S1.1 as the original data set of standard dynamic action;

[0052] In S1.3, the original data set is preprocessed to obtain the standard action preprocessed image; the specific process is:

[0053] S1.3.1. Extract the standard static action images from the original dataset of standard static actions described in S1.2.1; name them as preprocessed standard static action images;

[0054] S1.3.2. Extracting a preprocessed standard dynamic action image set based on the video data of the standard action in the original data set of the standard dynamic action described in S1.2.2;

[0055] The extraction process of each standard dynamic action in the pre-processed standard dynamic action image set is as follows:

[0056] Set the start time and end time of the standard dynamic action; extract each frame of the original image of the collected standard dynamic action video data at a frame rate of FPS = 30 as the pre-processed standard dynamic action image;

[0057] S1.3.3. The preprocessed standard static action image and the preprocessed standard dynamic action image set are used as standard action preprocessed images;

[0058] S1.4 processes the standard action pre-processed image obtained in S1.3 to obtain human skeleton point data; the specific process is:

[0059] Use the Hands and Pose models of the MediaPipe pose estimation algorithm to process the pre-processed images of standard actions to obtain skeleton point data;

[0060] The skeleton point diagram obtained after the Hands and Pose models are processed is as follows: Figure 2 2a in Figure 2 As shown in 2b;

[0061] The skeleton point data includes: human hand skeleton point data and body skeleton point data;

[0062] The human body's hand skeleton point data is the human body's two-hand skeleton point data;

[0063] The human body's two-hand skeleton point data includes information about the operator's left and right hands; the information about each hand includes coordinate information and position depth information of 21 hand skeleton points;

[0064] The body skeleton point data is the coordinate information and position depth information of 33 body skeleton points;

[0065] The human body skeleton point data and hand skeleton point data include the 3D coordinate point (x, y, z) of each hand skeleton point and body skeleton point in each image and the confidence information of the coordinate point;

[0066] The confidence level of the coordinate point represents the MediaPipe pose estimation algorithm's estimate of the probability of the accuracy of the coordinate point.

[0067] The other steps and parameters are the same as those in the first and second embodiments.

[0068] Specific embodiment 4: This embodiment differs from specific embodiments 1 to 4 in that:

[0069] The standard motion templates in S1.5 include: hand standard motion templates and body standard motion templates;

[0070] In S1.5, a standard action template is constructed based on the skeleton point data obtained in S1.4. The specific process is as follows:

[0071] S1.5.1. Construct a standard hand motion template based on the human hand skeleton point data;

[0072] The hand standard motion template includes a hand standard static motion template and a hand standard dynamic motion template;

[0073] S1.5.2. Construct a standard body movement template based on the body skeleton point data.

[0074] The body standard movement template includes a body standard static movement template and a body standard dynamic movement template;

[0075] The other steps and parameters are the same as those in the first to third embodiments.

[0076] Specific embodiment 5: This embodiment differs from specific embodiments 1 to 4 in that, in S1.5.1, a standard hand motion template is constructed based on the hand skeleton point data of the human body; the specific process is as follows:

[0077] S1.5.1.1. Determine the normalized coordinates of the hand skeleton points of each hand of the operator; the specific process is as follows:

[0078] According to the human body's two-hand skeleton point data in S1.4, the skeleton point at the wrist of the k-th hand is taken as the coordinate origin of the skeleton point of the k-th hand;

[0079] Where k = 1, 2; k = 1 means left hand, k = 2 means right hand;

[0080] Calculate the normalized coordinate value of the a-th hand skeleton point of the k-th hand; the specific process is:

[0081] For the k-th hand, the coordinate value of the a-th hand bone point is subtracted from the coordinate value of the coordinate origin of the hand bone point to obtain the normalized coordinate value of each hand bone point.

[0082] Where a=0,1,2,…,20;

[0083] The hand skeleton point when a=0, that is, the normalized coordinates of the coordinate origin of the hand skeleton point

[0084]

[0085] S1.5.1.2. Constructing the standard hand position feature vector set d H ; The specific process is:

[0086] For the k-th hand, the normalized coordinates of the a-th hand skeleton point Using the hand position feature vector calculation formula Get the standard hand position feature vector of each hand bone point

[0087] The hand standard position feature vector to be calculated Does not include the hand bone point coordinate origin That is, a≠0;

[0088] in Represents the vector of the hand bone point coordinates in the positive x direction, which is numerically the same as the hand bone point coordinates. The values ​​are equal;

[0089] in Represents the vector of the hand bone point coordinates in the positive y direction, which is numerically the same as the hand bone point coordinates. The values ​​are equal;

[0090] in Represents the vector of the hand bone point coordinates in the positive z direction, which is numerically the same as the hand bone point coordinates. The values ​​are equal;

[0091] Traverse all available k values ​​and a values ​​to obtain the standard hand position feature vector of each hand bone point of both hands Construct the hand standard position feature vector set d H ;

[0092] The hand standard position feature vector set d H Contains 20 standard hand position feature vectors for each hand

[0093] S1.5.1.3. Calculate the velocity feature vector of each hand bone point and construct the standard hand velocity feature vector set v H The hand standard velocity feature vector only exists in the hand standard dynamic movement; the specific process is:

[0094] For the k-th hand, the normalized coordinates of the a-th hand skeleton point For the standard dynamic hand movements containing n frames, the velocity feature vector calculation formula is used Get the hand standard velocity feature vector of the i-th frame (1≤i≤n0) of the hand standard dynamic motion of each hand bone point

[0095] in

[0096] in

[0097] in

[0098] in

[0099] Traverse all available hand standard dynamic movements, k values, a values, and i values ​​to obtain the hand standard velocity feature vector of each hand bone point of each hand standard dynamic movement Constitutes the hand standard velocity feature vector set v H ;

[0100] The hand standard velocity feature vector set v H Contains 21n standard hand velocity feature vectors for each hand standard dynamic action (n frames of image)

[0101] S1.5.1.4. Construct the hand standard angle feature vector set θ H ; The specific process is:

[0102] Calculate the cosine value of the angle between the hand bone point and the surrounding bone points Will Recorded as the standard characteristic angle of the hand skeleton point;

[0103] The standard feature angle of the hand bone point to be calculated Does not include the hand bone point coordinate origin The bone point at the top of the finger, that is, a≠4m; where m=0,1,2,3,4,5;

[0104] The angle between the hand skeleton point and the surrounding skeleton points is calculated when calculating the hand skeleton point with a=4m+1, and the coordinate origin of the hand is calculated. The standard characteristic angle of the hand bone point formed by the two hand bone points a+1

[0105] The angle of the surrounding skeleton points of the hand skeleton point, when calculating the hand skeleton points a=4m+2 and a=4m+3, calculate the standard characteristic angle of the hand skeleton point formed by the same hand skeleton point and the two hand skeleton points a-1 and a+1 of the same hand

[0106] The standard characteristic angle of the hand skeleton point The calculation formula is:

[0107]

[0108] in

[0109] Traverse all available k values ​​and a values ​​to obtain the standard feature angle of each hand bone point of both hands Constitutes the hand standard angle feature vector set θ H ;

[0110] The hand standard angle feature vector set θ H Contains standard feature angles of 15 hand bone points for each hand

[0111]

[0112] The cosine value of the angle between each hand bone point and the adjacent bone point is calculated, and key bone points are selected to construct a standard hand angle feature vector set containing 15 hand angle feature vectors, as shown in the following table:

[0113]

[0114]

[0115] S1.5.1.5. Constructing the hand standard distance feature vector set L H ; The specific process is:

[0116] Calculate the coordinate origin of each hand bone point and the same hand Euclidean distance Note it as the standard feature distance of the hand skeleton points;

[0117] The standard feature distance of the hand skeleton point to be calculated The coordinate origin does not include the hand bone point That is, a≠0;

[0118] The standard feature distance of the hand skeleton points The calculation formula is:

[0119]

[0120] Where a≠0;

[0121] Traverse all available k values ​​and a values ​​to obtain the standard feature distance of each hand bone point of both hands Construct the hand standard distance feature vector set L H ;

[0122] The hand standard distance feature vector set L H Contains standard feature distances of 20 hand skeleton points for each hand

[0123]

[0124] The Euclidean distance between each hand bone point and the wrist bone point is calculated, and a standard hand distance feature vector set containing 20 distance feature vectors of the hand is constructed, as shown in the following table:

[0125]

[0126] The hand standard position feature vector set d H , hand standard velocity feature vector set v H , the hand standard angle feature vector set θ H , hand standard distance feature vector set L H , together constitute the hand action feature vector set H;

[0127] Constructing a standard static hand motion template based on the hand motion feature vector set H;

[0128] The standard static hand motion template includes 10 standard hand motions: left hand 1, left hand 2, left hand 3, left hand 4, left hand 8, right hand 1, right hand 2, right hand 3, right hand 4, right hand 8;

[0129] The hand skeleton feature extraction is completed;

[0130] In S1.5.2, a standard body movement template is constructed based on the body skeleton point data; the specific process is as follows:

[0131] S1.5.2.1. Use the coordinate data of the skeletal point at the nose as the coordinate origin of the body; subtract the coordinate value of the origin of the body skeletal point from the coordinate values ​​of other skeletal points of the body to obtain the normalized coordinates of each joint skeletal point of the body.

[0132]

[0133] Where b = 0, 1, 2, ..., 32;

[0134] The body bone point when b=0, that is, the normalized coordinates of the coordinate origin of the body bone point

[0135] S1.5.2.2. Construct the body standard position feature vector set d P ; The specific process is:

[0136] For the normalized coordinates of the bth body bone point Using the body position eigenvector calculation formula Get the body standard position feature vector of each body bone point

[0137] The body standard position feature vector to be calculated Excluding the body bone point coordinate origin That is, b≠0;

[0138] in Represents the vector of the body bone point coordinates in the positive x direction, which is numerically the same as the body bone point coordinates. The values ​​are equal;

[0139] in Represents the vector of the body bone point coordinates in the positive y direction, which is numerically the same as the body bone point coordinates. The values ​​are equal;

[0140] in Represents the vector of the body bone point coordinates in the positive z direction, which is numerically the same as the body bone point coordinates. The values ​​are equal;

[0141] Traverse all available b values ​​to obtain the body standard position feature vector of each body bone point Constitutes the body standard position feature vector set d P ;

[0142] The body standard position feature vector set d P Contains 32 body standard position feature vectors

[0143] S1.5.2.3. Calculate the velocity feature vector of each body bone point and construct the body standard velocity feature vector set v H The body standard velocity feature vector only exists in the body standard dynamic movement; the specific process is:

[0144] For the normalized coordinates of the bth body bone point For the standard dynamic body motion containing n frames, the velocity feature vector calculation formula is used Get the body standard velocity feature vector of the i-th frame (1≤i≤n) of the body standard dynamic motion of each body bone point

[0145] in

[0146] in

[0147] in

[0148] in

[0149] Traverse all available body standard dynamic movements, b values, i values, and obtain the body standard velocity feature vector of each body bone point for each body standard dynamic movement Constitutes the body standard velocity feature vector set v P ;

[0150] The body standard velocity feature vector set v P Contains 33n body standard velocity feature vectors for each standard dynamic movement of the body (n frames of image)

[0151] S1.5.2.4. Construct the standard body angle feature vector set θ P ; The specific process is:

[0152] Calculate the cosine of the angle between the body bone point and the surrounding bone points Will Recorded as the standard characteristic angle of the body bone point;

[0153] The standard characteristic angle of the body bone point to be calculated Specifically include:

[0154] The angle between the two arms is denoted as Where b = 13, 14;

[0155] The angles between the upper and lower arms and the horizontal direction are recorded as Where b = 13, 14, 15, 16;

[0156] The angle between the two arms The angles between the upper and lower arms and the horizontal direction Constitutes the standard characteristic angles of the body's skeletal points

[0157] When b=13, it represents the 13th body bone point - the left elbow body bone point;

[0158] When b=14, it represents the 14th body bone point - the right elbow body bone point;

[0159] When b=15, it represents the 15th body bone point - the left wrist body bone point;

[0160] When b=16, it represents the 16th body bone point - the right wrist body bone point;

[0161] The standard characteristic angle between the major and minor arms of the two arms The calculation formula is:

[0162]

[0163] Where b = 13, 14;

[0164] The upper and lower arms of the two arms are respectively at standard characteristic angles θ' with the horizontal direction. P ' b The calculation formula is:

[0165]

[0166] Where b = 13, 14, 15, 16;

[0167] Traversal and All the b values ​​that can be obtained are used to obtain the standard characteristic angles of the body bone points Constitutes the body standard angle feature vector set θ P ;

[0168] The body standard angle feature vector set θ P Contains standard feature angles of 6 body bone points

[0169] The cosine value of the angle between each body bone point and the adjacent bone point is calculated, and key nodes are selected to construct a body standard angle feature vector set containing 6 body angle feature vectors, as shown in the following table:

[0170]

[0171] S1.5.2.5. Construct the body standard distance feature vector set L P ; The specific process is:

[0172] Calculate the coordinate origin of each body bone point and the body bone point Euclidean distance It is recorded as the standard feature distance of the body skeleton points;

[0173] The standard feature distance of the hand skeleton point to be calculated The coordinate origin does not include the hand bone point That is, b≠0;

[0174] The standard feature distance of the hand skeleton points The calculation formula is:

[0175]

[0176] Where b≠0;

[0177] Traverse all b values ​​to obtain the standard feature distance of each body bone point Constitutes the hand standard distance feature vector set L P ;

[0178] The hand standard distance feature vector set L P Contains standard feature distances of 32 hand skeleton points for each hand

[0179] The Euclidean distance between each body bone point and the nose bone point is calculated, and key bone nodes are selected to construct a body standard distance feature vector set containing 6 distance feature vectors of the body, as shown in the following table:

[0180]

[0181] The body standard position feature vector set d P , body standard velocity feature vector set v P , the body standard angle feature vector set θ P , body standard distance feature vector set L P , together constitute the body action feature vector set P;

[0182] Constructing a standard static body motion template according to the body motion feature vector set P;

[0183] The standard static body movement template includes 6 standard body movements: left arm vertical, right arm vertical, arms crossed, arms akimbo, waving left arm, and arms rotating alternately;

[0184] The 10 standard hand movements included in the standard hand static movement template and the 6 standard body movements included in the standard body static movement template can be divided into standard static movements and standard dynamic movements;

[0185] The standard static movements include 10 standard static gestures and 4 standard static body movements;

[0186] The standard dynamic movements include 2 standard dynamic body movements;

[0187] The 10 standard static gestures are left hand 1, left hand 2, left hand 3, left hand 4, left hand 8, right hand 1, right hand 2, right hand 3, right hand 4, right hand 8;

[0188] The four standard static body movements are left arm vertical, right arm vertical, arms crossed, and arms on hips;

[0189] The two standard dynamic movements are swinging the left arm and rotating both arms alternately;

[0190] The other steps and parameters are the same as those in the first to fourth embodiments.

[0191] Specific embodiment 6: This embodiment differs from specific embodiments 1 to 5 in that:

[0192] The motion control instructions for the robot arm motion in S1.6 include: a static control instruction for the robot arm and a dynamic control instruction for the robot arm; the control instructions for constructing the robot arm motion; the specific process is:

[0193] S1.6.1. Determine 17 motion modes of the robotic arm; construct 17 motion control instructions for the robotic arm based on the 17 motion modes;

[0194] The 17 motion control instructions include 15 robot arm static motion control instructions and 2 robot arm dynamic motion control instructions; each motion mode corresponds to a control instruction;

[0195] The 15 types of robot arm static motion control instructions include 5 robot arm static single motion control instructions that are only executed once, 4 robot arm static single motion control instructions that can be executed continuously, and 6 robot arm static double motion control instructions;

[0196] S1.6.2. Based on the static standard motion template and the robot arm motion mode, construct five static single motion control instructions for the robot arm that are executed only once: robot arm position initialization, end gripper release, end gripper tightening, motion instruction initialization, and entering dynamic motion control mode;

[0197] The standard static action templates corresponding to the instructions are: left hand 4, left hand 8, right hand 8, arms crossed, arms on hips;

[0198] S1.6.3. Based on the static standard motion template and the robot arm motion mode, construct four continuously executable static single motion control instructions for the robot arm: clockwise fine adjustment of the machine base axis, counterclockwise fine adjustment of the machine base axis, clockwise rotation of the end axis, and counterclockwise rotation of the end axis.

[0199] The standard static action templates corresponding to the instructions are: right hand 1, right hand 2, right hand 3, right hand 4;

[0200] S1.6.4. Based on the static standard motion template and the robot arm motion mode, construct six static dual-motion control instructions for the robot arm: end point forward motion, end point backward motion, end point left motion, end point right motion, end point upward motion, and end point downward motion;

[0201] The instructions correspond to the following standard static action templates of the two-hand manual movements: left hand 1 + left arm vertical, left hand 1 + right arm vertical, left hand 2 + left arm vertical, left hand 2 + right arm vertical, left hand 3 + left arm vertical, left hand 3 + right arm vertical;

[0202] S1.6.5. Based on the dynamic standard motion template and the robot arm motion mode, construct two robot arm dynamic motion control instructions: clockwise coarse adjustment of the machine base axis and counterclockwise coarse adjustment of the machine base axis;

[0203] The standard dynamic template actions corresponding to the instructions are waving the left arm and rotating both arms alternately;

[0204] According to the characteristics of the action and the movement mode of the UR5 robot, the motion control instructions and their corresponding relationships are designed. The action instructions are divided into static action instructions and dynamic action instructions. Among them, the static action instructions are further divided into static single action instructions (only executed once), static single action instructions (can be executed continuously) and static double action instructions. According to the classification of action instruction types, based on the movement mode of the six-axis robot,

[0205] The corresponding relationship of static single action instructions (executed only once) is shown in the following table:

[0206]

[0207] The corresponding relationship of static single action instructions (can be executed continuously) is shown in the following table:

[0208]

[0209] The corresponding relationship of static double action instructions is shown in the following table:

[0210]

[0211] The corresponding relationship of dynamic action instructions is shown in the following table:

[0212]

[0213] Next, we will build a real-time human action recognition system.

[0214] First, a static action recognition module is designed. Here, an action classification method based on template matching is adopted, which directly takes the original data or its statistical features as input to classify static actions.

[0215] For static gestures, the flexion and extension relationship of the five fingers and the angle characteristics of the extension and bending of the finger joints are specified respectively. Through the analysis of the physiological structure of the joints of the human hand, it is known that each finger in these hand movements has only two states: "straight" and "bent". Therefore, it is only necessary to define the angle vectors of "straight" and "bent" for each finger. Since the left and right hands are mirror-symmetrical relative to the body, the flexion and extension relationship of the fingers in the same static gesture of both hands is the same. The flexion and extension relationship of the five fingers for each static gesture is shown in the following table:

[0216]

[0217] Note: + represents straight fingers, - represents bent fingers

[0218] The angle characteristics of finger flexion and extension for each static gesture are shown in the following table:

[0219]

[0220] For static body movements, the angle characteristics of key joints and the corresponding distance characteristics expressed in coordinates are specified. By analyzing the skeletal point characteristics of human body movements and the physiological structure of arm joints, it can be seen that in the real-time recognition of static body movements, it is necessary to focus on the coordinate relationship between these joint points, including the vertical or horizontal relationship between the forearm or upper arm and the ground. According to the characteristic value conditions of each action in the table, the angle characteristic value L is used to identify the distance between the two joints. 01 and the distance eigenvalue L 02 , you can determine whether the static body movement in the current picture belongs to the set movement type. The angle characteristics and distance characteristics of each static body movement are shown in the following table:

[0221]

[0222] Using template matching, static gestures can be detected in real time. If a static gesture cannot be detected, it is classified as "Unknown." Experimental testing has shown that this solution achieves a recognition accuracy of over 99%, meeting technical requirements. Other steps and parameters are the same as those in Specific Implementations 1 through 5.

[0223] Specific embodiment 7: This embodiment differs from specific embodiments 1 to 6 in that:

[0224] The manipulator human-machine collaborative motion control model based on machine vision constructed in S2 includes: a static action recognition module and a dynamic action recognition module;

[0225] The static action recognition module is a template matching algorithm;

[0226] Template matching algorithm,First, a template image is defined, which is a specific pattern or object to be found in the target image.,The template is usually small in size and corresponds to a similar object or pattern that may appear in the target image;

[0227] Using similarity metrics, determine a method for measuring the similarity between the template and the local area of ​​the target image. Common metrics include:

[0228] Sum of Squared Differences (SSD): Calculate the square of the difference between each pixel value and sum them;

[0229] Normalized Cross-Correlation (NCC): Calculates the correlation between local regions of two images, making the comparison more robust and flexible;

[0230] Mean Absolute Difference (MAD): Calculates the absolute difference between each pixel value and averages them;

[0231] During the matching process, the template image is slid or traversed through each possible position of the target image starting from the upper left corner of the target image; at each position, the similarity measure between the template and the local area of ​​the target image is calculated; the position with the minimum difference (for SSD and MAD) or the maximum correlation (for NCC) is recorded, that is, the best matching position;

[0232] The dynamic recognition module is an LSTM neural network algorithm;

[0233] The LSTM neural network includes: an input layer, a first hidden layer, a first hidden layer, a first hidden layer, a first hidden layer, a first hidden layer, and an output layer;

[0234] The first hidden layer, the second hidden layer, and the third hidden layer are LSTM layers;

[0235] The fourth hidden layer, the fifth hidden layer and the output layer are Dense layers;

[0236] Three advantages of LSTM networks:

[0237] (1) Low data requirements. In an LSTM neural network, only a few dozen sets of data are needed for each set of actions to accurately and efficiently determine the corresponding dynamic actions. Using less data can produce higher accuracy.

[0238] (2) Simple and efficient structure. The neural network has a compact structure and only has about one million parameters, which is far less than other neural networks with millions of parameters, making the model training faster.

[0239] (3) Fast action recognition speed. The neural network is a whole and has a unique advantage in action recognition, with a faster recognition speed.

[0240] At the same time, the position and velocity feature vectors of key skeleton points in each frame are used to fuse feature data to create a dataset. For two or more adjacent frames of real-time images, the probability of generating the dynamic action judgment result at the current moment and the action judgment result with the highest probability are determined in real time.

[0241] Use the LSTM model to train a self-built dynamic motion dataset, and then evaluate and optimize the model. First, divide the self-built dynamic motion dataset into a training set and a test set. In this training, 95% of the dataset is used for training and 5% for testing. When dividing the dataset, choose an appropriate data partition to ensure an even distribution of sample size and motion types between the training and test sets to avoid dataset bias.

[0242] When training a dataset, the dataset must first be preprocessed. Preprocessing includes two steps: data cleaning and data standardization.

[0243] Data cleaning removes noise and outliers from the dataset and detects missing values. Based on the number of missing values ​​and the characteristics of the data, appropriate methods are selected to fill or delete missing values, such as mean filling, median filling, or deleting rows or columns containing missing values.

[0244] Handling Outliers: Detecting and identifying outliers in the data. Determining whether outliers need to be removed or transformed (such as truncation or smoothing) based on business logic or statistical methods to improve the robustness of the model;

[0245] Normalization is used to scale numerical features to a standard range, usually [0, 1]. Common normalization methods include minimum-maximum scaling and Z-score normalization. Scaling the data in the dataset to the same range improves the training effect of the model.

[0246] According to the preprocessed data set, a human dynamic action recognition network model based on LSTM is built. The input layer is set with 15 inputs, each with 258 dimensions; the hidden layer is set with 5 layers, the first three layers are LSTM layers, with 64 dimensions, 128 dimensions and 64 dimensions respectively; the last two layers are Dense layers, with 64 dimensions and 32 dimensions respectively; the last layer is the output layer Dense, and the output result is 2 dimensions. The specific structure of LSTM is shown in the figure below. Figure 4 shown.

[0247] During the training process, the Accuracy and Loss curves of human dynamic action recognition based on the LSTM neural network model were calculated. Through experimental analysis, it was found that both the Accuracy curve and the Loss curve showed a trend, that is, as the number of iterations increases, the Accuracy curve gradually decreases, while the Loss curve shows a trend of first increasing and then decreasing. The Accuracy curve of the LSTM model is shown in Figure 2. Figure 5 As shown in 5a, the Loss curve of the LSTM model is as follows Figure 5 As shown in 5b. Figure 5 5a in Figure 5 As shown in 5b, when the number of iterations reaches 20, the accuracy reaches over 99% and converges, indicating that the LSTM neural network model is highly effective in real-time dynamic motion recognition. The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0248] Specific embodiment eight: This embodiment differs from specific embodiments one to seven in that:

[0249] In S3, the human-machine collaborative motion control model of the manipulator based on machine vision is trained according to the standard action training set to obtain a trained human-machine collaborative motion control model of the manipulator based on machine vision. The specific process is as follows:

[0250] S3.1. Train the static action recognition module according to the standard static action dataset to obtain a trained static action recognition module;

[0251] S3.2. Train the dynamic action recognition module according to the standard dynamic action dataset to obtain a trained dynamic action recognition module;

[0252] S3.3. Integrate the trained static action recognition module and dynamic action recognition module to train a human-machine collaborative motion control model for the robotic arm based on machine vision.

[0253] In S3.1, the static action recognition module is trained according to the standard static action data set to obtain a trained static action recognition module; the specific process is:

[0254] S3.1.1. Input the standard static action dataset into the static action recognition module, calculate the training loss based on the input and output of the static action recognition module, and stop training when the training loss is minimized to obtain a trained static action recognition module;

[0255] In S3.2, the dynamic action recognition module is trained according to the standard dynamic action data set to obtain a trained dynamic action recognition module; the specific process is as follows:

[0256] S3.2.1. Preprocessing the standard dynamic action dataset to obtain a preprocessed standard dynamic action dataset;

[0257] S3.2.2. Input the preprocessed standard dynamic action data set into the dynamic action recognition module, calculate the training loss based on the input and output of the dynamic action recognition module, and stop training when the training loss is minimized to obtain a trained dynamic action recognition module.

[0258] The other steps and parameters are the same as those in the first to seventh embodiments.

[0259] Specific embodiment 9: This embodiment differs from specific embodiments 1 to 8 in that:

[0260] In S3.2.1, the standard dynamic action dataset is preprocessed to obtain a preprocessed standard dynamic action dataset; the specific process is as follows:

[0261] S3.2.1.1. Perform data cleaning on the standard dynamic action dataset to obtain a cleaned standard dynamic action dataset;

[0262] S3.2.1.2. Perform data standardization on the cleaned standard dynamic action dataset to obtain the preprocessed standard dynamic action dataset

[0263] The other steps and parameters are the same as those in the first to eighth embodiments.

[0264] Specific embodiment 10: This embodiment differs from specific embodiments 1 to 9 in that:

[0265] The specific process of inputting the robot arm control instruction into the robot arm to realize the motion control of the robot arm is as follows:

[0266] Use a static IP address to establish a connection with the command issuing end to control the UR5 robotic arm;

[0267] The second step is to collect real-time action images of the operator; the specific process is:

[0268] 21. Use Kinect camera to capture the operator's real-time action video stream;

[0269] 2. Use the MediaPipe pose estimation algorithm to process the image to obtain skeleton point data as the operator's real-time motion data;

[0270] Integrate static action recognition module and dynamic action recognition module to build a real-time human action recognition system. The process is as follows: Figure 6 As shown, the real-time recognition function of the action is realized.

[0271] Step 10. Use the Socket communication method in the TCP / IP protocol to build a motion control model for the UR5 robotic arm. Use a static IP address to establish a connection with the command issuer to control the UR5 robotic arm, thereby completing the design of the robotic arm control system.

[0272] Set the IP of the UR5 robot arm to 192.168.74.128, use a static address to establish a connection with the instruction issuer, set the IP of the instruction issuer to 192.168.74.1, and ensure that the robot arm and the instruction issuer are in the same local area network. Use Python3.8.0 to write a control program based on the Tensorflow framework at the instruction issuer, use the Socket interface to implement TCP / IP communication, and send the motion control instructions of the robot arm to the IP address and port 30003 of the UR5 robot arm. After receiving the motion control instructions sent by the instruction issuer, the robot arm uses the URScript scripting language to write a program, converts it into a motion control instruction that can be parsed, and performs the corresponding motion control operations. Finally, during the process of the robot arm executing the movement, the real-time motion status of the robot arm will be transmitted to the IP address and port of the instruction issuer through Socket programming to implement TCP / IP communication. The motion control process of the communication between the UR5 robot arm and the instruction issuer is as follows Figure 7 As shown in the figure, the control of the UR5 robotic arm is realized, and the design of the robotic arm control system is completed.

[0273] The real-time human motion recognition system and the robotic arm control system are integrated to build a real-time motion control system for the robotic arm based on human motion recognition technology. The structural block diagram of the motion control system for industrial robotic arm visual recognition is as follows: Figure 8 As shown in the figure, the entire system consists of two parts. The machine vision action recognition system consists of four parts: the visualization interface, image acquisition module, image processing module, and communication module; the robotic arm control system consists of three parts: the communication port, the robotic arm control system, and the UR5 robotic arm body.

[0274] When the program runs, it initializes the camera and communication modules and loads workpiece parameters. If any module fails to load, the system prompts the user, allowing the operator to easily identify the problem. If loading is successful, images are continuously captured and displayed on the main interface. When the Simulation Run button is clicked, the motion recognition module determines the current motion type and sends the recognized motion results via TCP / IP communication to the UR5 robotic arm, which then parses the data packet to achieve motion.

[0275] After obtaining the coordinates of the skeletal points of the hand and body movements in each frame, the motion control code of the UR5 robot arm is written in Python. This program is used to recognize the human body movements in the picture in real time, and the recognition results are converted into motion control instructions for the UR5, so that the robot arm moves according to the corresponding instructions. The flowchart of the entire system is shown below. Figure 1 The flowchart shows four main determination procedures: whether the camera is detected, whether it is a single action command, whether it is in dynamic action detection mode, and whether the detection has ended. The core of the entire program is to determine the real-time action type, generate the corresponding motion control code for the UR5 robot arm, and transmit the control command signal to the robot arm to make it move.

[0276] For static motion instructions, when the duration of the recognized instruction exceeds 1s, the corresponding control instruction is sent to the UR5 robot arm. If it is a single-action instruction that is only executed once, when the UR5 movement ends, the command ends and prepares to read the next action instruction; if it is a single-action instruction that can be executed continuously, after the instruction duration exceeds 1s, the UR5 starts to move, and checks every 0.5s to see if the action is maintained. If it is still maintained, it continues to move until the action disappears. If it is a double-action instruction, when the first action instruction is maintained for 1s, it will prompt that the second action can be indicated. When the second action instruction is maintained for 1s, the UR5 starts to move and does not stop until the second action disappears. For action instructions that can be executed continuously, if the action is interrupted and then restored to the original action, the UR5 movement can still be controlled.

[0277] For dynamic motion commands, the system can only begin recognizing dynamic motion after entering "Dynamic Motion Mode." This requires 15 consecutive frames of image recognition. If a dynamic motion is detected, the corresponding UR5 control command is sent, initiating movement. Test results show that the robotic arm can complete various motion recognition tasks and movement processes on-site.

[0278] After building the entire motion recognition-based robotic arm motion control system, although it could effectively recognize human motion and control the UR5 robotic arm's movements, the motion recognition results and the UR5 robotic arm's real-time motion status could not be monitored in real time, and the two results could not be displayed simultaneously in the same environment. Therefore, using the Python+Tkinter environment configuration, a user interface (UI) for the system was designed. This displays the motion recognition results and the robotic arm's real-time motion status on the same interface, making it easier for operators to observe the movements and the robotic arm's motion status.

[0279] Design interface such as Figure 9As shown in the figure, the interface clearly and intuitively displays real-time human motion images processed by MediaPipe, the real-time motion images of the robotic arm, the coordinate and rotation information of the robotic arm's end point, the real-time rotation angles of the six axes, real-time motion detection information for the left and right hands, and a table comparing motion semantics and control commands. Finally, field verification demonstrated the accurate and efficient display of video capture images, motion recognition results, and motion control commands, achieving integrated and convenient operation and observation. The remaining steps and parameters are the same as those in Specific Implementations 1 to 9.

[0280] The above only describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the above-mentioned specific implementation methods. Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. Any technician familiar with this profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical content disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent replacements and improvements made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A method for controlling human-machine collaborative motion of a robotic arm based on machine vision, characterized in that: The following steps are involved:

1. Construct a human-machine collaborative motion control model for a robotic arm based on machine vision; train and obtain a trained human-machine collaborative motion control model for a robotic arm based on machine vision; the specific process is as follows: S1: Collect videos of standard human movements as sample data of standard human movements, construct a standard human movement template based on the sample data of standard human movements, and then construct motion control instructions for the robotic arm based on the standard human movement template; Standard action templates include: hand standard action templates and body standard action templates; Construct a standard hand motion template based on the human hand skeleton point data; The standard hand motion templates include a standard static hand motion template and a standard dynamic hand motion template; Hand standard position feature vector set d H , hand standard velocity feature vector set v H , the hand standard angle feature vector set θ H , hand standard distance feature vector set L H , together constitute the hand action feature vector set H; Constructing a standard static hand motion template based on the hand motion feature vector set H; The standard static hand motion template includes 10 standard hand motions; Construct a standard body motion template based on body bone point data; The body standard movement template includes a body standard static movement template and a body standard dynamic movement template; Body standard position feature vector set d P , body standard velocity feature vector set v P , the body standard angle feature vector set θ P , body standard distance feature vector set L P , together constitute the body action feature vector set P; Constructing a standard static body motion template according to the body motion feature vector set P; Calculate the cosine of the angle between the body bone point and the surrounding bone points Will Recorded as the standard characteristic angles of body bone points; specifically including: The angle between the two arms is recorded as Where b = 13, 14; The angles between the upper and lower arms and the horizontal direction are recorded as Where b = 13, 14, 15, 16; The angle and angle Constitutes the standard characteristic angles of the body's skeletal points When b=13, it indicates the 13th body bone point - the left elbow body bone point; When b=14, it indicates the 14th body bone point - the right elbow body bone point; When b=15, it indicates the 15th body bone point - the left wrist body bone point; When b=16, it indicates the 16th body bone point - the right wrist body bone point; The standard characteristic angle between the major and minor arms of the two arms The calculation formula is: Where b = 13, 14; The upper and lower arms of the two arms are respectively at standard characteristic angles with the horizontal direction. The calculation formula is: Where b = 13, 14, 15, 16; The standard static body movement template includes 6 standard body movements: left arm vertical, right arm vertical, arms crossed, arms akimbo, waving left arm, and arms rotating alternately; The standard hand movements included in the standard hand static movement template and the standard body movements included in the standard body static movement template can be divided into standard static movements and standard dynamic movements; S2: Construct a human-machine collaborative motion control model for a robotic arm based on machine vision; S3: Using the human standard motion template as the standard motion training set, the machine vision-based human-machine collaborative motion control model of the robotic arm is trained according to the standard motion training set to obtain a trained machine vision-based human-machine collaborative motion control model of the robotic arm; 2. Collect real-time action images of operators; 3. Input the collected real-time action images of the operator into the trained machine vision-based human-machine collaborative motion control model of the robot arm to judge the real-time action of the operator and output the corresponding robot arm control instructions according to the action; 4. Input the robot arm control instructions into the robot arm to realize the motion control of the robot arm.

2. The method for controlling human-machine collaborative motion of a robot arm based on machine vision according to claim 1, characterized in that: In the step S1, a video of a human body's standard motion is collected as sample data of the human body's standard motion, a human body's standard motion template is constructed based on the sample data of the human body's standard motion, and then a motion control instruction of the robotic arm is constructed based on the human body's standard motion template. The specific process is as follows: S1.

1. Collect videos of standard human movements as sample data of standard human movements; S1.

2. Construct the original dataset based on the sample data of standard human movements; S1.

3. Preprocess the original data set to obtain the preprocessed images of standard actions; S1.4, processing the pre-processed image obtained in S1.3 to obtain human skeleton point data; S1.

5. Construct a human standard motion template based on the human skeleton point data obtained in S1.4; S1.

6. Based on the standard motion template obtained in S1.5, construct motion control instructions for the robot arm's motion.

3. The method for controlling human-machine collaborative motion of a robot arm based on machine vision according to claim 2, characterized in that: The sample data of standard human movements in S1.1 are collected. The specific process is: using a Kinect industrial-grade 2k depth camera to collect standard static action images and standard dynamic action videos of human movements as sample data of standard human movements; In S1.2, an original data set is constructed based on the sample data of standard human movements; the original data set includes a standard static movement original data set and a standard dynamic movement original data set; the specific process is as follows: S1.2.

1. Use the standard static action images collected in S1.1 as the original data set of standard static actions; The standard static action images include: standard static gesture images and standard static body action images; S1.2.

2. Use the standard dynamic action video collected in S1.1 as the original data set of standard dynamic action; In S1.3, the original data set is preprocessed to obtain the standard action preprocessed image; the specific process is: S1.3.

1. Extract the standard static action images from the original dataset of standard static actions described in S1.2.1; name them as preprocessed standard static action images; S1.3.

2. Extracting a preprocessed standard dynamic action image set based on the video data of the standard action in the original data set of the standard dynamic action described in S1.2.2; The extraction process of each standard dynamic action in the preprocessed standard dynamic action image set is as follows: Set the start time and end time of the standard dynamic action; extract each frame of the original image of the collected standard dynamic action video data at a frame rate of FPS=30 as the pre-processed standard dynamic action image; S1.3.

3. The preprocessed standard static action image and the preprocessed standard dynamic action image set are used as standard action preprocessed images; S1.4 processes the standard action pre-processed image obtained in S1.3 to obtain human skeleton point data; the specific process is: Use the Pose and Hands models of the MediaPipe pose estimation algorithm to process the pre-processed images of standard actions to obtain skeleton point data; The skeleton point data includes: human hand skeleton point data and body skeleton point data; The human body's hand skeleton point data is the human body's two-hand skeleton point data; The human body's two-hand skeleton point data includes information about the operator's left and right hands; each hand's information includes 21 hand skeleton point information; The body skeleton point data is information of 33 body skeleton points; The human body skeleton point data and hand skeleton point data include the 3D coordinate point (x, y, z) of each hand skeleton point and body skeleton point in each image and the confidence information of the coordinate point; The confidence level of the coordinate point represents the MediaPipe pose estimation algorithm's estimate of the probability of the accuracy of the coordinate point.

4. The method for controlling human-machine collaborative motion of a robot arm based on machine vision according to claim 3, characterized in that: The specific process of constructing the standard hand movement template is as follows: Determine the normalized coordinates of the hand skeleton points of each hand of the operator; the specific process is: According to the human body's two-hand skeleton point data in S1.4, the skeleton point at the wrist of the k-th hand is taken as the coordinate origin of the skeleton point of the k-th hand; Where k = 1, 2; k = 1 means left hand, k = 2 means right hand; Calculate the normalized coordinate value of the a-th hand skeleton point of the k-th hand; the specific process is: For the k-th hand, the coordinate value of the a-th hand bone point is subtracted from the coordinate value of the coordinate origin of the hand bone point to obtain the normalized coordinate value of each hand bone point. Where a=0,1,2,…,20; The hand skeleton point when a=0, that is, the normalized coordinates of the coordinate origin of the hand skeleton point Construct the hand standard position feature vector set d H ; The specific process is: For the k-th hand, the normalized coordinates of the a-th hand skeleton point Using the hand position feature vector calculation formula Get the standard hand position feature vector of each hand bone point The standard hand position feature vector to be calculated Does not include the hand bone point coordinate origin That is, a≠0; in Represents the vector of the hand bone point coordinates in the positive x direction, which is numerically the same as the hand bone point coordinates. The values ​​are equal; in Represents the vector of the hand bone point coordinates in the positive y direction, which is numerically the same as the hand bone point coordinates. The values ​​are equal; in Represents the vector of the hand bone point coordinates in the positive z direction, which is numerically the same as the hand bone point coordinates. The values ​​are equal; Traverse all available k values ​​and a values ​​to obtain the standard hand position feature vector of each hand bone point of both hands Construct the hand standard position feature vector set d H ; The hand standard position feature vector set d H Contains 20 standard hand position feature vectors for each hand Calculate the velocity feature vector of each hand bone point and construct the standard hand velocity feature vector set v H The hand standard velocity feature vector only exists in the hand standard dynamic movement; the specific process is: For the k-th hand, the normalized coordinates of the a-th hand skeleton point For the standard dynamic hand movements containing n frames, the velocity feature vector calculation formula is used Get the hand standard velocity feature vector of the i-th frame of the hand standard dynamic motion of each hand bone point in in in in Traverse all available hand standard dynamic movements, k values, a values, and i values ​​to obtain the hand standard velocity feature vector of each hand bone point of each hand standard dynamic movement Constitutes the hand standard velocity feature vector set v H ; The hand standard velocity feature vector set v H Contains 21n standard hand velocity feature vectors for each hand for each standard dynamic movement Construct the hand standard angle feature vector set θ H ; The specific process is: Calculate the cosine value of the angle between the hand bone point and the surrounding bone points Will Recorded as the standard characteristic angle of the hand skeleton point; The standard feature angle of the hand bone point to be calculated Does not include the hand bone point coordinate origin The bone point at the top of the finger, that is, a≠4m; where m=0,1,2,3,4,5; The angle between the hand skeleton point and the surrounding skeleton points is calculated when calculating the hand skeleton point with a=4m+1, and the coordinate origin of the hand is calculated. The standard characteristic angle of the hand bone point to be calculated between the two hand bone points a+1 The angle of the surrounding skeleton points of the hand skeleton point, when calculating the hand skeleton points a=4m+2 and a=4m+3, calculate the standard characteristic angle of the hand skeleton point formed by the same hand skeleton point and the two hand skeleton points a-1 and a+1 of the same hand The standard characteristic angle of the hand skeleton point The calculation formula is: in Traverse all available k values ​​and a values ​​to obtain the standard feature angle of each hand bone point of both hands Constitutes the hand standard angle feature vector set θ H ; The hand standard angle feature vector set θ H Contains standard feature angles of 15 hand bone points for each hand Construct the hand standard distance feature vector set L H ; The specific process is: Calculate the coordinate origin of each hand bone point and the same hand Euclidean distance Note it as the standard feature distance of the hand skeleton points; The standard feature distance of the hand skeleton points to be calculated The coordinate origin does not include the hand bone point That is, a≠0; The standard feature distance of the hand skeleton points The calculation formula is: Where a≠0; Traverse all available k values ​​and a values ​​to obtain the standard feature distance of each hand bone point of both hands Construct the hand standard distance feature vector set L H ; The hand standard distance feature vector set L H Contains standard feature distances of 20 hand skeleton points for each hand The 10 standard hand movements include: left hand 1, left hand 2, left hand 3, left hand 4, left hand 8, right hand 1, right hand 2, right hand 3, right hand 4, right hand 8; The body standard movement template is constructed based on the body skeleton point data; the specific process is: The coordinate data of the bone point at the nose is used as the coordinate origin of the body; the coordinate values ​​of the origin of the body bone point are subtracted from the coordinate values ​​of the other bone points of the body to obtain the normalized coordinates of each joint bone point of the body Where b = 0, 1, 2, ..., 32; The body bone point when b=0, that is, the normalized coordinates of the coordinate origin of the body bone point Construct the body standard position feature vector set d P ; The specific process is: For the normalized coordinates of the bth body bone point Using the body position eigenvector calculation formula Get the body standard position feature vector of each body bone point The body standard position feature vector to be calculated Excluding the body bone point coordinate origin That is, b≠0; in Represents the vector of the body bone point coordinates in the positive x direction, which is numerically the same as the body bone point coordinates. The values ​​are equal; in Represents the vector of the body bone point coordinates in the positive y direction, which is numerically the same as the body bone point coordinates. The values ​​are equal; in Represents the vector of the body bone point coordinates in the positive z direction, which is numerically the same as the body bone point coordinates. The values ​​are equal; Traverse all available b values ​​to obtain the body standard position feature vector of each body bone point Constitutes the body standard position feature vector set d P ; The body standard position feature vector set d P Contains 32 body standard position feature vectors Calculate the velocity feature vector of each body bone point and construct the body standard velocity feature vector set v H The body standard velocity feature vector only exists in the body standard dynamic movement; the specific process is: For the normalized coordinates of the bth body bone point For the standard dynamic body motion containing n frames, the velocity feature vector calculation formula is used Get the body standard velocity feature vector of the i-th frame of the body standard dynamic motion for each body bone point in in in in Traverse all available body standard dynamic movements, b values, i values, and obtain the body standard velocity feature vector of each body bone point for each body standard dynamic movement Constitutes the body standard velocity feature vector set v P ; The body standard velocity feature vector set v P Contains 33n body standard velocity feature vectors for each standard dynamic movement of the body Construct the body standard distance feature vector set L P ; The specific process is: Calculate the coordinate origin of each body bone point and the body bone point Euclidean distance It is recorded as the standard feature distance of the body skeleton points; The standard feature distance of the hand skeleton points to be calculated The coordinate origin does not include the hand bone point That is, b≠0; The standard feature distance of the hand skeleton points The calculation formula is: Where b≠0; Traverse all b values ​​to obtain the standard feature distance of each body bone point Constitutes the body standard distance feature vector set L P ; The body standard distance feature vector set L P Contains the standard feature distance of 32 body bone points for each body The standard static movements include 10 standard static gestures and 4 standard static body movements; The standard dynamic movements include 2 standard dynamic body movements; The 10 standard static gestures are left hand 1, left hand 2, left hand 3, left hand 4, left hand 8, right hand 1, right hand 2, right hand 3, right hand 4, right hand 8; The four standard static body movements are left arm vertical, right arm vertical, arms crossed, and arms on hips; The two standard dynamic body movements are swinging the left arm and rotating both arms alternately.

5. The method for controlling human-machine collaborative motion of a robot arm based on machine vision according to claim 4, characterized in that: The motion control instructions for the robot arm motion in S1.6 include: a static control instruction for the robot arm and a dynamic control instruction for the robot arm; the control instructions for constructing the robot arm motion; the specific process is: S1.6.

1. Determine 17 motion modes of the robotic arm; construct 17 motion control instructions for the robotic arm based on the 17 motion modes; The 17 types of robot arm motion control instructions include 15 types of robot arm static motion control instructions and 2 types of robot arm dynamic motion control instructions; each motion mode corresponds to a control instruction; The 15 types of robot arm static motion control instructions include 5 robot arm static single motion control instructions that are only executed once, 4 robot arm static single motion control instructions that can be executed continuously, and 6 robot arm static double motion control instructions; S1.6.

2. Based on the static standard motion template and the robot arm motion mode, construct five static single motion control instructions for the robot arm that are executed only once: robot arm position initialization, end gripper release, end gripper tightening, motion instruction initialization, and entering dynamic motion control mode; The standard static action templates corresponding to the static single action control instructions of the manipulator that are executed only once are: left hand 4, left hand 8, right hand 8, arms crossed, arms on hips; S1.6.

3. Based on the static standard motion template and the robot arm motion mode, construct four continuously executable static single motion control instructions for the robot arm: clockwise fine adjustment of the machine base axis, counterclockwise fine adjustment of the machine base axis, clockwise rotation of the end axis, and counterclockwise rotation of the end axis. The actions of the standard static action templates corresponding to the continuously executable static single action control instructions of the manipulator are: right hand 1, right hand 2, right hand 3, right hand 4; S1.6.

4. Based on the static standard motion template and the robot arm motion mode, construct six static dual-motion control instructions for the robot arm: end point forward motion, end point backward motion, end point left motion, end point right motion, end point upward motion, and end point downward motion; The static double-action control instructions of the manipulator correspond to the double-hand actions of the standard static action template respectively: left hand 1 + left arm vertical, left hand 1 + right arm vertical, left hand 2 + left arm vertical, left hand 2 + right arm vertical, left hand 3 + left arm vertical, left hand 3 + right arm vertical; S1.6.

5. Based on the dynamic standard motion template and the robot arm motion mode, construct two robot arm dynamic motion control instructions: clockwise coarse adjustment of the machine base axis and counterclockwise coarse adjustment of the machine base axis; The standard dynamic template actions corresponding to the dynamic action control instructions of the robotic arm are waving the left arm and rotating both arms alternately.

6. The method for controlling human-machine collaborative motion of a robot arm based on machine vision according to claim 5, characterized in that: The manipulator human-machine collaborative motion control model based on machine vision constructed in S2 includes: a static action recognition module and a dynamic action recognition module; The static action recognition module is a template matching algorithm; the dynamic recognition module is an LSTM neural network algorithm; The LSTM neural network algorithm includes: an input layer, a first hidden layer, a second hidden layer, a third hidden layer, a fourth hidden layer, a fifth hidden layer and an output layer; The first hidden layer, the second hidden layer, and the third hidden layer are LSTM layers; The fourth hidden layer, the fifth hidden layer and the output layer are Dense layers.

7. The method for controlling human-machine collaborative motion of a robot arm based on machine vision according to claim 6, characterized in that: The standard action training set in S3 includes a standard static action data set and a standard dynamic action data set; The human-machine collaborative motion control model of the robot arm based on machine vision is trained according to the standard action training set to obtain a trained human-machine collaborative motion control model of the robot arm based on machine vision. The specific process is as follows: S3.

1. Train the static action recognition module according to the standard static action dataset to obtain a trained static action recognition module; S3.

2. Train the dynamic action recognition module according to the standard dynamic action dataset to obtain a trained dynamic action recognition module; S3.

3. Integrate the trained static action recognition module and dynamic action recognition module to obtain a trained machine vision-based human-machine collaborative motion control model for the robotic arm; In S3.1, the static action recognition module is trained according to the standard static action data set to obtain a trained static action recognition module; the specific process is: S3.1.

1. Input the standard static action dataset into the static action recognition module, calculate the training loss based on the input and output of the static action recognition module, and stop training when the training loss is minimized to obtain a trained static action recognition module; In S3.2, the dynamic action recognition module is trained according to the standard dynamic action data set to obtain a trained dynamic action recognition module; the specific process is as follows: S3.2.

1. Preprocessing the standard dynamic action dataset to obtain a preprocessed standard dynamic action dataset; S3.2.

2. Input the preprocessed standard dynamic action dataset into the dynamic action recognition module, calculate the training loss based on the input and output of the dynamic action recognition module, stop training when the training loss is minimized, and obtain a trained dynamic action recognition module.

8. The method for controlling human-machine collaborative motion of a robot arm based on machine vision according to claim 7, characterized in that: In S3.2.1, the standard dynamic action dataset is preprocessed to obtain a preprocessed standard dynamic action dataset; the specific process is as follows: S3.2.1.

1. Perform data cleaning on the standard dynamic action dataset to obtain a cleaned standard dynamic action dataset; S3.2.1.

2. Perform data standardization on the cleaned standard dynamic motion dataset to obtain a preprocessed standard dynamic motion dataset.

9. The method for controlling human-machine collaborative motion of a robot arm based on machine vision according to claim 1, characterized in that: The specific process of inputting the robot arm control instruction into the robot arm to realize the motion control of the robot arm is as follows: Use a static IP address to establish a connection with the command issuing end to control the UR5 robotic arm; The second step is to collect real-time action images of the operator; the specific process is:

21. Use Kinect camera to collect real-time action video stream of operators; 2. Use the MediaPipe pose estimation algorithm to process the image to obtain skeleton point data as the operator's real-time motion data.

Images