A robot arm posture following control method and device
Through the positioning neural network and arm posture generation network, the color image of the human arm is used to identify joint angles, which solves the problem of inflexible movement of the robot arm when it appears at a singular point, and realizes the synchronous movement of the robot arm and the human arm, with high accuracy and versatility, suitable for a variety of robotic arm designs.
Patent Information
- Application Number
- CN202410253115.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-06
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-03-06
AI Technical Summary
The existing robotic arm control technology loses freedom when the singular point occurs, resulting in inflexibility of movement. The traditional method requires complex equipment and multimodal data processing, which lacks versatility.
Using the robotic arm posture following control method based on positioning neural network and arm posture generation network, the joint angle value is recognized in real time through the color image of the human arm, so as to realize the synchronous movement of the robotic arm and the human arm, avoid singular points, and simplify Equipment requirements.
It realizes the accurate and flexible movement of the robotic arm without singular points. It is suitable for all kinds of robotic arms, reducing system costs and calculation time, and has strong versatility.
Smart Images

Figure CN117921677B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of robot control technology, and in particular to a robot arm posture following control method and device. Background Art
[0002] A singularity is a specific point in the workspace of a robot that causes it to lose one or more degrees of freedom. The degrees of freedom of a robot are the number of independently controllable joints it has. For example, a 6-DOF robot has 6 independently movable joints. When a 6-DOF robot enters a singularity, one or more of its joints become useless, turning it into a robot with less than or equal to 5 degrees of freedom. The robot moves or stops moving in unexpected ways. Singularities are caused by the difference between the speed of motion control and the speed of real physics, that is, the speed of motion control is "infinite" and the speed in real physics is finite. The difference between the two will cause a lot of contradictions when programming the robot, which is why singularities in the robot are generated.
[0003] Early methods used image recognition technology to identify the position of a person's palm and arm, and based on the mapping rules between the arm and the robot, generated robot control instructions to guide the movement of the robot. Multimodal data including color images and depth images are also used to find the position of the human arm. The three-dimensional coordinates of the human arm joints are obtained and mapped to the three-dimensional coordinates of the robot arm joints to solve the joint values of the robot arm and move. Traditional image recognition-based methods require multimodal data, including color images, depth images, and three-dimensional point clouds, and require a lot of time to pre-process the multimodal data.
[0004] In addition, motion capture systems are also commonly used for remote control. The motion capture units are fixed to the waist, upper arm, lower arm and hand of the person, and communicate with the computer via Bluetooth to send instructions to the robot arm and move. However, the equipment of traditional motion capture systems is complex and expensive, and needs to be fixed to various parts of the human body.
[0005] Another common technology is trajectory generation. Through manual teaching to learn the trajectory of the robot arm, the regression curve obtained by the Gaussian model is used to find the mapping between the end of the human arm and the end of the robot arm, so that the robot arm can achieve track contour exploration; calculate the joint angle of the human arm, calculate the preset operation path according to the polynomial path planning method, and calculate the current joint angle of the robot arm according to the formula. However, in the traditional trajectory generation method, the robot arm can only move along a fixed trajectory and the movement is not flexible.
[0006] There is also a mapping system for humanoid robot arms, which uses the robot wrist joint to search for the robot elbow and shoulder joints through genetic algorithms, and uses the human wrist joint angle to adjust the robot wrist joint angle, so that the robot arm performs the terminal action. Traditional humanoid robot arm teleoperation requires that all robot arm joints are consistent with human arm joints, but this does not conform to all current robot arm designs and is not universal. Summary of the invention
[0007] The purpose of an embodiment of the present invention is to provide a method and device for controlling the posture of a robotic arm. Only by using a color image of the human arm, the robotic arm can follow the synchronous movement of the human arm in real time without any singularity points. It has the advantages of accurate, flexible and strong versatility in following, and is suitable for arm control of various types of robotic arms.
[0008] In order to solve the above technical problems, a first aspect of an embodiment of the present invention provides a robot arm posture following control method, comprising the following steps:
[0009] Acquire a real-time posture color image of a human body, and identify human arms and joints in the real-time posture color image based on a positioning neural network;
[0010] Based on the arm posture generation network, the 6D posture of the human arm is obtained, and the 6D posture is converted into the joint angle value of the human arm;
[0011] Based on the mapping relationship between the human arm and the robotic arm joints, the joint angle values of the human arm are assigned to several joints of the robotic arm to obtain the joint angle values of the robotic arm following the movement of the human arm, and the robotic arm is controlled to follow the synchronous movement of the human arm in real time.
[0012] Furthermore, the arm posture generation network is a posture generation algorithm model that includes an attention mechanism network and a joint loss function.
[0013] Furthermore, the posture generation algorithm model includes a reference neural network, an attention mechanism network and a posture matching loss function;
[0014] When there is no occlusion in the human body joints in the real-time posture color image, the real-time posture color image is input into the reference neural network, and the unoccluded human body posture is generated through the ResNet module;
[0015] When a human joint in the real-time posture color image is partially blocked, the real-time posture color image is input into the attention mechanism network, and the attention mechanism network assigns different weight values to each joint of the human body to generate a human posture, wherein the human joint with a higher correlation with the blocked joint has a larger weight value;
[0016] The posture matching loss function aligns the human body posture output by the reference neural network and the attention mechanism network.
[0017] Furthermore, the attention mechanism network input includes: a plurality of attention mechanism encoder layers connected in sequence, generating features layer by layer to obtain the human body posture.
[0018] Furthermore, the calculation formula of the joint loss function L is:
[0019] L=L_{P_i}+L_{J_i}+L_{S_i}+L_{M};
[0020] Among them, L_{P_i} is the posture loss of all joints of the human body, L_{J_i} is the position loss of all joints of the human body, L_{S_i} is the shape loss of all joints of the human body, i=1 is the reference neural network, i=2 is the attention mechanism network, P represents posture, J represents joint position, S represents human body shape, and each loss item is calculated by the mean absolute error.
[0021] Further, the arm posture generation network also includes: an SMPL human skeleton skin model;
[0022] The SMPL human skeleton skinning model generates skeleton joint position coordinates through posture, and the error between the skeleton joint position coordinates and the true label of the human joint position is used as part of the joint loss.
[0023] Further, the mapping relationship between the human arm and the mechanical arm joints includes: the shoulder joint of the human arm corresponds to a plurality of joints near the base of the mechanical arm, the elbow joint corresponds to a plurality of joints in the middle of the mechanical arm, and the wrist joint corresponds to a plurality of joints near the end of the mechanical arm;
[0024] The joints of the robotic arm corresponding to the shoulder joint, the elbow joint and the wrist joint are all different.
[0025] Furthermore, assigning the joint angle values of the human arm to several joints of the robotic arm includes:
[0026] Convert the 3D positions and 3D coordinates of the shoulder joint, elbow joint, and wrist joint of the human arm into joint angle values of the corresponding joints;
[0027] Based on the mapping relationship between the human arm and the robotic arm joints, the joint angle values of the human arm are assigned to the joints of the robotic arm.
[0028] Correspondingly, the second aspect of the embodiment of the present invention further provides a robot arm posture following control device, which controls the robot arm to follow the real-time synchronous movement of the human arm based on the above-mentioned robot arm posture following control method, including:
[0029] An image recognition module, which is used to obtain a real-time color image of a human body and recognize the arms and joints of the human body in the real-time color image of the human body based on a positioning neural network;
[0030] A posture generation module, which is used to obtain the 6D posture of the human arm based on the arm posture generation network, and convert it into the joint angle value of the human arm;
[0031] The joint mapping module is used to assign the joint angle values of the human arm to several joints of the robotic arm based on the mapping relationship between the human arm and the robotic arm joints, obtain the various joint angle values of the robotic arm following the movement of the human arm, and control the robotic arm to follow the synchronous movement of the human arm in real time.
[0032] Correspondingly, the third aspect of an embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the above-mentioned robotic arm posture following control method.
[0033] Correspondingly, the fourth aspect of the embodiments of the present invention further provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by a processor, the above-mentioned robot arm posture following control method is implemented.
[0034] The above technical solution of the embodiment of the present invention has the following beneficial technical effects:
[0035] Only by using the color image of the human arm, the robotic arm can follow the synchronous movement of the human arm in real time without any singularity. It has the advantages of accurate following, flexibility and strong versatility, and is suitable for arm control of various types of robotic arms. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 is a flow chart of a robot arm posture following control method provided by an embodiment of the present invention;
[0037] Figure 2 It is a schematic diagram of a robot arm posture following control algorithm provided by an embodiment of the present invention;
[0038] Figure 3 is a schematic diagram of a posture generation algorithm model provided by an embodiment of the present invention;
[0039] Figure 4is a schematic diagram of an arm posture generation network provided by an embodiment of the present invention;
[0040] Figure 5 It is a schematic diagram of the mapping relationship between the human arm and the mechanical arm joints provided by an embodiment of the present invention;
[0041] Figure 6 is a corresponding schematic diagram of a human arm coordinate system and a robotic arm base coordinate system provided by an embodiment of the present invention;
[0042] Figure 7a The present invention provides a schematic diagram of the rotation of the mechanical arm 0 joint. Figure 1 ;
[0043] Figure 7b The mechanical arm 1 provided in the embodiment of the present invention is a schematic diagram of the joint movement and rotation of the mechanical arm 1 Figure 2 ;
[0044] Figure 7c The embodiment of the present invention provides a schematic diagram of the rotation of the joint of the robot arm 2. Figure 3 ;
[0045] Figure 8 Schematic diagram of the range of motion of the elbow joint of a human arm around the Y axis and the Z axis respectively provided by an embodiment of the present invention;
[0046] Fig. 9 is a schematic diagram of the rotation of the elbow joint of a human arm around the Y axis provided by an embodiment of the present invention;
[0047] Fig.10 is a schematic diagram of the rotation of the elbow joint of a human arm around the Z axis provided by an embodiment of the present invention;
[0048] Fig.11 It is a module block diagram of a robot arm posture following control device provided by an embodiment of the present invention;
[0049] Fig.12 It is a block diagram of a joint mapping module provided by an embodiment of the present invention.
[0050] Reference numerals:
[0051] 1. Image recognition module, 2. Posture generation module, 3. Joint mapping module, 31. Posture conversion unit, 32. Joint amplitude unit. DETAILED DESCRIPTION
[0052] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.
[0053] Please refer to Figure 1 and Figure 2 A first aspect of an embodiment of the present invention provides a robot arm posture following control method, comprising the following steps:
[0054] Step S100, obtaining a real-time posture color image of a human body, and identifying human arms and joints in the real-time posture color image based on a positioning neural network.
[0055] Step S200, generating a network based on the arm posture, obtaining the 6D posture of the human arm, and converting it into the joint angle value of the human arm.
[0056] Step S300, based on the mapping relationship between the human arm and the robotic arm joints, assign the joint angle values of the human arm to several joints of the robotic arm, obtain the joint angle values of the robotic arm following the movement of the human arm, and control the robotic arm to follow the human arm in real time and move synchronously.
[0057] In the above-mentioned robot arm control method, based on a color image, each joint of the human arm and hand is calculated through the positioning neural network, and output to the human arm posture generation network to generate the 6D posture of the human arm, which is then converted into the human arm joint angle, and assigned to each joint of the robot arm based on the corresponding mapping relationship, so as to realize the real-time synchronous movement of the robot arm following the human hand. The required equipment is simple, and only an RGB color camera is needed to realize the movement of the robot arm following the human arm in real time. No motion capture equipment, no specific humanoid robot arm, no depth map or point cloud data are required, thus saving system costs and additional data processing time.
[0058] In addition, the 6D posture of the human arm is used to formulate the mapping rules between the human arm and the robotic arm, which reduces the singularity points caused by the traditional method of using inverse kinematics to solve the joint angles of the robotic arm and reduces the calculation time.
[0059] In order to address the self-occlusion that is prone to occur during movement, a human posture generation network based on the attention mechanism and the joint loss function is used to restore the occluded joint information based on the weights of each joint, and to restore the lost information by aligning the occluding information and the occluded information, making the posture generation results more accurate.
[0060] For details, please refer to Figure 3 ,The arm posture generation network is a posture generation algorithm model that includes an attention mechanism network and a joint loss function.
[0061] The present invention aims at the self-occlusion situations that often occur in human body movement, such as the joint occlusion when the elbow is bent forward 90 degrees or when the fingers are moving, and improves the posture generation model based on the attention mechanism and the joint loss. When the partially occluded human joints are input into the attention mechanism network, different weights are spontaneously assigned to the joints of the human body to generate the human body posture. At the same time, the present invention introduces a reference neural network, whose input is the complete human body joints, and outputs the posture through the convolution network.
[0062] The present invention designs a posture matching loss function to align the occluded and unobstructed human joints, so that the posture generated by the attention mechanism network is closer and closer to the unobstructed posture, and the obtained parameters can restore the missing human joint information as much as possible. At the same time, the present invention also involves a joint loss function, which includes the posture loss, joint position loss, and shape loss of the two networks, and each loss is matched with a pre-given label. As the joint loss decreases during the training process, the optimal model parameters are finally obtained, and the above model can be used to generate the human arm posture for the next step of "human arm and robotic arm" joint angle mapping.
[0063] Furthermore, the posture generation algorithm model includes a reference neural network, an attention mechanism network and a posture matching loss function. When the human body joints in the real-time posture color image are not blocked, the real-time posture color image is input into the reference neural network, and the unblocked human body posture is generated through the ResNet module; when the human body joints in the real-time posture color image are partially blocked, the real-time posture color image is input into the attention mechanism network, and the attention mechanism network assigns different weight values to each human body joint to generate the human body posture, wherein the human body joint with a higher correlation with the blocked joint has a larger weight value; the posture matching loss function aligns the human body posture output by the reference neural network and the attention mechanism network.
[0064] Please refer to Figure 4 In view of the possible self-occlusion during the movement of human arms, the present invention adopts an attention mechanism network to enable each joint point to find a node that is highly correlated with it. For example, when the human arm moves, the elbow joint has the highest correlation with the shoulder joint and the wrist joint. For the elbow joint, the weights of the shoulder joint and the wrist joint are the largest; on the contrary, the weights of the lower limb nodes relative to the elbow joint are relatively low. In this way, even if some joints of the human arm are occluded, the joints with high correlation can be used to assist in generating information about the occluded nodes according to the weights of other nodes, thereby overcoming the problem of inaccurate posture estimation under occlusion.
[0065] In the above attention mechanism network, there are three attention mechanism encoder layers. The input is partially occluded human joints, features are generated layer by layer, and finally an estimated posture is generated.
[0066] In order to align the occluded pose and the unoccluded pose to reduce the generation loss, the present invention introduces a reference neural network, which takes the unoccluded joint as input, passes through the classic ResNet module, generates the unoccluded pose, and generates the mask through a linear transformation. The occluded pose generated by the attention mechanism network is also linearly transformed to generate a predicted mask, and the matching loss L_{M} is calculated through these two masks, where M represents the mask. Each network also generates a human body shape, and the error with the real human body shape is used as part of the joint loss.
[0067] The attention mechanism network input further includes: a number of attention mechanism encoder layers connected in sequence, generating features layer by layer to obtain human body posture.
[0068] Furthermore, the calculation formula of the joint loss function L is:
[0069] L=L_{P_i}+L_{J_i}+L_{S_i}+L_{M};
[0070] Among them, L_{P_i} is the posture loss of all joints of the human body, L_{J_i} is the position loss of all joints of the human body, L_{S_i} is the shape loss of all joints of the human body, i=1 is the reference neural network, i=2 is the attention mechanism network, P represents posture, J represents joint position, S represents human body shape, and each loss term is calculated by the mean absolute error.
[0071] In another specific implementation of the embodiment of the present invention, the arm posture generation network also includes: an SMPL human skeleton skin model; the SMPL human skeleton skin model generates skeleton joint position coordinates through posture, and the error between the skeleton joint position coordinates and the true label of the human joint position is used as part of the joint loss.
[0072] The mapping relationship between the joints of the human arm and the joints of the robotic arm is clarified as follows:
[0073] For example, see Figure 5 , the 3D pose generated by the optimized human arm posture generation network is converted into three Euler angles in the XYZ coordinate system, and then according to the preset mapping rules, the joint angles of the three joint points of the human arm: shoulder joint, elbow joint, and wrist joint are mapped to the joint angles of the 7-axis robot arm, and the 21 joint angles of the human hand are mapped to the 21 joint angles of the robot.
[0074] Since the 21 joints of the human hand and the 21 joints of the robot can be mapped one by one, the specific mapping rules are: how to map the three joints of the human arm to the 7 joints of the 7-axis robot. The shoulder joint of the human arm corresponds to the 1st and 2nd joints of the robot, the elbow joint of the human arm corresponds to the 3rd and 4th joints of the robot, and the wrist joint of the human arm corresponds to the 5th, 6th and 7th joints of the robot.
[0075] The mapping relationship between the human arm and the robot arm joints includes: the shoulder joint of the human arm corresponds to several joints near the base of the robot arm, the elbow joint corresponds to several joints in the middle of the robot arm, and the wrist joint corresponds to several joints near the end of the robot arm. Among them, the shoulder joint, elbow joint and wrist joint correspond to different joints of the robot arm.
[0076] Furthermore, the step S300 of assigning the joint angle values of the human arm to the joints of the robot arm includes:
[0077] Step S310, converting the 3D posture and 3D coordinates of the shoulder joint, elbow joint, and wrist joint of the human arm into joint angle values of the corresponding joints.
[0078] Step S320, based on the mapping relationship between the human arm and the robotic arm joints, assign the joint angle values of the human arm to the joints of the robotic arm.
[0079] Please refer to Figure 6 , showing the correspondence between the human arm coordinate system and the robot arm coordinate system. It can be seen that both XYZ coordinate axes use the right-handed coordinate system. In the mapping rule, the 3D poses of the three joints of the human left arm: shoulder joint, elbow joint, and wrist joint are first converted into three Euler angles in the XYZ coordinate system. For example, the 3D coordinates of the left shoulder joint [x0, y0, z0], the 3D pose is converted into Euler angles:
[0080] [S_{P_x}], [S_{P_y}], [S_{P_z}]\rightarrow[S_{theta_x}], [S_{theta_y}], [S_{theta_z}];
[0081] Right shoulder joint 3D coordinates [xr, yr, zr], 3D Euler angles [S_{r_x}, S_{r_y}, S_{r_z}], left elbow joint 3D coordinates [x3, y3, z3], 3D pose converted to Euler angles:
[0082] [E_{P_x}], [E_{P_y}], [E_{P_z}]\rightarrow[E_{theta_x}], [E_{theta_y}], [E_{theta_z}];
[0083] The 3D coordinates of the left wrist joint [x5, y5, z5], 3D pose converted to Euler angles:
[0084] [W_{P_x}], [W_{P_y}], [W_{P_z}]\rightarrow[W_{theta_x}], [W_{theta_y}], [W_{theta_z}].
[0085] Joint 0: The left shoulder rotates around the y-axis, such as Figure 7a As shown. Joint 1: The left shoulder rotates around the z-axis, as shown Figure 7b As shown. Joint 2: The left elbow rotates around the x-axis, as shown Figure 7c As shown. Joint 3: Left elbow, divided into two situations: Figure 8 shown.
[0086] If |joint3|<\theta:
[0087] joint3 = E_{\theta_y}, #The elbow swings horizontally around the y-axis;
[0088] otherwise:
[0089] joint3 = E_{\theta_x}, #The elbow swings upward and downward around the z-axis;
[0090] If joint3>0:
[0091] joint2-=90, #Elbow upward, joint2 rotates 90 degrees clockwise.
[0092] The above content is the pseudo code of the algorithm, which is used to reflect the specific implementation logic. For easy recognition by human eyes, please refer to Fig. 9 and Fig.10 .
[0093] The mapping between human arm joint angles and robotic arm joints is shown in Table 1.
[0094] Table 1
[0095]
[0096] Accordingly, please refer to Fig.11 The second aspect of the embodiment of the present invention further provides a robot arm posture following control device, which controls the robot arm to follow the real-time synchronous movement of the human arm based on the above robot arm posture following control method, including:
[0097] Image recognition module 1, which is used to obtain a real-time posture color image of a human body, and recognize the human arms and joints in the real-time posture color image based on a positioning neural network;
[0098] Posture generation module 2, which is used to generate a network based on the arm posture, obtain the 6D posture of the human arm, and convert it into the joint angle value of the human arm;
[0099] The joint mapping module 3 is used to assign the joint angle values of the human arm to several joints of the robotic arm based on the mapping relationship between the human arm and the robotic arm joints, obtain the joint angle values of the robotic arm following the movement of the human arm, and control the robotic arm to follow the human arm in real time and move synchronously.
[0100] For details, please refer to Fig.12 , the joint mapping module 3 includes:
[0101] A posture conversion unit 31, which is used to convert the 3D posture and 3D coordinates of the shoulder joint, elbow joint and wrist joint of the human arm into joint angle values of corresponding joints;
[0102] The joint amplitude unit 32 is used to assign the joint angle value of the human arm to each joint of the robotic arm based on the mapping relationship between the human arm and the robotic arm joints.
[0103] The robot arm posture following control device controls the robot arm through the aforementioned robot arm posture following control method, recognizes the posture and joint angle of each joint of the arm based on the real-time image of the human arm, and assigns values to each joint of the robot arm according to the corresponding relationship, so that each joint of the robot arm maintains a following relationship with the joints of the human arm according to the corresponding relationship, and moves synchronously with the human arm.
[0104] Correspondingly, the third aspect of an embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the above-mentioned robotic arm posture following control method.
[0105] Correspondingly, the fourth aspect of the embodiments of the present invention further provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by a processor, the above-mentioned robot arm posture following control method is implemented.
[0106] The embodiment of the present invention aims to protect a method and device for controlling the posture following of a robot arm, wherein the method comprises the following steps: obtaining a real-time posture color image of a human body, identifying the human arm and joints in the real-time posture color image based on a positioning neural network; obtaining the 6D posture of the human arm based on an arm posture generation network, and converting it into joint angle values of the human arm; assigning the joint angle values of the human arm to several joints of the robot arm based on a mapping relationship between the human arm and the robot arm joints, obtaining the various joint angle values of the robot arm following the movement of the human arm, and controlling the robot arm to follow the synchronous movement of the human arm in real time. The above technical solution has the following effects:
[0107] Only by using the color image of the human arm, the robotic arm can follow the synchronous movement of the human arm in real time without any singularity. It has the advantages of accurate following, flexibility and strong versatility, and is suitable for arm control of various types of robotic arms.
[0108] It should be understood that the above specific embodiments of the present invention are only used to illustrate or explain the principles of the present invention, and do not constitute a limitation of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included in the protection scope of the present invention. In addition, the appended claims of the present invention are intended to cover all changes and modifications that fall within the scope and boundaries of the appended claims, or the equivalent forms of such scope and boundaries.
Claims
1. A robot arm posture following control method, characterized in that: The steps include: Acquire a real-time posture color image of a human body, and identify human arms and joints in the real-time posture color image based on a positioning neural network; Based on the arm posture generation network, the 6D posture of the human arm is obtained, and the 6D posture is converted into the joint angle value of the human arm; Based on the mapping relationship between the human arm and the mechanical arm joints, the joint angle values of the human arm are assigned to several joints of the mechanical arm, and the joint angle values of the mechanical arm following the movement of the human arm are obtained, and the mechanical arm is controlled to follow the human arm in real time and move synchronously; The arm posture generation network is a posture generation algorithm model including an attention mechanism network and a joint loss function; The posture generation algorithm model includes a reference neural network, an attention mechanism network and a posture matching loss function; When there is no occlusion in the human body joints in the real-time posture color image, the real-time posture color image is input into the reference neural network, and the unoccluded human body posture is generated through the ResNet module; When a human joint in the real-time posture color image is partially blocked, the real-time posture color image is input into the attention mechanism network, and the attention mechanism network assigns different weight values to each joint of the human body to generate a human posture, wherein the human joint with a higher correlation with the blocked joint has a larger weight value; The posture matching loss function aligns the human body posture output by the reference neural network and the human body posture output by the attention mechanism network; The attention mechanism network input includes: a plurality of attention mechanism encoder layers connected in sequence, generating features layer by layer to obtain the human body posture.
2. The robot arm posture following control method according to claim 1, characterized in that: The calculation formula of the joint loss function L is: L=L_{P_i}+L_{J_i}+L_{S_i}+L_{M}; Among them, L_{P_i} is the posture loss of all joints of the human body, L_{J_i} is the position loss of all joints of the human body, L_{S_i} is the shape loss of all joints of the human body, L_{M} is the mask matching loss, M is the mask, i=1 is the reference neural network, i=2 is the attention mechanism network, P represents posture, J represents joint position, S represents human body shape, and each loss item is calculated by the mean absolute error.
3. The robot arm posture following control method according to claim 1 or 2, characterized in that: The arm posture generation network also includes: an SMPL human skeleton skin model; The SMPL human skeleton skinning model generates skeleton joint position coordinates through posture, and the error between the skeleton joint position coordinates and the true label of the human joint position is used as part of the joint loss.
4. The robot arm posture following control method according to claim 1, characterized in that: The mapping relationship between the human arm and the mechanical arm joints includes: the shoulder joint of the human arm corresponds to a plurality of joints close to the base of the mechanical arm, the elbow joint corresponds to a plurality of joints in the middle of the mechanical arm, and the wrist joint corresponds to a plurality of joints close to the end of the mechanical arm; The joints of the robotic arm corresponding to the shoulder joint, the elbow joint and the wrist joint are all different.
5. The robot arm posture following control method according to claim 4, characterized in that: The step of assigning the joint angle values of the human arm to the joints of the robot arm comprises: Convert the 3D positions and 3D coordinates of the shoulder joint, elbow joint, and wrist joint of the human arm into joint angle values of the corresponding joints; Based on the mapping relationship between the human arm and the robotic arm joints, the joint angle values of the human arm are assigned to the joints of the robotic arm.
6. A robot arm posture following control device, characterized in that: The method for controlling the posture following control of a robotic arm according to any one of claims 1 to 5 controls the robotic arm to follow the real-time synchronous movement of a human arm, comprising: An image recognition module, which is used to obtain a real-time color image of a human body and recognize the arms and joints of the human body in the real-time color image of the human body based on a positioning neural network; A posture generation module, which is used to obtain the 6D posture of the human arm based on the arm posture generation network, and convert it into the joint angle value of the human arm; The joint mapping module is used to assign the joint angle values of the human arm to several joints of the robotic arm based on the mapping relationship between the human arm and the robotic arm joints, obtain the various joint angle values of the robotic arm following the movement of the human arm, and control the robotic arm to follow the synchronous movement of the human arm in real time.
7. An electronic device, characterized in that: include: at least one processor; And a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the robot arm posture following control method as described in any one of claims 1-5.
Citation Information
Patent Citations
Action mapping method and system for heterogeneous humanoid mechanical arm
CN111152218A
Monocular-camera-based method for following single arm of person by six-axis cooperative mechanical arm in real time
CN115805584A
3D human body posture estimation method based on complementary enhancement of key points and grid vertexes
CN116631064A
KR20230080938A
