Mechanical arm grabbing control method based on digital twinning

Abstract

The invention discloses a mechanical arm grabbing control method based on digital twinning, in the whole control process, a digital twinning technology can truly reproduce equipment position distribution, type, operating environment and operating state through virtual-real fusion and virtual-real mapping of a mechanical arm and a mechanical arm digital twinning body, external changes of the equipment can be seen, and the mechanical arm grabbing control method based on the digital twinning has the advantages of being simple in structure, convenient to operate and high in practicability. More importantly, the working state of each part in the equipment can be seen, real-time alarm is given for abnormal operation of the equipment, managers are assisted in visually mastering the operation state of the equipment, potential safety hazards of the equipment are found in time, the performance of the equipment is continuously improved, and the operation control cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of robot arm grasping control, and more specifically, to a robot arm grasping control method based on digital twin. Background Art

[0002] Industrial automation production line is an important component of manufacturing automation equipment, and industrial robot arm is an important part of it. The extensive use of robot arms in industrial production can not only effectively improve the labor productivity of the manufacturing industry, but also effectively improve the processing accuracy of industrial products. Therefore, the application of more effective and precise control technology is a requirement for the development of industrial robot arms. At present, in real robot arm control applications, the mainstream robot arm control method on the market is manual command control, that is, the teaching method, that is, the engineering and technical personnel plan the movement direction, operation steps and force control of the robot arm, and refine it to various parameters. This control technology method is relatively complex, time-consuming and labor-intensive. When faced with a complex environment of nonlinear structures, it is obviously extremely difficult to use manpower to solve the optimal solution. When the environment changes, it is necessary to teach again, which greatly consumes the energy of professionals. The existing technology usually uses forward and inverse kinematics calculation methods to output the collision-free joint path points of the robot arm when completing the task, so as to realize the movement trajectory planning of the robot arm, but the above method is difficult to realize high-dimensional space planning, and the amount of calculation is large; there are also sampling-based methods to realize the movement trajectory planning of the robot arm, using a smaller amount of calculation to realize high-dimensional space planning, but the above method requires frequent sampling, and the planning efficiency and planning accuracy of the operation path are low. In traditional robot arm grasping technology, it relies on two-dimensional data input, and it is impossible to effectively estimate the grasping posture when facing stacked objects, changing scenes and complex shapes. And in practical applications, it is impossible to carry out continuous learning to cope with new data appearing in changing scenes. Summary of the invention

[0003] 1. Technical issues to be resolved

[0004] In response to the problems existing in the prior art, the present invention provides a robotic arm grasping control method based on digital twins to solve the technical problems mentioned in the background technology.

[0005] (II) Technical solution

[0006] To achieve the above object, the present invention provides the following technical solution: a robot arm grasping control method based on digital twin, comprising the following steps:

[0007] Step 1: construct a digital twin bionic space according to the environmental information of the real environment and the joint posture information of the robotic arm; wherein the digital twin bionic space includes a digital bionic robotic arm, and the operation mode of each subsystem is controlled. The operation mode includes a local offline mode and an online cloud mode. The subsystem jumps to the local offline mode working process or the online cloud mode working process according to the selected operation mode, and collects data of the object to be grasped through a depth camera to obtain the point cloud data of the object to be grasped;

[0008] Step 2: Perform object grasping training on the digital bionic robotic arm based on the reinforcement learning algorithm. When the grasping accuracy of the digital bionic robotic arm meets the preset bionic threshold, obtain the bionic network parameters of the digital twin bionic space. When the robotic arm is running, collect state information periodically according to time. The state information is converted and used to construct the digital twin of the robotic arm. Use the deep reinforcement learning algorithm to train the constructed digital twin of the robotic arm.

[0009] Step 3: During the training process, the state information of the robot arm is analyzed and stored, and the point cloud data is input into a pre-acquired grasping detection model to obtain multiple grasping postures for grasping the object to be grasped;

[0010] Step 4: Input each grasping posture into the scoring network to obtain a score for each grasping posture, where the score for each grasping posture is used to represent a comprehensive score of the collision score and the force closure score of the grasping posture; if the score for each grasping posture obtained is less than a preset score, then, based on the point cloud data, obtain an object model corresponding to the point cloud data from a pre-configured 3D model database;

[0011] Step 5: Based on the object model, a data set is obtained, wherein the data set includes the degree of freedom grasping postures of the object model in different grasping scenarios; the grasping detection model is trained according to the data set to obtain an updated grasping detection model; the point cloud data is input into the updated grasping detection model for reasoning to determine the final grasping posture of the object to be grasped;

[0012] Step 6: configuring the real robotic arm of the real environment according to the bionic network parameters, performing a physical grasping test on the real robotic arm, and determining whether the grasping accuracy of the real robotic arm meets the real preset threshold; if not, returning to step 2; if yes, executing step 5;

[0013] Step seven: Based on the bionic network parameter output control model, use the control model to control the real robotic arm to perform the grasping task. After the training is completed, the optimal path planning is obtained. Through the optimal path planning, the possible fault points in the robotic arm control process are analyzed and compared with the historical data results to obtain the optimal path planning data. The optimal path planning data is sent to the robotic arm and the operation of the robotic arm is controlled. Through the above steps, the robotic arm is controlled to accurately reach the target point and complete the grasping action.

[0014] The present invention is further configured such that, before and after the deep reinforcement learning algorithm starts training, the fuzzy PID controller is used to optimize the parameters and adjust the errors of the robot arm, and the fuzzy PID algorithm is used to perform fuzzification, fuzzy reasoning, and defuzzification in sequence to obtain the output value; and then the output value is inserted into the fuzzy PID algorithm for adjustment.

[0015] The present invention is further configured such that the environmental information includes binocular camera parameters and posture information, obstacle parameters and posture information, and target object parameters and posture information; the robotic arm joint posture information includes robotic arm body parameters and posture information and robotic arm joint node posture information.

[0016] The present invention is further configured such that the digital twin bionic space also includes: a digital bionic binocular camera, a digital bionic obstacle and a digital bionic target object.

[0017] The present invention is further configured such that, based on the object model, a data set is obtained, including: generating an object stacking scene, placing the object on a virtual desktop to form a grasping scene, and performing scene-level grasping posture annotation, recording the 7-DOF grasping posture of the object model in each scene, and scoring each grasping posture to generate the data set.

[0018] The present invention is further configured that, in the step 2, communication is established between the intelligent terminal device of the intelligent manufacturing training sandbox and the cloud platform, a connection is established between the intelligent terminal device and the data processing center through the MQTT sub-node, the CoAP sub-node and the HTTP sub-node, and the data is saved in different databases through the data processing center to realize data display and application call, and the databases include rabbitmq information queue database, redis database, mysql database, mongodb database and logi time series log database.

[0019] The present invention is further configured such that step 2 synchronizes the operation information through a digital twin system and monitors the status of the smart terminal device in real time.

[0020] The present invention is further configured to include, after step four, performing a physical grasping test on the real robotic arm based on a preset time interval, and when the grasping error of the real robotic arm does not meet a preset error threshold, returning to step two.

[0021] The present invention is further configured such that the point cloud data is input into the updated grasping detection model for reasoning to determine the final grasping posture of the object to be grasped, including: inputting the point cloud data into the updated grasping detection model to obtain multiple new grasping postures; inputting each new grasping posture into the scoring network respectively to obtain a score corresponding to each new grasping posture; and obtaining the grasping posture with the highest score value from the multiple new grasping postures as the final grasping posture of the object to be grasped.

[0022] (III) Beneficial effects

[0023] Compared with the prior art, the present invention provides a robot arm grasping control method based on digital twin, which has the following beneficial effects:

[0024] The present invention provides a robotic arm grasping control method based on digital twins. During the entire control process, the digital twin technology can truly reproduce the location distribution, type, operating environment, and operating status of the equipment through the virtual-real fusion and virtual-real mapping of the robotic arm and the digital twin of the robotic arm. Not only can the changes on the outside of the equipment be seen, but more importantly, the working status of each component inside the equipment can be seen, and real-time alarms can be issued for abnormal equipment operation, which can assist managers to intuitively grasp the equipment operation status, promptly discover equipment safety hazards, continuously improve equipment performance, reduce the cost of control operation, improve the safety, reliability, and stability of equipment operation, and improve the health of equipment operation, thereby enhancing the competitiveness of equipment products in the market. In addition, by constructing a digital Twin bionic space, based on the reinforcement learning algorithm, conducts object grasping training on the digital bionic robotic arm of the digital twin bionic space, continuously adjusts the network parameters of the digital twin bionic space, obtains the bionic network parameters of the digital twin bionic space corresponding to the digital bionic robotic arm whose grasping accuracy meets the bionic preset threshold, and feeds the bionic network parameters back to the real environment, configures the real robotic arm in the real environment, adjusts the bionic network parameters according to the grasping effect of the real robotic arm, obtains the control model of the (high-precision) real robotic arm whose grasping accuracy meets the real preset threshold, and uses the control model to control the (high-precision) real robotic arm to perform grasping tasks, which effectively improves the operation path planning efficiency and planning accuracy of the robotic arm when performing grasping tasks, while also improving the operation efficiency and operation accuracy of the robotic arm. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1It is a schematic diagram of the process before bonding two substrates (substrate W1 and substrate W2) in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0026] Embodiment 1:

[0027] like Figure 1 As shown, the present invention discloses a robot arm grasping control method based on digital twin, comprising the following steps:

[0028] Step 1: construct a digital twin bionic space according to the environmental information of the real environment and the joint posture information of the robotic arm; wherein the digital twin bionic space includes a digital bionic robotic arm, and the operation mode of each subsystem is controlled. The operation mode includes a local offline mode and an online cloud mode. The subsystem jumps to the local offline mode working process or the online cloud mode working process according to the selected operation mode, and collects data of the object to be grasped through a depth camera to obtain the point cloud data of the object to be grasped;

[0029] Step 2: Perform object grasping training on the digital bionic robotic arm based on the reinforcement learning algorithm. When the grasping accuracy of the digital bionic robotic arm meets the preset bionic threshold, obtain the bionic network parameters of the digital twin bionic space. When the robotic arm is running, collect state information periodically according to time. The state information is converted and used to construct the digital twin of the robotic arm. Use the deep reinforcement learning algorithm to train the constructed digital twin of the robotic arm.

[0030] Step 3: During the training process, the state information of the robot arm is analyzed and stored, and the point cloud data is input into a pre-acquired grasping detection model to obtain multiple grasping postures for grasping the object to be grasped;

[0031] Step 4: Input each grasping posture into the scoring network to obtain a score for each grasping posture, where the score for each grasping posture is used to represent a comprehensive score of the collision score and the force closure score of the grasping posture; if the score for each grasping posture obtained is less than a preset score, then, based on the point cloud data, obtain an object model corresponding to the point cloud data from a pre-configured 3D model database;

[0032] Step 5: Based on the object model, a data set is obtained, wherein the data set includes the degree of freedom grasping postures of the object model in different grasping scenarios; the grasping detection model is trained according to the data set to obtain an updated grasping detection model; the point cloud data is input into the updated grasping detection model for reasoning to determine the final grasping posture of the object to be grasped;

[0033] Step 6: configuring the real robotic arm of the real environment according to the bionic network parameters, performing a physical grasping test on the real robotic arm, and determining whether the grasping accuracy of the real robotic arm meets the real preset threshold; if not, returning to step 2; if yes, executing step 5;

[0034] Step seven: Based on the bionic network parameter output control model, use the control model to control the real robotic arm to perform the grasping task. After the training is completed, the optimal path planning is obtained. Through the optimal path planning, the possible fault points in the robotic arm control process are analyzed and compared with the historical data results to obtain the optimal path planning data. The optimal path planning data is sent to the robotic arm and the operation of the robotic arm is controlled. Through the above steps, the robotic arm is controlled to accurately reach the target point and complete the grasping action.

[0035] In a further embodiment of the present invention, before and after the deep reinforcement learning algorithm starts training, the fuzzy PID controller is used to optimize the parameters and adjust the errors of the robotic arm, and the fuzzy PID algorithm is used to perform fuzzification, fuzzy reasoning, and defuzzification in sequence to obtain the output value; the output value is then inserted into the fuzzy PID algorithm for adjustment; the environmental information includes binocular camera parameters and posture information, obstacle parameters and posture information, and target object parameters and posture information; the robotic arm joint posture information includes robotic arm body parameters and posture information and robotic arm joint node posture information; the digital twin bionic space also includes: digital bionic binocular camera, digital bionic obstacles, and digital bionic target objects.

[0036] Embodiment 2:

[0037] A robot arm grasping control method based on digital twin includes the following steps:

[0038] Step 1: construct a digital twin bionic space according to the environmental information of the real environment and the joint posture information of the robotic arm; wherein the digital twin bionic space includes a digital bionic robotic arm, and the operation mode of each subsystem is controlled. The operation mode includes a local offline mode and an online cloud mode. The subsystem jumps to the local offline mode working process or the online cloud mode working process according to the selected operation mode, and collects data of the object to be grasped through a depth camera to obtain the point cloud data of the object to be grasped;

[0039] Step 2: Perform object grasping training on the digital bionic robotic arm based on the reinforcement learning algorithm. When the grasping accuracy of the digital bionic robotic arm meets the preset bionic threshold, obtain the bionic network parameters of the digital twin bionic space. When the robotic arm is running, collect state information periodically according to time. The state information is converted and used to construct the digital twin of the robotic arm. Use the deep reinforcement learning algorithm to train the constructed digital twin of the robotic arm.

[0040] Step 3: During the training process, the state information of the robot arm is analyzed and stored, and the point cloud data is input into a pre-acquired grasping detection model to obtain multiple grasping postures for grasping the object to be grasped;

[0041] Step 4: Input each grasping posture into the scoring network to obtain a score for each grasping posture, where the score for each grasping posture is used to represent a comprehensive score of the collision score and the force closure score of the grasping posture; if the score for each grasping posture obtained is less than a preset score, then, based on the point cloud data, obtain an object model corresponding to the point cloud data from a pre-configured 3D model database;

[0042] Step 5: Based on the object model, a data set is obtained, wherein the data set includes the degree of freedom grasping postures of the object model in different grasping scenarios; the grasping detection model is trained according to the data set to obtain an updated grasping detection model; the point cloud data is input into the updated grasping detection model for reasoning to determine the final grasping posture of the object to be grasped;

[0043] Step six: configure the real robotic arm in the real environment according to the bionic network parameters, perform a physical grasping test on the real robotic arm, and determine whether the grasping accuracy of the real robotic arm meets the real preset threshold. If not, return to step two, if yes, execute step five; output a control model based on the bionic network parameters, and use the control model to control the real robotic arm to perform the grasping task. After training, the optimal path planning is obtained. Through the optimal path planning, the possible fault points in the robotic arm control process are analyzed and compared with the historical data results to obtain the optimal path planning data. The optimal path planning data is sent to the robotic arm and the operation of the robotic arm is controlled; through the above steps, the robotic arm is controlled to accurately reach the target point and complete the grasping action.

[0044] In a further embodiment of the present invention, obtaining a data set based on the object model includes: generating an object stacking scene, placing the object on a virtual desktop to form a grasping scene, and performing scene-level grasping posture annotation, recording the 7-degree-of-freedom grasping posture of the object model in each scene, and scoring each grasping posture to generate the data set.

[0045] In a further embodiment of the present invention, in the step 2, communication is established between the intelligent terminal device and the cloud platform of the intelligent manufacturing training sandbox, and a connection is established between the intelligent terminal device and the data processing center through the MQTT sub-node, the CoAP sub-node and the HTTP sub-node, and the data is saved in different databases through the data processing center to realize data display and application call. The databases include rabbitmq information queue database, redis database, mysql database, mongodb database and logi time series log database.

[0046] In a further embodiment of the present invention, step two synchronizes the operation information through the digital twin system and monitors the status of the smart terminal device in real time; after step four, it also includes: based on a preset time interval, performing a physical grasping test on the real robotic arm, and when the grasping error of the real robotic arm does not meet the preset error threshold, returning to step two; the point cloud data is input into the updated grasping detection model for reasoning to determine the final grasping posture of the object to be grasped, including: inputting the point cloud data into the updated grasping detection model to obtain multiple new grasping postures; inputting each new grasping posture into the scoring network respectively to obtain a score corresponding to each new grasping posture; from the multiple new grasping postures, obtaining the grasping posture with the highest score as the final grasping posture of the object to be grasped.

[0047] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be listed as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A robot arm grasping control method based on digital twin, characterized in that: The following steps are involved: Step 1: construct a digital twin bionic space according to the environmental information of the real environment and the joint posture information of the robotic arm; wherein the digital twin bionic space includes a digital bionic robotic arm, and the operation mode of each subsystem is controlled. The operation mode includes a local offline mode and an online cloud mode. The subsystem jumps to the local offline mode working process or the online cloud mode working process according to the selected operation mode, and collects data of the object to be grasped through a depth camera to obtain the point cloud data of the object to be grasped; Step 2: Perform object grasping training on the digital bionic robotic arm based on the reinforcement learning algorithm. When the grasping accuracy of the digital bionic robotic arm meets the preset bionic threshold, obtain the bionic network parameters of the digital twin bionic space. When the robotic arm is running, collect state information periodically according to time. The state information is converted and used to construct the digital twin of the robotic arm. Use the deep reinforcement learning algorithm to train the constructed digital twin of the robotic arm. Step 3: During the training process, the state information of the robot arm is analyzed and stored, and the point cloud data is input into a pre-acquired grasping detection model to obtain multiple grasping postures for grasping the object to be grasped; Step 4: Input each grasping posture into the scoring network to obtain a score for each grasping posture, where the score for each grasping posture is used to represent a comprehensive score of the collision score and the force closure score of the grasping posture; if the score for each grasping posture obtained is less than a preset score, then, based on the point cloud data, obtain an object model corresponding to the point cloud data from a pre-configured 3D model database; Step 5: Based on the object model, a data set is obtained, wherein the data set includes the degree of freedom grasping postures of the object model in different grasping scenarios; the grasping detection model is trained according to the data set to obtain an updated grasping detection model; the point cloud data is input into the updated grasping detection model for reasoning to determine the final grasping posture of the object to be grasped; Step 6: configure the real robot arm of the real environment according to the bionic network parameters, perform a physical grasping test on the real robot arm, and determine whether the grasping accuracy of the real robot arm meets the real preset threshold. If not, return to step 2; if yes, execute step 5; Step seven: Based on the bionic network parameter output control model, use the control model to control the real robotic arm to perform the grasping task. After the training is completed, the optimal path planning is obtained. Through the optimal path planning, the possible fault points in the robotic arm control process are analyzed and compared with the historical data results to obtain the optimal path planning data. The optimal path planning data is sent to the robotic arm and the operation of the robotic arm is controlled. Through the above steps, the robotic arm is controlled to accurately reach the target point and complete the grasping action.

2. According to the digital twin-based robotic arm grasping control method of claim 1, it is characterized in that: Before and after the deep reinforcement learning algorithm starts training, the fuzzy PID controller is used to optimize the parameters and adjust the errors of the robotic arm. The fuzzy PID algorithm is used to perform fuzzification, fuzzy reasoning, and defuzzification in sequence to obtain the output value; the output value is then put into the fuzzy PID algorithm for adjustment.

3. According to a digital twin-based robotic arm grasping control method according to claim 1, it is characterized by: The environmental information includes binocular camera parameters and posture information, obstacle parameters and posture information, and target object parameters and posture information; the robotic arm joint posture information includes robotic arm body parameters and posture information and robotic arm joint node posture information.

4. According to a digital twin-based robotic arm grasping control method according to claim 3, it is characterized by: The digital twin bionic space also includes: a digital bionic binocular camera, digital bionic obstacles and digital bionic target objects.

5. The method for controlling a robot arm grasping based on digital twinning according to claim 4 is characterized in that: The method of obtaining a data set based on the object model includes: generating an object stacking scene, placing the object on a virtual desktop to form a grasping scene, and performing scene-level grasping posture annotation, recording the 7-DOF grasping posture of the object model in each scene, and scoring each grasping posture to generate the data set.

6. The method for controlling a robot arm grasping based on digital twinning according to claim 1 is characterized in that: In the step 2, communication is established between the intelligent terminal device and the cloud platform of the intelligent manufacturing training sandbox, and a connection is established between the intelligent terminal device and the data processing center through the MQTT sub-node, CoAP sub-node and HTTP sub-node. The data is saved in different databases through the data processing center to realize data display and application call. The databases include rabbitmq information queue database, redis database, mysql database, mongodb database and loki time series log database.

7. The method for controlling a robot arm grasping based on digital twinning according to claim 1 is characterized in that: The step 2 synchronizes the operation information through the digital twin system and monitors the status of the smart terminal device in real time.

8. According to the digital twin-based robotic arm grasping control method of claim 4, it is characterized in that: After step 4, the method further includes performing a physical grasping test on the real robotic arm based on a preset time interval, and when the grasping error of the real robotic arm does not meet the preset error threshold, returning to step 2.

9. The method for controlling a robot arm grasping based on digital twin according to claim 7 is characterized in that: The step of inputting the point cloud data into the updated grasping detection model for reasoning to determine the final grasping posture of the object to be grasped includes: inputting the point cloud data into the updated grasping detection model to obtain multiple new grasping postures; inputting each new grasping posture into the scoring network to obtain a score corresponding to each new grasping posture; and obtaining the grasping posture with the highest score value from the multiple new grasping postures as the final grasping posture of the object to be grasped.

Images