Object grabbing method, device and equipment of robot, medium and product

Abstract

The invention discloses an object grabbing method, device and equipment of a robot, a medium and a program product, and relates to the technical field of data processing.The method comprises the steps that a first digital twin corresponding to the robot is controlled to conduct simulation grabbing on a second digital twin corresponding to a to-be-grabbed object according to at least one set of candidate grabbing parameters, and a second digital twin corresponding to the to-be-grabbed object is obtained; determining a target grabbing parameter from the candidate grabbing parameters; determining a potential failure type corresponding to the target grabbing parameter, and obtaining a grabbing time sequence data template and a recovery action instruction matched with the potential failure type; the robot is controlled to grab the to-be-grabbed object according to the target grabbing parameters, and real process grabbing time sequence data are collected in real time; according to the collected real process grabbing time sequence data and the grabbing time sequence data template, anomaly detection is conducted on the grabbing process of the robot, and an anomaly detection result is obtained; and if the abnormal detection result is abnormal, the robot is controlled to execute the action recovery instruction, and the grabbing success rate and the production operation efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology and can be applied to the field of financial technology. In particular, it relates to a robot object grasping method, device, equipment, medium, and program product. Background Technology

[0002] In the fields of automated production lines, intelligent warehousing and logistics, and flexible manufacturing, there are scenarios where industrial robots perform tasks such as workpiece gripping, handling, and assembly. However, due to factors such as workpiece posture deviation, environmental interference, or fixture wear, gripping failures often occur.

[0003] Existing technologies rely on the fusion of visual and force sensor information to determine the grasping status online. Once a failure is detected, such as the workpiece slipping or not being grasped, a retry or reset operation is triggered. While this method can handle some failures, it is essentially a post-event remedy, meaning that the grasping has already failed before the retry or reset operation is triggered. There is a significant time delay from the occurrence of the failure to detection and then to the execution of recovery, which also affects production efficiency. Summary of the Invention

[0004] This invention provides a method, apparatus, device, medium, and program product for robot object grasping, so as to improve the success rate of robot object grasping and improve production efficiency.

[0005] In a first aspect, embodiments of the present invention provide a method for a robot to grasp objects, comprising:

[0006] By controlling the first digital twin corresponding to the robot to simulate grasping the second digital twin corresponding to the object to be grasped according to at least one set of candidate grasping parameters, the target grasping parameter with the highest grasping success rate is determined from each of the candidate grasping parameters.

[0007] Determine the potential failure type corresponding to the target capture parameters, and obtain the capture timing data template and recovery action instruction matching the potential failure type;

[0008] The robot is controlled to grasp the object to be grasped according to the target grasping parameters, and the grasping time sequence data of the actual process is collected in real time.

[0009] Based on the real-time collected actual process grasping time sequence data and the grasping time sequence data template, the robot's grasping process is subjected to real-time anomaly detection to obtain real-time anomaly detection results;

[0010] If the real-time anomaly detection result is abnormal, then the robot is controlled to execute the recovery action command.

[0011] Secondly, embodiments of the present invention also provide a robot object grasping device, comprising:

[0012] The simulation module is used to control the first digital twin corresponding to the robot to simulate grasping the second digital twin corresponding to the object to be grasped according to at least one set of candidate grasping parameters, and to determine the target grasping parameter with the highest grasping success rate from the candidate grasping parameters.

[0013] The determination module is used to determine the potential failure type corresponding to the target capture parameters, and to obtain the capture timing data template and recovery action instruction matching the potential failure type;

[0014] The grasping module is used to control the robot to grasp the object to be grasped according to the target grasping parameters, and to collect real-time grasping time sequence data.

[0015] The detection module is used to perform real-time anomaly detection on the robot's grasping process based on the real-time collected actual process grasping time-series data and the grasping time-series data template, and obtain real-time anomaly detection results.

[0016] The control module is used to control the robot to execute the recovery action command if the real-time anomaly detection result is abnormal.

[0017] Thirdly, embodiments of the present invention also provide an electronic device, comprising:

[0018] At least one processor; and

[0019] A memory that is communicatively connected to at least one processor; wherein

[0020] The memory stores instructions that can be executed by at least one processor, such that the at least one processor is able to perform the object grasping method of the robot provided in any embodiment of the present invention.

[0021] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions, which are used to cause a processor to execute and implement the object grasping method of the robot according to any embodiment of the present invention.

[0022] Fifthly, embodiments of the present invention also provide a computer program product, characterized in that the computer program product includes a computer program, which, when executed by a processor, implements the object grasping method of the robot according to any embodiment of the present invention.

[0023] This invention uses a digital twin for simulated grasping to determine the target grasping parameters with the highest success rate, thereby controlling the robot to grasp objects and improving the success rate. Before grasping, the potential failure types and recovery action instructions corresponding to the target grasping parameters are determined. During the process of controlling the robot to grasp objects, any abnormalities are detected in real time, and recovery action instructions are executed when abnormalities occur to avoid grasping failures and improve the grasping success rate. Compared with the prior art, retries and resets can be avoided, improving production efficiency.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a robot object grasping method according to Embodiment 1 of the present invention;

[0027] Figure 2 This is a flowchart of a robot object grasping method according to Embodiment 2 of the present invention;

[0028] Figure 3 This is a schematic diagram of the structure of a robot object grasping device according to Embodiment 3 of the present invention;

[0029] Figure 4 This is a structural diagram of an electronic device that implements a robot object grasping method according to an embodiment of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," "third," and "fourth," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] In the technical solutions of the embodiments of the present invention, the acquisition, storage, and application of the first digital twin and the second digital twin, etc., all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0033] Example 1

[0034] Figure 1 This is a flowchart of a robot object grasping method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where a robot is controlled to grasp objects. The method can be executed by a robot object grasping device, which can be implemented in hardware and / or software and specifically configured in an electronic device.

[0035] See Figure 1 The robot's object grasping method shown includes:

[0036] S101. By controlling the first digital twin corresponding to the robot to simulate grasping the second digital twin corresponding to the object to be grasped according to at least one set of candidate grasping parameters, the target grasping parameter with the highest grasping success rate is determined from each of the candidate grasping parameters.

[0037] S102. Determine the potential failure type corresponding to the target capture parameter, and obtain the capture timing data template and recovery action instruction matching the potential failure type.

[0038] S103. Control the robot to grasp the object to be grasped according to the target grasping parameters, and collect real-time grasping time sequence data.

[0039] S104. Based on the real-time collected actual process grasping time sequence data and the grasping time sequence data template, perform real-time anomaly detection on the robot's grasping process to obtain real-time anomaly detection results.

[0040] S105. If the real-time anomaly detection result is abnormal, then control the robot to execute the recovery action command.

[0041] In this embodiment, the first digital twin is the robot's digital twin; the second digital twin is the object to be grasped. Candidate grasping parameters may include, but are not limited to, grasping point position, grasping posture, gripping force of the grippers, and grasping lifting height. The grasping point position can be represented by three-dimensional coordinates (X, Y, Z); these three-dimensional coordinates are the coordinates of the grasping point relative to the object to be grasped. For example, (0, 5, 20) can represent the grasping point being 5mm in front of and 20mm above the center of the object to be grasped. The grasping posture can be represented by Euler angles (Rx, Ry, Rz); for example, (0°, 0°, 0°) indicates that the grippers are vertically downwards, with the two fingers parallel to the long axis of the workpiece. The target grasping parameters are the candidate grasping parameters with the highest grasping success rate.

[0042] Potential failure types can be those that could lead to failure when the robot attempts to grasp an object using grasping parameters. These can include, but are not limited to, object tilting, object slippage, gripper failure to grasp, and gripper collision. Object tilting refers to the object tilting during the lifting phase after grasping; object slippage refers to the object sliding during the lifting phase after grasping; failure to grasp refers to the gripper failing to grasp the object; and collision refers to the gripper colliding with the object to be grasped or other objects in the environment.

[0043] The grasping timing data template matching the potential failure type can be the grasping timing data of a grasping process that failed due to a potential failure type; the recovery action instruction matching the potential failure type can be an instruction used to control the robot to perform a corresponding recovery action to avoid the occurrence of the potential failure type. In an optional embodiment, the grasping timing data of a historical grasping process that failed due to a potential failure type is obtained as the grasping timing data template.

[0044] The actual grasping time sequence data can be the grasping time sequence data during the process of the robot grasping the object to be grasped; the grasping time sequence data can include, but is not limited to, torque time sequence data, gripper posture time sequence data, grasping point position time sequence data, and gripping force time sequence data.

[0045] Specifically, the robot is controlled to move to a preparatory pose above the object to be grasped; the preparatory pose includes a preparatory position and a preparatory posture; the preparatory pose can be set independently by technicians according to actual needs or practical experience, and this invention does not limit this; a high-precision point cloud of the object to be grasped is acquired through a point cloud acquisition device configured in the robot; using a point cloud registration algorithm, a virtual object model with the same pose and size as the physical world is generated in the digital twin environment based on the high-precision point cloud of the object to be grasped, namely the second digital twin; the digital twin environment also includes a virtual robot model with the same size as the robot, namely the first digital twin.

[0046] A Monte Carlo algorithm is used to perform random sampling and generate at least one set of candidate grasping parameters. Each set of candidate grasping parameters may include, but is not limited to, candidate grasping point position, candidate grasping posture, candidate gripping force, and candidate grasping lifting height. For each set of candidate grasping parameters, the candidate grasping parameters are perturbed to obtain at least one set of perturbed grasping parameters corresponding to the candidate grasping parameters. For each set of perturbed grasping parameters, in a digital twin environment, the first digital twin is controlled to grasp the second digital twin according to the robot's preparatory pose and the perturbed grasping parameters, and the grasping result corresponding to the perturbed grasping parameters is obtained. The grasping result includes success or failure. Based on the grasping results of each perturbed grasping parameter corresponding to the candidate grasping parameters, the grasping success rate of the candidate grasping parameters is determined. The candidate grasping parameter with the highest grasping success rate among all the candidate grasping parameters is determined as the target grasping parameter.

[0047] A specific algorithm is used to determine the potential failure type corresponding to the target grasping parameters, and to find the grasping timing data template and recovery action instruction that match the potential failure type. The robot is controlled to grasp the object to be grasped according to the target grasping parameters, and real-time grasping timing data is collected. A specific algorithm is used to perform real-time anomaly detection on the robot's grasping process based on the real-time collected real-time grasping timing data and the grasping timing data template, and to obtain real-time anomaly detection results. If the real-time anomaly detection result is abnormal, the robot is controlled to execute the recovery action instruction to stabilize the grasped object in time when the robot's grasping is abnormal and before it may fail, thereby avoiding grasping failure.

[0048] Optionally, the step of performing real-time anomaly detection on the robot's grasping process based on the real-time collected real-process grasping time-series data and the grasping time-series data template to obtain a real-time anomaly detection result includes: extracting auxiliary process grasping time-series data that matches the time sequence of the real-process grasping time-series data from the grasping time-series data template; calculating the similarity between the real-process grasping time-series data and the auxiliary process grasping time-series data; and determining that the real-time anomaly detection result is anomaly if the similarity is greater than or equal to a set threshold.

[0049] The auxiliary process capture time-series data refers to the time-series data in the capture time-series data template that matches the time sequence of the actual process capture time-series data. For example, the real-time collected real process capture time-series data includes real process capture time-series data of the first duration starting from the capture start time. Then, the capture time-series data of the first duration starting from the capture start time is extracted from the capture time-series data template as the auxiliary process capture time-series data. It should be noted that the threshold can be set independently by technicians according to actual needs or practical experience. This invention does not limit this, for example, it can be 90%.

[0050] Specifically, calculating the similarity between the real-process grasping time-series data and each time-series data item in the auxiliary-process grasping time-series data includes: calculating a first similarity between the fitted curve of the torque time-series data in the real-process grasping time-series data and the fitted curve of the torque time-series data in the auxiliary-process grasping time-series data; calculating a second similarity between the fitted curve of the gripper posture time-series data in the real-process grasping time-series data and the fitted curve of the gripper posture time-series data in the auxiliary-process grasping time-series data; calculating a third similarity between the fitted curve of the gripping point position time-series data in the real-process grasping time-series data and the fitted curve of the gripping point position time-series data in the auxiliary-process grasping time-series data; calculating a fourth similarity between the fitted curve of the clamping force time-series data in the real-process grasping time-series data and the fitted curve of the clamping force time-series data in the auxiliary-process grasping time-series data; if at least one of the first, second, third, and fourth similarities is greater than or equal to a set threshold, the real-time anomaly detection result is determined to be abnormal.

[0051] In an optional embodiment, the real process capture time-series data and the capture time-series data template can also be input into a trained lightweight machine learning model; the lightweight machine learning model directly outputs anomaly detection results.

[0052] It is understandable that by adopting the above technical solution, time-series matching auxiliary process time-series data can be extracted from the time-series data template. That is, auxiliary process time-series data with the same time-series data capture stage as the actual process time-series data can be extracted from the time-series data template, thereby calculating similarity and improving the accuracy of anomaly detection.

[0053] Optionally, before determining the target grasping parameter with the highest grasping success rate from the candidate grasping parameters by controlling the first digital twin corresponding to the robot to simulate grasping the object to be grasped according to at least one set of candidate grasping parameters, the method further includes: for each potential failure type, finding a recovery action that matches the potential failure type, and determining the recovery parameter corresponding to the recovery action; generating at least one candidate recovery action instruction based on the recovery parameter and the recovery action; the values ​​of the recovery parameters corresponding to each candidate recovery action instruction are different; generating a digital twin environment before failure based on the grasping timing data corresponding to the time before the grasping failure start time; for each candidate recovery action instruction, controlling the first digital twin to execute the candidate recovery action instruction in the digital twin environment before failure, and calculating the recovery success rate of the candidate recovery action instruction; and determining the candidate recovery action instruction with the highest recovery success rate as the recovery action instruction matching the potential failure type.

[0054] The recovery action can be an action that the robot needs to perform, such as downward displacement and increasing gripping force; the recovery parameter is the parameter corresponding to the recovery action; for example, if the recovery action is downward displacement, the recovery parameter is the displacement distance; if the recovery action is increasing gripping force, the recovery parameter is the gripping force; the candidate recovery action instruction can be used to instruct the robot to perform at least one recovery action. The grasping timing data template may also include position timing data, attitude timing data, and velocity timing data of the first digital twin, as well as position timing data, attitude timing data, and velocity timing data of the second digital twin.

[0055] Specifically, for each potential failure type, a recovery action matching the potential failure type is found, and the recovery parameters corresponding to the recovery action are determined; the value of the recovery parameter is randomly generated; based on the randomly generated recovery parameter value and the recovery action, at least one candidate recovery action instruction is generated; the recovery parameter values ​​corresponding to each candidate recovery action instruction are different; in the capture timing data template matching the potential failure type, the capture timing data corresponding to the previous moment of the capture failure start time is searched; based on the capture timing data corresponding to the previous moment of the capture failure start time, a digital twin environment before failure is generated; for each candidate recovery action instruction, the first digital twin is controlled to execute the candidate recovery action instruction in the digital twin environment before failure to obtain the recovery result and recovery duration; the operation of controlling the first digital twin to execute the candidate recovery action instruction in the digital twin environment is repeated multiple times to obtain multiple recovery results; based on the multiple recovery results, the recovery success rate of the candidate recovery action instruction is calculated; the candidate recovery action instruction with the highest recovery success rate is determined as the recovery action instruction matching the potential failure type.

[0056] It is understandable that by adopting the above technical solution, the recovery action instructions corresponding to the potential failure types can be determined in advance before the robot grasps the object to be grasped. Thus, after determining the potential failure types of the target grasping parameters, the recovery action instructions corresponding to the potential failure types can be directly executed when grasping anomalies are detected, thereby improving the efficiency of anomaly recovery and thus improving the efficiency of production operations.

[0057] In one optional embodiment, after the grasping task is completed, regardless of whether the robot successfully grasps the object or executes the recovery action command, the complete real-world grasping time-series data is recorded in a dataset. Using this complete real-world grasping time-series data and a Bayesian optimization algorithm, the friction coefficient of the second digital twin is fine-tuned. After multiple simulated grasping operations, the prediction accuracy of the digital twin is significantly improved, and the consistency between the virtual detection results and the physical world becomes increasingly higher.

[0058] This invention uses a digital twin for simulated grasping to determine the target grasping parameters with the highest success rate, thereby controlling the robot to grasp objects and improving the success rate. Before grasping, the potential failure types and recovery action instructions corresponding to the target grasping parameters are determined. During the process of controlling the robot to grasp objects, any abnormalities are detected in real time, and recovery action instructions are executed when abnormalities occur to avoid grasping failures and improve the grasping success rate. Compared with the prior art, retries and resets can be avoided, improving production efficiency.

[0059] Example 2

[0060] Figure 2This is a flowchart of a robot object grasping method provided in Embodiment 2 of the present invention. Based on the technical solution of the above embodiments, the present invention has optimized and improved the operation of determining potential failure types.

[0061] Furthermore, the step of "determining the potential failure type corresponding to the target grasping parameter" is refined to "randomly perturb the initial position parameters of the second digital twin to obtain an auxiliary digital twin; control the first digital twin to perform at least one simulated grasping operation on the auxiliary digital twin to obtain at least one simulated grasping result and simulated grasping time series data corresponding to each simulated grasping result; for each failed simulated grasping result, determine the potential failure type of the simulated grasping result based on the simulated grasping time series data corresponding to the simulated grasping result; count the first proportion of the simulated grasping results corresponding to each potential failure type in the failed simulated grasping results; and determine the potential failure type with the highest first proportion as the potential failure type corresponding to the target grasping parameter," thereby improving the operation of determining the potential failure type.

[0062] It should be noted that for any parts not described in detail in the embodiments of the present invention, please refer to the description in the foregoing embodiments.

[0063] See Figure 2 The robot's object grasping method shown includes:

[0064] S201. By controlling the first digital twin corresponding to the robot to simulate grasping the second digital twin corresponding to the object to be grasped according to at least one set of candidate grasping parameters, the target grasping parameter with the highest grasping success rate is determined from each of the candidate grasping parameters.

[0065] S202. Randomly perturb the initial position parameters of the second digital twin to obtain an auxiliary digital twin.

[0066] S203. Control the first digital twin to perform at least one simulation grasp on the auxiliary digital twin to obtain at least one simulation grasp result and simulation grasp timing data corresponding to each simulation grasp result.

[0067] S204. For each failed simulation crawling result, determine the potential failure type of the simulation crawling result based on the simulation crawling timing data corresponding to the simulation crawling result.

[0068] S205. Calculate the first proportion of the simulation crawling results corresponding to each of the potential failure types in the failed simulation crawling results.

[0069] S206. The potential failure type with the highest first proportion is determined as the potential failure type corresponding to the target crawling parameter.

[0070] S207. Obtain the capture time sequence data template and recovery action instruction that match the potential failure type.

[0071] S208. Control the robot to grasp the object to be grasped according to the target grasping parameters, and collect real-time grasping time sequence data.

[0072] S209. Based on the real-time collected actual process grasping time sequence data and the grasping time sequence data template, perform real-time anomaly detection on the robot's grasping process to obtain real-time anomaly detection results.

[0073] S210. If the real-time anomaly detection result is abnormal, then control the robot to execute the recovery action command.

[0074] In this embodiment, the auxiliary digital twin can be the second digital twin after the initial position parameters have been changed; the initial position parameters can be the initial position parameters of the second digital twin.

[0075] Specifically, the initial position parameters of the second digital twin are randomly perturbed to obtain an auxiliary digital twin; the first digital twin is controlled to perform at least one simulated grasping operation on the auxiliary digital twin to obtain at least one simulated grasping result and simulated grasping time series data corresponding to each simulated grasping result; for each failed simulated grasping result, a certain algorithm is used to determine the potential failure type of the simulated grasping result based on the simulated grasping time series data corresponding to the simulated grasping result; the first proportion of the simulated grasping results corresponding to each potential failure type in the failed simulated grasping results is calculated; the potential failure type with the highest first proportion is determined as the potential failure type corresponding to the target grasping parameter.

[0076] Optionally, determining the potential failure type of the simulation grasping result based on the simulation grasping time series data corresponding to the simulation grasping result includes: extracting gripper posture time series data and torque time series data within the lifting start time period from the simulation grasping time series data; the start time of the lifting start time period is the moment when the first digital twin lifts the second digital twin; for the first moment in the lifting start time period, determining the attitude angle change rate corresponding to the first moment based on the gripper attitude angle at the first moment and the gripper attitude angle at the second moment; the second moment is the moment before the first moment; detecting whether the second digital twin tilts at the first moment based on the attitude angle change rate and the torque corresponding to the first moment, and obtaining a detection result; if there is at least one detection result at the first moment indicating that the second digital twin tilts, then object tilting is determined as a potential failure type.

[0077] The lifting start time period can be the moment when the first digital twin grasps and lifts the second digital twin; the start time of the lifting start time period can be the instant when the first digital twin grasps and lifts the second digital twin; the gripper posture timing data can be the timing data of the gripper's posture angle of the first digital twin; the torque timing data can be the timing data of the robot's torque. Optionally, the recovery action corresponding to the object tilting can be the gripper moving downward and increasing the gripping force.

[0078] Specifically, from the simulated grasping time-series data, the gripper attitude time-series data and torque time-series data within the lifting start time period are extracted; for the first moment in the lifting start time period, the angle change between the gripper attitude angle at the first moment and the gripper attitude angle at the second moment is calculated; the interval between the first moment and the second moment is determined; the ratio between the angle change and the interval is determined as the attitude angle change rate corresponding to the first moment; if the attitude angle change rate corresponding to the first moment is greater than a set angle threshold, and the torque corresponding to the first moment is greater than a set torque threshold, then the detection result at the first moment is determined to be that the second digital twin is tilted; if there is at least one detection result at the first moment indicating that the second digital twin is tilted, then the object tilt is determined to be a potential failure type.

[0079] It should be noted that the angle threshold and torque threshold can be set independently by technicians, and this invention does not limit them. For example, the angle threshold can be 0.5 rad / s, and the torque threshold can be 5 Nm.

[0080] Understandably, by adopting the above technical solution, it is possible to determine whether the second digital twin grasped by the gripper of the first digital twin is tilted by determining the rate of change of the attitude angle and the torque at each first moment in the simulation grasping time series data, thereby determining whether there are candidate failure reasons for the object tilting in the target grasping parameters.

[0081] Optionally, determining the potential failure type of the simulated grasping result based on the simulated grasping time series data corresponding to the simulated grasping result includes: extracting the grasping point position time series data, clamping force time series data, and tangential force of the second digital twin from the simulated grasping time series data during the lifting start time period; determining the displacement velocity corresponding to the first moment based on the grasping point position at the first moment and the grasping point position at the second moment for the first moment in the lifting start time period; calculating the maximum static friction force at the first moment based on the clamping force at the first moment and the contact friction coefficient between the clamping surface of the first digital twin's gripper and the surface of the second digital twin; detecting whether the second digital twin slips at the first moment based on the displacement velocity corresponding to the first moment, the maximum static friction force at the first moment, and the tangential force at the first moment, and obtaining the detection result; if at least one detection result at the first moment indicates that the second digital twin has slipped, then object slippage is determined as a potential failure type.

[0082] The displacement velocity corresponding to the first moment can be the displacement velocity of the second digital twin. Optionally, the recovery action corresponding to the object's slippage can be increasing the gripping force and decreasing the robot's grasping and moving speed.

[0083] Specifically, for the first moment in the lifting start time period, the displacement distance between the gripping point position at the first moment and the gripping point position at the second moment is calculated; the interval between the first moment and the second moment is determined; the ratio between the displacement distance at the first moment and the interval is determined as the displacement velocity corresponding to the first moment; the clamping force at the first moment is multiplied by the contact friction coefficient between the gripping surface of the first digital twin's claw and the surface of the second digital twin to obtain the maximum static friction force at the first moment.

[0084] If the displacement velocity at the first moment is greater than the set velocity threshold, and the tangential force at the first moment is greater than the maximum static friction force at the first moment, then the detection result at the first moment is determined to be that the second digital twin has slipped; if there is at least one detection result at the first moment indicating that the second digital twin has slipped, then the object slippage is determined to be a potential failure type.

[0085] It is understandable that by adopting the above technical solution, by determining the displacement velocity and the maximum static friction force at each first moment in the simulation grasping time series data, it is possible to detect whether the second digital twin grasped by the gripper of the first digital twin slips, thereby determining whether there are candidate failure reasons for object slippage in the target grasping parameters.

[0086] In this embodiment of the invention, an auxiliary digital twin is obtained by randomly perturbing the initial position parameters of the second digital twin. The first digital twin is then controlled to perform multiple captures on the position-perturbed auxiliary digital twin, thereby determining the potential failure types corresponding to the multiple capture processes. The potential failure type with the highest proportion is then identified as the potential failure type, which improves the accuracy of the determined potential failure type.

[0087] Example 3

[0088] Figure 3 This is a schematic diagram of a robot object grasping device according to Embodiment 3 of the present invention. This embodiment of the invention is applicable to controlling a robot to grasp objects. The device can execute the robot's object grasping method. The robot object grasping device can be implemented in hardware and / or software, and can be configured in an electronic device.

[0089] like Figure 3 The robot's object grasping device shown includes: a simulation module 301, a determination module 302, a grasping module 303, a detection module 304, and a control module 305. Among these,

[0090] The simulation module 301 is used to control the first digital twin corresponding to the robot to simulate grasping the second digital twin corresponding to the object to be grasped according to at least one set of candidate grasping parameters, and to determine the target grasping parameter with the highest grasping success rate from the candidate grasping parameters.

[0091] The determination module 302 is used to determine the potential failure type corresponding to the target capture parameters, and to obtain the capture timing data template and recovery action instruction matching the potential failure type;

[0092] The grasping module 303 is used to control the robot to grasp the object to be grasped according to the target grasping parameters, and to collect real-time grasping time sequence data.

[0093] The detection module 304 is used to perform real-time anomaly detection on the robot's grasping process based on the real-time collected actual process grasping time sequence data and the grasping time sequence data template, and obtain real-time anomaly detection results.

[0094] The control module 305 is used to control the robot to execute the recovery action command if the real-time anomaly detection result is abnormal.

[0095] This invention uses a digital twin for simulated grasping to determine the target grasping parameters with the highest success rate, thereby controlling the robot to grasp objects and improving the success rate. Before grasping, the potential failure types and recovery action instructions corresponding to the target grasping parameters are determined. During the process of controlling the robot to grasp objects, any abnormalities are detected in real time, and recovery action instructions are executed when abnormalities occur to avoid grasping failures and improve the grasping success rate. Compared with the prior art, retries and resets can be avoided, improving production efficiency.

[0096] Optionally, module 302 includes:

[0097] A perturbation unit is used to randomly perturb the initial position parameters of the second digital twin to obtain an auxiliary digital twin;

[0098] The control unit is used to control the first digital twin to perform at least one simulated grasping operation on the auxiliary digital twin, to obtain at least one simulated grasping result and simulated grasping timing data corresponding to each simulated grasping result;

[0099] The first determining unit is used to determine the potential failure type of each failed simulation crawling result based on the simulation crawling timing data corresponding to the simulation crawling result.

[0100] The statistics unit is used to calculate the first proportion of the simulation crawling results corresponding to each of the potential failure types in the failed simulation crawling results;

[0101] The second determining unit is used to determine the potential failure type with the highest first proportion as the potential failure type corresponding to the target crawling parameter.

[0102] Optionally, the first determining unit is specifically used for:

[0103] From the simulated grasping time series data, extract the gripper posture time series data and torque time series data within the lifting start time period; the start time of the lifting start time period is the moment when the first digital twin lifts the second digital twin;

[0104] For the first moment in the lifting start time period, the attitude angle change rate corresponding to the first moment is determined based on the gripper attitude angle at the first moment and the gripper attitude angle at the second moment; the second moment is the moment before the first moment.

[0105] Based on the rate of change of attitude angle and the torque at the first moment, detect whether the second digital twin tilts at the first moment and obtain the detection result.

[0106] If at least one detection result at the first moment indicates that the second digital twin is tilted, then object tilt is identified as a potential failure type.

[0107] Optionally, the first determining unit is used for:

[0108] From the simulated grasping time series data, extract the grasping point position time series data, clamping force time series data, and tangential force of the second digital twin during the lifting start time period; the start time of the lifting start time period is the moment when the first digital twin lifts the second digital twin;

[0109] For the first moment in the lifting start time period, the displacement velocity corresponding to the first moment is determined based on the gripping point position at the first moment and the gripping point position at the second moment; the second moment is the moment before the first moment.

[0110] The maximum static friction force at the first moment is calculated based on the clamping force at the first moment and the contact friction coefficient between the clamping surface of the first digital twin's gripper and the surface between the second digital twin.

[0111] Based on the displacement velocity at the first moment, the maximum static friction force at the first moment, and the tangential force at the first moment, the detection result is obtained by detecting whether the second digital twin slips at the first moment.

[0112] If at least one detection result at the first moment indicates that the second digital twin has slipped, then object slippage is identified as a potential failure type.

[0113] Optional, the detection module 304 is specifically used for:

[0114] Extract auxiliary process time series data that matches the time series data of the actual process from the time series data template;

[0115] Calculate the similarity between the real process capture time series data and the auxiliary process capture time series data;

[0116] If the similarity is greater than or equal to the set threshold, the real-time anomaly detection result is determined to be abnormal.

[0117] Optionally, the device may also include:

[0118] The lookup module is used to find a recovery action that matches each potential failure type and determine the recovery parameters corresponding to the recovery action for each potential failure type.

[0119] The generation module is used to generate at least one candidate recovery action instruction based on the recovery parameters and the recovery action; the values ​​of the recovery parameters corresponding to each candidate recovery action instruction are different;

[0120] The restoration module is used to generate a digital twin environment before the failure based on the capture time sequence data corresponding to the time before the start time of the capture failure.

[0121] The statistics module is used to control the first digital twin to execute the candidate recovery action instruction in the digital twin environment before the failure for each candidate recovery action instruction, and to count the recovery success rate of the candidate recovery action instruction.

[0122] The instruction determination module is used to determine the candidate recovery action instruction with the highest recovery success rate as the recovery action instruction that matches the potential failure type.

[0123] The robot object grasping device provided in the embodiments of the present invention can execute the robot object grasping method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the robot object grasping method.

[0124] Example 4

[0125] Figure 4 A schematic diagram of a robot object grasping device 410, which can be used to implement embodiments of the present invention, is shown. The robot object grasping device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The robot object grasping device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0126] like Figure 4As shown, the robot's object grasping device 410 includes at least one processor 411 and a memory, such as a read-only memory (ROM) 412 or a random access memory (RAM) 413, communicatively connected to the at least one processor 411. The memory stores computer programs executable by the at least one processor. The processor 411 can perform various appropriate actions and processes based on the computer program stored in the ROM 412 or loaded from storage unit 418 into the RAM 413. The RAM 413 can also store various programs and data required for the operation of the robot's object grasping device 410. The processor 411, ROM 412, and RAM 413 are interconnected via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0127] Multiple components in the robot's object grasping device 410 are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, optical disk, etc.; and a communication unit 419, such as a network card, modem, wireless transceiver, etc. The communication unit 419 allows the robot's object grasping device 410 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0128] Processor 411 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 411 performs the various methods and processes described above, such as object grasping methods for robots.

[0129] In some embodiments, the robot's object grasping method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed onto the robot's object grasping device 410 via ROM 412 and / or communication unit 419. When the computer program is loaded into RAM 413 and executed by processor 411, one or more steps of the robot's object grasping method described above may be performed. Alternatively, in other embodiments, processor 411 may be configured to perform the robot's object grasping method by any other suitable means (e.g., by means of firmware).

[0130] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable robot's object grasping device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0132] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0133] To provide interaction with a user, the systems and techniques described herein can be implemented on a robot's object grasping device, which includes: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the robot's object grasping device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0134] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0135] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability.

[0136] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0137] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for object grasping by a robot, characterized in that, The method includes: By controlling the first digital twin corresponding to the robot to simulate grasping the second digital twin corresponding to the object to be grasped according to at least one set of candidate grasping parameters, the target grasping parameter with the highest grasping success rate is determined from each of the candidate grasping parameters. Determine the potential failure type corresponding to the target capture parameters, and obtain the capture timing data template and recovery action instruction matching the potential failure type; The robot is controlled to grasp the object to be grasped according to the target grasping parameters, and the grasping time sequence data of the actual process is collected in real time. Based on the real-time collected actual process grasping time sequence data and the grasping time sequence data template, the robot's grasping process is subjected to real-time anomaly detection to obtain real-time anomaly detection results; If the real-time anomaly detection result is abnormal, then the robot is controlled to execute the recovery action command.

2. The method according to claim 1, characterized in that, Determining the potential failure type corresponding to the target crawling parameters includes: The initial position parameters of the second digital twin are randomly perturbed to obtain an auxiliary digital twin; The first digital twin is controlled to perform at least one simulated grasping operation on the auxiliary digital twin to obtain at least one simulated grasping result and simulated grasping timing data corresponding to each simulated grasping result. For each failed simulation crawling result, the potential failure type of the simulation crawling result is determined based on the simulation crawling timing data corresponding to the simulation crawling result; The first proportion of the simulation crawling results corresponding to each of the potential failure types in the failed simulation crawling results is calculated. The highest proportion of the first potential failure type is determined as the potential failure type corresponding to the target crawling parameter.

3. The method according to claim 2, characterized in that, The step of determining the potential failure type of the simulation crawling result based on the simulation crawling timing data corresponding to the simulation crawling result includes: From the simulated grasping time series data, extract the gripper posture time series data and torque time series data within the lifting start time period; the start time of the lifting start time period is the moment when the first digital twin lifts the second digital twin; For the first moment in the lifting start time period, the attitude angle change rate corresponding to the first moment is determined based on the gripper attitude angle at the first moment and the gripper attitude angle at the second moment; the second moment is the moment before the first moment. Based on the rate of change of attitude angle and the torque at the first moment, detect whether the second digital twin tilts at the first moment and obtain the detection result. If at least one detection result at the first moment indicates that the second digital twin is tilted, then object tilt is identified as a potential failure type.

4. The method according to claim 3, characterized in that, The step of determining the potential failure type of the simulation crawling result based on the simulation crawling timing data corresponding to the simulation crawling result includes: From the simulated grasping time series data, extract the grasping point position time series data, clamping force time series data, and tangential force of the second digital twin within the initial time period of the lifting process; For the first moment in the lifting start time period, the displacement velocity corresponding to the first moment is determined based on the gripping point position at the first moment and the gripping point position at the second moment. The maximum static friction force at the first moment is calculated based on the clamping force at the first moment and the contact friction coefficient between the clamping surface of the first digital twin's gripper and the surface between the second digital twin. Based on the displacement velocity at the first moment, the maximum static friction force at the first moment, and the tangential force at the first moment, the detection result is obtained by detecting whether the second digital twin slips at the first moment. If at least one detection result at the first moment indicates that the second digital twin has slipped, then object slippage is identified as a potential failure type.

5. The method according to claim 1, characterized in that, The step of performing real-time anomaly detection on the robot's grasping process based on real-time collected actual process grasping time-series data and the grasping time-series data template, and obtaining real-time anomaly detection results, includes: Extract auxiliary process time series data that matches the time series data of the actual process from the time series data template; Calculate the similarity between the real process capture time series data and the auxiliary process capture time series data; If the similarity is greater than or equal to the set threshold, the real-time anomaly detection result is determined to be abnormal.

6. The method according to claim 1, characterized in that, Before determining the target grasping parameter with the highest grasping success rate from among the candidate grasping parameters by simulating grasping of the second digital twin corresponding to the object to be grasped by controlling the first digital twin corresponding to the robot according to at least one set of candidate grasping parameters, the process further includes: For each potential failure type, find the recovery action that matches the potential failure type, and determine the recovery parameters corresponding to the recovery action; Based on the recovery parameters and the recovery action, at least one candidate recovery action instruction is generated; the values ​​of the recovery parameters corresponding to each candidate recovery action instruction are different. Based on the capture timing data corresponding to the time before the start time of the capture failure, a digital twin environment before the failure is generated; For each candidate recovery action instruction, the first digital twin is controlled to execute the candidate recovery action instruction in the digital twin environment before the failure, and the recovery success rate of the candidate recovery action instruction is calculated. The candidate recovery action instruction with the highest recovery success rate is determined as the recovery action instruction that matches the potential failure type.

7. A robot's object grasping device, characterized in that, The device includes: The simulation module is used to control the first digital twin corresponding to the robot to simulate grasping the second digital twin corresponding to the object to be grasped according to at least one set of candidate grasping parameters, and to determine the target grasping parameter with the highest grasping success rate from the candidate grasping parameters. The determination module is used to determine the potential failure type corresponding to the target capture parameters, and to obtain the capture timing data template and recovery action instruction matching the potential failure type; The grasping module is used to control the robot to grasp the object to be grasped according to the target grasping parameters, and to collect real-time grasping time sequence data. The detection module is used to perform real-time anomaly detection on the robot's grasping process based on the real-time collected actual process grasping time-series data and the grasping time-series data template, and obtain real-time anomaly detection results. The control module is used to control the robot to execute the recovery action command if the real-time anomaly detection result is abnormal.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the object grasping method of the robot according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the object grasping method of the robot according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the object grasping method of the robot according to any one of claims 1-6.

Images