Dynamic grabbing method and system

Abstract

The invention discloses a dynamic grabbing method and system, and relates to the technical field of automatic manufacturing. The dynamic grabbing method comprises the following steps that S1, positioning and pose recognition are conducted on to-be-grabbed parts based on a deep learning model, and the to-be-grabbed parts are divided into grabbable parts or non-grabbable parts based on pose recognition; s2, calculating an affine transformation matrix based on camera calibration to convert a camera coordinate system into a robot coordinate system; s3, constructing a part track motion model based on the multi-frame image, and predicting a grabbable part track based on the part track motion model; s4, the robot implements a grabbing action and obtains feedback information of the grabbing action, and track planning parameters or time compensation values are adjusted based on the feedback information; realizing time synchronization of the camera and the robot based on the time compensation value; and the optimal adsorption position and speed are calculated according to the predicted grabbable part track and the preset triggering time, so that the movement of the robot is more efficient and smoother.

Description

Technical Field

[0001] This application relates to the field of automated manufacturing technology, and particularly to a dynamic grasping method and system. Background Art

[0002] With the improvement of industrial automation, the demand for automatic recognition and grasping of small parts in a dynamic environment is increasing.

[0003] However, due to problems such as small part size, changing conveyor belt speed, and possible different poses, precise and efficient dynamic grasping has become a challenge. Traditional static grasping methods are difficult to adapt to such complex situations. Summary of the Invention

[0004] To solve the above problems, this application discloses a dynamic grasping method, which calculates the optimal adsorption position and speed through the predicted trajectory of the graspable part and the preset trigger time, making the movement of the robot more efficient and smooth; at the same time, a corresponding dynamic grasping system is proposed to implement the dynamic grasping method under different circumstances.

[0005] The first technical solution adopted by this application is: to provide a dynamic grasping method, including the following steps:

[0006] S1: Locate and identify the pose of the part to be grasped based on a deep learning model, and divide the part to be grasped into a graspable part or a non-graspable part based on the pose identification;

[0007] S2: Calculate the affine transformation matrix based on camera calibration to convert the camera coordinate system into the robot coordinate system;

[0008] S3: Construct a part trajectory motion model based on multiple frames of images, and predict the trajectory of the graspable part based on the part trajectory motion model; calculate the position and speed of the robot to adsorb the graspable part based on the predicted trajectory of the graspable part and the preset trigger time;

[0009] S4: The robot performs a grasping action and obtains feedback information of the grasping action, and adjusts the trajectory planning parameters or the time compensation value based on the feedback information; realize the time synchronization between the camera and the robot based on the time compensation value.

[0010] Among them, the trajectory points of a single graspable part are obtained based on the part pose and target tracking in the multiple frames of images; the speed information of a single graspable part is obtained based on the position change of a single target between adjacent frames and the time interval between adjacent frames.

[0011] Among them, trajectory points of multiple graspable parts are obtained based on the part poses and target tracking in the multiple frames of images; speed information of multiple graspable parts is obtained based on the position changes of multiple targets between adjacent frames and the time interval between adjacent frames, and the median value of the speeds of multiple graspable parts is obtained as the part speed to improve the grasping accuracy.

[0012] Among them, the grasping action includes a downward-slanting following stage, a horizontal following stage, and an upward-slanting following stage; adsorption between the suction cup and the graspable part is achieved based on the downward-slanting following stage; the adsorption stability between the suction cup and the graspable part is improved based on the horizontal following stage; and the graspable part is safely separated from the tray based on the upward-slanting following stage.

[0013] Among them, the speed component of the robot in the horizontal direction is equal to the speed of the graspable part to improve the grasping accuracy.

[0014] Among them, the feedback information includes part position deviation and adsorption stability; trajectory planning parameters or time compensation values are adjusted based on the part position deviation and the adsorption stability.

[0015] Among them, a trigger signal is sent after the graspable part reaches a preset position, the current timestamp is obtained after receiving the trigger signal, and a time compensation value is calculated based on the current timestamp to achieve time synchronization between the camera and the robot.

[0016] The second technical solution adopted in this application is: providing a dynamic grasping system that can apply the dynamic grasping method described in any one of the above, including:

[0017] A camera module for collecting multiple frames of images and performing pose recognition on the parts in the multiple frames of images;

[0018] A calculation module for constructing a part trajectory motion model based on the multiple frames of images and predicting the trajectory of the graspable part according to the part trajectory motion model; and also for calculating an affine transformation matrix based on camera calibration to convert the camera coordinate system into the robot coordinate system;

[0019] A robot control module for performing a grasping action and obtaining feedback information of the grasping action according to the data provided by the calculation module; and also for adjusting trajectory planning parameters or time compensation values based on the feedback information and achieving time synchronization between the camera and the robot based on the time compensation value;

[0020] A time synchronization module, which is configured in the upper computer and responds immediately when receiving a trigger signal.

[0021] The third technical solution adopted in this application is: providing an electronic device, which includes: a memory and a processor coupled to each other, and the processor is configured to execute program instructions stored in the memory to implement the steps of the dynamic grasping method described in any one of the above.

[0022] The fourth technical solution adopted in this application is: providing a computer-readable storage medium, which stores program data, and the program data can be executed by a processor to implement the steps of the dynamic grasping method described in any one of the above.

[0023] Due to the adoption of the above technical solutions in this application, compared with the prior art, it has at least one of the following beneficial effects:

[0024] 1. The optimal adsorption position and speed calculated through the predicted trajectory of the graspable part and the preset trigger time make the movement of the robot more efficient and smooth.

[0025] 2. The deep learning model is used to locate and recognize the pose of the part to be grasped, and the part to be grasped is divided into graspable or non-graspable categories, ensuring that only the parts suitable for grasping will be attempted to be adsorbed by the robot.

[0026] 3. Using the feedback information to adjust the trajectory planning parameters or the time compensation value to achieve time synchronization between the camera and the robot, reducing the operation error caused by time asynchronization.

[0027] 4. When implementing the grasping action, a three-segment following linear motion is adopted, which not only helps the suction cup to firmly adsorb the part, but also effectively avoids the risk of collision with the tray or other obstacles. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of this application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0029] Among them:

[0030] Figure 1 is a schematic flowchart of an embodiment of the dynamic grasping method provided by this application;

[0031] Figure 2 is a schematic flowchart of an embodiment of obtaining graspable part information based on a single-frame image in this application;

[0032] Figure 3 is a schematic flowchart of an embodiment of camera calibration provided by this application;

[0033] Figure 4 Schematic flowchart of an embodiment for obtaining the robot motion trajectory in this application;

[0034] Figure 5 Schematic diagram for obtaining the median speed of a single-frame multi-part image provided in this application;

[0035] Figure 6 Schematic diagram for obtaining the median speed of multi-frame multi-part images provided in this application;

[0036] Figure 7 Schematic diagram of an embodiment of the entire process of dynamic grasping provided in this application;

[0037] Figure 8 Schematic framework diagram of an embodiment of the dynamic grasping system provided in this application;

[0038] Figure 9 Schematic structural diagram of an embodiment of the computer device in this application;

[0039] Figure 10 Schematic structural diagram of an embodiment of the computer-readable storage medium in this application. Detailed implementation manners

[0040] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. It can be understood that the specific embodiments described herein are only used to explain this application, rather than limiting this application. Additionally, it should be noted that for the sake of description, only parts related to this application are shown in the accompanying drawings, rather than all structures. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope protected by this application.

[0041] The terms "first", "second", etc. in this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.

[0042] References to "embodiments" in this specification mean that the particular features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0043] When dealing with small parts moving at high speeds, existing dynamic grasping techniques often struggle to meet the required high-precision requirements. For example, when the conveyor belt speed is relatively fast, there may be deviations in predicting the part position and selecting the grasping timing, resulting in grasping failures. In contrast, the dynamic grasping method of the present application calculates the optimal adsorption position and speed through the predicted graspable part trajectory and preset trigger time, making the robot's movement more efficient and smooth. As Figure 1 shown, Figure 1 is a flowchart of an embodiment of the dynamic grasping method provided by the present application, including the following steps:

[0044] S1: Locate and identify the pose of the parts to be grasped based on a deep learning model, and divide the parts to be grasped into graspable parts or non-graspable parts based on the pose identification; the dataset used to train the deep learning model, which includes part images in different poses; label the images to determine whether they are graspable parts and their accurate positions and orientations.

[0045] As Figure 2 shown, Figure 2 is a flowchart of an embodiment of the present application for obtaining graspable part information based on a single-frame picture.

[0046] Obtain a frame of image as input, use the YOLOv8 segmentation model to analyze this frame of image, identify and classify all parts in the image, and determine whether they are facing up or down; in this embodiment, the parts facing up are graspable parts; if a part facing down is detected, the robot can be controlled to turn over the reverse part before grasping.

[0047] It should be clear that in this embodiment, whether a part is a graspable part is determined based on the front and back of the part. In other embodiments, whether a part is a graspable part can be determined from features such as the material or shape of the part, and no limitations are imposed on this.

[0048] If a part facing up or down is detected, it enters the target tracking stage. Through the target tracking function of YOLOv8, the system can track the movement of these parts between consecutive frames to help predict their future positions and orientations; if no part facing up or down is detected, it returns to the YOLOv8 segmentation model to continue the analysis of the next frame.

[0049] In the multi-target scenario, calculate the relative distances between all tracked targets and obtain the targets that are far from other targets; preferentially select the parts with a high degree of outlier for grasping, which facilitates the operation of the robot and reduces the risk of collision with other parts.

[0050] For the selected target part, use the SAM2 large model for more accurate segmentation. SAM2 can provide higher accuracy than YOLOv8, especially suitable for situations that require very accurate edge information, ensuring a more precise selection of the grasping point; according to the segmentation result provided by SAM2, calculate the centroid position of the part contour, and this centroid will be used as the grasping center point of the robot to ensure that the suction cup can accurately adsorb at the best position of the part; by calculating the minimum circumscribed rectangle of the part contour, the main axis of the part can be obtained, and then control the robot to adjust its grasping posture to match the direction of the part, improving the adsorption stability.

[0051] As Figure 3 shown, Figure 3 is a schematic flow chart of an embodiment of the camera calibration provided by this application.

[0052] S2: Calculate the affine transformation matrix based on the camera calibration to convert the camera coordinate system to the robot coordinate system; the affine transformation matrix obtained through an accurate camera calibration process can achieve a high-precision mapping from the camera coordinate system to the robot coordinate system. This means that the target position detected by the vision system can be accurately converted into the actual position where the robot needs to perform operations, greatly improving the success rate and accuracy of grasping or operating tasks.

[0053] The following details the acquisition process of the camera-robot affine matrix:

[0054] The camera takes pictures of the scene containing the standard checkerboard calibration board, and multiple images can be taken from multiple angles and positions. Detect all the corner points of the checkerboard in each image, and the positions of the corner points represent the coordinates in the camera coordinate system.

[0055] If all the necessary corner points are successfully recognized, use the teaching function of the robot to let the robotic arm touch these corner points one by one, and record the corresponding robot plane coordinates of each corner point; if the corner points are not successfully recognized, re-acquire the pictures by adjusting the angle or lighting conditions of the camera to obtain clearer images.

[0056] Calculate the root mean square error (RMSE) based on the corner coordinates in the camera coordinate system and the corresponding robot plane coordinates to evaluate the conversion accuracy; judge the accuracy of the affine transformation matrix based on the comparison between the RMSE and the preset threshold. If the RMSE is less than the preset threshold, the current affine transformation matrix is used to perform the operation of converting the camera coordinate system to the robot coordinate system; if the RMSE is greater than or equal to the preset threshold, the robot performs touch teaching again, and the robot re-obtains the affine transformation matrix based on adding samples or optimizing the touch accuracy.

[0057] It should be clear that in this embodiment, the size of the preset threshold is 0.2; in other embodiments, the size of the preset threshold can be set to other values, and no limitation is imposed on this.

[0058] As Figure 4 shown, Figure 4 is a schematic flowchart of an embodiment for obtaining the robot motion trajectory in this application.

[0059] S3: Construct a part trajectory motion model based on multiple frames of images, and predict the trajectory of the graspable part based on the part trajectory motion model; by analyzing the pose changes of the part in multiple frames of images, the future motion trajectory of the part can be predicted more accurately; calculate the position and speed of the robot to adsorb the graspable part based on the predicted graspable part trajectory and the preset trigger time; using the predicted part trajectory and the preset trigger time, the optimal path of the robot to adsorb the part can be calculated in advance, including the position and speed. This not only reduces unnecessary movement and waiting time, but also improves the operating efficiency of the entire system.

[0060] The following details the detailed process of obtaining the robot motion trajectory in this embodiment:

[0061] Based on the part poses (t, x, y, z, u) in multiple frames of images, obtain the trajectory points of a single part between consecutive frames [(t, x, y, z, u),...] through target tracking technology; where t is time, x represents the horizontal axis coordinate, y represents the vertical axis coordinate, z represents the vertical axis coordinate, and u represents the rotation angle or pose information of the part.

[0062] The part trajectory motion model uses a first-order linear regression model f(t, x, y, z, u) to fit the trajectory points to predict the future motion path of the part; it should be clear that the part in this embodiment is located on the conveyor belt and maintains a stable linear motion; in other embodiments, different motion models can be selected for trajectory prediction based on different part motion states, and no limitation is imposed on this.

[0063] Calculate the path of the robot based on the preset trigger time and the part trajectory, that is, calculate the path of the robot based on the robot action time point and the part trajectory; obtain the motion speed of the robot through reversible operation based on the robot path points.

[0064] Due to the inherent time difference between the actual robot execution timeline and the camera timeline, there will be a problem of time asynchronization during the first attempt at grasping; based on the horizontal following position, the time synchronization is fine-tuned, and after the timelines are aligned, the motion trajectory and speed of the robot are output.

[0065] S4: The robot performs the grasping action and obtains the feedback information of the grasping action, and adjusts the trajectory planning parameters or the time compensation value based on the feedback information; after completing the trajectory planning, the robot starts to perform the grasping action according to the calculated path points and speed.

[0066] The grasping action includes moving to a specified position, adjusting the posture, and finally adsorbing the part; during or after the grasping process, the system collects relevant feedback information; based on the feedback information, it is judged whether there is an error in the current trajectory planning or time synchronization. If there is an error, the trajectory planning parameters or the time synchronization value are adjusted. The trajectory planning parameters include the position of the path point, the motion speed, and the acceleration; the time synchronization between the camera and the robot is achieved based on the time compensation value; based on the adjusted time compensation value, the timelines of the camera and the robot are recalibrated to eliminate the time difference between the two; the time synchronization is achieved during the trial grasping process in step S3.

[0067] It should be clear that in this embodiment, a single adjustment is to adjust the trajectory planning parameters or the time compensation value; in other embodiments, a single adjustment can simultaneously adjust the trajectory planning parameters and the time compensation value, and no limitation is made thereto.

[0068] By obtaining the feedback information of the grasping action in real time, the system can quickly identify and correct the deviation during the grasping process. For example, if the part position does not match the prediction, the grasping path can be corrected by adjusting the trajectory planning parameters, thereby significantly improving the success rate and accuracy of grasping; the feedback mechanism can avoid ineffective grasping attempts caused by incorrect trajectory planning or time asynchronization, reduce unnecessary robotic arm movements and energy consumption, and improve the overall production efficiency.

[0069] In summary, the dynamic grasping method of this embodiment includes the following steps: S1: Based on a deep learning model, locate and identify the pose of the part to be grasped, and divide the part to be grasped into a graspable part or a non-graspable part based on the pose identification; S2: Calculate the affine transformation matrix based on camera calibration to convert the camera coordinate system into the robot coordinate system; S3: Construct a part trajectory motion model based on multiple frames of images, and predict the trajectory of the graspable part based on the part trajectory motion model; Calculate the position and speed of the robot to adsorb the graspable part based on the predicted trajectory of the graspable part and the preset trigger time; S4: The robot performs a grasping action and obtains feedback information of the grasping action, and adjusts the trajectory planning parameters or the time compensation value based on the feedback information; Achieve time synchronization between the camera and the robot based on the time compensation value; The optimal adsorption position and speed calculated through the predicted trajectory of the graspable part and the preset trigger time make the movement of the robot more efficient and smooth.

[0070] In one embodiment, trajectory points of a single graspable part are obtained based on the part pose and target tracking in multiple frames of images; speed information of a single graspable part is obtained based on the position change of a single target between adjacent frames and the time interval between adjacent frames; By accurately obtaining the trajectory points and speed information of the part, the future motion state of the part can be predicted more accurately, thereby guiding the robot to perform grasping at the optimal time and position, significantly improving the success rate and accuracy of grasping; This method can update the motion state of the part in real time and is applicable to handling the rapid movement of parts in dynamic environments such as conveyor belts. Even when the part speed changes or a non-linear motion pattern appears, it can be adapted by adjusting the tracking algorithm.

[0071] In one embodiment, trajectory points of multiple graspable parts are obtained based on the part pose and target tracking in multiple frames of images; speed information of multiple graspable parts is obtained based on the position change of multiple targets between adjacent frames and the time interval between adjacent frames, and the median value of the speeds of multiple graspable parts is obtained as the part speed to improve the grasping accuracy; By calculating the median value of the speeds of multiple graspable parts, the error caused by the abnormal motion of a single part can be effectively reduced. This method can provide a more stable and reliable average speed estimate, thereby improving the overall grasping accuracy. Using the median speed can help predict the future position of the part more accurately, which helps the robot better plan its grasping path and ensure grasping at the correct moment and position.

[0072] As Figures 5 - 6 shown, Figure 5 is a schematic diagram for obtaining a median value of speeds of a single-frame multi-part picture provided by this application; Figure 6 is a schematic diagram for obtaining a median value of speeds of a multi-frame multi-part picture provided by this application.

[0073] The median value of the speeds of a single-frame multi-part is

[0074] When grasping multiple times at the same speed, the median value of multiple speeds can be taken as the part movement speed, that is, the part movement speed V5 is the median speed of the first frame of the picture, V6 is the median speed of the second frame of the picture, and V7 is the median speed of the third frame of the picture.

[0075] It should be clear that in this embodiment, the trajectory of the part is predicted through multiple frames of pictures and the timestamps of the pictures; in other embodiments, the trajectory of the part can also be predicted by a single frame of picture + pre-calibrating the conveyor belt direction and speed in advance or a single frame of picture + conveyor belt encoder, and no limitation is made thereto.

[0076] In one embodiment, the grasping action includes a downward-slanting following stage, a horizontal following stage, and an upward-slanting following stage. The working processes of the downward-slanting following stage, the horizontal following stage, and the upward-slanting following stage are described in detail below:

[0077] Based on the downward-slanting following stage, adsorption between the suction cup and the graspable part is achieved; in the downward-slanting stage, the robot slides downward at a certain angle to approach the part. Based on the design of the downward-slanting following stage, it can be ensured that the suction cup can smoothly contact the part surface, avoiding impacts or unstable contacts caused by vertical descent.

[0078] Based on the horizontal following stage, the adsorption stability between the suction cup and the graspable part is improved; in the horizontal following stage, the robot moves horizontally relative to the part, enabling the suction cup to form a more secure seal on the part surface, thereby enhancing the adsorption force.

[0079] Based on the upward-slanting following stage, the graspable part is safely separated from the tray; in the upward-slanting following stage, the part leaves the tray or other supporting surfaces at a relatively gentle angle, reducing part deformation or damage that may be caused by sudden pulling.

[0080] As Figure 7 shown, Figure 7 is a schematic diagram of an embodiment of the entire dynamic grasping process provided by this application, including the following steps:

[0081] After completing the trajectory calculation, the robot gripper moves to the GP0003 grasping trigger starting point to wait and is triggered at the preset trigger time T0.

[0082] The host computer enables the trigger mode and sends an instruction to the robot to continue the program when the trigger timestamp is reached.

[0083] Maintain a state of not moving in the horizontal position with the part and follow downward-slantingly for t1 time.

[0084] Maintain a state of not moving in the horizontal position with the part and follow horizontally for t2 time.

[0085] Maintain the state of the part in a horizontal position without movement, and follow obliquely upward for t3 time to get out of the height of the tray.

[0086] Based on the trajectory motion model of the part, the xy position of the part at time T0, and the xy positions at times T0+t1, T0+t1+t2, and T0+t1+t2+t3 can be calculated. Combining the preset safety height, grasping height, and separation height, the positions GP0003, GP0004, GP0005, and GP0006 can be obtained. Combining the expected arrival time and calculating in reverse, the robot speeds in the downward oblique following stage, horizontal following stage, and upward oblique following stage can be obtained.

[0087] The horizontal velocity component of the robot is equal to the velocity of the part that can be grasped to improve the grasping accuracy; when the horizontal velocity of the robot is equal to the velocity of the part, the position error caused by the relative velocity difference can be significantly reduced. This "follow-up" state enables the suction cup to adsorb more stably on the surface of the part, thereby improving the success rate and accuracy of grasping.

[0088] The part is attached to the tray and moves with the conveyor belt. If the horizontal velocity component of the robot is not equal to the velocity of the part that can be grasped, it is easy to cause a collision between the robot gripper and the tray.

[0089] In one embodiment, the feedback information includes the part position deviation and adsorption stability; the part position deviation is the difference between the actual grasping position and the preset target position, and the adsorption force of the suction cup on the part is monitored by a sensor to determine whether the adsorption is stable; the sensor can be a torque sensor, a pressure sensor, or any other sensor that can achieve the same technical effect, and no limitation is made in this regard.

[0090] Adjust the trajectory planning parameters or time compensation value based on the part position deviation and adsorption stability; if it is found that the part position deviation is large, the target path of the robot can be corrected by finely adjusting the coordinates of the robot; adjust the motion speed and / or acceleration of the robot based on the magnitude of the part position deviation to ensure that it can reach the target position smoothly without being too fast and causing new deviations.

[0091] If the adsorption stability is insufficient, it means that there are problems with the speed or angle of the robot when contacting the part. At this time, the time synchronization between the robot and the camera can be finely adjusted by adjusting the time compensation value to ensure that the visual detection result is more coordinated with the mechanical operation. In other embodiments, the duration of the horizontal following stage can be appropriately extended to improve the adsorption stability.

[0092] In one embodiment, after the graspable part reaches the preset position, a trigger signal is sent. After receiving the trigger signal, the current timestamp is obtained, and a time compensation value is calculated based on the current timestamp to achieve time synchronization between the camera and the robot. By calculating and applying the time compensation value in real time, the error accumulation caused by the inherent system delay can be effectively reduced, making the entire grasping process more stable and reliable. The ability to dynamically adjust the time compensation value enables the system to adapt to different working environments and condition changes. Whether it is the change in the production line speed or the influence of external interference factors, it can respond flexibly.

[0093] This application also provides a dynamic grasping system, which can apply the dynamic grasping method described in any of the above embodiments, as Figure 8 shown, Figure 8 is a schematic framework diagram of an embodiment of the dynamic grasping system provided by this application, including:

[0094] A camera module, which is used to collect multiple frames of images and perform pose recognition on the parts in the multiple frames of images;

[0095] A calculation module, which is used to construct a part trajectory motion model based on the multiple frames of images, and predict the trajectory of the graspable part according to the part trajectory motion model; it is also used to calculate an affine transformation matrix based on camera calibration to convert the camera coordinate system into the robot coordinate system;

[0096] A robot control module, which is used to execute a grasping action and obtain feedback information of the grasping action according to the data provided by the calculation module; it is also used to adjust the trajectory planning parameters or the time compensation value based on the feedback information, and achieve time synchronization between the camera and the robot based on the time compensation value;

[0097] A time synchronization module, which is configured in the upper computer and responds immediately when receiving the trigger signal.

[0098] The following details the working process of the dynamic grasping system:

[0099] The camera module continuously collects multiple frames of images, and uses a deep learning model to perform pose recognition and classification of the parts; then, the calculation module constructs a part trajectory motion model based on these images, predicts the future position of the parts, and calculates an affine transformation matrix through camera calibration to achieve coordinate system conversion; subsequently, the robot control module plans and executes a grasping action according to the calculation results, and simultaneously obtains real-time grasping feedback information for adjusting the trajectory planning parameters or the time compensation value; when the time synchronization module receives the trigger signal that the part reaches the preset position, it records the current timestamp and calculates the time compensation value to ensure time synchronization between the camera and the robot, thereby achieving efficient and accurate part grasping; the entire process realizes automated and intelligent dynamic grasping operations through the close cooperation between the modules.

[0100] For the above embodiments, the present application provides a computer device. Please refer to Figure 9 , Figure 9 which is a schematic structural diagram of an embodiment of the computer device of the present application. The computer device includes a memory and a processor. Among them, the memory and the processor are coupled to each other. Program data is stored in the memory, and the processor is configured to execute the program data to implement the steps of any one of the above dynamic capture methods.

[0101] In this embodiment, the processor can also be referred to as a CPU (Central Processing Unit). The processor may be an integrated circuit chip with signal processing capabilities. The processor can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0102] For the method of the above embodiments, it can be implemented in the form of a computer program. Therefore, the present application proposes a computer-readable storage medium. Please refer to Figure 10 , Figure 10 which is a schematic structural diagram of an embodiment of the computer-readable storage medium of the present application. Program data that can be run by the processor is stored in the computer-readable storage medium, and the program data can be executed by the processor to implement the steps of any one of the above dynamic capture methods.

[0103] The computer-readable storage medium of this embodiment can be a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc, etc., which can store program data, or it can also be a server storing the program data. The server can send the stored program data to other devices for running, or it can also run the stored program data by itself.

[0104] In several implementation manners provided by the present application, it should be understood that the disclosed method and device can be implemented in other ways. For example, the device implementation manner described above is only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0105] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0106] In addition, each functional unit in various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0107] The above are only the embodiments of the present application, and do not limit the patent scope of the present application accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present application.

Claims

1. A dynamic scraping method, characterized in that, The steps include: S1: Positioning and recognizing the posture of the parts to be grasped based on the deep learning model, and classifying the parts to be grasped into graspable parts or non-graspable parts based on the posture recognition; S2: Calculate the affine transformation matrix based on the camera calibration to transform the camera coordinate system into the robot coordinate system; S3: constructing a part trajectory motion model based on multiple frame images, and realizing prediction of the graspable part trajectory based on the part trajectory motion model; calculating the position and speed of the robot adsorbing the graspable part based on the predicted graspable part trajectory and the preset trigger time; S4: The robot performs a grasping action and obtains feedback information of the grasping action, and adjusts trajectory planning parameters or time compensation values based on the feedback information; and realizes time synchronization between the camera and the robot based on the time compensation value.

2. The dynamic grabbing method according to claim 1, wherein Based on the part pose and target tracking in the multiple frames of images, the trajectory points of a single graspable part are obtained; based on the position change of a single target between adjacent frames and the time interval between adjacent frames, the speed information of the single graspable part is obtained.

3. The dynamic grabbing method according to claim 1, characterized in that Based on the part posture and target tracking in the multiple frames of images, trajectory points of multiple graspable parts are obtained; based on the position changes of multiple targets between adjacent frames and the time interval between adjacent frames, speed information of multiple graspable parts is obtained, and the median speed of multiple graspable parts is obtained as the part speed to improve the grasping accuracy.

4. The dynamic grabbing method according to claim 2 or claim 3, characterized in that The grasping action includes a diagonally downward following stage, a horizontal following stage and a diagonally upward following stage; based on the diagonally downward following stage, the adsorption of the suction cup and the graspable part is achieved; based on the horizontal following stage, the adsorption stability of the suction cup and the graspable part is improved; based on the diagonally upward following stage, the graspable part is safely separated from the material tray.

5. The dynamic grabbing method according to claim 4, wherein The speed component of the robot in the horizontal direction is equal to the speed of the graspable part to improve the grasping accuracy.

6. The dynamic grabbing method according to claim 5, wherein, The feedback information includes part position deviation and adsorption stability; and the trajectory planning parameters or time compensation values are adjusted based on the part position deviation and the adsorption stability.

7. The dynamic grabbing method according to claim 6, wherein The graspable part sends out a trigger signal after reaching the preset position, obtains the current timestamp after receiving the trigger signal, and calculates the time compensation value based on the current timestamp to achieve time synchronization between the camera and the robot.

8. A dynamic grabbing system, which can apply the dynamic grabbing method described in any one of claims 1-7, characterized in that include: A camera module, used for collecting multiple frames of images and performing position and posture recognition on parts in the multiple frames of images; A calculation module, used to construct a part trajectory motion model based on the multiple frames of images, and predict the trajectory of the graspable part according to the part trajectory motion model; and also used to calculate an affine transformation matrix based on camera calibration to convert the camera coordinate system into the robot coordinate system; A robot control module, used to perform a grasping action and obtain feedback information of the grasping action according to the data provided by the calculation module; and also used to adjust trajectory planning parameters or time compensation values based on the feedback information, and realize time synchronization between the camera and the robot based on the time compensation value; A time synchronization module is configured in the host computer and responds immediately upon receiving a trigger signal.

9. A computer device, characterized in that, The computer device includes a memory and a processor coupled to each other, and program data is stored in the memory. The processor is configured to execute the program data to implement the steps of the dynamic scraping method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Program data that can be run by a processor is stored, and the program data is used to implement the steps of the dynamic scraping method according to any one of claims 1-7.

Images