A dynamic scraping method and system

Abstract

The application discloses a dynamic grabbing method and system, and relates to the technical field of automatic manufacturing.The dynamic grabbing method comprises the following steps: S1: positioning and pose recognition of a part to be grabbed based on a deep learning model, and division of the part to be grabbed into a grabbable part or an ungraspable part based on the pose recognition; S2: calculation of an affine transformation matrix based on camera calibration to convert a camera coordinate system into a robot coordinate system; S3: construction of a part trajectory motion model based on multiple images, and realization of trajectory prediction of the graspable part based on the part trajectory motion model; S4: implementation of a grabbing action by a robot and acquisition of feedback information of the grabbing action, adjustment of trajectory planning parameters or a time compensation value based on the feedback information, time synchronization of the camera and the robot based on the time compensation value, and calculation of the best adsorption position and speed based on the predicted trajectory of the graspable part and a preset trigger time, so that the movement of the robot is more efficient and smooth.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automation manufacturing technology, in particular to a dynamic grabbing method and system. BACKGROUND

[0002] With the improvement of industrial automation, there is an increasing demand for automatic identification and grabbing of small parts in a dynamic environment.

[0003] However, due to the small size of the parts, the change of the speed of the conveyor belt, and the different poses that may occur, it is a challenge to accurately and efficiently grab in a dynamic environment. Traditional static grabbing methods are difficult to adapt to such complex situations. SUMMARY

[0004] To solve the above problems, the present application discloses a dynamic grabbing method, which calculates the optimal adsorption position and speed based on the predicted graspable part trajectory and the preset trigger time, so that the motion of the robot is more efficient and smooth. At the same time, a corresponding dynamic grabbing system is proposed to realize the dynamic grabbing method under different conditions.

[0005] The first technical solution adopted by the present application is to provide a dynamic grabbing method, comprising the following steps:

[0006] S1: positioning and pose recognition of the part to be grabbed based on a deep learning model, and dividing the part to be grabbed into graspable parts or ungraspable parts based on the pose recognition;

[0007] S2: calculating an affine transformation matrix based on camera calibration to convert the camera coordinate system to the robot coordinate system;

[0008] S3: constructing a part trajectory motion model based on multiple images, realizing the prediction of the graspable part trajectory based on the part trajectory motion model, and calculating the position and speed of the robot adsorbing the graspable part based on the predicted graspable part trajectory and the preset trigger time;

[0009] S4: the robot implements a grabbing action and obtains feedback information of the grabbing action, adjusts the trajectory planning parameters or time compensation value based on the feedback information, and realizes the time synchronization of the camera and the robot based on the time compensation value.

[0010] Wherein, the trajectory point of a single graspable part is obtained based on the part pose in the multiple images and target tracking; the speed information of a single graspable part is obtained based on the position change of a single target between adjacent frames and the time interval between adjacent frames.

[0011] The trajectory points of the plurality of graspable parts are obtained based on the part poses in the plurality of images and target tracking; the speed information of the plurality of graspable parts is obtained based on the position changes of the plurality of targets between adjacent frames and the time intervals between adjacent frames, and the median value of the speed of the plurality of graspable parts is obtained as the part speed to improve the grasping precision.

[0012] The grasping action includes a diagonal downward following stage, a horizontal following stage and a diagonal upward following stage; the adsorption of the suction disc and the graspable part is realized based on the diagonal downward following stage; the adsorption stability of the suction disc and the graspable part is improved based on the horizontal following stage; the graspable part is safely separated from the tray based on the diagonal upward following stage.

[0013] The speed component of the robot in the horizontal direction is equal to the speed of the graspable part to improve the grasping precision.

[0014] The feedback information includes part position deviation and adsorption stability; the trajectory planning parameter or the time compensation value is adjusted based on the part position deviation and the adsorption stability.

[0015] The trigger signal is sent after the graspable part reaches the preset position, the current timestamp is obtained after the trigger signal is received, and the time compensation value is calculated based on the current timestamp to realize the time synchronization of the camera and the robot.

[0016] The second technical solution adopted in the application is to provide a dynamic grasping system, which can apply the dynamic grasping method as described in any of the above, comprising:

[0017] A camera module is configured to collect a plurality of images and identify the poses of parts in the plurality of images.

[0018] A calculation module is configured to construct a part trajectory motion model based on the plurality of images, and predict the trajectory of the graspable part according to the part trajectory motion model; and calculate an affine transformation matrix based on camera calibration to convert the camera coordinate system into the robot coordinate system.

[0019] A robot control module is configured to execute a grasping action and obtain feedback information of the grasping action according to the data provided by the calculation module; and adjust the trajectory planning parameter or the time compensation value based on the feedback information, and realize the time synchronization of the camera and the robot based on the time compensation value.

[0020] A time synchronization module is configured in the upper computer and responds immediately when receiving the trigger signal.

[0021] The third technical solution adopted by the present application is to provide an electronic device, comprising a memory and a processor coupled with each other, wherein the processor is configured to execute program instructions stored in the memory to implement the steps of the dynamic grabbing method according to any one of the above.

[0022] The fourth technical solution adopted by the present application is to provide a computer readable storage medium, which stores program data, wherein the program data can be executed by a processor to implement the steps of the dynamic grabbing method according to any one of the above.

[0023] Compared with the prior art, the present application has at least one of the following beneficial effects:

[0024] 1. The optimal suction position and speed calculated by the predicted graspable part trajectory and the preset trigger time make the robot motion more efficient and smooth.

[0025] 2. The deep learning model is used to locate and recognize the pose of the part to be grabbed, and the part to be grabbed is classified into graspable or ungraspable categories, so that only the parts suitable for grabbing will be tried to be sucked by the robot.

[0026] 3. The feedback information is used to adjust the trajectory planning parameters or time compensation value, so as to realize the time synchronization between the camera and the robot, and reduce the operation error caused by the time asynchronization.

[0027] 4. When the grabbing action is performed, three straight line motions are adopted, which not only helps the suction cup to firmly suck the part, but also effectively avoids the risk of collision with the tray or other obstacles. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0029] Among them:

[0030] Figure 1 The flowchart of an embodiment of the dynamic grabbing method provided by the present application is shown in the figure;

[0031] Figure 2 The flowchart of an embodiment of the present application based on single frame image to obtain graspable part information is shown in the figure;

[0032] Figure 3 The flowchart of an embodiment of the camera calibration provided by the present application is shown in the figure;

[0033] Figure 4 A flowchart of an embodiment of acquiring a robot motion trajectory for the present application is shown in FIG. 1.

[0034] Figure 5 An acquisition diagram of an embodiment of providing a single-frame multi-part picture speed median for the present application is shown in FIG. 2.

[0035] Figure 6 An acquisition diagram of an embodiment of providing a multi-frame multi-part picture speed median for the present application is shown in FIG. 3.

[0036] Figure 7 A diagram of an embodiment of a dynamic grabbing whole process for the present application is shown in FIG. 4.

[0037] Figure 8 A framework diagram of an embodiment of a dynamic grabbing system for the present application is shown in FIG. 5.

[0038] Figure 9 A structural diagram of an embodiment of a computer device for the present application is shown in FIG. 6.

[0039] Figure 10 A structural diagram of an embodiment of a computer readable storage medium for the present application is shown in FIG. 7. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It can be understood that the specific embodiments described herein are only used to explain the present application, and not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, and not all the structures. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0041] The terms "first", "second", and the like in the present application are used to distinguish different objects, and are not used to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product, or device.

[0042] Reference to“an embodiment” herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase“in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all

[0043] In the process of handling small parts with high-speed motion, the existing dynamic grabbing technology often fails to meet the required high-precision requirements. For example, in the case of a fast conveyor belt, the prediction of the position of the part and the selection of the grabbing timing may deviate, resulting in a failed grab. The dynamic grabbing method of the present application calculates the optimal adsorption position and speed based on the predicted trajectory of the grabbable part and the preset trigger time, making the robot's motion more efficient and smooth. For example, Figure 1 as shown in FIG. 1, Figure 1 The flowchart of an embodiment of the dynamic grabbing method provided by the present application includes the following steps:

[0044] S1: Positioning and pose recognition of the part to be grabbed based on a deep learning model, and dividing the part to be grabbed into a grabbable part or a non-grabable part based on the pose recognition; the data set used to train the deep learning model includes part images in different poses; the images are labeled to determine whether they are grabbable parts and their accurate positions and directions.

[0045] As shown in FIG. 2, Figure 2 as shown in FIG. 3, Figure 2 The flowchart of an embodiment of the present application for obtaining information of a grabbable part based on a single frame of image.

[0046] A frame of image is obtained as input, and a YOLOv8 segmentation model is used to analyze the frame of image, identify and classify all parts in the image, and determine whether they are front-facing or back-facing; in this embodiment, the part facing the front is a grabbable part; if the back of the part is detected to be facing up, the robot can be controlled to flip the part and then grab it.

[0047] It should be noted that in this embodiment, the front and back of the part are used to determine whether the part is a grabbable part, and in other embodiments, the material or shape of the part can be used to determine whether it is a grabbable part, which is not limited in this regard.

[0048] If a part facing the front or back is detected, the target tracking stage is entered. Through the target tracking function of YOLOv8, the system can track the movement of these parts between consecutive frames, helping to predict their future positions and directions; if no part facing the front or back is detected, the YOLOv8 segmentation model is returned to continue analysis of the next frame.

[0049] In the multi-target case, the relative distances between all tracked targets are calculated, and the target that is farthest from the other targets is obtained; the part with high outlying degree is preferentially selected for grasping, facilitating the operation of the robot and reducing the risk of collision with other parts.

[0050] For the selected target part, a SAM2 large model is used for more accurate segmentation. SAM2 can provide higher accuracy than YOLOv8, especially for cases that require very accurate edge information, ensuring that the selection of the grasping point is more accurate; according to the segmentation result provided by SAM2, the centroid position of the part contour is calculated, which will be used as the grasping center point of the robot, ensuring that the suction cup can be accurately attached to the best position of the part; by calculating the minimum circumscribed rectangle of the part contour, the main axial direction of the part can be obtained, and then the robot can adjust its grasping posture to match the direction of the part, improving the stability of the suction.

[0051] As shown in Figure 3 , the flowchart of an embodiment of camera calibration provided by the present application is shown. Figure 3

[0052] S2: Calculate the affine transformation matrix based on camera calibration to convert the camera coordinate system to the robot coordinate system; the affine transformation matrix obtained through the accurate camera calibration process can realize high-precision mapping from the camera coordinate system to the robot coordinate system. This means that the target position detected by the vision system can be accurately converted to the actual position where the robot needs to perform operations, greatly improving the success rate and accuracy of grasping or operation tasks.

[0053] The process of obtaining the camera-robot affine matrix is described in detail below:

[0054] The camera captures the scene containing the standard checkerboard calibration board, and multiple images can be taken from multiple angles and positions. All corner points of the checkerboard in each image are detected, and the positions of the corner points represent the coordinates in the camera coordinate system.

[0055] If all the necessary corner points are successfully identified, the robot's teaching function is used to let the mechanical arm touch these corner points one by one, and the robot plane coordinates of each corner point are recorded; if the corner points cannot be successfully identified, the image is reacquired by adjusting the angle of the camera or the lighting conditions to obtain a clearer image.

[0056] ​The root mean square error (RMSE) is calculated based on the corner coordinates in the camera coordinate system and the corresponding robot planar coordinates to evaluate the transformation accuracy. The accuracy of the affine transformation matrix is ​​judged by comparing the RMSE with a preset threshold. If the RMSE is less than the preset threshold, the current affine transformation matrix is ​​used to realize the operation of transforming from the camera coordinate system to the robot coordinate system. If the RMSE is greater than or equal to the preset threshold, the robot re-performs touch teaching, and the robot re-acquires the affine transformation matrix based on increasing the number of samples or optimizing the touch accuracy.

[0057] It should be clarified that in this embodiment, the preset threshold value is 0.2; in other embodiments, the preset threshold value can be other values, and there is no limitation on this.

[0058] like Figure 4 As shown, Figure 4 This is a schematic diagram illustrating an embodiment of obtaining a robot motion trajectory according to this application.

[0059] S3: A part trajectory motion model is constructed based on multi-frame images, and the trajectory of the graspable part is predicted based on this model. By analyzing the part's pose changes in multi-frame images, the future motion trajectory of the part can be predicted more accurately. Based on the predicted graspable part trajectory and a preset trigger time, the position and speed at which the robot adsorbs the graspable part are calculated. Using the predicted part trajectory and the preset trigger time, the optimal path for the robot to adsorb the part, including its position and speed, can be calculated in advance. This not only reduces unnecessary movement and waiting time but also improves the overall system efficiency.

[0060] The following describes in detail the process of obtaining the robot's motion trajectory in this embodiment:

[0061] Based on the part pose (t,x,y,z,u) in multiple frames of images, the trajectory points [(t,x,y,z,u),...] of a single part between consecutive frames are obtained through target tracking technology; where t is time, x represents the horizontal axis coordinate, y represents the vertical axis coordinate, z represents the vertical axis coordinate, and u represents the rotation angle or posture information of the part.

[0062] The part trajectory motion model uses a first-order linear regression model f(t,x,y,z,u) to fit the trajectory points in order to predict the future motion path of the part. It should be noted that in this embodiment, the part is located on the conveyor belt and maintains stable linear motion. In other embodiments, different motion models can be selected for trajectory prediction based on different part motion states, and no limitation is made in this regard.

[0063] The robot's path is calculated based on the preset trigger time and part trajectory, that is, the robot's path is calculated based on the robot's action time point and part trajectory; the robot's movement speed is obtained based on the robot path point reversible operation.

[0064] Due to the inherent time difference between the actual robot execution timeline and the camera timeline, there is a problem of time desynchronization in the first attempt to grasp; fine-tune the time synchronization based on the horizontal following position, and output the robot's motion trajectory and speed after aligning the timelines.

[0065] S4: The robot implements the grasping action and obtains feedback information of the grasping action, adjusts the trajectory planning parameters or time compensation value based on the feedback information; after completing the trajectory planning, the robot starts to implement the grasping action according to the calculated path points and speed.

[0066] The grasping action includes moving to a specified position, adjusting the posture, and finally absorbing the part; during the grasping process or after the grasping is completed, the system will collect relevant feedback information; based on the feedback information, it is determined whether there is an error in the current trajectory planning or time synchronization, and if there is an error, the trajectory planning parameters or time synchronization value is adjusted; the trajectory planning parameters include the position of the path point, the motion speed and the acceleration; based on the time compensation value, the time synchronization of the camera and the robot is realized; based on the adjusted time compensation value, the time axes of the camera and the robot are recalibrated to eliminate the time difference between the two; the time synchronization is realized in the trial grasping process in step S3.

[0067] It should be noted that in the present embodiment, a single adjustment is to adjust the trajectory planning parameters or the time compensation value; in other embodiments, a single adjustment can adjust the trajectory planning parameters and the time compensation value at the same time, and no limitation is made to this.

[0068] By obtaining the feedback information of the grasping action in real time, the system can quickly identify and correct the deviation in the grasping process. For example, if the position of the part does not match the prediction, the grasping path can be corrected by adjusting the trajectory planning parameters, thereby significantly improving the success rate and accuracy of grasping; the feedback mechanism can avoid invalid grasping attempts caused by incorrect trajectory planning or time desynchronization, reduce unnecessary mechanical arm movement and energy consumption, and improve overall production efficiency.

[0069] In summary, the dynamic grasping method of this embodiment includes the following steps: S1: Based on a deep learning model, the part to be grasped is located and its pose is identified. Based on the pose identification, the part to be grasped is divided into graspable parts or non-graspable parts; S2: Based on camera calibration, an affine transformation matrix is ​​calculated to convert the camera coordinate system into the robot coordinate system; S3: Based on multiple frames of images, a part trajectory motion model is constructed. Based on the part trajectory motion model, the trajectory of the graspable part is predicted. Based on the predicted trajectory of the graspable part and a preset trigger time, the position and speed at which the robot adsorbs the graspable part are calculated; S4: The robot performs the grasping action and obtains feedback information of the grasping action. Based on the feedback information, the trajectory planning parameters or time compensation value are adjusted. Based on the time compensation value, the time synchronization between the camera and the robot is achieved. The optimal adsorption position and speed calculated by the predicted trajectory of the graspable part and the preset trigger time make the robot's movement more efficient and smooth.

[0070] In one embodiment, the trajectory points of a single graspable part are obtained based on the part's pose and target tracking in multiple frames of images; the velocity information of a single graspable part is obtained based on the positional changes of a single target between adjacent frames and the time interval between adjacent frames; by accurately obtaining the trajectory points and velocity information of the part, the future motion state of the part can be predicted more accurately, thereby guiding the robot to grasp at the optimal time and position, significantly improving the success rate and accuracy of grasping; this method can update the motion state of the part in real time and is suitable for handling the rapid movement of parts in dynamic environments such as conveyor belts. Even if the part's velocity changes or nonlinear motion patterns occur, it can adapt by adjusting the tracking algorithm.

[0071] In one embodiment, trajectory points of multiple graspable parts are obtained based on part poses and target tracking in multiple frames of images; velocity information of multiple graspable parts is obtained based on position changes of multiple targets between adjacent frames and time intervals between adjacent frames, and the median velocity of multiple graspable parts is obtained as the part velocity to improve grasping accuracy; by calculating the median velocity of multiple graspable parts, errors caused by abnormal movement of individual parts can be effectively reduced. This method can provide a more stable and reliable average velocity estimate, thereby improving the overall grasping accuracy. Using the median velocity can help predict the future position of the part more accurately, which helps the robot better plan its grasping path and ensure grasping at the correct time and position.

[0072] like Figures 5-6 As shown, Figure 5 A schematic diagram illustrating an embodiment of obtaining the median velocity of a single-frame multi-part image provided in this application; Figure 6 This is a schematic diagram illustrating an embodiment of obtaining the median velocity of multi-frame, multi-part images provided in this application.

[0073] The median velocity of multiple parts in a single frame is

[0074] When grabbing multiple times at the same speed, the median value among the multiple speeds can be taken as the part's movement speed, i.e., the part's movement speed. V5 is the median velocity of the first frame, V6 is the median velocity of the second frame, and V7 is the median velocity of the third frame.

[0075] It should be clarified that in this embodiment, the trajectory of the part is predicted by multiple frames of images and the timestamps of the images; in other embodiments, the trajectory of the part can also be predicted by a single frame of image + pre-calibrated conveyor belt direction and speed or a single frame of image + conveyor belt encoder, and there is no limitation on this.

[0076] In one embodiment, the grasping action includes a downward following phase, a horizontal following phase, and an upward following phase. The workflow of the downward following phase, the horizontal following phase, and the upward following phase is described in detail below:

[0077] The suction cup adheres to the graspable part through the downward following phase. In the downward following phase, the robot slides down at a certain angle to approach the part. The design based on the downward following phase can ensure that the suction cup can make stable contact with the surface of the part, avoiding impact or unstable contact caused by vertical descent.

[0078] The horizontal following phase improves the adhesion stability between the suction cup and the graspable part; during the horizontal following phase, the robot maintains horizontal movement relative to the part, allowing the suction cup to form a stronger seal on the part surface, thereby enhancing the adhesion force.

[0079] The upward following phase allows the grippable parts to be safely removed from the tray; the upward following phase allows the parts to leave the tray or other support surfaces at a relatively gentle angle, reducing the deformation or damage to the parts that may be caused by sudden pulling.

[0080] like Figure 7 As shown, Figure 7 A schematic diagram of an embodiment of the dynamic data capture process provided in this application includes the following steps:

[0081] After completing the trajectory calculation, the robot gripper moves to the GP0003 gripping trigger starting point and waits for the preset trigger time T0 to be triggered.

[0082] The host computer activates the trigger mode and sends a command to the robot to continue the program when the trigger timestamp is reached.

[0083] Maintaining the horizontal position of the part, follow the downward slope for time t1.

[0084] Maintain the horizontal position of the part and keep it stationary, following the horizontal movement for time t2.

[0085] Keeping the part and the horizontal position of the part stationary, the oblique upward following t3 time, the disengagement height of the material disc.

[0086] Based on the trajectory motion model of the part, the xy point position of the part at T0 time, and the xy positions at T0+t1, T0+t1+t2, T0+t1+t2+t3 time can be calculated, combined with the preset safety height, grabbing height, disengagement height, the GP0003, GP0004, GP0005, GP0006 point positions can be obtained. Combined with the predicted arrival time, the reverse calculation can obtain the robot speed in the oblique downward following stage, the horizontal following stage and the oblique upward following stage.

[0087] The horizontal velocity component of the robot is equal to the velocity of the graspable part to improve the grabbing accuracy; when the horizontal velocity of the robot is equal to the velocity of the part, the position error caused by the relative velocity difference can be significantly reduced. This "follow-up" state enables the suction cup to be more stably adsorbed on the surface of the part, thereby improving the success rate and accuracy of grabbing.

[0088] The part attached to the material disc moves with the conveyor belt, and if the horizontal velocity component of the robot is not equal to the velocity of the graspable part, it is easy to cause the robot gripper to collide with the material disc.

[0089] In an embodiment, the feedback information includes part position deviation and adsorption stability; the part position deviation is the difference between the actual grabbing position and the preset target position, and the adsorption force of the suction cup on the part is monitored through a sensor to determine whether the adsorption is stable; the sensor can be a torque sensor, a pressure sensor or any other sensor that can achieve the same technical effect, and no limitation is made on this.

[0090] Adjust the trajectory planning parameters or time compensation value based on the part position deviation and adsorption stability; if the part position deviation is found to be large, the target path of the robot can be corrected by fine-tuning the coordinates of the robot; adjust the motion speed and / or acceleration of the robot based on the size of the part position deviation to ensure that the target position can be reached smoothly without being too fast to cause new deviations.

[0091] If the adsorption stability is insufficient, it indicates that there is a problem with the speed or angle of the robot when it contacts the part. At this time, the time synchronization between the robot and the camera can be fine-tuned by adjusting the time compensation value to ensure that the visual detection result and the mechanical operation are more coordinated and consistent. In other embodiments, the duration of the horizontal following stage can be appropriately extended to improve the adsorption stability.

[0092] In an embodiment, a trigger signal is sent when the graspable part reaches the preset position, a current timestamp is obtained after receiving the trigger signal, and a time compensation value is calculated based on the current timestamp to achieve time synchronization between the camera and the robot; by calculating and applying the time compensation value in real time, the accumulation of errors caused by inherent system delays can be effectively reduced, making the entire grasping process more stable and reliable; the ability to dynamically adjust the time compensation value enables the system to adapt to different working environments and changes in conditions, whether it is changes in production line speed or external interference factors, and can respond flexibly.

[0093] The application also provides a dynamic grasping system, which can apply the dynamic grasping method as described in any of the above embodiments, such as Figure 8 as shown, Figure 8 The framework diagram of an embodiment of the dynamic grasping system provided by the application includes:

[0094] A camera module for capturing multiple images and recognizing the pose of the parts in the multiple images;

[0095] A calculation module for constructing a part trajectory motion model based on the multiple images and predicting the trajectory of the graspable part according to the part trajectory motion model; and for calculating an affine transformation matrix based on camera calibration to convert the camera coordinate system to the robot coordinate system;

[0096] A robot control module for executing a grasping action and obtaining feedback information of the grasping action according to the data provided by the calculation module; and for adjusting the trajectory planning parameters or the time compensation value based on the feedback information, and achieving time synchronization between the camera and the robot based on the time compensation value;

[0097] A time synchronization module, which is configured in the upper computer and responds immediately when receiving the trigger signal.

[0098] The working process of the dynamic grasping system is described in detail as follows:

[0099] The camera module continuously captures multiple images, and uses a deep learning model to recognize and classify the pose of the parts; then, the calculation module constructs a part trajectory motion model based on these images, predicts the future position of the part, and calculates an affine transformation matrix through camera calibration to achieve coordinate system conversion; subsequently, the robot control module plans and executes a grasping action according to the calculation results, while obtaining real-time grasping feedback information for adjusting the trajectory planning parameters or the time compensation value; the time synchronization module records the current timestamp and calculates the time compensation value when it receives the trigger signal that the part has reached the preset position, ensuring time synchronization between the camera and the robot, thereby achieving efficient and accurate part grasping; through the close cooperation between the modules, the entire process realizes automatic and intelligent dynamic grasping operation.

[0100] For the above-mentioned embodiments, the present application provides a computer device, please refer to Figure 9 , Figure 9 is a structural schematic diagram of an embodiment of the computer device of the present application. The computer device comprises a memory and a processor, wherein the memory and the processor are coupled to each other, the memory stores program data, and the processor is configured to execute the program data to realize the steps of any embodiment of the dynamic crawling method.

[0101] In the embodiment, the processor can also be referred to as a CPU (Central Processing Unit). The processor can be an integrated circuit chip having a processing capability of signals. The processor can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0102] For the method of the above-mentioned embodiments, it can be realized in the form of a computer program, and therefore the present application provides a computer readable storage medium, please refer to Figure 10 , Figure 10 is a structural schematic diagram of an embodiment of the computer readable storage medium of the present application. The computer readable storage medium stores program data capable of being executed by a processor, and the program data can be executed by the processor to realize the steps of any embodiment of the dynamic crawling method.

[0103] The computer readable storage medium of the embodiment can be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc. which can store program data, or it can also be a server storing the program data, which can send the stored program data to other devices for running, or it can also run the stored program data by itself.

[0104] In the several embodiments provided by the present application, it should be understood that the disclosed method and device can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the modules or units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0105] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment scheme.

[0106] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0107] The above is only an embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent flow transformation using the content of the present application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A dynamic scraping method, characterized in that, Comprising the following steps: S1: positioning and pose recognition of the part to be grabbed based on a deep learning model, dividing the part to be grabbed into a graspable part or an ungraspable part based on the pose recognition; S2: calculating an affine transformation matrix based on camera calibration to convert the camera coordinate system into the robot coordinate system; S3: constructing a part trajectory motion model based on multiple images, predicting the trajectory of the graspable part based on the part trajectory motion model, and calculating the position and speed of the robot to adsorb the graspable part based on the predicted trajectory of the graspable part and the preset trigger time; S4: the robot implements the grabbing action and obtains the feedback information of the grabbing action, adjusts the trajectory planning parameters or the time compensation value based on the feedback information, and realizes the time synchronization of the camera and the robot based on the time compensation value; the grabbing action includes a diagonal downward following stage, a horizontal following stage and a diagonal upward following stage; the adsorption of the suction cup and the graspable part is realized based on the diagonal downward following stage; the adsorption stability of the suction cup and the graspable part is improved based on the horizontal following stage; the duration of the horizontal following stage can be extended; the graspable part is safely separated from the tray based on the diagonal upward following stage.

2. The dynamic grasping method according to claim 1, wherein, The trajectory points of a single graspable part are obtained based on the part pose and target tracking in the multiple images; the speed information of a single graspable part is obtained based on the position change of a single target between adjacent frames and the time interval between adjacent frames.

3. The dynamic grasping method of claim 1, wherein, The trajectory points of multiple graspable parts are obtained based on the part pose and target tracking in the multiple images; the speed information of multiple graspable parts is obtained based on the position change of multiple targets between adjacent frames and the time interval between adjacent frames, and the median value of the speed of multiple graspable parts is taken as the part speed to improve the grabbing precision.

4. The dynamic grasping method according to claim 2 or claim 3, characterized in that, The speed component of the robot in the horizontal direction is equal to the speed of the graspable part to improve the grabbing precision.

5. The dynamic grasping method according to claim 4, wherein, The feedback information includes part position deviation and adsorption stability; the trajectory planning parameters or the time compensation value are adjusted based on the part position deviation and the adsorption stability.

6. The dynamic grasping method according to claim 5, wherein, The graspable part sends a trigger signal after reaching a preset position, a current timestamp is obtained after receiving the trigger signal, and a time compensation value is calculated based on the current timestamp to realize the time synchronization of the camera and the robot.

7. A dynamic picking system, to which the dynamic picking method according to any one of claims 1 to 6 can be applied, characterized in that Comprising: A camera module for collecting multiple images and recognizing the pose of the parts in the multiple images; A calculation module for constructing a part trajectory motion model based on the multiple images, and predicting the trajectory of a graspable part according to the part trajectory motion model; and for calculating an affine transformation matrix based on camera calibration to convert the camera coordinate system into the robot coordinate system; A robot control module for executing a grabbing action and obtaining the feedback information of the grabbing action according to the data provided by the calculation module; and for adjusting the trajectory planning parameters or the time compensation value based on the feedback information, and realizing the time synchronization of the camera and the robot based on the time compensation value; A time synchronization module configured in the upper computer, which responds immediately when receiving a trigger signal.

8. A computer device, comprising: The computer device comprises a memory and a processor coupled to each other, the memory stores program data, and the processor is configured to execute the program data to implement the steps of the dynamic crawling method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer device comprises a memory and a processor coupled to each other, the memory stores program data, and the processor is configured to execute the program data to implement the steps of the dynamic crawling method according to any one of claims 1-6.

Images