A mobile workpiece dynamic grabbing method and system based on visual driving

Abstract

The application discloses a kind of mobile workpiece dynamic grabbing method and system based on vision driving, belong to intelligent manufacturing technical field, can solve the problem that existing technology cannot adapt to the production demand of quick change production, it is difficult to be applicable to high beat production line.The method comprises the following steps: S1, the datum image of workpiece is located at the preset trigger position on the production line and is in the preset pose, the datum grabbing pose of robot is determined according to the preset pose;S2, the real-time position of workpiece is obtained during conveying on the production line, when the real-time position matches the preset trigger position, the acquisition signal is sent to the image acquisition device using hardware triggering method, and the real-time image of workpiece collected by the image acquisition device according to the acquisition signal is received;S3, the pixel pose deviation of workpiece in image coordinate system is determined according to the datum image and the real-time image;S4, the robot is controlled to grab workpiece according to the datum grabbing pose and the pixel pose deviation.The application is used for grabbing mobile workpiece.

Description

Technical Field

[0001] This invention relates to a vision-driven method and system for dynamic grasping of moving workpieces, belonging to the field of intelligent manufacturing technology. Background Technology

[0002] With the development of industrial automation, robot dynamic grasping technology is gradually being applied to scenarios such as sorting and assembly. Unlike static grasping, dynamic grasping presents challenges to accurately calculating the grasping pose because the workpiece moves continuously on the production line.

[0003] Existing technologies are mostly based on building deep learning models using object detection and tracking algorithms, and then using deep learning models to perform visual servoing on moving workpieces, thereby achieving dynamic grasping.

[0004] However, relying on deep learning models for dynamic grasping requires pre-collecting a large amount of data for specific workpieces and pre-training the deep learning model. Therefore, each new workpiece necessitates re-collecting data, retraining the model, and re-deploying the system. In 24 / 7 production scenarios with multiple product categories, this leads to slow task response and insufficient flexibility, making existing technologies unsuitable for rapid production changeovers. Furthermore, the deep learning model's processing chain, from image acquisition and deep learning inference to complex tracking calculations, is computationally heavy and has high latency. On high-speed, high-paced production lines, this can easily result in missed grasps due to insufficient processing speed, making existing technologies unsuitable for high-paced production lines. Additionally, deep learning models based on past experience are only applicable to the stable scenarios they were trained on, and are ill-suited to sudden changes in workpiece speed or severe vibrations, resulting in limited versatility. Summary of the Invention

[0005] This invention provides a vision-driven method and system for dynamic grasping of moving workpieces, which can solve the problems that existing technologies cannot adapt to the production needs of rapid changeover, are difficult to apply to high-cycle production lines, and have weak versatility.

[0006] On one hand, the present invention provides a vision-driven dynamic grasping method for moving workpieces, the method comprising: S1. Obtain a reference image of the workpiece when it is located at a preset trigger position on the production line and in a preset pose, and determine the robot's reference grasping pose based on the preset pose. S2. Obtain the real-time position of the workpiece being conveyed on the production line. When the real-time position matches the preset trigger position, send a collection signal to the image acquisition device using a hardware triggering method, and receive the real-time image of the workpiece collected by the image acquisition device according to the collection signal. S3. Determine the pixel pose deviation of the workpiece in the image coordinate system based on the reference image and the real-time image; S4. Control the robot to grasp the workpiece according to the reference grasping pose and the pixel pose deviation.

[0007] Optionally, S3 specifically includes: S31. Determine the reference feature region corresponding to the workpiece in the reference image, and determine the real-time feature region corresponding to the workpiece in the real-time image; S32. Determine the pixel pose deviation of the workpiece in the image coordinate system based on the reference feature region and the real-time feature region.

[0008] Optionally, determining the real-time feature region corresponding to the workpiece in the real-time image in step S31 specifically includes: A reference template for the workpiece is generated based on the reference feature region; The real-time feature region corresponding to the workpiece is determined in the real-time image based on the reference template.

[0009] Optionally, determining the real-time feature region corresponding to the workpiece in the real-time image based on the reference template specifically includes: Determine the delay time between sending the acquisition signal and acquiring the real-time image, and define a search area in the real-time image based on the delay time; The real-time feature region corresponding to the workpiece is determined in the search area based on the reference template.

[0010] Optionally, determining the real-time feature region corresponding to the workpiece specifically includes: The real-time feature region corresponding to the workpiece is determined using a template matching algorithm.

[0011] Optionally, S4 specifically includes: S41. Perform coordinate system transformation on the pixel pose deviation to obtain the spatial pose deviation of the workpiece in the robot coordinate system; S42. Control the robot to grasp the workpiece according to the reference grasping pose and the spatial pose deviation.

[0012] Optionally, S42 specifically includes: The robot's real-time grasping pose is determined based on the reference grasping pose and the spatial pose deviation; The robot is controlled to grasp the workpiece based on the real-time grasping pose.

[0013] Optionally, controlling the robot to grasp the workpiece based on the real-time grasping pose specifically includes: The robot's grasping trajectory is determined based on the real-time grasping pose; The robot is controlled to grasp the workpiece according to the grasping trajectory.

[0014] Optionally, controlling the robot to grasp the workpiece according to the grasping trajectory specifically includes: Based on the conveying speed of the production line, the movement speed of the robot when it is less than a preset distance from the workpiece is determined, and the velocity component of the movement speed along the conveying speed direction is equal to the conveying speed. The robot is controlled to grasp the workpiece based on the grasping trajectory and the movement speed.

[0015] On the other hand, the present invention provides a vision-driven dynamic grasping system for moving workpieces, the system comprising: The reference acquisition module is used to acquire a reference image of the workpiece when it is located at a preset trigger position on the production line and in a preset pose, and to determine the robot's reference grasping pose based on the preset pose. The real-time acquisition module is used to acquire the real-time position of the workpiece as it is conveyed on the production line. When the real-time position matches the preset trigger position, it sends an acquisition signal to the image acquisition device using a hardware triggering method and receives the real-time image of the workpiece acquired by the image acquisition device based on the acquisition signal. The deviation determination module is used to determine the pixel pose deviation of the workpiece in the image coordinate system based on the reference image and the real-time image. The gripping module is used to control the robot to grip the workpiece based on the reference gripping pose and pixel pose deviation.

[0016] The beneficial effects that this invention can produce include: This invention eliminates the reliance on deep learning models, requiring no large-scale data collection and model training. For new workpieces, a baseline template can be created with just one "photo-teaching" session, a significantly shorter timeframe compared to existing technologies. This enables rapid deployment, adapting to 24 / 7 production demands involving multiple product categories and small batches, achieving a highly flexible "produce on demand, grab on change" production model. Furthermore, this invention utilizes a hardware-triggered acquisition method, drastically reducing the latency between sending the acquisition signal and acquiring the real-time image. This significantly minimizes the random latency caused by software communication, ensuring high synchronization between the signal transmission and real-time image acquisition. This greatly reduces workpiece position deviation in the baseline image, facilitating accurate and rapid determination of the workpiece position in the real-time image with relatively low computational power. Consequently, the computational complexity of this invention is far lower than that of deep learning models, with processing time down to the millisecond level, and a response speed far exceeding that of deep learning models, meeting the dynamic grabbing requirements of high-paced production lines. Furthermore, this invention employs a "relative offset" calculation strategy, so the final grasping accuracy does not depend on the absolute positioning accuracy, but rather on the relative positioning repetition accuracy of template matching. Under the premise of feature stability, sub-pixel level accuracy can be achieved, and it is more robust to slow changes in overall illumination. Moreover, the stability and reliability of positioning are far superior to pure vision technology solutions that rely on absolute image coordinate calculation. Attached Figure Description

[0017] Figure 1 A flowchart of a vision-driven dynamic grasping method for moving workpieces provided in an embodiment of the present invention; Figure 2 This is a top-view schematic diagram of a grasping scenario provided in an embodiment of the present invention.

[0018] Figure label: 1. Upper production line; 2. Lower production line; 3. Pallet; 4. Target workpiece. Detailed Implementation

[0019] The present invention will now be described in detail with reference to the embodiments, but the present invention is not limited to these embodiments.

[0020] This invention provides a vision-driven method for dynamically grasping moving workpieces, such as... Figure 1 As shown, the method includes: S1. Obtain a reference image of the workpiece when it is located at a preset trigger position on the production line and in a preset pose, and determine the robot's reference grasping pose based on the preset pose.

[0021] Specifically, the preset trigger position refers to a fixed position on the production line, which can be flexibly set according to the robot's response time and action time. For example, such as... Figure 2 As shown, the preset trigger position is Figure 2The position corresponding to the medium-thick dashed line is perpendicular to the conveying direction of the production line.

[0022] The preset position of the workpiece includes a preset placement position of the workpiece along the conveying direction perpendicular to the production line, and a preset placement posture of the workpiece at the preset placement position. For example, such as... Figure 2 As shown, this embodiment defines Figure 2 The target workpiece 4 on the upper production line 1 is in a preset position. Figure 2 On the upper production line 1, the target workpiece 4 is pre-positioned at the center line of the production line, while... Figure 2 On the lower production line 2, the target workpiece 4 is positioned off-center from the center line of the production line, indicating that the target workpiece 4 on the lower production line 2 has deviated from its preset placement position. Furthermore, referring to the placement posture of the pallet 3, it can be seen that the target workpiece 4 on the lower production line 2 has also deviated from its preset placement posture.

[0023] By acquiring a reference image of the workpiece at a preset trigger position and in a preset pose on the production line, a reference benchmark can be provided for the robot to grasp the workpiece, making it easier for the robot to accurately grasp the workpiece in subsequent processes.

[0024] In this embodiment, S1 specifically includes: First, hand-eye calibration is performed to determine the pose transformation relationship between the camera coordinate system of the image acquisition device and the robot coordinate system. Then, the image acquisition device acquires a reference image of the workpiece located at a preset trigger position and in a preset pose on the production line. Finally, based on the pose transformation relationship, the preset pose of the workpiece in the reference image is converted into the robot's reference grasping pose in the robot coordinate system. Adopting this reference grasping pose ensures successful grasping of the workpiece in that preset pose during subsequent grasping processes.

[0025] Through the above process, when a new workpiece is introduced to the production line for the first time, this embodiment does not require large-scale data acquisition and model training. It only needs to use an image acquisition device to photograph the workpiece located at a preset trigger position and in a preset pose on the production line to obtain a reference image of the workpiece. Simultaneously, based on the pose transformation relationship, the reference gripping pose of the robot's end effector (such as a gripper or suction cup) is determined, thus clearly defining the precise gripping relationship between the robot's end effector and the workpiece in the preset pose. Since the above process takes very little time (usually only a few minutes), unlike existing technologies that require a lengthy model training process, this embodiment can achieve instant acquisition of precise gripping relationships, achieving a "learn and use" technical effect. In practical applications, a teach pendant can be used to guide the image acquisition device to take pictures when acquiring the reference image, thereby improving the quality of the reference image.

[0026] S2. Obtain the real-time position of the workpiece being conveyed on the production line. When the real-time position matches the preset trigger position, send a collection signal to the image acquisition device using a hardware triggering method, and receive the real-time image of the workpiece collected by the image acquisition device based on the collection signal.

[0027] This embodiment utilizes positioning sensors, rotary encoders, and other devices to acquire the real-time position of the workpiece as it is conveyed on the production line. Taking a rotary encoder as an example, when the real-time position of the workpiece reaches a preset trigger position, the rotary encoder immediately sends a hardware-triggered acquisition signal to an image acquisition device. The image acquisition device immediately acquires a real-time image of the workpiece upon receiving the acquisition signal. Using a hardware-triggered method to trigger the acquisition signal significantly shortens the delay time between sending the acquisition signal and acquiring the real-time image, especially mitigating the random delay caused by software communication. This ensures high synchronization between the transmission time of the acquisition signal and the acquisition time of the real-time image, thereby significantly reducing the workpiece position deviation in the reference image.

[0028] S3. Determine the pixel pose deviation of the workpiece in the image coordinate system based on the reference image and the real-time image. Specifically, this includes: S31. Determine the reference feature region corresponding to the workpiece in the reference image, and determine the real-time feature region corresponding to the workpiece in the real-time image, including: 1) Extract the feature regions that can characterize the stable and unique features on the workpiece from the reference image, such as contours, specific marker points, textures, etc., to obtain the reference feature regions; 2) Generate a reference template for the workpiece based on the reference feature area; 3) Determine the time delay between sending the acquisition signal and acquiring the real-time image, and delineate the search area in the real-time image based on the time delay; Specifically, since the workpiece continues to move with the production line during the delay time, it will inevitably have a slight offset relative to the preset trigger position when the real-time image is acquired. The offset distance of the workpiece relative to the preset trigger position at the moment of image acquisition can be calculated based on the delay time and the conveying speed of the production line. Furthermore, by superimposing the offset distance of the workpiece in the conveying direction of the production line onto the preset placement position of the workpiece in the reference image, the search area can be defined in the real-time image.

[0029] By defining a search area, the location of the workpiece in the real-time image can be accurately determined. This eliminates the need for full-image processing in subsequent steps; only the search area needs to be processed, saving computational resources and improving efficiency. Furthermore, since this embodiment uses a hardware-triggered signal transmission method with extremely short latency, the defined search area closely approximates the workpiece's coverage area, resulting in more precise workpiece positioning and further improved computational efficiency.

[0030] 4) Determine the real-time feature region corresponding to the workpiece in the search area based on the reference template.

[0031] Specifically, this embodiment utilizes a template matching algorithm to quickly match the feature region most similar to the reference template of the workpiece within the search area of ​​the real-time image, thereby obtaining the real-time feature region corresponding to the workpiece.

[0032] The template matching algorithm can be a normalized cross-correlation (NCC) algorithm, a zero-mean normalized cross-correlation (ZNCC) algorithm, or an edge feature matching algorithm, etc.

[0033] S32. Determine the pixel pose deviation of the workpiece in the image coordinate system based on the reference feature region and the real-time feature region.

[0034] This embodiment calculates the pixel pose deviation of the real-time feature region relative to the reference feature region in the image coordinate system. This pixel pose deviation includes the workpiece's position offset and orientation offset. The position offset can be represented by pixel coordinates, and the orientation offset can be represented by the tilt angle of the pixel relative to the horizontal line. For example, the position offset can be expressed as ( , ),in This indicates the column offset of the pixel corresponding to the geometric center of the workpiece. This indicates the offset row number of the pixel corresponding to the geometric center of the workpiece.

[0035] Pixel pose deviation directly reflects the deviation of the workpiece's real-time pose from the preset pose. This deviation may come from placement errors in previous processes, or from sudden events such as sudden changes in workpiece speed or violent shaking.

[0036] S4. Control the robot to grasp the workpiece based on the reference grasping pose and pixel pose deviation. Specifically, this includes: S41. Perform coordinate system transformation on the pixel pose deviation to obtain the spatial pose deviation of the workpiece in the robot coordinate system.

[0037] This embodiment converts pixel pose deviation into spatial pose deviation of the robot's end effector in the robot coordinate system. This spatial pose deviation is a relative quantity, representing the deviation between the robot's real-time grasping pose and the reference grasping pose during the current grasping action. Spatial pose deviation includes spatial position deviation and spatial attitude deviation, such as... Figure 2 As shown, the spatial position deviation can be expressed as ( , ),in This represents the offset distance of the robot's end effector along the horizontal axis. This represents the offset distance of the robot's end effector along the vertical axis.

[0038] S42. Control the robot to grasp the workpiece based on the reference grasping pose and spatial pose deviation, including: 1) Determine the robot's real-time grasping pose based on the reference grasping pose and the spatial pose deviation; specifically, the real-time grasping pose can be obtained by superimposing the spatial pose deviation on the reference grasping pose. 2) Determine the robot's grasping trajectory based on the real-time grasping pose; 3) Obtain the conveyor speed of the production line, and determine the robot's movement speed when the distance between the robot and the workpiece is less than the preset distance (i.e., when the robot is in the approach and gripping phases) based on the conveyor speed. Make the velocity component of the movement speed along the conveyor speed direction equal to the conveyor speed. This ensures that the robot's movement along the conveyor speed direction is synchronized with the movement of the production line during the approach and gripping phases, thus maintaining relative stillness with the production line and the workpiece. This effectively avoids relative slippage between the robot and the workpiece during the gripping process, which could lead to gripping failure or workpiece displacement. The preset distance can be flexibly set according to the robot's gripping trajectory and action time. 4) Control the robot to grasp the workpiece according to the grasping trajectory and movement speed.

[0039] Another embodiment of the present invention provides a vision-driven dynamic grasping system for moving workpieces, the system comprising: The reference acquisition module is used to acquire a reference image of the workpiece when it is located at a preset trigger position on the production line and in a preset pose, and to determine the robot's reference grasping pose based on the preset pose. The real-time acquisition module is used to acquire the real-time position of the workpiece as it is conveyed on the production line. When the real-time position matches the preset trigger position, it sends an acquisition signal to the image acquisition device using a hardware triggering method and receives the real-time image of the workpiece acquired by the image acquisition device based on the acquisition signal. The deviation determination module is used to determine the pixel pose deviation of the workpiece in the image coordinate system based on the reference image and the real-time image. The gripping module is used to control the robot to grip the workpiece based on the reference gripping pose and pixel pose deviation.

[0040] This invention eliminates the reliance on deep learning models, requiring no large-scale data collection and model training. For new workpieces, a baseline template can be created with just one "photo-teaching" session, a significantly shorter timeframe compared to existing technologies. This enables rapid deployment, adapting to 24 / 7 production demands involving multiple product categories and small batches, achieving a highly flexible "produce on demand, grab on change" production model. Furthermore, this invention utilizes a hardware-triggered acquisition method, drastically reducing the latency between sending the acquisition signal and acquiring the real-time image. This significantly minimizes the random latency caused by software communication, ensuring high synchronization between the signal transmission and real-time image acquisition. This greatly reduces workpiece position deviation in the baseline image, facilitating accurate and rapid determination of the workpiece position in the real-time image with relatively low computational power. Consequently, the computational complexity of this invention is far lower than that of deep learning models, with processing time down to the millisecond level, and a response speed far exceeding that of deep learning models, meeting the dynamic grabbing requirements of high-paced production lines. Furthermore, this invention employs a "relative offset" calculation strategy, so the final grasping accuracy does not depend on the absolute positioning accuracy, but rather on the relative positioning repetition accuracy of template matching. Under the premise of feature stability, sub-pixel level accuracy can be achieved, and it is more robust to slow changes in overall illumination. Moreover, the stability and reliability of positioning are far superior to pure vision technology solutions that rely on absolute image coordinate calculations.

[0041] The above description is merely a few embodiments of this application and is not intended to limit this application in any way. Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any changes or modifications made by those skilled in the art without departing from the scope of the technical solution of this application using the disclosed technical content are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A vision-driven dynamic grasping method for moving workpieces, characterized in that, The method includes: S1. Obtain a reference image of the workpiece when it is located at a preset trigger position on the production line and in a preset pose, and determine the robot's reference grasping pose based on the preset pose. S2. Obtain the real-time position of the workpiece being conveyed on the production line. When the real-time position matches the preset trigger position, send a collection signal to the image acquisition device using a hardware triggering method, and receive the real-time image of the workpiece collected by the image acquisition device according to the collection signal. S3. Determine the pixel pose deviation of the workpiece in the image coordinate system based on the reference image and the real-time image; S4. Control the robot to grasp the workpiece according to the reference grasping pose and the pixel pose deviation; S3 specifically includes: In the reference image, a reference feature region corresponding to the workpiece is determined, and a reference template of the workpiece is generated based on the reference feature region; Determine the delay time between sending the acquisition signal and acquiring the real-time image, and define a search area in the real-time image based on the delay time; The real-time feature region corresponding to the workpiece is determined in the search area based on the reference template. The pixel pose deviation of the workpiece in the image coordinate system is determined based on the reference feature region and the real-time feature region.

2. The method according to claim 1, characterized in that, Determining the real-time feature region corresponding to the workpiece specifically includes: The real-time feature region corresponding to the workpiece is determined using a template matching algorithm.

3. The method according to claim 1, characterized in that, S4 specifically includes: S41. Perform coordinate system transformation on the pixel pose deviation to obtain the spatial pose deviation of the workpiece in the robot coordinate system; S42. Control the robot to grasp the workpiece according to the reference grasping pose and the spatial pose deviation.

4. The method according to claim 3, characterized in that, S42 specifically includes: The robot's real-time grasping pose is determined based on the reference grasping pose and the spatial pose deviation; The robot is controlled to grasp the workpiece based on the real-time grasping pose.

5. The method according to claim 4, characterized in that, Controlling the robot to grasp the workpiece based on the real-time grasping pose specifically includes: The robot's grasping trajectory is determined based on the real-time grasping pose; The robot is controlled to grasp the workpiece according to the grasping trajectory.

6. The method according to claim 5, characterized in that, Controlling the robot to grasp the workpiece according to the grasping trajectory specifically includes: Based on the conveying speed of the production line, the movement speed of the robot when it is less than a preset distance from the workpiece is determined, and the velocity component of the movement speed along the conveying speed direction is equal to the conveying speed. The robot is controlled to grasp the workpiece based on the grasping trajectory and the movement speed.

7. A system based on the vision-driven dynamic grasping method for moving workpieces according to any one of claims 1 to 6, characterized in that, The system includes: The reference acquisition module is used to acquire a reference image of the workpiece when it is located at a preset trigger position on the production line and in a preset pose, and to determine the robot's reference grasping pose based on the preset pose. The real-time acquisition module is used to acquire the real-time position of the workpiece as it is conveyed on the production line. When the real-time position matches the preset trigger position, it sends an acquisition signal to the image acquisition device using a hardware triggering method and receives the real-time image of the workpiece acquired by the image acquisition device based on the acquisition signal. The deviation determination module is used to determine the pixel pose deviation of the workpiece in the image coordinate system based on the reference image and the real-time image. The gripping module is used to control the robot to grip the workpiece based on the reference gripping pose and pixel pose deviation.

Images