A 3D camera-based motion-guided grasping visual guidance method and system

Abstract

This invention belongs to the field of visual inspection technology and provides a 3D camera-based follow-up grasping visual guidance method and system. It utilizes a relative calibration method to transform the visual coordinates of the 3D camera into the mechanical coordinates of the robot. After receiving the conveyor belt operation signal transmitted by the encoder, the 3D camera scans the conveyor belt, acquiring multiple consecutive images. An image stitching algorithm is used to stitch these consecutive images together. A multiple filtering algorithm is used to filter duplicate carriers within different conveyor belt images, merging duplicate carriers and obtaining their coordinates. A sorting algorithm is used to sort and output the carrier coordinates, and the robot grasps the carrier based on these coordinates. This invention uses a 3D camera combined with relative calibration technology, offering a wide field of view, high guidance accuracy, and avoiding errors caused by lens distortion when using 2D camera solutions, thus improving grasping accuracy and compatibility with a wide variety of products.

Description

Technical Field

[0001] This invention belongs to the field of visual inspection technology, specifically relating to a 3D camera-based follow-up grasping visual guidance method and system. Background Technology

[0002] In the field of industrial automation, the transfer links in traditional three-way catalyst production workshops mainly rely on manual handling or 2D vision-guided technology. Manual handling is limited by the surface temperature of the carrier, which can reach hundreds of degrees Celsius after calcination. It requires intermittent operation with the aid of heat insulation equipment, resulting in low handling efficiency and personnel safety hazards, making it difficult to meet the needs of continuous production lines. 2D vision-guided technology suffers from multiple drawbacks due to the physical characteristics of the camera: its depth of field is typically less than 200mm. When dealing with carriers of different models with height differences of up to 1500mm, repeated adjustments to the camera height via mechanical lifting mechanisms are necessary, leading to cumulative positioning errors of ±5mm, and the risk of collisions damaging the carrier due to mechanism jamming or misoperation. Furthermore, lens distortion necessitates the deployment of multiple cameras for stitching in 2000mm wide-field scenes, significantly increasing system complexity and maintenance costs. Furthermore, multi-camera collaborative calibration errors further reduce grasping accuracy. Therefore, existing technologies have not effectively solved key technical challenges such as 3D spatial perception, high-precision coordinate transformation, and reliable grasping of continuously moving targets in large-scale dynamic scenes, severely hindering the flexible upgrading and intelligent transformation of production lines. 2D cameras require a fixed distance between the object's imaging surface and the camera. However, the carrier has varying heights, necessitating the mounting of the 2D camera on a vertically movable mechanism. Producing carriers of different heights requires this moving mechanism, which may result in mismatched heights and image inaccuracies. Furthermore, 2D cameras require an external light source for illumination; different carrier surface colors lead to varying lighting effects, resulting in less stable image quality compared to 3D solutions. Therefore, this invention provides a 3D camera-based follow-up grasping visual guidance method and system. Summary of the Invention

[0003] The purpose of this invention is to overcome the existing defects and provide a 3D camera-based motion-guided grasping visual guidance method and system.

[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0005] A 3D camera-based motion-guided grasping visual guidance method includes:

[0006] The visual coordinates of the 3D camera are transformed into the mechanical coordinates of the robot using a relative calibration method;

[0007] After receiving the conveyor belt operation signal transmitted by the encoder, the 3D camera scans the conveyor belt and acquires multiple consecutive images of the conveyor belt.

[0008] Multiple consecutive conveyor belt images are stitched together using an image stitching algorithm. A multiple filtering algorithm is used to filter the carriers obtained repeatedly in different conveyor belt images, merge the repeated carriers and obtain the carrier coordinates.

[0009] Using a sorting algorithm, the coordinates of the carrier are sorted and output, and the robot grasps the carrier based on the obtained coordinates.

[0010] Furthermore, the method of converting the visual coordinates of the 3D camera to the mechanical coordinates of the robot using a relative calibration method includes:

[0011] Multiple calibration components are evenly placed within a section of the conveyor belt;

[0012] Set the 3D camera output parameters and use the 3D camera to acquire calibration images containing all calibration components;

[0013] Image processing algorithms are used to sequentially find the pixel coordinates of feature points on the calibration component in the calibration image;

[0014] The conveyor belt moves multiple calibration parts to a position within the robot's arm span. The robot picks up the multiple calibration parts in sequence and records the mechanical coordinates of the parts picked up by the robot in sequence, thus obtaining a set of robot coordinates.

[0015] The robot coordinates are transformed according to the relative calibration formula. The transformed robot coordinates and pixel coordinates are then input into the calibration tool for calibration. Points with large errors are deleted or modified until the calibration error meets the production requirements.

[0016] Furthermore, the relative calibration formula is expressed as:

[0017]

[0018] in, For the machine coordinate system, Let R be the image coordinate system, M be the rotation coefficient, and N be the linear motion distance of the robot to reach the same point in the calibration image using the workpiece coordinate system.

[0019] Furthermore, the step of stitching together multiple consecutive conveyor belt images using an image stitching algorithm includes:

[0020] Obtain the first stitched image and the second stitched image, and obtain the image width and height parameters;

[0021] Generate an output image with a length equal to the sum of the lengths of the first and second stitched images, and a width equal to the width of either the first or second stitched image;

[0022] Set the image stitching alignment point, and copy the data of the first stitched image and the second stitched image to the output image according to the alignment point coordinates.

[0023] Furthermore, in setting the image stitching alignment points, the (0,0) point of the output image corresponds to the (0,0) point of the first stitched image, and the (0, output image width / 2) point of the output image corresponds to the (0,0) point of the second stitched image.

[0024] Furthermore, the step of using a multiple filtering algorithm to filter repeatedly obtained carriers within different conveyor belt images, merging the repeated carriers, and obtaining the carrier coordinates includes:

[0025] Define the carrier as a class, which contains the carrier's X, Y, and angle information, the row and column it belongs to, single or double grasp information, and carrier spacing information;

[0026] Blob analysis and circle finding analysis are used to process the image to obtain the X, Y, and angle information of the corresponding carrier, the row and column information, single or double capture information, and carrier spacing information.

[0027] Create a carrier queue, create carrier objects, and assign values ​​to the properties of the carrier objects;

[0028] Perform an enqueue operation on the carrier class, perform the first carrier filtering based on the carrier's coordinate attributes to filter out duplicate carriers, and enqueue the filtering results.

[0029] A second carrier filtering is performed based on the carrier spacing information to filter carriers whose spacing does not meet the grasping conditions, and the results of this filtering are then enqueued.

[0030] Furthermore, the step of using a sorting algorithm to sort and output the carrier coordinates includes:

[0031] Create a carrier object, dequeue the queue after the second filtering, and assign the information of the dequeue carrier to the newly created carrier object.

[0032] The carrier object is divided into rows based on its Y-coordinate information;

[0033] Sort the carrier objects after row division according to their X coordinates, and enqueue the sorted carrier objects.

[0034] Perform a dequeue operation on the carrier objects sorted by X coordinate, row by row.

[0035] Another object of the present invention is to provide a 3D camera-based motion-guided grasping visual guidance system, comprising:

[0036] A 3D line scan camera is used to scan the conveyor belt after receiving the conveyor belt running signal transmitted by the encoder, and to acquire multiple consecutive images of the conveyor belt.

[0037] The coordinate transformation module is used to transform the visual coordinates of the 3D camera to the mechanical coordinates of the robot using a relative calibration method;

[0038] The image carrier processing module is used to stitch together multiple consecutive conveyor belt images using an image stitching algorithm, filter duplicate carriers obtained in different conveyor belt images using a multiple filtering algorithm, merge duplicate carriers and obtain carrier coordinates.

[0039] The carrier coordinate output module is used to sort and output the carrier coordinates using a sorting algorithm.

[0040] A grasping robot is used to grasp carriers based on the corresponding coordinates of each carrier obtained.

[0041] In combination with the above technical solutions, the beneficial effects of the present invention compared with the prior art are as follows:

[0042] This invention employs a 3D camera combined with relative calibration technology, offering a wide field of view. It achieves an ultra-wide field of view coverage of 2000mm without mechanical adjustments, solving the problem of requiring multiple workstations for processing large fields of view with 2D cameras. It boasts high guidance accuracy, with pure visual accuracy reaching ±0.5mm and overall equipment accuracy reaching ±2mm, avoiding errors caused by lens distortion when using 2D camera solutions and improving grasping accuracy. It is compatible with a wide range of products, supporting carriers with a maximum height difference of 1500mm without hardware modifications, eliminating the need to change camera positions when changing product models using 2D cameras.

[0043] In summary, this invention solves the problems of low efficiency in manual handling, cumbersome and error-prone operation of 2D camera-guided changeover, and improves the production efficiency and product yield of the production line. This invention has been successfully applied to several pieces of equipment in the production workshop and is operating well.

[0044] This invention also solves the problem that 2D cameras need to change their lighting scheme to be compatible with different colored carriers, i.e., many light sources may need to be hung on the same production line, and different light sources need to be turned on to produce different models of carriers; it also solves the problem that 2D cameras need to be mounted on a lifting mechanism to be compatible with carriers of different heights, and the camera needs to be raised and lowered to a fixed height in advance when producing carriers of different heights; and it solves the problem that the accuracy of 2D cameras cannot meet the requirements when the field of view is large. When the field of view is large, the lens equipped with 2D cameras will have great distortion, especially at the edge of the field of view, while 3D cameras have no distortion. Attached Figure Description

[0045] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0046] Figure 1 This is a flowchart of a 3D camera-based motion-guided grasping visual guidance method provided in an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of the calibration effect provided in an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the image stitching algorithm provided in the embodiment of the present invention, wherein (a) is the first stitched image, (b) is the second stitched image, and (c) is the stitched image. Detailed Implementation

[0049] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0050] Example 1:

[0051] like Figure 1 The image shown is an embodiment of the 3D camera-based motion-guided grasping visual guidance method provided by the present invention, comprising:

[0052] S1: Use a relative calibration method to transform the visual coordinates of the 3D camera into the mechanical coordinates of the robot;

[0053] S2: After receiving the conveyor belt operation signal transmitted by the encoder, the 3D camera scans the conveyor belt and acquires multiple consecutive images of the conveyor belt.

[0054] S3: Use an image stitching algorithm to stitch together multiple consecutive conveyor belt images, use a multiple filtering algorithm to filter the repeated carriers in different conveyor belt images, merge the repeated carriers and obtain the carrier coordinates.

[0055] S4: Using a sorting algorithm, the coordinates of the carrier are sorted and output. The robot then grasps the carrier based on the obtained coordinates.

[0056] Preferably, in step S1 of this embodiment of the invention, the process of converting the visual coordinates of the 3D camera to the mechanical coordinates of the robot using a relative calibration method includes:

[0057] Step 1: Distribute the calibration pieces (at least 4, usually 9) evenly on the conveyor belt within a 600mm length (this distance can be flexibly set according to the site conditions).

[0058] Step 2: Set the 3D camera output parameters to ensure that all calibration parts are scanned when the 3D camera outputs the image, and save the output calibration image locally.

[0059] Step 3: Find the calibration image and use image processing algorithms to find the pixel coordinates of the feature points on the calibration piece (the coordinates of this point in the 3D camera coordinate system). At this time, a set of pixel coordinates is obtained.

[0060] Different image processing algorithms can be used depending on the appearance of the calibration part, as shown in the following examples:

[0061] For circular calibration parts: you can use the BLOB tool to find the centroid of the calibration part; you can also use the circle finding tool to find the center of the calibration part; or you can use the PMA tool to find the center of the calibration part.

[0062] For rectangular calibration parts: the centroid of the calibration part can be found using the BLOB tool; the center of the calibration part can also be found using the line finding tool and line fitting tool; or the center of the calibration part can be found using the PMA tool.

[0063] Step 4: The conveyor belt continues to move forward and stops when it reaches a position accessible to the robot's arm. The robot's gripper moves sequentially to the position of the calibration part in the robot workpiece coordinate system, and the coordinates of the robot gripper at that time are recorded. At this point, a set of robot coordinates is obtained.

[0064] Step 5: Transform the robot coordinates according to the relative calibration formula.

[0065] Step 6: Input the converted robot coordinates and pixel coordinates into the calibration tool for calibration. If there are points with large errors, they can be deleted or modified until the calibration error meets the requirements (the implementer needs to control this based on experience).

[0066] The application data in this embodiment is shown in the table below, and the calibration results are as follows: Figure 2 As shown.

[0067] Table 1. Calibration Application Data Details

[0068]

[0069] The relative calibration algorithm in this step is derived from the existing absolute calibration algorithm, and has been optimized based on the existing absolute calibration algorithm. The algorithm is described below:

[0070] The purpose of calibration is to transform the image coordinates (pixels) of the 3D camera to the robot's mechanical coordinate system (millimeters). The correspondence between the two coordinate systems is as follows:

[0071] Mechanical coordinate system = rotation coefficient * image coordinates + displacement coefficient

[0072] Represented as:

[0073]

[0074] After expansion, we get the following formula:

[0075] X′=aX+bY+c

[0076] Y′=a′X+b′Y+c′

[0077] This can be transformed into finding the coefficients of the following linear equation in three variables:

[0078]

[0079] They are represented as follows:

[0080]

[0081] and

[0082]

[0083] Where X′ and Y′ are the coordinate parameters of the machine coordinate system, X and Y are the coordinate parameters of the image coordinate system, R is the rotation coefficient, M is the displacement coefficient, a, b, a′, b′ represent the rotation coefficient parameters when transforming the image coordinate system to the machine coordinate system, and c, c′ are the displacement coefficient parameters when transforming the image coordinate system to the machine coordinate system. When we have at least three sets of corresponding pixel coordinate points and machine coordinate points in the application process, we can solve for a, b, c, a′, b′, c′ by substituting them into a system of three linear equations. X0′ and Y0′ are the coordinate parameters of the first set of machine coordinate systems, X1′ and Y1′ are the coordinate parameters of the second set of machine coordinate systems, X2′ and Y2′ are the coordinate parameters of the third set of machine coordinate systems, X0 and Y0 are the coordinate parameters of the first set of image coordinate systems, X1 and Y1 are the coordinate parameters of the second set of image coordinate systems, and X2 and Y2 are the coordinate parameters of the third set of image coordinate systems.

[0084] As can be seen from the above formula, to calculate the rotation coefficient R and the displacement coefficient M, we need points in the aligned mechanical coordinate system and points in the image coordinate system, and at least three points are required.

[0085] In this embodiment, the mechanical coordinate system originates from the robot, and the image coordinate system originates from the 3D line scan camera. Because there is a certain distance between the robot and the 3D camera during implementation, it is not possible to directly provide the coordinates of the same point in both the mechanical and image coordinate systems. To solve this problem, high-precision measuring equipment was introduced externally to measure the data on site, leading to the following formula:

[0086]

[0087] While both N and M are coefficient matrices, N is a known quantity, representing the distance the robot travels from the image of the same point in the 3D camera during calibration to the point the robot can reach using the workpiece coordinate system. M is a quantity that needs to be solved for. The rotation coefficient R is an unknown quantity that needs to be solved for, obtained through relative calibration. The purpose of relative calibration is to solve for the rotation and displacement coefficients.

[0088] The calibration method used in the 2D camera approach involves counting squares on a checkerboard pattern. Since 3D cameras cannot image a checkerboard pattern, this relative calibration method solves the calibration problem when converting 3D images into 2D images for 2D applications.

[0089] Preferably, in step S3 of this embodiment of the invention, multiple consecutive conveyor belt images are stitched together using an image stitching algorithm, including:

[0090] Obtain the first and second stitched images (the images are temporarily stored in the industrial control computer's memory and assigned values ​​through logic), and obtain the image length and width parameters;

[0091] Generate an output image with a length equal to the sum of the lengths of the first and second stitched images, and a width equal to the width of either the first or second stitched image (the widths of the first and second stitched images are the same).

[0092] Set the image stitching alignment point, and copy the data of the first and second stitched images to the output image based on the alignment point coordinates. Here, "data" refers to the information within the image (0 to 255). Based on the alignment point, determine the correct positions of the first and second stitched images in the final stitched image, and then directly copy their data to the desired locations. The stitched image is created based on the width and height information of the first and second stitched images, and then the data from these images is copied over (after conversion).

[0093] Preferably, in setting the image stitching alignment points, the (0,0) point of the output image corresponds to the (0,0) point of the first stitched image, and the (0, output image width / 2) point of the output image corresponds to the (0,0) point of the second stitched image.

[0094] Specifically, because the position of the carrier on the conveyor belt is uncertain and the 3D camera output image requires fixed distance parameters, the 3D camera may capture only a portion of the carrier in an image. Direct image processing could lead to missing carrier segments or incorrect carrier positioning. To address this issue, this invention employs an image stitching algorithm.

[0095] like Figure 3 As shown, Figure 3 (a) in the image is the first stitched image. Figure 3 (b) in the image is the second stitched image. Figure 3 (c) in the image is the stitched image, where Figure 3 The portion within box (c) in the image represents the stitched carrier. Only a portion of the two bottom carriers in the first stitched image is shown; the remaining portions of the two top carriers in the second stitched image are the bottom two carriers shown in the first stitched image. The stitched image after image stitching is as follows: Figure 3 As shown in (c), the box contains the spliced ​​carriers. It can be seen that the two carriers are spliced ​​together very well after splicing. After multiple verifications, the splicing accuracy of this splicing algorithm can reach ±0.005mm.

[0096] Preferably, in step S3 of this embodiment of the invention, the repeated carriers obtained in different conveyor belt images are filtered using a multiple filtering algorithm, and the repeated carriers are merged to obtain the carrier coordinates, including:

[0097] Step 1: Define the carrier as a class, which includes the carrier's X, Y, and angle information, the row and column it belongs to, single or double grab information, and carrier spacing information, etc.

[0098] Step 2: Use Blob analysis and circle finding analysis to process the image and obtain the X, Y, and angle information of the corresponding carrier, the row and column information, single or double capture information, carrier spacing information, etc.

[0099] Step 3: Create a carrier queue (to store carrier objects), create carrier objects, and assign values ​​to the properties of the carrier objects.

[0100] Step 4: Perform an enqueue operation on the carrier class, perform the first carrier filtering based on the carrier's coordinate attributes (this filtering filters out duplicate carriers), and enqueue the results of this filtering (this queue is the queue after the first filtering).

[0101] Step 5: Perform a second carrier filtering based on the carrier spacing information (this filtering is for carriers whose carrier spacing does not meet the grabbing conditions), and enqueue the results of this filtering (this queue is the queue after the second filtering).

[0102] It should be noted that after using image stitching algorithms to solve the problem of not being able to capture a complete carrier during image acquisition, a corresponding phenomenon arises: each image captured by the 3D camera is processed twice (because the first image in each image stitching operation is used as the second image in the next image stitching operation). Therefore, some carriers may be detected twice by the visual algorithm. To eliminate these carriers detected twice, a multiple filtering algorithm is employed.

[0103] Preferably, after obtaining the carrier coordinates, the carrier coordinates need to be sorted and output to prevent carrier interference during the robot's grasping process, including:

[0104] Step 1: Create the carrier object;

[0105] Step 2: Dequeue the queue after the second filtering and assign the dequeued carrier to the newly created carrier object.

[0106] Step 3: Divide the carrier into rows based on the carrier's Y-coordinate information;

[0107] Step 4: Sort the carriers after row division according to their X coordinates, and enqueue the sorted carriers.

[0108] Step 5: Dequeue the carriers sorted by X coordinate row by row.

[0109] The existing algorithms are one-dimensional sorting algorithms. This algorithm is a multi-dimensional sorting algorithm derived for the specific situation. It can sort multi-dimensional data and ensure that the sorted data is not disordered.

[0110] Example 2:

[0111] This invention provides a 3D camera-based motion-guided grasping visual guidance system, comprising:

[0112] A 3D line scan camera is used to scan the conveyor belt after receiving the conveyor belt running signal transmitted by the encoder, and to acquire multiple consecutive images of the conveyor belt.

[0113] The coordinate transformation module is used to transform the visual coordinates of the 3D camera to the mechanical coordinates of the robot using a relative calibration method;

[0114] The image carrier processing module is used to stitch together multiple consecutive conveyor belt images using an image stitching algorithm, filter duplicate carriers obtained in different conveyor belt images using a multiple filtering algorithm, merge duplicate carriers and obtain carrier coordinates.

[0115] The carrier coordinate output module is used to sort and output the carrier coordinates using a sorting algorithm.

[0116] A grasping robot is used to grasp carriers based on the corresponding coordinates of each carrier obtained.

[0117] The operation flow of this invention embodiment is as follows:

[0118] Step 1: Set the parameters of the 3D line scan camera and start the vision guidance system.

[0119] Step 2: After the PLC controller of the conveyor belt detects that there is material on the conveyor belt, it controls the conveyor belt to run. The 3D line scan camera acquires the first image with a length of 300mm (this distance can be flexibly set according to the site conditions). (The conveyor belt moves forward, and the encoder sends an encoder signal to the 3D line scan camera. When the conveyor belt reaches 300mm, the 3D line scan camera will output an image).

[0120] Step 3: The image is uploaded to the visual guidance system via network cable to begin algorithm processing. First, the image is calibrated using the coordinate transformation module. Then, the image carrier processing module performs image stitching (if it is the first time the image is acquired, stitching is not performed temporarily; after the first acquisition, stitching is performed every time). After that, carrier search, sorting, and filtering are performed in sequence to finally obtain the detection result.

[0121] Step 4: The carrier coordinate output module sends the detection results to the grasping robot via TCP / IP protocol. The grasping robot processes the results and grasps the carrier based on the received grasping information.

[0122] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in the present invention, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0123] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0124] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A 3D camera based on-the-fly grasp vision guidance method, characterized in that, The method comprises the following steps: (1) converting the visual coordinates of the 3D camera to the mechanical coordinates of the robot by using a relative calibration method; (2) after receiving the conveying belt running signal transmitted by the encoder, the 3D camera scans the conveying belt to obtain a plurality of continuous conveying belt images; (3) splicing the plurality of continuous conveying belt images by using an image splicing algorithm, filtering the carriers repeatedly obtained in different conveying belt images by using a multiple filtering algorithm, merging the repeated carriers and obtaining carrier coordinates; The multiple filtering algorithm for filtering the carriers repeatedly obtained in different conveying belt images, merging the repeated carriers and obtaining carrier coordinates comprises: defining the carrier as a class, wherein the class contains carrier X, Y, angle information, the row and column to which the carrier belongs, single grab, double grab information, and carrier spacing information; processing the image by using Blob analysis and round analysis to obtain the X, Y, angle information of the corresponding carrier, the row and column information to which the carrier belongs, the single grab, double grab information, and the carrier spacing information; creating a carrier queue, creating a carrier object and assigning values to the attributes of the carrier object; carrying out a queue operation on the carrier class, carrying out a first carrier filtering according to the coordinate attributes of the carrier, filtering the repeated carriers, and carrying out a queue operation on the filtering results; carrying out a second carrier filtering according to the spacing information of the carrier, filtering the carriers whose spacing does not meet the grabbing condition, and carrying out a queue operation on the filtering results; (4) sorting and outputting the carrier coordinates by using a class sorting algorithm, and grabbing the carriers by the robot according to the obtained carrier coordinates.

2. The 3D camera-based follow-the-grip visual guidance method of claim 1, wherein, The relative calibration method for converting the visual coordinates of the 3D camera to the mechanical coordinates of the robot comprises: uniformly placing a plurality of calibration members in a length range of the conveying belt; setting the 3D camera output image parameters, and collecting calibration images containing all the calibration members by using the 3D camera; finding the pixel coordinates of the feature points on the calibration members in the calibration images in sequence by using an image processing algorithm; moving the plurality of calibration members to positions within the arm length range of the robot by the conveying belt, moving the robot gripper to the positions of the plurality of calibration members in sequence and recording the mechanical coordinates of the robot in sequence to obtain a set of robot coordinates; converting the robot coordinates according to a relative calibration formula, inputting the converted robot coordinates and the pixel coordinates into a calibration tool for calibration, deleting or modifying the points with large errors until the calibration error meets the production requirements.

3. The 3D camera-based follow-the-grip visual guidance method of claim 2, wherein, The relative calibration formula is expressed as: wherein, is a mechanical coordinate system, is an image coordinate system, R is a rotation coefficient, M is a displacement coefficient, and N is the linear motion distance of the robot from the workpiece coordinate system to the same point in the calibrated image.

4. The 3D camera-based follow-the-grip visual guidance method of claim 1, wherein, The image splicing algorithm for splicing the plurality of continuous conveying belt images comprises: obtaining a first splicing image and a second splicing image, and obtaining the length and width parameters of the images; generating an output image with a length equal to the sum of the first splicing image and the second splicing image and a width equal to the width of the first splicing image or the second splicing image; setting the image splicing alignment points, and respectively copying the data of the first splicing image and the second splicing image to the output image according to the alignment point coordinates.

5. The 3D camera-based follow-the-grip visual guidance method of claim 4, wherein, In the setting of the image splicing alignment points, the (0, 0) point of the output image corresponds to the (0, 0) point of the first splicing image, and the (0, output image width / 2) point of the output image corresponds to the (0, 0) point of the second splicing image.

6. The 3D camera-based follow-the-grip visual guidance method of claim 5, wherein, The sorting algorithm is used for sorting and outputting the carrier coordinates, including: Creating a carrier object, dequeuing the queue after the second filtering, and assigning the information of each dequeued carrier to the newly created carrier object; Dividing the carrier objects into rows according to the Y coordinate information of the carrier objects; Sorting the divided carrier objects according to the X coordinate, and performing an enqueue operation on the sorted carrier objects; Performing a dequeue operation on the carrier objects sorted according to the X coordinate by row.

7. A 3D camera based on-the-fly grasping vision guidance system, characterized in that, The system comprises: A 3D line scanning camera for scanning the conveyor belt and obtaining multiple continuous conveyor belt images after receiving the conveyor belt running signal transmitted by the encoder; A coordinate conversion module for converting the visual coordinates of the 3D camera to the mechanical coordinates of the robot using a relative calibration method; An image carrier processing module for splicing multiple continuous conveyor belt images using an image splicing algorithm, filtering the carrier repeatedly obtained in different conveyor belt images using a multiple filtering algorithm, merging the repeated carriers, and obtaining carrier coordinates; A carrier coordinate output module for sorting and outputting the carrier coordinates using a sorting algorithm; A grabbing robot for grabbing the carriers according to the corresponding coordinates of each carrier obtained; The multiple filtering algorithm is used to filter the repeatedly obtained carriers in different conveyor belt images, merge the repeated carriers, and obtain the carrier coordinates, including: Defining the carrier as a class, which contains carrier X, Y, angle information, row and column information, single and double grabbing information, and carrier spacing information; Using Blob analysis and circle finding analysis to process the image and obtain the X, Y, angle information, row and column information, single and double grabbing information, and carrier spacing information of the corresponding carrier; Creating a carrier queue, creating a carrier object, and assigning attributes to the carrier object; Performing an enqueue operation on the carrier class, performing a first carrier filtering according to the coordinate attributes of the carrier, filtering the repeated carriers, and performing an enqueue operation on the filtering results; Performing a second carrier filtering according to the spacing information of the carrier, filtering the carriers whose spacing does not meet the grabbing condition, and performing an enqueue operation on the filtering results.

Images