Follow-up grabbing visual guidance method and system based on 3D camera

Abstract

The invention belongs to the technical field of visual inspection, and provides a follow-up grabbing visual guidance method and system based on a 3D camera, and the method comprises the steps: converting a visual coordinate of the 3D camera into a mechanical coordinate of a robot through a relative calibration method; after the 3D camera receives the conveyor belt operation signal transmitted by the encoder, the conveyor belt is scanned, and a plurality of continuous conveyor belt images are obtained; splicing a plurality of continuous conveyor belt images by using an image splicing algorithm, filtering carriers repeatedly obtained in different conveyor belt images by using a multi-filtering algorithm, and merging the repeated carriers to obtain carrier coordinates; and the carrier coordinates are sorted and output through a class sorting algorithm, and the robot grabs the carriers according to the obtained carrier coordinates. The 3D camera is combined with the relative calibration technology, the compatible view field is wide, the guiding precision is high, errors caused by lens distortion when a 2D camera scheme is used are avoided, the grabbing precision is improved, and the compatible product types are multiple.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual detection, and in particular relates to a 3D camera-based follow-up grasping visual guidance method and system. Background Art

[0002] In the field of industrial automation, traditional three-way catalyst production workshops rely primarily on manual handling or 2D vision-guided technology for transfer operations. Manual handling is limited by the surface temperature of the calcined carrier, which can reach hundreds of degrees Celsius. It requires intermittent operation with the aid of thermal insulation equipment, resulting in low handling efficiency and personnel safety hazards, making it difficult to meet the requirements of continuous production lines. 2D vision-guided technology, however, suffers from multiple drawbacks due to the physical characteristics of the camera. Its depth of field is typically less than 200mm. When handling different carriers with height differences of up to 1500mm, the camera's height must be repeatedly adjusted using a mechanical lifting mechanism, resulting in cumulative positioning errors of up to ±5mm and the risk of collision damage due to mechanism jamming or misoperation. Furthermore, due to lens distortion, multiple cameras must be deployed for stitching in a 2000mm field of view, significantly increasing system complexity and maintenance costs. Furthermore, errors in multi-camera coordinated calibration can further reduce grasping accuracy. Consequently, existing technologies have yet to effectively address key technical challenges such as three-dimensional spatial perception, high-precision coordinate transformation, and reliable grasping of continuously moving objects in large-scale dynamic scenes, severely hindering the flexible upgrade and intelligent transformation of production lines. 2D cameras require a fixed distance between the object's imaging surface and the camera, but the carrier has height differences. Therefore, using a 2D camera requires the camera to be mounted on a mechanism that can move up and down. When producing carriers of different heights, the mechanism needs to be moved, and the height mismatch may occur, resulting in imaging deviations. 2D cameras require an external light source for illumination, and the different colors of the carrier surface will result in different lighting effects, resulting in unstable imaging compared to 3D solutions. Therefore, the present invention provides a visual guidance method and system for follow-up grasping based on a 3D camera. Summary of the Invention

[0003] The purpose of the present invention is to overcome the existing defects and provide a 3D camera-based follow-up grasping vision guidance method and system.

[0004] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0005] A 3D camera-based follow-up grasping vision guidance method includes:

[0006] Use relative calibration method to convert the visual coordinates of 3D camera to the mechanical coordinates of robot;

[0007] After receiving the conveyor belt running signal transmitted by the encoder, the 3D camera scans the conveyor belt and obtains multiple continuous conveyor belt images;

[0008] Use image stitching algorithm to stitch multiple continuous conveyor belt images, use multiple filtering algorithms to filter repeated carriers in different conveyor belt images, merge repeated carriers and obtain carrier coordinates;

[0009] The carrier coordinates are sorted and output using a class sorting algorithm, and the robot grabs the carrier based on the obtained carrier coordinates.

[0010] Furthermore, the method of converting the visual coordinates of the 3D camera into the mechanical coordinates of the robot using a relative calibration method includes:

[0011] Place multiple calibration pieces evenly over a length of the conveyor belt;

[0012] Set the 3D camera output parameters and use the 3D camera to collect calibration images containing all calibration parts;

[0013] Use image processing algorithms to find the pixel coordinates of the feature points on the calibration part in the calibration image in sequence;

[0014] The conveyor belt drives multiple calibration pieces to a position within the robot's arm span. The robot takes the calibration pieces in turn and records the mechanical coordinates of the pieces taken by the robot in turn to obtain a set of robot coordinates.

[0015] The robot coordinates are converted according to the relative calibration formula, and the converted robot coordinates and pixel coordinates are input into the calibration tool for calibration. Points with large errors are deleted or modified until the calibration error meets the production requirements.

[0016] Furthermore, the relative calibration formula is expressed as:

[0017]

[0018] in, is the mechanical coordinate system, is the image coordinate system, R is the rotation coefficient, M is the displacement coefficient, and N is the linear motion distance that the robot uses the workpiece coordinate system to reach the same point in the calibration image.

[0019] Furthermore, the step of stitching a plurality of continuous conveyor belt images using an image stitching algorithm includes:

[0020] Obtain the first stitched image, the second stitched image, and the image length and width parameters;

[0021] Generate an output image whose length is the sum of the first stitched image and the second stitched image and whose width is the width of the first stitched image or the second stitched image;

[0022] Set the image stitching alignment point, and copy the data of the first stitching image and the second stitching image to the output image according to the alignment point coordinates.

[0023] Furthermore, in setting the image stitching alignment point, the (0,0) point of the output image corresponds to the (0,0) point of the first stitched image, and the (0, output image width / 2) point of the output image corresponds to the (0,0) point of the second stitched image.

[0024] Furthermore, the method of filtering the carriers repeatedly obtained in different conveyor belt images using a multiple filtering algorithm, merging the repeated carriers and obtaining the carrier coordinates includes:

[0025] Define the carrier as a class, which contains the carrier's X, Y, angle information, the row and column it belongs to, single grasping, double grasping information, and carrier spacing information;

[0026] Use blob analysis and circle analysis to process the image and obtain the X, Y, and angle information of the corresponding carrier, the row and column information, single capture and double capture information, and carrier spacing information;

[0027] Create a carrier queue, create a carrier object and assign values ​​to the attributes of the carrier object;

[0028] The carrier class is queued, and the first carrier filtering is performed according to the carrier's coordinate attributes to filter out duplicate carriers, and the filtering results are queued.

[0029] A second carrier filtering is performed based on the carrier spacing information to filter out carriers whose carrier spacing does not meet the crawling conditions, and the results of this filtering are queued.

[0030] Furthermore, the method of using a class sorting algorithm to sort and output the carrier coordinates includes:

[0031] Create a carrier object, dequeue the queue after the second filtering, and assign the information of each dequeue carrier to the newly created carrier object;

[0032] Separate the carrier objects into rows according to their Y coordinate information;

[0033] Sort the carrier objects after the rows are divided according to the X coordinates, and put the sorted carrier objects into the queue;

[0034] Dequeue the carrier objects sorted by X coordinates row by row.

[0035] Another object of the present invention is to provide a 3D camera-based follow-up grasping vision guidance system, comprising:

[0036] 3D line scan camera, which is used to scan the conveyor belt after receiving the conveyor belt running signal transmitted by the encoder, and obtain multiple continuous conveyor belt images;

[0037] The coordinate conversion module is used to convert the visual coordinates of the 3D camera into the mechanical coordinates of the robot using the relative calibration method;

[0038] An image carrier processing module is used to stitch multiple continuous conveyor belt images using an image stitching algorithm, filter repeated carriers in different conveyor belt images using a multiple filtering algorithm, merge repeated carriers and obtain carrier coordinates;

[0039] The carrier coordinate output module is used to sort and output the carrier coordinates using a class sorting algorithm;

[0040] The grasping robot is used to grasp the carriers according to the corresponding coordinates of each carrier obtained.

[0041] In combination with the above technical solutions, the present invention has the following beneficial effects compared with the prior art:

[0042] The present invention uses a 3D camera combined with relative calibration technology, which is compatible with a wide field of view. It can achieve 2000mm ultra-wide field of view coverage without mechanical adjustment, solving the problem that 2D cameras need to use multiple stations for processing when the field of view is large. The guidance accuracy is high, the pure visual accuracy can reach ±0.5mm, and the overall accuracy of the equipment can reach ±2mm, avoiding the error caused by lens distortion when using a 2D camera solution and improving the grasping accuracy. It is compatible with many types of products. Without changing the hardware, it can be compatible with carriers with a maximum height difference of 1500mm, solving the problem of changing the camera position when changing the product model when using a 2D camera.

[0043] In summary, the present invention solves the problems of low manual handling efficiency and cumbersome and error-prone operation of using 2D cameras to guide mold change, thereby improving the production efficiency and product yield of the production line. The present invention has been successfully applied to several devices in the production workshop and all of them are running well.

[0044] The present invention also solves the problem that a 2D camera needs to change its lighting scheme in order to be compatible with carriers of different colors, that is, multiple light sources may need to be hung on the same production line, and different light sources need to be used to produce carriers of different models; it solves the problem that a 2D camera needs to be installed on a lifting mechanism in order to be compatible with carriers of different heights, and the camera needs to be raised and lowered to a fixed height in advance when producing carriers of different heights; it solves the problem that a 2D camera cannot meet the accuracy requirements when the field of view is large. When the field of view is large, the lens equipped with the 2D camera will have a large distortion, especially at the edge of the field of view, while a 3D camera has no distortion. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0046] Figure 1 Flowchart of a 3D camera-based visual guidance method for follow-up grasping provided by an embodiment of the present invention;

[0047] Figure 2 is a schematic diagram of the calibration effect provided by an embodiment of the present invention;

[0048] Figure 3 3 is a principle diagram of the image stitching algorithm provided by an embodiment of the present invention, wherein (a) is the first stitched image, (b) is the second stitched image, and (c) is the stitched image. DETAILED DESCRIPTION

[0049] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0050] Example 1:

[0051] like Figure 1 FIG. 1 is an embodiment of a 3D camera-based follow-up grasping visual guidance method provided by the present invention, comprising:

[0052] S1: Use the relative calibration method to convert the visual coordinates of the 3D camera to the mechanical coordinates of the robot;

[0053] S2: After receiving the conveyor belt running signal transmitted by the encoder, the 3D camera scans the conveyor belt and obtains multiple continuous conveyor belt images;

[0054] S3: Using an image stitching algorithm to stitch multiple continuous conveyor belt images, using a multiple filtering algorithm to filter the repeated vectors in different conveyor belt images, merging the repeated vectors and obtaining the vector coordinates;

[0055] S4: Use the class sorting algorithm to sort and output the carrier coordinates, and the robot grabs the carrier according to the obtained carrier coordinates.

[0056] Preferably, in step S1 in the embodiment of the present invention, converting the visual coordinates of the 3D camera to the mechanical coordinates of the robot using a relative calibration method includes:

[0057] Step 1: Evenly distribute the calibration pieces (at least 4, usually 9) on the conveyor belt within a length of 600mm (this distance can be flexibly set according to the site conditions).

[0058] Step 2: Set the 3D camera output parameters to ensure that all calibration parts are scanned when the 3D camera outputs the image, and save the output calibration image locally.

[0059] Step 3: Find the calibration image and use the image processing algorithm to find the pixel coordinates of the feature points on the calibration part in sequence (the coordinates of this point in the 3D camera coordinate system). At this time, a set of pixel coordinates is obtained.

[0060] Different image processing algorithms can be used depending on the appearance of the calibration parts. Examples are as follows:

[0061] Circular calibration part: You can use the BLOB tool to find the center of mass of the calibration part; you can also use the Find Circle tool to find the center of the calibration part; you can also use the PMA tool to find the center of the calibration part.

[0062] Rectangular calibration part: You can use the BLOB tool to find the center of mass of the calibration part; you can also use the Line Finder tool and Line Fitting tool to find the center of the calibration part; you can also use the PMA tool to find the center of the calibration part.

[0063] Step 4: The conveyor belt continues to move forward and stops at a position that the robot's arm can reach. The robot's gripper moves to the position of the calibration workpiece in the robot's workpiece coordinate system and records the coordinates of the robot's gripper at that time. At this point, a set of robot coordinates is obtained.

[0064] Step 5: Convert the robot coordinates according to the relative calibration formula.

[0065] Step 6: Input the converted robot coordinates and pixel coordinates into the calibration tool for calibration. If there are points with large errors, they can be deleted or modified until the calibration error meets the requirements (implementers need to control it based on their experience).

[0066] The application data in this embodiment are shown in the following table. Figure 2 shown.

[0067] Table 1 Calibration application data details

[0068]

[0069] The relative calibration algorithm in this step is derived from the existing absolute calibration algorithm and is optimized based on the existing absolute calibration algorithm. The algorithm is described as follows:

[0070] The purpose of calibration is to convert the image coordinates (pixels) of the 3D camera into the mechanical coordinate system (mm) of the robot. The correspondence between the two coordinate systems is as follows:

[0071] Mechanical coordinate system = rotation coefficient * image coordinate + displacement coefficient

[0072] Expressed as:

[0073]

[0074] After expansion, we get the following formula:

[0075] X′=aX+bY+c

[0076] Y′=a′X+b′Y+c′

[0077] It can be transformed into the problem of finding the coefficients of the following three-variable linear equation:

[0078]

[0079] Respectively expressed as:

[0080]

[0081] and

[0082]

[0083] Here, X′ and Y′ are the coordinate parameters of the mechanical coordinate system, X and Y are the coordinate parameters of the image coordinate system, R is the rotation coefficient, M is the displacement coefficient, a, b, a′, and b′ represent the rotation coefficient parameters when converting the image coordinate system to the mechanical coordinate system, and c and c′ are the displacement coefficient parameters when converting the image coordinate system to the mechanical coordinate system. Once we have at least three sets of corresponding pixel coordinate points and mechanical coordinate points during the application process, we can substitute them into the three-variable linear equation to solve for a, b, c, a′, b′, and c′. X0′ and Y0′ are the coordinate parameters of the first set of mechanical coordinate systems, X1′ and Y1′ are the coordinate parameters of the second set of mechanical coordinate systems, and X2′ and Y2′ are the coordinate parameters of the third set of mechanical coordinate systems. X0 and Y0 are the coordinate parameters of the first set of image coordinate systems, X1 and Y1 are the coordinate parameters of the second set of image coordinate systems, and X2 and Y2 are the coordinate parameters of the third set of image coordinate systems.

[0084] It can be seen from the above formula that to calculate the rotation coefficient R and the displacement coefficient M, it is necessary to have aligned points in the mechanical coordinate system and the image coordinate system, and at least three points are required.

[0085] In this embodiment, the mechanical coordinate system is derived from the robot, and the image coordinate system is derived from the 3D line scan camera. Due to the distance between the robot and the 3D camera during implementation, it is impossible to directly determine the coordinates of the same point in both the mechanical and image coordinate systems. To address this issue, high-precision measurement equipment was introduced to measure the on-site data, resulting in the following formula:

[0086]

[0087] Although both N and M are coefficient matrices, N is a known quantity, representing the distance the linear body travels from the point imaged in the 3D camera image to the point the robot can reach using the workpiece coordinate system during calibration. M is the quantity to be solved. The rotation coefficient R is an unknown quantity to be solved and is obtained through relative calibration. The purpose of relative calibration is to solve for the rotation coefficient and displacement coefficient.

[0088] The calibration method used for 2D camera solutions is to count the number of squares on a checkerboard. Because 3D cameras cannot image a checkerboard, this relative calibration method solves the calibration problem when converting 3D camera images into 2D images for 2D applications.

[0089] Preferably, in step S3 of the embodiment of the present invention, the multiple continuous conveyor belt images are stitched together using an image stitching algorithm, including:

[0090] Obtain the first stitched image and the second stitched image (the images are temporarily stored in the industrial computer memory and assigned values ​​through logic), and obtain the image length and width parameters;

[0091] Generate an output image whose length is the sum of the first stitched image and the second stitched image and whose width is the width of the first stitched image or the second stitched image (the first stitched image and the second stitched image have the same image width);

[0092] Set the image stitching alignment point and copy the data of the first and second stitched images to the output image based on the alignment point coordinates. The data here refers to the information within the image (0 to 255). Based on the alignment point, the first and second stitched images should be converted to the position in the stitched image. Then, the data of the first and second stitched images are directly copied to the position where the stitched images were captured. The stitched image is created based on the length and width information of the first and second stitched images, and then the data of the first and second stitched images are copied to them (after conversion).

[0093] Preferably, in setting the image stitching alignment points, the (0,0) point of the output image corresponds to the (0,0) point of the first stitched image, and the (0, output image width / 2) point of the output image corresponds to the (0,0) point of the second stitched image.

[0094] Specifically, because the position of the carrier on the conveyor belt is uncertain and the 3D camera output image requires a fixed distance parameter, when the 3D camera captures an image, it may only capture a portion of a carrier. If the image is directly processed, the carrier may be missed or incorrectly positioned. To solve this problem, the present invention uses an image stitching algorithm.

[0095] like Figure 3 As shown, Figure 3 (a) in the figure is the first stitched image. Figure 3 (b) in the figure is the second stitched image. Figure 3 (c) in the figure is the spliced ​​image, where Figure 3 The part in the (c) box is the stitched carrier. The two carriers at the bottom of the first stitched image are only part of it, and the two carriers at the top of the second stitched image are the remaining parts of the two carriers at the bottom of the first stitched image. The stitched image after image stitching is as follows Figure 3 As shown in (c), the box is the spliced ​​carrier. It can be seen that the two carriers are perfectly spliced ​​together after splicing. After multiple verifications, the splicing accuracy of this splicing algorithm can reach ±0.005mm.

[0096] Preferably, in step S3 of the embodiment of the present invention, using a multiple filtering algorithm to filter the carriers repeatedly obtained in different conveyor belt images, merging the repeated carriers and obtaining the carrier coordinates includes:

[0097] Step 1: Define the carrier as a class, which includes the carrier's X, Y, angle information, the row and column it belongs to, single grabbing, double grabbing information, carrier spacing information, etc.

[0098] Step 2: Use Blob analysis and circle analysis to process the image to obtain the X, Y, and angle information of the corresponding carrier, the row and column information, single capture and double capture information, carrier spacing information, etc.

[0099] Step 3: Create a carrier queue (used to store carrier objects), create a carrier object and assign values ​​to the attributes of the carrier object.

[0100] Step 4: Put the carrier class into a queue, perform the first carrier filtering according to the coordinate attributes of the carrier (this filtering filters out duplicate carriers), and put the result of this filtering into a queue (this queue is the queue after the first filtering).

[0101] Step 5: Perform a second carrier filtering based on the carrier spacing information (this time, the carriers whose carrier spacing does not meet the crawling conditions are filtered), and put the results of this filtering into a queue (this queue is the queue after the second filtering).

[0102] It should be noted that while the image stitching algorithm solves the problem of incomplete carrier acquisition, a corresponding phenomenon occurs: each image captured by the 3D camera is processed twice (because the first image in each image stitching operation is used as the second image in the next image stitching operation). As a result, some carriers will be found twice by the vision algorithm. To eliminate these carriers found twice, a multi-filter algorithm is used.

[0103] Preferably, after obtaining the carrier coordinates, the carrier coordinates need to be sorted and output to prevent carrier interference when the robot grasps the carrier, including:

[0104] Step 1: Create a carrier object;

[0105] Step 2: Dequeue the queue after the second filtering, and assign the carrier of each dequeue to the newly created carrier object;

[0106] Step 3: divide the carrier into rows according to the Y coordinate information of the carrier;

[0107] Step 4: Sort the carriers after the rows are sorted according to their X coordinates and put them into a queue;

[0108] Step 5: Dequeue the carriers sorted by X coordinates in rows.

[0109] The existing algorithm is a one-dimensional sorting algorithm. This algorithm is a multi-dimensional sorting algorithm derived from on-site conditions. It can sort multi-dimensional data and ensure that the data after sorting will not be disordered.

[0110] Example 2:

[0111] The embodiment of the present invention provides a 3D camera-based follow-up grasping vision guidance system, comprising:

[0112] 3D line scan camera, which is used to scan the conveyor belt after receiving the conveyor belt running signal transmitted by the encoder, and obtain multiple continuous conveyor belt images;

[0113] The coordinate conversion module is used to convert the visual coordinates of the 3D camera into the mechanical coordinates of the robot using the relative calibration method;

[0114] An image carrier processing module is used to stitch multiple continuous conveyor belt images using an image stitching algorithm, filter repeated carriers in different conveyor belt images using a multiple filtering algorithm, merge repeated carriers and obtain carrier coordinates;

[0115] The carrier coordinate output module is used to sort and output the carrier coordinates using a class sorting algorithm;

[0116] The grasping robot is used to grasp the carriers according to the corresponding coordinates of each carrier obtained.

[0117] The operation process of the embodiment of the present invention is as follows:

[0118] Step 1: Set the parameters of the 3D line scan camera and start the vision guidance system.

[0119] Step 2: After the conveyor belt's PLC controller detects that there is material on the conveyor belt, it controls the conveyor belt to run, and the 3D line scan camera obtains the first image with a length of 300 mm (this distance can be flexibly set according to the on-site conditions) (the conveyor belt moves forward, and the encoder sends an encoder signal to the 3D line scan camera. When the conveyor belt reaches 300 mm, the 3D line scan camera will output an image).

[0120] Step 3: The image is uploaded to the vision guidance system via a network cable, and algorithm processing begins. The image is first calibrated and converted using the coordinate conversion module, followed by image stitching using the image carrier processing module (splicing is not performed if this is the first time the image is acquired; it will be performed every time after the first time). Then, carrier search, sorting, and filtering are performed sequentially to obtain the final detection result.

[0121] Step 4: The carrier coordinate output module sends the detection results to the grasping robot through the TCP / IP protocol. The grasping robot processes the results and grasps the carrier based on the received grasping information.

[0122] It should be understood that, although the various steps in the flow charts of the various embodiments of the present invention are shown in sequence according to the instructions of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless otherwise specified in the present invention, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the various embodiments may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0123] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-described methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0124] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A 3D camera-based follow-up grasping visual guidance method, characterized in that: The method comprises: Use relative calibration method to convert the visual coordinates of 3D camera to the mechanical coordinates of robot; After receiving the conveyor belt running signal transmitted by the encoder, the 3D camera scans the conveyor belt and obtains multiple continuous conveyor belt images; Use image stitching algorithm to stitch multiple continuous conveyor belt images, use multiple filtering algorithms to filter repeated carriers in different conveyor belt images, merge repeated carriers and obtain carrier coordinates; The carrier coordinates are sorted and output using a class sorting algorithm, and the robot grabs the carrier based on the obtained carrier coordinates.

2. The 3D camera-based follow-up grasping visual guidance method according to claim 1, characterized in that: The method of converting the visual coordinates of the 3D camera into the mechanical coordinates of the robot using the relative calibration method includes: Place multiple calibration pieces evenly over a length of the conveyor belt; Set the 3D camera output parameters and use the 3D camera to collect calibration images containing all calibration parts; Use image processing algorithms to find the pixel coordinates of the feature points on the calibration part in the calibration image in sequence; The conveyor belt drives multiple calibration pieces to positions within the robot's arm span. The robot gripper moves to the multiple calibration piece positions in sequence and records the robot's mechanical coordinates in sequence to obtain a set of robot coordinates. The robot coordinates are converted according to the relative calibration formula, and the converted robot coordinates and pixel coordinates are input into the calibration tool for calibration. Points with large errors are deleted or modified until the calibration error meets the production requirements.

3. The 3D camera-based follow-up grasping visual guidance method according to claim 2, characterized in that: The relative calibration formula is expressed as: in, is the mechanical coordinate system, is the image coordinate system, R is the rotation coefficient, M is the displacement coefficient, and N is the linear motion distance that the robot uses the workpiece coordinate system to reach the same point in the calibration image.

4. The 3D camera-based follow-up grasping visual guidance method according to claim 1, characterized in that: The method of stitching a plurality of continuous conveyor belt images using an image stitching algorithm includes: Obtain the first stitched image, the second stitched image, and the image length and width parameters; Generate an output image whose length is the sum of the first stitched image and the second stitched image and whose width is the width of the first stitched image or the second stitched image; Set the image stitching alignment point, and copy the data of the first stitching image and the second stitching image to the output image according to the alignment point coordinates.

5. The 3D camera-based follow-up grasping visual guidance method according to claim 4, characterized in that: In setting the image stitching alignment point, the (0,0) point of the output image corresponds to the (0,0) point of the first stitched image, and the (0, output image width / 2) point of the output image corresponds to the (0,0) point of the second stitched image.

6. The 3D camera-based follow-up grasping visual guidance method according to claim 1, characterized in that: The method of filtering the repeated carriers in different conveyor belt images using a multiple filtering algorithm, merging the repeated carriers and obtaining the carrier coordinates includes: Define the carrier as a class, which contains the carrier's X, Y, angle information, the row and column it belongs to, single grasping, double grasping information, and carrier spacing information; Use blob analysis and circle analysis to process the image and obtain the X, Y, and angle information of the corresponding carrier, the row and column information, single capture and double capture information, and carrier spacing information; Create a carrier queue, create a carrier object and assign values ​​to the attributes of the carrier object; The carrier class is queued, and the first carrier filtering is performed according to the carrier's coordinate attributes to filter out duplicate carriers, and the filtering results are queued. A second carrier filtering is performed based on the carrier spacing information to filter out carriers whose carrier spacing does not meet the crawling conditions, and the results of this filtering are queued.

7. The 3D camera-based follow-up grasping visual guidance method according to claim 6, characterized in that: The method of using a class sorting algorithm to sort and output the carrier coordinates includes: Create a carrier object, dequeue the queue after the second filtering, and assign the information of each dequeue carrier to the newly created carrier object; Separate the carrier objects into rows according to their Y coordinate information; Sort the carrier objects after the rows are divided according to the X coordinates, and put the sorted carrier objects into the queue; Dequeue the carrier objects sorted by X coordinates row by row.

8. A 3D camera-based follow-up grasping visual guidance system, characterized in that: The system comprises: 3D line scan camera, which is used to scan the conveyor belt after receiving the conveyor belt running signal transmitted by the encoder, and obtain multiple continuous conveyor belt images; The coordinate conversion module is used to convert the visual coordinates of the 3D camera into the mechanical coordinates of the robot using the relative calibration method; An image carrier processing module is used to stitch multiple continuous conveyor belt images using an image stitching algorithm, filter repeated carriers in different conveyor belt images using a multiple filtering algorithm, merge repeated carriers and obtain carrier coordinates; The carrier coordinate output module is used to sort and output the carrier coordinates using a class sorting algorithm; The grasping robot is used to grasp the carriers according to the corresponding coordinates of each carrier obtained.

Images