A pallet recognition and posture calculation method

Abstract

The present invention discloses a method for pallet recognition and posture calculation, comprising: 1. using a forklift robot's own time-of-flight camera to acquire a camera initial point cloud, and performing matrix transformation to obtain a world initial point cloud in the forklift robot's world coordinate system; 2. preprocessing the world initial point cloud to obtain a world point cloud, and converting the world point cloud into a grayscale image through projection, matrix transformation, and dimensionality reduction; 3. performing template matching on the grayscale image to obtain a pallet confidence frame; 4. extracting the pallet end face point cloud and the pallet leg point cloud within the confidence frame; 5. fitting the pallet end face equation using the pallet end face point cloud, and calculating the pallet socket midpoint coordinates using the pallet leg point cloud, ultimately obtaining the complete pallet posture, including the pallet yaw angle, pallet midpoint coordinates, and pallet socket spacing. The present invention can accurately and quickly calculate pallet posture in complex environments, thereby enabling the forklift robot to perform automated insertion and extraction tasks in a working environment.

Description

Technical Field

[0001] The present invention relates to the field of pallet recognition, detection and posture calculation. Specifically, it is a method that enables a forklift robot to automatically recognize a pallet and accurately calculate the pallet's posture in different working environments, thereby guiding the forklift robot to accurately insert the pallet. Technical Background

[0002] Pallets, as core transfer containers in production processes and warehousing and logistics, are key pick-up targets for intelligent forklift robots. With the increasing development of automated factories, the application scope of forklift robots is expanding, and the penetration rate of unmanned operations is also increasing. Forklift robots are widely deployed in scenarios such as repetitive handling tasks, high-intensity handling work, harsh working environments, and those with high environmental requirements. To achieve a high level of automation for forklift robots, it is crucial to enable them to automatically identify pallets and accurately calculate their position and posture (pose) to automatically complete insertion and removal operations. However, due to complex and changing environmental factors, the previous forklift robot's ability to recognize pallets through images has been unable to meet the accuracy and speed requirements of practical applications. Therefore, developing a real-time pallet detection and pose calculation method that combines sufficient accuracy with good real-time performance has become a key issue that needs to be addressed.

[0003] Traditional pallet recognition methods mainly include the following: one is to identify the pallet by capturing its color or geometric features in RGB images; another method is to attach labels such as QR codes to the pallet and then indirectly identify the pallet by identifying these labels; and another method is to use laser sensors to obtain point cloud data of the pallet and use point cloud template matching technology based on this point cloud data to complete pallet recognition.

[0004] The three traditional pallet recognition methods mentioned above all have certain limitations. The first method is highly dependent on lighting conditions, and sufficient lighting is often not guaranteed in the working environment of forklift robots, which leads to a significant decrease in recognition accuracy. Although the second method has achieved pallet recognition to a certain extent, it is still guided in nature and has not reached the level of full automation. In the complex environment where forklift robots are actually used, markers such as QR codes are easily damaged, and long-term use is more likely to cause problems in pallet recognition. Although the third method is based on point cloud data obtained by laser sensors for recognition, due to the huge amount of point cloud data, the computing resources required for point cloud template matching are very huge. However, in actual applications, forklift robots often lack sufficient computing power resources, resulting in a long recognition process and poor real-time performance, which makes it difficult to meet the needs of actual handling work. Summary of the Invention

[0005] The present invention aims to address the deficiencies of the above-mentioned prior art and proposes a pallet recognition and posture calculation method, in order to achieve high-precision and high-speed pallet recognition and posture detection, thereby realizing the goal of fully automated handling operations of forklift robots.

[0006] In order to solve the technical problem, the present invention adopts the following technical solution:

[0007] The present invention provides a method for pallet recognition and posture calculation, which is characterized by being applied to forklift robot handling operations and is performed in the following steps:

[0008] Step 1: Establish the camera coordinate system and the world coordinate system of the forklift robot;

[0009] Step 2: The original 3D point cloud containing the pallet features in the world coordinate system of the forklift robot After preprocessing, the preprocessed 3D point cloud containing the pallet features is obtained , and record The serial numbers corresponding to all points in ;

[0010] Step 3: Define K projection planes;

[0011] Step 4: Initialize k=1;

[0012] Step 5: Project to the kth projection plane, then use the dimensionality reduction algorithm to convert the projected points into a two-dimensional point set, and then use the overlapping point screening algorithm to remove all overlapping points in the two-dimensional point set to obtain a two-dimensional point set without overlap ;

[0013] Step 6: Use the conversion algorithm to convert the two-dimensional point set without overlapping points Convert to target grayscale image ;

[0014] Step 7: According to the parameters of the pallet used by the forklift robot, Make a proportional pallet template M with a height of H and a width of W. The left and right holes of the pallet inserted by the forklift robot are marked as ;

[0015] Step 8: Use an image template matching method to obtain the target grayscale image If the pallet template M fails to match, k+1 is assigned to k. If k≤K, return to step 5. If k>K, determine There is no pallet in the table; if the pallet template M is matched successfully, go to step 9;

[0016] Step 9: Combine the matching result of the pallet template M with the pallet template to obtain the pallet pose.

[0017] The method for pallet identification and posture calculation according to the present invention is also characterized in that step 1 comprises:

[0018] The midpoint of the contact surface between the forklift robot and the forklift robot body is taken as the origin , taking the forklift robot's forward direction as The positive direction of the axis is perpendicular to the ground and upward Positive axis direction, determined by the right-hand rule Axis, thereby establishing the world coordinate system of the forklift robot ;

[0019] Take the TOF camera light source as , the TOF camera is directly in front of Axis positive direction, TOF camera vertically downward The positive direction of the axis is determined by the right-hand rule Axis, thus establishing the camera coordinate system .

[0020] Furthermore, the specific steps of the pre-processing method in step 2 include:

[0021] Step 2.1: The forklift robot uses its own TOF camera to obtain the original 3D point cloud containing the pallet features. After coordinate transformation, the original three-dimensional point cloud containing the pallet features in the world coordinate system of the forklift robot is obtained. ;

[0022] Step 2.2: Use a straight-through filter to Perform background removal and retain the world coordinate system Axis origin The distance range is The point cloud inside is obtained to obtain the filtered three-dimensional point cloud containing the pallet features. ,in, is the closest point of the pallet placement range, The farthest point of the pallet placement range;

[0023] Step 2.3: Use voxel filtering to Sampling is performed to obtain a downsampled 3D point cloud containing pallet features. ;

[0024] Step 2.4: Use statistical filter to After performing noise reduction to remove outlier point clouds, the pre-processed 3D point cloud containing the pallet features is obtained. ,in, express The nth point in express Total number of midpoints, recorded exist The sequence number in is recorded as .

[0025] Furthermore, the step 3 includes:

[0026] Define perpendicular to K projection planes of the plane The distance from the TOF camera is And the world coordinate system of the forklift robot The axis angles are plane, where Represents the projection plane With the world coordinate system The angle between the axes, Represents the world coordinate system The axis angle is The projection plane of , K represents the total number of projection planes.

[0027] Furthermore, the step 5 includes:

[0028] Step 5.1, calculation Each point in distance ,in, express The nth point in arrive distance;

[0029] Step 5.2, All points in the projection are And rotate and translate all the projected points so that all the projected points move to On the plane, the kth transition 3D point cloud is obtained ;

[0030] Step 5.3, The three-dimensional coordinates of all points in , converted into a two-dimensional point set The two-dimensional coordinate set of ,in, express The three-dimensional coordinates of the nth point in , express The two-dimensional coordinates of the nth point in ;

[0031] Step 5.4, follow , ,Will The two-dimensional coordinates of all points are proportionally reduced times, and The center of the image is translated to the coordinates , and then after translation The two-dimensional coordinates of all points in are rounded to obtain the preprocessed two-dimensional point set , whose coordinates are ,in, Respectively The maximum and minimum values ​​in Respectively The maximum and minimum values ​​in , S is a positive integer, Represents the two-dimensional coordinates of the nth point after preprocessing;

[0032] Step 5.5: If There are coincident points with the same two-dimensional coordinates in , then all the coincident points are Get the serial number in All coincident points of corresponding serial numbers in the plane The distance between the two points is kept, and the point with the maximum distance is retained. Delete the remaining coincident points to obtain a two-dimensional point set without overlap The two-dimensional coordinate set of ,in, for Middle The coordinates of the points, express The total number of midpoints.

[0033] Furthermore, the step 6 includes:

[0034] Step 6.1, The center is located in the two-dimensional coordinate The nth point at The sequence number in the matrix Middle Rank The element value of the column will be The center is located in the two-dimensional coordinate The nth point at The serial number in the corresponding point to The distance as a distance matrix Middle Rank The element values ​​of the column, thus obtaining two matrices of dimension S×S and distance matrix ;

[0035] Step 6.2: Calculate using formula (1) Middle Gray value of a point , and used as the target grayscale image Middle Rank The element value of the column and divide The element values ​​of the remaining positions other than the two-dimensional coordinates are set to 0, thus obtaining a target grayscale image with a dimension of S×S ;

[0036] V k , n ′ = int [ ( l ″ k , n ′ − l k ,min ) / ( l k ,max − l k ,min ) × Gray ] (1)

[0037] In formula (1), Indicates rounding operation, for Located in the middle Rank The element value of the column, is a matrix The minimum value in is a matrix The maximum value in is the maximum grayscale value.

[0038] Furthermore, the image template matching method in step 8 includes:

[0039] Step 8.1: Combine the pallet template M and the target grayscale image , use formula (4) to calculate the upper left corner vertex of M and Coordinates on Similarity score when overlapped ;

[0040] (2)

[0041] In formula (2), is the coordinate in M The gray value at yes The standardized and normalized values, yes Center coordinates The gray value at yes Standardized and normalized values; is the coordinate in R The gray value at Coordinates in M The grayscale value at is standardized and normalized, yes Center coordinates The grayscale value at is standardized and normalized;

[0042] Step 8.2: If Exceeding the preset value When , it means that on the k-th projection plane The pallet on the is successfully identified, and the k-th recognition result is output, including: the confidence box of the pallet location The upper left corner is Coordinates on and a confidence box for where the pallet is located The height H and width W of ; otherwise, it represents the k-th projection plane Identification failed.

[0043] Furthermore, the step 9 includes:

[0044] Step 9.1: Extract the confidence box of the tray location based on M The local two-dimensional pixel coordinates of the pallet end surface area corresponding to , the local two-dimensional pixel coordinates corresponding to the end surface area of ​​the tray are obtained The corresponding 3D point cloud , then, from Extract the initial point cloud of the pallet end surface ;

[0045] Step 9.2: Use the least squares method to calculate the initial point cloud Perform fitting to obtain the pallet end face equation , and thus calculate the yaw angle of the pallet ;

[0046] Step 9.3, screening Equations of the center and tray end faces The distance between the corresponding planes is less than the threshold Point cloud , extraction tray Point cloud of the middle pallet legs The relative position of the left jack of the tray is obtained Local point clouds of the left leg columns on both sides , right leg column local point cloud and the right jack of the tray Local point clouds of the left leg columns on both sides , right leg column local point cloud ;

[0047] Step 9.4, according to Calculate left jack The midpoint of the left leg column point cloud;

[0048] according to Calculate left jack The midpoint of the right leg column point cloud;

[0049] According to the hole The midpoint of the left leg column point cloud and the midpoint of the right leg column point cloud are used to calculate the left socket The midpoint coordinates of

[0050] according to Calculate right jack The midpoint of the left leg column point cloud;

[0051] according to Calculate right jack The midpoint of the right leg column point cloud;

[0052] According to the hole The midpoint of the left leg column point cloud and the midpoint of the right leg column point cloud are used to calculate the right socket The midpoint coordinates of

[0053] Step 9.5, according to the left jack The midpoint coordinates and right jack The midpoint coordinates of The midpoint distance of

[0054] Step 9.6, from the right jack The midpoint coordinates of the left jack The midpoint coordinates of The midpoint distance and the yaw angle of the pallet The posture of the tray.

[0055] The electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the tray identification and posture calculation method, and the processor is configured to execute the program stored in the memory.

[0056] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the pallet identification and posture calculation method when the computer program is executed by a processor.

[0057] Compared with the prior art, the beneficial effects of the present invention are embodied in:

[0058] 1. By processing point cloud data, the present invention effectively avoids the influence of lighting factors compared to processing image data, and significantly improves the robustness of pallet recognition.

[0059] 2. The present invention projects point cloud data onto a set plane and assigns corresponding grayscale values ​​to the projected points based on their distance from the plane, thereby significantly enhancing the contrast between the pallet and its surroundings and further improving the accuracy of pallet recognition.

[0060] 3. The present invention converts three-dimensional point cloud data into two-dimensional image data and performs template matching. This method effectively reduces the computational complexity and improves the real-time performance of pallet recognition, enabling the forklift robot to quickly and accurately complete pallet recognition and posture calculation.

[0061] 4. The present invention recognizes pallets through image template matching, without any marking on the pallets, and realizes fully automatic pallet recognition and posture calculation.

[0062] 5. The present invention simplifies the template production so that it is not limited by the type of pallet and is applicable to pallets of different materials, different numbers of jacks and different shapes. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a flow chart of the pallet identification and posture calculation of the present invention;

[0064] Figure 2 Schematic diagram of coordinate conversion of the present invention;

[0065] Figure 3 A schematic diagram of a practical application of the test scenario of the present invention;

[0066] Figure 4 Schematic diagram of template matching results of the present invention;

[0067] Figure 5 Schematic diagram of the pallet template of the present invention. DETAILED DESCRIPTION

[0068] In this embodiment, a pallet recognition and posture calculation method is applied to forklift robot handling operations, such as Figure 1 As shown, the following steps are followed:

[0069] Step 1: Take the midpoint of the contact surface between the forklift robot and the forklift robot body as the origin , taking the forklift robot's forward direction as The positive direction of the axis is perpendicular to the ground and upward Positive axis direction, determined by the right-hand rule Axis, thereby establishing the world coordinate system of the forklift robot ;

[0070] Take the TOF camera light source as , the TOF camera is directly in front of Axis positive direction, TOF camera vertically downward The positive direction of the axis is determined by the right-hand rule Axis, thus establishing the camera coordinate system ; The coordinate relationship is as follows Figure 2 shown.

[0071] Step 2: The forklift robot uses its own TOF camera to obtain the original 3D point cloud containing the pallet features. After coordinate transformation, the original three-dimensional point cloud containing the pallet features in the world coordinate system of the forklift robot is obtained. The coordinate transformation in this embodiment is to calculate the rotation matrix and translation matrix based on the forklift's own values ​​and the camera installation position to obtain the transformation matrix.

[0072] Step 3: Use a straight-through filter to Perform background removal and retain the world coordinate system Axis origin The distance range is The point cloud inside is obtained to obtain the filtered three-dimensional point cloud containing the pallet features. ,in, is the closest point of the pallet placement range, The farthest point of the pallet placement range;

[0073] Step 4: Use voxel filtering to Sampling is performed to obtain a downsampled 3D point cloud containing pallet features. ;

[0074] Step 5: Use statistical filter to After performing noise reduction to remove outlier point clouds, the processed 3D point cloud containing the pallet features is obtained. ,in, express The nth point in express Total number of midpoints; record exist The sequence number in is recorded as ;

[0075] Step 6. Define perpendicular to K projection planes of the plane The distance from the TOF camera is And the world coordinate system of the forklift robot The axis angles are plane, where Represents the projection plane With the world coordinate system The angle between the axes, Represents the world coordinate system The axis angle is The projection plane, K represents the total number of projection planes;

[0076] In this embodiment, the projection plane and Axle angle setting and sorting: Set the maximum angle and minimum angle based on the possible yaw angle range of the pallet in actual applications, divide the range of the maximum angle and the minimum angle into K equal angles, and then sort them according to the actual probability.

[0077] Step 7: Initialize k=1;

[0078] Step 8. Calculation Each point in distance ,in, express The nth point in arrive distance;

[0079] Step 9: Use equations (1) and (2) to convert All points in the projection are And rotate and translate all the projected points so that all the projected points move to On the plane, the kth transition 3D point cloud is obtained ;

[0080] (1)

[0081] (2)

[0082] In formula (1) and formula (2), yes The projection coefficient, yes The three-dimensional coordinates of the nth point in , yes The plane equation, yes The nth point is projected on The coordinates on the plane; A, B, C, and D are the four coefficients of the plane equation respectively;

[0083] Rotate and translate the projection to The plane can keep its original shape characteristics unchanged and The axis coordinates are reset to zero and converted into an approximate two-dimensional image.

[0084] Step 10: The three-dimensional coordinates of all points in , converted into a two-dimensional point set The two-dimensional coordinate set of ,in, express The three-dimensional coordinates of the nth point in , express The two-dimensional coordinates of the nth point in ;

[0085] Step 11: Follow , ,Will The two-dimensional coordinates of all points are proportionally reduced times, and The center of the image is translated to the coordinates , and then after translation The two-dimensional coordinates of all points in are rounded to obtain the preprocessed two-dimensional point set , whose coordinates are ,in, Respectively The maximum and minimum values ​​in Respectively The maximum and minimum values ​​in , S is a positive integer, Represents the two-dimensional coordinates of the nth point after preprocessing.

[0086] Step 12: If There are coincident points with the same two-dimensional coordinates in , then all the coincident points are Get the serial number in All coincident points of corresponding serial numbers in the plane The distance between the two points is kept, and the point with the maximum distance is retained. Delete the remaining coincident points to obtain a two-dimensional point set without overlap Two-dimensional coordinate set ,in, for Middle The coordinates of the points, express The total number of midpoints.

[0087] In this embodiment, the principle of filtering out overlapping points is as follows:

[0088] Because in actual pallet recognition work, obstacles between the pallet end face and the camera have been processed in the obstacle detection stage, when two or more point clouds are projected to the same pixel point, the distance between the pallet end face point cloud and the projection plane must be the larger value.

[0089] Step 13: The center is located in the two-dimensional coordinate The nth point at The sequence number in the matrix Middle Rank The element value of the column will be The center is located in the two-dimensional coordinate The nth point at The serial number in the corresponding point to The distance as a distance matrix Middle Rank The element values ​​of the column, thus obtaining two matrices of dimension S×S and distance matrix ;

[0090] Step 14: Calculate using formula (3) Middle Gray value of a point , and used as the target grayscale image Middle Rank The element value of the column and divide The element values ​​of the remaining positions other than the two-dimensional coordinates are set to 0, thus obtaining the target grayscale image of dimension S×S ;

[0091] V k , n ′ = i n t [ ( l ″ k , n ′ − l k ,min ) / ( l k ,max − l k ,min ) × Gray ] (3)

[0092] In formula (3), Indicates rounding operation, for Located in the middle Rank The element value of the column, is a matrix The minimum value in is a matrix The maximum value in is the maximum gray value; Indicates rounding down;

[0093] The environmental fusion process in this embodiment is as follows: when the pallet point cloud and the background point cloud are projected on the same plane, if a binary image is generated, the contour will be lost, causing the pallet projection and the background projection to be fused into a group of pixels, thereby greatly reducing the accuracy of subsequent template matching.

[0094] Step 15: According to the parameters of the pallet used by the forklift robot, Make a proportional pallet template M with a height of H and a width of W. The left and right holes of the pallet inserted by the forklift robot are marked as ;

[0095] The template making process in this embodiment is as follows: measure the width and height of the tray, the width and height of the jack, and the coordinates of the upper left vertex of the jack on the end face of the tray, and then scale down the measured values ​​to make a binary image. The grayscale value of the tray end face is set to 255, and the grayscale value of the jack is set to 0. Figure 5 The black and white part of the image.

[0096] Step 16: Combine the pallet template M and the target grayscale image , use formula (4) to calculate the upper left corner vertex of M and Coordinates on Similarity score when overlapped ;

[0097] (4)

[0098] In formula (4), is the coordinate in M The gray value at yes The standardized and normalized values, yes Center coordinates The gray value at yes Standardized and normalized values; is the coordinate in R The gray value at Coordinates in M The grayscale value at is standardized and normalized, yes Center coordinates The grayscale value at is standardized and normalized;

[0099] Step 17: If Exceeding the preset value When , it means that on the k-th projection plane The pallet on the is successfully identified, and the k-th recognition result is output, including: the confidence box of the pallet location The upper left corner is Coordinates on and a confidence box for where the pallet is located The height H and width W of , and execute step 18; otherwise, it represents the k-th projection plane After the recognition fails and k+1 is assigned to k, return to step 8 and execute sequentially until k>K. If the Kth projection plane If the recognition still fails, then The target tray was not recognized.

[0100] Step 18: Extract the confidence box of the tray location based on M The local two-dimensional pixel coordinates of the pallet end surface area corresponding to , the local two-dimensional pixel coordinates corresponding to the end surface area of ​​the tray are obtained The corresponding 3D point cloud , then, from Extract the initial point cloud of the pallet end surface ;

[0101] Step 19: Use the least squares method to calculate the initial point cloud Perform fitting and obtain the tray end face equation from formula (5): , and thus calculate the yaw angle of the pallet ;

[0102] (5)

[0103] In formula (5), is the equation of the tray end face The normal vector of is the yaw angle of the tray.

[0104] Step 20: Screening Equations of the center and tray end faces The distance between the corresponding planes is less than the threshold Point cloud , extraction tray Point cloud of the middle pallet legs The relative position of the left jack of the tray is obtained Local point clouds of the left leg columns on both sides , right leg column local point cloud and the right jack of the tray Local point clouds of the left leg columns on both sides , right leg column local point cloud .

[0105] Step 21: According to Calculate left jack The midpoint of the left leg column point cloud;

[0106] according to Calculate left jack The midpoint of the right leg column point cloud;

[0107] According to the hole The midpoint of the left leg column point cloud and the midpoint of the right leg column point cloud are used to calculate the left socket The midpoint coordinates of

[0108] Step 22: According to Calculate right jack The midpoint of the left leg column point cloud;

[0109] according to Calculate right jack The midpoint of the right leg column point cloud;

[0110] According to the hole The midpoint of the left leg column point cloud and the midpoint of the right leg column point cloud are used to calculate the right socket The midpoint coordinates of .

[0111] Step 23: According to the left jack The midpoint coordinates and right jack The midpoint coordinates of The midpoint distance of

[0112] Step 24: From the right jack The midpoint coordinates of the left jack The midpoint coordinates of The midpoint distance and the yaw angle of the pallet The posture of the tray.

[0113] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0114] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

[0115] Pallet recognition and posture calculation method test:

[0116] exist Figure 3 In the process shown, first a frame of point cloud captured by the camera is Figure 3 Part (a) of the figure is preprocessed, including removing out-of-range point clouds, implementing point cloud downsampling, and removing outliers to effectively reduce the number of point clouds, thereby obtaining the processed image as shown in the figure. Figure 3 Then, these point clouds are projected onto a specific projection plane to generate images such as Figure 3 In part (c), during this process, it is found that the tray and the background have a high degree of integration. To this end, the grayscale value is assigned to the projection point according to the distance between the projection point and the projection plane, and then the image is generated as follows: Figure 3 Then, template matching was performed and the matching results were as follows: Figure 4 As shown. The pose calculation visualization is shown in the figure Figure 3As shown in part (e), the green rectangle is the fitted pallet end face, and the red point set is the local point set of the pallet end face and the local point set of the pallet legs. A significant advantage of point cloud data is that it is not affected by lighting conditions and exhibits strong lighting robustness. By preprocessing the point cloud, not only is the time consumption of the projection process significantly reduced, but the key shape features of the pallet are also successfully retained. In addition, the method of converting distance information into grayscale values ​​ensures sufficient contrast between the pallet and the background, thereby further highlighting the features of the pallet and greatly improving the accuracy of template matching. By extracting the pallet leg point cloud from the template matching result, the pallet end face parameters and the coordinates of the midpoint of the pallet socket can be effectively and accurately calculated, and then the complete position and posture of the pallet can be obtained.

[0117] This test fully demonstrated the reliability of this pallet recognition and pose calculation method. Even in complex and changing working environments, the forklift robot can achieve accurate pallet recognition and pose calculation using this method. Its performance is not limited by lighting conditions, and it maintains excellent performance even at night or in low-light environments. Furthermore, this method has a relatively low computational load and can run at high speed on mobile devices with limited computing resources, ensuring that the forklift robot can complete its handling tasks quickly and smoothly. Experimental results demonstrate that this method fully meets the pallet recognition and pose calculation needs of the forklift robot and has never failed under the preset conditions, further verifying its excellent stability.

Claims

1. A pallet recognition and posture calculation method is characterized by being applied to forklift robot handling operations and is performed according to the following steps: Step 1: Establish the camera coordinate system and the world coordinate system of the forklift robot; Step 2: The original 3D point cloud containing the pallet features in the world coordinate system of the forklift robot After preprocessing, the preprocessed 3D point cloud containing the pallet features is obtained , and record The serial numbers corresponding to all points in ; Step 3: Define K projection planes; Step 4: Initialize k=1; Step 5: Project to the kth projection plane, then use the dimensionality reduction algorithm to convert the projected points into a two-dimensional point set, and then use the overlapping point screening algorithm to remove all overlapping points in the two-dimensional point set to obtain a two-dimensional point set without overlap ; Step 6: Use the conversion algorithm to convert the two-dimensional point set without overlapping points Convert to target grayscale image ; Step 6.1, The center is located in the two-dimensional coordinate The nth point at The sequence number in the matrix Middle Rank The element value of the column will be The center is located in the two-dimensional coordinate The nth point at The serial number in the corresponding point to The distance as a distance matrix Middle Rank The element values ​​of the column, thus obtaining two matrices of dimension S×S and distance matrix ; Step 6.2: Calculate using formula (1) Middle Gray value of a point , and used as the target grayscale image Middle Rank The element value of the column and divide The element values ​​of the remaining positions other than the two-dimensional coordinates are set to 0, thus obtaining a target grayscale image with a dimension of S×S ; (1) In formula (1), Indicates rounding operation, for Located in the middle Rank The element value of the column, is a matrix The minimum value in is a matrix The maximum value in is the maximum gray value; Step 7: According to the parameters of the pallet used by the forklift robot, Make a proportional pallet template M with a height of H and a width of W. The left and right holes of the pallet inserted by the forklift robot are marked as ; Step 8: Use an image template matching method to obtain the target grayscale image If the pallet template M fails to match, k+1 is assigned to k. If k≤K, return to step 5. If k>K, determine There is no pallet in the table; if the pallet template M is matched successfully, go to step 9; Step 9: Combine the matching result of the pallet template M with the pallet template to obtain the pallet pose.

2. A pallet recognition and posture calculation method according to claim 1, characterized in that: The step 1 comprises: The midpoint of the contact surface between the forklift robot and the forklift robot body is taken as the origin , taking the forklift robot's forward direction as The positive direction of the axis is perpendicular to the ground and upward Positive axis direction, determined by the right-hand rule Axis, thereby establishing the world coordinate system of the forklift robot ; Take the TOF camera light source as , the TOF camera is directly in front of Axis positive direction, TOF camera vertically downward The positive direction of the axis is determined by the right-hand rule Axis, thus establishing the camera coordinate system .

3. A pallet recognition and posture calculation method according to claim 2, characterized in that: The specific steps of the pre-processing method in step 2 include: Step 2.1: The forklift robot uses its own TOF camera to obtain the original 3D point cloud containing the pallet features. After coordinate transformation, the original three-dimensional point cloud containing the pallet features in the world coordinate system of the forklift robot is obtained. ; Step 2.2: Use a straight-through filter to Perform background removal and retain the world coordinate system Axis origin The distance range is The point cloud inside is obtained to obtain the filtered three-dimensional point cloud containing the pallet features. ,in, is the closest point of the pallet placement range, The farthest point of the pallet placement range; Step 2.3: Use voxel filtering to Sampling is performed to obtain a downsampled 3D point cloud containing pallet features. ; Step 2.4: Use statistical filter to After performing noise reduction to remove outlier point clouds, the pre-processed 3D point cloud containing the pallet features is obtained. ,in, express The nth point in express Total number of midpoints, recorded exist The sequence number in is recorded as .

4. A pallet recognition and posture calculation method according to claim 3, characterized in that: The step 3 comprises: Define perpendicular to K projection planes of the plane The distance from the TOF camera is And the world coordinate system of the forklift robot The axis angles are plane, where Represents the projection plane With the world coordinate system The angle between the axes, Represents the world coordinate system The axis angle is The projection plane of , K represents the total number of projection planes.

5. A pallet recognition and posture calculation method according to claim 4, characterized in that: The step 5 comprises: Step 5.1, calculation Each point in distance ,in, express The nth point in arrive distance; Step 5.2, All points in the projection are And rotate and translate all the projected points so that all the projected points move to On the plane, the kth transition 3D point cloud is obtained ; Step 5.3, The three-dimensional coordinates of all points in , converted into a two-dimensional point set The two-dimensional coordinate set of ,in, express The three-dimensional coordinates of the nth point in , express The two-dimensional coordinates of the nth point in ; Step 5.4, follow , ,Will The two-dimensional coordinates of all points are proportionally reduced times, and The center of the image is translated to the coordinates , and then after translation The two-dimensional coordinates of all points in are rounded to obtain the preprocessed two-dimensional point set , whose coordinates are ,in, Respectively The maximum and minimum values ​​in Respectively The maximum and minimum values ​​in , S is a positive integer, Represents the two-dimensional coordinates of the nth point after preprocessing; Step 5.5: If There are coincident points with the same two-dimensional coordinates in , then all the coincident points are Get the serial number in All coincident points of corresponding serial numbers in the plane The distance between the two points is kept, and the point with the maximum distance is retained. Delete the remaining coincident points to obtain a two-dimensional point set without overlap The two-dimensional coordinate set of ,in, for Middle The coordinates of the points, express The total number of midpoints.

6. A pallet recognition and posture calculation method according to claim 5, characterized in that: The image template matching method in step 8 includes: Step 8.1: Combine the pallet template M and the target grayscale image , use formula (4) to calculate the upper left corner vertex of M and Coordinates on Similarity score when overlapped ; (2) In formula (2), is the coordinate in M The gray value at yes The standardized and normalized values, yes Center coordinates The gray value at yes Standardized and normalized values; is the coordinate in R The gray value at Coordinates in M The grayscale value at is standardized and normalized, yes Center coordinates The grayscale value at is standardized and normalized; Step 8.2: If Exceeding the preset value When , it means that on the k-th projection plane The pallet on the is successfully identified, and the k-th recognition result is output, including: the confidence box of the pallet location The upper left corner is Coordinates on and a confidence box for where the pallet is located The height H and width W of ; otherwise, it represents the k-th projection plane Identification failed.

7. A pallet recognition and posture calculation method according to claim 6, characterized in that: The step 9 comprises: Step 9.1: Extract the confidence box of the tray location based on M The local two-dimensional pixel coordinates of the pallet end surface area corresponding to , the local two-dimensional pixel coordinates corresponding to the end surface area of ​​the tray are obtained The corresponding 3D point cloud , then, from Extract the initial point cloud of the pallet end surface ; Step 9.2: Use the least squares method to calculate the initial point cloud Perform fitting to obtain the pallet end face equation , and thus calculate the yaw angle of the pallet ; Step 9.3, screening Equations of the center and tray end faces The distance between the corresponding planes is less than the threshold Point cloud , extraction tray Point cloud of the middle pallet legs The relative position of the left jack of the tray is obtained Local point clouds of the left leg columns on both sides , right leg column local point cloud and the right jack of the tray Local point clouds of the left leg columns on both sides , right leg column local point cloud ; Step 9.4, according to Calculate left jack The midpoint of the left leg column point cloud; according to Calculate left jack The midpoint of the right leg column point cloud; According to the hole The midpoint of the left leg column point cloud and the midpoint of the right leg column point cloud are used to calculate the left socket The midpoint coordinates of according to Calculate right jack The midpoint of the left leg column point cloud; according to Calculate right jack The midpoint of the right leg column point cloud; According to the hole The midpoint of the left leg column point cloud and the midpoint of the right leg column point cloud are used to calculate the right socket The midpoint coordinates of Step 9.5, according to the left jack The midpoint coordinates and right jack The midpoint coordinates of The midpoint distance of Step 9.6, from the right jack The midpoint coordinates of the left jack The midpoint coordinates of The midpoint distance and the yaw angle of the pallet The posture of the tray.

8. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the pallet identification and posture calculation method according to any one of claims 1 to 7, and the processor is configured to execute the program stored in the memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the pallet identification and posture calculation method according to any one of claims 1 to 7 are executed.

Images