A method and system for pallet recognition and positioning based on a 3D sensor

Abstract

The present invention provides a pallet recognition and positioning method and system based on a 3D sensor. The steps of the recognition method include: S1 determining the parameters of each end face of the target pallet for recognition and establishing a feature template; S2 establishing a target point cloud based on the sensing data collected by the 3D sensor, and preprocessing the target point cloud. After filtering the ground point cloud, the point clouds of the same end face of the target pallet are merged to obtain a planar point cloud block; S3 converting the planar point cloud block into a 2D plane in the camera coordinate system through the external parameters of the 3D sensor; S4 establishing a coding map and placing the 2D plane therein, constructing a double-line sliding window with a preset feature recognition distance in the coding map and synchronously moving according to a preset step length to perform line segment fitting scanning on the points on the 2D plane encountered on the path. When it is determined that the current line lengths respectively fitted by the double-line sliding windows both conform to the feature template, the recognition result corresponding to the feature module is output. Thereby, a variety of 3D sensors are made universal to improve versatility, and at the same time, the requirements for computing performance are reduced.

Description

Technical Field

[0001] The present invention relates to robot vision positioning technology, and in particular to a method and system for identifying and positioning the shape of a pallet surface based on data collected by a traditional 3D sensor. Background Art

[0002] The robot referred to in the present invention is a mobile robot capable of automatic operation, such as a wheeled robot, which can be classified into: cleaning robot, disinfection robot, inspection robot, handling robot, etc. according to different working attributes.

[0003] In the field of industrial applications, automated robots with moving and handling functions are gradually replacing manual intervention in the existing production system to achieve tasks such as material handling and picking. However, the reason why these robots can achieve automated control mainly depends on the continuously developing robot recognition and positioning technology.

[0004] For example, an automatic forklift robot transfers goods by picking up a pallet. If the specific pose of the pallet is not known in advance, it is very easy to fail in the picking process. Therefore, how to quickly identify and locate the position of such a pallet is the goal that the present field has been iteratively improving.

[0005] In the current existing technologies, relatively mature technologies usually use an RGB-D camera to determine the position of the pallet according to features. This technology is also a widely used technology at present, but the problem with this technology is that it is not highly versatile and depends on specific depth camera devices.

[0006] On the other hand, the technology based on sample learning is also a relatively popular technology at present, also called deep learning. Deep learning can achieve a very high accuracy in recognition, but it has high requirements for computing performance. Summary of the Invention

[0007] Therefore, the main object of the present invention is to provide a method and system for pallet recognition and positioning based on a 3D sensor, so as to be applicable to a variety of 3D sensors to improve versatility and at the same time reduce the requirements for computing performance.

[0008] To achieve the above object, according to one aspect of the present invention, there is provided a method for pallet recognition based on a 3D sensor, the steps of which include:

[0009] S1 Determine the parameters of each end face of the target pallet for recognition, and establish a feature template;

[0010] S2 Based on the sensing data collected by the 3D sensor, establish a target point cloud, and preprocess the target point cloud. After filtering out the ground point cloud, merge the point clouds of the same end face of the target pallet to obtain a planar point cloud block;

[0011] S3 converts the planar point cloud block into a 2D plane in the camera coordinate system through the external parameters of the 3D sensor;

[0012] S4 constructs an encoding map based on the 3D sensor resolution, places the 2D plane therein, constructs a double-line sliding window for identifying the preset feature distance in the encoding map, and synchronously moves according to the preset step size to perform line segment fitting scanning on the points on the 2D plane encountered on the path. When it is determined that the current line lengths respectively fitted by the double-line sliding windows both conform to the feature template, the recognition result corresponding to the feature module is output.

[0013] In a possible preferred embodiment, the feature template includes: a first sub-feature, which is a continuous line segment feature and its length exists within the end face parameter range of the target pallet; a second sub-feature, which is a plurality of spaced line segment features, and the lengths and spacing distances of each spaced line segment exist within the end face parameter range of the target pallet.

[0014] In a possible preferred embodiment, the steps of preprocessing the target point cloud in step S2 include:

[0015] S21 performs filtering processing on the target point cloud, converts the target point cloud to the robot coordinate system according to the external parameters of the 3D sensor to obtain the corresponding point cloud coordinates, and thus filters out the mismatched target point cloud with reference to the height parameter of the target pallet;

[0016] S22 performs statistical filtering processing on the target point cloud processed in step S21 to remove outliers.

[0017] In a possible preferred embodiment, the steps of filtering out the ground point cloud in step S2 include:

[0018] S23 randomly extracts multiple points from the target point cloud multiple times respectively to fit out multiple reference planes;

[0019] S24 counts the number of corresponding points within the tolerance distance range between each reference plane and all the points of the target point cloud;

[0020] S25 selects the reference plane with the largest number of corresponding points as the ground, attributes all the points on this reference plane to the ground component to be excluded, and the remaining points are attributed to the object component.

[0021] In a possible preferred embodiment, the steps of merging the point clouds of the same end face of the target pallet to obtain a planar point cloud block in step S2 include:

[0022] S26 randomly selects a seed point from the object component point cloud, and determines whether the seed point and the non-seed points around the seed point are in the same plane, where the normal vector of the seed point is perpendicular to the ground normal vector. When it is determined that the seed point and the non-seed points are in the same plane, it is determined to use the non-seed point as a new seed point;

[0023] S27 iteratively determines whether the new seed points and the non-seed points around them are in the same plane, so as to count all the seed points by the method of point cloud region growth;

[0024] S28 determines whether the number of the counted seed points is within a preset range. When the number of seed points is within the range, the planar point cloud block is constructed based on the counted seed points, and at the same time, the merging process is performed on all the planar point cloud blocks determined to be the same surface of the same object.

[0025] In a possible preferred embodiment, the step of determining the same surface of the same object includes: converting each planar point cloud block into a plane equation ax + by + cz = 1, and determining whether the three factors a, b, and c in the plane equations of each planar point cloud block are similar and the absolute value of the difference between the factors is less than a preset threshold. If it meets the condition, it is determined to be the same surface of the same object.

[0026] In a possible preferred embodiment, the step of converting the planar point cloud block to a 2D plane in the camera coordinate system through the external parameters of the 3D sensor in step S3 includes:

[0027] S31 encodes each point in the planar point cloud block as (h, w, z, a, b, c, yaw), where h and w are its coordinates in the camera coordinate system, and a, b, and c are the factors of the plane equation ax + by + cz = 1 of the planar point cloud, and then calculates ;

[0028] S32 rotates each planar point cloud by yaw with the center point of the planar point cloud block to make it parallel to the h and w axes, so as to obtain a 2D plane.

[0029] To achieve the above object, according to another aspect of the present invention, a pallet positioning method based on a 3D sensor is further provided, and its steps include:

[0030] S1 According to the pallet recognition method based on the 3D sensor as described in any one of claims 1 to 7, obtain the average value of the center points of the matching double-line sliding window fitting lines as the center point coordinates of the currently recognized target pallet end face, and use this as the x, y, and z parameters of the 6D pose;

[0031] S2 Set the angles of roll and pitch to 0;

[0032] S3 According to the plane equation ax + by + cz = 1 established by the planar point cloud block, calculate the yaw parameter value , so as to obtain the 6D pose of the target pallet.

[0033] To achieve the above object, according to another aspect of the present invention, there is also provided a pallet recognition system based on a 3D sensor, which includes:

[0034] A storage unit that stores a program for implementing the steps of the pallet recognition method based on a 3D sensor as described in any one of claims 1 to 7, for the control unit and the processing unit to retrieve and execute as appropriate;

[0035] The control unit controls the 3D sensor to collect the target point cloud in the scene and send it to the processing unit;

[0036] The processing unit filters out the ground point cloud from the target point cloud, merges the point clouds of the same end face of the target pallet to obtain a planar point cloud block; then converts the planar point cloud block into a 2D plane in the camera coordinate system through the external parameters of the 3D sensor, and then establishes an encoding map based on the 3D sensor resolution to place the 2D plane therein, and constructs a double-line sliding window with a preset feature recognition distance in the encoding map and synchronously moves according to a preset step length to perform line segment fitting scanning on the points on the 2D plane encountered on the path. When it is judged that the current line lengths fitted by the double-line sliding windows both conform to the feature template, the recognition result corresponding to the feature module is output.

[0037] To achieve the above object, according to another aspect of the present invention, there is also provided a pallet positioning system based on a 3D sensor, which includes:

[0038] A storage unit that stores a program for implementing the steps of the pallet recognition method based on a 3D sensor as described in any one of claims 1 to 7, for the control unit and the processing unit to retrieve and execute as appropriate;

[0039] The control unit controls the 3D sensor to collect the target point cloud in the scene and send it to the processing unit;

[0040] The processing unit filters out the ground point cloud from the target point cloud, merges the point clouds of the same end face of the target pallet to obtain a planar point cloud block; then converts the planar point cloud block into a 2D plane in the camera coordinate system through the external parameters of the 3D sensor, and then establishes an encoding map based on the 3D sensor resolution to place the 2D plane therein, and constructs a double-line sliding window with a preset feature recognition distance in the encoding map and synchronously moves according to a preset step length to perform line segment fitting scanning on the points on the 2D plane encountered on the path. When it is judged that the current line lengths fitted by the double-line sliding windows both conform to the feature template, the recognition result corresponding to the feature module is output;

[0041] The processing unit further takes the average value of the center points of the matching double-line fitting lines as the center point coordinates of the current recognized end face of the target pallet, and uses this as the x, y, z parameters of the 6D pose. At the same time, the angles of roll and pitch are set to 0, and the yaw parameter value is calculated to obtain the 6D pose of the target pallet.

[0042] Through the pallet recognition and positioning method and system based on 3D sensors provided by the present invention, it is not only particularly suitable for the recognition and positioning of pallets, but also for other objects that need to interact with robots. As long as there are certain continuous surface features of the object at the angles that the sensor can scan, the object and its position can be accurately and quickly recognized. And the extensibility is very strong. For example, for the pallet in the example of this case, whether it is a standard pallet or a non-standard pallet, only the respective structural parameters of the pallet need to be obtained in advance to perform the subsequent matching and recognition steps.

[0043] In addition, the present invention can also be applicable to a variety of 3D sensors with strong versatility. Whether it is a depth camera, a multi-line lidar or a solid-state lidar, it can be applied, and calculations can be directly performed without the need to splice point clouds. On the other hand, the solution of the present invention does not require pre-training of samples and can perform real-time recognition and judgment and target pose calculation. Therefore, compared with the deep learning solution, it occupies lower computing performance and is more ingenious. Brief Description of the Drawings

[0044] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0045] Figure 1 It is a schematic diagram of the end face parameters of the target pallet in the example of the present invention;

[0046] Figure 2 It is a schematic diagram of the steps of the pallet recognition method based on 3D sensors of the present invention;

[0047] Figure 3 It is a schematic diagram of the 3D sensor coordinate system (camera coordinate system) of the present invention;

[0048] Figure 4 It is a schematic diagram of the double-line sliding window fitting the end face point cloud of the pallet in the coding diagram in the pallet recognition method based on 3D sensors of the present invention;

[0049] Figure 5 It is a schematic diagram of the double-line sliding window fitting and matching the end face point cloud of the pallet in the coding diagram in the pallet recognition method based on 3D sensors of the present invention and stopping when the feature template is met;

[0050] Figure 6 It is a schematic diagram of the feature template in the pallet recognition method based on 3D sensors of the present invention;

[0051] Figure 7 It is a schematic diagram of the structure of the pallet recognition and positioning system based on 3D sensors of the present invention. Detailed Embodiments

[0052] In order to enable those skilled in the art to better understand the technical solution of the present invention, the following will, in conjunction with embodiments, clearly and completely describe the specific technical solution of the present invention to help those skilled in the art further understand the present invention. Obviously, the embodiments described in this case are only part of the embodiments of the present invention, rather than all the embodiments. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention and without conflict with each other, the embodiments in this application and the features in the embodiments can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative labor shall fall within the scope of disclosure and protection of the present invention.

[0053] In addition, in the description of the present invention, the terms "first", "second", "S1", "S2", etc. in the specification, claims and drawings of the present invention are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order other than those described here. At the same time, the terms "include" and "have" in the present invention and any variations thereof are intended to cover non-exclusive inclusion. Unless otherwise clearly defined and limited, the terms "set", "arrange", "install", "connect", "couple" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in this case can be understood in combination with the prior art according to specific circumstances.

[0054] It should be noted that in the examples of the present invention, the 3D sensor can be a 3D laser sensor, or a 3D camera sensor can be used and only the depth information is used. Specifically, the solution of the present invention is applicable to 3D lidars in non-repetitive scanning mode, but is also applicable to mechanically scanned 3D lidars and depth cameras. For mechanically scanned 3D lidars, calculations can be directly performed without point cloud stitching; for depth cameras, the depth map needs to be first converted into a 3D point cloud, and calculations can also be performed without point cloud stitching.

[0055] Furthermore, in the concept of the solution of the present invention, by specially encoding the discrete 3D point cloud into a 2D plane and using the method of double-line traversal, the pallet can be identified and located only through simple calculations. Therefore, it is applicable to various 3D sensors such as depth cameras, multi-line lidars, and solid-state lidars.

[0056] In addition, although the recognition and positioning process of a pallet with a specific structure is exemplified in the following examples, those skilled in the art can understand from the following embodiments that the solution of the present invention can also be applied to recognize multiple different types of pallets, and can even be extended to non-standard pallets. Only by pre-configuring the parameters of the pallet and establishing a feature template can the pose of the pallet be quickly and accurately recognized and positioned.

[0057] Specifically, as Figures 1 to 6 shown, the method for pallet recognition based on a 3D sensor provided by the present invention includes the following steps:

[0058] Step S1

[0059] Determine the parameters of each end face of the target pallet for recognition, and establish a feature template.

[0060] Specifically, as Figure 1 shown, first determine the various parameters on the end face of the target pallet, and then establish a feature template according to the characteristics of different end faces of the pallet. For example, the pallet exemplified in this case has a three-leg shape. Therefore, in order to correspond to the double-line traversal method proposed in this case, the feature template can be exemplified as two line segments with different sizes and shapes at specific positions on the end face of the pallet.

[0061] For example, the feature template includes: a first sub-feature, which is a continuous line segment feature, and its length is within the range of the end face parameters of the target pallet; a second sub-feature, which is a plurality of spaced line segment features, and the length of each spaced line segment and the spacing distance are within the range of the end face parameters of the target pallet.

[0062] As Figure 6 shown, since the pallet has 3 legs, the line at L1 represents the continuous feature of the upper plate body of the pallet. At the same time, its length dimension is referenced by a1, and the spaced lines at L2 represent the plate body features where the 3 legs are spaced apart. At the same time, the length of each line itself is referenced by b1 and c1 respectively, and the spacing distance is referenced by d1. In addition, the spacing between L1 and L2 is referenced by e1. Thus, a feature template is established through these preset recognition constraints and used for subsequent recognition.

[0063] Wherein a1, b1, c1, d1, and e1 are respectively dimensions corresponding to and substantially the same or similar to the respective parameters a, b, c, d, and e of the pallet.

[0064] In addition, those skilled in the art should be aware that the above feature template is only an example and is not subject to any restrictions. Those skilled in the art can also make similar settings according to the specific structure of the pallet without departing from the inventive concept of this case, and such settings also fall within the scope of disclosure of the present invention.

[0065] Step S2

[0066] The target point cloud is established based on the sensor data collected by the 3D sensor. The target point cloud is preprocessed. After filtering out the ground point cloud, the point cloud on the same end face of the target pallet is merged to obtain a planar point cloud block.

[0067] Specifically, the target point cloud preprocessing step first requires filtering the pallet point cloud data, and then separating the ground component and the object component from the filtered pallet point cloud data. The filtering processing operation can specifically assign the acquired pallet point cloud data to point cloud coordinates corresponding to the natural three-axis coordinate system, and then obtain the required height of the pallet to be identified, and then filter out the pallet point cloud data that does not match the required height under the point cloud coordinates according to the required height.

[0068] For example, in the example, when the height of the pallet is 15 cm, if the pallet is placed on the horizontal ground, a straight-through filter can be used to obtain only the point cloud 0 to 15 cm from the ground, and if the pallet is stacked on two layers of pallets of the same height, a straight-through filter can be used to obtain the point cloud 30 to 45 cm from the ground.

[0069] Then, a statistical filter is applied to the point cloud obtained after the above processing to remove outliers. In this way, noise and redundant data can be effectively reduced through multiple filtering preprocessing methods, so as to improve the accuracy and operation speed of subsequent algorithms, as well as the robustness and real-time performance of the algorithms.

[0070] Furthermore, the step of filtering out the ground point cloud includes: selecting the reference plane with the largest number of points within the tolerance distance range from all points in the target point cloud data to the reference plane as the ground, and attributing all points on the plane to the ground component, and the remaining points to the object component.

[0071] Specifically, to remove the ground point cloud from the point cloud after the above preprocessing, it is first necessary to extract the ground component and object component from the pallet point cloud data respectively. For this purpose, in this example, a random sampling consensus algorithm and a plane model matching method are used to separate the ground component and object component from the point cloud data. Specifically, multiple points can be randomly extracted from the pallet point cloud data multiple times, and multiple reference planes are fitted accordingly; then, the number of corresponding points whose distances from all points in the point cloud data to multiple reference planes are within the tolerance distance range is counted; then, the number of counted points is compared to determine the ground component from multiple reference planes.

[0072] For example, the reference plane with the largest number of points within the tolerance distance from all points in the point cloud data to the reference plane may be selected as the ground, and all points on the plane may be attributed to the ground component, thereby separating the ground component and the object component in the pallet point cloud data.

[0073] Further, the steps of merging the point clouds on the same end face of the target pallet to obtain a planar point cloud block include:

[0074] Extract a planar point cloud block perpendicular to the ground. For example, randomly select seed points from the object component point cloud data, and determine whether the seed points and the non-seed points around the seed points are in the same plane. The normal vector of the seed points is perpendicular to the normal vector of the ground. When it is determined that the seed points and the non-seed points are in the same plane, determine that the non-seed point is a new seed point.

[0075] Then iteratively determine whether the new seed points and the non-seed points around them are in the same plane, so as to count all the seed points by the method of point cloud region growth.

[0076] Based on the aforementioned counted seed points, construct a planar point cloud block perpendicular to the ground. Specifically, it is to judge whether the number of the counted seed points is within a preset number range, and when the number of the seed points is within the number range, a planar point cloud block perpendicular to the ground can be constructed based on the counted seed points.

[0077] At the same time, if the number of the seed points is too high or too low, it can be determined that the plane corresponding to the counted seed points does not belong to the planar point cloud block, and this method can also obtain the plane equation parameters.

[0078] After that, after obtaining all the planar point cloud blocks, in order to ensure the integrity of the same face of the same object, in this example, further merging processing is performed on all the planar point cloud blocks perpendicular to the ground. For example, make a judgment through the aforementioned plane equation and distance. If the factors of the plane equation are similar and the distance meets the threshold, the two point cloud blocks are merged into the same point cloud block.

[0079] As in the example, any plane equation can be written as ax + by + cz = 1. If the three factors a, b, and c in the plane equations of the two planar point cloud blocks are similar and the absolute value of the difference between the factors is less than the threshold, the two planar point cloud blocks can be merged into the same point cloud block. This ensures the integrity of the same face of the same object.

[0080] In addition, for the convenience of subsequent step calculations, the center point of the planar point cloud block can be further calculated. In the present invention, since the pallet in the example is symmetric, the maximum and minimum values of the planar point cloud block on the z-axis in the camera coordinate system can be directly calculated, and the center coordinates of the planar point cloud block can be obtained by averaging the two points of the cloud.

[0081] Step S3

[0082] Convert the planar point cloud block into a 2D plane in the camera coordinate system through the external parameters of the 3D sensor. Specifically, first encode each point in the planar point cloud block as (h, w, z, a, b, c, yaw), where h and w are the translation coordinates of the y and x axes in the camera coordinate system. Here, the origin can be transformed to the upper left corner, and a, b, c are the factors of the plane equation ax + by + cz = 1 of the planar point cloud. Then, it can be calculated that .

[0083] At this time, since the center coordinates of the planar point cloud block have been obtained in step S2, each planar point cloud can be rotated by yaw around the center point of the planar point cloud block, so that all planar point clouds can be rotated to be parallel to the x and y axes, thereby obtaining the end face 2D plane of the target pallet in the camera coordinate system, as Figure 4 shown, for subsequent recognition.

[0084] Step S4

[0085] As Figures 4 to 5 shown, first establish an encoding map. The size of the encoding map can be constructed with reference to the resolution size supported by the 3D sensor, such as the encoding map with the size of W*H as Figures 4 to 5 shown, and then put the 2D plane obtained in step S3 into it.

[0086] Furthermore, construct a double-line sliding window starting from the uppermost end of the encoding map. L1 and L2 of the double-line sliding window are set parallel to each other, and the interval distance is preferably e1 slightly smaller than the pallet parameter e in this example.

[0087] Furthermore, set the moving step of the double-line sliding window, and make the double-line sliding window move parallel to the W axis along the positive H direction, and the distance between the double-line sliding windows remains unchanged. After each movement, the points in this double-line sliding window need to be calculated, that is, perform line segment fitting scanning on the points on L1 and L2. The fitting is based on the distance between adjacent points and the a, b, c of the encoding parameters of adjacent points. Since the equation ax + by + cz = 1 is satisfied on the same plane, and the distance between points can determine whether they are on the same plane.

[0088] Then perform length judgment. Since the length example of the first sub-feature to be recognized by L1 in the feature template is in the a1 parameter range corresponding to the a size of the pallet; the length and interval distance of the second sub-feature to be recognized by L2 are in the b1, c1, d1 parameter ranges corresponding to the b, c, d sizes of the pallet, so the template matching judgment can be made according to the following logic:

[0089]

[0090] where is the error threshold.

[0091] Continuously move the double-line sliding window in this way for matching and judgment. When it is judged that the current line lengths fitted by the double-line sliding windows respectively meet the feature template, record this result. Next, judge whether the sliding is over. If it is not over, continue to slide; otherwise, end the recognition and output the recognition result corresponding to this feature module.

[0092] It should be noted here that in this example Figure 5 the end face of the pallet shown is an ideal surface and is shown relatively completely. However, in the actual process, at the three feet of the pallet, there may be a phenomenon of partial point cloud missing. Therefore, as Figure 5 shown, during the forward movement of the double-line sliding window, the highest matching degree during the whole process can be selected as the recognition result for output.

[0093] Through the recognition process of the above steps S1 to S4, it can be judged whether the object represented by the target point cloud is the corresponding pallet. On the other hand, it can also provide a basis for calculating the pose of the pallet.

[0094] For this reason, on the other hand, the present invention also provides a pallet positioning method based on a 3D sensor, and its steps include:

[0095] Step S1

[0096] According to the above-mentioned pallet recognition method based on a 3D sensor, obtain the average value of the center points of the lines fitted by the matching double-line sliding window as the center point coordinates of the current recognized target pallet end face, and use this as the x, y, z parameters of the 6D pose.

[0097] Specifically, the pose of the pallet has 6 degrees of freedom in space, namely the translation amounts x, y, z and the rotation amounts roll, pitch, yaw, that is, 6D pose. Therefore, according to the optimal line segments that meet the conditions fitted by the double-line sliding window, select the average value of the midpoints of a1 and c1 as the center position of the recognized pallet. Then, restore this center point to the camera coordinate system, and the x, y, z parameters can be obtained.

[0098] Step S2

[0099] Since the pallet and the forklift robot are both default in the same space and the ground is horizontal, at this time, it can be considered that the pallet is parallel to the forklift forks, that is, there are no roll and pitch angles. Therefore, the roll and pitch angles can be set to 0.

[0100] At this time, only the yaw angle value needs to be calculated to obtain the 6D pose of the pallet.

[0101] Step S3

[0102] Based on the plane equation ax + by + cz = 1 of the planar point cloud of the pallet surface and the three equation factors a, b, and c obtained in the steps of the previous pallet recognition method based on a 3D sensor, the yaw parameter value can be calculated according to

[0103]

[0104] to obtain the complete 6D pose of the target pallet, thus completing its positioning.

[0105] On the other hand, corresponding to the above recognition method, please refer to Figure 7 as shown, the present invention also provides a pallet recognition system based on a 3D sensor, which includes:

[0106] A storage unit that stores a program for implementing the steps of the pallet recognition method based on a 3D sensor as described in the above embodiments, for the control unit and the processing unit to retrieve and execute timely.

[0107] The control unit controls the 3D sensor to collect the target point cloud in the scene and send it to the processing unit.

[0108] The processing unit filters out the ground point cloud from the target point cloud, merges the point clouds of the same end face of the target pallet to obtain a planar point cloud block; then transforms the planar point cloud block into a 2D plane in the camera coordinate system through the external parameters of the 3D sensor, and then establishes a coding map to place the 2D plane in it, and constructs a double-line sliding window with a preset feature recognition distance in the coding map and synchronously moves according to a preset step length to perform line segment fitting scanning on the points on the 2D plane encountered on the path. When it is determined that the current line lengths of the double-line sliding windows respectively fitted are all in line with the feature template, the recognition result corresponding to the feature module is output.

[0109] On the other hand, corresponding to the above recognition method, please refer to Figure 7 as shown, the present invention also provides a pallet positioning system based on a 3D sensor, which includes:

[0110] A storage unit that stores a program for implementing the steps of the pallet recognition method based on a 3D sensor as described in the above embodiments, for the control unit and the processing unit to retrieve and execute timely.

[0111] The control unit controls the 3D sensor to collect the target point cloud in the scene and send it to the processing unit.

[0112] The processing unit filters out the ground point cloud from the target point cloud, merges the point clouds on the same end face of the target pallet to obtain a planar point cloud block; then converts the planar point cloud block into a 2D plane in the camera coordinate system through the external parameters of the 3D sensor, and then establishes a coding map to place the 2D plane therein, and constructs a double-line sliding window with a preset feature recognition distance in the coding map and synchronously moves according to a preset step length to perform line segment fitting scanning on the points on the 2D plane encountered on the path. When it is judged that the current line lengths fitted by the double-line sliding windows both conform to the feature template, the recognition result corresponding to the feature module is output.

[0113] The processing unit further takes the average value of the central points of the double-line fitting lines that match as the central point coordinates of the current recognized end face of the target pallet, and uses this as the x, y, and z parameters of the 6D pose. At the same time, the angles of roll and pitch are set to 0, and the yaw parameter value is calculated. , so as to obtain the 6D pose of the target pallet, and thus complete its positioning.

[0114] In summary, the pallet recognition and positioning method and system based on a 3D sensor provided by the present invention are not only particularly suitable for the recognition and positioning of pallets, but also for other objects that need to interact with the robot. As long as the object has certain continuous surface features at the angles that the sensor can scan, the object and its position can be accurately and quickly recognized. And the expandability is very strong. For example, for the pallet in the example of this case, whether it is a standard pallet or a non-standard pallet, only the respective structural parameters of the pallet need to be obtained in advance to perform the subsequent matching and recognition steps.

[0115] In addition, the present invention can also be applicable to a variety of 3D sensors, with strong versatility. Whether it is a depth camera, a multi-line lidar or a solid-state lidar, it can be applicable, and direct calculation can be performed without the need to splice the point cloud. On the other hand, the solution of the present invention does not require pre-training of samples, and can perform recognition judgment and target pose calculation in real time. Therefore, it occupies lower computing performance and is more ingenious than the deep learning solution.

[0116] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the present invention, so that those skilled in the art can well understand and utilize the present invention. The present invention is only limited by the claims and their full scope and equivalents. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

[0117] Those skilled in the art can understand that, in addition to implementing the systems, devices, and their respective modules provided by the present invention in the form of pure computer-readable program codes, the method steps can be logically programmed to enable the systems, devices, and their respective modules provided by the present invention to be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Therefore, the systems, devices, and their respective modules provided by the present invention can be considered as a kind of hardware component, and the modules included therein for implementing various programs can also be regarded as the structures within the hardware component; the modules for implementing various functions can also be regarded as either software programs for implementing the methods or the structures within the hardware component.

[0118] In addition, all or part of the steps in the methods of the above embodiments can be completed by instructing relevant hardware through a program, and the program is stored in a storage medium, including several instructions for enabling a single-chip microcomputer, a chip, or a processor to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs.

[0119] In addition, any combination can be made among various different embodiments of the embodiments of the present invention, as long as it does not violate the idea of the embodiments of the present invention, and it should also be regarded as the content disclosed in the embodiments of the present invention.

Claims

1. A pallet recognition method based on a 3D sensor, characterized in that the steps Including: S1 Determine the end-face parameters for identifying the target pallet and establish a feature template; S2 Based on the sensing data collected by the 3D sensor, establish a target point cloud, and preprocess the target point cloud. After filtering out the ground point cloud, merge the point clouds of the same end face of the target pallet to obtain a planar point cloud block; S3 Transform the planar point cloud block into a 2D plane in the camera coordinate system through the external parameters of the 3D sensor; S4 Based on the resolution of the 3D sensor, establish a coding map and place the 2D plane into it. Construct a double-line sliding window with a preset feature recognition distance in the coding map and synchronously move it according to a preset step size to perform line segment fitting scanning on the points on the 2D plane encountered on the path. When it is judged that the current line lengths fitted by the two double-line sliding windows both conform to the feature template, output the recognition result corresponding to the feature module.

2. The pallet recognition method based on a 3D sensor according to claim 1, wherein The feature template includes: a first sub-feature, which is a continuous line segment feature, and its length exists within the range of the end-face parameters of the target pallet; a second sub-feature, which is a plurality of spaced line segment features, and the lengths and spacing distances of each spaced line segment exist within the range of the end-face parameters of the target pallet.

3. The pallet recognition method based on a 3D sensor according to claim 1, characterized in that, The steps for preprocessing the target point cloud in step S2 include: S21 Perform filtering processing on the target point cloud, convert the target point cloud to the robot coordinate system according to the external parameters of the 3D sensor to obtain the corresponding point cloud coordinates, and thus filter out the mismatched target point cloud with reference to the height parameter of the target pallet; S22 Perform statistical filtering processing on the target point cloud processed in step S21 to remove outliers.

4. The pallet recognition method based on a 3D sensor according to claim 1, wherein The steps for filtering out the ground point cloud in step S2 include: S23 Randomly extract multiple points from the target point cloud multiple times to fit out multiple reference planes; S24 Statistically count the number of points corresponding to all points of the target point cloud within the tolerance distance range of each reference plane; S25 Select the reference plane with the largest number of corresponding points as the ground, and attribute all the points on this reference plane to the ground component for elimination, while the remaining points are attributed to the object component.

5. The pallet recognition method based on a 3D sensor according to claim 4, wherein, The steps for merging the point clouds of the same end face of the target pallet to obtain a planar point cloud block in step S2 include: S26 Randomly select seed points from the object component point cloud, and judge whether the seed points and the non-seed points around the seed points are in the same plane. The normal vector of the seed points is perpendicular to the normal vector of the ground. When it is determined that the seed points and the non-seed points are in the same plane, determine to use the non-seed points as new seed points; S27 Iteratively judge whether the new seed points and the non-seed points around them are in the same plane to statistically count all the seed points by the method of point cloud region growth; S28 Judge whether the number of the statistically counted seed points is within a preset number range, and when the number of seed points is within the number range, construct the planar point cloud block based on the statistically counted seed points, and at the same time perform merging processing on all the planar point cloud blocks of the same face judged to be the same object.

6. The pallet recognition method based on a 3D sensor according to claim 5, characterized in that, The steps of determining the same surface of the same object include: converting each surface point cloud block into a plane equation ax + by + cz = 1, and determining whether the three factors a, b, and c in the plane equations of each surface point cloud block are similar and the absolute value of the difference between the factors is less than a preset threshold. If it meets the criteria, it is determined to be the same surface of the same object.

7. The pallet recognition method based on a 3D sensor according to claim 1, characterized in that In step S3, the steps of converting the surface point cloud block to a 2D plane in the camera coordinate system through the external parameters of the 3D sensor include: S31 encodes each point in the surface point cloud block as (h, w, z, a, b, c, yaw), where h and w are its coordinates in the camera coordinate system, and a, b, c are the factors of the plane equation ax + by + cz = 1 of the surface point cloud, and then calculates ; S32 rotates each surface point cloud by yaw with the center point of the surface point cloud block to make it parallel to the h and w axes, so as to obtain a 2D plane.

8. A pallet positioning method based on a 3D sensor, characterized in that the steps Include: S1 According to the pallet recognition method based on a 3D sensor described in any one of claims 1 to 7, obtain the average value of the center points of the lines fitted by the matching double-line sliding window as the center point coordinates of the target pallet end face currently recognized, and use this as the x, y, z parameters of the 6D pose. S2 Set the angles of roll and pitch to 0. S3 Based on the plane equation ax + by + cz = 1 established from the planar point cloud block, calculate the yaw parameter value according to this plane equation and its three factors a, b, and c to obtain the 6D pose of the target pallet.

9. A pallet recognition system based on a 3D sensor, characterized in that Include: A storage unit that stores a program for implementing the steps of the pallet recognition method based on a 3D sensor described in any one of claims 1 to 7 for the control unit and the processing unit to retrieve and execute as appropriate; The control unit controls the 3D sensor to collect the target point cloud in the scene and send it to the processing unit; The processing unit filters out the ground point cloud from the target point cloud, merges the point clouds of the same end face of the target pallet to obtain a surface point cloud block; then converts the surface point cloud block into a 2D plane in the camera coordinate system through the external parameters of the 3D sensor, and then establishes an encoding map based on the 3D sensor resolution and puts the 2D plane into it. And construct a double-line sliding window with a preset feature recognition distance in the encoding map and move synchronously according to the preset step length to perform a line segment fitting scan on the points on the 2D plane encountered on the path. When it is determined that the current line lengths fitted by the double-line sliding windows both meet the feature template, output the recognition result corresponding to the feature module.

10. A pallet positioning system based on a 3D sensor, characterized in that Include: A storage unit that stores a program for implementing the steps of the pallet recognition method based on a 3D sensor described in any one of claims 1 to 7 for the control unit and the processing unit to retrieve and execute as appropriate; The control unit controls the 3D sensor to collect the target point cloud in the scene and send it to the processing unit; The processing unit filters out the ground point cloud from the target point cloud, merges the point clouds of the same end face of the target pallet to obtain a surface point cloud block; then converts the surface point cloud block into a 2D plane in the camera coordinate system through the external parameters of the 3D sensor, and then establishes an encoding map based on the 3D sensor resolution and puts the 2D plane into it. And construct a double-line sliding window with a preset feature recognition distance in the encoding map and move synchronously according to the preset step length to perform a line segment fitting scan on the points on the 2D plane encountered on the path. When it is determined that the current line lengths fitted by the double-line sliding windows both meet the feature template, output the recognition result corresponding to the feature module; The processing unit further takes the average value of the center points of the double-line fitting lines that match as the center point coordinates of the currently recognized target pallet end face, uses this as the x, y, and z parameters of the 6D pose, sets the angles of roll and pitch to 0 at the same time, and calculates the yaw parameter value to obtain the 6D pose of the target pallet.

Images