Tray identification and positioning method based on point cloud perception

Abstract

The invention discloses a tray identification and positioning method based on point cloud perception, and relates to the technical field of industrial vehicle automatic driving, and the method comprises the steps: obtaining and preprocessing an original point cloud set of a visual field area, calculating the normal vector of each point in the preprocessed point cloud set, constructing a point cloud plane composed of points of the same type, and carrying out the calculation of the normal vector of each point in the point cloud plane; calculating a normal vector of the point cloud plane by using a principal component analysis algorithm, and calculating a yaw angle and a pitch angle of the point cloud plane relative to a camera coordinate system according to the normal vector; projecting the preprocessed point cloud into a grid of a plane along the axis direction to obtain a grid map, and setting the projection height as a grid gray value; carrying out contour extraction and template matching on the grid map by adopting an edge detection algorithm, and determining a to-be-detected contour as a tray; positioning the center and the pose of the tray according to the pixel points of the whole tray grid map; according to the method, the height of the tray, the jack position, the inclination angle and other key parameters can be calculated in real time in a dynamically changing working environment.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology for industrial vehicles, and in particular to a pallet recognition and positioning method based on point cloud perception. Background Technology

[0002] Currently, AGV systems face limitations in their positioning and sensing methods when automating pallet picking, handling, and placement. They typically rely on pre-set fixed tracks, external markers (such as magnetic strips or QR codes), or pre-built static environmental maps. This approach not only results in high deployment costs and limited flexibility, but also increases maintenance burden due to damaged markers. More importantly, it lacks adaptability, failing to achieve accurate, real-time sensing of pallet position and orientation in unstructured or dynamically changing complex environments, thus hindering further improvements in operational efficiency and reliability.

[0003] With the continuous advancement of sensor technologies such as depth cameras and LiDAR, AGV systems have initially acquired autonomous environmental perception capabilities, laying the foundation for pallet identification, positioning, and path planning. However, in outdoor high-light environments, the application of depth cameras for AGV end-effector identification faces three main technical challenges: 1. Depth Information Loss and Sharp Drop in Signal-to-Noise Ratio: Depth cameras (especially those based on structured light) rely on actively projected coded light spots for ranging. Under strong illumination (especially including near-infrared light), ambient light creates extremely high background noise, completely "drowsing" the camera's self-projected light spot signal, leading to a sharp deterioration in the signal-to-noise ratio. This not only significantly reduces the effective pixels in the acquired depth map, resulting in numerous holes and noise, but may also cause the depth information to become completely invalid due to sensor saturation.

[0004] 2. Deterioration of visible light imaging quality: Strong light can cause overexposure of the object's surface, blurring or even eliminating key visual features of the tray (such as texture, edges, QR codes, etc.). At the same time, the resulting heavy shadows and intense light contrast can create incorrect contours and textures, severely interfering with the initial detection and classification of the tray by 2D image-based visual recognition algorithms.

[0005] 3. Insufficient Dynamic Range and Stereo Vision Failure: Outdoor lighting conditions change rapidly, placing extremely high demands on the dynamic range of cameras. Ordinary depth cameras struggle to capture details in both brightly lit and shadowed areas simultaneously, resulting in the inability to fully represent the target in a single image. Furthermore, for depth cameras based on binocular vision, glare from strong light and uneven image quality severely affect the accuracy and quantity of feature point matching, directly leading to depth calculation failure. Summary of the Invention

[0006] Based on the technical problems existing in the background technology, this invention proposes a pallet recognition and positioning method based on point cloud perception, which can calculate key parameters such as pallet height, socket position and tilt angle in real time in dynamically changing working environments (including outdoor strong light environment).

[0007] This invention proposes a tray recognition and localization method based on point cloud perception, comprising: The original point cloud of the field of view is obtained and preprocessed. The principal component analysis algorithm is used to calculate the normal vector of each point in the preprocessed point cloud. One of the points is used as a seed point. When the angle between the normal vector of the seed point and the normal vector of the surrounding neighboring points is less than a set threshold, the two points are classified into the same class. A point cloud plane composed of points of the same class is constructed. The normal vector of the point cloud plane is calculated using the principal component analysis algorithm. Based on this, the yaw angle and pitch angle of the point cloud plane relative to the camera coordinate system are calculated. Using the sensor viewpoint as the origin, and the normal vector of the point cloud plane as... Establish a coordinate system using the axis unit vector, and then plot the preprocessed point cloud along... Projected along the axis In a planar grid, a raster image is obtained, and the projection height is set to the raster grayscale value; An edge detection algorithm is used to extract the contour of the raster image and perform template matching to determine that the contour to be detected is a tray; the center and pose of the tray are located based on the pixels of the entire tray raster image.

[0008] Furthermore, the original point cloud of the acquired lidar field of view is preprocessed, and the preprocessing process is as follows: 3D voxel sizes are determined based on the minimum volume surrounding the cloud of origin points; Based on point cloud coordinates, each point is divided into its corresponding voxel. The centroid of each voxel is calculated using the coordinates of each point within the voxel, and the centroid is used as the coordinates of each point within the voxel. Filter out abrupt noise points at the edges.

[0009] Furthermore, the filtering of abrupt noise points appearing at the edges specifically involves: A point whose maximum depth difference with its Q neighboring points is greater than a set depth threshold is defined as a transition point; Randomly select a point, and form a point set with the W nearest surrounding points of that point.

[0010] Calculate the average depth of W+1 points within the point set. Calculate the maximum depth difference between the selected point and the remaining points in the point set. ,when When this point is considered a jump noise point, it is filtered out. The threshold parameter is set.

[0011] Furthermore, before calculating the normal vector of each point in the preprocessed point cloud using the principal component analysis algorithm, a region growing algorithm based on normal vector constraints is used to segment the preprocessed point cloud, specifically as follows: Along each row of the pallet Search along the axis direction; when a set number of rows are found to be in the same direction... When the number of jump points near the coordinates exceeds the jump threshold, determine the location of the tray. Axis region; Where the tray is located Within the axis region, search along The periodic change in depth along the axis, capturing one cycle to obtain the value of a single tray. Axis region.

[0012] Furthermore, the process of generating the normal vector for each point is as follows: Set search neighborhood parameters traverse one of the points For each of the nearest neighbors, construct a neighborhood covariance matrix; The eigenvector corresponding to the smallest eigenvalue in the neighborhood covariance matrix is ​​taken as the normal vector of that point. Calculate the angle between the normal vector corresponding to the point and the z-axis of the camera coordinate system (i.e., vector (0, 0, 1)). If the angle is less than 0, then flip the normal vector.

[0013] Furthermore, before using one of the points as a seed point, an angle threshold is set. When the angle between the normal vectors of the points is less than the angle threshold, they are considered close. An upper limit and a lower limit are set for the number of points in the growth region, and points that exceed the upper limit or fall below the lower limit are removed. After obtaining the yaw and pitch angles of the point cloud plane relative to the camera coordinate system, the normal vectors corresponding to yaw angles less than 0 and pitch angles less than 0 are flipped.

[0014] Furthermore, in the process of preprocessing the point cloud along... Projected along the axis Before the grid in the plane, a threshold range for the yaw angle is set. If the calculated yaw angle is not within the threshold range, the point cloud plane corresponding to that yaw angle is removed to initially identify interfering planes.

[0015] Furthermore, the edge detection algorithm is used to extract the contour of the raster image and perform template matching to determine whether the extracted contour is a tray. The template matching process is as follows: Calculate the Hu invariant moments of the extracted contour; Construct the aspect ratio of the extracted contour and duty cycle ; Based on the actual vehicle's drawings or dimensions, generate a frontal outline of the vehicle, and obtain outlines in different directions through perspective transformation, thereby constructing multiple template outlines. The template contours are traversed using the flight angle and pitch angle corresponding to the extracted contours, and the top-H template contours that are closest to the flight angle and pitch angle are selected as candidate template contours, where H is an integer; Using Hu invariant moments and profile aspect ratio and duty cycle Calculate the similarity between the extracted contour and the candidate template contour, and determine whether the similarity is greater than the set similarity threshold. If it is, the extracted contour is a tray; otherwise, it is not a tray.

[0016] Furthermore, the extracted contour With candidate template contours The formula for calculating the similarity between them is as follows: ; in, for The Euclidean distance between them is used as a similarity measure. for Hu invariant moment, for Hu invariant moment, For value index, for The scale and proportion characteristics, for The scale and proportion characteristics.

[0017] Furthermore, the step of locating the center and pose of the tray legs based on the pixels of the entire tray grid image specifically involves: For the entire tray raster image, the pixels are traversed from top to bottom and from left to right in each row. First-order difference operation is performed to extract the pixels with a pixel value difference of 255 between adjacent points and put them into the set, which are the edge points. Set a row distance threshold to retain points whose distance between adjacent edge points in the same row is less than or equal to the row distance threshold; for dual-hole trays, set a row of pixels to have R edge points; the distance between adjacent edge points cannot exceed the vertical distance threshold; For vehicle tilt, calculate the minimum bounding rectangle of the template outline, correct the rotation according to the direction of the minimum bounding rectangle, and then perform detection and template matching. The identified coordinates are then rotated again. After extracting the left and right contours of each pallet leg, calculate the mean of the coordinates of the left and right contours of each pallet leg, and calculate the centroid of the original point cloud projected onto the grid image to obtain the center point and pose of the pallet.

[0018] The advantages of the pallet recognition and positioning method based on point cloud perception provided by this invention are: it can accurately and in real time acquire the three-dimensional pose information of the pallet (including horizontal position, vertical height and tilt angle, etc.), and can calculate key parameters such as the height of the pallet, the position of the socket and the tilt angle in real time in dynamically changing working environments (including outdoor strong light environment). Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the structure of the present invention; Figure 2 For along A schematic diagram illustrating the location of a large depth jump along the axial direction; Figure 3 To find along A schematic diagram of the periodic variation of depth on the axis; Figure 4 A flowchart illustrating the principal component analysis algorithm; Figure 5 The template outline is generated as a schematic diagram. (a) is the front view, and (b) is the rotated view. Detailed Implementation

[0020] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0021] like Figures 1 to 5 As shown, the present invention proposes a pallet recognition and localization method based on point cloud perception, comprising: Step 1: Obtain the original point cloud of the field of view and preprocess it. Use the principal component analysis algorithm to calculate the normal vector of each point in the preprocessed point cloud. Step 2: Take one of the points as the seed point. When the angle between the normal vector of the seed point and the normal vector of the surrounding neighboring points is less than a set threshold, classify the two points into the same class and construct a point cloud plane composed of points of the same class. Use the principal component analysis algorithm to calculate the normal vector of the point cloud plane, and then calculate the yaw angle and pitch angle of the point cloud plane relative to the camera coordinate system. Step 3: Using the sensor viewpoint as the origin, and the normal vector of the point cloud plane as... Establish a coordinate system using the axis unit vector, and then plot the preprocessed point cloud along... Projected along the axis In a planar grid, a raster image is obtained, and the projection height is set to the raster grayscale value; Step 4: Use an edge detection algorithm to extract the contour of the grid image and perform template matching to determine that the contour to be detected is a tray; locate the center and pose of the tray legs based on the pixels of the entire tray grid image.

[0022] This embodiment addresses the limitations of depth cameras in edge recognition under strong outdoor light by proposing a 3D perception scheme based on point cloud fusion sensing. The scheme aims to construct a recognition and localization system that is inherently robust to changes in lighting conditions and does not rely on 2D visual features.

[0023] This embodiment can accurately acquire the three-dimensional pose information of the pallet (including horizontal position, vertical height and tilt angle, etc.) in real time, and can calculate key parameters such as the height of the pallet, the position of the socket and the tilt angle in real time in dynamically changing working environments (including outdoor strong light environment).

[0024] In one embodiment, step one involves obtaining and preprocessing the original point cloud of the view area, and using principal component analysis to calculate the normal vector of each point in the preprocessed point cloud. Specifically: (a1) Point cloud preprocessing; improve the quality of point cloud data to make it more suitable for subsequent applications and processing tasks.

[0025] Point cloud data is typically characterized by irregularity, sparsity, and noise, resulting in large data volumes, high computational complexity, and long computation times. Preprocessing can help improve the accuracy of point clouds, reduce noise and data volume, thereby effectively improving the efficiency and accuracy of algorithms.

[0026] The original point cloud contains a large number of unevenly distributed points, making computation difficult and prone to getting trapped in local optima. Therefore, downsampling is necessary. Downsampling reduces the number of points in the point cloud, decreasing data complexity while preserving essential geometric structure. Downsampling, also known as voxelization, uses voxels, which are small 3D spaces. The point cloud data is divided into 3D voxels, and within each voxel, the centroid is used to approximate all points. Downsampling requires checking all voxels, replacing the point set within each voxel with its centroid, thus reducing the number of points, decreasing computational complexity, and achieving a more uniform point density while maintaining the point cloud shape.

[0027] The preprocessing process specifically includes (b1) to (b4): (b1) Obtain the maximum value of the point cloud on the three-dimensional coordinate axis by comparison and sorting. , , and minimum value , , Find the side lengths of the minimum volume enclosing the point cloud, and then determine an appropriate voxel size based on the side lengths of the minimum volume. For example, setting the voxel size to .

[0028] (b2) Calculate the corresponding coordinates in the point cloud. voxel position on axis The point cloud is divided into its respective voxels. For each point, calculations are performed and the voxel is assigned to it. ; in, Representing the three-dimensional coordinates (integer coordinates) of a voxel, physically indicating which voxel cube in space the current point belongs to, and thus... The first The three-dimensional coordinates of the voxel to which each point belongs. The first The three-dimensional coordinates of each point.

[0029] (b3) Calculate the centroid of each voxel using the coordinates of each point cloud within the voxel, and use the centroid coordinates of the voxel. Replace all point cloud data coordinates within the voxel: ; in, The number of points within a voxel. Points within a voxel.

[0030] (b4) Three-dimensional point cloud data may contain abrupt noise at the edges of objects, resulting in jagged or noisy points that need to be filtered out. Specifically: Points whose maximum depth difference with their Q (e.g., Q=8) nearest neighbors exceeds a set depth threshold are defined as transition points. A point is randomly selected, and its W nearest neighbors (e.g., W=8) are grouped with the selected point to form a point set. The average depth of the W+1 (e.g., W+1=9) points within this point set is calculated and denoted as _____. Calculate the maximum depth difference between the selected point and the remaining points in the point set, denoted as . A point is considered a jump noise point and filtered out when the following conditions are met: ; in, These are the threshold parameters that need to be set.

[0031] (a2) Use the principal component analysis algorithm to calculate the normal vector of each point in the preprocessed point cloud.

[0032] (d1) The point cloud is segmented using a region growing algorithm based on normal vector constraints; The region growing algorithm based on normal vectors may not be able to distinguish these trays (the calculated normal vectors may be too close), and subsequent projection may be affected. Therefore, it is necessary to introduce prior information to segment these trays first.

[0033] Step 1: Due to the significant depth difference between the tray and the background, you can follow... Find locations with significant depth jumps along the axis. That is, for each row, along... Search along the axis direction; when a set number of rows are found to be in the same direction... When the number of transition points near the coordinates exceeds the set transition threshold, the location of the tray can be determined. Axial region, such as Figure 2 As shown.

[0034] Step 2: Because the trays are facing close together, The depth of the pallet along the axial direction will vary periodically; that is, in the section where the pallet is located as determined in the first step... Within the axis region, search along The periodic change in depth along the axis, by capturing one cycle, yields the value of a single pallet. Axial region, such as Figure 3 As shown.

[0035] (d2) Calculate the normal vector of each point in the preprocessed point cloud using the principal component analysis algorithm; Traversal of For each of the nearest neighbors, construct a neighborhood covariance matrix. : ; in, For the centroid within the neighborhood, and These are the covariance matrices The 1 eigenvalue and eigenvector, For the first Neighboring points.

[0036] Find the minimum eigenvalue in the neighborhood covariance matrix. Corresponding feature vector As the normal vector corresponding to that point .

[0037] Calculate the angle between the normal vector corresponding to the point and the z-axis of the camera coordinate system (i.e., vector (0, 0, 1)). If the angle is less than 0, then flip the normal vector.

[0038] The principal component analysis algorithm described above is used to calculate the normal vectors for all point clouds, thereby obtaining the normal vector corresponding to each point in the original point cloud set.

[0039] A region growing algorithm based on normal vector constraints is used to segment the point cloud, and principal component analysis is used to calculate the normal vector of each point in the preprocessed point cloud set. To avoid the large amount of computation required for global analysis, PCA provides high-quality "features" for region growing, and fast segmentation is achieved by using the angle between simplified normal vectors, which effectively avoids huge computational resource overhead.

[0040] In one embodiment, step two involves using one of the points as a seed point. When the angle between the normal vector of the seed point and its surrounding neighboring points is less than a set threshold, the two points are grouped into the same class, constructing a point cloud plane composed of points of the same class. Principal component analysis is used to calculate the normal vector of the point cloud plane, and based on this, the yaw and pitch angles of the point cloud plane relative to the camera coordinate system are calculated. Specifically: (e1) Construction of the point cloud plane; Set an angle threshold; points are considered close if the angle between their normal vectors is less than this threshold. Set upper and lower limits for the number of points in the growth region, discarding points exceeding the upper limit or falling below the lower limit. Randomly select a point cloud as a seed point, calculate the angle between the seed point and the normal vectors of surrounding point clouds; if the angle is less than the set threshold, classify the two points into the same category. Repeat this classification operation until the upper limit for the point set is reached or no more point clouds meet the conditions. Then, use Principal Component Analysis (PCA) again to estimate the normal vectors of the segmented point cloud plane, thus obtaining the normal vectors of the point cloud plane. .

[0041] (e2) After obtaining the normal vector of the point cloud plane, the yaw angle (the angle between the point cloud normal vector and the z-axis) and pitch angle (the angle between the point cloud normal vector and the y-axis) of the point cloud plane relative to the camera coordinate system can be calculated, which is convenient for subsequent matching: Yaw angle Calculation formula: ； Pitch angle Calculation formula: .

[0042] It should be noted that the normal vectors of the calculated points and the normal vectors of the point cloud plane are calculated using the same principal component analysis algorithm, such as... Figure 4 As shown. Similarly, in obtaining... and Next, it is necessary to make a judgment. or If any of them are less than 0, then the normal vector of the point cloud plane needs to be flipped.

[0043] In one embodiment, step three involves using the sensor viewpoint as the origin and the normal vector of the point cloud plane as... Establish a coordinate system using the axis unit vector, and then plot the preprocessed point cloud along... Projected along the axis In a planar grid, a raster image is obtained, and the projection height is set to the raster grayscale value, specifically: Take sensor viewpoint The normal vector of the point cloud plane is the origin. As Axial unit vector Establish a coordinate system.

[0044] Set a threshold range for the yaw angle. If the calculated yaw angle is not within the threshold range, the point cloud plane corresponding to the yaw angle is removed to initially identify interfering planes and improve the matching speed.

[0045] Set the width and height of a single-channel raster image. The physical dimensions of each grid cell: length and width and .

[0046] The point cloud set (i.e., the preprocessed point cloud after removing interference planes) is along... axial direction in Project onto the plane. The coordinates after projection. for: ; ; in, Here are the coordinates of the point cloud.

[0047] Project all points into the corresponding grid according to the above rules, and set the projection height to the grid grayscale value.

[0048] In one embodiment, step four involves using an edge detection algorithm to extract the contour of the grid image and perform template matching to determine that the contour to be detected is a tray; and locating the center and pose of the tray legs based on the pixels of the entire tray grid image.

[0049] (f1) The raster image is a single-channel grayscale image. Target edges can be detected and contours extracted through pixel changes. The extraction process is as follows: The edge detection algorithm (Canny operator) employs non-maximum suppression for gradient magnitude, preserving pixels with the largest local gradient. It uses a double threshold algorithm to detect and connect edges, achieving excellent detection results for blurred edges. Therefore, this embodiment uses the Canny operator to perform edge detection on the raster image.

[0050] Among them, the dual threshold algorithm is a smart algorithm used to filter out real edges and eliminate edge breaks. That is, two thresholds are set (a high threshold and a low threshold). (1) First, preliminary screening: any pixel with a gradient magnitude higher than the high threshold is immediately marked as a "strong edge" pixel, and it is believed that these are real edges. Any pixel with a gradient magnitude lower than the low threshold is immediately suppressed (not considered as an edge). Any pixel with a gradient magnitude between the low threshold and the high threshold is marked as a "weak edge" pixel. These may be real edges or noise, which need to be further judged. (2) Connecting edges (hysteresis connection): only when a "weak edge" pixel is connected to a "strong edge" pixel (e.g., within an 8-neighborhood) will the "weak edge" pixel be promoted to a "strong edge" and considered as part of a real edge. If there are no "strong edge" pixels around a "weak edge" pixel, it will be discarded.

[0051] Considering the sparse point cloud at the edge of the tray, morphological operations are needed to connect and repair discontinuous edges based on edge detection, and to extract their outer contours. For the segmented tray, to preserve the edge contour information of its front surface, closing operations are used to remove surface holes and smooth edge information.

[0052] Morphological operations are based on a kernel called a structuring element (usually a small matrix). Dilation expands the white areas (foreground) in an image, connecting adjacent white areas and filling small holes. Erosion shrinks the white areas (foreground) in an image, eliminating small, isolated white noise and separating connected objects.

[0053] The closing operation removes surface pores and smooths edge information by: first, dilating: thickening the edge lines, connecting previously broken, closely spaced edge segments; then, eroding: "slimming down" the thickened edges back to their original approximate thickness, while preserving the connections formed by the dilution, thus repairing the break.

[0054] Perform pixel-level retrieval from a point on the edge along a certain direction until you return to that edge point, and then extract the contour using the method described above.

[0055] (f2) Template matching; The contour boundary exhibits good rotation, translation, and scaling invariance, and its pixel count is significantly lower than that of the complete target area, thus providing a relatively complete description of the object. Hu invariant moments and scale-proportion features can be fused as the tray contour feature vector, and a similarity function between the target contour and the tray template contour can be established for tray detection and recognition. Moment features are primarily used to describe the ensemble features of the image, possessing rotation, translation, and scaling invariance, effectively adapting to tray offsets in terms of lateral distance and angle, making tray detection more robust under offset conditions. The template matching algorithm is as follows: Calculate the Hu invariant moments of the profile for any non-negative integer. , The length and width are respectively and Discrete digital image of pixels Corresponding central moments The expression is: ; in, For point cloud coordinates, These are the averages.

[0056] Normalized central moments Defined as: ; in, for and The center distance corresponding to when all parameters are 0 is taken. .

[0057] During image matching, only two Hu invariant moments are well preserved, namely: ; ; in, for and The normalized central moments corresponding to taking 2 and 0 respectively, for and The normalized central moments corresponding to taking values ​​of 0 and 2 respectively. for and The normalized central moments corresponding to taking values ​​of 1 and 1 respectively. These are two Hu invariant moments, respectively.

[0058] For any contour C, using pixels as the basic unit, the length and width of its bounding rectangle are defined as follows: and The perimeter of the outline is The outline area is This is used to construct the aspect ratio of the outline. and duty cycle Two scale proportions: ; .

[0059] Before the experiment, different template outlines need to be prepared. Based on the operational scenario, a frontal outline of the vehicle should be generated in advance according to the actual vehicle's drawings or dimensions. Then, perspective transformation is used to simulate the vehicle's outline deflection from -30° to 30°. In a soft-pack stacking scenario, outlines with pitch angles can be constructed to form the projected template outlines of different vehicle structures, such as... Figure 5 As shown. This allows the solution to effectively adapt to any standard or non-standard vehicle, and also to pallets. Rotation between them.

[0060] In practical application, based on the point cloud of the vehicle actually captured, the front plane is extracted according to the above steps, projected into a raster image, the contour is extracted, and then matched according to the following formula: For the extracted contour and template outline Contour similarity is established using Euclidean distance.

[0061] ; Among them, template outline For the pre-defined template outline, for The Euclidean distance between them for Hu invariant moment, for Hu invariant moment, For value index, for The scale and proportion characteristics, for The scale and proportion characteristics.

[0062] in, and according to The values ​​1 and 2 correspond to , , and The calculation process is as described above. and Special note and refer to The calculation formula, and refer to The calculation formula.

[0063] Similarly, and according to The values ​​1 and 2 correspond to respectively , , and The calculation process is as described above. and Special note and refer to The calculation formula, and refer to The calculation formula.

[0064] The lower the similarity, the greater the deviation between the extracted contour and the template contour. Based on the yaw and pitch angles calculated in the above steps, the similarity between the extracted contour and several template contours with the closest angles can be calculated. The contour with the highest similarity is compared with a set threshold. If the similarity is greater than the threshold, the extracted contour is determined to be a tray.

[0065] It should be noted that multiple template contours may be set as needed. Before similarity calculation, the template contours are traversed based on the yaw angle and pitch angle calculated in step two. The top-H template contours that are closest to the yaw angle and pitch angle are selected as candidate template contours, where H is an integer. The similarity between the extracted contour and the candidate template contours is calculated respectively. The maximum similarity is then compared with a set threshold to determine whether the extracted contour is a tray.

[0066] (f3) Locate the center and pose of the tray legs based on the pixels of the entire tray grid; Based on the contour of the front plane of the tray, the contour can be further processed to locate the center of the tray legs. The tray pose is calculated using the centers of multiple legs (at least two legs are included). The algorithm flow is as follows: Step 1: For the entire tray raster image, traverse the pixels in each row from top to bottom and from left to right, perform a first-order difference operation, and extract the pixels whose pixel value difference between adjacent points is 255 and put them into a set, which are the edge points.

[0067] Step 2: Set more conditions to filter out noise generated by goods and the external environment. For example: 1. Set a row distance threshold to retain points whose distance between adjacent edge points in the same row is less than or equal to the row distance threshold, ensuring that the distance between adjacent edge points in the same row is not too close or too far. 2. For double-hole trays, set the number of edge points in a row of pixels to R (e.g., R=6). 3. The distance between adjacent edge points vertically cannot exceed the vertical distance threshold, ensuring that the distance between adjacent edge points vertically is not too large.

[0068] Step 3: For special scenarios, such as vehicle tilt, the minimum bounding rectangle of the template outline can be calculated, the rotation can be corrected according to the direction of the minimum bounding rectangle, and then detection can be performed. Finally, the identified coordinates can be rotated again. For deformable pallets, after correctly extracting the front plane outline of the pallet, the outline of the pallet legs can also be extracted based on the number of jump points in a search row or by adding additional conditions, thereby identifying the center.

[0069] Step 4: After extracting the left and right contours of each pallet leg, calculate the average coordinates of the left and right contours of each pallet leg, and then calculate the centroids of the original 3D point cloud projected onto the grid. This will give you the center coordinates of each pallet leg, and from this, you can obtain the center point and pose of the pallet.

[0070] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for pallet recognition and localization based on point cloud perception, the method comprising: The method comprises the following steps: acquire the original point cloud set of the field of view area and pre-process, calculate the normal vector of each point in the pre-processed point cloud set using principal component analysis algorithm; take one of the points as a seed point, when the angle between the normal vector of the seed point and the normal vector of the surrounding neighborhood point is less than a set threshold, the two points are classified into the same class, construct a point cloud plane composed of points in the same class, calculate the normal vector of the point cloud plane using principal component analysis algorithm, and calculate the yaw angle and pitch angle of the point cloud plane relative to the camera coordinate system; Using the sensor viewpoint as the origin, and the normal vector of the point cloud plane as... Establish a coordinate system using the axis unit vector, and then plot the preprocessed point cloud along... Projected along the axis A raster image is obtained from a flat grid, and the projection height is set to the raster grayscale value; use edge detection algorithm to extract the contour of the grid map and template matching to determine that the detected contour is a tray; locate the center and pose of the tray according to the pixel points of the entire tray grid map.

2. The method of claim 1, wherein, The original point cloud set of the field of view area of the laser radar is acquired and pre-processed, and the pre-processing process is as follows: determine the 3D voxel size based on the minimum volume surrounding the original point cloud set; based on the point cloud coordinates, divide each point into the voxel it belongs to, and calculate the center of gravity of each voxel based on the coordinates of each point in the voxel; filter out the jumping noise points appearing at the edge.

3. The method of claim 1, wherein, The jumping noise points appearing at the edge are filtered out, specifically as follows: define the point with the maximum depth difference between the current point and the adjacent Q points as a jumping point if the maximum depth difference is greater than a set depth threshold; randomly select a point, and form a point set with the nearest W points around the point. calculating the average depth of W+1 points within the point set calculating the maximum depth difference between the selected point and the rest of the points in the point set when considering the point as a jump noise point and filtering it out, wherein is a set threshold parameter.

4. The method of claim 1, wherein, Before the normal vector of each point is calculated using the principal component analysis algorithm, the pre-processed point cloud set is segmented using a region growing algorithm based on normal vector constraint, specifically as follows: The search is performed in the direction of the axis for each row of the tray When the number of jump points appearing in the vicinity of the same coordinate exceeds the jump setting threshold value, the axis region in which the tray is located is determined When the number of jump points appearing in the vicinity of the same coordinate exceeds the jump setting threshold value, the axis region in which the tray is located is determined axis Within the axial region of the tray search for periodic changes in depth along the axis, and take a single tray's axial region.

5. The method of claim 1, wherein, The generation process of the normal vector of each point is as follows: Setting search neighborhood parameters , traversing one of the points nearest neighbors, constructing a neighborhood covariance matrix for each point; take the eigenvector corresponding to the minimum eigenvalue in the neighborhood covariance matrix as the normal vector corresponding to the point; The angle between the point corresponding normal vector and the z-axis of the camera coordinate system is calculated, and if the angle is less than 0, the normal vector is flipped. The angle between the point corresponding normal vector and the z-axis of the camera coordinate system is calculated, and if the angle is less than 0, the normal vector is flipped.

6. The method of claim 5, wherein, Before taking one of the points as a seed point, set an angle threshold, and consider that the points are close when the angle between the normal vectors of the points is less than the angle threshold; set the upper limit and lower limit of the number of point sets in the growing region, and remove the points exceeding the upper limit or below the lower limit; After obtaining the yaw angle and pitch angle of the point cloud plane relative to the camera coordinate system, flip the normal vector corresponding to the yaw angle less than 0 or the pitch angle less than 0.

7. The method of claim 1, wherein, The preprocessed point cloud along Projected along the axis Before the grid in the plane, a threshold range for the yaw angle is set. If the calculated yaw angle is not within the threshold range, the point cloud plane corresponding to that yaw angle is removed to initially identify interfering planes.

8. The method of claim 1, wherein, The edge detection algorithm is used to extract the contour of the grid map and template matching to determine whether the extracted contour is a tray, wherein the template matching process is as follows: calculate the Hu invariant moment of the extracted contour; constructing an aspect ratio of the extracted contour and a duty cycle ; generate a contour map of the front of the actual carrier according to the drawing or size of the carrier, and obtain the contours in different directions through perspective transformation, so as to construct multiple template contours; use the yaw angle and pitch angle corresponding to the extracted contour to traverse the template contours, select the top-H template contours closest to the yaw angle and pitch angle as the candidate template contours, and H is an integer; Hu moment, contour length-width ratio and duty cycle The similarity between the extracted contour and the candidate template contour is calculated, and it is determined whether the similarity is greater than a set similarity threshold. If yes, the extracted contour is a tray, otherwise, it is not a tray.

9. The method of claim 8, wherein, the extracted contour the similarity between the candidate template contour The similarity calculation formula between the candidate template contour and the extracted contour is as follows: ； wherein, is Euclidean distance between is Hu moment of is Hu moment of is the value index, is scale ratio feature of is scale ratio feature of 10. The method of claim 1, wherein, The center and pose of the tray leg are located according to the pixel points of the entire tray grid map, specifically as follows: perform first-order difference operation on the pixel points in each row from left to right in the pixel points of the entire tray grid map from top to bottom, extract the pixel points with a pixel value difference of 255 between adjacent points into a set, which are the edge points. Set the row distance threshold, the distance between adjacent edge points of the same row is less than or equal to the point within the row distance threshold range is retained; for double-hole tray, set a row of pixel points to exist R edge points; the distance between the adjacent edge points above and below cannot exceed the up-down distance threshold; For the tilt of the carrier, the minimum circumscribed rectangle of the template contour is calculated, the direction of the minimum circumscribed rectangle is corrected for rotation, and then detection and template matching are performed, and the recognized coordinates are rotated again; After extracting the left and right contours of each tray leg, the mean value of the left and right contour coordinates of each tray leg is calculated, and the centroid of all the original point clouds projected onto the grid map is calculated, so as to obtain the center point and pose of the tray.

Images