Tray detecting and positioning method, system and equipment for unmanned forklift and medium

Abstract

The invention belongs to the technical field of robot sensing and positioning, and discloses a tray detecting and positioning method, system and device of an unmanned forklift and a medium. The method comprises the steps of obtaining original laser radar point cloud data of a tray, performing preprocessing, and projecting the preprocessed laser radar point cloud data into an image; processing the image by a preset detection model to obtain a tray identification result and a corresponding confidence coefficient, obtaining an effective tray according to the confidence coefficient, and performing digital image processing based on the laser radar point cloud data corresponding to the effective tray to obtain the position of the center of the tray. According to the laser radar point cloud data of the effective tray, plane fitting is carried out, a normal vector is calculated, and yaw angle information of the tray relative to a vehicle body is obtained; and based on the yaw angle information and the position of the center of the tray, the deviation position from the fork tooth edge to the center of the tray when the forklift forks the tray is calculated, and a task is forked based on the deviation position. The distances from the edges of the prongs to the center of the tray are compared, and it is ensured that the tray is forked correctly.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot perception and positioning, and particularly relates to a pallet detection and positioning method, system, device and medium for a unmanned forklift. BACKGROUND

[0002] In an intelligent logistics and automated warehousing system, accurate identification and positioning of a pallet is a key technology for realizing automated loading and unloading. Pallet detection refers to automatically extracting a pallet from laser radar point cloud data or camera data. Pallet end positioning refers to obtaining the center position of the pallet, which is generally the geometric center position of the pallet. The position is then output to a planning / control module to realize pallet forking and other operations. Current unmanned forklifts mostly use image processing or laser radar point cloud filtering methods for pallet detection, which may have certain false detection or missed detection. In end positioning, template matching is generally relied on, and is easily affected by pallet size and pallet shape, resulting in poor robustness and end positioning accuracy. SUMMARY

[0003] The present application provides a pallet detection and positioning method, system, device and medium for a unmanned forklift, which solves the problems of large end positioning error and high requirements for pallet specifications of the unmanned forklift, and improves the accuracy of pallet end positioning.

[0004] The present application provides a pallet detection and positioning method for a unmanned forklift, comprising: Obtaining original laser radar point cloud data of a pallet, preprocessing, and projecting the preprocessed laser radar point cloud data into an image; A preset detection model processes the image to obtain a pallet recognition result and a corresponding confidence, and an effective pallet is obtained according to the confidence; Based on the laser radar point cloud data corresponding to the effective pallet, a digital image is processed to obtain the position of the center of the pallet, a plane is fitted according to the laser radar point cloud data of the effective pallet, and a normal vector is calculated to obtain the yaw angle information of the pallet relative to the vehicle body; Based on the yaw angle information and the position of the center of the pallet, the offset position of the tine edge of the forklift to the center of the pallet when the forklift forks the pallet is calculated, and the forking task is executed based on the offset position.

[0005] Optionally, the step of processing the laser radar point cloud data corresponding to the effective pallet to obtain the position of the center of the pallet comprises: The laser radar point cloud data points corresponding to the effective pallet are projected and converted into a digital image according to the pixel resolution of the yz plane point cloud; Gaussian and normalization processing is performed, and adaptive threshold segmentation is performed by using the maximum inter-class variance method; The digital image is opened to remove small noise and fine connection, the outer contour of the tray is extracted, noise regions with small areas are filtered out through area size, and the position of the tray center in the digital image is obtained.

[0006] Optionally, the step of obtaining the yaw angle information of the tray relative to the vehicle body according to the laser radar point cloud data of the effective tray by fitting a plane and calculating a normal vector comprises: randomly selecting a minimum point set from the laser radar point cloud data of the effective tray by using a random sample consensus algorithm; selecting an inner point set with the most inner points as a plane model after iterating the minimum point set multiple times; converting the nonlinear fitting into a nonlinear least squares problem to obtain an optimal plane equation of the fitted inner points; calculating an included angle between the optimized plane and a reference direction as the yaw angle information of the tray relative to the vehicle body by using a dot product.

[0007] Optionally, the step of randomly selecting a minimum point set from the laser radar point cloud data of the effective tray by using a random sample consensus algorithm and selecting an inner point set with the most inner points as a plane model after iterating the minimum point set multiple times comprises: using a random number generation method to randomly select a plurality of points from all possible points to form a minimum point set; calculating equation parameters of a plane according to coordinates of the minimum point set as an initial plane model; calculating distances from each point to the initial plane model, determining points with distances less than a preset distance threshold as inner points, and counting the number of the inner points; performing next iteration according to the current inner point set and corresponding model parameters until a preset iteration number is reached, and selecting an inner point set with the most inner points as a plane model.

[0008] Optionally, the step of converting the nonlinear fitting into a nonlinear least squares problem to obtain an optimal plane equation of the fitted inner points comprises:

[0009] constructing a residual function by using the fitted inner point set and the plane equation; constructing a Jacobian matrix based on the residual function using automatic differentiation to obtain a plane parameter update amount, and iterating until convergence to obtain an optimized plane equation.

[0010] Optionally, the preset detection model is constructed in the following process: preliminarily screening from a yolo series, ssd, fasterRcnn and a segmentation-based method according to a tray detection requirement to determine a model corresponding to the method; building a corresponding deep learning framework according to the selected model, and inputting a training data set required for tray detection; The performance indicators of the model are monitored during the training process to adjust parameters, and after the training is completed, the model is evaluated and optimized to obtain a detection model.

[0011] Optionally, the step of performing the fork picking task based on the offset position comprises: If the y and z deviations of the fork tooth edge to the pallet center do not exceed the threshold value, the fork picking task is performed. Otherwise, the vehicle body posture needs to be adjusted again, the step of obtaining the original laser radar point cloud data of the pallet is repeated until the fork picking task is performed, or the number of adjustments is greater than the preset number of times, and the pallet fork picking task is exited.

[0012] According to another aspect of the present application, a pallet detection and positioning system of an unmanned forklift is provided, comprising: A preprocessing unit configured to obtain original laser radar point cloud data of a pallet, perform preprocessing, and project the preprocessed laser radar point cloud data into an image; A detection unit configured to process the image using a preset detection model to obtain a pallet recognition result and a corresponding confidence level, and obtain an effective pallet according to the confidence level; An end positioning unit configured to perform digital image processing on the laser radar point cloud data corresponding to the effective pallet to obtain the position of the center of the pallet, fit a plane according to the laser radar point cloud data of the effective pallet, and calculate a normal vector to obtain the yaw angle information of the pallet relative to the vehicle body; An execution unit configured to calculate the offset position of the fork tooth edge to the center of the pallet when the forklift picks up the pallet based on the yaw angle information and the position of the center of the pallet, and perform a fork picking task based on the offset position.

[0013] According to another aspect of the present application, an electronic device is provided, comprising: At least one processor; and A memory in communication connection with the at least one processor; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the pallet detection and positioning method of the unmanned forklift according to any one of the embodiments of the present application.

[0014] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the pallet detection and positioning method of the unmanned forklift according to any one of the embodiments of the present application when executed.

[0015] The technical scheme of the embodiment of the application, the method comprises: acquiring original laser radar point cloud data of a pallet, pre-processing, projecting the pre-processed laser radar point cloud data into an image; a preset detection model processes the image to obtain a pallet recognition result and a corresponding confidence, and acquires an effective pallet according to the confidence; based on the laser radar point cloud data corresponding to the effective pallet, a digital image processing is performed to obtain the position of the center of the pallet, a plane is fitted according to the laser radar point cloud data of the effective pallet, and a yaw angle information of the pallet relative to the vehicle body is obtained by calculating the normal vector; based on the yaw angle information and the position of the center of the pallet, the offset position of the tine edge to the center of the pallet when the pallet is forked by the forklift is calculated, and if the offset position deviation does not exceed a threshold value, the forking task is executed.

[0016] The application converts the laser radar point cloud data of the effective pallet into a digital image, extracts position information, realizes accurate mapping from three-dimensional points to a two-dimensional plane, calculates the yaw angle information of the pallet relative to the vehicle body through nonlinear fitting, and accurately reflects the relative attitude between the pallet and the vehicle body, and calculates the position and attitude information of the pallet.

[0017] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the application, nor is it used to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0019] Figure 1 The pallet detection and positioning method flow chart of the unmanned forklift provided by the application.

[0020] Figure 2 The pallet detection and positioning system module diagram of the unmanned forklift provided by the application.

[0021] Figure 3 The structural schematic diagram of an embodiment of the computer equipment provided by the application.

[0022] Figure 4 The structural schematic diagram of an embodiment of the computer readable storage medium provided by the application.

[0023] Figure 5 The detection algorithm flow chart of the pallet detection and positioning method of the unmanned forklift provided by the application.

[0024] Figure 6 The positioning algorithm flowchart of the tray detection and positioning method of the unmanned forklift provided by the present application.

[0025] Figure 7 The laser radar collection and identification diagram of the tray detection and positioning method of the unmanned forklift provided by the present application.

[0026] Figure 8 The projection confidence diagram of the tray detection and positioning method of the unmanned forklift provided by the present application.

[0027] Figure 9 The center position diagram of the tray with a confidence of p0 of the tray detection and positioning method of the unmanned forklift provided by the present application.

[0028] Figure 10 The center position diagram of the tray with a confidence of p1 of the tray detection and positioning method of the unmanned forklift provided by the present application. DETAILED DESCRIPTION

[0029] In order to make the personnel in the art better understand the present application scheme, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0030] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0031] This invention employs a combination of coarse and fine positioning. Coarse positioning uses deep learning-based methods to acquire complete pallet laser point cloud data; fine positioning uses image processing and nonlinear fitting algorithms to obtain the pallet's attitude (x, y, z, yaw) relative to the vehicle body. Specifically, it involves obtaining the pallet's yz cross-section from the point cloud data, then using digital image processing techniques such as filtering and contour extraction to achieve fine end-effector positioning of the pallet, and finally outputting the pallet's yaw angle by fitting a plane and calculating the normal vector. When the forklift is about to pick up the pallet, the position from the edge of the fork teeth to the center of the pallet is calculated for secondary recognition to ensure the forklift correctly picks up the pallet and guarantees the safety of the vehicle and goods. If the y and z deviations from the edge of the fork teeth to the center of the pallet simultaneously meet preset thresholds, the picking task is executed; if one or more of them do not meet the preset thresholds, the vehicle attitude needs to be readjusted, and the above process is repeated. When the number of adjustments exceeds 3 times or other preset thresholds, the pallet picking task is terminated. This solves the problems of large pallet end-effector positioning errors and high pallet specification requirements of unmanned forklifts, improving the accuracy of pallet end-effector positioning.

[0032] According to one aspect of the invention, please refer to Figure 1 and Figure 5 It provides a method for pallet detection and positioning of unmanned forklifts, including: S1. Obtain the original LiDAR point cloud data of the tray, perform preprocessing, and project the preprocessed LiDAR point cloud data into an image. S2. The preset detection model processes the image to obtain the tray recognition result and the corresponding confidence level, and the valid tray is obtained based on the confidence level; S3. Based on the lidar point cloud data corresponding to the effective pallet, perform digital image processing to obtain the position of the pallet center. Based on the lidar point cloud data of the effective pallet, fit a plane and calculate the normal vector to obtain the yaw angle information of the pallet relative to the vehicle body. S4. Calculate the offset position from the edge of the fork teeth to the center of the pallet when the forklift picks up the pallet based on the yaw angle information and the position of the pallet center, and perform the picking task based on the offset position.

[0033] Pallet detection primarily involves acquiring LiDAR point cloud data, evaluating the LiDAR point cloud data, converting valid LiDAR point cloud data into images, processing the images using deep learning algorithms, and processing the model's output. Based on confidence levels, it determines whether a pallet is valid. If a pallet is valid, the extracted pallet point cloud data is output to the end-effector localization unit. The specific details are as follows: LiDAR point cloud data can be obtained either directly from the network or via ROS messages.

[0034] The evaluation of lidar point cloud data can be performed by determining whether the point cloud is empty, or by identifying null values, the number of point clouds, and other relevant information.

[0035] Point cloud to image conversion mainly involves converting point cloud data from the yz plane into an image based on a preset resolution.

[0036] The end-point positioning unit primarily outputs the pallet's position (x, y, z) and its yaw angle relative to the vehicle body. The input data for the end-point positioning unit is pallet data, which is in the format of LiDAR point cloud data. The specific implementation method of this end-point positioning unit is described below.

[0037] Acquire tray data, primarily the tray point cloud data output by the tray detection module. This can be obtained via ROS topic messages or other methods.

[0038] The data anomaly module mainly determines whether the tray data is null, the number of point clouds is too small, or the point cloud position does not conform to the detection range of the lidar.

[0039] By directly acquiring and preprocessing the raw LiDAR point cloud data of the pallet, the integrity and accuracy of the data can be ensured, avoiding errors introduced by improper data conversion or processing. The preprocessed LiDAR point cloud data is projected into an image, converting three-dimensional spatial information into two-dimensional image information. A pre-defined detection model processes the image, enabling rapid and accurate identification of the pallet and providing a corresponding confidence score. Valid pallets are selected based on the confidence score, improving the reliability of the identification.

[0040] The yaw angle of the pallet relative to the vehicle body is obtained by nonlinear fitting calculation, which can accurately reflect the relative attitude between the pallet and the vehicle body. The position and yaw angle information is visualized to facilitate remote monitoring and manual intervention.

[0041] In some embodiments, please refer to Figure 1 The step of performing digital image processing based on the lidar point cloud data corresponding to the effective tray to obtain the position of the tray center includes: The point cloud data points of the LiDAR corresponding to the effective tray are converted into digital images by projecting the point cloud onto the yz plane according to the pixel resolution. The data are subjected to Gaussian and normalization processing, and adaptive threshold segmentation is performed using the Otsu's method. The digital image is processed to remove small noise and fine connections, the outer contour of the tray is extracted, and smaller noise areas are filtered out by area size to obtain the position of the center of the tray in the digital image.

[0042] Point cloud projection converts the point cloud projection on the yz plane into an image based on pixel resolution, which facilitates the subsequent extraction of the pallet centroid.

[0043] Image preprocessing mainly involves Gaussian processing to filter out noise. Normalization maps floating-point numbers to the range of 0-255.

[0044] For morphological processing, Otsu's method (maximum inter-class variance method) is first used to achieve adaptive thresholding. Then, the image is opened to remove small noise and fine connections.

[0045] Contour extraction is performed to extract the outer contour of the tray. Then, smaller noise areas are filtered out based on their area size.

[0046] Calculate the center pixel region, calculate the minimum bounding square of the effective contour, and then obtain the geometric center point. Then perform coordinate transformation and output the position (x, y, z), where the unit can be meters or centimeters.

[0047] The preprocessed LiDAR point cloud data points are projected onto the yz plane and converted into images based on pixel resolution, transforming the complex 3D point cloud data into a more easily processed 2D image format. The converted images are then subjected to Gaussian and normalization processing, and adaptive threshold segmentation is performed using the Otsu's method. Gaussian processing effectively smooths the image, reduces noise interference, and makes image features clearer; normalization unifies the data scale and enhances data comparability. The Otsu's method adaptively determines the threshold for segmentation, accurately distinguishing the tray area from the background area. Compared to fixed threshold segmentation, it is more adaptable to differences in images under different scenarios, greatly improving the accuracy and robustness of segmentation.

[0048] The opening operation removes small noise points and fine connections from the image, making the tray's outer contour clearer and more complete. Then, by extracting the tray's outer contour and filtering out smaller noise areas based on their size, the image is further purified. The resulting digital image, after this series of processing steps, not only retains the tray's key feature information but also removes a large amount of irrelevant noise interference, ensuring the reliability and accuracy of the entire tray recognition and positioning system.

[0049] In some embodiments, please refer to Figure 1 and Figure 6 , The step of obtaining the yaw angle information of the pallet relative to the vehicle body by fitting a plane and calculating the normal vector based on the effective pallet's lidar point cloud data includes: The minimum set of points is randomly selected based on the LiDAR point cloud data of the valid tray using a random sampling consensus algorithm. After iterating through the minimum point set multiple times, the set of interior points with the most interior points is selected as the planar model. The nonlinear fitting is transformed into a nonlinear least squares problem, and the optimal plane equation for fitting interior points is obtained. The angle between the optimized plane and the reference direction is calculated using dot product as the yaw angle information of the pallet relative to the vehicle body.

[0050] The yaw angle calculation mainly involves performing plane fitting on the extracted pallet point cloud data and calculating the normal vector to obtain the yaw angle information of the pallet relative to the vehicle body.

[0051] Randomly sampling points from complex point cloud data avoids model bias caused by local data features, ensuring the selected point set better represents the overall data distribution. Continuous filtering and optimization gather inliers that conform to the characteristics of a planar model. Constructing a planar model using the largest inlier set significantly improves the model's accuracy and reliability, effectively reducing the impact of noise points and outliers, resulting in a planar model that more closely reflects reality. Nonlinear least squares seeks the optimal solution by minimizing the sum of squared errors. Applying it to solving the plane equations allows for precise determination of the plane's parameters, minimizing the error between the fitted plane and the actual data points, thus yielding a more accurate plane equation.

[0052] The angle between the optimized plane and the reference direction is used as the yaw angle information of the pallet relative to the vehicle body by calculating the dot product. The dot product calculation can accurately reflect the angular relationship between the two vectors. By performing the dot product operation between the optimized plane normal vector and the reference direction, the yaw angle can be obtained quickly and accurately.

[0053] In some embodiments, please refer to Figure 1 The step of randomly selecting a minimum point set using a random sampling consensus algorithm, and then selecting the set of interior points with the most interior points as the planar model after iterating through the minimum point set multiple times, includes: Use random number generation methods to randomly select a number of points from all possible points to form a minimum point set; The equation parameters of the plane are calculated based on the coordinates of the minimum point set, and used as the initial plane model. Calculate the distance from each point to the initial planar model, identify points whose distance is less than a preset distance threshold as interior points, and count the number of interior points; The next iteration is performed based on the current set of interior points and the corresponding model parameters until the preset number of iterations is reached. The set of interior points with the most interior points is then selected as the planar model.

[0054] The yaw angle calculation of the end-positioning unit is mainly based on the extracted pallet point cloud data, which is then fitted to a plane and the normal vector is calculated to obtain the yaw angle information of the pallet relative to the vehicle body.

[0055] The Random Sample Consensus Algorithm (RANSAC) is a robust parameter estimation method. For the plane equation ax + by + cz + d = 0, any three non-collinear points can uniquely determine a plane.

[0056] The RANSAC plane is calculated by randomly sampling the minimum set of points. The plane equation is: ax + by + cz + d = 0. A simplified plane equation, ax + by + c = 0 (lacking the z term), can also be used. Interior point statistics and optimal model selection are then performed. RANSAC calculates model parameters by randomly selecting a minimum set of points (3 points are needed for the plane), then counts the number of points (interiors) that fit the model. After multiple iterations, the model with the most interiors is selected.

[0057] RANSAC calculates model parameters by randomly selecting a minimum set of points (3 points for a plane), then counting the number of points that conform to the model (interior points), iterating multiple times, and finally selecting the model with the most interior points. Its implementation process is as follows: randomly select 3 points to calculate the plane equation, calculate the distance from all points to the plane, count the points whose distance is less than a threshold (interior points), repeat this process multiple times, select the plane model with the most interior points, and recalculate more accurate plane parameters using all interior points.

[0058] It can randomly extract samples from massive point cloud data, avoiding biases caused by human selection or local data features. This randomness ensures that the selected minimum set of points is representative and can cover data points from different regions and features.

[0059] Distance thresholding accurately distinguishes between data points that conform to the characteristics of the planar model and noise points, separating the qualified inliers. Multiple iterations continuously optimize the planar model, gradually approximating the true data distribution. Each iteration adjusts the model based on the previous result, progressively eliminating unqualified points and adding qualified ones, thus continuously improving the model's accuracy and reliability. Selecting the set of inliers with the most inliers as the final planar model ensures that the model fits the actual data to the greatest extent possible.

[0060] In some embodiments, please refer to Figure 1 The steps of transforming the nonlinear fitting into a nonlinear least squares problem and obtaining the optimal plane equation for the fitted interior points include:

[0061] The residual function is constructed by fitting the set of interior points and the plane equation; Based on the residual function, the Jacobian matrix is ​​constructed using automatic differentiation. The updated plane parameters are obtained by solving the problem. The process is iterated until convergence, and the optimized plane equation is obtained.

[0062] Nonlinear optimization primarily transforms plane fitting into a least-squares optimization problem. Specifically, it involves extracting and filtering interior points using a nonlinear least-squares method, or alternatively, employing the Levenberg-Marquardt algorithm. The implementation process can be as follows: define the residual function (distance from a point to the plane), calculate the Jacobian matrix using automatic differentiation, construct the normal equation and solve for the parameter update, iterating until convergence.

[0063] The angle between the optimized plane and the reference direction is calculated mainly by using the dot product to calculate the angle between the two vectors.

[0064] The residual function accurately measures the deviation between the actual data points and the fitted plane, transforming the complex fitting problem into a pursuit of minimizing the residual. It reflects the gap between the current plane equation and the true data, converting the originally abstract nonlinear fitting problem into a quantifiable and operable concrete problem. Automatic differentiation techniques can automatically and accurately calculate the derivative of the residual function with respect to the plane parameters, thereby constructing the Jacobian matrix. The Jacobian matrix contains information on the variation of the residual function in different parameter directions, guiding how to adjust the plane parameters to minimize the residual.

[0065] The plane parameter update is obtained by solving the Jacobian matrix, and the process is iterated until convergence, thus obtaining an optimized plane equation that fits the actual data well. When the convergence condition is met, the set of interior points is fitted to the maximum extent, accurately describing the characteristics of the plane containing the pallet.

[0066] In some embodiments, please refer to Figure 1 The construction process of the preset detection model is as follows: Based on the requirements of pallet detection, a preliminary screening was conducted among YOLO series, SSD, Faster R-CNN and segmentation-based methods to determine the corresponding models for each method. Based on the selected model, build the corresponding deep learning framework and input the training dataset required for tray detection; During training, the model's performance metrics are monitored and parameters are adjusted. After training is completed, the model is evaluated and optimized to obtain the detection model.

[0067] Deep learning models, including classic and novel methods such as YOLO, YOLOx, SSD, and Faster R-CNN, are trained on pre-labeled pallet data to produce models suitable for pallet detection. These models are then deployed on unmanned forklifts. The input data for this model is an image converted from LiDAR point clouds, and the output is all pallet information in the image, including bounding boxes and confidence scores.

[0068] The YOLO series of algorithms are known for their speed, capable of processing a large number of images in a short time, making them suitable for scenarios with high real-time requirements; the SSD algorithm achieves a good balance between speed and accuracy, meeting most common detection needs; the Faster R-CNN algorithm has high accuracy and performs well in pallet detection in complex scenes; while segmentation-based methods can extract pallet contour information more precisely.

[0069] In some embodiments, please refer to Figure 1 The step of performing the fork task based on the offset position includes: If the y and z deviations from the edge of the fork tines to the center of the pallet do not exceed the threshold, then the fork picking task is executed; Conversely, the vehicle's posture needs to be adjusted again, and the steps of acquiring the original LiDAR point cloud data of the pallet need to be repeated until the pallet picking task is executed, or the pallet picking task is exited if the number of adjustments exceeds the preset number.

[0070] When a forklift is about to pick up a pallet, the position from the edge of the fork tines to the center of the pallet is calculated for secondary identification to ensure that the forklift correctly picks up the pallet and to guarantee the safety of the vehicle and goods. The specific process can be as follows.

[0071] Point cloud data containing the pallet and fork teeth is obtained through pre-defined region filtering. This point cloud data is converted into an image. For the pallet's yz plane (pallet entry plane) and fork tooth images, Gaussian filtering and Canndy edge detection techniques are used to obtain the lateral and vertical distances between the fork tooth edge lines and the geometric center of the pallet's yz plane. If the y and z deviations from the fork tooth edge to the pallet center simultaneously meet preset thresholds, the pallet picking task is executed. If one or more of these thresholds are not met, the vehicle's posture needs to be adjusted again, and the above process is repeated. When the number of adjustments exceeds three times or other preset thresholds, the pallet picking task is terminated.

[0072] According to another aspect of the present invention, a pallet detection and positioning system for unmanned forklifts is provided; please refer to [link to relevant documentation]. Figure 2 ,include: The preprocessing unit 100 is configured to acquire the original lidar point cloud data of the tray, perform preprocessing, and project the preprocessed lidar point cloud data into an image. The detection unit 200 is configured to process the image using a preset detection model to obtain the tray recognition result and the corresponding confidence level, and to obtain the valid tray based on the confidence level. The end-positioning unit 300 is configured to perform digital image processing based on the lidar point cloud data corresponding to the effective pallet to obtain the position of the pallet center, and to perform plane fitting and calculate the normal vector based on the lidar point cloud data of the effective pallet to obtain the yaw angle information of the pallet relative to the vehicle body. The execution unit 400 is configured to calculate the offset position of the fork tooth edge from the center of the pallet when the forklift picks up the pallet based on the yaw angle information and the position of the pallet center, and to perform the picking task based on the offset position.

[0073] This invention comprises a pallet detection unit and an end-effector localization unit. Pallet detection primarily acquires LiDAR point cloud data, evaluates the LiDAR point cloud data, converts valid LiDAR point cloud data into images, processes the images using deep learning algorithms, and processes the model's output. Based on confidence levels, it determines whether a pallet is valid. If a pallet is valid, the extracted pallet point cloud data is output to the end-effector localization unit. The end-effector localization unit mainly includes the pallet itself.

[0074] Pallet detection is adaptable to various pallet shapes, including zigzag, grid, and sun-shaped pallets, as well as pallets made of different materials such as metal, wood, and plastic. It can operate in both day and night conditions, unaffected by lighting conditions. It is not limited by pallet shape or size, and can accommodate various specifications and sizes of pallets, offering better robustness and making it more suitable for industrial applications. The yaw angle for pallet end-positioning is calculated using plane fitting and normal vector solving, ensuring reliable efficiency and accuracy. It can be used with sensors other than LiDAR, such as cameras. For abnormal data handling, the acquired sensor data is first anomaly detected, including but not limited to checking for null values, data boundaries, and empty data. LiDAR point cloud data is used, including but not limited to converting the LiDAR point cloud to an image based on resolution, converting the image coordinate system to the LiDAR coordinate system based on resolution, and converting the LiDAR coordinate system to the vehicle coordinate system. The detection outputs the vertical and horizontal distances between the forklift edge and the pallet hole to ensure the forklift can be properly inserted into the pallet hole, guaranteeing the safety of vehicles and goods. The output sensor can display all trays within its field of view, which can be a single tray or multiple trays.

[0075] In some embodiments, please refer to Figure 7 , Figure 8 , Figure 9 and Figure 10 The pallet detection and positioning process of the present invention is as follows: like Figure 7 As shown, the raw point cloud data of the LiDAR is obtained, with two trays as an example.

[0076] The raw point cloud data from the LiDAR radar is processed through ground removal, filtering, and cropping to convert it into an image. The preset detection model is a depth model, and the result of this image processing is as follows. Figure 8 As shown, Figure 8 The example shows two forklifts with double forks picking up two pallets. The double forks are just an example; four-forklifts or other multi-forklifts that are actually used in real life can also be used. The confidence scores of each set of pallets detected are p0=0.9 and p1=0.93, respectively. The confidence score threshold selected in the example is 0.85. Therefore, both sets of pallets are valid pallets.

[0077] The pallet point cloud is filtered based on the confidence level output by the detection model. After processing by the end-effector localization unit, the result is the center position and yaw angle of the pallet. Figure 9 and Figure 10 As shown in the diagram, the small red dot indicates the center position of each tray.

[0078] When the forklift is about to pick up the pallet, the position from the edge of the fork teeth to the center of the pallet is calculated for secondary identification to ensure that the forklift picks up the pallet correctly and to ensure the safety of the vehicle and the goods.

[0079] If the y and z deviations from the edge of the fork tines to the center of the pallet both meet the preset thresholds, then the fork picking task is executed. The y deviation is the lateral distance from the center line of the fork tines to the center of the pallet, and the corresponding threshold is selected between 15cm and 35cm. The z deviation is the longitudinal distance from the center line of the fork tines to the center of the pallet, and the corresponding threshold is selected between 5 and 12cm. The thresholds selected vary depending on the size of the fork tines.

[0080] For the fork width of fork tin A, with a corresponding lateral distance threshold of 35cm and a longitudinal distance threshold of 12cm, after performing the detection and positioning according to this invention, the calculated y-deviation measurement value is 25cm. Since 25cm is less than 35cm, the y-deviation is considered acceptable. The measured z-deviation value is 8cm. Since 8cm is less than 12cm, the z-deviation is also considered acceptable. The conclusion is that fork tin A, when handling pallets, exhibits acceptable y-deviation and z-deviation, and can safely and accurately complete the handling task.

[0081] If one or more of the preset thresholds are not met, the vehicle's posture must be adjusted again, and the above process must be repeated. If the number of adjustments exceeds three times or other preset thresholds, the pallet picking task will be terminated.

[0082] Fork tooth B is narrow. The corresponding lateral distance threshold is set to 15cm, and the longitudinal distance threshold to 5cm. The measured y-deviation is 18cm, which is considered to exceed 15cm, therefore the y-deviation is unacceptable. The measured z-deviation is 3cm, which is considered to be within 5cm, therefore the z-deviation is acceptable.

[0083] Because the y-deviation was unacceptable, the fork tooth position needed adjustment. After three adjustments, the y-deviation measurement of fork tooth B became 16cm. Since 16cm exceeds 15cm, the y-deviation was deemed unacceptable. Therefore, fork tooth B was removed from the pallet picking task.

[0084] This disclosure also provides an electronic device, including: at least one processor; a memory for storing processor-executable instructions; wherein the at least one processor is configured to execute the instructions to implement the methods disclosed in this disclosure.

[0085] Figure 3 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this disclosure. For example... Figure 3 As shown, the electronic device 300 includes at least one processor 301 and a memory 302 coupled to the processor 301, which can perform the corresponding steps in the methods disclosed in the embodiments of this disclosure.

[0086] The processor 301 described above can also be called a central processing unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in this embodiment can be implemented by the integrated logic circuitry in the processor 301 or by software instructions. The processor 301 can be a general-purpose processor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this embodiment can be directly implemented by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can be located in the memory 302, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor 301 reads information from the memory 302 and, in conjunction with its hardware, completes the steps of the method described above.

[0087] Furthermore, various operations / processes according to this disclosure, implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, such as... Figure 4The computer system 1900 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including those described above. Figure 4 A block diagram of a computer system provided for an exemplary embodiment of this disclosure.

[0088] Computer System 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0089] like Figure 4 As shown, the computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 into a random access memory (RAM) 1903. The RAM 1903 may also store various programs and data required for the operation of the computer system 1900. The computing unit 1901, ROM 1902, and RAM 1903 are interconnected via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.

[0090] Multiple components in computer system 1900 are connected to I / O interface 1905, including: input unit 1906, output unit 1907, storage unit 1908, and communication unit 1909. Input unit 1906 can be any type of device capable of inputting information into computer system 1900. Input unit 1906 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 1907 can be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1908 may include, but is not limited to, hard disks and optical disks. Communication unit 1909 allows computer system 1900 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0091] The computing unit 1901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1901 performs the various methods and processes described above. In some embodiments, the methods disclosed in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1908. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 1902 and / or communication unit 1909. In some embodiments, the computing unit 1901 can be configured by any other suitable means (by means of firmware) to perform the methods disclosed in this disclosure.

[0092] This disclosure also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the methods disclosed in this disclosure.

[0093] The computer-readable storage medium in this disclosure can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The aforementioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specifically, the aforementioned computer-readable storage medium may include electrical connections based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0094] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0095] This disclosure also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements the methods disclosed in the embodiments of this disclosure.

[0096] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer.

[0097] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. Two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0098] The modules, components, or units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0099] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. Exemplary hardware logic components that may be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0100] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0101] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0102] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0103] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0104] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0105] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0106] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0107] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for pallet detection and positioning of an unmanned forklift, characterized in that, include: The original LiDAR point cloud data of the tray is acquired, preprocessed, and then projected into an image. The preset detection model processes the image to obtain the tray recognition result and the corresponding confidence score, and the valid tray is obtained based on the confidence score; Digital image processing is performed based on the lidar point cloud data corresponding to the effective pallet to obtain the position of the pallet center. The yaw angle information of the pallet relative to the vehicle body is obtained by fitting a plane and calculating the normal vector based on the lidar point cloud data of the effective pallet. Based on the yaw angle information and the position of the pallet center, the offset position of the fork tooth edge from the pallet center when the forklift picks up the pallet is calculated, and the picking task is performed based on the offset position.

2. The pallet detection and positioning method for an unmanned forklift according to claim 1, characterized in that, The step of performing digital image processing based on the lidar point cloud data corresponding to the effective tray to obtain the position of the tray center includes: The point cloud data points of the LiDAR corresponding to the effective tray are converted into digital images by projecting the point cloud onto the yz plane according to the pixel resolution. The data are subjected to Gaussian and normalization processing, and adaptive threshold segmentation is performed using the Otsu's method. The digital image is processed to remove small noise and fine connections, the outer contour of the tray is extracted, and smaller noise areas are filtered out by area size to obtain the position of the center of the tray in the digital image.

3. The pallet detection and positioning method for an unmanned forklift according to claim 2, characterized in that, The steps of fitting a plane and calculating the normal vector based on the effective pallet's lidar point cloud data to obtain the pallet's yaw angle information relative to the vehicle body include: The minimum set of points is randomly selected based on the LiDAR point cloud data of the valid tray using a random sampling consensus algorithm. After iterating through the minimum point set multiple times, the set of interior points with the most interior points is selected as the planar model. The nonlinear fitting is transformed into a nonlinear least squares problem, and the optimal plane equation for fitting interior points is obtained. The angle between the optimized plane and the reference direction is calculated using dot product as the yaw angle information of the pallet relative to the vehicle body.

4. The pallet detection and positioning method for an unmanned forklift according to claim 3, characterized in that, The step of randomly selecting a minimum point set using a random sampling consensus algorithm, and then selecting the set of interior points with the most interior points as the planar model after iterating through the minimum point set multiple times, includes: Use random number generation methods to randomly select a number of points from all possible points to form a minimum point set; The equation parameters of the plane are calculated based on the coordinates of the minimum point set, and used as the initial plane model. Calculate the distance from each point to the initial planar model, identify points whose distance is less than a preset distance threshold as interior points, and count the number of interior points; The next iteration is performed based on the current set of interior points and the corresponding model parameters until the preset number of iterations is reached. The set of interior points with the most interior points is then selected as the planar model.

5. The pallet detection and positioning method for an unmanned forklift according to claim 4, characterized in that, The steps of transforming the nonlinear fitting into a nonlinear least squares problem and obtaining the optimal plane equation for the fitted interior points include: The residual function is constructed by fitting the set of interior points and the plane equation; Based on the residual function, the Jacobian matrix is ​​constructed using automatic differentiation. The updated plane parameters are obtained by solving the problem. The process is iterated until convergence, and the optimized plane equation is obtained.

6. The pallet detection and positioning method for an unmanned forklift according to claim 1, characterized in that, The process of constructing the preset detection model is as follows: Based on the requirements of pallet detection, a preliminary screening was conducted among YOLO series, SSD, Faster R-CNN and segmentation-based methods to determine the corresponding models for each method. Based on the selected model, build the corresponding deep learning framework and input the training dataset required for tray detection; During training, the model's performance metrics are monitored and parameters are adjusted. After training is completed, the model is evaluated and optimized to obtain the detection model.

7. The pallet detection and positioning method for an unmanned forklift according to claim 1, characterized in that, The step of performing the fork task based on the offset position includes: If the y and z deviations from the edge of the fork tines to the center of the pallet do not exceed the threshold, then the fork picking task is executed; Conversely, the vehicle's posture needs to be adjusted again, and the steps of acquiring the original LiDAR point cloud data of the pallet need to be repeated until the pallet picking task is executed, or the pallet picking task is exited if the number of adjustments exceeds the preset number.

8. A pallet detection and positioning system for unmanned forklifts, characterized in that, include: The preprocessing unit is configured to acquire the original LiDAR point cloud data of the tray, perform preprocessing, and project the preprocessed LiDAR point cloud data into an image. The detection unit is configured to process the image using a preset detection model to obtain the tray recognition result and the corresponding confidence level, and to obtain the valid tray based on the confidence level. The end-positioning unit is configured to perform digital image processing based on the lidar point cloud data corresponding to the effective pallet to obtain the position of the pallet center, and to perform plane fitting and normal vector calculation based on the lidar point cloud data of the effective pallet to obtain the yaw angle information of the pallet relative to the vehicle body. The execution unit is configured to calculate the position from the edge of the fork teeth to the center of the pallet when the forklift picks up the pallet based on the yaw angle information, and to perform the picking task based on the offset position.

9. An electronic device, characterized in that, include: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method as described in any one of claims 1 to 7.

Images