A pallet detection and positioning method and system for unmanned forklift applications

Abstract

The present invention discloses a pallet detection and positioning method and system for unmanned forklift applications, comprising: obtaining a depth image and an RGB image acquired by an RDG-D camera module; establishing a pallet image dataset to train a pallet detector to predict the pallet area and the support column area in the RGB image, and to eliminate incomplete pallets in the pallet area; taking a complete pallet as a target pallet and its support column area as a region of interest, aligning the RGB image and the depth image, extracting the depth information of the support column area, and converting the depth information of the region of interest into three-dimensional point cloud data in a camera coordinate system based on pre-calibrated camera parameters; segmenting the support column surface of the target pallet and calculating the centroid coordinates of each support column surface; extracting the support column triplet of the fork-picking surface of the target pallet, calculating the position and steering angle of the pallet in the camera coordinate system, and converting the pallet posture from the camera coordinate system to the forklift coordinate system, thereby realizing pallet positioning.

Description

Technical Field

[0001] The present invention belongs to the field of robot perception technology, and in particular relates to a pallet detection and positioning method and system for unmanned forklift applications. Background Art

[0002] Unmanned forklifts, also known as forklift AGVs or fork-mounted mobile robots, combine forklift and AGV technologies. Compared to conventional AGVs, unmanned forklifts not only handle material from point to point but also enable logistics transportation that connects multiple production processes. The application of unmanned forklifts can address the challenges of large logistics volumes and the high labor intensity of manual handling in industrial production and warehousing logistics operations, promoting the transformation and upgrading of industrial manufacturing and improving enterprise production efficiency and economic benefits.

[0003] In a dynamic, unstructured industrial environment, the position and posture of pallets are highly uncertain due to multiple factors such as operating procedures, equipment accuracy, and manual operation. How to efficiently and accurately detect and locate pallets has become an urgent problem that needs to be solved in the application of unmanned forklifts.

[0004] In related technologies, multi-sensor fusion is currently a major research direction. RGB images obtained by cameras contain rich texture and color information, while lidar or depth cameras can obtain high-precision depth information. By fusing multi-sensor information, pallet detection and pose estimation are more accurate.

[0005] Patent CN 112907666 A proposes an RGB-D-based pallet pose estimation method, system, and device. This method calculates the compressed grid size of sensor-captured images, performs template matching to determine the pallet's region of interest, and extracts the coordinates of the pallet bracket based on the pallet's region of interest. This method uses template matching to detect pallets. However, when the pallet is tilted at a certain angle, the match between the pallet in the image and the template decreases. Furthermore, occlusion by cargo can lead to inaccurate pixel classification, reducing pallet detection accuracy.

[0006] Patent CN 113409397 A proposes a warehouse pallet detection and positioning method based on an RGBD camera. This method uses a pre-trained YOLOv5 model for pallet detection, selects the pallet area in the RGB image, and calculates the pallet's position based on the depth and distance information of the pallet area. Patent CN 115272275 A proposes a pallet and obstacle detection and positioning system and method based on an RGB-D camera and a neural network model. This method uses a neural network to detect the pallet area in the RGB image and performs operations such as point cloud filtering, edge extraction, and calculation of the target spatial position information. Both methods use pallet edge features to locate the pallet. This can easily lead to pallet positioning failure or large positioning errors when the pallet is at a large angle or when the pallet edge is obstructed.

[0007] Patent CN 1 14972968A proposes a pallet recognition and pose estimation method based on multiple neural networks. It uses a UNet network to segment the target pallet in the image, and then uses an improved PointNet to process the point cloud of the extracted pallet area and output the pallet pose. This method uses a deep neural network to directly segment the pallet point cloud and estimate the pallet pose. The model trained using the generated pallet data cannot guarantee positioning accuracy in actual scenarios and requires a large amount of high-precision pallet point cloud annotation data for training. At the same time, the deep neural network for point cloud processing is computationally intensive and difficult to guarantee real-time performance.

[0008] In summary, most of the existing pallet detection and positioning methods based on multi-sensor fusion have high requirements for the placement of the pallet and cannot accurately handle situations where the pallet has a large deflection angle. Moreover, these methods do not consider problems such as cargo blocking the pallet and false detection in actual applications. The pallet posture estimation has the problems of low accuracy and poor robustness. Summary of the Invention

[0009] The purpose of the embodiments of the present application is to provide a pallet detection and positioning method and system for unmanned forklift applications, so as to solve the technical problems existing in the related art, such as the lack of safe handling of false detections and the low accuracy and poor robustness in the pallet pose estimation stage.

[0010] According to a first aspect of an embodiment of the present application, a pallet detection and positioning method for unmanned forklift applications is provided, comprising:

[0011] (1) Image acquisition: Acquire the depth image and RGB image captured by the RDG-D camera module;

[0012] (2) Pallet detection: A pallet image dataset is established to train a pallet detector. The trained pallet detector is used to predict the pallet area and support column area in the RGB image, and incomplete pallets in the pallet area are eliminated.

[0013] (3) Data fusion: The complete pallet obtained in step (2) is used as the target pallet, and its support column area is used as the region of interest. The RGB image and the depth image are aligned, and the depth information of the support column area is extracted. Based on the pre-calibrated camera parameters, the depth information of the region of interest is converted into three-dimensional point cloud data in the camera coordinate system;

[0014] (4) Surface segmentation: Perform point cloud filtering, plane segmentation, and geometric information extraction on the three-dimensional point cloud data of the support column area, segment the support column surface of the target pallet, and calculate the center of mass coordinates of each support column surface;

[0015] (5) Posture calculation: Extract the support column triplet of the target pallet fork picking surface, calculate the position and steering angle of the pallet in the camera coordinate system, and convert the pallet pose from the camera coordinate system to the forklift coordinate system to achieve pallet positioning.

[0016] Furthermore, step (2) includes:

[0017] (2.1) Obtain several pallet images and manually annotate the pallet area and support column area in each pallet image using rectangular frames. Perform image preprocessing on the annotated pallet images to establish a pallet image dataset.

[0018] (2.2) using the pallet image dataset to train a deep learning-based object detection network, thereby obtaining a trained pallet detector;

[0019] (2.3) Using the trained pallet detector, the RGB image is processed to obtain a plurality of pallet region rectangular frames and support column region rectangular frames;

[0020] (2.4) Based on the pallet area rectangular frame and the support column area rectangular frame, for each pallet detection result, the support column belonging to the pallet is determined according to the overlap of the rectangular frames, and the integrity of the pallet is verified, and all incomplete pallets are eliminated.

[0021] Furthermore, step (3) includes:

[0022] (3.1) Using the complete pallet obtained in step (2) as the target pallet and its support column area as the region of interest, align the RGB image and depth image captured by the RGB-D camera module, obtain the support column area in the depth image, and extract the pixel coordinates (u, v) and depth D of the support column area in the depth image;

[0023] (3.2) Based on the pre-calibrated camera parameters, the three-dimensional coordinates (X, Y, Z) of the pixel points in the support column area in the camera coordinate system are calculated using the following formula to generate three-dimensional point cloud data:

[0024]

[0025]

[0026] Z=D

[0027] where c x 、f x 、c y 、f y are pre-calibrated camera parameters.

[0028] Furthermore, step (4) includes:

[0029] (4.1) Preprocessing the three-dimensional point cloud data using straight-through filtering and voxel grid filtering;

[0030] (4.2) Segmenting the support column surfaces of the target pallet in the preprocessed point cloud data using a random sampling consistency method;

[0031] (4.3) For the segmented point cloud data of the support column surface, the following formula is used to calculate the centroid coordinates C (p x , p y , p z );

[0032]

[0033] Where n is the number of points on the support column surface, (x i ,y i , z i ) is the spatial coordinate of the point.

[0034] Furthermore, step (4.2) includes:

[0035] (4.2.1) Randomly extract three sample points from the preprocessed point cloud data, fit the plane equation, and estimate the four parameters a, b, c, and d of the plane equation aX+bY+cX+d=0 to obtain the undetermined plane;

[0036] (4.2.2) Calculating the distance between each point in the preprocessed point cloud data and the undetermined plane, and counting the number of points whose distance is less than the tolerance range d, i.e., the number of inliers;

[0037] (4.2.3) If the number of inliers in the current undetermined plane is greater than the threshold T, refit the plane using all the current inliers to obtain the point cloud data of the support column surface;

[0038] (4.2.4) If the current number of inliers is less than the threshold T, return to the above step (4.2.1).

[0039] Furthermore, step (5) includes:

[0040] (5.1) For the support pillar surface segmented from the point cloud data in step (3), all support pillar surface centroid triplets are checked according to the collinearity and symmetry of the pallet support pillar structure, and the centroid triplet of the target pallet is extracted;

[0041] (5.2) Use the target pallet’s center of mass triplet to calculate the pallet’s pose in the camera coordinate system, and use the pre-calibrated coordinate transformation to transform the pallet’s pose in the camera coordinate system to the unmanned forklift coordinate system.

[0042] Furthermore, in step (5.2), the center of mass triplet of the target pallet is used to calculate the position of the pallet in the camera coordinate system, including:

[0043] The center of mass coordinates of the middle support column of the pallet is used as the pallet position, and the center of mass coordinates of the support columns on both sides are used to calculate the rotation angle of the pallet in the camera coordinate system. Where Δz is the difference in z-coordinates between the centroids of the two support columns, and Δx is the difference in x-coordinates between the centroids of the two support columns.

[0044] According to a second aspect of an embodiment of the present application, a pallet detection and positioning system for unmanned forklift applications is provided, comprising:

[0045] RDG-D camera module, used to acquire depth images and RGB images;

[0046] a pallet detection module, configured to establish a pallet image dataset to train a pallet detector, use the trained pallet detector to predict the pallet region and support column region in the RGB image, and remove incomplete pallets in the pallet region;

[0047] A data fusion module is used to use the support column area obtained by the pallet detection module as the region of interest, align the RGB image and the depth image, extract the depth information of the support column area, and convert the depth information of the region of interest into three-dimensional point cloud data in the camera coordinate system based on pre-calibrated camera parameters;

[0048] The surface segmentation module is used to perform point cloud filtering, plane segmentation, and geometric information extraction on the three-dimensional point cloud data of the support column area, segment the support column surface of the target pallet, and calculate the center of mass coordinates of each support column surface;

[0049] The pose calculation module is used to extract the support column triplet of the target pallet fork picking surface, calculate the position and steering angle of the pallet in the camera coordinate system, and convert the pallet pose from the camera coordinate system to the forklift coordinate system to achieve pallet positioning.

[0050] According to a third aspect of the embodiments of the present application, there is provided an electronic device, including:

[0051] one or more processors;

[0052] a memory for storing one or more programs;

[0053] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in the first aspect.

[0054] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0055] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0056] It can be seen from the above embodiments that the present application fuses RGB images and depth images, and the deep learning-based pallet detector can quickly detect pallet information and determine the image area of ​​the pallet, greatly reducing the subsequent computing overhead. At the same time, the positioning method based on the pallet support column information can handle target pallets with larger angles, thereby improving the scope of use of the present invention. The present invention proposes for the first time a pallet detection method for simultaneously detecting two types of pallet information. While almost no increase in computing overhead is required, the detection results can be verified using the prior information of the pallet model to eliminate false detection information. The present invention uses the centroid of the pallet support column to calculate the pallet position for the first time, and uses the prior information of the pallet model to extract the support column centroid triplet of the target pallet, thereby improving the positioning range of the target pallet while ensuring the positioning accuracy.

[0057] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0059] Figure 1 This is a flow chart showing a pallet detection and positioning method for unmanned forklift applications according to an exemplary embodiment;

[0060] Figure 2 2 is a schematic diagram of an RGB-D camera installation and coordinate system according to an exemplary embodiment, wherein (a) is a side view and (b) is a top view;

[0061] Figure 3 is a schematic diagram showing a pallet detection result according to an exemplary embodiment;

[0062] Figure 4 is a schematic diagram of a pallet support column mass center triplet according to an exemplary embodiment;

[0063] Figure 5 is a schematic diagram of calculating the position and posture of a pallet according to an exemplary embodiment;

[0064] Figure 6 is a schematic diagram showing a framework of a pallet detection and positioning system for unmanned forklift applications according to an exemplary embodiment;

[0065] Figure 7 is a schematic diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0066] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this application.

[0067] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0068] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of this application. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0069] The core technology of this invention is to fuse different types of sensor data, use a deep learning-based object detector to quickly detect pallet information in RGB images, combine depth information for point cloud processing, and consider the prior information of the pallet model to achieve an efficient pallet detection and positioning system and method.

[0070] The present invention proposes a pallet detection and positioning method for unmanned forklift applications, such as Figure 1 As shown, the following steps are included:

[0071] (1) Image acquisition: Acquire the depth image and RGB image captured by the RDG-D camera module;

[0072] Specifically, the RGB-D camera module includes an RGB camera and a depth sensor. The RGB sensor is used to collect color and texture information of the environment, and the depth sensor is used to collect depth information of the environment. The camera's intrinsic parameter matrix is ​​determined in advance using Zhang Zhengyou's camera calibration method:

[0073]

[0074] Through hand-eye calibration, determine the homogeneous transformation matrix from the RGB-D camera coordinate system {C} to the unmanned forklift coordinate system {R}

[0075] In one embodiment, the RGB-D camera used is Intel RealSense D455, which includes an RGB sensor and an infrared ranging sensor. The camera's intrinsic parameters are automatically calibrated by a built-in algorithm at startup. The RGB-D camera is fixedly installed on the unmanned forklift, located in the middle of the fork arm, and faces the same direction as the fork arm. The coordinate system settings of the unmanned forklift platform and the sensor are shown in [1]. Figure 2 , including the unmanned forklift coordinate system {R}, the XOY plane is parallel to the ground, the X-axis coincides with the forward motion direction of the robot, and the Z-axis is perpendicular to the ground and upward; the camera coordinate system {C}, the Z-axis coincides with the optical axis and points to the front of the camera, the X-axis is parallel to the photosensitive plane to the right, and the Y-axis is perpendicular to the ground and downward; the pallet coordinate system {P}, the XOY plane is parallel to the ground, the origin of the coordinate system coincides with the center of the pallet fork picking surface, the X-axis is perpendicular to the pallet fork picking surface and forward, and the Z-axis is perpendicular to the ground and upward.

[0076] Homogeneous transformation matrix from RGB-D camera coordinate system {C} to unmanned forklift coordinate system {R} It is determined by hand-eye calibration method after the camera is installed.

[0077] (2) Pallet detection: Establish a pallet image dataset to train a pallet detector, use the trained pallet detector to predict the pallet area and support column area in the RGB image, and remove incomplete pallets in the pallet area; specifically, the following sub-steps are included:

[0078] (2.1) Obtain several pallet images and manually annotate the pallet area and support column area in each pallet image using rectangular frames. Perform image preprocessing on the annotated pallet images to establish a pallet image dataset.

[0079] Specifically, pallet images were collected using public image datasets and / or on-site sensors. Two types of pallet information were manually annotated in the images using rectangular boxes: the pallet and the pallet support column. The annotated images were then enhanced using image enhancement methods such as random mosaics, random grayscale, and random flipping to increase the diversity of the dataset.

[0080] (2.2) using the pallet image dataset to train a deep learning-based object detection network, thereby obtaining a trained pallet detector;

[0081] Specifically, the pallet image dataset is split into a training set and a test set. A deep learning-based object detection network is trained offline using a GPU platform, and the network parameters are saved to obtain a trained pallet detector. The object detection network can be Faster R-CNN, SSD, YOLO series, etc.

[0082] This example uses YOLOv5. The network structure of YOLOv5 is divided into four parts: Input, Backbone, Neck, and Head. To adapt to the pallet detection scenario, the output dimension of YOLOv5 is set to 3×(5+2), where 3 represents the three template boxes predicted for each grid, 5 represents the coordinates (x, y, w, h) and confidence c of each predicted box, and 2 represents the two information labels of the pallet dataset. The pre-trained weights provided by YOLOv5 on the COCO dataset are used as the initial weights. The YOLOv5-based object detection network is trained offline using the Nvidia GPU platform, and the network parameters are saved to obtain the trained pallet detector.

[0083] (2.3) Using the trained pallet detector, the RGB image is processed to obtain several pallet area rectangular frames and support column area rectangular frames, such as Figure 3 As shown;

[0084] Specifically, all rectangular frames are represented in the form of (x, y, w, h), where x and y represent the horizontal and vertical coordinates of the center of the rectangular frame, respectively, and w and h represent the width and length of the rectangular frame, respectively.

[0085] (2.4) Based on the pallet area rectangular frame and the support column area rectangular frame, for each pallet inspection result, determine the support column belonging to the pallet according to the overlap of the rectangular frames, verify the integrity of the pallet, and reject all incomplete pallets;

[0086] Specifically, based on the prior information of the pallet model, the pallet information detected in step (1.2) is verified. First, a pallet detection frame R is selected. i (x i ,y i , w i , h i ), for each support column detection frame R j (x j ,y j , w j , h j ), calculate the area S of the support column detection frame j The overlapping area S of the pallet detection frame and the support column detection frame ij :

[0087] Sj =w j ×h j

[0088] S ij =max(0,x2-x1)×max(0,y2-y1)

[0089] Where x1=max(x i -w i / 2,x j -w j / 2), y1=max(y i -h i / 2,y j -h j / 2) is the coordinate of the upper left corner of the overlapping area, x2=min(x i +w i / 2,x j +w j / 2), y2=min(y i +h i / 2,y j +h j / 2) is the lower right corner of the overlapping area.

[0090] Finally, calculate the overlapping area S ij The proportion k in the support column detection frame:

[0091]

[0092] Support pillars with k ≥ 0.5 are identified as belonging to the pallet. If the pallet has fewer than 3 support pillars, it is considered an incomplete pallet and is discarded from the detection results. Completeness checks are performed on all pallets to obtain complete pallet detection results. The final number of complete pallets may be 1 or 0. If the number of complete pallets is 1, the target pallet has been successfully detected and the next steps can be performed.

[0093] (3) Data fusion: The complete pallet obtained in step (2) is used as the target pallet, and its support column area is used as the region of interest. The RGB image and the depth image are aligned, and the depth information of the support column area is extracted. Based on the pre-calibrated camera parameters, the depth information of the region of interest is converted into three-dimensional point cloud data in the camera coordinate system. Specifically, the following sub-steps are included:

[0094] (3.1) Using the complete pallet obtained in step (2) as the target pallet and the support column rectangular frame as the region of interest, align the RGB image and depth image captured by the RGB-D camera module to obtain the support column region in the depth image, and extract the pixel coordinates (u, v) and depth D of the support column region in the depth image;

[0095] Specifically, assuming that the target pallet to be picked up is detected in the RGB image, its pallet support column detection frame is R k (k≥3), since the RGB image is aligned with the depth image, R is also selected in the depth image k The region is regarded as the region of interest, and each pixel point in the depth image region can be represented by pixel coordinates (u, v) and depth D.

[0096] (3.2) Based on the pre-calibrated camera parameters, the three-dimensional coordinates (X, Y, Z) of the pixel points in the support column area in the camera coordinate system {C} are calculated using the following formula to generate three-dimensional point cloud data:

[0097]

[0098]

[0099] Z=D

[0100] Among them, the camera's intrinsic parameter matrix is ​​determined in advance by Zhang Zhengyou's camera calibration method:

[0101]

[0102] (4) Performing point cloud filtering, plane segmentation, and geometric information extraction on the three-dimensional point cloud data of the support column area, segmenting the support column surface of the target pallet, and calculating the center of mass coordinates of each support column surface; specifically, the following sub-steps are included:

[0103] (4.1) Preprocessing the three-dimensional point cloud data using straight-through filtering and voxel grid filtering;

[0104] Specifically, the through-filter determines the position range (x min , x max ,y min ,y max , z min , z max ), retain the points within the effective detection range, where z min Set according to the fork arm length of the unmanned forklift, z max According to the maximum effective detection depth setting of the RGB-D camera, y min and y max According to the camera installation height and pallet placement height settings, eliminate the ground and too high data, x min and x max It is set according to the camera viewing angle and detection distance, and is reduced to a certain extent to ensure detection accuracy.

[0105] Voxel grid filtering is used to downsample the point cloud data within the effective detection range to remove noise and outliers, so that the point cloud has the same density at different distances.

[0106] (4.2) Using the Random Sample Consensus (RANSAC) method, segment the support pillar surfaces of the target pallet in the preprocessed point cloud data;

[0107] The surface of the pallet support column is represented by a plane equation aX+bY+cX+d=0. The steps for using RANSAC to segment the plane are as follows:

[0108] (4.2.1) Since three points determine a plane, randomly select three sample points P from the point cloud data. i (x i ,y i , z i ), i = 1, 2, 3, substitute into the plane equation, calculate the four parameters a, b, c, d of the plane equation, and obtain the plane to be determined;

[0109] (4.2.2) Calculate the distance from other points in the preprocessed point cloud data to the undetermined plane in sequence. If the distance is less than the threshold d, the point is considered to belong to the undetermined plane, that is, an inlier, and the number of inliers is counted;

[0110] (4.2.3) If the number of inliers in the current undetermined plane is greater than the threshold T, refit the plane using all the current inliers to obtain the point cloud data of the support column surface;

[0111] (4.2.4) If the current number of inliers is less than the threshold T, repeat the above steps.

[0112] The above steps (4.2.1) to (4.2.4) are standard steps of the RANSAC method and will not be repeated here.

[0113] (4.3) For the segmented point cloud data of the support column surface, the following formula is used to calculate the centroid coordinates C (p x , p y , p z );

[0114]

[0115] Where n is the number of points on the support column surface, (x i ,y i , z i ) is the spatial coordinate of the point.

[0116] In one embodiment, the point cloud processing in step (4) is implemented through the PCL point cloud library.

[0117] (5) Posture calculation: Extract the support column triplet of the target pallet fork pick-up surface (i.e., the surface where the forklift enters), calculate the position p and steering angle θ of the pallet in the camera coordinate system {C}, and transform the pallet pose from the camera coordinate system {C} to the forklift coordinate system {R} to achieve pallet positioning; specifically, it includes the following sub-steps:

[0118] (5.1) For the support pillar surface segmented from the point cloud data in step (3), all support pillar surface centroid triplets are checked according to the collinearity and symmetry of the pallet support pillar structure, and the centroid triplet of the target pallet is extracted;

[0119] Specifically, randomly select a centroid triple (C1, C2, C3) and calculate two random line segments and The slopes k1 and k2, if Determine the collinearity of the centroid triples; sort them according to the position where the centroid appears. If the line segment Determine the symmetry of the centroid triplet. If the centroid triplet is collinear and symmetrical, it is the centroid triplet of the front of the support column and is used as the centroid triplet of the target pallet. Figure 4 This step can eliminate falsely detected support columns or support columns that do not belong to the pallet fork pick-up surface, ensuring the accuracy of the subsequent pallet posture calculation based on the support column center of mass.

[0120] (5.2) Use the target pallet’s center of mass triplet (C1, C2, C3) to calculate the pallet’s pose in the camera coordinate system, and use the pre-calibrated coordinate transformation to transform the pallet’s pose in the camera coordinate system to the unmanned forklift coordinate system;

[0121] Specifically, considering that the pallet to be picked up is generally placed on the ground or parallel to the ground, the pallet state estimation is represented by spatial coordinates (x, y, z) and rotation angle θ. The center of mass coordinates C2 (p x , p y , p z ) represents the position of the pallet, represents the center of the pallet fork, and uses the center of mass of the support columns on both sides to calculate the rotation angle of the pallet in the camera coordinate system Where Δz is the difference in z coordinates between the centroids of the two support columns, and Δx is the difference in x coordinates between the centroids of the two support columns. Figure 5 A schematic diagram of the pallet pose calculation is given. Finally, the homogeneous transformation matrix from the camera coordinate system {C} to the unmanned forklift coordinate system {R} is used. Complete the conversion of the pallet's posture and convert it into the unmanned coordinate system, which can be used for subsequent forking tasks.

[0122] This embodiment improves the processing speed of pallet detection and positioning and the detectable range of the target pallet by integrating pallet detection based on deep learning and positioning methods based on pallet support column information. At the same time, this embodiment takes into account the prior information of the pallet model, eliminates false detection information, and ensures the accuracy and robustness of pallet detection and positioning.

[0123] This application also provides a pallet detection and positioning system for unmanned forklift applications, see Figure 6 , the system may include:

[0124] RDG-D camera module, used to acquire depth images and RGB images;

[0125] a pallet detection module, configured to establish a pallet image dataset to train a pallet detector, use the trained pallet detector to predict the pallet region and support column region in the RGB image, and remove incomplete pallets in the pallet region;

[0126] A data fusion module is used to use the support column area obtained by the pallet detection module as the region of interest, align the RGB image and the depth image, extract the depth information of the support column area, and convert the depth information of the region of interest into three-dimensional point cloud data in the camera coordinate system based on pre-calibrated camera parameters;

[0127] The surface segmentation module is used to perform point cloud filtering, plane segmentation, and geometric information extraction on the three-dimensional point cloud data of the support column area, segment the support column surface of the target pallet, and calculate the center of mass coordinates of each support column surface;

[0128] The pose calculation module is used to extract the support column triplet of the target pallet fork picking surface, calculate the position and steering angle of the pallet in the camera coordinate system, and convert the pallet pose from the camera coordinate system to the forklift coordinate system to achieve pallet positioning.

[0129] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0130] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The system embodiment described above is only illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this application. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0131] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned pallet detection and positioning method for unmanned forklift applications. Figure 7 As shown in FIG, a hardware structure diagram of a pallet detection and positioning method for unmanned forklift applications provided by an embodiment of the present invention is a device with data processing capabilities, except Figure 7 In addition to the processor, memory, and network interface shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0132] Accordingly, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the pallet detection and positioning method for unmanned forklift applications as described above. The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities as described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0133] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed in this application.

[0134] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A pallet detection and positioning method for unmanned forklift applications, characterized in that: include: (1) Image acquisition: Acquire the depth image and RGB image captured by the RDG-D camera module; (2) Pallet detection: Establish a pallet image dataset to train a pallet detector, use the trained pallet detector to predict the pallet area and support column area in the RGB image, and remove incomplete pallets in the pallet area; (3) Data fusion: The complete pallet obtained in step (2) is used as the target pallet, and its support column area is used as the region of interest. The RGB image and the depth image are aligned, and the depth information of the support column area is extracted. Based on the pre-calibrated camera parameters, the depth information of the region of interest is converted into three-dimensional point cloud data in the camera coordinate system; (4) Surface segmentation: Perform point cloud filtering, plane segmentation, and geometric information extraction on the three-dimensional point cloud data of the support column area, segment the support column surface of the target pallet, and calculate the center of mass coordinates of each support column surface; (5) Posture calculation: Extract the support column triplet of the target pallet fork pick-up surface, calculate the position and steering angle of the pallet in the camera coordinate system, and convert the pallet pose from the camera coordinate system to the forklift coordinate system to achieve pallet positioning; Wherein, step (2) includes: (2.1) Obtain several pallet images and manually annotate the pallet area and support column area in each pallet image using rectangular boxes. Perform image preprocessing on the annotated pallet images to establish a pallet image dataset. (2.2) using the pallet image dataset to train a deep learning-based object detection network, thereby obtaining a trained pallet detector; (2.3) Processing the RGB image using the trained pallet detector to obtain a number of pallet area rectangular frames and support column area rectangular frames; (2.4) Based on the pallet area rectangular frame and the support column area rectangular frame, for each pallet inspection result, determine the support column belonging to the pallet according to the overlap of the rectangular frames, verify the integrity of the pallet, and reject all incomplete pallets; Step (3) includes: (3.1) Take the complete pallet obtained in step (2) as the target pallet and its support column area as the region of interest, align the RGB image and depth image captured by the RGB-D camera module, obtain the support column area in the depth image, and extract the pixel coordinates of the support column area in the depth image. and depth ; (3.2) Based on the pre-calibrated camera parameters, the three-dimensional coordinates of the pixel points in the support column area in the camera coordinate system are calculated using the following formula: , generate 3D point cloud data: ， in 、 、 、 are pre-calibrated camera parameters; Step (4) includes: (4.1) preprocessing the three-dimensional point cloud data using straight-through filtering and voxel grid filtering; (4.2) Use the RANSAC method to segment the surface of each support column of the target pallet in the preprocessed point cloud data; (4.3) For the segmented point cloud data of the support column surface, the centroid coordinates of the point cloud of the support column surface in the camera coordinate system are calculated using the following formula: ; ， Where n is the number of points on the support column surface, is the spatial coordinate of the point.

2. The method according to claim 1, characterized in that Step (4.2) includes: (4.2.1) Randomly extract 3 sample points from the preprocessed point cloud data, fit the plane equation, and estimate the plane equation The four parameters 、 、 、 , get the undetermined plane; (4.2.2) Calculate the distance between each point in the pre-processed point cloud data and the undetermined plane, and count the distances that are less than the tolerance range. The points are the number of interior points; (4.2.3) If the number of inner points of the current undetermined plane is greater than the threshold , then all the current internal points are used to refit the plane to obtain the point cloud data of the support column surface; (4.2.4) If the current number of inliers is less than the threshold , then return to the above step (4.2.1).

3. The method according to claim 1, characterized in that Step (5) includes: (5.1) For the support column surface segmented from the point cloud data in step (3), check all support column surface centroid triplets based on the collinearity and symmetry of the pallet support column structure, and extract the centroid triplet of the target pallet; (5.2) Use the target pallet’s center of mass triplet to calculate the pallet’s pose in the camera coordinate system, and use the pre-calibrated coordinate transformation to transform the pallet’s pose in the camera coordinate system to the unmanned forklift coordinate system.

4. The method according to claim 3, characterized in that In step (5.2), the center of mass triplet of the target pallet is used to calculate the pallet's position in the camera coordinate system, including: The center of mass coordinates of the middle support column of the pallet is used as the pallet position, and the center of mass coordinates of the support columns on both sides are used to calculate the rotation angle of the pallet in the camera coordinate system. ,in is the difference in the z coordinates of the centroids of the two support columns, is the difference in x-coordinates between the centroids of the two support columns.

5. A pallet detection and positioning system for unmanned forklift applications, characterized in that: include: RDG-D camera module, used to acquire depth images and RGB images; a pallet detection module, configured to establish a pallet image dataset to train a pallet detector, use the trained pallet detector to predict the pallet region and support column region in the RGB image, and remove incomplete pallets in the pallet region; A data fusion module is used to use the support column area obtained by the pallet detection module as the region of interest, align the RGB image and the depth image, extract the depth information of the support column area, and convert the depth information of the region of interest into three-dimensional point cloud data in the camera coordinate system based on pre-calibrated camera parameters; The surface segmentation module is used to perform point cloud filtering, plane segmentation, and geometric information extraction on the three-dimensional point cloud data of the support column area, segment the support column surface of the target pallet, and calculate the center of mass coordinates of each support column surface; The pose calculation module is used to extract the support column triplet of the target pallet fork pick-up surface, calculate the position and steering angle of the pallet in the camera coordinate system, and convert the pallet pose from the camera coordinate system to the forklift coordinate system to achieve pallet positioning; The processing of the pallet detection module includes: (2.1) Obtain several pallet images and manually annotate the pallet area and support column area in each pallet image using rectangular boxes. Perform image preprocessing on the annotated pallet images to establish a pallet image dataset. (2.2) using the pallet image dataset to train a deep learning-based object detection network, thereby obtaining a trained pallet detector; (2.3) Processing the RGB image using the trained pallet detector to obtain a number of pallet area rectangular frames and support column area rectangular frames; (2.4) Based on the pallet area rectangular frame and the support column area rectangular frame, for each pallet inspection result, determine the support column belonging to the pallet according to the overlap of the rectangular frames, verify the integrity of the pallet, and reject all incomplete pallets; The processing of the data fusion module includes: (3.1) Take the complete pallet obtained in step (2) as the target pallet and its support column area as the region of interest, align the RGB image and depth image captured by the RGB-D camera module, obtain the support column area in the depth image, and extract the pixel coordinates of the support column area in the depth image. and depth ; (3.2) Based on the pre-calibrated camera parameters, the three-dimensional coordinates of the pixel points in the support column area in the camera coordinate system are calculated using the following formula: , generate 3D point cloud data: ， in 、 、 、 are pre-calibrated camera parameters; The processing flow of the surface segmentation module includes: (4.1) preprocessing the three-dimensional point cloud data using straight-through filtering and voxel grid filtering; (4.2) Use the RANSAC method to segment the surface of each support column of the target pallet in the preprocessed point cloud data; (4.3) For the segmented point cloud data of the support column surface, the centroid coordinates of the point cloud of the support column surface in the camera coordinate system are calculated using the following formula: ; ， Where n is the number of points on the support column surface, is the spatial coordinate of the point.

6. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images