RGB-D-based van carton stacking 3D point cloud identification and positioning method

Abstract

The invention discloses an RGB-D-based van carton stacking 3D point cloud recognition and positioning method, and belongs to the field of machine vision and automatic logistics. An existing visual detection algorithm is poor in recognition and positioning accuracy, complex in calculation and low in detection efficiency. The method comprises the following steps: acquiring RGB images of a target object under two visual angles and depth images of the target object under the two visual angles, performing denoising processing on the acquired images, combining the denoised images of the same visual angle to obtain a three-dimensional point cloud image under the corresponding visual angle, and fusing the three-dimensional point cloud images under the two visual angles to generate a complete three-dimensional point cloud image; the method comprises the steps of optimizing point cloud data in a point cloud image, inputting an optimized two-dimensional RGB point cloud image into a pre-trained YOLO target detection model, outputting a detection image, correcting center coordinates of sub-objects in the detection image, obtaining corrected two-dimensional center point coordinates of all the sub-objects, further obtaining three-dimensional coordinates, and finally obtaining a three-dimensional point cloud image. The method is used for identifying coordinate points of each sub-object.

Description

Technical Field

[0001] The invention belongs to the field of machine vision and automated logistics. Background Art

[0002] With the booming commodity economy and the rapid rise of internet e-commerce, the logistics industry is experiencing unprecedented development opportunities. The scale and complexity of freight transportation have significantly increased, and market demand is growing rapidly. However, traditional manual unloading methods suffer from significant deficiencies in efficiency, precision, and safety, making them difficult to meet the stringent requirements of modern logistics for efficient, high-quality, safe, and cost-effective operations. Therefore, computer vision-based intelligent recognition algorithms, due to their flexibility and efficiency, have become an important research direction for improving the adaptability of automated unloading technology.

[0003] Traditional visual detection algorithms mainly rely on hand-crafted feature extraction and geometric analysis methods, such as the combination of Histogram of Oriented Gradients (HOG) and Support Vector Machines (SVM), as well as template matching technology. Although these algorithms perform well in small-scale, standardized scenes, their detection performance degrades significantly when lighting conditions change, objects are occluded, or the background is complex, making them difficult to adapt to dynamic and complex environments. In recent years, deep learning-based visual detection algorithms have gradually emerged, which automatically learn target features through deep neural networks, significantly improving detection performance and applicability. However, these methods are highly dependent on large-scale, high-quality annotated data, and the model training and inference processes require high computing resources, which limits their application in practical engineering.

[0004] Furthermore, existing visual inspection algorithms can be categorized into two main categories, based on the input data type: RGB image-based and point cloud-based. RGB image-based algorithms can capture rich texture and color information, but lack depth information, making it difficult to accurately resolve the target's 3D spatial position. This is particularly true in scenes with severe occlusion or complex stacking. While point cloud-based algorithms can directly obtain the target's 3D geometric information, they lack texture and color information, resulting in limitations in target feature recognition and classification.

[0005] In summary, existing visual inspection algorithms have the following main shortcomings:

[0006] (1) Insufficient adaptability: Traditional algorithms are sensitive to lighting changes, background complexity, and target occlusion, while learning-based algorithms are highly dependent on large-scale labeled data and computing resources, which limits their practical engineering applications.

[0007] (2) Data limitations: RGB images lack depth information, making it difficult to accurately analyze three-dimensional spatial positions; point cloud data lacks texture information, which limits the ability to recognize target features.

[0008] (3) Lack of modularity and scalability: Existing algorithms are mostly customized for specific hardware platforms or task scenarios, lacking flexible structural design and standardized interfaces, making them difficult to transplant and expand in different environments.

[0009] Therefore, in summary, the existing visual detection algorithms have poor recognition and positioning accuracy and complex calculations, resulting in low detection efficiency. Summary of the Invention

[0010] The purpose of the present invention is to solve the problems of poor recognition and positioning accuracy and low detection efficiency caused by complex calculations in existing visual inspection algorithms; the present invention provides a 3D point cloud recognition and positioning method for box carton palletizing based on RGB-D.

[0011] A 3D point cloud recognition and positioning method for palletizing cartons based on RGB-D, comprising the following steps:

[0012] Step 1: Acquire RGB images of the target object at two viewing angles and depth images of the target object at the two viewing angles, wherein the two viewing angles refer to the viewing angles containing the top edge of the target object and the viewing angle containing the bottom edge of the target object, respectively;

[0013] Step 2: Image denoising: Denoise and normalize the depth image and RGB image of the target object under two viewing angles in turn to obtain the processed depth image and RGB image under two viewing angles;

[0014] Step 3: Point cloud generation: Based on the processed depth image and RGB image at the same viewing angle, a pixel-level 3D point cloud image at the corresponding viewing angle is obtained. The point cloud data on the two pixel-level 3D point cloud images are converted to the same horizontal reference plane. The converted point cloud data of the two viewing angles are fused to generate a complete 3D point cloud image.

[0015] Step 4: Point cloud optimization: After removing discrete points from the complete 3D point cloud image and visualizing it, the visualized 3D point cloud image is mapped to a 2D plane through orthogonal projection to generate an RGB point cloud image.

[0016] Step 5: Target detection: Input the RGB point cloud image into the pre-trained YOLO target detection model, and output the bounding box coordinates of each sub-object in the target object, the coordinates of the center point of each sub-object, and the confidence score of each sub-object. The coordinates of the center point of each sub-object are used as the detection center point coordinates.

[0017] Step 6: Position adjustment: Select the sub-object center coordinate corresponding to the maximum value from the confidence scores of all sub-objects as the reference benchmark value. Based on the reference benchmark value and the prior knowledge of the sizes of all sub-objects, obtain the theoretical center coordinates of all sub-objects. Use the theoretical center coordinates of all sub-objects to correct the detected center coordinates of all sub-objects to obtain the corrected two-dimensional center coordinates of all sub-objects.

[0018] Step 7: Based on the corrected 2D center point coordinates of all sub-objects, obtain a 3D point cloud of all sub-objects. The 3D point cloud includes the 3D coordinates, ID, and row and column numbers of each sub-object, thereby achieving 3D point cloud recognition and positioning.

[0019] Preferably, in step 2, the depth image and the RGB image of the target object under the two viewing angles are processed by a median filtering algorithm and a bilateral filtering algorithm in sequence for denoising.

[0020] Preferably, the median filtering algorithm is expressed as:

[0021] I1(i,j)=median({I(i,j)|(i,j)∈N(u,v)}) Formula 1,

[0022] Where I1(i,j) is the denoised pixel value obtained by the median filter algorithm, I(i,j) is the pixel value in the image, N(u,v) is the neighborhood value selected from each pixel value, and median represents the median value.

[0023] The bilateral filtering algorithm is expressed as:

[0024]

[0025] Where, I out (u, v) is the denoised pixel value obtained by the bilateral filtering algorithm, W(u, v) is the normalization factor, w s (i,j) is the spatial domain weight, w r (i,j) is the gray value range weight, W(u,v)=∑ (i,j)∈N(u,v) w s (i,j)·w r (i,j),σ s is the variance of the Gaussian kernel in the spatial domain, σ r is the variance of the grayscale Gaussian kernel.

[0026] Preferably, in step 2, the normalization process is: scaling each pixel value after denoising to the range of [0, 255]:

[0027]

[0028] Where, I norm (u,v) Each pixel value after normalization, I(u,v) is equal to I median (u,v) or I median (u,v),I min is the minimum depth value of all pixels in the current image, I max is the maximum depth value of all pixels in the current image.

[0029] Preferably, in step 3, the point cloud data on the two three-dimensional point cloud images are converted to the same horizontal reference plane, specifically:

[0030] The points in the point cloud data on the two 3D point cloud images are converted into points in the camera coordinate system using formulas 4 and 5:

[0031]

[0032] Where (u,v) is the pixel coordinate, (c x , c y ) is the principal point coordinate of the camera, (f x , f y ) is the focal length of the camera, point P in the camera coordinate system c for:

[0033] d is the depth value;

[0034] Convert the point in the camera coordinate system to the point in the world coordinate system using formula 6:

[0035] P base =T·P c Formula 6,

[0036] Where, P base is the point in the world coordinate system, T is the transformation matrix;

[0037] The point cloud data in the world coordinate system is depth-scaled, and the processed point cloud data is converted to point cloud data under the same horizontal reference plane.

[0038] Preferably, the point cloud data in the world coordinate system is subjected to depth value scaling processing:

[0039] P′ base [:3]=P base [:3]·depth_scale formula 7,

[0040] Where depth_scale is the scaling factor, P base [:3] is to get P base The first three components x, y and z, P'base [:3] is the scaled data point.

[0041] Preferably, in step 4, the complete three-dimensional point cloud image is processed to remove discrete points, specifically:

[0042] The complete 3D point cloud is divided into several cubic voxels, and the points in each voxel are replaced by a representative point:

[0043] P down =voxel_down_sample(p all ,voxel_size) Formula 8,

[0044] Where, P down is the point cloud data after voxel filtering, voxel_down_sample() is the voxel grid filtering function, p all For the point cloud data of the complete 3D point cloud, voxel size is the side length of the voxel grid;

[0045] Use statistical filtering to remove discrete points:

[0046] P flitered =remove_stasistical_outlier(p down ,nb neighbors ,std_ratio) formula 9,

[0047] Where, P flitered is the processed point, remove_stasistical_outlier() is the statistical filtering function, nb_neighbors is the number of neighborhood points, and std_ratio is the standard deviation multiple threshold.

[0048] Preferably, in step 6, the theoretical center coordinates of all sub-objects are obtained according to the reference benchmark value, and the detection center coordinates of all sub-objects are corrected using the theoretical center coordinates of all sub-objects, specifically:

[0049] Step 61: Cluster the theoretical center point coordinates of all sub-objects using the DBSCAN algorithm to obtain cluster labels for each column and row, and sort the cluster labels to obtain sorted cluster labels.

[0050] Step 62: Perform least square fitting on the position difference between the theoretical center point coordinates and the detection center point coordinates of each column and each row according to the sorted cluster labels to obtain a straight line equation, and correct the detection center point coordinates using the straight line equation.

[0051] Preferably, in step 7, a three-dimensional point cloud image of all sub-objects is obtained according to the two-dimensional true center point coordinates of all sub-objects, specifically:

[0052] According to the two-dimensional real center point coordinates of all sub-objects and the depth value of each center point coordinate in the depth image, the three-dimensional center point coordinates of each sub-object are calculated, and the three-dimensional point cloud images of all sub-objects are obtained based on the coordinates.

[0053] Preferably, the three-dimensional center point coordinates (X, Y, Z) are expressed as:

[0054]

[0055] Y=d·depth_scale Formula 12,

[0056] Where x_offset and z_offset are the offsets of the projected image in the x and z directions respectively, scale is the scaling factor between image pixels and three-dimensional space coordinates, depth_scale is the depth map unit conversion coefficient, and c x is the horizontal coordinate of the center pixel of the detection box of the sub-object in the projected image, c y is the central pixel ordinate of the detection box of the sub-object in the projected image, d is the original depth value of the corresponding pixel position in the depth map, and x_min is the minimum value in the x direction in the point cloud.

[0057] The beneficial effects brought by the present invention are:

[0058] This paper adopts multimodal data fusion technology to combine the texture information of RGB images with the spatial information of depth images, making full use of the advantages of both and improving the adaptability and adaptability of the YOLO target detection model to images.

[0059] The present invention performs downsampling, filtering and clustering of point clouds through image denoising, point cloud generation and point cloud optimization, removes noise and outliers, and eliminates the problem of poor accuracy of YOLO model recognition caused by perspective deviation.

[0060] The present invention uses the YOLO model to detect the bounding box and center point of the carton, and adjusts the center point position through the least squares method to adapt to complex stacking scenarios.

[0061] The present invention realizes task automation, that is, generates a visual task list, and supports the precise grasping of the automated unloading robot.

[0062] The present invention achieves efficient and accurate identification and positioning of carton stacks in complex stacking scenarios, significantly improves unloading efficiency, reduces labor costs, and provides a reliable intelligent solution for the modern logistics industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a flowchart of the RGB-D-based 3D point cloud recognition and positioning method for palletizing cartons;

[0064] Figure 2 This is a comparison chart of the center coordinates of the child object before and after position adjustment;

[0065] Figure 3 The output 3D point cloud image. DETAILED DESCRIPTION

[0066] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0067] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0068] Example 1:

[0069] This example, applied to an automated unloading robot system for cartons loaded on vans, proposes a method for identifying and locating carton stacks based on RGB and depth images (RGB-D). By fusing RGB images and depth information, combined with the YOLO object detection model and point cloud processing technology, this method achieves efficient and accurate recognition and positioning of complex stacks of cartons, providing reliable visual support for the automated unloading robot.

[0070] This embodiment uses a TOF (Time of Flight) depth camera to obtain RGB images and depth information of the carton stack. Through multimodal data fusion, point cloud generation and optimization, target detection and position adjustment, etc., accurate recognition and positioning of the carton stack is achieved.

[0071] A 3D point cloud recognition and positioning method for palletizing cartons based on RGB-D, comprising the following steps:

[0072] Step 1: Acquire RGB images of the target object at two viewing angles and depth images of the target object at the two viewing angles, wherein the two viewing angles refer to the viewing angles containing the top edge of the target object and the viewing angle containing the bottom edge of the target object, respectively;

[0073] Step 2: Image denoising: Denoise and normalize the depth image and RGB image of the target object under two viewing angles in turn to obtain the processed depth image and RGB image under two viewing angles;

[0074] Step 3: Point cloud generation: Based on the processed depth image and RGB image at the same viewing angle, a 3D point cloud image at the corresponding viewing angle is obtained. The point cloud data on the two 3D point cloud images are converted to the same horizontal reference plane. The converted point cloud data of the two viewing angles are fused to generate a complete 3D point cloud image.

[0075] Step 4: Point cloud optimization: After removing discrete points from the complete 3D point cloud image and visualizing it, the visualized 3D point cloud image is mapped to a 2D plane through orthogonal projection to generate an RGB point cloud image.

[0076] Step 5: Target detection: Input the RGB point cloud image into the pre-trained YOLO target detection model, and output the bounding box coordinates of each sub-object in the target object, the coordinates of the center point of each sub-object, and the confidence score of each sub-object. The coordinates of the center point of each sub-object are used as the detection center point coordinates.

[0077] Step 6: Position adjustment: Select the sub-object center coordinate corresponding to the maximum value from the confidence scores of all sub-objects as the reference benchmark value. Based on the reference benchmark value and the prior knowledge of the sizes of all sub-objects, obtain the theoretical center coordinates of all sub-objects. Use the theoretical center coordinates of all sub-objects to correct the detected center coordinates of all sub-objects to obtain the corrected two-dimensional center coordinates of all sub-objects.

[0078] Step 7: Based on the corrected 2D center point coordinates of all sub-objects, obtain a 3D point cloud of all sub-objects. The 3D point cloud includes the 3D coordinates, ID, and row and column numbers of each sub-object, thereby achieving 3D point cloud recognition and positioning.

[0079] Specifically, if the target object is a stack of cartons, and each child object is a carton, the RGB image and depth image of the stack captured by the TOF (Time of Flight) depth camera in step 1 are obtained from two perspectives: the high and low perspectives, respectively, including the top and bottom edges of the target object. The denoising operation in step 2 is intended to eliminate noise interference and improve data usability.

[0080] Since there is a perspective deviation in the image directly taken by the camera, this deviation will cause the geometric deformation and feature distortion of the target object, thereby affecting the recognition accuracy of the YOLO model. To solve this problem, this embodiment proposes a correction method based on multi-perspective image fusion. Specifically, by collecting two sets of images of the target object at high and low positions respectively, and using image registration and geometric transformation algorithms, the two sets of images are synthesized and spliced ​​into a vertical perspective view (same horizontal reference plane). This method effectively eliminates the geometric distortion caused by perspective deviation and significantly improves the feature extraction capability and recognition accuracy of the YOLO model for the target object.

[0081] In step 6, target detection is specifically as follows:

[0082] The 2D RGB point cloud image is used as input and the pre-trained YOLO (You Only Look Once) object detection model is loaded to detect cartons in the image in real time. The model outputs the coordinates (x1, y1, x2, y2) of the bounding box of each detected object and its center point (c x ,c y ) and provides a detection confidence score. The detection results include the precise bounding box location and center point coordinates of the carton, providing key information for subsequent 3D spatial positioning and task planning.

[0083] The input data is prepared as follows:

[0084] (1) 2D RGB point cloud image: generated by projecting the 3D point cloud onto a 2D plane, containing the texture and color information of the carton.

[0085] (2) YOLO model: A pre-trained object detection model that can efficiently identify object bounding boxes and categories in images.

[0086] The output of the YOLO model is:

[0087] (1) Bounding box: The bounding box of each detected carton, represented as (x1, y1, x2, y2).

[0088] (2) Detection center point: The coordinates of the center point of each bounding box (c x ,c y ).

[0089] (3) Confidence: The confidence score of the detection result, which indicates the confidence level of the model in the detection result.

[0090] Next, we display the projected image and use OpenCV to display the projected 2D RGB point cloud image. Then, we display the detection result image and draw the detected bounding box and center point on the 2D RGB point cloud image to generate the detection result image.

[0091] The following further defines the denoising method:

[0092] In step 2, the depth image and RGB image of the target object under two viewing angles are processed by the median filtering algorithm and the bilateral filtering algorithm respectively for denoising.

[0093] The median filter algorithm is expressed as:

[0094] I1(u,v)=median({I(i,j)|(i,j)∈N(u,v)}) Formula 1,

[0095] Where I1(u,v) is the denoised pixel value obtained by the median filter algorithm, I(i,j) is the pixel value in the image, N(u,v) is the neighborhood value selected from each pixel value, and median represents the median value.

[0096] The bilateral filtering algorithm is expressed as:

[0097]

[0098] Where, I out (U, v) is the denoised pixel value obtained by the bilateral filtering algorithm, W(u, v) is the normalization factor, w s (i,j) is the spatial domain weight, w r (i,j) is the gray value range weight, W(u,v)=∑ (i,j)∈N(u,v) w s (i,j)·w r (i,j),σ s is the variance of the Gaussian kernel in the spatial domain (used to control the degree to which the weight changes with spatial distance), σ r is the variance of the grayscale Gaussian kernel (used to control the degree to which the weight changes with grayscale differences).

[0099] Specifically, the median filter algorithm is explained as follows: for each pixel I(u,v) in the depth image, the median value of its neighborhood N(u,v) is taken as the filtered value. Median filtering is used to remove salt and pepper noise, while bilateral filtering is used to remove noise while preserving edges.

[0100] The normalization process is further defined below:

[0101] Scale each pixel value after denoising to the range [0,255]:

[0102]

[0103] Where, I norm (u,v) After normalization of each pixel value, I(u,v) is equal to I median (u,v) or I median (u,v),I min is the minimum depth value of all pixels in the current image, I max is the maximum depth value of all pixels in the current image.

[0104] It is further defined below that in step 3, the point cloud data on the two three-dimensional point cloud images are converted to point cloud data under the same horizontal reference plane, specifically:

[0105] The points in the point cloud data on the two 3D point cloud images are converted into points in the camera coordinate system using formulas 4 and 5:

[0106]

[0107] Where (u,v) is the pixel coordinate, (c x , c y ) is the principal point coordinate of the camera, (f x , f y ) is the focal length of the camera, point P in the camera coordinate system c for:

[0108] d is the depth value;

[0109] Convert the point in the camera coordinate system to the point in the world coordinate system using formula 6:

[0110] P base =T·P c Formula 6,

[0111] Where, P base is the point in the world coordinate system, T is the transformation matrix;

[0112] Depth value scaling: In the algorithm, point cloud coordinates need to be converted from millimeters to meters, so the P base The first three components (P base For one of the pixels (x, y, z, 1), 1 represents the grayscale value or intensity value of the pixel, and the first three components refer to the x, y, z in (x, y, z, 1) for scaling:

[0113] P' base [:3]=P base [:3]·depth_scale formula 7,

[0114] Where depth_scale is the scaling factor, usually 0.001 (millimeter to meter conversion), P base [:3] is to get P base The first three components x, y and z, P′ base [:3] is the scaled data point.

[0115] Specifically, T in Equation 6 is a transformation matrix that contains rotation and translation information. T is a 4×4 transformation matrix used to transform points from the camera coordinate system to the world coordinate system.

[0116] Through hand-eye calibration, the transformation matrices T1 and T2 of the camera at the high and low positions relative to the base coordinate system are obtained to facilitate point cloud conversion. Based on the registered depth map and RGB image obtained by the RGBD camera, 3D point clouds of the high and low position cardboard stacks relative to the camera are obtained through coordinate transformation. Subsequently, the high and low position point cloud data are uniformly converted to the same horizontal reference plane and fused to generate a complete 3D point cloud image.

[0117] Merging of point cloud data: Before merging, the high-position and low-position point cloud data have been transformed to the same horizontal reference plane through the transformation matrices T1 and T2. Specifically:

[0118] Point cloud data at high position: P high =T1·P gihh,camera

[0119] Point cloud data at low position: P low =T2·P low,camera

[0120] Among them, P high,camera and P low,camera They are the point cloud data of high position and low position in the camera coordinate system respectively.

[0121] Merged point cloud data P all Contains all point cloud data at high and low positions, and these point cloud data have been converted to the same horizontal reference plane. all Expressed as:

[0122]

[0123] Assumption: The point cloud data at high position is P high , the shape is (N1,3), where N1 is the number of high position point clouds. The low position point cloud data is P low , with shape (N2,3), where N2 is the number of low-level point clouds.

[0124] After synthesis all The shape is (N1+N2,3).

[0125] The merging of color information is the same as above.

[0126] The following further defines point cloud optimization: The complete 3D point cloud image undergoes voxel downsampling, statistical filtering, and cluster analysis to remove noise and outliers and optimize point cloud quality. The processed point cloud is then visualized using the Open3D library. Orthogonal projection is used to map the 3D point cloud onto a 2D plane, generating an RGB point cloud image. This provides high-quality input data for subsequent object detection and localization.

[0127] Voxel downsampling: The point cloud space is divided into a voxel grid, and the points within each voxel are replaced by a representative point:

[0128] P down =voxel_down_sample(p all ,voxel_size) Formula 8,

[0129] Where, P down is the point cloud data after voxel filtering, voxel_down_sample() is the voxel grid filtering function (used to split the point cloud into several cubic voxels and replace all the points of the voxel with the centroid or representative point in each voxel to achieve downsampling and homogenization), p all The point cloud data of the complete three-dimensional point cloud (representing the three-dimensional coordinates of all points), voxel size is the side length of the voxel grid (used to control the downsampling accuracy, the smaller the value, the more points are retained, and the larger the value, the stronger the simplification);

[0130] Statistical filtering: remove outliers and retain points with higher density:

[0131] P flitered =remove_stasistical_outlier(p down ,nb neighbors ,std_ratio) formula 9,

[0132] Where, P flitered is the processed point, remove_stasistical_outlier() is a statistical filtering function (used to remove outliers with abnormally large local neighborhood distances), nb_neighbors is the number of neighborhood points (used to calculate the average neighbor distance of each point), and std_ratio is the standard deviation multiple threshold (usually set to 1-2).

[0133] The following further defines the specific process of position adjustment:

[0134] Based on the known carton size parameters, the carton position with the highest detection confidence is selected as the reference benchmark to determine the coordinates of the center points of all cartons in the theoretical grid. Subsequently, the least squares method is used to fit the positional deviation between the theoretical center point and the detection center point of each column and row, and the straight line equations of the center points of each row and column are obtained respectively. By solving the intersection of the row and column straight line equations, the adjusted center point coordinates are calculated. This adjustment process effectively eliminates detection errors and ensures that the distribution of the carton center points in three-dimensional space conforms to the geometric constraints of the theoretical grid, providing high-precision positioning information for subsequent grasping and unloading tasks.

[0135] The specific process is as follows:

[0136] Point cloud clustering and sorting: First, use the DBSCAN algorithm to cluster the point cloud data of the RGB point cloud image to obtain cluster labels for each column and row. Sorting the cluster labels ensures that the cartons are arranged in row and column order.

[0137] Generate theoretical grid points: Select the carton position with the highest recognition confidence as the reference point. Generate theoretical grid points based on the reference point and carton size parameters (such as row spacing and column spacing).

[0138] Adjust the point cloud position: Perform a least squares fit on the position difference between the theoretical center point and the detection center point of each column and row to obtain the equation of a straight line. Use this equation to adjust the position of the detection center point to conform to the theoretical distribution.

[0139] Generate box objects: For each adjusted point cloud data, generate a corresponding box object, including the box ID, row and column numbers, 3D coordinates, width, height and other information.

[0140] Box grouping: To facilitate the crawling operation, the KMeans algorithm is used to group the boxes left and right, dividing the boxes into two groups: left and right.

[0141] The following further defines the specific process of step 7 as follows:

[0142] Based on the coordinates of the center points of the cartons in the two-dimensional projection image and the depth values ​​of the corresponding positions in the depth image, the coordinates of the center points of each carton in three-dimensional space are calculated through projection parameters and geometric transformation. Each carton center point is mapped from two-dimensional projection to three-dimensional space, and a three-dimensional point cloud is drawn using visualization tools (such as Matplotlib) to intuitively display the distribution of cartons in three-dimensional space.

[0143] The specific process is as follows:

[0144] Read the depth value: According to the two-dimensional center point (c x ,c y ), read the corresponding depth value d from the depth image.

[0145] Calculate 3D coordinates: Use the projection parameters and depth value to calculate the 3D center point coordinates (X, Y, Z) of each carton.

[0146]

[0147] Y=d·depth scale Formula 12,

[0148] in:

[0149] x_offset and z_offset are the offsets of the projected image in the x and z directions respectively, scale is the scaling factor between image pixels and three-dimensional space coordinates, depth_scale is the depth map unit conversion coefficient (usually 0.001, converting millimeters to meters), c x is the horizontal coordinate of the center pixel of the detection box of the sub-object in the projected image, c y is the central pixel ordinate of the detection box of the sub-object in the projected image, d is the original depth value of the corresponding pixel position in the depth map, and x_min is the minimum value in the x direction in the point cloud.

[0150] Draw a 3D point cloud: Use the Matplotlib library to draw the 3D center point in 3D space.

[0151] Display task information: Display detailed information of each carton in the visual interface, such as ID, row and column numbers, 3D coordinates, etc.

[0152] By reading the distance information from the depth image, the center points of the cartons projected on the 2D projection are mapped into 3D space, generating the 3D coordinates of each carton's center point. Finally, a visualization tool displays the distribution of the cartons, providing intuitive task support for the automated unloading robot. This step achieves precise conversion from 2D detection results to 3D spatial positioning, laying a solid foundation for subsequent grasping and unloading tasks.

[0153] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be employed in conjunction with other described embodiments.

Claims

1. The RGB-D-based 3D point cloud recognition and positioning method for carton palletizing in vans is characterized by: The method includes the following: Step 1: Acquire RGB images of the target object at two viewing angles and depth images of the target object at the two viewing angles, wherein the two viewing angles refer to the viewing angles containing the top edge of the target object and the viewing angle containing the bottom edge of the target object, respectively; Step 2: Image denoising: Denoise and normalize the depth image and RGB image of the target object under two viewing angles in turn to obtain the processed depth image and RGB image under two viewing angles; Step 3: Point cloud generation: Based on the processed depth image and RGB image at the same viewing angle, a pixel-level 3D point cloud image at the corresponding viewing angle is obtained. The point cloud data on the two pixel-level 3D point cloud images are converted to the same horizontal reference plane. The converted point cloud data of the two viewing angles are fused to generate a complete 3D point cloud image. Step 4: Point cloud optimization: After removing discrete points from the complete 3D point cloud image and visualizing it, the visualized 3D point cloud image is mapped to a 2D plane through orthogonal projection to generate an RGB point cloud image. Step 5: Target detection: Input the RGB point cloud image into the pre-trained YOLO target detection model, and output the bounding box coordinates of each sub-object in the target object, the coordinates of the center point of each sub-object, and the confidence score of each sub-object. The coordinates of the center point of each sub-object are used as the detection center point coordinates. Step 6: Position adjustment: Select the sub-object center coordinate corresponding to the maximum value from the confidence scores of all sub-objects as the reference benchmark value. Based on the reference benchmark value and the prior knowledge of the sizes of all sub-objects, obtain the theoretical center coordinates of all sub-objects. Use the theoretical center coordinates of all sub-objects to correct the detected center coordinates of all sub-objects to obtain the corrected two-dimensional center coordinates of all sub-objects. Step 7: Based on the corrected 2D center point coordinates of all sub-objects, obtain a 3D point cloud of all sub-objects. The 3D point cloud includes the 3D coordinates, ID, and row and column numbers of each sub-object, thereby achieving 3D point cloud recognition and positioning.

2. The RGB-D based 3D point cloud recognition and positioning method for palletizing cartons according to claim 1 is characterized in that: In step 2, the depth image and RGB image of the target object under two viewing angles are processed by the median filtering algorithm and the bilateral filtering algorithm for denoising.

3. The RGB-D based 3D point cloud recognition and positioning method for palletizing cartons according to claim 2 is characterized in that: The median filter algorithm is expressed as: I1(i,j)=median({I(i,j)|(i,j)∈N(u,v)}) Formula 1, Where I1(i,j) is the denoised pixel value obtained by the median filter algorithm, I(i,j) is the pixel value in the image, N(u,v) is the neighborhood value selected from each pixel value, and median represents the median value. The bilateral filtering algorithm is expressed as: Where, I out (u, v) is the denoised pixel value obtained by the bilateral filtering algorithm, W(u, v) is the normalization factor, w s (i,j) is the spatial domain weight, w r (i,j) is the gray value range weight, W(u,v)=∑ (i,j)∈N(u,v) w s (i,j)·w r (i,j),σ s is the variance of the Gaussian kernel in the spatial domain, σ r is the variance of the grayscale Gaussian kernel.

4. The RGB-D-based 3D point cloud recognition and positioning method for palletizing cartons according to claim 3 is characterized in that: In step 2, the normalization process is to scale each pixel value after denoising to the range of [0, 255]: Where, I norm (u,v) After normalization of each pixel value, I(u,v) is equal to I nedian (u,v) or I nedian (u,v),I min is the minimum depth value of all pixels in the current image, I max is the maximum depth value of all pixels in the current image.

5. The RGB-D based 3D point cloud recognition and positioning method for carton palletizing in a van according to claim 1 or 4, characterized in that: In step 3, the point cloud data on the two 3D point cloud images are converted to the same horizontal reference plane, specifically: The points in the point cloud data on the two 3D point cloud images are converted into points in the camera coordinate system using formulas 4 and 5: Where (u,v) is the pixel coordinate, (c x , c y ) is the principal point coordinate of the camera, (f x , f y ) is the focal length of the camera, point P in the camera coordinate system c for: d is the depth value; Convert the point in the camera coordinate system to the point in the world coordinate system using formula 6: P base =T·P c Formula 6, Where, P base is the point in the world coordinate system, T is the transformation matrix; The point cloud data in the world coordinate system is depth-scaled, and the processed point cloud data is converted to point cloud data under the same horizontal reference plane.

6. The RGB-D-based 3D point cloud recognition and positioning method for palletizing cartons according to claim 5 is characterized in that: Scale the depth value of the point cloud data in the world coordinate system: P' base [:3]=P base [:3]·depth_scale formula 7, Where depth_scale is the scaling factor, P base [:3] is to get P base The first three components x, y and z, P' base [:3] is the scaled data point.

7. The RGB-D based 3D point cloud recognition and positioning method for palletizing cartons according to claim 1 or 6, characterized in that: In step 4, the complete 3D point cloud image is processed to remove discrete points, specifically: The complete 3D point cloud is divided into several cubic voxels, and each cubic voxel is replaced by a representative point: P down =voxel_down_sample(p all ,voxel_size) Formula 8, Where, P down is the point cloud data after voxel filtering, voxel_down_sample() is the voxel grid filtering function, p all For the point cloud data of the complete 3D point cloud, voxel size is the side length of the voxel grid; Use statistical filtering to remove discrete points: P flitered = remove_stasisical_outlier(p down ,nb neighbors ,std_ratio) Formula 9, Where, P flitered is the processed point, remove_stasistical_outlier() is the statistical filtering function, nb_neighbors is the number of neighborhood points, and std_ratio is the standard deviation multiple threshold.

8. The RGB-D based 3D point cloud recognition and positioning method for palletizing cartons according to claim 1 is characterized in that: In step 6, the coordinates of the detection center points of all sub-objects are corrected using the theoretical center point coordinates of all sub-objects, specifically: Step 61: Cluster the theoretical center point coordinates of all sub-objects using the DBSCAN algorithm to obtain cluster labels for each column and row, and sort the cluster labels to obtain sorted cluster labels. Step 62: Perform least square fitting on the position difference between the theoretical center point coordinates and the detection center point coordinates of each column and each row according to the sorted cluster labels to obtain a straight line equation, and correct the detection center point coordinates using the straight line equation.

9. The RGB-D based 3D point cloud recognition and positioning method for palletizing cartons according to claim 1 is characterized in that: In step 7, the 3D point cloud images of all sub-objects are obtained based on the 2D true center point coordinates of all sub-objects, specifically: According to the two-dimensional real center point coordinates of all sub-objects and the depth value of each center point coordinate in the depth image, the three-dimensional center point coordinates of each sub-object are calculated, and the three-dimensional point cloud images of all sub-objects are obtained based on the coordinates.

10. The RGB-D based 3D point cloud recognition and positioning method for palletizing cartons according to claim 9 is characterized in that: The three-dimensional center point coordinates (X, Y, Z) are expressed as: Y=d·depth_scale Formula 12, Where x_offset and z_offset are the offsets of the projected image in the x and z directions respectively, scale is the scaling factor between image pixels and three-dimensional space coordinates, depth_scale is the depth map unit conversion coefficient, and c x is the central pixel horizontal coordinate of the detection box of the sub-object in the projected image, c y is the central pixel ordinate of the detection box of the sub-object in the projected image, d is the original depth value of the corresponding pixel position in the depth map, and x_min is the minimum value in the x direction in the point cloud.

Images