Warehouse carton detection method and system based on instance segmentation and RGBD point cloud fusion

Abstract

The application provides a warehouse carton detection method and system based on instance segmentation and RGBD point cloud fusion, which takes a single frame of RGB image and a corresponding depth map as input, generates a box and layer structure instance mask through a YOLO instance segmentation model, and generates a color point cloud in combination with depth information back projection; through point cloud denoising and multi-plane RANSAC fitting, coordinate correction is realized, an extreme value smoothing strategy is used to calculate the volume of the overall stack and the single box body bounding box, and the stack layer number, the number of top boxes and the single layer box number are estimated through the volume ratio. The application generates a pixel-level mask through instance segmentation, realizes the deep fusion of two-dimensional instance information and three-dimensional geometric information by combining the depth information of the RGBD point cloud, overcomes the defects that the traditional two-dimensional visual detection cannot process depth occlusion and box overlap, accurately distinguishes the carton target and the background and invalid area, avoids the false detection and missed detection caused by background interference, and further improves the detection accuracy compared with the prior art.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, point cloud processing and intelligent warehousing technology, and in particular to a warehouse carton detection method and system based on instance segmentation and RGBD point cloud fusion. Background Technology

[0002] In warehousing operations, cardboard boxes are often stacked in multiple layers on pallets or shelves. Quickly calculating the number of stack layers, the number of boxes on top, and the number of boxes per layer is crucial for inbound inspection, outbound verification, inventory counting, warehouse utilization assessment, and the development of automated replenishment strategies.

[0003] Traditional manual inventory methods are significantly affected by factors such as ambient lighting, obstructed views, work pace, and personnel experience, resulting in problems such as low efficiency, large errors, and difficulty in standardization.

[0004] Existing inventory counting technologies mainly include methods based on 2D detection / counting, methods based on depth thresholding or point cloud clustering, and methods based on 3D reconstruction and fine modeling. Among these, 2D detection / counting methods count box instances using RGB image target detection; while simple to implement, these methods are prone to missed or false detections when shelves are obstructed, top boxes have similar appearances, or box boundaries are blurred. Furthermore, 2D counting struggles to reflect the 3D stacking structure, making it difficult to infer the number of boxes per layer or the number of layers. Methods based on depth thresholding or point cloud clustering use depth sensors to directly segment and cluster point clouds to estimate the number or volume of voxels. However, in warehouse environments, depth data noise, reflective materials, and edge flypoints can cause point cloud fragmentation; simultaneously, without prior instance information, clustering is susceptible to interference from background shelves, stretch film, and pallet structures, resulting in insufficient robustness. Methods based on 3D reconstruction and fine modeling obtain high-quality point clouds through multi-view reconstruction or SLAM followed by geometric analysis. While offering high accuracy, these methods are costly in terms of equipment and computation, and involve complex processes, making them unsuitable for deployment in high-rate-of-time inventory counting scenarios with single frames / single views.

[0005] Therefore, there is an urgent need to provide a warehouse carton detection method and system based on instance segmentation and RGBD point cloud fusion. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a warehouse carton detection method and system based on instance segmentation and RGBD point cloud fusion. This invention integrates two-dimensional instance segmentation and three-dimensional point cloud geometric calculation to solve counting errors caused by occlusion, noise, and posture changes, and achieves automated, high-precision, and traceable detection of the number of stacked layers and the number of boxes per layer in warehouses.

[0007] In a first aspect, the technical solution provided by this invention is: a warehouse cardboard box detection method based on instance segmentation and RGBD point cloud fusion, comprising the following steps:

[0008] S1) Obtain the RGB image of the location to be tested and the corresponding depth image aligned to the same resolution;

[0009] S2) The YOLO instance segmentation model is used to perform instance segmentation and detection on the RGB image of the storage location to be tested, obtain the instance prediction results, and generate instance masks. After confidence filtering and non-maximum suppression, an effective set of instance masks is obtained.

[0010] S3) Generate an instance color mask map and overlay visualization based on the valid instance mask set; count the number of instances in the layer structure. Number of top box instances ;

[0011] S4) Perform pixel-level fusion of the depth image and the instance color mask image, and back-project the effective pixels into 3D points based on the pinhole camera model to generate a colored point cloud;

[0012] S5) Perform radius outlier removal on the colored point cloud objects to obtain a clean point cloud set;

[0013] S6) Perform multiplane RANSAC fitting on the clean point cloud set to obtain at least two sets of approximately orthogonal principal planes;

[0014] S7) Construct an orthogonal basis based on the normal vectors of the two sets of principal planes and correct the point cloud coordinates to obtain a point cloud with coordinate system aligned.

[0015] S8) Estimate the 3D bounding box of the point cloud set aligned with the coordinate system, use the extreme value smoothing strategy to calculate the volume of the overall stacked 3D bounding box, and calculate the volume of a single box on the top box point set to obtain the estimated number of boxes in a single layer.

[0016] Secondly, the technical solution of the present invention is: a warehouse cardboard box detection method based on instance segmentation and RGBD point cloud fusion, comprising:

[0017] The data acquisition module is used to acquire the RGB image of the location to be tested and the corresponding depth image aligned to the same resolution;

[0018] The instance segmentation and mask generation module uses the YOLO instance segmentation model to perform instance segmentation and detection on the RGB image of the test location, obtains the instance prediction results, and generates instance masks. After confidence filtering and non-maximum suppression, an effective set of instance masks is obtained.

[0019] The instance statistics module is used to generate instance color mask images and overlay visualizations based on the set of valid instance masks; it also counts the number of instances in the layer structure. Number of top box instances ;

[0020] The point cloud reconstruction module is used to perform pixel-level fusion of depth images and instance color mask images, and back-project effective pixels into three-dimensional points based on the pinhole camera model to generate colored point clouds.

[0021] The point cloud filtering module is used to remove outliers by radius from colored point cloud objects to obtain a clean point cloud set.

[0022] The fitting module is used to perform multiplane RANSAC fitting on a clean point cloud set to obtain at least two sets of approximately orthogonal principal planes.

[0023] The point cloud alignment module is used to construct an orthogonal basis based on the normal vectors of the two sets of principal planes and correct the point cloud coordinates to obtain a point cloud with coordinate system alignment.

[0024] The single-layer box number estimation module is used to estimate the 3D bounding box of a point cloud set aligned with the coordinate system. It uses an extreme value smoothing strategy to calculate the overall stacked 3D bounding box volume and calculates the volume of a single box on the top box point set to obtain the estimated value of the single-layer box number.

[0025] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the warehouse carton detection method based on instance segmentation and RGBD point cloud fusion.

[0026] The beneficial effects of this invention are as follows:

[0027] 1. This invention generates pixel-level masks through instance segmentation. By combining the depth information of RGBD (RGB image and depth image) point cloud, it achieves deep fusion of two-dimensional instance information and three-dimensional geometric information, overcoming the shortcomings of traditional two-dimensional visual inspection that cannot handle depth occlusion and box overlap. It accurately distinguishes cardboard box targets from background and invalid areas, avoiding false detection and missed detection caused by background interference. The detection accuracy is further improved compared with the existing technology.

[0028] 2. This invention assigns differentiated colors to different box instances and different layer structures by using instance color mask maps, which can clearly distinguish each carton instance and stacking layer at the image level. It solves the problems of boundary blurring and instance confusion caused by box overlap and occlusion in traditional two-dimensional images, provides accurate instance constraints for three-dimensional point cloud back projection, and significantly reduces the impact of background and interference areas on detection results.

[0029] 3. This invention can effectively remove isolated noise points in point clouds caused by depth noise, environmental reflection, and camera measurement errors by using radius outliers; combined with the high outlier tolerance of the multi-plane RANSAC fitting algorithm, even if there are a large number of outliers in the point cloud, it can still accurately fit the horizontal and vertical planes of the cardboard box stack, avoiding plane fitting deviation caused by noise. Compared with traditional pure point cloud detection methods, the noise resistance is significantly improved, and it can adapt to complex situations such as changes in lighting and slight occlusion in the warehouse environment, greatly enhancing the robustness of detection.

[0030] 4. This invention uses multi-plane RANSAC fitting to fit two sets of approximately orthogonal principal planes, constructs a standard orthogonal basis to complete coordinate correction, and transforms the point cloud in the camera coordinate system into a target coordinate system consistent with the principal plane of the cardboard box stack. This completely eliminates the detection deviation caused by the camera's placement posture and shooting angle, ensuring that the detection results are consistent under different shooting angles and camera positions. It solves the problem of volume calculation and layer count deviation caused by camera posture deviation in the prior art, and further improves the accuracy and consistency of the detection results.

[0031] 5. This invention uses an extreme value smoothing strategy to calculate the overall stacked 3D bounding box volume. By using a sliding window to smooth and denoise the extreme values ​​of the coordinates, it effectively avoids the bounding box offset caused by residual noise at the edge of the point cloud. Compared with the traditional method of directly extracting extreme values ​​to calculate the volume, the volume calculation error is significantly reduced, and the true volume of the overall stack can be accurately obtained, providing accurate quantitative support for warehouse inventory and location utilization assessment. Attached Figure Description

[0032] Figure 1 This is a flowchart illustrating the method of Embodiment 1 of the present invention;

[0033] Figure 2 This is a schematic diagram of the process for generating a colored point cloud in Embodiment 1 of the present invention. Detailed Implementation

[0034] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0035] Example 1

[0036] like Figure 1 As shown, this embodiment provides a warehouse cardboard box detection method based on instance segmentation and RGBD point cloud fusion, including the following steps:

[0037] S1) Obtain the RGB image of the location to be tested and the corresponding depth image aligned to the same resolution;

[0038] S2) The YOLO instance segmentation model is used to perform instance segmentation and detection on the RGB image of the storage location to be tested, obtain the instance prediction results, and generate instance masks. After confidence filtering and non-maximum suppression, an effective set of instance masks is obtained.

[0039] In this embodiment, the YOLO instance segmentation model consists of a backbone network, a neck network, and a detection head. The backbone network adopts a CSPDarknet or ViT-Transformer structure to extract multi-scale image features from the RGB image of the location to be tested. The neck network adopts an FPN or PAN structure to fuse feature maps of different scales and enhance the feature representation capability of small targets (cardboard boxes). The detection head outputs predictions, target confidence, and class probabilities.

[0040] In this embodiment, the YOLO instance segmentation model is used to perform instance segmentation and detection on the RGB image of the storage location under test, as follows:

[0041] S21) Extract multi-scale feature maps from the RGB image of the storage location to be tested using the backbone network. , , ;in, This is a shallow feature map; This is a mid-level feature map; For deep feature maps;

[0042] S22) The neck network achieves multi-scale feature maps through upsampling and element-wise summation. , , The fusion yields a fused feature map. ;

[0043] S23) The detection head is used to fuse the feature map. Perform convolutional prediction and output the prediction result corresponding to the anchor box; the prediction result includes:

[0044] Predicted box offset ,in, Offset to center coordinates; The aspect ratio;

[0045] Target confidence score, used to predict the probability that the anchor box contains the target. ;

[0046] Category probability, the one-hot coding probability used to predict the carton category. .

[0047] S24) Anchor frame prior information Offset from the prediction box Combined, the true prediction box is obtained. ,Right now:

[0048] ;

[0049] ;

[0050] ;

[0051] ;

[0052] ;

[0053] In the formula, The coordinates of the prior center of the anchor frame; The center coordinates of the prediction box; These are the prior height and width of the anchor frame, respectively; These represent the height and width of the prediction box, respectively.

[0054] In this embodiment, an instance mask is generated as follows:

[0055] S25), for each detected prediction box Perform RoIAlign operation to extract a fixed-size mask feature map. ;

[0056] ;

[0057] S26) By analyzing the mask feature map Perform transposed convolution upsampling to output the mask prototype Proto and mask coefficients Coeff; specifically:

[0058] Pair of mask feature maps Perform transposed convolution upsampling and simultaneously adjust the number of channels using 1×1 convolution to obtain the enhanced mask feature map. ;

[0059] Enhanced mask feature map Perform multiple rounds of convolution and upsampling operations to output the mask prototype Proto;

[0060] Simultaneously, through the enhanced mask feature map Perform convolution and global pooling operations on the mask feature map It is compressed into a one-dimensional vector, and then the dimension is adjusted through a fully connected layer to output the mask coefficient Coeff;

[0061] S27) By linearly combining the mask prototype and mask coefficients through matrix multiplication, a binary mask for each target is generated, i.e.:

[0062] ;

[0063] in, For flattening operation;

[0064] For mask logical values After reshaping, the probability mask is obtained by activating the Sigmoid function. ,Right now:

[0065] ;

[0066] Finally, a threshold is set for binarization to obtain the final instance mask set. ,Right now:

[0067]

[0068] In the formula, This is an indicator function; it outputs 1 if the condition is met, and 0 otherwise. For the first The binary mask of each target instance in the image coordinates The value at that location.

[0069] In this embodiment, after confidence filtering and non-maximum suppression, a set of effective instance masks is obtained; specifically:

[0070] S28) Calculate the overall confidence of each detection box, filter the predicted boxes based on the confidence threshold, and perform non-maximum suppression through IOU to eliminate overlapping boxes;

[0071] In this embodiment, the overall confidence level of each detection box is calculated based on the target confidence level and the class probability. And remove the overall confidence level Predicted bounding boxes with a confidence level below the threshold are used to obtain a candidate box set; the overall confidence level... The calculation formula is expressed as:

[0072] .

[0073] In this embodiment, non-maximum suppression is performed using IOU to eliminate overlapping boxes; specifically as follows:

[0074] All predicted boxes within the candidate box set are evaluated based on their overall confidence level. Sort in descending order;

[0075] Selecting the overall confidence level The highest predicted bounding box A is selected, and the Interchange of Union (IOU) value between predicted bounding box A and other predicted bounding boxes B is calculated. Redundant boxes with IOU values ​​greater than the IOU threshold are removed.

[0076] ;

[0077] This represents the pixel area of ​​the intersection region of prediction boxes A and B; The pixel area of ​​the union region of bounding boxes A and B is predicted.

[0078] Repeat this process until all boxes have been processed, resulting in a valid set of instance masks.

[0079] S3) Generate an instance color mask map and overlay visualization based on the valid instance mask set; count the number of instances in the layer structure. Number of top box instances The details are as follows:

[0080] S31) For each instance mask, assign a corresponding color according to its category, where top box instances are mapped to random colors, and layer structure instances or reference categories are mapped to fixed colors.

[0081] S32) Generate an instance color mask map by covering the corresponding mask area according to the assigned color. ,Right now:

[0082] ;

[0083] S33) Instance color mask image With preset transparency and the original RGB image Pixel-level fusion is performed to obtain an overlay visualization. ,Right now:

[0084] ;

[0085] In the formula, The preset transparency factor;

[0086] S34) Employing a text-based obstacle avoidance strategy for overlaying visualization images The text box position is iteratively offset, specifically as follows:

[0087] Calculate the center coordinates of each instance mask And use it as the initial text anchor point;

[0088] Calculate the width and height of the text box based on the font and font size. ;

[0089] Place text labels starting from the center of the mask. If the current text box intersects a rectangle with any existing text box, move the text box along the y-direction by a step size. = After shifting downwards, the detection is repeated, and the loop iterates until the objects no longer intersect. If the position is confirmed, it is added to the text box list. If the image exceeds the lower boundary after shifting, the position is returned to the initial position or the position is cropped to the image boundary before the process ends.

[0090] S35), Statistical Hierarchical Structure Instance Count Number of top box instances ,Right now:

[0091] ;

[0092] ;

[0093] In the formula, Indicates the number of elements in the set; For instance indexing, For the first Binary masks for each instance; It is a collection of all instances of the layer structure; This is the collection of all top box instances.

[0094] S4) Perform pixel-level fusion of the depth image and the instance color mask image, and back-project the effective pixels into 3D points based on the pinhole camera model to generate a colored point cloud; such as Figure 2 As shown, the details are as follows:

[0095] S41) Pair each pixel position in the depth image one by one Perform detection to determine the corresponding pixel position. Check if the corresponding depth value is greater than 0. If the depth value is greater than 0, it is considered valid, and the pixel position is set. Mark it as valid; otherwise, determine it as invalid and discard it as an invalid depth point to obtain a valid depth mask;

[0096] S42) Based on the color rules of the instance color mask, determine the position of each pixel in the instance color mask pixel by pixel. The color, if the corresponding pixel position If the color is the background color or a preset invalid category color, it is marked as invalid background or invalid category, and the pixels with invalid background or invalid category are removed to obtain the color filter mask;

[0097] S43) Perform a bitwise AND operation between the depth-validated mask and the color-filtered mask to obtain the final retained mask. ;

[0098] S44) For the final reserved mask Each pixel in Extract its corresponding depth value in the depth image. And, combined with preset camera intrinsic parameters, calculate pixels through back projection. The corresponding three-dimensional point coordinates are:

[0099]

[0100] ;

[0101] In the formula, For three-dimensional point coordinates; Let these be the coordinates of the camera's principal point. , For focal length scale coefficients; all are camera intrinsic parameters;

[0102] S45) The final preserved mask Each corresponding pixel The RGB color values ​​of the instance color mask are normalized to the [0,1] interval as the point color, and then compared with the corresponding 3D point coordinates. Binding allows each 3D point to have corresponding instance color information, resulting in a point cloud object. .

[0103] S5) Perform radius outlier culling on the colored point cloud objects to obtain a clean point cloud set; specifically as follows:

[0104] S51) Set the filter radius and minimum neighborhood number ;

[0105] S52), for each point cloud object Point cloud objects Centered on the set filter radius Construct a spherical neighborhood space for the range;

[0106] S53) Count the number of other point cloud objects falling within the spherical neighborhood space; if the number of point cloud objects within the spherical neighborhood space is greater than or equal to the minimum neighborhood point count... Then the point cloud object is considered Points located in dense areas are considered valid target points and will be retained; otherwise, they will be identified as outliers and removed directly from the point cloud.

[0107] S54) After traversing and judging all point cloud objects in the point cloud object set, all the remaining valid point cloud objects are recombined to form a clean point cloud after radius filtering and noise reduction.

[0108] S6) Perform multiplane RANSAC fitting on the clean point cloud set to obtain at least two sets of approximately orthogonal principal planes;

[0109] S61), Set distance threshold Number of sampling points and the maximum number of iterations ;

[0110] S62) Randomly select from the clean point cloud set in each iteration. Fit candidate planes at points ;in, , , The normal vector components of the plane, the normal vector ; This is the offset of the plane from the origin of the coordinate system; , , These are the three-dimensional coordinates of any point in space;

[0111] S63) Calculate the distance from all points in the clean point cloud set to the candidate plane, and set the distances less than the distance threshold. The point cloud is determined to be an interior point of the candidate plane; that is:

[0112]

[0113] In the formula, For point clouds Distance to the candidate plane; , , For point clouds 3D coordinates;

[0114] S64), repeated iterations The plane with the most interior points and whose number of interior points is greater than the minimum threshold for the number of interior points in a plane is selected as the effective fitting plane.

[0115] S65) Perform multiple rounds of RANSAC plane fitting, and save the effective fitted plane after each fitting.

[0116] S66) Select the two planes with the most interior points from all the effective fitted planes of the multi-round RANSAC plane fitting, and take the plane with the most interior points as the first plane and the other plane as the second plane. Determine whether the second plane and the first plane are approximately orthogonal. If they are not orthogonal, remove the second plane and re-select the second plane from the effective fitted planes until it is orthogonal to the first plane.

[0117] In this embodiment, the orthogonality is determined by the angle θ between the normal vectors of the second plane and the first plane; that is:

[0118] ;

[0119] like If the first plane is perpendicular to the second plane, then the second plane is orthogonal to the first plane; otherwise, they are not orthogonal. The orthogonal first plane and the second plane are taken as the first principal plane and the second principal plane, respectively.

[0120] The plane whose normal vector is approximately perpendicular to the ground is taken as the horizontal plane of the corresponding cardboard box stack, and the plane whose normal vector is approximately parallel to the ground is determined as the vertical plane of the corresponding side of the cardboard box stack.

[0121] S7) Construct an orthogonal basis based on the normal vectors of the two sets of principal planes and correct the point cloud coordinates to obtain a point cloud with an aligned coordinate system; specifically as follows:

[0122] S71) Based on the two sets of principal planes mentioned above, construct the orthogonal basis of the target coordinate system. ,Right now:

[0123] Take the unit normal vector of the horizontal plane as the positive z-axis direction of the target coordinate system. ;

[0124] Take the unit normal vector of the vertical plane as the positive x-axis direction of the target coordinate system. ;

[0125] With the positive x-axis With the positive z-axis direction The dot product is used as the positive direction of the y-axis. ,Right now:

[0126] ;

[0127] orthogonal basis of construction After normalization, the three coordinate axes of the target coordinate system are finally obtained as a standard orthogonal basis.

[0128] S72), based on orthogonal basis The 3D coordinates of all points in the clean point cloud set. Transform from camera coordinate system to the constructed target coordinate system ;

[0129] ;

[0130] ;

[0131] In the formula, Coordinate transformation matrix The inverse matrix is ​​used to perform coordinate mapping from the camera coordinate system to the target coordinate system; The origin of the target coordinate system;

[0132] After all clean transformations are completed, a point cloud set with coordinates aligned is obtained.

[0133] S8) Estimate the 3D bounding box of the coordinate-aligned point cloud set, calculate the overall stacked 3D bounding box volume using an extreme value smoothing strategy, and calculate the volume of a single box on the top box point set to obtain the estimated number of boxes per layer; as detailed below:

[0134] S81), a point cloud set aligned with the coordinate system. Take M minimum values ​​and M maximum values ​​for each of the three dimensions, and calculate the mean of each as the smoothed boundary. , , , , , ;

[0135] S82) Construct a stacked 3D bounding box based on the smoothed boundary and calculate its volume. ,Right now:

[0136]

[0137] The volume of the stacked 3D bounding box Compared with the preset volume range, if the volume of the overall stacked 3D bounding box is... If the value is greater than 0 and less than the preset maximum volume threshold, the volume is considered valid; if... If the volume exceeds the preset maximum volume threshold, then step S81 will be executed again.

[0138] S83) Remove the layer structure point set corresponding to a fixed color from the point cloud color to obtain the candidate point set of the top box; group the candidate point set of the top box by color and select the color group with the most points as the single box point set.

[0139] (S84) Fit at least two approximately orthogonal planes onto the point set of a single box to ensure the geometric observability of the box, and obtain the volume of the single box boundary box based on extreme value smoothing. ;

[0140] S85), Volume of a 3D bounding box based on global stacking Volume of a single box boundary frame Calculate the estimated number of boxes per layer ;

[0141]

[0142] S86) Estimation based on the number of boxes per layer Number of layered structure instances Number of top box instances The total number of all boxes is calculated.

[0143] Example 2

[0144] This embodiment provides a warehouse cardboard box detection method based on instance segmentation and RGBD point cloud fusion, including:

[0145] The data acquisition module is used to acquire the RGB image of the location to be tested and the corresponding depth image aligned to the same resolution;

[0146] The instance segmentation and mask generation module uses the YOLO instance segmentation model to perform instance segmentation and detection on the RGB image of the test location, obtains the instance prediction results, and generates instance masks. After confidence filtering and non-maximum suppression, an effective set of instance masks is obtained.

[0147] The instance statistics module is used to generate instance color mask images and overlay visualizations based on the set of valid instance masks; it also counts the number of instances in the layer structure. Number of top box instances ;

[0148] The point cloud reconstruction module is used to perform pixel-level fusion of depth images and instance color mask images, and back-project effective pixels into three-dimensional points based on the pinhole camera model to generate colored point clouds.

[0149] The point cloud filtering module is used to remove outliers by radius from colored point cloud objects to obtain a clean point cloud set.

[0150] The fitting module is used to perform multiplane RANSAC fitting on a clean point cloud set to obtain at least two sets of approximately orthogonal principal planes.

[0151] The point cloud alignment module is used to construct an orthogonal basis based on the normal vectors of the two sets of principal planes and correct the point cloud coordinates to obtain a point cloud with coordinate system alignment.

[0152] The single-layer box number estimation module is used to estimate the 3D bounding box of a point cloud set aligned with the coordinate system. It uses an extreme value smoothing strategy to calculate the overall stacked 3D bounding box volume and calculates the volume of a single box on the top box point set to obtain the estimated value of the single-layer box number.

[0153] Example 3

[0154] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the warehouse carton detection method based on instance segmentation and RGBD point cloud fusion described in Embodiment 1.

[0155] In this embodiment, the memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. A processor, coupled to the memory, is used to execute computer programs stored in the memory.

[0156] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form.

[0157] The embodiments and descriptions above are merely illustrative of the principles and preferred embodiments of the present invention. Various changes and modifications may be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.

Claims

1. A warehouse cardboard box detection method based on instance segmentation and RGBD point cloud fusion, characterized in that, Includes the following steps: S1) Obtain the RGB image of the location to be tested and the corresponding depth image aligned to the same resolution; S2) The YOLO instance segmentation model is used to perform instance segmentation and detection on the RGB image of the storage location to be tested, obtain the instance prediction results, and generate instance masks. After confidence filtering and non-maximum suppression, an effective set of instance masks is obtained. S3) Generate an instance color mask map and overlay visualization based on the valid instance mask set; count the number of instances in the layer structure. Number of top box instances ; S4) Perform pixel-level fusion of the depth image and the instance color mask image, and back-project the effective pixels into 3D points based on the pinhole camera model to generate a colored point cloud; S5) Perform radius outlier removal on the colored point cloud objects to obtain a clean point cloud set; S6) Perform multi-plane RANSAC fitting on the clean point cloud set to obtain at least two sets of approximately orthogonal principal planes; S7) Construct an orthogonal basis based on the normal vectors of the two sets of principal planes and correct the point cloud coordinates to obtain a point cloud with an aligned coordinate system. S8) Estimate the 3D bounding box of the point cloud set aligned with the coordinate system, use the extreme value smoothing strategy to calculate the volume of the overall stacked 3D bounding box, and calculate the volume of a single box on the top box point set to obtain the estimated number of boxes in a single layer.

2. The warehouse cardboard box detection method based on instance segmentation and RGBD point cloud fusion according to claim 1, characterized in that: In step S2), the instance prediction results include: Predicted box offset ,in, Offset to center coordinates; The aspect ratio; Target confidence score, used to predict the probability that the anchor box contains the target. ; Category probability, the one-hot coding probability used to predict the carton category. ; By anchor frame prior information Offset from the prediction box Combined, the true prediction box is obtained. .

3. The warehouse cardboard box detection method based on instance segmentation and RGBD point cloud fusion according to claim 2, characterized in that: In step S1), the generation and filtering of the instance mask are as follows: S25), for each detected prediction box Perform RoIAlign operation to extract a fixed-size mask feature map. ; ； S26) By analyzing the mask feature map Perform transposed convolution upsampling to output the mask prototype (Proto) and mask coefficients (Coeff); specifically: Pair of mask feature maps Perform transposed convolution upsampling and simultaneously adjust the number of channels using 1×1 convolution to obtain the enhanced mask feature map. ; Enhanced mask feature map Perform multiple rounds of convolution and upsampling operations to output the mask prototype Proto; Simultaneously, through the enhanced mask feature map Perform convolution and global pooling operations on the mask feature map It is compressed into a one-dimensional vector, and then the dimension is adjusted through a fully connected layer to output the mask coefficient Coeff; S27) By linearly combining the mask prototype and mask coefficients through matrix multiplication, a binary mask for each target is generated, i.e.: ； in, For flattening operation; For mask logical values After reshaping, the probability mask is obtained by activating the Sigmoid function. ,Right now: ； Finally, a threshold is set for binarization to obtain the final instance mask set. ,Right now: In the formula, This is an indicator function; it outputs 1 if the condition is met, and 0 otherwise. For the first The binary mask of each target instance in the image coordinates The value at; S28) Calculate the overall confidence of each detection box, filter the predicted boxes based on the confidence threshold, and perform non-maximum suppression through IOU to eliminate overlapping boxes and obtain a set of effective instance masks.

4. The warehouse cardboard box detection method based on instance segmentation and RGBD point cloud fusion according to claim 3, characterized in that: In step S3), an instance color mask image and an overlay visualization image are generated based on the valid instance mask set, as detailed below: S31) For each instance mask, assign a corresponding color according to its category, where top box instances are mapped to random colors, and layer structure instances or reference categories are mapped to fixed colors. S32) Generate an instance color mask map by covering the corresponding mask area according to the assigned color. , S33) Instance color mask image With preset transparency and the original RGB image Pixel-level fusion is performed to obtain an overlay visualization. ; S34) Employing a text-based obstacle avoidance strategy for overlaying visualization images The text box position is iteratively offset, specifically as follows: Calculate the center coordinates of each instance mask And use it as the initial text anchor point; Calculate the width and height of the text box based on the font and font size. ; Place text labels starting from the center of the mask. If the current text box intersects a rectangle with any existing text box, move the text box along the y-direction by a step size. = After shifting downwards, the detection is repeated, and the loop iterates until the objects do not intersect. If the position is confirmed, it is added to the text box list. If the position exceeds the lower boundary of the image after shifting, the position is returned to the initial position or the position is cropped to the range of the image boundary before the process ends.

5. The warehouse cardboard box detection method based on instance segmentation and RGBD point cloud fusion according to claim 4, characterized in that: In step S4), the depth image and the instance color mask image are fused pixel-level, and the effective pixels are back-projected into 3D points based on the pinhole camera model to generate a colored point cloud; specifically as follows: S41) Pair each pixel position in the depth image one by one Perform detection to determine the corresponding pixel position. Check if the corresponding depth value is greater than 0. If the depth value is greater than 0, it is considered valid, and the pixel position is set. Mark it as valid; otherwise, determine it as invalid and discard it as an invalid depth point to obtain a valid depth mask; S42) Based on the color rules of the instance color mask, determine the position of each pixel in the instance color mask pixel by pixel. The color, if the corresponding pixel position If the color is the background color or a preset invalid category color, it is marked as invalid background or invalid category, and the pixels with invalid background or invalid category are removed to obtain the color filter mask; S43) Perform a bitwise AND operation between the depth-validated mask and the color-filtered mask to obtain the final retained mask. ; S44) For the final reserved mask Each pixel in Extract its corresponding depth value in the depth image. And, combined with preset camera intrinsic parameters, calculate pixels through back projection. The corresponding three-dimensional point coordinates; S45) The final preserved mask Each corresponding pixel The RGB color values ​​of the instance color mask are normalized to the [0,1] interval as the point color, and then compared with the corresponding 3D point coordinates. Binding allows each 3D point to have corresponding instance color information, resulting in a point cloud object. .

6. The warehouse cardboard box detection method based on instance segmentation and RGBD point cloud fusion according to claim 5, characterized in that: In step S5), outlier removal by radius is performed on the colored point cloud objects to obtain a clean point cloud set; specifically as follows: S51) Set the filter radius with minimum neighborhood number ; S52), for each point cloud object Point cloud objects Centered on the set filter radius Construct a spherical neighborhood space for the range; S53) Count the number of other point cloud objects falling within the spherical neighborhood space; if the number of point cloud objects within the spherical neighborhood space is greater than or equal to the minimum neighborhood point count... Then the point cloud object is considered Located in a densely populated area, it is a valid target point and should be retained. Otherwise, it is judged as an outlier and noise point and directly deleted from the point cloud; S54) After traversing and judging all point cloud objects in the point cloud object set, all the remaining valid point cloud objects are recombined to form a clean point cloud after radius filtering and noise reduction.

7. The warehouse cardboard box detection method based on instance segmentation and RGBD point cloud fusion according to claim 6, characterized in that: In step S6), multi-plane RANSAC fitting is performed on the clean point cloud set to obtain at least two sets of approximately orthogonal principal planes; S61), Set distance threshold Number of sampling points and the maximum number of iterations ; S62) Randomly select from the clean point cloud set in each iteration. Fit candidate planes at points ;in, , , The normal vector components of the plane, the normal vector ; This is the offset of the plane from the origin of the coordinate system; , , These are the three-dimensional coordinates of any point in space; S63) Calculate the distance from all points in the clean point cloud set to the candidate plane, and set the distances less than the distance threshold. The point cloud is determined to be an interior point of the candidate plane; that is: In the formula, For point clouds Distance to the candidate plane; , , For point clouds 3D coordinates; S64), repeated iterations The plane with the most interior points and whose number of interior points is greater than the minimum threshold for the number of interior points in a plane is selected as the effective fitting plane. S65) Perform multiple rounds of RANSAC plane fitting, and save the effective fitted plane after each fitting. S66) Select the two planes with the most interior points from all the effective fitted planes of the multi-round RANSAC plane fitting, and take the plane with the most interior points as the first plane and the other plane as the second plane. Determine whether the second plane and the first plane are approximately orthogonal. If they are not orthogonal, remove the second plane and re-select the second plane from the effective fitted planes until it is orthogonal to the first plane. Whether the planes are orthogonal is determined by the angle θ between the normal vectors of the second plane and the first plane; that is: ； like If the first plane is perpendicular to the second plane, then the second plane is orthogonal to the first plane; otherwise, they are not orthogonal. The orthogonal first plane and the second plane are taken as the first principal plane and the second principal plane, respectively. The plane whose normal vector is approximately perpendicular to the ground is taken as the horizontal plane of the corresponding cardboard box stack, and the plane whose normal vector is approximately parallel to the ground is determined as the vertical plane of the corresponding side of the cardboard box stack.

8. The warehouse cardboard box detection method based on instance segmentation and RGBD point cloud fusion according to claim 7, characterized in that: In step S8), 3D bounding box estimation is performed on the coordinate-aligned point cloud set. An extreme value smoothing strategy is used to calculate the overall stacked 3D bounding box volume, and the volume of a single box is calculated on the top box point set to obtain the estimated number of boxes per layer; specifically as follows: S81), a point cloud set aligned with the coordinate system. Take M minimum values ​​and M maximum values ​​for each of the three dimensions, and calculate the mean of each as the smoothed boundary. , , , , , ; S82) Construct a stacked 3D bounding box based on the smoothed boundary and calculate its volume. ,Right now: The volume of the stacked 3D bounding box Compared with the preset volume range, if the volume of the overall stacked 3D bounding box is... If the value is greater than 0 and less than the preset maximum volume threshold, the volume is considered valid; if... If the volume exceeds the preset maximum volume threshold, then step S81 will be executed again. S83) Remove the layer structure point set corresponding to a fixed color from the point cloud color to obtain the candidate point set of the top box; group the candidate point set of the top box by color and select the color group with the most points as the single box point set. (S84) Fit at least two approximately orthogonal planes onto the point set of a single box to ensure the geometric observability of the box, and obtain the volume of the single box boundary box based on extreme value smoothing. ; S85), Volume of a 3D bounding box based on global stacking Volume of a single box boundary frame Calculate the estimated number of boxes per layer ; S86) Estimation based on the number of boxes per layer Number of layered structure instances Number of top box instances The total number of all boxes is calculated.

9. A warehouse cardboard box detection method based on instance segmentation and RGBD point cloud fusion, characterized in that, include: The data acquisition module is used to acquire the RGB image of the location to be tested and the corresponding depth image aligned to the same resolution; The instance segmentation and mask generation module uses the YOLO instance segmentation model to perform instance segmentation and detection on the RGB image of the test location, obtains the instance prediction results, and generates instance masks. After confidence filtering and non-maximum suppression, an effective set of instance masks is obtained. The instance statistics module is used to generate instance color mask images and overlay visualizations based on the set of valid instance masks; it also counts the number of instances in the layer structure. Number of top box instances ; The point cloud reconstruction module is used to perform pixel-level fusion of depth images and instance color mask images, and back-project effective pixels into three-dimensional points based on the pinhole camera model to generate colored point clouds. The point cloud filtering module is used to remove outliers by radius from colored point cloud objects to obtain a clean point cloud set. The fitting module is used to perform multiplane RANSAC fitting on a clean point cloud set to obtain at least two sets of approximately orthogonal principal planes. The point cloud alignment module is used to construct an orthogonal basis based on the normal vectors of the two sets of principal planes and correct the point cloud coordinates to obtain a point cloud with coordinate system alignment. The single-layer box number estimation module is used to estimate the 3D bounding box of a point cloud set aligned with the coordinate system. It uses an extreme value smoothing strategy to calculate the overall stacked 3D bounding box volume and calculates the volume of a single box on the top box point set to obtain the estimated value of the single-layer box number.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the warehouse carton detection method based on instance segmentation and RGBD point cloud fusion as described in any one of claims 1-8.

Images