Tray box target detection and quantity statistics method based on multi-view depth camera

Abstract

The invention relates to the technical field of intelligent logistics storage, in particular to a tray box target detection and quantity statistics method based on a multi-view depth camera, and the method comprises the following steps: 1, building a system architecture; a pretreatment process; step 3, a target identification process; step 4, a depth camera continuous image acquisition process; and step 5, an output process: adopting the training model to carry out tray box body identification on a plurality of visual angles, and carrying out fusion and three-dimensional positioning on detection result information of the plurality of visual angles to complete quantity statistics. The positions, the depths and the stacking layers of the box bodies on the tray and the number of the box bodies on each layer can be quickly and accurately identified, and the cargo management efficiency in logistics storage and industrial production is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent logistics warehousing technology, and specifically to a pallet box target detection and quantity counting method based on a multi-view depth camera. Background Art

[0002] Pallets are a common means of transporting goods in numerous fields, including logistics, warehousing, and industrial production. Rapid and accurate identification of the number and status of stacked boxes on pallets is crucial for inventory management, logistics scheduling, and automated production processes. Traditional manual counting methods are inefficient and prone to errors. Standard image object detection based on a single perspective struggles to fully capture the three-dimensional information of boxes, making it impossible to accurately determine the number of stacked boxes and the number of boxes per layer, making it difficult to meet the demands of modern, efficient, and precise production operations. Summary of the Invention

[0003] The purpose of the present invention is to address the shortcomings of the existing technology and provide a pallet box target detection and quantity counting method based on a multi-view depth camera, which can quickly and accurately identify the position, depth, number of stacking layers and the number of boxes on each layer of the boxes on the pallet, thereby improving the efficiency of cargo management in logistics warehousing and industrial production.

[0004] In view of the above technical problems, the technical solutions adopted by the present invention are as follows:

[0005] The method for detecting and counting pallet box objects based on a multi-view depth camera includes the following steps:

[0006] Step 1: System architecture construction process: properly fix depth cameras on the top, front, back, left, and right of the pallet to ensure that the field of view of each camera can fully cover the pallet and the range of boxes that may be stacked on it;

[0007] Step 2: Preprocessing step, by obtaining the box data set and labeling and preprocessing the data set;

[0008] Step 3: Target recognition process: train the box data set using the improved YOLV8 box target recognition algorithm and save the training model.

[0009] Step 4: Continuous image acquisition process of the depth camera. The depth camera is connected to a computing unit with image processing capabilities to receive and process image data from each camera in real time.

[0010] Step 5: Output process: Use the trained model to identify the pallet box from multiple perspectives, and fuse the detection results from multiple perspectives with three-dimensional positioning to complete quantity statistics.

[0011] As a preferred embodiment, in the step 1, in the actual application scenario, according to the pallet size and workspace layout, a depth camera model such as the Intel RealSense series is selected, and a sturdy and stable mounting bracket is designed to fix the five depth cameras around the pallet in a symmetrical distribution up and down, front and back, and left and right to ensure that there are no visual blind spots. At the same time, standard tools such as the checkerboard calibration plate are used to perform the camera calibration process; first, multiple groups of images of the calibration plate under each camera perspective are captured, and the camera intrinsic parameters are solved by extracting and matching the corner points of the calibration plate, and using classic algorithms such as the Zhang Zhengyou calibration method. Then, by moving the calibration plate, image pairs at different positions are obtained. Based on the correspondence between the feature points, the bundle adjustment method is used to calculate the camera extrinsic parameters to complete the precise calibration of the entire camera system, and the calibration parameters are stored for subsequent image coordinate conversion.

[0012] As a preferred embodiment, in the step 2, different types of box images in a logistics warehousing environment are collected to form a training data set for pallet boxes; the preprocessed RGB images are sent to the YOLO target detection model inference engine. The model is built based on the deep learning framework PyTorch, performs efficient calculations on the GPU, and quickly outputs the two-dimensional detection box coordinates, category confidence and other information of the box in each perspective image. The confidence threshold can be dynamically adjusted according to the actual detection accuracy requirements to balance the false detection rate and the missed detection rate.

[0013] As a preferred embodiment, in the step three, the model is constructed by adding the YOLOv8 model to the ELA attention mechanism and the RVB module, the data set is input into the improved YOLOv8n network model for pre-training, the pre-training weight with the smallest training loss value is taken for transfer learning, the data set and the pre-training weight are combined for training to obtain the optimal weight; the image to be detected is detected according to the optimal weight, the detection result is evaluated, and the box detection is realized.

[0014] As a preferred embodiment, in the step three, on the basis of YOLOv8n, the C2f-RVB module is constructed by introducing RepViTBlock to replace the C2f module of YOLOv8 to reduce the number of model parameters and speed up the inference time. The idea of ​​advanced feature fusion pyramid HSFPN and efficient local attention mechanism are used to design the ELA-HSFPN structure to enhance the local feature extraction capability of the model. The background interference is reduced by adding rotation angle regression, and the Wisel-IoU loss function is used to balance the sample quality of the data set to further improve the accuracy and generalization ability of the algorithm.

[0015] As a preferred embodiment, in the step four, the computing unit accelerates the coordinate conversion process based on the stored camera extrinsics using parallel computing technologies such as CUDA, synchronously converts the coordinates of the two-dimensional detection frame of the box under different perspectives into the world coordinate system, and obtains the three-dimensional coordinates of the box in combination with the depth value in the corresponding depth image. In this process, a multi-threaded synchronization mechanism is set to ensure the timeliness and consistency of data processing at each perspective.

[0016] As a preferred embodiment, in the step 4, for the multi-view fusion link, a method based on SURF feature point extraction and matching is adopted to find the feature points corresponding to the box between each view, and the RANSAC algorithm is combined to eliminate the mismatched points to further optimize the three-dimensional positioning result of the box; at the same time, the detection frame is fine-tuned using regional overlap analysis to remove edge misjudgments caused by occlusion, reflection and other factors, thereby ensuring the accuracy of box positioning.

[0017] As another preferred embodiment, in step five, for the multi-view fusion link, a method based on SURF feature point extraction and matching is used to find feature points corresponding to the box between each view, and the RANSAC algorithm is combined to eliminate mismatched points to further optimize the three-dimensional positioning result of the box; at the same time, the detection frame is fine-tuned using regional overlap analysis to remove edge misjudgments caused by occlusion, reflection and other factors, thereby ensuring the accuracy of box positioning.

[0018] Beneficial effects of the present invention:

[0019] 1. The detection accuracy of box targets in this invention has been improved by about 5%. It can comprehensively and accurately obtain the three-dimensional spatial information of the boxes on the pallet, overcome the limitations of single-view detection, greatly improve the accuracy of box recognition and positioning, and achieve stable detection even in complex stacking and occlusion conditions. It realizes the automation of cargo quantity counting, which is more efficient than traditional manual counting.

[0020] In summary, this equipment has the advantages of improving the accuracy of box identification and positioning, and improving counting efficiency, and is particularly suitable for the field of intelligent logistics and warehousing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 This is a process flow chart for the pallet box object detection and quantity counting method based on a multi-view depth camera;

[0023] Figure 2 It is a schematic diagram of the algorithm model;

[0024] Figure 3 Schematic diagram of the precision, recall and mAP@0.5 curves;

[0025] Figure 4 Schematic diagram of experimental results;

[0026] Figure 5 Schematic diagram of the improved algorithm model. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present invention are clearly and completely described below with reference to the accompanying drawings.

[0028] Example 1

[0029] like Figures 1 to 2 As shown in the figure, the pallet box object detection and quantity counting method based on the multi-view depth camera includes the following steps:

[0030] Step 1: System architecture construction process: properly fix depth cameras on the top, front, back, left, and right of the pallet to ensure that the field of view of each camera can fully cover the pallet and the range of boxes that may be stacked on it;

[0031] Step 2: Preprocessing step, by obtaining the box data set and labeling and preprocessing the data set;

[0032] Step 3: Target recognition process: train the box data set using the improved YOLV8 box target recognition algorithm and save the training model;

[0033] Step 4: Continuous image acquisition process of the depth camera. The depth camera is connected to a computing unit with image processing capabilities to receive and process image data from each camera in real time.

[0034] Step 5: Output process: Use the trained model to identify the pallet box from multiple perspectives, and fuse the detection results from multiple perspectives with three-dimensional positioning to complete quantity statistics.

[0035] In the step 1, in the actual application scenario, according to the pallet size and workspace layout, a depth camera model such as the Intel RealSense series is selected, and a sturdy and stable mounting bracket is designed. The five depth cameras are fixed around the pallet in a symmetrical distribution in the upper and lower, front and back, and left and right directions to ensure that there are no blind spots in vision. At the same time, standard tools such as a checkerboard calibration plate are used to perform a camera calibration process. First, multiple sets of images of the calibration plate under each camera perspective are captured, and the camera intrinsic parameters are solved by extracting and matching the corner points of the calibration plate and applying classic algorithms such as the Zhang Zhengyou calibration method. Then, by moving the calibration plate, image pairs at different positions are obtained. Based on the correspondence between the feature points, the bundle adjustment method is used to calculate the camera extrinsic parameters to complete the accurate calibration of the entire camera system, and the calibration parameters are stored for subsequent image coordinate conversion.

[0036] In step 2, different types of box images are collected in a logistics warehousing environment to form a training dataset for pallet boxes. The preprocessed RGB images are fed into the YOLO target detection model inference engine. The model is built based on the deep learning framework PyTorch and performs efficient calculations on the GPU to quickly output the two-dimensional detection box coordinates, category confidence, and other information of the box in each perspective image. The confidence threshold can be dynamically adjusted according to the actual detection accuracy requirements to balance the false detection rate and the missed detection rate.

[0037] In the step three, the model is constructed by adding the YOLOv8 model to the ELA attention mechanism and the RVB module. The data set is input into the improved YOLOv8n network model for pre-training, and the pre-training weight with the smallest training loss value is taken for transfer learning. The data set and the pre-training weight are combined for training to obtain the optimal weight; the image to be detected is detected according to the optimal weight, and the detection result is evaluated to achieve box detection.

[0038] In step 3, based on YOLOv8n, the C2f-RVB module is constructed by introducing RepViTBlock to replace the C2f module of YOLOv8 to reduce the number of model parameters and speed up the inference time. The idea of ​​advanced feature fusion pyramid HSFPN and efficient local attention mechanism are used to design the ELA-HSFPN structure to enhance the local feature extraction capability of the model. The background interference is reduced by adding rotation angle regression, and the Wisel-IoU loss function is used to balance the quality of dataset samples to further improve the accuracy and generalization ability of the algorithm.

[0039] In step 4, the computing unit uses parallel computing technologies such as CUDA to accelerate the coordinate conversion process based on the stored camera extrinsics, synchronously converts the coordinates of the two-dimensional detection frame of the box under different perspectives into the world coordinate system, and combines the depth value in the corresponding depth image to obtain the three-dimensional coordinates of the box. In this process, a multi-threaded synchronization mechanism is set to ensure the timeliness and consistency of data processing at each perspective.

[0040] In step 4, for the multi-view fusion link, a method based on SURF feature point extraction and matching is used to find feature points corresponding to the box between each viewpoint, and the RANSAC algorithm is combined to eliminate mismatched points to further optimize the three-dimensional positioning result of the box. At the same time, the detection frame is fine-tuned using regional overlap analysis to remove edge misjudgments caused by occlusion, reflection and other factors, thereby ensuring the accuracy of box positioning.

[0041] In step five, for the multi-view fusion link, a method based on SURF feature point extraction and matching is used to find feature points corresponding to the box between each viewpoint, and the RANSAC algorithm is combined to eliminate mismatched points to further optimize the three-dimensional positioning result of the box. At the same time, the detection frame is fine-tuned using regional overlap analysis to remove edge misjudgments caused by occlusion, reflection and other factors, thereby ensuring the accuracy of box positioning.

[0042] Example 2

[0043] like Figures 1 to 5 As shown in the figure, the REW-YOLO algorithm is improved based on YOLOv8. The main improvements include:

[0044] S1.C2f-RVB module: Reduces model parameters and accelerates inference by introducing the RepViTBlock module;

[0045] S2.ELA-HSFPN structure: Combining the efficient local attention mechanism ELA and the advanced feature fusion pyramid HSFPN enhances the model's ability to extract local features;

[0046] S3. Rotation Angle Regression: Add rotation angle regression to reduce background interference;

[0047] S4.Wise-IoU loss function: The Wise-IoU loss function is used to balance the quality of dataset samples and improve the algorithm accuracy and generalization ability.

[0048] The network structure of REW-YOLO includes a backbone network, a feature pyramid network and a detection head. The backbone network is used to extract image features, the feature pyramid network is used to fuse multi-scale features, and the detection head outputs the target category and bounding box information.

[0049] The C2f-RVB module includes 1×1 dilated convolution, 3×3 depth-separable convolution, and spatial feature extraction SE attention module. Through structural reparameterization, depth-separable convolution and feedforward module, the computational complexity is significantly reduced. Its mathematical expression is as follows: Input feature map X, dimension is H×W×C, use 1×1 dilated convolution to increase channel dimension: Xexp=Conv1×1(X), use depth-separable convolution to capture spatial information: Xdw=DWConv3×3(Xexp), apply SE (Squeeze-and-Excitation) layer to enhance channel attention: Xse=SE(Xdw), and the final feature map is obtained by 1×1 projection convolution: Y=

[0050] Conv1×1(Xexp).

[0051] The ELA-HSFPN structure enhances the model's ability to extract local features through the efficient local attention mechanism ELA and the advanced feature fusion pyramid HSFPN. The ELA mechanism captures spatial attention through 1D convolution and group normalization (GN), which is defined as follows: Use strip pooling to capture horizontal and vertical feature vectors: Zh =

[0052] AvgPoolH×1(X),Zw=AvgPool1×W(X), apply 1D convolution to capture local interactions: yh=σ(GN(Conv1×k(zh))),yw=σ(GN(Convk×1(zw))) where σ represents the nonlinear activation function ReLU and k is the convolution kernel size. Combining horizontal and vertical attention predictions, we get the position attention map: A=yh×ywA=yh×yw Apply position attention to the input feature map to get the final feature map: Y=X×A.

[0053] Traditional horizontal bounding box detection methods tend to include too much background information when processing tilted boxes, resulting in reduced detection accuracy. REW-YOLO can more accurately represent the box boundaries and reduce background interference by adding rotation angle regression.

[0054] REW-YOLO uses Wise-IoU (WIoU) as the bounding box regression loss function. WIoU reduces large or harmful gradients in extreme samples through a dynamic non-monotonic mechanism and gradient gain distribution strategy, thereby improving the generalization ability of the model. Its calculation formula is as follows:

[0055] L IoU =1-IoU

[0056] L WIoUV1 =R WIoU ×L WIoUV1 =R WIoU ×L IoU

[0057]

[0058] Among them, RWIoU is the dynamic non-monotonic focus coefficient, and the coordinates of the center point of the actual box are expressed as and The coordinates of the center point of the prediction box are given by and The width and height of the minimum perimeter rectangle between the predicted box and the actual box are denoted as c w and c h .

[0059] This experiment was trained on a single high-performance NVIDIA GeForce RTX 4060 graphics card. The deep learning framework was Python 3.11 and CUDA 118. The resolution of all input images was uniformly converted to 640 × 640 pixels. The model was trained for 300 epochs, the number of training samples per batch was 16, and the initial learning rate was set to 0.01. To verify the performance of the REW-YOLO model in box detection, experiments were conducted on the box dataset using the baseline model YOLOv8n and the improved model, as shown in the following figure. Figure 3 As shown in Figure 2, the accuracy, recall, and mAP@0.5 curves of the two models throughout the training cycle are shown. The figure clearly shows that the performance of the REW-YOLO model is always better than the baseline model. The final P and R values ​​of the improved model are 88.3% and 69.9% respectively, and the mAP@0.5 value is 80.3%, which are 2.7%, 1.6%, and 2.4% higher than the baseline YOLOv8n model.

[0060] In order to test the effectiveness of the improved algorithm, this study constructed five sets of ablation experiments, using consistent equipment and datasets for training and testing to ensure consistency in comparison. These five experiments include the original YOLOv8n, YOLOv8n and C2f-RVB, YOLOv8n and HSFPN, YOLOv8s and ELA, and the proposed comprehensive method. The experimental results are shown in Figure 2. Figure 4As shown in the figure, the results show that after adding the C2fRVB module, although the inference speed of the model decreases from 131.1FPS to 83.9FPS, the number of parameters of the model is reduced by about 24%, and mAP@0.5 is improved by 1.6%. This shows that the module improves detection accuracy while reducing computational costs. The HSFPN module significantly improves the recall rate, from 68.3% to 72.0%. In addition, although the FLOPs are slightly increased compared to the C2f-RVB module, the number of parameters is further reduced, and the inference speed is increased to 150.4FPS. The ELA module significantly improves the accuracy from 85.6% to 87.4%, while maintaining a high recall rate. Although the number of parameters is slightly higher than that of the C2fRVB and HSFPN modules, the inference speed reaches the highest 173.5FPS, showing a good performance balance. After using all the optimized modules in combination, the model achieved the greatest improvement in accuracy, from 85.6% to 88.3%. Although the recall rate decreased slightly, it was still higher than the basic version. In addition, the number of parameters and FLOPs were significantly reduced, and the inference speed was maintained at a high level of 130.8FPS, demonstrating the effectiveness of these optimized modules in improving the overall performance of the model.

[0061] Of course, in this technical solution, those skilled in the art should understand that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple, and the term "one" should not be understood as a limitation on the quantity.

[0062] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art based on the technical guidance of the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for detecting and counting pallet box objects based on a multi-view depth camera, characterized in that: The following steps are involved: Step 1: System architecture construction process: properly fix depth cameras on the top, front, back, left, and right of the pallet to ensure that the field of view of each camera can fully cover the pallet and the range of boxes that may be stacked on it; Step 2: Preprocessing step, by obtaining the box data set and labeling and preprocessing the data set; Step 3: Target recognition process: train the box data set using the improved YOLV8 box target recognition algorithm and save the training model. Step 4: Continuous image acquisition process of the depth camera. The depth camera is connected to a computing unit with image processing capabilities to receive and process image data from each camera in real time. Step 5: Output process: Use the trained model to identify the pallet box from multiple perspectives, and fuse the detection results from multiple perspectives with three-dimensional positioning to complete quantity statistics.

2. The method for detecting and counting pallet box objects based on a multi-view depth camera according to claim 1, characterized in that: In the step 1, in the actual application scenario, according to the pallet size and workspace layout, a depth camera model such as the Intel RealSense series is selected, and a sturdy and stable mounting bracket is designed. The five depth cameras are fixed around the pallet in a symmetrical distribution in the upper and lower, front and back, and left and right directions to ensure that there are no blind spots in vision. At the same time, standard tools such as a checkerboard calibration plate are used to perform a camera calibration process. First, multiple sets of images of the calibration plate under each camera perspective are captured, and the camera intrinsic parameters are solved by extracting and matching the corner points of the calibration plate and applying classic algorithms such as the Zhang Zhengyou calibration method. Then, by moving the calibration plate, image pairs at different positions are obtained. Based on the correspondence between the feature points, the bundle adjustment method is used to calculate the camera extrinsic parameters to complete the accurate calibration of the entire camera system, and the calibration parameters are stored for subsequent image coordinate conversion.

3. The method for detecting and counting pallet box objects based on a multi-view depth camera according to claim 1, characterized in that: In the second step, different types of box images are collected in a logistics warehousing environment to form a training data set for pallet boxes; The preprocessed RGB images are fed into the YOLO object detection model inference engine. This model, built on the deep learning framework PyTorch, performs efficient computations on the GPU and quickly outputs information such as the two-dimensional detection box coordinates and category confidence of the boxes in each perspective image. The confidence threshold can be dynamically adjusted based on the actual detection accuracy requirements to balance the false detection rate and missed detection rate.

4. The method for detecting and counting pallet box objects based on a multi-view depth camera according to claim 1, characterized in that: In the step three, the model is constructed by adding the YOLOv8 model to the ELA attention mechanism and the RVB module. The data set is input into the improved YOLOv8n network model for pre-training, and the pre-training weight with the smallest training loss value is taken for transfer learning. The data set and the pre-training weight are combined for training to obtain the optimal weight; the image to be detected is detected according to the optimal weight, and the detection result is evaluated to achieve box detection.

5. The method for detecting and counting pallet box objects based on a multi-view depth camera according to claim 1, characterized in that: In step 3, based on YOLOv8n, the C2f-RVB module is constructed by introducing RepViTBlock to replace the C2f module of YOLOv8 to reduce the number of model parameters and speed up the inference time. The idea of ​​advanced feature fusion pyramid HSFPN and efficient local attention mechanism are used to design the ELA-HSFPN structure to enhance the local feature extraction capability of the model. The background interference is reduced by adding rotation angle regression, and the Wisel-IoU loss function is used to balance the quality of dataset samples to further improve the accuracy and generalization ability of the algorithm.

6. The method for detecting and counting pallet box objects based on a multi-view depth camera according to claim 1, characterized in that: In step 4, the computing unit uses parallel computing technologies such as CUDA to accelerate the coordinate conversion process based on the stored camera extrinsics, synchronously converts the coordinates of the two-dimensional detection frame of the box under different perspectives into the world coordinate system, and combines the depth value in the corresponding depth image to obtain the three-dimensional coordinates of the box. In this process, a multi-threaded synchronization mechanism is set to ensure the timeliness and consistency of data processing at each perspective.

7. The method for detecting and counting pallet box objects based on a multi-view depth camera according to claim 1, characterized in that: In step 4, for the multi-view fusion link, a method based on SURF feature point extraction and matching is used to find feature points corresponding to the box between each viewpoint, and the RANSAC algorithm is combined to eliminate mismatched points to further optimize the three-dimensional positioning result of the box. At the same time, the detection frame is fine-tuned using regional overlap analysis to remove edge misjudgments caused by occlusion, reflection and other factors, thereby ensuring the accuracy of box positioning.

8. The method for detecting and counting pallet box objects based on a multi-view depth camera according to claim 1, characterized in that: In step five, for the multi-view fusion link, a method based on SURF feature point extraction and matching is used to find feature points corresponding to the box between each viewpoint, and the RANSAC algorithm is combined to eliminate mismatched points to further optimize the three-dimensional positioning result of the box. At the same time, the detection frame is fine-tuned using regional overlap analysis to remove edge misjudgments caused by occlusion, reflection and other factors, thereby ensuring the accuracy of box positioning.

Images