Special-shaped part discrimination algorithm based on 3D target detection

Abstract

The invention provides a special-shaped part discrimination algorithm based on 3D target detection, and belongs to the technical field of logistics, and the algorithm comprises the following steps: 1, installing a plurality of 3D cameras according to the installation requirements; 2, respectively obtaining wrapped RGB images and depth images of three visual angles through a camera, and aligning the RGB images and the depth images; step 3, marking package coordinates and bottom coordinates thereof, modifying a yov8 segmentation model and training; 4, reasoning is carried out, and parcel coordinates, bottom surface coordinates and calculation projection coordinates of the parcel coordinates and the bottom surface coordinates are obtained; 5, fitting the calculated projection coordinates of the bottom surface and the coordinates of the bottom surface of model regression, and performing weighted fusion to obtain the final position coordinates of the bottom surface; 6, calculating the coordinates and curvature of the bottom surface through a bottom surface reduction algorithm, and correcting the shape of the bottom surface; 7, comprehensively judging the package of the special-shaped part to obtain a conclusion whether the package is a special-shaped part or not; the bottom surface features of the parcels can be comprehensively captured, the parcel feeding accuracy is improved, and the probability of wrong sorting is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of logistics technology, and more specifically, particularly relates to a special-shaped part discrimination algorithm based on 3D target detection. Background Art

[0002] In recent years, 3D vision-based target detection technology has gradually matured and has been widely used in real life and production, such as logistics package detection and positioning. 3D target detection is the task of locating and identifying targets in three-dimensional scenes. Currently, target detection schemes are mainly divided into two categories. The first method only uses a single image as input without any additional information, but this method has low accuracy. The second method uses additional data, such as multiple views, depth maps, LIDAR point clouds, and CAD models, to obtain additional information and enhance detection. This method is widely used in the industrial field.

[0003] However, due to the influence of object height and camera viewing angle, existing 3D target detection technology can only relatively accurately locate the coordinate information of the target's upper surface. In logistics scenarios that require more precise bottom surface coordinates, there are great limitations, especially in the case of special-shaped items, such as smart logistics sorting machines or visual induction tables. If the bottom surface coordinates cannot be accurately located, the package cannot be accurately loaded to the center of the cart during loading, causing the package to easily float or fail to load.

[0004] With the increasing application of visual technology in the field of logistics, in order to achieve full automation of logistics, the existing methods lack solutions for bottom coordinate restoration and the identification of special-shaped parts. Therefore, an algorithm for identifying special-shaped parts is needed to enable the system to have more accurate positioning of the bottom surface of the package, effectively improving the controllability and accuracy of single-piece separation and visual feeding. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides an algorithm for distinguishing special-shaped parts based on 3D target detection, so as to solve the problem in the prior art that traditional target detection technology can only locate the coordinate information of the upper surface of the target, but lacks the positioning of the bottom surface coordinates. When loading the package, it will make it impossible to accurately load the package to the center of the trolley, resulting in the technical problem that the package is prone to floating or loading failure.

[0006] The purpose and effect of the special-shaped part identification algorithm based on 3D target detection of the present invention are achieved by the following specific technical means:

[0007] An algorithm for distinguishing irregular parts based on 3D target detection includes the following steps:

[0008] Step 1: Install multiple 3D cameras on the supply belt in the logistics production scene according to the installation requirements to ensure that the object to be measured is within the working distance range of the camera;

[0009] Step 2: Obtain the package RGB images and depth images of the three perspectives through the camera, and align them using the alignment algorithm;

[0010] Step 3: Annotate the parcel coordinates and bottom coordinates of the data aligned above, modify the yolov8 segmentation model to adapt it to RGBD input and add a detection head, perform specified training on the modified segmentation model to obtain the yolov8-RGBD segmentation model;

[0011] Step 4: Perform reasoning to obtain the package coordinates, bottom coordinates and their calculated projection coordinates;

[0012] Step 5: Fit the calculated projection coordinates of the bottom surface and the bottom surface coordinates regressed by the model, perform weighted fusion, correct the bottom surface shape, and obtain the final bottom surface position coordinates;

[0013] Step 6: Using the two sets of data from the side view, calculate the bottom coordinates and bottom curvature using the bottom restoration algorithm;

[0014] Step 7: Make a comprehensive judgment on the special-shaped package to determine whether it is a special-shaped package.

[0015] In a preferred embodiment, in step one, the installation requirements include: there are three 3D cameras, one of which is mounted directly above the center of the supply belt, and the other two 3D cameras are mounted on the left and right sides of the belt, 50 cm above the belt surface and parallel to the belt surface.

[0016] In a preferred embodiment, in step three, the specified training includes using 20% ​​of the aligned data as a test set and the remaining 80% as a training set, and using the training set to train the modified segmentation model to obtain the yolov8-RGBD segmentation model that can output the package coordinates and its bottom coordinates.

[0017] In a preferred embodiment, in step 4, the reasoning process includes:

[0018] Repeat step 2, input the top view data into the yolov8-RGBD segmentation model to obtain the package coordinates and the bottom coordinates;

[0019] According to the object height information in the depth map of the viewing angle data directly above, the package coordinates obtained by the model are corrected by transmission transformation to obtain the calculated projection coordinates wrapped on the bottom surface.

[0020] In a preferred embodiment, in step six, the bottom surface restoration algorithm includes aligning the object according to the positional relationship between the side camera and the top camera, and then correcting the bottom surface shape based on the bottom surface position coordinates and the depth information combined with the IPM image inverse perspective algorithm.

[0021] In a preferred embodiment, in step seven, the comprehensive determination includes determining the packaging of the special-shaped part according to the bottom surface shape, bottom surface curvature, and packaging aspect ratio information.

[0022] In a preferred embodiment, the determination requirements include that the intersection ratio of the bottom shape and the surface shape is greater than 0.5, the curvature of the bottom curvature is required to be less than 1 / 50 per centimeter, and the aspect ratio of the package is less than 50.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1. Through the setting of multiple 3D cameras, combined with depth information and RGBD image processing technology, the position and shape information of the bottom surface of the package can be obtained. In the logistics production scenario, the traditional target detection technology can only obtain the coordinate information of the upper surface of the package, but cannot perceive the bottom surface features, resulting in offset or failure during automatic loading. However, this algorithm can fully capture the bottom surface features of the package through camera data collection from different perspectives, specific model training and bottom surface restoration algorithm, improve the accuracy of loading and reduce the probability of incorrect sorting.

[0025] 2. The complementary relationship between the two bottom surface position branches (coordinate calculation branch and coordinate regression branch) is realized, which can effectively reduce the overall prediction error; combining multiple bottom surface position clues, the bottom surface position estimation is formulated as a collection of multiple bottom surface position predictions, which can effectively correct the offset of the bottom surface position information. In practical applications, by integrating multiple bottom surface position clues and weightedly fusing them, the algorithm can more accurately correct the offset in the bottom surface information, thereby improving the controllability and accuracy of the system.

[0026] 3. The bottom surface restoration algorithm can make full use of multi-perspective depth information to better restore the bottom surface information of the object. Based on the package coordinates, according to the depth information, combined with the IPM image inverse perspective algorithm, the bottom surface shape is corrected. It can accurately restore the bottom surface shape in various complex situations, improve the accuracy of package shape recognition, and can more accurately identify abnormal packages, thereby effectively avoiding problems in sorting, loading and other links.

[0027] 4. At the model level, this algorithm uses yolov8 as the base model and customizes it for RGBD input, so that it can process depth maps at the same time and add detection heads to achieve end-to-end output of package coordinates and bottom surface prediction coordinates. Compared with existing 3D detection algorithms, this algorithm is faster and more convenient to deploy. At the same time, while achieving better accuracy, it reduces the demand for computing resources. For logistics companies, this means that they can improve the efficiency of logistics detection without increasing a lot of hardware costs.

[0028] 5. The algorithm for determining special-shaped parts is used to determine whether a package is a special-shaped part by the bottom surface shape, bottom surface curvature, package outer contour, and package aspect ratio. When these data exceed the size of the trolley to be loaded or the user's requirements, the package is defined as a special-shaped part, and deceleration or removal is taken during the transportation process; this solves the problem of package loading caused by the lack of bottom surface coordinate positioning in traditional target detection technology. In logistics transportation, accurate judgment of special-shaped parts can avoid the floating of packages or failure of loading during transportation, thereby improving the stability and reliability of logistics transportation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of the steps of an algorithm for distinguishing special-shaped parts based on 3D target detection of the present invention.

[0030] Figure 2 It is a training block diagram of the yolov8-RGBD segmentation model in the special-shaped part discrimination algorithm based on 3D target detection of the present invention. DETAILED DESCRIPTION

[0031] In order to further understand the present invention, preferred embodiments of the present invention are described below in conjunction with the accompanying drawings and examples. However, it should be understood that these descriptions are only for further illustrating the features and advantages of the present invention, rather than limiting the claims of the present invention.

[0032] Example:

[0033] Please refer to Figure 1 and 2 As shown, the present invention provides a special-shaped part discrimination algorithm based on 3D target detection, which is characterized by comprising the following steps:

[0034] Step 1: Taking the logistics production site as the application background, multiple 3D cameras are used to collect perception data. Multiple 3D cameras need to be installed according to the installation requirements to ensure that the object to be measured is always within the working range of the camera. Usually, the camera needs to be arranged along the logistics conveyor belt. At the same time, the overlap of the field of view between the cameras needs to be considered to achieve multi-angle observation of the same object. In addition, the installation height and inclination of the camera also need to be adjusted to ensure that the acquired 3D information can fully describe the shape and position characteristics of the package;

[0035] Step 2: After the cameras are installed and deployed, multiple 3D cameras will simultaneously acquire the RGB image and depth information of the package to form three-dimensional perception data. Considering the differences in the installation positions and angles of different cameras, it is necessary to align these heterogeneous 3D data to ensure that the geometric features of the same package at different viewing angles can completely overlap, and the originally scattered multi-view 3D data will eventually be unified into the same coordinate system to form a complete 3D point cloud model;

[0036] Step 3: After acquiring the 3D data, based on the aligned RGBD data, the upper surface coordinates and bottom surface coordinates of the package are manually formed into the data set required for training; based on these annotated data, the algorithm then performs targeted transformation on the yolov8 target detection model, and adjusts the adaptability of the original yolov8 backbone network to RGBD input, so that it can process color images and depth information at the same time; at the same time, the setting of the detection head is added to predict the upper surface coordinates and bottom surface coordinates of the package. This customized network structure ensures that the model can make full use of the rich information in the 3D perception data. Finally, based on the aforementioned annotated data set, the optimized yolov8-RGBD model is trained in a targeted manner so that it can accurately extract the location coordinates and bottom surface features of the package; compared with the simple use of 2D images for detection, this method of integrating 3D information not only improves the detection accuracy, but also gives the model a stronger generalization ability, and can cope with various complex package forms;

[0037] Step 4: By running the optimized yolov8-RGBD model, it can quickly and accurately obtain the upper surface coordinates and bottom surface coordinates of the package. In logistics production scenarios, traditional target detection can often only obtain the position information of the upper surface of the package, but cannot perceive its bottom surface features, resulting in errors caused by package offset or unclear bottom features during automatic loading. This algorithm uses RGBD data collected by a multi-view 3D camera, and through a deep learning model, it can fully capture the upper surface coordinates and bottom surface information of the package; the inference module is used to obtain the coordinates and bottom surface features of the package, and then its projection coordinates on the plane are calculated based on this information, which improves the accuracy of loading and reduces the probability of incorrect sorting.

[0038] Step 5: With the upper surface coordinates and bottom surface information of the package obtained in the above reasoning stage, the calculated projection coordinates are compared and weightedly fused with the bottom surface coordinates predicted by model regression. This fusion of multi-source information can not only make full use of the advantages of different algorithms, but also improve the reliability and accuracy of the final result; Next, the algorithm will perform targeted shape correction based on the fused bottom surface feature information. Specifically, it will combine the geometric features of the bottom surface and further optimize and correct the initially obtained bottom surface position coordinates through modeling and fitting methods; This modeling and correction of the bottom surface shape helps to improve the automatic recognition and sorting accuracy of special-shaped parts, because for many special-shaped packages, relying solely on the upper surface information is far from enough. Only by fully perceiving and restoring its bottom surface features can the degree of special shape of the package be more accurately judged, thereby better realizing automatic sorting; By fusing multi-source bottom surface information and combining geometric modeling and other methods, the bottom surface position coordinates are finally obtained;

[0039] Step 6: In order to further improve the perception accuracy of bottom surface features, another set of data from the side view is introduced for assistance. Through the bottom surface restoration algorithm, the three-dimensional shape information of the bottom surface of the package can be more comprehensively captured and the bottom surface curvature can be calculated. The restoration algorithm will be based on the package coordinates, combined with the depth information of multiple viewpoints, and use IPM image inverse perspective and other technologies to correct and optimize the initially acquired bottom surface coordinates. It can not only accurately restore the bottom surface shape under various complex situations, but also further improve the cognitive accuracy of the overall geometric features of the package. For many packages with special shapes, it is far from enough to rely solely on the upper surface information. Only by fully perceiving and restoring its bottom surface features can the algorithm more accurately judge the degree of the package's special shape, thereby effectively avoiding problems in sorting, loading and other links;

[0040] Step 7: Comprehensively judge the parcel of special-shaped parts and get the conclusion whether it is a special-shaped part; combine the bottom surface shape, curvature and the aspect ratio of the parcel and other indicators to comprehensively identify the special-shaped parts; this method of judging special-shaped parts based on the bottom surface geometric features is different from the traditional method that only relies on target detection technology. In the past, due to the lack of accurate perception of the bottom surface of the parcel, the problem of parcel floating or loading failure often occurred during the automatic loading process; and through the comprehensive 3D modeling of the algorithm, not only can the bottom surface shape of the parcel be accurately captured, but also its aspect ratio and other features can be deeply analyzed to determine whether it is a special-shaped part; specifically, the algorithm will require that the intersection ratio between the bottom surface shape and the surface shape of the parcel is greater than 0.5, the bottom curvature is less than 1 / 50 per centimeter, and the aspect ratio is less than 50. When any of the indicators exceeds these thresholds, it will be defined as a special-shaped part; such a judgment standard not only ensures the stability and reliability in logistics and transportation, avoids the problem of floating and loading failure, but also provides a reliable foundation for subsequent automatic sorting and loading.

[0041] The installation requirements in step one include: deploying three high-precision 3D cameras, one of which is installed above the center of the supply belt, and the other two are located on the left and right sides of the belt, about 50 cm above the belt surface and parallel to the belt surface; this three-point layout can not only fully cover the target objects on the belt, but also obtain their three-dimensional information from different angles, providing a reliable data basis for subsequent 3D reconstruction and analysis; the reason for adopting this special three-camera configuration is that compared with the perception of a single perspective, multi-angle 3D data acquisition can greatly improve the algorithm's ability to recognize the shape of the package. On the one hand, the central camera can obtain the front information of the package, including length and width size data; and the cameras on the left and right sides can supplement the depth information of the side of the package, providing a basis for subsequent bottom reconstruction. On the other hand, the data fusion of multiple perspectives is also conducive to reducing the impact of factors such as occlusion on perception accuracy and improving the overall 3D perception effect.

[0042] In step three, the specified training includes dividing 20% ​​of the aligned and corrected 3D data samples as a test set and the remaining 80% as a training set, using the training set to train the modified segmentation model to obtain the yolov8-RGBD segmentation model that can output the coordinates of the package and its bottom surface coordinates; using the yolov8 model as a base, and customizing the RGBD input so that it can simultaneously process image data containing depth information; in addition, an additional detection head is added to the model structure so that it can not only output the overall coordinate position of the package, but also predict the coordinates of the bottom surface of the package, so as to achieve end-to-end output of the package coordinates and the bottom surface predicted coordinates, thereby improving the perception efficiency of the algorithm; compared with the existing 3D detection-based solution, this improved YOLOv8-RGBD segmentation model has a faster reasoning speed and lower demand for computing resources, which means that efficient deployment in logistics scenarios can be achieved without increasing a lot of hardware costs; at the same time, while achieving better accuracy, the demand for computing resources is reduced, which means that for logistics companies, the efficiency of logistics detection can be improved without increasing a lot of hardware costs.

[0043] In step 4, the reasoning process includes: repeating the data preprocessing steps mentioned above, inputting the 3D data from the top view into the previously trained YOLOv8-RGBD segmentation model. Through this operation, the model can not only output the overall coordinate position of the package in space, but also predict the coordinates of the bottom surface of the package. In order to further improve the accuracy of coordinate prediction, the algorithm will also use the height information of the package in the depth map from the top view to perform transmission transformation correction on the coordinate results given by the model. By analyzing the elevation data of the package in the depth image, its three-dimensional height information in the actual space can be obtained. Combined with this data, the algorithm can convert the package coordinates predicted by the model into its calculated projection coordinates on the ground plane, which can further improve the accuracy of the prediction results. In practical applications, by integrating multiple bottom surface position clues and weighted fusion them, the algorithm can more accurately correct the offset in the bottom surface information, thereby improving the controllability and accuracy of the system.

[0044] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An algorithm for distinguishing irregular parts based on 3D object detection, characterized by: The following steps are included: Step 1: Install multiple 3D cameras on the supply belt in the logistics production scene according to the installation requirements to ensure that the object to be measured is within the working distance range of the camera; Step 2: Obtain the package RGB images and depth images of the three perspectives through the camera, and align them using the alignment algorithm; Step 3: Annotate the parcel coordinates and bottom coordinates of the data aligned above, modify the yolov8 segmentation model to adapt it to RGBD input and add a detection head, perform specified training on the modified segmentation model to obtain the yolov8-RGBD segmentation model; Step 4: Perform inference calculation to obtain the package coordinates, bottom coordinates and their calculated projection coordinates; Step 5: Fit the calculated projection coordinates of the bottom surface and the bottom surface coordinates regressed by the model, perform weighted fusion, correct the bottom surface shape, and obtain the final bottom surface position coordinates; Step 6: Using the two sets of data from the side view, calculate the bottom coordinates and bottom curvature using the bottom restoration algorithm; Step 7: Make a comprehensive judgment on the special-shaped package to determine whether it is a special-shaped package.

2. According to claim 1, the algorithm for distinguishing irregular parts based on 3D target detection is characterized in that: In step 1, the installation requirements include: there are three 3D cameras, one of which is mounted directly above the center of the supply belt, and the other two 3D cameras are mounted on the left and right sides of the belt, 50 cm above the belt surface and parallel to the belt surface.

3. The algorithm for distinguishing irregular parts based on 3D target detection according to claim 1 is characterized in that: In step three, the specified training includes using 20% ​​of the aligned data as a test set and the remaining 80% as a training set, and using the training set to train the modified segmentation model to obtain the yolov8-RGBD segmentation model that can output the package coordinates and its bottom coordinates.

4. The algorithm for distinguishing irregular parts based on 3D target detection according to claim 1 is characterized in that: In step 4, the inference calculation includes: Repeat step 2, input the top view data into the yolov8-RGBD segmentation model to obtain the package coordinates and the bottom coordinates; According to the object height information in the depth map of the viewing angle data directly above, the package coordinates obtained by the model are corrected by transmission transformation to obtain the calculated projection coordinates wrapped on the bottom surface.

5. The algorithm for distinguishing irregular parts based on 3D target detection according to claim 1 is characterized in that: In step six, the bottom surface restoration algorithm includes aligning the object according to the positional relationship between the side camera and the top camera, and then correcting the bottom surface shape based on the bottom surface position coordinates, according to the depth information, and combining the IPM image inverse perspective algorithm.

6. The algorithm for distinguishing irregular parts based on 3D target detection according to claim 1, characterized in that: In step seven, the comprehensive determination includes determining the packaging of the special-shaped part according to the bottom surface shape, bottom surface curvature, and packaging aspect ratio information.

7. The algorithm for distinguishing irregular parts based on 3D target detection according to claim 6 is characterized in that: The judgment requirements include that the intersection ratio of the bottom shape and the surface shape is greater than 0.5, the curvature of the bottom curvature is required to be less than 1 / 50 per centimeter, and the aspect ratio of the package is less than 50.

Images