Automatic counting system and method for different products on conveyor belt

Abstract

The invention discloses a counting system for products on a conveying belt, and particularly relates to an automatic counting system and method for different products on the conveying belt. The system comprises a visual sensor for acquiring 2D image data; the structured light sensor is used for collecting 3D point cloud data; a deep learning neural network algorithm is adopted to carry out target detection and classification on collected 2D image data and 3D point cloud data, a 2D image and 3D point cloud multi-modal registration algorithm is adopted to carry out accurate registration on the 2D image data and the 3D point cloud data, point cloud surface reconstruction and point cloud segmentation are carried out on the 3D point cloud data, the point cloud volume of a corresponding product is calculated, and the 3D point cloud volume of the corresponding product is calculated. And the calculation module is used for matching and counting different product volumes. The technical problem of accurately counting different products on the conveying belt is solved. According to the invention, unused products on the conveyor belt can be counted, whether the products have obvious volume defects or not can be distinguished, and qualified products and unqualified products can be counted separately.

Description

Technical Field

[0001] The invention relates to a counting system for products on a conveyor belt, in particular to an automatic counting system for different products on a conveyor belt, and an automatic counting method for different products on a conveyor belt. Background Art

[0002] In modern industrial production, conveyor belts are widely used in assembly line operations, especially in mass production environments, to transport various products. With the continuous improvement of production efficiency, how to accurately and quickly count the number of products on the conveyor belt has become one of the key needs for enterprises to improve their production management level. The traditional manual counting method is not only time-consuming and labor-intensive, but also prone to errors and cannot meet the needs of efficient production. Therefore, it is of great practical significance and application value to develop an automated counting method for products on the conveyor belt to improve the counting accuracy and production efficiency.

[0003] In recent years, with the development of image processing technology and sensor technology, automatic counting systems / methods based on these technologies have gradually become a research hotspot. For example, the existing patent documents related to automatic counting systems / methods include: a counting method and system for conveyor belt conveyed items recorded in the published Chinese invention patent application document with application number CN202211143455.X; and an intelligent method for counting and sorting eggs on a conveyor belt recorded in the published Chinese invention patent application document with application number CN202310946274.9.

[0004] However, the existing systems and methods still have the following technical defects. Specifically, the existing systems and methods usually use a single visual sensor or photoelectric sensor for counting, which is prone to product counting errors, or can only be used for counting a single product (such as eggs). If the product or multiple products are replaced, it will be impossible to count, and it is also impossible to determine whether the product has obvious volume defects.

[0005] Therefore, the applicant proposes the present invention. Summary of the invention

[0006] The purpose of the present invention is to solve the above-mentioned deficiencies in the prior art and to provide an automatic counting system and method for different products on a conveyor belt.

[0007] In order to achieve the above object, the present invention provides an automatic counting system for different products on a conveyor belt, comprising: Vision sensors that collect 2D image data; Structured light sensors that collect 3D point cloud data; In addition, a 2D deep neural network algorithm is used to perform target detection and classification on the collected 2D image data, and a 3D deep neural network algorithm is used to perform target detection on the collected 3D point cloud data. A 2D image and 3D point cloud multimodal registration algorithm is used to accurately align the 2D image data with the 3D point cloud data, and invalid detection frames are further removed by the overlap between the 2D and 3D detection frames. The 3D point cloud data is then reconstructed and segmented, and the point cloud volume of the corresponding product is calculated. A calculation module is also provided to match and count the volumes of different products.

[0008] In the above-mentioned automatic counting system for different products on a conveyor belt, the calculation module preferably determines qualified and unqualified products with volume defects in each category of products by setting a volume threshold value, and counts them.

[0009] The above-mentioned computing module is further preferably selected to include: an artificial intelligence 2D detection and classification neural network for detecting 2D image targets and their classification; an artificial intelligence 3D detection neural network for detecting 3D point cloud targets; a registration algorithm for multimodal registration of 2D images and 3D point clouds; and a post-processing algorithm for post-matching data preprocessing, point cloud surface reconstruction, point cloud segmentation, segmentation volume calculation, counting of each category of products through classification label information, setting a threshold value according to the volume of each product, setting the products within the threshold as qualified products, and setting the products outside the threshold as unqualified products.

[0010] Furthermore, the above-mentioned automatic counting system for different products on a conveyor belt further includes: Display module for data visualization.

[0011] In addition, the present invention provides an automatic counting method for different products on a conveyor belt, which comprises the following steps: Step 1 of collecting 2D image data using a visual sensor and collecting 3D point cloud data using a structured light sensor; Use a 2D deep neural network algorithm to perform target detection and classification on the collected 2D image data, use a 3D deep neural network algorithm to perform target detection on the collected 3D point cloud data, and use a 2D image and 3D point cloud multimodal registration algorithm to accurately align the 2D image data with the 3D point cloud data, further remove invalid detection frames by comparing the overlap between the 2D and 3D detection frames, reconstruct the point cloud surface of the 3D point cloud data, segment the point cloud, calculate the point cloud volume of the corresponding product, and match and count the volumes of different products. Step 2.

[0012] In the above step 2, it is also preferred to set a volume threshold to determine qualified products and unqualified products with volume defects in each category of products, and count them.

[0013] Compared with the prior art, the automatic counting system and method of different products on a conveyor belt obtained by the present invention has the following technical effects: The present invention can accurately count different products on the conveyor belt in different time periods, can also synchronously and accurately count multiple products on the conveyor belt in the same time period, and can also calculate the number of qualified products and unqualified products with obvious defects in product volume.

[0014] That is to say, if there are multiple products such as A, B, C being transported on the conveyor belt, and due to process or management problems, these products are likely to be unqualified. Through the patent of this invention, we can count multiple products such as A, B, C, etc. separately, and also calculate the number of qualified products and unqualified products. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Schematic diagram of an automated counting system for different products on a conveyor belt. DETAILED DESCRIPTION

[0016] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0017] As an implementation mode of the present invention, this embodiment provides an automatic counting system and method for different products on a conveyor belt.

[0018] This system mainly involves three hardware modules, namely a visual sensor for collecting 2D image data, a structured light sensor for collecting 3D point cloud data, a computing module including a graphics card, and a display module.

[0019] Among them, the calculation module mainly involves four algorithm modules, namely, an artificial intelligence 2D detection and classification neural network for detecting 2D image targets and their classification, an artificial intelligence 3D detection neural network for detecting 3D point cloud targets, a registration algorithm for multimodal registration of 2D images and 3D point clouds, and a data result post-processing algorithm, including post-matching data preprocessing, point cloud surface reconstruction, point cloud segmentation, segmentation volume calculation, counting of each category of products through classification label information, setting a threshold according to the volume of each product, setting the products within the threshold as qualified products, and setting the products outside the threshold as unqualified products.

[0020] The overall algorithm module is as follows: 1. Deep learning 2D image data detection and classification neural network: This module uses the current advanced deep learning neural network algorithm to detect and classify the collected 2D image data. It mainly includes the following steps: Data preprocessing: First, the original 2D image data collected from the visual sensor is preprocessed. This process includes image scaling, grayscale processing, image enhancement (such as histogram equalization, contrast enhancement, etc.), and noise removal to ensure that the input data meets the input requirements of the neural network and can improve the accuracy of detection and classification.

[0021] Feature extraction: Next, the preprocessed image data is subjected to feature extraction through a convolutional neural network (CNN). CNN gradually extracts low-level and high-level features from the image through multiple layers of convolution, pooling, and activation function operations, and ultimately generates a compact feature representation that contains key information about the target object.

[0022] Object detection: After feature extraction, an object detection algorithm (such as Faster R-CNN, YOLO, or SSD) is used to detect objects in the image. This process generates multiple bounding boxes and locates and scores the objects in each box to determine the location of the target object.

[0023] Classification: The detected targets are classified through the fully connected layer and the Softmax classifier. The neural network will predict the target category in the input image based on the pre-trained model weights and output the classification result.

[0024] Post-processing: After obtaining the preliminary classification results, non-maximum suppression (NMS) is applied to remove redundant bounding boxes, and the bounding box with the highest confidence is retained as the final target detection result. At the same time, the corresponding classification label is generated according to the detection result and passed to the subsequent modules for further multimodal fusion and processing.

[0025] During the training process, the neural network used a large number of annotated image data sets and improved the detection and classification accuracy in specific scenarios through transfer learning technology. Through continuous iterative training and optimization, this module can effectively identify a variety of target objects in images and accurately classify them.

[0026] 2. Deep learning 3D point cloud data target detection neural network: This module involves using deep learning technology to detect objects in 3D point cloud data. The following is a detailed description of the module: Data preprocessing: Input data: 3D point cloud data acquired from a structured light sensor.

[0027] Noise reduction: Use filtering algorithms to remove noise from point clouds to improve data quality.

[0028] Downsampling: Downsample the point cloud to reduce the amount of computation while retaining key features.

[0029] Neural Network Model: Network architecture: Use PointNet, PointNet++ or other advanced 3D convolutional neural networks.

[0030] Feature extraction: The network extracts spatial features from the point cloud through convolutional layers.

[0031] Feature fusion: Fusion of features at different scales to enhance detection capabilities.

[0032] Training and Optimization: Loss function: Use a combination of loss functions such as cross entropy loss and IoU loss.

[0033] Data enhancement: Enhance point cloud data through rotation, scaling, translation, etc. to improve the generalization ability of the model.

[0034] Optimization algorithm: Use Adam or SGD for model training and adjust the learning rate to speed up convergence.

[0035] Object Detection: Segmentation and classification: Segment the point cloud and assign a category label to each segmented area.

[0036] Bounding Box Generation: Generate 3D bounding boxes to accurately locate objects.

[0037] Confidence score: Provides a confidence score for each detection result to assess the reliability of the detection.

[0038] Output: Detection results: Output detection results including target category, location and confidence.

[0039] Visualization: Display the test results through the display device and conduct further analysis.

[0040] Model deployment: Optimization and acceleration: Use model pruning and quantization techniques to optimize the model to adapt to the hardware resources of the computing module.

[0041] Real-time detection: Ensure real-time detection on the microcontroller computing module.

[0042] 3. Multimodal registration algorithm of 2D image and 3D point cloud: The main goal of this module is to accurately align 2D image data with 3D point cloud data to achieve more accurate target detection and classification. The specific steps are as follows: Feature extraction: To extract key feature points from 2D images, algorithms such as SIFT (Scale Invariant Feature Transform) or ORB (Orbital Rapid Feature Points) can be used.

[0043] Extract geometric features such as normals and curvature from 3D point clouds.

[0044] Feature Matching: Use a matching algorithm such as FLANN or KNN to match 2D image features with 3D point cloud features.

[0045] The RANSAC (Random Sample Consensus) algorithm is used to remove incorrect matches and retain the best matching pairs.

[0046] Initial registration: Using the matched eigenpairs, the initial transformation matrix is ​​computed. This typically involves estimating the rotation and translation matrices.

[0047] Fine registration: The ICP (Iterative Closest Point) algorithm or its variants are used to further optimize the registration accuracy.

[0048] Through continuous iteration, the distance between the 2D image feature points and the 3D point cloud is minimized.

[0049] Verification of registration results: Calculate the registration error and verify it with visualization tools (e.g. overlay of 2D and 3D data).

[0050] If the error exceeds the set threshold, feature matching and registration are performed again.

[0051] Multimodal data fusion: The registered 2D and 3D data are fused to generate comprehensive target recognition results.

[0052] Use the fused data for subsequent object classification and detection.

[0053] 4. Data post-processing, point cloud surface reconstruction, point cloud segmentation, segmentation volume calculation, threshold setting and count calculation: This module is responsible for a series of post-processing steps on the registered data to facilitate further analysis and decision making. The specific steps are as follows: Remove invalid target frames: Match the 2D target box with the 3D target box and calculate the overlap IOU. The target box with IOU lower than 90% is retained as the largest target box, and the target detection accuracy is submitted.

[0054] Point cloud surface reconstruction: Use algorithms such as Poisson Surface Reconstruction or Ball-Pivoting to convert discrete point cloud data into a continuous 3D surface.

[0055] The reconstructed surface is used to more clearly represent the shape and structure of the object.

[0056] Point cloud segmentation: Apply a clustering algorithm (such as DBSCAN or K-Means) to segment the point cloud into different parts.

[0057] The goal is to distinguish different objects or structures in the point cloud.

[0058] Segment volume calculation: Calculate the volume of each segmented part.

[0059] Use volume computation methods such as voxelization or convex hull for estimation.

[0060] Threshold setting and count calculation: Based on prior knowledge or experimental data, thresholds for qualified and unqualified volumes are set for each category.

[0061] Count the products in each category based on the category label information.

[0062] The results are classified into categories: products with volumes within the threshold range are qualified products, and products with volumes outside the range are unqualified products.

[0063] Data Visualization and Report Generation: Generate analysis reports, including registration effects, segmentation results, and statistics of qualified and unqualified products.

[0064] Use a graphical interface to display the results, supporting further manual inspection and verification.

[0065] The overall algorithm framework and process are as follows Figure 1 shown.

[0066] The present invention is not limited to the above-mentioned optimal implementation mode. Anyone can derive other various forms of products under the inspiration of the present invention. However, no matter what changes are made in the shape or structure, all technical solutions that are the same or similar to those of the present application fall within the protection scope of the present invention.

Claims

1. An automatic counting system for different products on a conveyor belt, characterized by include: Vision sensors that collect 2D image data; Structured light sensors that collect 3D point cloud data; In addition, a 2D deep neural network algorithm is used to perform target detection and classification on the collected 2D image data, and a 3D deep neural network algorithm is used to perform target detection on the collected 3D point cloud data. A 2D image and 3D point cloud multimodal registration algorithm is used to accurately align the 2D image data with the 3D point cloud data, and invalid detection frames are further removed by the overlap between the 2D and 3D detection frames. The 3D point cloud data is then reconstructed and segmented, and the point cloud volume of the corresponding product is calculated. A calculation module is also provided to match and count the volumes of different products.

2. An automatic counting system for different products on a conveyor belt according to claim 1, characterized in that: The calculation module also determines qualified products and unqualified products with volume defects in each category of products by setting volume thresholds, and counts them.

3. An automatic counting system for different products on a conveyor belt according to claim 2, characterized in that The computing module mainly includes: an artificial intelligence 2D detection and classification neural network for detecting 2D image targets and their classification; an artificial intelligence 3D detection neural network for detecting 3D point cloud targets; a registration algorithm for multimodal registration of 2D images and 3D point clouds; and a post-processing algorithm for post-matching data preprocessing, point cloud surface reconstruction, point cloud segmentation, segmentation volume calculation, counting of each category of products through classification label information, setting a threshold value according to the volume of each product, setting the products within the threshold as qualified products, and setting the products outside the threshold as unqualified products.

4. An automatic counting system for different products on a conveyor belt according to any one of claims 1 to 3, characterized in that Also includes: Display module for data visualization.

5. A method for automatically counting different products on a conveyor belt, characterized in that The steps include: Step 1 of collecting 2D image data using a visual sensor and collecting 3D point cloud data using a structured light sensor; Use a 2D deep neural network algorithm to perform target detection and classification on the collected 2D image data, use a 3D deep neural network algorithm to perform target detection on the collected 3D point cloud data, and use a 2D image and 3D point cloud multimodal registration algorithm to accurately align the 2D image data with the 3D point cloud data, further remove invalid detection frames by comparing the overlap between the 2D and 3D detection frames, reconstruct the point cloud surface of the 3D point cloud data, segment the point cloud, calculate the point cloud volume of the corresponding product, and match and count the volumes of different products. Step 2.

6. The method for automatically counting different products on a conveyor belt according to claim 5, characterized in that: In the above step 2, the qualified and unqualified products with volume defects in each category of products are judged and counted by setting the volume threshold.

Images