A combined inspection system for packaged irregular parts based on vision and photoelectric sensors

Abstract

The present invention discloses a combined detection system for parcel irregular parts based on vision and photoelectric sensors in the field of automated logistics sorting. When mail passes through the camera detection area, the cross photoelectric row installed on the bottom of the belt undergoes a level change. At this time, the controller calculates the bottom shape and area of ​​the parcel through the level change of the photoelectric row, and triggers the stereo camera to take pictures. The point cloud scanning and color plane image information of the obtained parcel are respectively processed using improved point cloud processing and plane image parcel detection technology, and combined with the bottom scanning information, the number of parcels and the properties of irregular parts in the detection area are comprehensively judged. The whole method includes a preprocessing module, a parcel point cloud extraction and irregular part detection module, a bottom scanning detection module, a plane image detection module, a result synthesis module, a communication and log management and a file management and visualization module. The present invention can quickly and accurately judge whether a parcel contains double or irregular parts, saving manpower.

Description

Technical Field

[0001] The present invention relates to the field of automated logistics sorting, and in particular to a combined detection system for packaged irregular-shaped parts based on vision and photoelectric sensors. Background Art

[0002] In recent years, the single-piece separation system has been successfully integrated into the sorting system. It uses visual intelligent detection and electronic control technology to position, separate, and intelligently queue packages, providing stable spacing and sequential packages for trolley supply and barcode scanning, realizing fully automatic transmission, detection, and sorting of the system.

[0003] Generally, there is a certain percentage of unqualified parcels among the parcels separated by the single-piece system, mainly including irregular pieces that cannot be put on the sorting cart or double pieces, which can easily lead to sorting errors or system failures. Double pieces are generally defined as double or multiple pieces with small spacing or overlap. Irregular pieces are generally defined as the following four situations:

[0004] (1) According to the size of the trolley, large items that exceed the size are defined as special-shaped items (generally refers to packages with a length and width exceeding 400 mm);

[0005] (2) Round and spherical mail that easily rolls over and cannot be loaded onto a cart are also a type of special-shaped items, such as cylindrical container packages and bottled objects;

[0006] (3) Mail in the form of thin rods or odd shapes can easily cause the system machinery to jam and damage the equipment. Mail such as iron rods, flowers, iron rakes, iron spoons, laundry detergent, etc. are also considered as odd-shaped items.

[0007] (4) Damaged mail and scattered small objects are also a type of irregular parts, such as damaged cartons, potatoes, stones, etc. The occurrence of duplicate and irregular parts is inevitable, which reduces the efficiency of the system sorting to a certain extent.

[0008] Therefore, it is necessary to quickly remove the double and special-shaped packages after the single pieces are separated to ensure the safe and stable operation of the system and improve the system efficiency. Summary of the invention

[0009] The technical problem to be solved by the present invention is to provide a joint detection system for package irregularities based on vision and photoelectric sensors, so as to quickly and accurately judge whether there are duplicates or irregularities in the package, and cooperate with the control system to complete the rejection action in real time, so as to avoid the influence of duplicates and irregularities on the sorting efficiency and accuracy of the system.

[0010] The object of the present invention is achieved as follows: A combined detection system for packaged irregular parts based on vision and photoelectric sensors, comprising a preprocessing module, a package three-dimensional point cloud extraction and irregular part detection module, a bottom surface scanning detection module, a plane image detection module, a result synthesis module, a communication and log management and a file management and visualization module;

[0011] Preprocessing module: The depth map and color map of the camera are acquired synchronously by triggering the shutter of the stereo camera, and the color map and the depth map are pixel-aligned to obtain the depth of each pixel. Based on the internal parameters of the color camera, the image is distorted and corrected, and then the spatial position of each pixel in the camera coordinate system is calculated to generate a color point cloud. The point cloud processing is accelerated using CUDA parallel programming technology.

[0012] Package 3D point cloud extraction and special-shaped parts detection module: Use color point cloud as input, use background removal, point cloud segmentation and package 3D modeling technology to calculate the location, size and posture of the package, and judge special-shaped parts based on the package size and surface shape;

[0013] Bottom surface scanning detection module: Use the cross photoelectric sensors on the bottom of the belt to calculate the bottom area and shape of the package by using the position spacing and time difference of high and low level changes of the photoelectric sensors and the belt speed information;

[0014] Plane image detection module: Use color image information to improve the single-stage convolutional neural network Yolo, SSD or Efficient Det / Net with good detection performance, add a classification detection network, judge the overlap, and output the number of packages;

[0015] Result synthesis module: input the results of package 3D point cloud extraction and special-shaped part detection, bottom surface scanning detection and plane image detection, and judge the number of packages by integrating the information of package 3D point cloud extraction and plane image detection, and judge whether the package is a special-shaped part by integrating the information of point cloud special-shaped part detection and bottom surface scanning;

[0016] Communication, log management, file management and visualization modules: responsible for system stable operation and fault diagnosis;

[0017] The processing method of the package three-dimensional point cloud extraction and special-shaped part detection module comprises the following steps:

[0018] S1: Input the color point cloud generated by the preprocessing module, perform background modeling on the belt surface, obtain the plane parameters of the belt surface, and preset the parameters into the detection program;

[0019] S2: According to the position, interval and size of the detected package, the ROI area of ​​the point cloud is fixed;

[0020] S3: Use the lattice method to downsample the point cloud to separate the foreground and background;

[0021] S4: For the foreground point cloud, use the method of region growing and Euclidean distance segmentation to obtain the point cloud segmentation of the package; remove outliers from each segmented point cloud; perform secondary segmentation on the projection contour of the package point cloud; if the number of segmented point clouds is 1, calculate the minimum bounding box of the package, and then perform special-shaped parts detection on the point cloud, and publish the number of point cloud segmentations, segmented color point clouds, the size, position and posture of the minimum bounding box, and the special-shaped parts detection results through ROS messages; otherwise, publish the number of point cloud segmentations and each segmented color point cloud through ROS messages;

[0022] The secondary segmentation of the projection contour of the package point cloud comprises the following steps:

[0023] S1: Based on the assumption that the package shape is rectangular, secondary detection and segmentation are performed on the point cloud with irregular projection shape;

[0024] S2: Project each segmented point cloud onto the belt plane to form a projected binary image;

[0025] S3: Perform a closing operation on the binary image; use the Canny operator to calculate the contour of the package; calculate the concave points of the polygonal contour; connect the nearest concave point pairs; use polygonal contour fitting to calculate the plane rotation angle of the minimum bounding box, then rotate the point cloud according to the calculated angle to obtain the length and width of the segmented point cloud, calculate the maximum height of the point cloud based on the belt plane parameters, and obtain the point cloud height;

[0026] The detection steps of the package 3D point cloud extraction and special-shaped parts detection module are as follows:

[0027] S1: Size judgment: Compare the length, width and height of the minimum bounding box of the parcel point cloud with the size of the trolley. If they are greater than a certain threshold, it is judged as an oversized part.

[0028] S2: Plane curvature calculation: Project the package point cloud onto the xy, yz, and zx planes to generate binary images. Use image morphology to smooth the images, extract the contours, calculate the area of ​​each point of the contours, and calculate the histogram distribution. When the curvature exceeds a certain threshold and the distribution exceeds a certain ratio, it is judged that the curvature of the package surface is large and easy to roll over, and thus it is judged as a special-shaped part.

[0029] S3: Calculation of volume occupancy: Calculate the integral volume of the wrapped point cloud and compare it with the minimum bounding box volume. When the ratio of the integral volume to the minimum bounding box volume is less than a certain ratio, it means that the point cloud is hollow or approximates to a cone or ring, and has a strange shape, so it is judged as an irregular part. The above steps are executed in sequence. If a step is judged to be an irregular part, the program terminates; conversely, if all calculation steps do not judge it as an irregular part, the program returns to a normal part.

[0030] Preferably, after the stereo camera is triggered to take a photo in the preprocessing module, a color image is first obtained for distortion correction, and then a depth image is obtained to generate a point cloud, so as to shorten the overall preprocessing time.

[0031] Preferably, the detection method of the bottom surface scanning detection module is as follows: photoelectric sensors are arranged in a row at equal intervals and perpendicular to the direction of belt movement and placed below the belt gap. When the package is located above the gap, the potential of the photoelectric sensor at the position blocked by the package will be pulled up and down, and the controller records the moment when the potential of the photoelectric sensor changes, and integrates and calculates the shape and area of ​​the bottom surface of the package based on the position spacing of the photoelectric sensors, the time difference between the high and low level changes, and the belt speed information.

[0032] Preferably, the detection method of the plane image detection module is as follows: a two-stage convolutional neural network structure is designed, in which a detection network Yolo, SSD or Efficient Det / Net with good real-time detection performance is used in the first stage, and the image is cropped in combination with the output frame of the network, and input into the second-stage dual classification network; the classifier for detecting double items uses a simple neural network with a small number of convolutional layers to classify whether the package images overlap, and if they overlap, 1 is added to the number of packages; a first-stage package detection network and a second-stage dual classification network.

[0033] Preferably, the two-stage convolutional neural network structure is accelerated using the TensorRT tool.

[0034] Preferably, the preprocessing module, package three-dimensional point cloud extraction and special-shaped parts detection module, bottom surface scanning detection module, plane image detection module, result integration module, communication and log management and file management and visualization module are integrated under the ROS framework, each module is a startup node of ROS, and inter-module communication uses ROS message subscription and distribution services.

[0035] Preferably, the comprehensive module uses the calculation results of the color plane image detection and the package point cloud extraction module, combined with the average height of the package point cloud, to increase or decrease the confidence of the point cloud segmentation result, and performs weighted summation with the confidence of the plane image detection to obtain the double judgment confidence. When its value is greater than a certain threshold, it is judged as a double piece. For the judgment of special-shaped parts, the point cloud special-shaped part detection and bottom surface scanning module information are used to judge the special-shaped parts in two steps: if the point cloud special-shaped part detection module judges it as a special-shaped part, the comprehensive module judges it as a special-shaped part; otherwise, the area obtained by the bottom surface scanning is compared with the area of ​​the upper surface of the minimum bounding box. When the ratio of the two is less than a certain ratio, it means that the bottom surface of the package occupies a small area, resulting in unstable posture, and it is also judged as a special-shaped part; finally, the comprehensive module combines the double judgment and special-shaped part detection results. If the package is judged to be a double piece or a special-shaped part, a rejection command is returned to the controller; otherwise, a pass command is returned to the controller.

[0036] Compared with the prior art, the advantages of the present invention are as follows: the present invention uses multiple sensors to obtain the planar and three-dimensional features of the package, and through specific image detection, stereo vision algorithm, and combined with GPU programming, the system can quickly and accurately determine whether the package contains double and special-shaped parts, saving manpower. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of the module composition and information interaction diagram of the present invention.

[0038] Figure 2 Schematic diagram of the processing flow of the preprocessing module.

[0039] Figure 3 Schematic diagram of the package point cloud extraction and special-shaped parts detection processing flow.

[0040] Figure 4 Schematic diagram of the planar image package detection processing flow. DETAILED DESCRIPTION

[0041] The present invention is further described below in conjunction with specific embodiments and drawings.

[0042] A combined inspection system for packaged irregular parts based on vision and photoelectric sensors, including a preprocessing module, a package 3D point cloud extraction and irregular part detection module, a bottom surface scanning detection module, a plane image detection module, a result synthesis module, a communication and log management module, and a file management and visualization module;

[0043] Preprocessing module: The depth map and color map of the camera are acquired synchronously by triggering the shutter of the stereo camera, and the color map and the depth map are pixel-aligned to obtain the depth of each pixel. Based on the internal parameters of the color camera, the image is distorted and corrected, and then the spatial position of each pixel in the camera coordinate system is calculated to generate a color point cloud. The point cloud processing is accelerated using CUDA parallel programming technology.

[0044] Package 3D point cloud extraction and special-shaped parts detection module: Use color point cloud as input, use background removal, point cloud segmentation and package 3D modeling technology to calculate the location, size and posture of the package, and judge special-shaped parts based on the package size and surface shape;

[0045] Bottom surface scanning detection module: Use the cross photoelectric sensors on the bottom of the belt to calculate the bottom area and shape of the package by using the position spacing and time difference of high and low level changes of the photoelectric sensors and the belt speed information;

[0046] Plane image detection module: Use color image information to improve the single-stage convolutional neural network Yolo, SSD or Efficient Det / Net with good detection performance, add a classification detection network, judge the overlap, and output the number of packages;

[0047] Result synthesis module: input the results of package 3D point cloud extraction and special-shaped part detection, bottom surface scanning detection and plane image detection, and judge the number of packages by integrating the information of package 3D point cloud extraction and plane image detection, and judge whether the package is a special-shaped part by integrating the information of point cloud special-shaped part detection and bottom surface scanning;

[0048] Communication, log management, file management and visualization modules: responsible for system stable operation and fault diagnosis;

[0049] The processing method of the package three-dimensional point cloud extraction and special-shaped part detection module comprises the following steps:

[0050] S1: Input the color point cloud generated by the preprocessing module, perform background modeling on the belt surface, obtain the plane parameters of the belt surface, and preset the parameters into the detection program;

[0051] S2: According to the position, interval and size of the detected package, the ROI area of ​​the point cloud is fixed;

[0052] S3: Use the lattice method to downsample the point cloud to separate the foreground and background;

[0053] S4: For the foreground point cloud, the method of region growing and Euclidean distance segmentation is used to obtain the point cloud segmentation of the package; outliers are removed from each segmented point cloud; the projection contour of the package point cloud is segmented twice; if the number of segmented point clouds is 1, the minimum bounding box of the package is calculated, and then the point cloud is detected for special-shaped parts, and the number of point cloud segmentations, segmented colored point clouds, the size, position and posture of the minimum bounding box, and the special-shaped part detection results are published by ROS messages; otherwise, the number of point cloud segmentations and each segmented colored point cloud are published by ROS messages.

[0054] The secondary segmentation of the projection contour of the package point cloud comprises the following steps:

[0055] S1: Based on the assumption that the package shape is rectangular, secondary detection and segmentation are performed on the point cloud with irregular projection shape;

[0056] S2: Project each segmented point cloud onto the belt plane to form a projected binary image;

[0057] S3: Perform a closing operation on the binary image; use the Canny operator to calculate the contour of the package; calculate the concave points of the polygonal contour; connect the nearest concave point pairs; use polygonal contour fitting to calculate the plane rotation angle of the minimum bounding box, then rotate the point cloud according to the calculated angle to obtain the length and width of the segmented point cloud, calculate the maximum height of the point cloud based on the belt plane parameters, and obtain the point cloud height;

[0058] The detection steps of the package 3D point cloud extraction and special-shaped parts detection module are as follows:

[0059] S1: Size judgment: Compare the length, width and height of the minimum bounding box of the parcel point cloud with the size of the trolley. If they are greater than a certain threshold, it is judged as an oversized part.

[0060] S2: Plane curvature calculation: Project the package point cloud onto the xy, yz, and zx planes to generate binary images. Use image morphology to smooth the images, extract the contours, calculate the area of ​​each point of the contours, and calculate the histogram distribution. When the curvature exceeds a certain threshold and the distribution exceeds a certain ratio, it is judged that the curvature of the package surface is large and easy to roll over, and thus it is judged as a special-shaped part.

[0061] S3: Volume occupancy calculation: The integral volume of the wrapped point cloud is calculated and compared with the minimum bounding box volume. When the ratio of the integral volume to the minimum bounding box volume is less than a certain ratio, it means that the point cloud is hollow or approximates to a cone or ring, and the shape is strange, so it is judged as a special-shaped part. The above steps are executed in sequence. If a step is judged as a special-shaped part, the program terminates; conversely, if all calculation steps do not judge a special-shaped part, the program returns to a normal part.

[0062] Preferably, after the stereo camera is triggered to take a photo in the preprocessing module, a color image is first obtained for distortion correction, and then a depth image is obtained to generate a point cloud, so as to shorten the overall preprocessing time.

[0063] Preferably, the detection method of the bottom surface scanning detection module is as follows: photoelectric sensors are arranged in a row at equal intervals and perpendicular to the direction of belt movement and placed below the belt gap. When the package is located above the gap, the potential of the photoelectric sensor at the position blocked by the package will be pulled up and down. The controller calculates the shape and area of ​​the bottom surface of the package by integration based on the position spacing of the photoelectric sensors, the time difference between the high and low level changes, and the belt speed information when the potential of the photoelectric sensor changes.

[0064] Preferably, the detection method of the plane image detection module is as follows: a two-stage convolutional neural network structure is designed, in which a detection network Yolo, SSD or Efficient Det / Net with good real-time detection performance is used in the first stage, and the image is cropped in combination with the output frame of the network, and input into the second-stage dual classification network; the classifier for detecting double items uses a simple neural network with a small number of convolutional layers to classify whether the package images overlap, and if they overlap, 1 is added to the number of packages; a first-stage package detection network and a second-stage dual classification network.

[0065] Preferably, the two-stage convolutional neural network structure is accelerated using the TensorRT tool.

[0066] Preferably, the preprocessing module, package three-dimensional point cloud extraction and special-shaped parts detection module, bottom surface scanning detection module, plane image detection module, result integration module, communication and log management and file management and visualization module are integrated under the ROS framework, each module is a startup node of ROS, and inter-module communication uses ROS message subscription and distribution services.

[0067] Preferably, the comprehensive module uses the calculation results of the color plane image detection and the package point cloud extraction module, combined with the average height of the package point cloud, to increase or decrease the confidence of the point cloud segmentation result, and performs weighted summation with the confidence of the plane image detection to obtain the double judgment confidence. When its value is greater than a certain threshold, it is judged as a double piece. For the judgment of special-shaped parts, the point cloud special-shaped part detection and bottom surface scanning module information are used to judge the special-shaped parts in two steps: if the point cloud special-shaped part detection module judges it as a special-shaped part, the comprehensive module judges it as a special-shaped part; otherwise, the area obtained by the bottom surface scanning is compared with the area of ​​the upper surface of the minimum bounding box. When the ratio of the two is less than a certain ratio, it means that the bottom surface of the package occupies a small area, resulting in unstable posture, and it is also judged as a special-shaped part; finally, the comprehensive module combines the double judgment and special-shaped part detection results. If the package is judged to be a double piece or a special-shaped part, a rejection command is returned to the controller; otherwise, a pass command is returned to the controller.

[0068] The present invention is not limited to the above-mentioned embodiments. On the basis of the technical solution disclosed in the present invention, technicians in this field can make some substitutions and deformations to some technical features therein according to the disclosed technical content without creative labor, and these substitutions and deformations are all within the protection scope of the present invention.

Claims

1. A combined detection system for packaged irregular parts based on vision and photoelectric sensors, characterized in that: It includes pre-processing module, package 3D point cloud extraction and special-shaped parts detection module, bottom surface scanning detection module, plane image detection module, result synthesis module, communication and log management and file management and visualization module; Preprocessing module: The depth map and color map of the camera are acquired synchronously by triggering the shutter of the stereo camera, and the color map and the depth map are pixel-aligned to obtain the depth of each pixel. Based on the internal parameters of the color camera, the image is distorted and corrected, and then the spatial position of each pixel in the camera coordinate system is calculated to generate a color point cloud. The point cloud processing is accelerated using CUDA parallel programming technology. Package 3D point cloud extraction and special-shaped parts detection module: Use color point cloud as input, use background removal, point cloud segmentation and package 3D modeling technology to calculate the location, size and posture of the package, and judge special-shaped parts based on the package size and surface shape; Bottom surface scanning detection module: Use the cross photoelectric sensors on the bottom of the belt to calculate the bottom area and shape of the package by using the position spacing and time difference of high and low level changes of the photoelectric sensors and the belt speed information; Plane image detection module: Use color image information to improve the single-stage convolutional neural network Yolo, SSD or Efficient Det / Net with good detection performance, add a classification detection network, judge the overlap, and output the number of packages; Result synthesis module: input the results of package 3D point cloud extraction and special-shaped part detection, bottom surface scanning detection and plane image detection, and judge the number of packages by integrating the information of package 3D point cloud extraction and plane image detection, and judge whether the package is a special-shaped part by integrating the information of point cloud special-shaped part detection and bottom surface scanning; Communication, log management, file management and visualization modules: responsible for system stable operation and fault diagnosis; The processing method of the package three-dimensional point cloud extraction and special-shaped part detection module comprises the following steps: S1: Input the color point cloud generated by the preprocessing module, perform background modeling on the belt surface, obtain the plane parameters of the belt surface, and preset the parameters into the detection program; S2: According to the position, interval and size of the detected package, the ROI area of ​​the point cloud is fixed; S3: Use the lattice method to downsample the point cloud to separate the foreground and background; S4: For the foreground point cloud, use the method of region growing and Euclidean distance segmentation to obtain the point cloud segmentation of the package; remove outliers from each segmented point cloud; perform secondary segmentation on the projection contour of the package point cloud; if the number of segmented point clouds is 1, calculate the minimum bounding box of the package, and then perform special-shaped parts detection on the point cloud, and publish the number of point cloud segmentations, segmented color point clouds, the size, position and posture of the minimum bounding box, and the special-shaped parts detection results through ROS messages; otherwise, publish the number of point cloud segmentations and each segmented color point cloud through ROS messages; The secondary segmentation of the projection contour of the package point cloud comprises the following steps: S1: Based on the assumption that the package shape is rectangular, secondary detection and segmentation are performed on the point cloud with irregular projection shape; S2: Project each segmented point cloud onto the belt plane to form a projected binary image; S3: Perform a closing operation on the binary image; use the Canny operator to calculate the contour of the package; calculate the concave points of the polygonal contour; connect the nearest concave point pairs; calculate the color and depth similarity of the point cloud near the connecting line, and use the connecting line to segment the point cloud with large similarity differences; The detection steps of the package 3D point cloud extraction and special-shaped parts detection module are as follows: S1: Determine whether it is an oversized part based on the minimum bounding box size of the segmented point cloud; S2: Project the point cloud onto the xy, yz, and zx planes to generate a binary projection image, extract the point cloud contour, calculate the curvature of each point, and determine the special-shaped part based on the distribution of the curvature; S3: Integrate and calculate the volume of the package, and compare the integrated volume with the bounding box volume to determine the irregular-shaped part.

2. According to claim 1, a combined detection system for packaged irregular parts based on vision and photoelectric sensors is characterized in that: After the stereo camera is triggered to take a photo in the preprocessing module, a color image is first obtained for distortion correction, and then a depth image is obtained to generate a point cloud, so as to shorten the overall preprocessing time.

3. The combined detection system for packaged irregular parts based on vision and photoelectric sensors according to claim 1 is characterized in that: The detection method of the bottom surface scanning detection module is as follows: photoelectric sensors are arranged in a row at equal intervals and perpendicular to the direction of belt movement and placed below the belt gap. When the package is located above the gap, the potential of the photoelectric sensor at the position blocked by the package will be pulled up and down. The controller records the moment when the potential of the photoelectric sensor changes, and integrates and calculates the shape and area of ​​the bottom surface of the package based on the position spacing of the photoelectric sensors, the time difference between the high and low level changes, and the belt speed information.

4. The combined detection system for packaged irregular parts based on vision and photoelectric sensors according to claim 1 is characterized in that: The detection method of the plane image detection module is as follows: a two-stage convolutional neural network structure is designed. The network first deploys a single-stage convolutional neural network Yolo, SSD or Efficient Det / Net with good detection performance, and uses the output frame of the network to crop the image. The cropped image is input into a classifier for detecting double items. The classifier uses a simple convolutional neural network to determine whether the package images overlap.

5. A combined detection system for packaged irregular parts based on vision and photoelectric sensors according to claim 4, characterized in that: The two-stage convolutional neural network structure is accelerated using the TensorRT tool.

6. The combined detection system for packaged irregular parts based on vision and photoelectric sensors according to claim 1 is characterized in that: The preprocessing module, package 3D point cloud extraction and special-shaped parts detection module, bottom surface scanning detection module, plane image detection module, result integration module, communication and log management and file management and visualization module are integrated under the ROS framework, each module is a startup node of ROS, and the communication between modules uses ROS message subscription and distribution service.

Images