A Visual Positioning System and Method for Carton Depalletizing

Abstract

The present invention discloses a visual positioning system for carton palletizing. A top camera for collecting the image below is provided directly above the palletizing area, and a side camera for collecting the side image of the palletizing area is provided on the side of the palletizing area. Both the top camera and the side camera are connected and output the collected 2D image and 3D point cloud signals to the control host. The present invention uses a 3D camera to collect 2D images and 3D point clouds and uses machine vision methods for carton positioning, which can achieve carton positioning more quickly and accurately; combining the characteristics of 2D images and 3D point cloud data for carton positioning, the positioning speed is faster, and the positioning accuracy and stability are better; through visual automatic positioning of cartons, robots can be used for carton depalletizing, which can reduce costs, improve production efficiency and the level of industrial automation.

Description

Technical Field

[0001] The present invention relates to the field of visual positioning, and particularly to the field of realizing carton segmentation and positioning through machine vision processing. Background Art

[0002] Carton depalletizing is a labor-intensive industry. It requires manual workers to take out cartons from the pallet stack and place them at the designated position. Some cartons are large in volume and weight, resulting in a high labor intensity for depalletizing. It is even more difficult to use manual labor for depalletizing large pallet stacks.

[0003] With the continuous development of robot technology, there are more and more scenarios where robots are used to replace humans in performing high-intensity and repetitive tasks. Using robots to replace humans for carton depalletizing is also a new solution. The most crucial part of using a robot for carton depalletizing lies in positioning the cartons in the pallet stack. In recent years, with the development of 3D point cloud technology, when people conduct industrial inspections, they have started to use the point cloud images collected by 3D cameras for inspection. This is a new technology. This new technology can solve many problems in the industrial inspection process through point cloud images, replacing the backward manual inspection method, improving the inspection efficiency and the level of industrial automation. Moreover, using 3D point cloud for inspection does not require contact with the product to be inspected, will not cause damage, has high inspection accuracy, and is more intelligent. Currently, there are no domestic carton depalletizing positioning methods and related visual inspection and positioning solutions. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to realize an automatic carton positioning system and method based on 3D vision to solve the problem of carton positioning during the carton depalletizing process.

[0005] To achieve the above object, the technical solution adopted by the present invention is as follows: A visual positioning system for carton depalletizing, a top camera for collecting the image below is provided directly above the palletizing area, and a side camera for collecting the side image of the palletizing area is provided on the side of the palletizing area. Both the top camera and the side camera are connected and output the collected 2D image and 3D point cloud signal to the control host.

[0006] The side camera is fixed at the end of the robot's arm. There are columns on both sides of the palletizing area, and the tops of the two columns are connected by a connecting rod. The top camera is fixed on the connecting rod.

[0007] Both the top camera and the side camera are 3D cameras, and the control host is connected and outputs a control signal to the robot.

[0008] A visual positioning method for carton depalletizing includes the following steps:

[0009] Step 1, obtain the 2D image of the carton and the corresponding 3D point cloud data in real time;

[0010] Step 2: Segment the carton according to the 2D image data of the carton;

[0011] Step 3: Segment the cartons stacked together in the image into individual carton areas;

[0012] Step 4: Calculate the coordinates of an individual carton based on the 3D point cloud data of the carton and the segmented individual carton area;

[0013] Step 5: Obtain the positioning of each carton.

[0014] In the above Step 1, data is obtained in real time through two dimensions, directly above and on the side of the palletizing area.

[0015] Steps 2 and 3 include the following steps:

[0016] 1) Extract the carton area based on the positional relationship between the carton and the background and the 3D point cloud data, and filter out the non-carton background area in the image;

[0017] 2) Binarize the image using an adaptive binarization method to enhance the gap area between cartons in the image;

[0018] 3) Process the image morphologically to filter out the burrs in the gap area, connect the discontinuous gaps, and optimize the gap area;

[0019] 4) Use the connected component analysis contour extraction method to extract the carton and gap contours;

[0020] 5) Calculate the contour corner point features;

[0021] 6) Segment the individual carton areas in the image through feature matching.

[0022] Steps 4 and 5 include the following steps:

[0023] 1) In the individual cartons segmented from the cartons to be depalletized, calculate the positional relationship of each carton relative to the depalletizing robot, and select the carton closest to the depalletizing robot for depalletizing;

[0024] 2) According to the segmented individual carton area, extract the 3D point cloud data of the corresponding area;

[0025] 3) Use the point cloud boundary extraction algorithm based on normal estimation to extract the boundary of the point cloud data of the individual carton area;

[0026] 4) Use the random sampling least squares method to fit a straight line on the point cloud boundary to obtain 4 straight lines, which form the 4 sides of the carton rectangle;

[0027] 5) Calculate the boundary coordinates of the carton according to the intersection points of the 4 calculated straight lines;

[0028] 6) Finally, the central coordinates of the carton are calculated through the intersection point of the connection lines of two groups of diagonal intersection points, and the positioning position of the carton is obtained.

[0029] The present invention uses a 3D camera to collect 2D images and 3D point clouds and uses machine vision methods for carton positioning, which can achieve carton positioning more quickly and accurately; combining the characteristics of 2D images and 3D point cloud data for carton positioning, the positioning speed is faster, and the positioning accuracy and stability are better; through visual automatic positioning of the carton, a robot can be used for carton depalletizing, which can reduce costs, improve production efficiency and the level of industrial automation. Brief Description of the Drawings

[0030] The following briefly describes the content expressed in each drawing in the specification of the present invention and the marks in the drawings:

[0031] Figure 1 It is the schematic diagram of the visual positioning system for carton depalletizing;

[0032] Figure 2 It is the flow chart of the visual positioning method for carton depalletizing;

[0033] Figure 3 It is the flow chart of the 2D segmentation process of the carton;

[0034] Figure 4 It is the flow chart of the 3D positioning process of the carton;

[0035] The marks in the figure are as follows: 1. Top camera; 2. Carton stack; 3. Robot; 4. Side camera. Detailed Embodiments

[0036] The following, with reference to the drawings, through the description of the embodiments, the specific embodiments of the present invention, such as the shapes, structures of the various components involved, the mutual positions and connection relationships between the various parts, the functions and working principles of the various parts, the manufacturing process and the operation and use methods, etc., are further described in detail to help those skilled in the art have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0037] As Figure 1 shown, the visual positioning system for carton depalletizing is provided with two camera positions, one at the top of the palletizing area and one on the side of the palletizing area. A top camera 1 for collecting the lower image is provided directly above the palletizing area, and a side camera 4 for collecting the side image of the palletizing area is provided on the side of the palletizing area. Generally, the side camera 4 is fixed at the end of the arm of the robot 3. There are columns on both sides of the palletizing area, and the tops of the two columns are connected by a connecting rod. The top camera 1 is fixed on the connecting rod.

[0038] Both the top camera 1 and the side camera 4 are connected and output the collected 2D images and 3D point cloud signals to the control host. Both the top camera 1 and the side camera 4 are 3D cameras, and the control host is connected and outputs control signals to the robot 3. After the control host obtains the signals, it identifies the cartons. For the visual positioning method of carton depalletizing, first, the top camera 1 on the front collects the 2D and 3D data of the top of the carton stack 2 for top segmentation and positioning. Then, according to the top segmentation and positioning results, the carton grasping sequence is planned. The robot 3 moves the side 3D camera installed on the robot 3 to a suitable position according to the position of the carton to be grasped, collects the 2D and 3D data of the side of the carton stack 2, and then calls an algorithm for side segmentation and positioning. Finally, according to the side segmentation and positioning results, the carton is grasped from the side.

[0039] As Figure 2 shown, first, 2D images and corresponding 3D point cloud data are collected by a 3D camera. Then, the cartons are segmented according to the 2D image data of the cartons. The cartons stacked together in the image are segmented into individual carton areas. Finally, according to the 3D point cloud data of the cartons and the segmented individual carton areas, the coordinates of the individual cartons are calculated to achieve carton positioning.

[0040] The 2D segmentation process of the cartons is as Figure 3 shown: 1. Extract the carton area based on the positional relationship between the cartons and the background and the 3D point cloud data, and filter out the non-carton background areas in the image; 2. Binarize the image using an adaptive binarization method to strengthen the gap areas between the cartons in the image; 3. Process the image morphologically to filter out the burrs in the gap areas and connect the discontinuous gaps to optimize the gap areas; 4. Use a connected component analysis contour extraction method to extract the carton and gap contours; 5. Calculate the contour corner point features; 6. Segment the individual carton areas in the image through feature matching.

[0041] The 3D positioning process of the cartons is as Figure 4 shown: 1. In the individual cartons segmented from the cartons to be depalletized, calculate the positional relationship of each carton relative to the depalletizing robot 3, and select the carton closest to the depalletizing robot 3 for depalletizing; 2. According to the segmented individual carton areas, extract the 3D point cloud data of the corresponding areas; 3. Use a point cloud boundary extraction algorithm based on normal estimation to extract the boundaries of the 3D point cloud data of the individual carton areas; 4. Use the random sample consensus least squares method to fit lines on the point cloud boundary to obtain 4 lines, which are the 4 sides of the carton rectangle; 5. Calculate the boundary coordinates of the carton according to the intersection points of the 4 calculated lines; 6. Finally, calculate the center coordinates of the carton through the intersection point of the connection lines of 2 sets of diagonal intersection points to obtain the positioning position of the carton.

[0042] The present invention has been described by way of example in conjunction with the accompanying drawings. Obviously, the specific implementation of the present invention is not limited by the above methods. As long as various non-substantive improvements are made by adopting the method concept and technical solution of the present invention, or the concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A visual positioning method for carton depalletizing, characterized in that: Applied to the visual positioning system for carton depalletizing, a top camera for collecting the image below is provided directly above the palletizing area, and a side camera for collecting the side image of the palletizing area is provided on the side of the palletizing area. Both the top camera and the side camera are connected and output the collected 2D image and 3D point cloud signals to the control host; The side camera is fixed at the end of the robot's arm. There are columns on both sides of the palletizing area, and the tops of the two columns are connected by a connecting rod. The top camera is fixed on the connecting rod; Both the top camera and the side camera are 3D cameras, and the control host is connected and outputs control signals to the robot; The visual positioning method for carton depalletizing includes the following steps: Step 1: Real-time obtain the 2D image of the carton and the corresponding 3D point cloud data; Step 2: Segment the carton according to the 2D image data of the carton; Step 3: Segment the cartons stacked together in the image into individual carton areas; Step 4: Calculate the coordinates of an individual carton according to the 3D point cloud data of the carton and the segmented individual carton area; Step 5: Obtain the positioning of each carton; In the above Step 1, the real-time data acquisition is carried out through two dimensions, directly above and on the side of the palletizing area; Steps 2 and 3 include the following steps: 1) Extract the carton area based on the positional relationship between the carton and the background and the 3D point cloud data, and filter out the non-carton background area in the image; 2) Binarize the image using the adaptive binarization method to strengthen the gap area between the cartons in the image; 3) Perform morphological processing on the image to filter out the burrs in the gap area, connect the discontinuous gaps, and optimize the gap area; 4) Use the connected domain analysis contour extraction method to extract the carton and gap contours; 5) Calculate the contour corner point features; 6) Segment the individual carton areas in the image through feature matching.

2. The visual positioning method for carton depalletizing according to claim 1, characterized in that: Steps 4 and 5 include the following steps: 1) In the individual cartons segmented from the cartons to be depalletized, calculate the positional relationship of each carton relative to the depalletizing robot, and select the carton closest to the depalletizing robot for depalletizing; 2) According to the segmented individual carton area, extract the 3D point cloud data of the corresponding area; 3) Use the point cloud boundary extraction algorithm based on normal estimation to extract the boundary of the 3D point cloud data of the individual carton area; 4) Use the random sampling least squares method to fit a straight line on the point cloud boundary to obtain 4 straight lines, which form the 4 sides of the carton rectangle; 5) Calculate the boundary coordinates of the carton according to the intersection points of the 4 calculated straight lines; 6) Finally, calculate the center coordinates of the carton through the intersection point of the connection lines of 2 groups of diagonal intersection points to obtain the positioning position of the carton.

Images