A carton detection method and system based on quadrilateral fitting

Abstract

The application discloses a carton detection method and system based on quadrilateral fitting, and belongs to the fields of machine vision and logistics. The method comprises the following steps: simultaneously acquiring a two-dimensional color image and a three-dimensional point cloud image of a scene to be detected; inputting the two-dimensional color image into a trained anchor frame-based detection network to obtain the two-dimensional coordinates of the corner points of each quadrilateral of the carton in the two-dimensional color image; performing plane segmentation on the three-dimensional point cloud image to obtain a plurality of planes; obtaining the three-dimensional coordinates of the carton in the scene to be detected according to the two-dimensional coordinates of the corner points on the segmented planes; and the anchor frame-based detection network is trained by using a training data set {two-dimensional color image of a carton, two-dimensional coordinates of the corner points of the quadrilateral of the carton exposed in the two-dimensional color image}. The application can more accurately obtain the three-dimensional coordinates of the carton by predicting the four corner points of a certain face of the carton through the anchor frame-based detection network, and can adapt to various angles.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of machine vision and logistics, and more particularly, relates to a carton detection method and system based on quadrilateral fitting. BACKGROUND

[0002] With the progress of the times and the development of technology, the intelligent degree of the logistics industry is continuously improved, and robots are widely used in the logistics industry to replace manual work to complete the loading and carrying of cartons, which not only can save costs and improve efficiency, but also can avoid close contact between workers and some dangerous goods. The first step to realize the automatic carrying of cartons is to accurately identify and segment the cartons from the background, and then guide the mechanical hand to grasp.

[0003] In order to realize the automatic carrying work of cartons, various attempts have been made in the industry. The traditional automatic carrying operation needs to pre-set the position of the carton, and the mechanical arm carries out the carrying operation according to the fixed trajectory. This way needs to know the size, position and arrangement of the carton in advance, which is very difficult to do in most cases. Later, the industry proposed a carton segmentation algorithm based on 3D point cloud, which can detect cartons through point cloud segmentation and feature point matching, but this method needs expensive high-precision 3D sensors, and cannot distinguish objects similar in size and shape to cartons, nor can it distinguish different types of cartons. The carton recognition technology based on 3D point cloud can be combined with radio frequency identification technology to obtain relevant data of the carton through wireless transmission, and then classify the carton, but this way needs to pre-install a chip on the carton, which is very difficult to do in most cases. Another method is a carton detection algorithm based on 2D color image, including edge detection-based detection algorithm, template matching-based detection algorithm, etc. These methods are very sensitive to environmental changes, and the stacking of cartons needs to be more orderly.

[0004] In recent years, deep learning has developed rapidly and achieved excellent results in the field of machine vision. Some algorithms such as YOLO and Mask R-CNN have also been applied to carton detection. However, the detection result of YOLO is a horizontal rectangle, which cannot detect scattered cartons, and also has high requirements for the relative position of the camera and the carton. Although Mask R-CNN can detect cartons at any angle, it is slow and difficult to meet the real-time requirement. SUMMARY

[0005] In view of the defects of the prior art, the purpose of the present application is to provide a carton detection method and system based on quadrilateral fitting, which aims to solve the problem that the existing detection method cannot adapt to any angle, while considering real-time and high precision.

[0006] To achieve the above object, in a first aspect, the application provides a carton detection method based on quadrilateral fitting, which comprises:

[0007] S1. Simultaneously acquiring a two-dimensional color image and a three-dimensional point cloud image of a scene to be detected, inputting the two-dimensional color image into a trained anchor box-based detection network to obtain two-dimensional coordinates of corner points of each quadrilateral of a carton in the two-dimensional color image;

[0008] S2. Performing plane segmentation on the three-dimensional point cloud image to obtain a plurality of planes;

[0009] S3. Obtaining three-dimensional coordinates of the carton in the scene to be detected according to the two-dimensional coordinates of the corner points on the segmented planes;

[0010] The anchor box-based detection network is trained by using a training data set {two-dimensional color image of a carton, two-dimensional coordinates of corner points of a quadrilateral of the carton exposed in the two-dimensional color image}.

[0011] Preferably, the training data set covers two-dimensional color images under different light conditions, different backgrounds, different angles between a camera and a carton.

[0012] Preferably, the two-dimensional coordinates of the corner points are offset amounts of four corner points of an anchor box to corner points of a target quadrilateral.

[0013] Preferably, in the anchor box-based detection network, at least one anchor box in each scale adopts a diamond anchor box composed of midpoints of each side of an original rectangular anchor box.

[0014] Preferably, in the anchor box-based detection network, an anchor box regression branch assigns a target quadrilateral to different anchor boxes according to the closeness of the shape of the target quadrilateral to the shape of the anchor box, and then uses different methods to determine the corresponding relationship between the corner points of the target quadrilateral and the corner points of the anchor box for regression.

[0015] Preferably, the calculation formula of the closeness of the shape of the target quadrilateral to the shape of the anchor box is as follows:

[0016] k = Δh / h

[0017] wherein Δh represents the difference between the y coordinates of the first two corner points of the quadrilateral from top to bottom, and h represents the difference between the y coordinates of the uppermost corner point and the lowermost corner point of the quadrilateral.

[0018] If k ≤ α, α represents a preset threshold value, the target quadrilateral is closer to a horizontal rectangle, the corner points of the target quadrilateral are sequentially numbered clockwise as 1, 2, 3 and 4 from the upper left corner point of the anchor box, the anchor box is sequentially numbered clockwise as 1, 2, 3 and 4 from the upper left corner point, and the corner points of the target quadrilateral are regressed from the anchor box corner points with the same serial number.

[0019] If k > a, the target quadrilateral is closer to a rhombus, and the corresponding rhombus anchor frame is sequentially numbered 1, 2, 3, and 4 clockwise from the upper corner point. The rhombus anchor frame is sequentially numbered 1, 2, 3, and 4 clockwise from the upper corner point. Each corner point is regressed from the anchor frame of the same corner point.

[0020] To achieve the above object, in a second aspect, the present application provides a carton detection system based on quadrilateral fitting, comprising: a processor and a memory; the memory is used for storing computer execution instructions; the processor is used for executing the computer execution instructions, so that the method of the first aspect is executed.

[0021] Overall, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects:

[0022] (1) The present application detects the carton by quadrilateral fitting, predicts the four corner points of a certain face of the carton based on the anchor frame detection network, and obtains the three-dimensional coordinates of the carton. Compared with the target detection algorithm of horizontal rectangular frame and rotated rectangular frame, the three-dimensional coordinates of the carton can be more accurately obtained, and it can adapt to various angles. In addition, only the cooperation of the depth camera is needed, and the cost is low.

[0023] (2) For the anchor frame based target detection algorithm, the selection of anchor frame directly affects the final detection result. The traditional target detection algorithm generally uses horizontal rectangular anchor frame, which cannot well adapt to the demand of quadrilateral detection. In view of this problem, the present application changes part of the anchor frame to rhombus anchor frame, divides the target into two categories of more close to rectangle and more close to rhombus, respectively matches different anchor frames, so that the prior frame is more accurate, and the final positioning accuracy is improved.

[0024] (3) The quadrilateral is regressed from the corresponding corner point of the anchor frame, and the corner points of the two are in one-to-one correspondence. This corresponding relationship always exists in a critical state, resulting in a large error in the algorithm when the target quadrilateral is close to the critical state. In view of this problem, in the training, different corner point corresponding relationships are adopted for different quadrilaterals, so that in most cases, when one corresponding relationship is in a critical state, the other corresponding relationship is far from the critical state, the critical problem of quadrilateral regression is improved, and the detection accuracy is improved. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a quadrilateral fitting based carton detection method flow chart provided by the present application.

[0026] Figure 2 is an improved anchor frame shape diagram provided by the present application.

[0027] Figure 3 is a diagram of the coefficient k = Ah / h provided by the present application.

[0028] Figure 4 is a schematic diagram of a regression process provided by the present application, which is allocated to different anchor frame target quadrilaterals.

[0029] Figure 5 is a schematic diagram of the final detection effect provided by the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0031] The present application proposes a carton detection method based on quadrilateral fitting for the loading and unloading and carrying scene of cartons, which can quickly and accurately identify and locate the cartons, and then guide the robot hand to grasp the cartons.

[0032] As shown in Figure 1 The present application provides a carton detection method based on quadrilateral fitting, which specifically includes the following steps:

[0033] Step S1. Collecting a two-dimensional color image of the carton through a camera, including different angles and different light conditions, and labeling the target in the collected image.

[0034] In this embodiment, the carton image is collected by an RGB-D camera.

[0035] When collecting the carton image, the arrangement of the cartons can be in order or scattered, and the appropriate arrangement can be selected according to the requirements.

[0036] If the angle of the carton during detection is fixed, the angle close to the detection can also be selected for sampling, otherwise, as many carton images as possible are collected from different angles.

[0037] When collecting the carton image, the light condition close to the detection environment is selected; if the detection environment is not fixed, sampling can be performed under different backgrounds and light conditions.

[0038] For the collected carton image, labeling is performed through an image labeling tool, and the labeling target is the outer surface of the quadrilateral of the carton exposed in the image; if there are multiple surfaces, all of them need to be labeled.

[0039] Step S2. Data augmentation is performed on the collected image.

[0040] Due to the characteristics of the neural network, the same target exhibits different characteristics in different situations. In order to enable the network to learn as many characteristics of the target as possible, a large number of samples are needed to train the network. Limited by manpower and material resources, the number of collected samples is generally not large, and it is difficult to meet the needs of network training, so the data needs to be expanded to improve the generalization of the prediction model.

[0041] The specific steps of the data expansion of the embodiment are as follows:

[0042] (1) Rotate the collected n images by 0-360° random angle to obtain n sample images;

[0043] (2) Perform horizontal mirroring and upside-down flipping on the collected n images to obtain 2n sample images;

[0044] (3) Perform 0.5-1.5 times random scaling and translation on the collected n images to obtain n sample images;

[0045] (4) Add Gaussian noise to the collected n images to obtain n sample images;

[0046] (5) Perform perspective transformation on the collected n images to obtain n sample images. The absolute value of the difference between the corner coordinates of the perspective transformed image and the original coordinates should be less than 1 / 4 of the length and width of the original image, otherwise the image will be distorted and affect the detection effect of the model.

[0047] When expanding the data set, the target should be avoided to be out of the image range. After the data expansion in the above manner, 6n sample images are finally obtained for model training.

[0048] Step S3. Improve the anchor-based target detection algorithm and train it with the expanded data.

[0049] The improved YOLOv5 algorithm is used as the anchor-based detection network in the embodiment of the application.

[0050] (1) Improve the prediction result.

[0051] The prediction result of YOLOv5 is a target rectangular frame, and the center point coordinates x, y and length and width w, h are obtained by the following formula:

[0052]

[0053] Where x0, y0, w0, h0 are the x, y coordinates and length and width of the anchor frame center, and Δx, Δy, Δw, Δh are the output results of the YOLOv5 output end.

[0054] The prediction result of the improved YOLOv5 is a target quadrangle, and the coordinates x i , y i of the four corners (i = 1, 2, 3, 4) are obtained by the following formula:

[0055]

[0056] wherein, x , y i are the corner coordinates of the anchor frame, Δx i , Δy i (i = 1, 2, 3, 4) are the output results of the output end of the improved YOLOv5.

[0057] (2) Improve the anchor frame.

[0058] The output layer of YOLOv5 contains three different scales, and each scale contains three anchor frames with different length-width ratios, as shown in FIG. 1. Since YOLOv5 outputs a rectangular target frame, the anchor frame is also rectangular. However, for a quadrangular target frame, a rectangular anchor frame cannot play a good prior role. Figure 2

[0059] In order to better predict quadrangles, the two anchor frames in each scale of YOLOv5 are changed to new anchor frames composed of the midpoints of their edges in the embodiment. As shown in FIG. 2, this way has anchor frames with two different shapes, rectangular and rhombus, so as to better adapt to the needs of quadrangle detection. Figure 2

[0060] (3) Improve the corresponding relationship of the corners.

[0061] Which corner of the anchor frame each corner should be regressed from depends on the corresponding relationship between the corners. There are many corresponding relationships between the anchor frame and the corners of the quadrangle, but there will be a sudden change in the corresponding relationship of the corners, which will affect the accuracy of the prediction. The present application adopts different corresponding relationships for rectangular anchor frames and rhombus anchor frames, and according to the closeness of the target quadrangle to the shape of the anchor frame, specifically, by a coefficient k, it is allocated to different anchor frames.

[0062] The target quadrangle is divided into two categories according to the coefficient k, and the coefficient k is defined as:

[0063] k = Δh / h

[0064] As shown in FIG. 3, Δh is the difference between the y coordinates of the first two corners of the quadrangle from top to bottom, and h is the difference between the y coordinates of the uppermost corner and the lowermost corner of the quadrangle. Figure 3

[0065] ​The quadrilateral with k<=a is closer to a horizontal rectangle, and the horizontal rectangle anchor frame is sequentially numbered as 1, 2, 3, and 4 from the top-left corner point in a clockwise direction, and each corner point is regressed from the anchor frame corner point with the same serial number; the quadrilateral with k>a is closer to a diamond, and the diamond anchor frame is sequentially numbered as 1, 2, 3, and 4 from the top corner point in a clockwise direction, and each corner point is regressed from the anchor frame corner point with the same serial number. a is a set threshold, and the value range is [0.2, 0.3].

[0066] As shown in Figure 4 , the left and right two figures respectively represent the process of corner point regression of the target quadrilateral from the anchor frame when k<=a and k>a.

[0067] Step S4. The two-dimensional color image of the carton is input into the trained neural network to obtain the two-dimensional coordinates of the carton.

[0068] The trained network model is used to detect the sample, and the result is shown in Figure 5 .

[0069] Step S5. The point cloud image of the carton is segmented, and the three-dimensional coordinates of the carton are obtained on the segmented plane according to the two-dimensional coordinates.

[0070] The RANSAC algorithm is used to segment the plane in the depth map of the sample to obtain a plurality of planes. Each detected target is assigned to a plane, and the three-dimensional coordinates of the carton are obtained according to the two-dimensional coordinates of the detected corner points.

[0071] The three-dimensional plane is represented by the formula k1x+k2y+k3z=0, and the two-dimensional coordinates x, y of the corner points obtained in step S4 are brought into the three-dimensional plane formula to obtain the z coordinates of the corner points.

[0072] The present application is not limited to YOLOv5 in the above-mentioned embodiments, but is applicable to all anchor-based regression type target detection algorithms.

[0073] Those skilled in the art will readily understand that the above description is only a preferred embodiment of the present application and is not intended to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A quadrangle fitting based carton detection method, characterized in that, The method comprises: S1. Simultaneously acquiring a two-dimensional color image and a three-dimensional point cloud image of a scene to be detected, inputting the two-dimensional color image into a trained anchor box-based detection network to obtain two-dimensional coordinates of corner points of each quadrangle of a carton in the two-dimensional color image; S2. Performing plane segmentation on the three-dimensional point cloud image to obtain a plurality of planes; S3. Obtaining three-dimensional coordinates of the carton in the scene to be detected on the segmented planes according to the two-dimensional coordinates of the corner points. The anchor box-based detection network is trained by using a training data set {a two-dimensional color image of a carton, two-dimensional coordinates of corner points of a quadrangle of the carton exposed in the two-dimensional color image}, at least one anchor box in each scale in the anchor box-based detection network adopts a diamond anchor box composed of midpoints of each side of an original rectangular anchor box, anchor box regression branches assign a target quadrangle to different anchor boxes according to the closeness of the shape of the target quadrangle to the shape of the anchor box, and then determine the corresponding relationship between the corner points of the target quadrangle and the corner points of the anchor box by using different methods for regression; and the closeness of the shape of the target quadrangle to the shape of the anchor box is calculated according to the following formula: wherein, represents the difference between the y coordinates of the upper two corner points of the quadrangle, represents the difference between the y coordinates of the uppermost corner point and the lowermost corner point of the quadrangle. If , represents a preset threshold, the target quadrilateral is closer to a horizontal rectangle, and the corresponding rectangular anchor frame is sequentially numbered clockwise from the upper left corner point as 1, 2, 3, and 4, and the corner points of the target quadrilateral are regressed from the corner points with the same anchor frame serial number. If , the target quadrilateral is closer to a rhombus, and a corresponding rhombus anchor frame is obtained, the rhombus anchor frame is sequentially numbered as 1, 2, 3, and 4 clockwise from the upper corner point, and each corner point is regressed from the corner point with the same anchor frame number.

2. The method of claim 1, wherein, The training data set covers two-dimensional color images under different light conditions, different backgrounds, different angles between a camera and a carton.

3. The method of claim 1, wherein, The two-dimensional coordinates of the corner points are offsets of four corner points of an anchor box to corner points of a target quadrangle.

4. A quadrangle fitting based carton detection system, characterized in that, Comprise: A processor and a memory; The memory is used for storing computer execution instructions; The processor is used for executing the computer execution instructions, so that the method in any one of claims 1 to 3 is executed.

Images