A cargo box recognition method based on the combination of image and point cloud

Abstract

The present invention provides a cargo box recognition method based on the combination of images and point clouds, including: using a depth camera to collect cargo box images and point clouds, and processing the collected cargo box images to form a training set; training the images in the training set based on the deep learning network Mask R-CNN to obtain weights; setting up a communication interface between the camera and the industrial control computer (IPC), connecting the camera, and using the obtained weights to predict new cargo box images input during operation, obtain a mask for each cargo box, and output the pixel coordinates of the mask; based on the pixel coordinates, calculating the index range of the corresponding point in the point cloud; based on the index range, performing region growing segmentation on the point cloud, and determining whether the damage is serious based on the point cloud segmentation results. The present invention can provide an important basis for different methods of grasping and subsequent processing of intact and severely damaged cargo boxes. This method can effectively address the identification and destacking of damaged cargo boxes, such as those in cold chain containers.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and point cloud processing technology, and more specifically, to a cargo box recognition method based on the combination of images and point clouds. Background Art

[0002] As cold chain transportation plays an increasingly important role in the import of goods, a large amount of manpower and material resources are invested in the destacking of goods in containers. Therefore, a fully automatic visually guided destacking solution is needed. However, the cargo boxes of cold chain goods are of various types, sizes, colors, and damages. The existing two-dimensional vision is affected by many factors such as the surface texture of the pattern and lighting, and cannot be well identified. 3D point cloud vision has been gradually applied in recent years because it is not affected by these factors. Therefore, the present invention proposes a method that combines the two-dimensional plane level and the 3D point cloud level to well identify these cartons with complex surface textures and easily damaged during transportation.

[0003] Deep learning methods for 2D vision are widely used for their superior recognition performance. This paper builds on the Mask R-CNN algorithm proposed by Kaiming He in 2017 for 2D recognition. Mask R-CNN not only achieves higher recognition accuracy than traditional visual recognition algorithms, but also holds a high accuracy rate among existing deep learning algorithms. The region growing algorithm for 3D vision is simple and easy to implement. Its main principle is point-by-point clustering, making it suitable for small-scale point clouds and offering high accuracy and impressive recognition speed. Summary of the Invention

[0004] To address the aforementioned technical issues, a method for cargo box recognition based on a combination of images and point clouds is provided. This method uses a depth camera to capture cargo box images, labels the captured data using LabelMe, and trains the data using the deep learning method Mask R-CNN. After obtaining the trained weights, the method connects the camera to perform predictions. After each predicted cargo box, a point cloud segmentation method is used to determine whether any of the boxes are severely damaged and cannot be grasped with a suction cup.

[0005] The technical means adopted in the present invention are as follows:

[0006] A cargo box recognition method based on the combination of image and point cloud, comprising:

[0007] Use a depth camera to collect cargo box images, and process the collected cargo box images to form a training set;

[0008] The images in the training set are trained based on the deep learning network Mask R-CNN to obtain weights;

[0009] Set up the communication interface between the camera and the industrial computer IPC, connect the camera, use the obtained weights to predict the new cargo box images input at work, obtain the mask of each cargo box, and output the pixel coordinates of the mask;

[0010] According to the pixel coordinates, calculate the index range of the corresponding point in the point cloud;

[0011] According to the index range, the point cloud is segmented by region growing, and whether the damage is serious is determined based on the point cloud segmentation results.

[0012] Furthermore, the method of using a depth camera to collect cargo box images and processing the collected cargo box images to form a training set includes:

[0013] Perform image enhancement on the collected images and use LabelMe software for data annotation;

[0014] Generate train.json, val.json, and test.json files that conform to the COCO dataset format from the json files generated by the images and annotations.

[0015] Furthermore, the deep learning network Mask R-CNN is used to train the images in the training set to obtain weights, including:

[0016] Set the initial learning rate to 1e-4, select Adam as the optimizer, and train for 50 rounds to obtain the trained weights.

[0017] Furthermore, the obtained weights are used to predict new cargo box images input during work, obtain masks for each cargo box, and output pixel coordinates of the masks, including:

[0018] Use the ResNet50 backbone network of the deep learning network Mask R-CNN for feature extraction;

[0019] Use bounding box regression to translate and scale the ROI area;

[0020] Use non-maximum suppression to obtain the target frame;

[0021] The final mask is obtained using the ROIAlign layer of the deep learning network Mask R-CNN;

[0022] After obtaining the mask, use the findcontour function of the skimage library to extract the edge of the mask and obtain all the pixel coordinates of the edge contour of the mask.

[0023] Furthermore, the step of calculating the index range of the corresponding point in the point cloud according to the pixel coordinates includes:

[0024] After the image recognizes the mask of each container, the point cloud is segmented according to the index range of the pixel coordinates of each mask. The pixel coordinates of the depth camera and the index in the point cloud have the following correspondence:

[0025] Index=512×y+x

[0026] Where (x, y) is the pixel coordinate of a point in the image, and the camera resolution is 512×424.

[0027] Furthermore, performing region growing segmentation on the point cloud according to the index range and judging whether the damage is serious according to the point cloud segmentation result include:

[0028] Sort the point cloud according to curvature;

[0029] Add the point with the minimum curvature to the seed set, and use the k-dimensional tree to search for nearby points to calculate the angle difference with the normal line of the seed point;

[0030] If the difference is less than the smoothing threshold and the curvature of the point is less than the curvature threshold, then the point is added to the seed set and considered to be of the same type as the seed point; otherwise, continue searching for nearby points;

[0031] Repeat the above steps until all points are traversed;

[0032] If the number of clusters in the region growing result is 1 or the number of clusters is greater than 1 but the cluster with the largest number of cluster points far exceeds the sum of the remaining classes, the occurrence of either of these two conditions indicates that the cargo box is intact and can be grabbed; otherwise, it is considered that the cargo box corresponding to the mask is severely damaged and cannot be grabbed.

[0033] Furthermore, the cargo box recognition method based on the combination of image and point cloud also includes: when the cargo box is judged to be complete based on the point cloud segmentation result, the edge of the mask is further processed to obtain the pixel coordinates of the highest point and the lowest point close to the center point of the edge. Since the pixel coordinates of the depth camera and the camera coordinates have a natural correspondence, the camera coordinates of the upper and lower edges of each cargo box can be obtained, and then the camera coordinates are converted to world coordinates:

[0034] (x w ,y w ,z w ) T =R·(x c ,y c ,z c ) T +T1+T2

[0035]

[0036]

[0037] Among them, R represents the rotation matrix, T1 represents the translation matrix, and T2 is the measurement value from the origin of the world coordinate system. (x w ,y w ,z w ) T Represents a world coordinate point, (x c ,y c ,z c ) T Represents a point in the camera coordinate system.

[0038] Compared with the prior art, the present invention has the following advantages:

[0039] 1. The cargo box recognition method based on the combination of images and point clouds provided by the present invention uses a depth camera to collect some cargo box images, uses LabelMe to annotate the collected data, and uses the deep learning method Mask R-CNN for training. After obtaining the trained weights, it connects the camera to perform prediction. After predicting each cargo box, it uses the point cloud segmentation method to determine whether these cargo boxes are severely damaged and cannot be grasped by the suction cup.

[0040] 2. The cargo box recognition method based on the combination of images and point clouds provided by the present invention can provide an important basis for subsequent processing such as grabbing intact cargo boxes and severely damaged cargo boxes. This method can effectively deal with the identification and destacking of damaged cargo boxes in cold chain containers.

[0041] Based on the above reasons, the present invention can be widely promoted in the fields of computer vision, point cloud processing, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 Flow chart of the method of the present invention.

[0044] Figure 2 This is a diagram of the overall architecture of the deep learning network Mask R-CNN provided by an embodiment of the present invention.

[0045] Figure 3 The embodiment of the present invention provides original image data of damaged cartons.

[0046] Figure 4 This figure illustrates how the Mask R-CNN provided in this embodiment of the present invention recognizes a damaged cardboard stack.

[0047] Figure 5 The region growing segmentation method provided in the embodiment of the present invention is used to identify a stack of damaged cartons. DETAILED DESCRIPTION

[0048] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0050] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0051] Unless otherwise specifically stated, the relative arrangement of the parts and steps, numerical expressions and numerical values ​​described in these embodiments do not limit the scope of the present invention. At the same time, it should be clear that, for ease of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship. The techniques, methods and equipment known to ordinary technicians in the relevant fields may not be discussed in detail, but where appropriate, the techniques, methods and equipment should be considered as part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as merely exemplary and not as a limitation. Therefore, other examples of the exemplary embodiments may have different values. It should be noted that similar numbers and letters represent similar items in the following figures, so once an item is defined in one figure, it does not need to be further discussed in subsequent figures.

[0052] In the description of the present invention, it should be understood that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "horizontal, vertical, vertical, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the device or element referred to must have a specific direction or be constructed and operated in a specific direction. Therefore, they cannot be understood as limiting the scope of protection of the present invention: the directional words "inside and outside" refer to the inside and outside relative to the outline of each component itself.

[0053] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used herein to describe the spatial positional relationship of a device or feature to other devices or features as shown in the figures. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figures. For example, if the device in the drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below their position devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used here are interpreted accordingly.

[0054] In addition, it should be noted that the use of terms such as "first" and "second" to limit components is only for the convenience of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of the present invention.

[0055] like Figure 1 As shown, the present invention provides a cargo box recognition method based on the combination of image and point cloud, comprising:

[0056] S1. Use a depth camera to collect cargo box images and process the collected cargo box images to form a training set;

[0057] S2. Train the images in the training set based on the deep learning network Mask R-CNN to obtain weights;

[0058] S3. Set up the communication interface between the camera and the IPC, connect the camera, use the obtained weights to predict the new cargo box images input during work, obtain the mask of each cargo box, and output the pixel coordinates of the mask;

[0059] S4. Calculate the index range of the corresponding point in the point cloud based on the pixel coordinates;

[0060] S5. Perform region growing segmentation on the point cloud according to the index range, and determine whether the damage is serious based on the point cloud segmentation results.

[0061] In a specific implementation, as a preferred embodiment of the present invention, step S1 uses a depth camera to collect cargo box images, and processes the collected cargo box images to form a training set, including:

[0062] Perform image enhancement on the collected images and use LabelMe software for data annotation;

[0063] Generate train.json, val.json, and test.json files that conform to the COCO dataset format from the json files generated by the images and annotations.

[0064] In a specific implementation, as a preferred embodiment of the present invention, step S2 trains the images in the training set based on the deep learning network Mask R-CNN to obtain weights, including:

[0065] Set the initial learning rate to 1e-4, select Adam as the optimizer, and train for 50 epochs to obtain the trained weights. In this embodiment, the Mask R-CNN structure includes a backbone network, an RPN region proposal network, a ROIAlign layer, and a fully connected layer.

[0066] In a specific implementation, as a preferred embodiment of the present invention, step S3 uses the obtained weights to predict a new cargo box image input during work, obtains a mask for each cargo box, and outputs the pixel coordinates of the mask, including:

[0067] Feature extraction is performed using the backbone network ResNet50 of the deep learning network Mask R-CNN. In this embodiment, ResNet can be seen as a combination of two blocks (Conv Block and Identity Block), where the input and output dimensions of ConvBlock are different, so they cannot be connected in series continuously. Its function is to change the dimension of the network. The input and output dimensions of Identity Block are the same and can be connected in series to deepen the network. After the first cloth in ResNet undergoes a 7×7 convolution, the activation function Relu multiplied by it is modified to Mish. Compared with Relu, Mish has the characteristics of continuous differentiability, which makes it easier to optimize the gradient during the back propagation of the neural network and the network generalization effect is better. The formulas of the two activation functions are as follows:

[0068]

[0069] σ(x)=x·tanh(ln(1+ex ))

[0070] Use bounding box regression to translate and scale the ROI area;

[0071] Use non-maximum suppression to obtain the target frame;

[0072] The final mask is obtained using the ROIAlign layer of the deep learning network Mask R-CNN;

[0073] After obtaining the mask, use the findcontour function of the skimage library to extract the edge of the mask and obtain all the pixel coordinates of the edge contour of the mask.

[0074] In specific implementation, as a preferred embodiment of the present invention, step S4 calculates the index range of the corresponding point in the point cloud according to the pixel coordinates, including:

[0075] After the image recognizes the mask of each container, the point cloud is segmented according to the index range of the pixel coordinates of each mask. The pixel coordinates of the depth camera and the index in the point cloud have the following correspondence:

[0076] Index=512×y+x

[0077] Where (x, y) is the pixel coordinate of a point in the image, and the camera resolution is 512×424.

[0078] In a specific implementation, as a preferred embodiment of the present invention, step S5 performs region growing segmentation on the point cloud according to the index range, and determines whether the damage is serious according to the point cloud segmentation result, including:

[0079] Sort the point cloud according to curvature;

[0080] Add the point with the minimum curvature to the seed set, and use the k-dimensional tree to search for nearby points to calculate the angle difference with the normal line of the seed point;

[0081] If the difference is less than the smoothing threshold and the curvature of the point is less than the curvature threshold, then the point is added to the seed set and considered to be of the same type as the seed point; otherwise, continue searching for nearby points;

[0082] Repeat the above steps until all points are traversed;

[0083] If the number of clusters in the region growing result is 1 or the number of clusters is greater than 1 but the cluster with the largest number of cluster points far exceeds the sum of the remaining classes, the occurrence of either of these two conditions indicates that the cargo box is intact and can be grabbed; otherwise, it is considered that the cargo box corresponding to the mask is severely damaged and cannot be grabbed.

[0084] In this embodiment, point cloud segmentation has strict requirements on the spacing between different objects. That is, if there are serious damage or cracks, they are easy to segment out in the point cloud, while image segmentation tends to identify the entire carton area as a whole, and it is difficult to annotate the damaged cargo box data set. Therefore, the point cloud method is added on the basis of the image method.

[0085] Example

[0086] Connect a camera and take images of cargo boxes of different sizes as a dataset.

[0087] The collected data set was annotated using LabelMe, and then the data set was divided into a training set and a validation set at a ratio of 9:1.

[0088] Process the labeled images and annotation information to generate a json file that conforms to the COCO format.

[0089] The activation function after the 7×7 convolution of the Mask R-CNN backbone network is changed to Mish, which is continuously differentiable and has better generalization performance.

[0090] Train Mask R-CNN with a learning rate of 1e-4, 50 training rounds, and SGD optimizer. Fine-tune the training using the pre-trained Mask R-CNN weights from the COCO dataset.

[0091] Use the weights of the trained dataset to perform instance segmentation on each box fed into the instance segmentation network Mask R-CNN, and obtain the mask of each box as follows Figure 4 shown.

[0092] To filter out the identified non-cargo box objects, you can use the z-distance to filter. The distance between the non-cargo box objects and the depth camera is much greater than the distance between the cargo box and the depth camera, and use this as the filtering condition.

[0093] The edge coordinates of each cargo box mask are screened, and the findcontour function in the skimage library of Python is used to extract the edge pixel coordinates.

[0094] After obtaining the edge pixel coordinates of the cargo box, convert them into world coordinates. Here, it is necessary to measure the length d, width w (not marked in the figure), and height H of the servo gimbal carrying the camera from the optical center of the camera. In this way, the camera coordinate to world coordinate conversion formula when the camera is tilted by θ degrees can be obtained:

[0095] (x w ,y w ,z w ) T =R·(xc ,y c ,z c ) T +T1+T2

[0096]

[0097]

[0098] Among them, R represents the rotation matrix, T represents the translation matrix, (x w ,y w ,z w ) T Represents a world coordinate point, (x c ,y c ,z c ) T Represents a point in the camera coordinate system.

[0099] The above steps are for identifying and grabbing normal cartons without serious damage. Figure 4 It can be seen that a damaged box was also identified with a complete mask. If specific grasping methods are not adopted at this time, the grasping may fail and have a serious impact on subsequent grasping.

[0100] Therefore, a method combining images and point clouds is proposed here to solve this problem. After the image recognizes the mask of each container, the point cloud is segmented according to the index range of the pixel coordinates of each mask. The pixel coordinates of the depth camera and the index in the point cloud have the following correspondence:

[0101] Index=512×y+x

[0102] Where (x, y) is the pixel coordinate of a point in the image, and the camera resolution is 512×424.

[0103] According to the above index calculation method, we can get an index range, and perform region growing instance segmentation on the point cloud in this range. We can see the region growing segmentation result as follows: Figure 5 As shown. Region growing is essentially a clustering algorithm. The calculation formula involved is as follows:

[0104] The least squares local plane P is fitted between each point and its neighbors:

[0105]

[0106] Covariance matrix M, kdtree search method sets the number of neighbor search points k, here k = 30:

[0107]

[0108] Decompose the eigenvalue of M and obtain the eigenvalue:

[0109] λM-E=0

[0110] The eigenvector corresponding to the smallest eigenvalue of M is the normal vector of the fitted plane between neighboring points: λ0, λ1, λ2 are the eigenvalues ​​of M, and λ0≤λ1≤λ2; the surface curvature calculation formula is:

[0111]

[0112] In this embodiment, the operation of point cloud is performed using the python-pcl library. PCL is very convenient for point cloud operation and has many built-in functions available for use.

[0113] Region growing clustering can be sorted by the number of points in each cluster. In the region growing segmentation of the point cloud, we can see that the damaged box in the image is represented by multiple, extremely incomplete clusters in the point cloud segmentation. Within this index range, there are only two cases where a complete box can be considered: there is only one cluster within the range, or there are multiple clusters but the number of points in the top cluster is far greater than that of the other clusters.

[0114] In summary, this method of combining images and point clouds can provide an important basis for the selection of grabbing methods and post-processing of intact and severely damaged cargo boxes.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cargo box recognition method based on the combination of image and point cloud, characterized in that: include: Use a depth camera to collect cargo box images, and process the collected cargo box images to form a training set; The images in the training set are trained based on the deep learning network Mask R-CNN to obtain weights; Set up the communication interface between the camera and the industrial computer IPC, connect the camera, use the obtained weights to predict the new cargo box images input at work, obtain the mask of each cargo box, and output the pixel coordinates of the mask; According to the pixel coordinates, calculate the index range of the corresponding point in the point cloud; Perform region growing segmentation on the point cloud based on the index range, and determine whether the damage is serious based on the point cloud segmentation results, including: Sort the point cloud according to curvature; Add the point with the minimum curvature to the seed set, and use the k-dimensional tree to search for nearby points to calculate the angle difference with the normal line of the seed point; If the difference is less than the smoothing threshold and the curvature of the point is less than the curvature threshold, then the point is added to the seed set and considered to be of the same type as the seed point; otherwise, continue searching for nearby points; Repeat the above steps until all points are traversed; If the number of clusters in the region growing result is 1 or the number of clusters is greater than 1 but the cluster with the largest number of cluster points far exceeds the sum of the remaining classes, the occurrence of either of these two conditions indicates that the cargo box is intact and can be grabbed; otherwise, it is considered that the cargo box corresponding to the mask is severely damaged and cannot be grabbed.

2. The container recognition method based on image and point cloud combination according to claim 1 is characterized in that: The method of using a depth camera to collect cargo box images and processing the collected cargo box images to form a training set includes: Perform image enhancement on the collected images and use LabelMe software for data annotation; Generate train.json, val.json, and test.json files that conform to the COCO dataset format from the json files generated by the images and annotations.

3. The cargo box recognition method based on image and point cloud combination according to claim 1 is characterized in that: The deep learning network Mask R-CNN is used to train the images in the training set to obtain weights, including: Set the initial learning rate to 1e-4, select Adam as the optimizer, and train for 50 rounds to obtain the trained weights.

4. The cargo box recognition method based on image and point cloud combination according to claim 1 is characterized in that: The obtained weights are used to predict the new cargo box images input during work, obtain the mask of each cargo box, and output the pixel coordinates of the mask, including: Use the ResNet50 backbone network of the deep learning network Mask R-CNN for feature extraction; Use bounding box regression to translate and scale the ROI area; Use non-maximum suppression to obtain the target frame; The final mask is obtained using the ROIAlign layer of the deep learning network Mask R-CNN; After obtaining the mask, use the findcontour function of the skimage library to extract the edge of the mask and obtain all the pixel coordinates of the edge contour of the mask.

5. The cargo box recognition method based on image and point cloud combination according to claim 1 is characterized in that: Calculating the index range of the corresponding point in the point cloud according to the pixel coordinates includes: After the image recognizes the mask of each container, the point cloud is segmented according to the index range of the pixel coordinates of each mask. The pixel coordinates of the depth camera and the index in the point cloud have the following correspondence: Index = 512× y + x in,( x , y ) is the pixel coordinate of a point in the image, and the resolution of the camera is 512×424.

6. The cargo box recognition method based on the combination of image and point cloud according to claim 1 is characterized in that: Also includes: When the container is judged to be complete based on the point cloud segmentation results, the edge of the mask is processed again to obtain the pixel coordinates of the highest and lowest points close to the center point of the edge. Since the pixel coordinates of the depth camera and the camera coordinates have a natural correspondence, the camera coordinates of the upper and lower edges of each container can be obtained, and then converted from the camera coordinates to the world coordinates: in, represents the rotation matrix, T 1 represents the translation matrix, T 2 is the measured value from the origin of the world coordinate system, Represents a world coordinate point, Represents a point in the camera coordinate system.

Images