Automatic container stacking method based on vision system

Abstract

The present invention provides a method for automatic stacking of containers based on a vision system, comprising the following steps: S1: installing cameras on the four corners of a sling, respectively, and each camera respectively acquiring an image of a container corner area at a corresponding position below; S2: acquiring the intrinsic parameter matrix, distortion parameters, and posture information of the camera in a fixed state through the Zhang Zhengyou calibration method, and constructing a mapping relationship between coordinate points in the two-dimensional coordinate system of the camera image and the world coordinate system; S3: training a box edge segmentation model using a deep learning-based method, and obtaining straight lines and box corner feature points of the box edge of the camera image after iterative learning and optimization; S4: under the working condition of automatic container stacking, using the trained box edge segmentation model to obtain the box corner feature points of the grabbed container and the box corner feature points of the container to be stacked, obtaining a positioning result and converting it into the world coordinate system; S5: the sling executes an automatic stacking instruction.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic crane control, and in particular to an automatic container stacking method based on a vision system. Background Art

[0002] With the advancement of economic globalization, the role and status of modern ports in socioeconomic development have undergone profound changes. Various solutions to improve operational efficiency have been proposed and implemented, and port operation automation has gradually become a trend. Traditional container lifting is difficult and requires experienced operators. Currently, all ports are moving towards automation and intelligentization, and fully automatic container stacking is a core feature of this.

[0003] Although some ports have adopted yard automation solutions, there are still shortcomings such as difficulty in calibration, easy deviation of large vehicle benchmarks, inconvenience in detecting the current posture of the spreader, and time-consuming posture adjustment, which prevents effective improvement in work efficiency. Existing automatic stacking functions are mostly based on lidar solutions, which guide the spreader to stack boxes by analyzing point cloud data. The Chinese patent application with publication number CN113902814A discloses a unified calibration method for multiple cameras of automated crane spreaders. It is necessary to convert the coordinate system of each spreader camera into the same world coordinate system, which has the disadvantages of large computational complexity, high system resource usage and low efficiency. In order to improve the degree of automation of containers and improve the efficiency of container stacking, the present invention proposes an automatic container stacking method based on a visual system. Summary of the Invention

[0004] In view of this, the present invention proposes a method for automatic container stacking based on a vision system, which has a simple structure, convenient calibration and high precision.

[0005] The technical solution of the present invention is implemented as follows: The present invention provides a method for automatically stacking containers based on a vision system, comprising the following steps:

[0006] S1: Cameras are installed at the four corners of the spreader, pointing toward the ground. The four cameras capture images of the container corners at the corresponding locations below and output the images.

[0007] S2: Use the Zhang Zhengyou calibration method to obtain the camera's intrinsic parameter matrix, distortion parameters, and pose information in a fixed state, and construct the mapping relationship between the coordinate points in the two-dimensional coordinate system in the camera image and the world coordinate system;

[0008] S3: After the camera captures an image containing the box corner area, the box edge area to be segmented is manually annotated. Then, a box edge segmentation model is trained using a deep learning-based method. After iterative learning and optimization, the box edge segmentation model is used to segment the box edge area of ​​the box corner area image. The segmented box edge point set is fitted to obtain the box edge straight line and box corner feature points.

[0009] S4: During automatic container stacking, when the spreader reaches the set height above the container to be stacked, each camera captures an image containing the container corners. The trained edge segmentation model is used to obtain the corner feature points of the grasped container and the container to be stacked, and the positioning results are obtained. The corner feature points corresponding to the positioning results are converted to the world coordinate system.

[0010] S5: The spreader executes the automatic stacking instruction, selects two positioning results from the four cameras, obtains the offset between the grabbed container and the container to be stacked based on the two positioning results, adjusts the position and posture of the spreader based on the offset, and determines whether the deviation between the corner feature points of the grabbed container and the container to be stacked under the adjusted spreader posture exceeds the set threshold. The spreader then lands the container or makes further adjustments based on the deviation result.

[0011] On the basis of the above technical solution, preferably, in step S1, cameras are installed on the four corners of the spreader respectively so that the corner area of ​​the container to be stacked is located at the center of the camera image, and the extension direction of the camera image is parallel to the extension direction of the edge contour of the container to be stacked.

[0012] Preferably, in step S2, the intrinsic parameter matrix, distortion parameters and posture information of the camera in a fixed state are obtained by the Zhang Zhengyou calibration method, so that the chessboard covers the entire screen of each camera, and the horizontal axis direction of the chessboard is placed according to the running direction of the vehicle and is located at the center of the camera field of view; let the coordinates of point P in the world coordinate system be [X, Y, Z], the coordinates of the imaging point P' in the camera coordinate system be [X', Y', Z'], the focal length of the camera is f, the pixel coordinate system is O'-uv, the pixel coordinates are scaled by α times on the u axis and β times on the v axis, and the origin of the pixel coordinate system is translated by [u0, v0] T , according to Zhang Zhengyou calibration method, the camera internal parameter M and distortion coefficient K are obtained. where f s =αf,f y =βf, and the rotation matrix R and translation matrix T are obtained; after the pixel coordinate point (u, v) is corrected according to the distortion coefficient K, its coordinates in the world coordinate system are obtained by the following mapping relationship: Where s is the scale factor from the world coordinate system to the pixel coordinate system.

[0013] Further preferably, when the intrinsic parameter matrix, distortion parameters and posture information of the camera in a fixed state are obtained by the Zhang Zhengyou calibration method in step S2, the height of the sling is consistent with the height of the stacking box adjustment.

[0014] Further preferably, when the intrinsic parameter matrix, distortion parameters and pose information of the camera in a fixed state are obtained by the Zhang Zhengyou calibration method in step S2, the four cameras are calibrated separately, and the zero point of the world coordinate system corresponding to each camera corresponds to the zero point position of the chessboard when calibrating the external parameters.

[0015] Further preferably, step S3 uses a deep learning-based method to train the box edge segmentation model. After iterative learning and optimization, the box edge segmentation model is used to segment the box edge area of ​​the box corner area image, and the segmented box edge point set is fitted to obtain the straight line and box corner feature points of the box edge. The box edge segmentation model is trained based on the semantic segmentation method; the box edge segmentation model segments the edge of the container to obtain a point set of the box edge mask, and the boundary of the box edge mask is fitted with a straight line of two adjacent box edges of the container using a straight line fitting method. The intersection of the two straight lines of the box edges is the box corner feature point.

[0016] Preferably, in step S5, the spreader executes an automatic stacking instruction, selects two positioning results from four cameras, obtains an offset between the grabbed container and the container to be stacked based on the two positioning results, and adjusts the position and posture of the spreader based on the offset. When the spreader drives the grabbed container to move to the height adjusted for stacking, two corner feature points in the diagonal direction of the grabbed container are selected to form a first straight line; two corner feature points in the diagonal direction of the container to be stacked are obtained to form a second straight line; and the translation distance and rotation angle of the corner feature points at the endpoints of the first straight line relative to the corner feature points at the endpoints of the second straight line in the world coordinate system are calculated, so that the distance or angle between the ground projections of the corner feature points at the endpoints of the first straight line and the corner feature points at the endpoints of the second straight line is minimized, and the area of ​​the container to be stacked exceeding the rectangular area enclosed by the corner feature points of the grabbed container is minimized.

[0017] Further preferably, the threshold value is set such that the horizontal length extension direction of the container to be stacked is the lateral direction, the horizontal width extension direction of the container to be stacked is the longitudinal direction, and the lateral allowable deviation between the corresponding box edges of the grabbed container and the container to be stacked does not exceed 20 mm, and the longitudinal allowable deviation does not exceed 30 mm.

[0018] On the basis of the above technical solution, preferably, the containers to be stacked or the containers to be grabbed are all 20-foot containers or 40-foot containers.

[0019] The present invention provides a method for automatically stacking containers based on a vision system, which has the following advantages over the prior art:

[0020] (1) This solution is based on a deep learning visual positioning method to segment and locate the edges of the container being grabbed and the container to be stacked under the spreader. The offset required to adjust the spreader is calculated by combining the results of multiple cameras. By continuously adjusting the position and posture of the spreader, the spreader can automatically stack the containers below. The calibration process is simple. The four cameras can be calibrated separately and can be used without unifying the four cameras to the same world coordinate system. Using four cameras can avoid the problem of some corners of the container being blocked, thereby improving the fault tolerance of the system positioning.

[0021] (2) This solution has strong real-time performance in segmenting box edges and locating box corners. Using GPU real-time inference, it can continuously judge the results of multiple frames of images in a short period of time. Through simple filtering processing, the robustness of the system is further enhanced.

[0022] (3) By optimizing effective positioning results and a reasonable spreader offset determination threshold, the accuracy of box stacking after the spreader is lowered is ensured, and the frequency of manual intervention is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 This is a flow chart of a method for automatic container stacking based on a vision system according to the present invention;

[0025] Figure 2 This is a schematic diagram of the installation position of the spreader camera of the automatic container stacking method based on the vision system of the present invention;

[0026] Figure 3 This is a schematic diagram of the locations of corner feature points of a grabbed container and a container to be stacked in an automatic container stacking method based on a vision system according to the present invention;

[0027] Figure 4 This is another schematic diagram of the positions of the corner feature points of the container in the automatic container stacking method based on the vision system of the present invention;

[0028] Figure 5 This is a schematic diagram of calculating the translation amount of a spreader in a method for automatically stacking containers based on a visual system according to the present invention;

[0029] Figure 6 This is a schematic diagram of calculating the rotation amount of the spreader in the automatic container stacking method based on the vision system of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0031] like Figure 1 As shown, the present invention provides a method for automatically stacking containers based on a vision system, which specifically includes the following steps:

[0032] S1: Cameras are mounted at each corner of the spreader, pointing toward the ground. Each camera captures an image of the container corner at the corresponding location below. The corner of the container to be stacked is always centered in the camera image, and the camera image extends parallel to the edge of the container to be stacked. The image is then output. Specifically, a checkerboard pattern is placed at the center of the container placement area, with two pairs of orthogonal edges at the vertices of the checkerboard pattern. The edges of the checkerboard pattern are parallel to the edges of the container. The edges of the plane images captured by the cameras are kept as parallel as possible to the edges of the container to be stacked, with the container to be stacked centered in each camera's image.

[0033] S2: The Zhang Zhengyou calibration method is used to obtain the intrinsic parameter matrix, distortion parameters, and pose information of the camera in a fixed state, and to construct the mapping relationship between the coordinate points in the two-dimensional coordinate system of the camera image and the world coordinate system.

[0034] The specific process of calibration is to make the chessboard cover the entire screen of each camera, and the horizontal axis of the chessboard is placed according to the direction of the vehicle's movement and is located at the center of the camera's field of view; let the coordinates of point P in the world coordinate system be [X, Y, Z], the coordinates of the imaging point P' in the camera coordinate system be [X', Y', Z'], the focal length of the camera is f, the pixel coordinate system is O'-uv, and the pixel coordinates are scaled by α times on the u axis and β times on the v axis. At the same time, the origin of the pixel coordinate system is translated by [u0, v0] T , according to Zhang Zhengyou calibration method, the camera internal parameter M and distortion coefficient K are obtained. where f x =αf,f y =βf, and the rotation matrix R and translation matrix T are obtained. After the point (u, v) in the pixel coordinate system is corrected according to the distortion coefficient K, its coordinates in the world coordinate system are obtained by the following mapping relationship: Where s is the scale factor from the world coordinate system to the pixel coordinate system. The scale factor s can be calculated by setting Z = 0 in the above mapping relationship. The calibration process is the process of transforming the planar pixel points in the pixel coordinate system to the camera coordinate system, and finally to the world coordinate system.

[0035] During this calibration step, the spreader height is aligned with the stacking height. This means that the stacking heights are adjusted differently for stacking containers when the spreader is loading a container on the ground without a container, loading a container on top of a single layer of containers, or loading a container on top of more than one layer of containers. These heights are all set to the same preset values.

[0036] It should be noted that when this scheme adopts Zhang Zhengyou calibration method to obtain the intrinsic parameter matrix, distortion parameters and posture information of the camera in a fixed state, the four cameras installed on the sling are calibrated separately, and the zero point of the world coordinate system corresponding to each camera corresponds to the zero point position of the chessboard when calibrating the extrinsic parameters. There is no need to correspond all camera coordinate systems to the same world coordinate system, that is, there is no need to consider the relative positions and postures between different cameras, which can simplify the amount of calculation and improve the calibration efficiency.

[0037] S3: After the camera captures an image containing the box corner area, the box edge area to be segmented is manually labeled, and then a box edge segmentation model is trained using a deep learning-based method. After iterative learning and optimization, the box edge segmentation model is used to segment the box edge area of ​​the box corner area image, and the segmented box edge point set is fitted to obtain the box edge straight line and box corner feature points.

[0038] Specifically, a semantic segmentation method is used to train a box edge segmentation model; the box edge segmentation model segments the edge of the container to obtain a point set of the box edge mask, and a straight line fitting method is used to fit the boundary of the box edge mask to the two adjacent box edges of the container. The intersection of the two box edge straight lines is the box corner feature point.

[0039] S4: During automatic container stacking, when the spreader reaches the set height above the container to be stacked, each camera captures an image containing the corner area. The trained edge segmentation model is used to obtain the corner feature points of the grasped container and the container to be stacked, and the positioning results are obtained. The corner feature points corresponding to the positioning results are converted to the world coordinate system.

[0040] In step S4, each camera is mapped to two distinct corner feature points: one for the container being grasped, and the other for the container to be stacked. Locating these two corner feature points ensures the accuracy of subsequent automatic stacking instructions. Analysis using multiple consecutive image frames has two advantages: first, it eliminates the impact of jumps in the detection results of the deep learning-trained container edge segmentation model; second, it reduces detection errors caused by the combined shaking of the camera and spreader.

[0041] S5: The spreader executes the automatic stacking instruction, selects two positioning results from the four cameras, obtains the offset between the grabbed container and the container to be stacked based on the two positioning results, adjusts the position and posture of the spreader based on the offset, and determines whether the deviation between the corner feature points of the grabbed container and the container to be stacked under the adjusted spreader posture exceeds the set threshold. The spreader then lands the container or makes further adjustments based on the deviation result.

[0042] The specific process of step S5 is that when the spreader drives the grabbed container to move to the specified stacking adjustment height, such as when there is no container on the ground, when the container is placed on the top of a single-layer container, or when the container is placed on the top of more than one layer, there are four groups of eight different box corner feature points due to the four cameras; the box corner feature points in the diagonal direction are preferentially selected because only two positions are needed to determine the positioning result for calculating the offset, and secondly, the cameras on the diagonal line have the largest spacing, which can ensure a better stacking effect. Only when the positioning results of the two groups of diagonal cameras are not ideal, are two groups of box corner feature points adjacent to each other in a certain edge direction considered for calculating the positioning result.

[0043] When the two corner feature points of the captured container in the diagonal direction are selected, that is, Figure 5 The points P1 and P2 shown in the figure are used to form a first straight line; the two corner feature points C1 and C2 in the diagonal direction of the container to be stacked are obtained to form a second straight line; the translation distance and rotation angle of the corner feature points at the end points of the first straight line relative to the corner feature points at the end points of the second straight line in the world coordinate system are calculated, so that the distance or angle between the ground projections of the corner feature points at the end points of the first straight line and the corner feature points at the end points of the second straight line is minimized, and the area of ​​the container to be stacked exceeding the rectangular area enclosed by the corner feature points of the grabbed container is minimized.

[0044] like Figure 5As shown, obtaining the translation distance and rotation angle of the corner feature point at the endpoint of the first straight line relative to the corner feature point at the endpoint of the second straight line in the world coordinate system is performed in two steps. First, the translation distance between the corner feature points P1 and C1 is determined. Let the horizontal length of the container to be stacked be the x-axis direction, the horizontal width be the y-axis direction, the first straight line intersects the second straight line, and the two endpoints of the first straight line are translated in a direction equivalent to the distance between the two endpoints of the second straight line, so that Figure 5 The first and second straight lines in the figure satisfy dx1=-dx2, dy1=-dy2 at both ends. The first and second straight lines after translation are as follows: Figure 6 As shown, the corner feature points C1 and C2 at the endpoints of the second straight line remain unchanged, and the corner feature points at the endpoints of the first straight line after translation are P10 and P20. The angle θ between the first and second straight lines is calculated based on their slopes, that is, the rotation angle of the grabbed container relative to the container to be stacked. The rotating spreader causes the grabbed container to rotate in the direction of reducing the angle θ.

[0045] Step S5 mentions setting thresholds, where the horizontal length of the container to be stacked is defined as the lateral direction, and the horizontal width of the container to be stacked is defined as the longitudinal direction. The allowable lateral deviation between the edges of the grasped container and the corresponding edges of the container to be stacked is no more than 20 mm, and the allowable longitudinal deviation is no more than 30 mm. If, after the aforementioned translation and rotation operations, the allowable deviation between the edges of the grasped container and the corresponding edges of the container to be stacked does not exceed the set threshold, the grasped container is vertically lowered, completing automatic stacking. If, after the aforementioned translation and rotation operations, the allowable deviation between the edges of the grasped container and the corresponding edges of the container to be stacked does exceed the set threshold, after a period of waiting, the corner feature points of the grasped container and the container to be stacked are repositioned, and the translation and rotation operations are attempted again. Automatic landing is completed until the deviation does not exceed the set threshold, or if the maximum number of retries is reached and still cannot be less than the set threshold, the automatic stacking is terminated and an alarm signal is issued.

[0046] In this solution, the containers to be stacked or the containers to be grabbed are both 20-foot containers or 40-foot containers. That is, the specifications of the containers to be stacked and the containers to be grabbed are the same.

[0047] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. The automatic container stacking method based on the vision system is characterized by: The steps include: S1: Cameras are installed at the four corners of the spreader, pointing toward the ground. The four cameras capture images of the container corners at the corresponding locations below and output the images. S2: Use the Zhang Zhengyou calibration method to obtain the camera's intrinsic parameter matrix, distortion parameters, and pose information in a fixed state, and construct the mapping relationship between the coordinate points in the two-dimensional coordinate system in the camera image and the world coordinate system; S3: After the camera captures an image containing the box corner area, the box edge area to be segmented is manually annotated. Then, a box edge segmentation model is trained using a deep learning-based method. After iterative learning and optimization, the box edge segmentation model is used to segment the box edge area of ​​the box corner area image. The segmented box edge point set is fitted to obtain the box edge straight line and box corner feature points. In step S3, the box edge segmentation model is trained using a deep learning-based method. After iterative learning and optimization, the box edge segmentation model is used to segment the box edge area of ​​the box corner area image, and the segmented box edge point set is fitted to obtain the straight line and box corner feature points of the box edge. The box edge segmentation model is trained based on the semantic segmentation method; the box edge segmentation model segments the edge of the container to obtain a point set of the box edge mask, and the boundary of the box edge mask is fitted with a straight line of two adjacent box edges of the container using a straight line fitting method. The intersection of the two straight lines of the box edge is the box corner feature point; S4: During automatic container stacking, when the spreader reaches the set height above the container to be stacked, each camera captures an image containing the container corners. The trained edge segmentation model is used to obtain the corner feature points of the grasped container and the container to be stacked, and the positioning results are obtained. The corner feature points corresponding to the positioning results are converted to the world coordinate system. S5: The spreader executes the automatic stacking command and selects two positioning results from the four cameras. Based on the two positioning results, it obtains the offset between the grasped container and the container to be stacked. The position and posture of the spreader are adjusted according to the offset. It is determined whether the deviation between the corner feature points of the grasped container and the container to be stacked in the adjusted spreader posture exceeds the set threshold. The spreader then lands the container or makes further adjustments based on the deviation result. In step S5, the spreader executes the automatic stacking instruction, selects two positioning results from four cameras, obtains the offset between the grabbed container and the container to be stacked based on the two positioning results, and adjusts the position and posture of the spreader based on the offset. When the spreader drives the grabbed container to move to the height adjusted for stacking, two corner feature points in the diagonal direction of the grabbed container are selected to form a first straight line; two corner feature points in the diagonal direction of the container to be stacked are obtained to form a second straight line; the translation distance and rotation angle of the corner feature points at the endpoints of the first straight line relative to the corner feature points at the endpoints of the second straight line in the world coordinate system are calculated, so that the distance or angle between the ground projections of the corner feature points at the endpoints of the first straight line and the corner feature points at the endpoints of the second straight line is minimized, and the area of ​​the container to be stacked exceeding the rectangular area enclosed by the corner feature points of the grabbed container is minimized.

2. The automatic container stacking method based on a vision system according to claim 1 is characterized in that: In step S1, cameras are installed at the four corners of the spreader so that the corner area of ​​the container to be stacked is located at the center of the camera image, and the extension direction of the camera image is parallel to the extension direction of the edge contour of the container to be stacked.

3. The automatic container stacking method based on a vision system according to claim 2 is characterized in that: In step S2, the intrinsic parameter matrix, distortion parameters and pose information of the camera in a fixed state are obtained by the Zhang Zhengyou calibration method, so that the chessboard covers the entire screen of each camera, and the horizontal axis direction of the chessboard is placed according to the direction of the vehicle's movement and is located at the center of the camera's field of view; let the coordinates of point P in the world coordinate system be [X, Y, Z], the coordinates of the imaging point P' in the camera coordinate system be [X', Y', Z'], the focal length of the camera is f, the pixel coordinate system is O'-uv, and the pixel coordinates are scaled by α times on the u axis and β times on the v axis. At the same time, the origin of the pixel coordinate system is translated by [u0, v0] T , according to Zhang Zhengyou calibration method, the camera internal parameter M and distortion coefficient K are obtained. where f x =αf,f y =βf, and the rotation matrix R and translation matrix T are obtained; after the pixel coordinate point (u, v) is corrected according to the distortion coefficient K, its coordinates in the world coordinate system are obtained by the following mapping relationship: Where s is the scale factor from the world coordinate system to the pixel coordinate system.

4. The automatic container stacking method based on a vision system according to claim 3 is characterized in that: When the intrinsic parameter matrix, distortion parameters and posture information of the camera in a fixed state are obtained by the Zhang Zhengyou calibration method in step S2, the height of the sling is consistent with the height of the stacking box adjustment.

5. The automatic container stacking method based on a vision system according to claim 3 is characterized in that: When the Zhang Zhengyou calibration method is used to obtain the intrinsic parameter matrix, distortion parameters, and pose information of the camera in a fixed state in step S2, the four cameras are calibrated separately, and the zero point of the world coordinate system corresponding to each camera corresponds to the zero point position of the chessboard when calibrating the extrinsic parameters.

6. The automatic container stacking method based on a vision system according to claim 1, characterized in that: The threshold values ​​are set such that the horizontal length of the container to be stacked is defined as the lateral direction, the horizontal width of the container to be stacked is defined as the longitudinal direction, and the lateral deviation between the corresponding edges of the container to be grasped and the container to be stacked is allowed to be no more than 20 mm, and the longitudinal deviation is allowed to be no more than 30 mm.

7. The automatic container stacking method based on a vision system according to claim 1 is characterized in that: The containers to be stacked or the containers to be grabbed are all 20-foot containers or 40-foot containers.

Images