A shore-based bridge container arrangement adaptive positioning method

Abstract

The application provides a shore bridge ship container arrangement adaptive positioning method, comprising the following steps: installing a depth camera on a gantry beam and installing a laser scanner at the bottom of a shore bridge trolley; establishing a reference coordinate system at a PLC; the depth camera acquires an RGB image and a depth image of a clamp plate region, and positions the clamp plate region in the RGB image; the clamp plate region in the RGB image is point by point to obtain three-dimensional coordinates in the depth camera coordinate system, to obtain a clamp plate point cloud of the depth camera, and then the clamp plate point cloud of the depth camera is uniformly converted into the reference coordinate system at the PLC; the ship body point cloud acquired by the laser scanner is converted into the reference coordinate system at the PLC, and the ship body point cloud generated by the laser scanner in the reference coordinate system is positioned and conditionally filtered to obtain the point cloud of the clamp plate part in the ship body point cloud generated by the laser scanner in the reference coordinate system at the PLC; and the position of the container to be placed in the running direction of the shore bridge trolley is further determined.

Description

Technical Field

[0001] This invention relates to the field of port automation control technology, and in particular to an adaptive positioning method for quay crane container arrangement. Background Technology

[0002] With the continuous development of port automation, automated quay crane operation systems are constantly being improved and upgraded. In traditional automated operations, laser scanners are often used to scan containers on the ship to obtain the position of the containers in the direction of the trolley, replacing human eyes. Then, given the position of each row of containers on the ship, the height of each row of containers below the spreader (left adjacent sill, main sill, right adjacent sill) is analyzed. Subsequently, the spreader's path is planned to complete the automated operation.

[0003] Many quay crane operations utilize automated path planning for spreader configurations, primarily employing laser scanners (such as the SICK511 or GLAXY LiDAR) to scan and model the containers on the quay crane vessel. This analysis determines the height of each row of containers below the spreader (left, main, and right sills), thus enabling automated path planning. This approach relies on the assumption that the positions of each row of containers on the ship are roughly fixed in the trolley direction. By locating the center of the first row of containers in the quay crane's trolley direction, and using the container width (2.43m), the position of each subsequent row in the trolley's direction is deduced. Then, given the known positions of each row of containers on the ship, the height of each row below the spreader (left, main, and right sills) is analyzed. However, this method of locating the positions of each row of containers in the trolley direction is not suitable for all ship types. In reality, inland waterway vessels come in all shapes and sizes, and their deck widths vary. In addition, the distance between the side of the vessel and the shore when it is moored is often inconsistent. This directly leads to the inconsistency in the position of the first row of containers on the ship in the direction of the trolley, and there may even be a large deviation. This affects the positioning of each row of containers on the ship in the direction of the trolley, and may even cause misalignment of the stack height of each row of containers on the ship.

[0004] Therefore, it is necessary to provide an adaptive positioning method for container placement on quay cranes to improve the problem of container positioning failures on board due to differences in ship type, ship deck, and distance between the quay and the ship, thereby improving the safety and accuracy of spreader path planning in automated quay crane operations. Summary of the Invention

[0005] In view of this, the present invention proposes an adaptive positioning method for quay crane container arrangement that identifies the boundaries of the clamping area of ​​different types of ships and facilitates the finding of the starting position of container arrangement.

[0006] The technical solution of this invention is implemented as follows: This invention provides an adaptive positioning method for quay crane container placement, comprising the following steps:

[0007] For data acquisition, a depth camera is installed on the gantry crane beam, and a laser scanner is installed at the bottom of the trolley on the shore top. The coordinate systems of the depth camera and the laser scanner are established respectively. A PLC is set up at the gantry crane, and a reference coordinate system is established at the PLC.

[0008] A depth camera acquires RGB and depth images of the clamping plate area, and locates the clamping plate area in the RGB image, wherein the clamping plate area is located at the edge of the hull;

[0009] By combining the depth image, the three-dimensional coordinates of the clamping area in the RGB image are obtained point by point in the depth camera coordinate system to obtain the clamping point cloud of the depth camera. Then, the clamping point cloud of the depth camera is uniformly transformed into the reference coordinate system at the PLC.

[0010] The point cloud of the ship hull acquired by the laser scanner is converted to the reference coordinate system at the PLC, and the point cloud of the clamp part in the point cloud of the ship hull generated by the laser scanner is obtained in the reference coordinate system at the PLC.

[0011] Based on the point cloud of the hull point cloud generated by the laser scanner in the reference coordinate system at the PLC, the position of the container to be placed in the direction of travel of the quay crane trolley is further determined.

[0012] Based on the above technical solutions, preferably, the location of the clamping plate region in the RGB image is achieved by using a deep learning semantic segmentation method to identify and segment the clamping plate region in the RGB image, thereby obtaining the pixel points of the clamping plate region in the RGB image.

[0013] Preferably, the step of combining the depth image and calculating the corresponding three-dimensional coordinates of the clamp region in the RGB image in the depth camera coordinate system to obtain the clamp point cloud of the depth camera is performed using the following formula: Where (u, v) are the pixels of the RGB image, [X C Y C Z C ] T It is the three-dimensional coordinate in the depth camera coordinate system, Z C f represents depth. x and f y Let (u0, v0) be the focal length along the x and y axes of the RGB image, (u0, v0) be the center of the RGB image, and M be the symbol of the intrinsic parameter matrix. The three-dimensional coordinates in the depth camera coordinate system are obtained by converting each pixel of the RGB image of the clamping area acquired by the depth camera, which is the clamping point cloud of the depth camera corresponding to the depth camera.

[0014] More preferably, the transformation of the depth camera's clamp point cloud to the reference coordinate system at the PLC is performed using the following transformation relationship: P plc =R c2plc ·P c +T c2plc P plc The point cloud is located in the reference coordinate system of the PLC, with dimensions of 3×1; R c2plc The rotation matrix from the point cloud of the clamp generated in the depth camera coordinate system to the point cloud in the reference coordinate system of the PLC, with dimensions of 3×3; P c The depth camera's cladding point cloud, generated for the depth camera, has dimensions of 3×1; T c2plc This is the translation matrix from the point cloud of the clamp in the depth camera coordinate system to the point cloud in the reference coordinate system of the PLC, with a dimension of 3×1.

[0015] More preferably, the conversion of the hull point cloud acquired by the laser scanner to the reference coordinate system at the PLC adopts the following conversion relationship: P plc =R s2plc ·P s +T s2plc P s The point cloud of the ship's hull generated by the laser scanner has a dimension of 3×1; R s2 p lc The rotation matrix is ​​3×3, representing the point cloud of the ship's hull in the laser scanner coordinate system and the point cloud in the reference coordinate system of the PLC. s2plc This is the translation matrix from the point cloud of the ship's hull in the laser scanner coordinate system to the point cloud in the reference coordinate system of the PLC, with a dimension of 3×1.

[0016] In a further preferred embodiment, the positioning and conditional filtering in the reference coordinate system involves transferring the point cloud of the clamping plate generated by the depth camera and the point cloud of the hull acquired by the scanner to the reference coordinate system at the PLC. Then, statistical filtering is performed on the point cloud of the clamping plate generated by the depth camera in the reference coordinate system at the PLC to remove noise. Next, its bounding box is obtained, and the bounding space is determined based on the bounding box. Finally, conditional filtering is performed on the point cloud of the hull in the reference coordinate system at the PLC along the X, Y, and Z axes. The point cloud obtained after filtering is the point cloud of the clamping plate portion in the point cloud of the hull generated by the laser scanner in the reference coordinate system at the PLC.

[0017] Preferably, the method also includes a fine positioning step; specifically, the point cloud of the hull part of the plate in the laser scanner generated in the reference coordinate system of the PLC is further fitted using a sampling consistency algorithm of a three-dimensional planar model, and the fitted interior points are used as the finely positioned point cloud of the plate part in the hull point cloud generated by the laser scanner in the reference coordinate system of the PLC.

[0018] Further preferably, the step of determining the position of the container to be placed in the direction of travel of the quay crane trolley based on the point cloud of the hull generated by the laser scanner in the reference coordinate system at the PLC involves performing bounding box extraction on the point cloud of the quay crane generated by the depth camera in the reference coordinate system at the PLC. Taking the Y-axis as the direction of movement of the quay crane trolley in the reference coordinate system at the PLC, the maximum Y-value of the point cloud within the bounding box space is the boundary position of the quay crane, denoted as Y. max Let D be the axial dimension of the container along the direction of travel of the quay crane trolley. Based on the container width, we can deduce the position of each row in the direction of travel of the quay crane trolley, and then we have R. n =Y max +(n-0.5)D,R n Let n be the coordinate of the direction of movement of the container on the quay crane trolley, and n represent the row number.

[0019] The adaptive positioning method for quay crane container arrangement provided by this invention has the following advantages compared with the prior art:

[0020] (1) This solution uses a combination of depth camera and laser scanner to reliably identify the non-container-placement plate area of ​​the ship's hull, and uses the boundary of the plate area as the starting position for container width arrangement, thus achieving the safety and reliability of container path planning.

[0021] (2) Deep learning semantic segmentation technology is used to train and identify the deck parts of the ship, improve the classification and identification of the plate areas of different ship edges, and improve the accuracy of classification. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the steps of an adaptive positioning method for quay crane container placement according to the present invention.

[0024] Figure 2 This is a schematic diagram showing the installation positions of the depth camera and laser scanner in the adaptive positioning method for quay crane container placement according to the present invention.

[0025] Figure 3 This is a top view of the hull of the quay crane container adaptive positioning method of the present invention.

[0026] Figure 4 This is a schematic diagram of the coordinate systems of the adaptive positioning method for quay crane container arrangement according to the present invention. Detailed Implementation

[0027] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0028] like Figure 1-4 As shown, the present invention provides an adaptive positioning method for quay crane container placement, comprising the following steps:

[0029] S1: Sensing data acquisition. A depth camera is installed on the gantry crane beam, and a laser scanner is installed at the bottom of the trolley on the shore. The coordinate systems of the depth camera and the laser scanner are established respectively. A PLC is set up at the gantry crane, and a reference coordinate system is established at the PLC.

[0030] S2: The depth camera acquires RGB and depth images of the clamping area, and locates the clamping area in the RGB image, which is located at the edge of the hull. The area between the clamping areas on both sides of the hull is the space for stacking containers. The clamping area cannot be used to place containers, so it is necessary to obtain its maximum boundary away from the ship's side.

[0031] The method described here for locating the clamping area in an RGB image is to use a deep learning semantic segmentation method to identify and segment the clamping area in the RGB image, thereby obtaining the pixel points of the clamping area in the RGB image.

[0032] Semantic segmentation is a fundamental task in computer vision. It involves categorizing visual input into different semantically interpretable classes to correspond to specific objects in the real world, understanding the image content at the pixel level, and assigning an object class to each pixel. Several mature databases exist for training semantic segmentation models, such as Pascal VOC2012, Cityscapes, and Pascal Cintext. The semantic segmentation method used in this solution is a conventional semantic segmentation technique, and semantic segmentation is common knowledge in the field for those skilled in the art.

[0033] S3: Combining the depth image, the three-dimensional coordinates of the clamping area in the RGB image are obtained point by point in the depth camera coordinate system to obtain the clamping point cloud of the depth camera. Then, the clamping point cloud of the depth camera is uniformly transformed into the reference coordinate system of the PLC. This solution involves three coordinate systems, namely the depth camera coordinate system, the laser scanner coordinate system and the reference coordinate system of the PLC. The obtained point clouds will all be transformed into the same reference coordinate system of the PLC.

[0034] This step, obtaining the point cloud of the depth camera's clamping device, involves conversion using the following formula: Where (u, v) are the pixels of the RGB image, [X C Y C Z C ] T It is the three-dimensional coordinate in the depth camera coordinate system, Z C f represents depth. x and f y Let (u0, v0) be the focal length along the x and y axes of the RGB image, (u0, v0) be the center of the RGB image, and M be the symbol of the intrinsic parameter matrix. The three-dimensional coordinates in the depth camera coordinate system are obtained by converting each pixel of the RGB image of the clamping area acquired by the depth camera.

[0035] Subsequently, the point cloud of the depth camera's clamping device is uniformly transformed to the reference coordinate system at the PLC using the following transformation relationship: P plc =R c2plc ·P c +T c2plc P plc The point cloud is located in the reference coordinate system of the PLC, with dimensions of 3×1; R c2plc The rotation matrix from the point cloud of the clamp generated in the depth camera coordinate system to the point cloud in the reference coordinate system of the PLC, with dimensions of 3×3; P c The depth camera's cladding point cloud, generated for the depth camera, has dimensions of 3×1; T c2plc This is the translation matrix from the point cloud of the clamp in the depth camera coordinate system to the point cloud in the reference coordinate system of the PLC, with a dimension of 3×1.

[0036] S4: Convert the hull point cloud acquired by the laser scanner to the reference coordinate system at the PLC, locate it in the reference coordinate system and perform conditional filtering to obtain the point cloud of the hull point cloud generated by the laser scanner in the reference coordinate system at the PLC for the clamp part.

[0037] Specifically, the conversion of the hull point cloud acquired by the laser scanner to the reference coordinate system at the PLC is performed using the following conversion relationship: P plc =R s2plc ·P s +T s2plc P s The point cloud of the ship's hull generated by the laser scanner has a dimension of 3×1; R s2plc The rotation matrix is ​​3×3, representing the point cloud of the ship's hull in the laser scanner coordinate system and the point cloud in the reference coordinate system of the PLC. s2plc P is the translation matrix from the point cloud of the ship's hull in the laser scanner coordinate system to the point cloud in the reference coordinate system of the PLC, with dimensions of 3×1; plc The explanation has already been given in step S3, and will not be repeated here.

[0038] The aforementioned conditional filtering involves transferring the point cloud of the hull generated by the depth camera and the point cloud of the ship's hull acquired by the scanner to the reference coordinate system at the PLC. Then, statistical filtering is performed on the point cloud of the hull generated by the depth camera in the reference coordinate system at the PLC to remove noise. Next, its bounding box is calculated, and the bounding space is determined based on the bounding box. Finally, conditional filtering is performed on the point cloud of the ship's hull in the reference coordinate system at the PLC along the X, Y, and Z axes. The resulting point cloud is the point cloud of the hull portion of the point cloud generated by the laser scanner in the reference coordinate system at the PLC.

[0039] The statistical filtering mentioned here, which filters out noise, calculates the average distance between each ship point cloud and its k nearest ship point clouds, makes the distance between the ship point cloud and non-current ship point clouds follow a Gaussian distribution, and, given the mean and variance, removes ship point clouds that are outside the variance.

[0040] The conditional filtering mentioned here selects the point cloud of the hull plate area by limiting the region of the hull point cloud in the X, Y and Z axes, and discards the hull point cloud outside the region.

[0041] As a preferred embodiment of this solution, in order to improve the positioning accuracy of the point cloud of the hull part of the clamping plate in the point cloud of the hull generated by the laser scanner in the reference coordinate system of the PLC, a fine positioning step can be further introduced. Specifically, the point cloud of the clamping plate in the point cloud of the hull generated by the laser scanner in the reference coordinate system of the PLC is further fitted using the sampling consistency algorithm of the three-dimensional planar model, and the fitted interior points are used as the finely positioned point cloud of the clamping plate in the point cloud of the hull generated by the laser scanner in the reference coordinate system of the PLC.

[0042] The sampling consistency algorithm for the 3D planar model specifically adopts a planar fitting method. Since the clamping plate is a planar region, the point cloud of the clamping plate needs to be fitted to a planar plane. The point cloud located in the fitting plane is used as the point cloud of the clamping plate in the hull point cloud generated by the laser scanner in the reference coordinate system of the PLC after fine positioning. The point cloud of the clamping plate located in the fitting plane is discarded.

[0043] S5: Based on the point cloud of the hull point cloud generated by the laser scanner in the reference coordinate system at the PLC, further determine the position of the container to be placed in the direction of travel of the quay crane trolley.

[0044] Based on the point cloud of the hull portion generated by the laser scanner in the reference coordinate system at the PLC, or the point cloud of the hull portion generated by the laser scanner in the reference coordinate system at the PLC obtained in the previous step after fine positioning, the position of the container to be placed in the running direction of the quay crane trolley is further determined. This involves extracting the bounding box of the quay plate point cloud generated by the depth camera in the reference coordinate system at the PLC. Using the Y-axis as the moving direction of the quay crane trolley in the reference coordinate system at the PLC, the maximum Y-value of the point cloud within the bounding box space is the boundary position of the quay plate, denoted as Y. max Let D be the axial dimension of the container along the direction of travel of the quay crane trolley. Based on the container width, we can deduce the position of each row in the direction of travel of the quay crane trolley, and then we have R. n =Y max +(n-0.5)D,R n Let n be the coordinate of the direction of movement of the container on the quay crane trolley, and n represent the row number.

[0045] As can be seen from the above steps, the specific content of this solution is as follows: Acquiring perceptual information such as RGB images, depth images, or 3D point clouds using a depth camera and a laser scanner; obtaining the ship's hull point cloud using 3D reconstruction with a depth camera, further obtaining the hull point cloud of the depth camera in the depth camera coordinate system; transforming the hull point cloud of the depth camera in the depth camera coordinate system and the ship's hull point cloud in the laser radar coordinate system to the reference coordinate system at the PLC through 3D calibration, generating the hull point cloud of the hull portion generated by the laser scanner in the reference coordinate system at the PLC, thereby obtaining the movement position and width information of the hull plates on the ship in the direction of the quay crane trolley in the reference coordinate system at the PLC, and then accurately obtaining the position of each row in the direction of movement of the quay crane trolley based on the container width, thus achieving accurate positioning of the container arrangement on the ship.

[0046] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for adaptive positioning of a container on a quay crane, characterized in that, The method comprises the following steps: Sensing data acquisition, installing a depth camera on the beam of the gantry crane and a laser scanner at the bottom of the shore-top bridge trolley, and establishing a depth camera coordinate system and a laser scanner coordinate system respectively; setting a PLC at the gantry crane and establishing a reference coordinate system at the PLC; The depth camera acquires an RGB image and a depth image of the clamping plate region, and positions the clamping plate region in the RGB image, which is located at the edge of the ship body; In combination with the depth image, the clamping plate region in the RGB image is point-by-point calculated for corresponding three-dimensional coordinates in the depth camera coordinate system to obtain clamping plate point cloud of the depth camera, and then the clamping plate point cloud of the depth camera is uniformly converted into the reference coordinate system at the PLC; The ship body point cloud acquired by the laser scanner is converted into the reference coordinate system at the PLC, and the clamping plate part of the ship body point cloud generated by the laser scanner in the reference coordinate system at the PLC is positioned and conditionally filtered; According to the clamping plate part of the ship body point cloud generated by the laser scanner in the reference coordinate system at the PLC, the position of the container to be placed in the running direction of the shore-top bridge trolley is further determined.

2. The method of claim 1, wherein, The positioning of the clamping plate region in the RGB image is to identify and separate the clamping plate region in the RGB image by using a deep learning semantic segmentation method, so as to obtain the pixel points of the clamping plate region in the RGB image.

3. The method of claim 2, wherein, The clamping plate region in the RGB image is converted into three-dimensional coordinates in the depth camera coordinate system point by point to obtain the clamping plate point cloud of the depth camera by using the following formula: Where (u, v) is a pixel point of the RGB image, [X C Y C Z C ] T is a three-dimensional coordinate in the depth camera coordinate system, Z C represents a depth, f x and f y are focal lengths of the x-axis and y-axis directions of the RGB image, (u0, v0) is the center of the RGB image, and M is an internal parameter matrix symbol; each pixel of the RGB image of the clamping plate region obtained by the depth camera is converted to obtain a three-dimensional coordinate in the depth camera coordinate system, i.e., the clamping plate point cloud of the corresponding depth camera of the depth camera.

4. The method of claim 3, wherein, The conversion of the clamping plate point cloud of the depth camera into the reference coordinate system at the PLC is performed using the following conversion relationship: P plc = R c2plc · P c + T c2plc , wherein P plc is the point cloud in the reference coordinate system at the PLC, with a dimension of 3x1; R c2plc is a rotation matrix of the clamping plate point cloud generated by the depth camera coordinate system to the point cloud in the reference coordinate system at the PLC, with a dimension of 3x3; P c is the clamping plate point cloud of the depth camera generated by the depth camera, with a dimension of 3x1; and T c2plc is a translation matrix of the clamping plate point cloud of the depth camera coordinate system to the point cloud in the reference coordinate system at the PLC, with a dimension of 3x1.

5. The method of claim 4, wherein, The conversion of the ship body point cloud acquired by the laser scanner into the reference coordinate system at the PLC is performed using the following conversion relationship: plc P s2plc = R s · P s2plc + T s , where P is the ship body point cloud generated by the laser scanner, with a dimension of 3x1; R s2plc is a rotation matrix of the ship body point cloud in the laser scanner coordinate system to the point cloud in the reference coordinate system at the PLC, with a dimension of 3x3; and T s2plc is a translation matrix of the ship body point cloud in the laser scanner coordinate system to the point cloud in the reference coordinate system at the PLC, with a dimension of 3x1.

6. The method of claim 5, wherein, The positioning and conditional filtering in the reference coordinate system are as follows: after the clamping plate point cloud generated by the depth camera and the ship body point cloud acquired by the scanner are all converted into the reference coordinate system at the PLC, the clamping plate point cloud generated by the depth camera in the reference coordinate system at the PLC is statistically filtered to filter out noise points, and then a bounding box is calculated, the surrounding space is determined according to the bounding box, and then the ship body point cloud in the reference coordinate system at the PLC is conditionally filtered in X, Y and Z three axes, and the point cloud obtained after the filtering is the clamping plate part of the ship body point cloud generated by the laser scanner in the reference coordinate system at the PLC.

7. The method of claim 6, wherein, Further comprising a fine positioning step; Specifically, the clamping plate part of the ship body point cloud generated by the laser scanner in the reference coordinate system at the PLC is further fitted by using a sampling consistency algorithm of a three-dimensional plane model, and the inliers after the fitting are taken as the clamping plate part of the ship body point cloud generated by the laser scanner in the reference coordinate system at the PLC after the fine positioning.

8. A method of positioning a container on a quay crane according to any one of claim 6 or claim 7, characterized in that, The point cloud of the clamp plate part in the ship body point cloud generated by the laser scanner under the reference coordinate system at the PLC further determines the position of the container to be placed in the direction of the shore top bridge trolley operation. The bounding box of the depth camera clamp plate point cloud generated by the depth camera under the reference coordinate system at the PLC is extracted, and in the reference coordinate system at the PLC, the Y axis is the moving direction of the shore top bridge trolley, the maximum Y value of the point cloud in the bounding box space range is the boundary position of the clamp plate, and is recorded as Y max ; the axial dimension of the container along the direction of the shore top bridge trolley operation is D, the position of each row in the direction of the shore top bridge trolley movement is inferred according to the width of the container, R n =Y max +(n-0.5)D, R n is the coordinate of the container in the direction of the shore top bridge trolley movement, and n represents the row number.

Images