A ship cabin unloading grabbing control method based on laser point cloud recognition

Abstract

The present application relates to the technical field of port transportation intelligence, in particular to a ship cabin unloading grabbing control method based on laser point cloud recognition, comprising: acquiring ship point cloud data of a target ship in real time; identifying the ship cabin and the cargo in real time according to the ship point cloud data, and calculating the optimal grabbed point coordinates of the cargo in the ship cabin; converting the optimal grabbed point coordinates of the cargo to the coordinate system of the ship unloader to obtain the grabbed point coordinates of the ship unloader; generating the unloading point coordinates of the ship unloader in real time according to the position of the target truck for loading the cargo; and controlling the ship unloader to perform the cargo grabbing of the target ship and the cargo loading of the target truck in real time according to the grabbed point coordinates and the unloading point coordinates of the ship unloader. The present application accurately identifies the positions of the ship cabin and the cargo through the ship point cloud data, calculates the optimal grabbed point coordinates of the ship unloader according to the cargo distribution, and further realizes the automatic grabbing control of the ship cabin cargo, thereby improving the accuracy and efficiency of the ship cabin unloading grabbing.

Description

Technical Field

[0001] This invention relates to the field of intelligent port transportation technology, specifically to a method for controlling unloading and grasping materials in ship holds based on laser point cloud recognition. Background Technology

[0002] Grab unloaders, as crucial unloading equipment at bulk cargo terminals, currently operate primarily using a combination of semi-automatic and manual modes, resulting in low levels of automation and unmanned operation. Existing grab unloader operations rely on drivers analyzing cargo accumulation and manually planning the unloading process based on their experience, leading to low efficiency and a high risk of errors. In today's global context of promoting artificial intelligence and intelligent manufacturing, the development of unmanned grab unloaders has become an industry trend.

[0003] To this end, Chinese Patent No. CN116161443A discloses "A Method for Unloading Ships Using an Unmanned Grab Bucket Ship Unloader", which includes: dividing the work area into several work units and selecting an initial work unit; a grab direction control strategy: controlling the grab bucket to reach the work unit, completing the work process, and determining whether there is a risk of the grab bucket being buried; a trolley direction control strategy: controlling the grab bucket to move step by step towards the sea to the next work unit; a trolley direction control strategy: dividing all longitudinal work combinations into two groups; wherein the grab bucket is controlled to move step by step from one work unit to other longitudinal work combinations within the group to which the work unit belongs, and then moves step by step to the other group, until all work units in the work area have completed unloading.

[0004] The existing solutions described above have, to some extent, addressed the low efficiency issues caused by conventional grab unloader operations relying heavily on driver experience for planning. However, these solutions still cannot accurately identify and locate the cargo position when controlling the unloading and grabbing of cargo from the hold, resulting in large errors and low efficiency. Therefore, improving the accuracy and efficiency of cargo unloading and grabbing is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, the technical problem to be solved by this invention is: how to provide a ship hold unloading and grasping control method based on laser point cloud recognition, which accurately identifies the position of the ship hold and cargo through ship point cloud data, and calculates the optimal grasping point coordinates of the unloader based on the cargo distribution, thereby realizing automated grasping control of cargo in the ship hold and improving the accuracy and efficiency of ship hold unloading and grasping.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A method for controlling unloading and grasping materials in a ship's hold based on laser point cloud recognition includes:

[0008] S1: Acquire point cloud data of the target vessel in real time;

[0009] S2: Identify the ship's hold and cargo in real time based on the ship's point cloud data, and calculate the optimal coordinates of the cargo in the hold to be grabbed;

[0010] S3: Transform the coordinates of the best grab point of the cargo to the coordinate system of the unloader to obtain the coordinates of the grab point of the unloader;

[0011] S4: Generate the unloading point coordinates of the unloader in real time based on the location of the target truck used to load the cargo;

[0012] S5: Based on the coordinates of the grabbing point and unloading point of the unloader, control the unloader in real time to perform cargo grabbing of the target vessel and cargo loading of the target truck.

[0013] Preferably, the optimal coordinates of the cargo's grabbing point are calculated using the following steps:

[0014] S201: Determine the coordinate system of the ship point cloud data;

[0015] S202: Preprocess the ship point cloud data, remove point cloud data that is not related to the ship hull, and realize two-dimensional gridding and binarization to obtain the corresponding ship hull grid map.

[0016] S203: Extract the hull outline from the hull grid map, and extract lines on the hull outline to obtain the corresponding hull outline lines.

[0017] S204: Calculate the tilt angle of the hull outline relative to the coordinate system of the point cloud data, and rotate the hull grid map according to the tilt angle so that the hull outline is parallel to the coordinate axis of the coordinate system.

[0018] S205: Perform a scan line scan on the rotated hull grid map to obtain the corresponding cabin boundary box;

[0019] S206: Rotate and restore the hull grid map that has determined the cabin boundary frame so that the hull outline maintains the original tilt angle with the coordinate axis of the coordinate system.

[0020] S207: Obtain cargo point cloud data within the ship compartment boundary box in the hull grid map, calculate the cargo height in each grid, and then select the center point coordinates of the grid with the highest cargo height as the optimal capture point coordinates.

[0021] Preferably, the ship point cloud data is preprocessed through the following steps:

[0022] S2021: Downsample the ship point cloud data to obtain downsampled point cloud data;

[0023] S2022: Remove all point cloud data except for the hull body from the downsampled point cloud data to obtain the hull point cloud data;

[0024] S2023: Perform two-dimensional gridding processing on the hull point cloud data to obtain the hull point cloud grid image;

[0025] S2024: Perform image binarization on the hull point cloud grid image to obtain the hull grid map.

[0026] Preferably, the hull grid map is first scanned from top to bottom to determine the upper and lower boundaries of the cabin. The upper boundary of the cabin is determined by black pixels above and white pixels below, and the lower boundary of the cabin is determined by white pixels above and below and black pixels above the boundary line. Then, the hull grid map is scanned from left to right to determine the left and right boundaries of the cabin. The left boundary of the cabin is determined by black pixels on the left and white pixels on the right, and the right boundary of the cabin is determined by white pixels on the left and black pixels on the right.

[0027] Preferably, the coordinates of the optimal grab point of the cargo are transformed to the coordinate system of the unloader through the following steps:

[0028] S301: A photosensitive reflective strip is installed at the end of the gripper of the ship unloader. The position A of the photosensitive reflective strip in the coordinate system of the ship unloader is calculated using the ship unloader's attitude sensor. i ;

[0029] S302: Using lidar to acquire ship point cloud data, identify the photosensitive reflective strip set at the end of the unloader's gripper claws, and calculate the position B of the photosensitive reflective strip in the lidar coordinate system. i ;

[0030] S303: Obtain multiple sets of position A i and position B i Calculate the coordinate transformation relationship T;

[0031] The formula is described as follows:

[0032] A i *T=B i (i = 0, 1, 2, ...);

[0033] S304: Transform the coordinates of the optimal grab point to the coordinate system of the unloader through coordinate transformation relationship T;

[0034] The formula is described as follows:

[0035] B z / T=A z ;

[0036] In the formula: Bz Indicates the optimal coordinates of the captured point (x) B ,y B A z B z Corresponding grab point coordinates (x A ,y A ).

[0037] Preferably, through multiple sets of positions A i and position B i The following calibration algorithm is constructed:

[0038]

[0039] A*T = B;

[0040] In the formula: A and B represent multiple sets of positions A i and position B i The matrix formed; i represents the group number; T represents the coordinate transformation relationship to be calibrated;

[0041] The coordinate transformation relationship T is calculated using the following formula:

[0042] T = (A T′ A) -1 A T′ B;

[0043] In the formula: T′ represents a multi-point exponential relationship.

[0044] Preferably, the corresponding load capacity is obtained through the basic information of the target truck. The number of times the unloader grabs cargo is calculated based on the load capacity of the target truck and the weight of the cargo fully loaded in the unloader's grab bucket. Then, the loaded amount is determined based on the number of times the target truck has been loaded. If not, the unloading point coordinates of the unloader are generated based on the point cloud data of the target truck. If so, cargo loading is not performed on the target truck.

[0045] Preferably, the coordinates of the unloading point of the target truck are calculated through the following steps:

[0046] S401: Determine the coordinate system of the target truck point cloud data;

[0047] S402: Preprocess the point cloud data of the target truck, remove point cloud data that is irrelevant to the truck body, and realize two-dimensional gridding and binarization to obtain the corresponding truck body grid map.

[0048] S403: Extract the vehicle outline from the vehicle grille map, and extract lines on the vehicle outline to obtain the corresponding vehicle outline line;

[0049] S404: Calculate the tilt angle of the carriage outline relative to the coordinate system of the point cloud data, and rotate the carriage grille map according to the tilt angle so that the carriage outline is parallel to the coordinate axis of the coordinate system.

[0050] S405: Perform a scan line scan on the rotated carriage grille map to obtain the corresponding carriage boundary box;

[0051] S406: Rotate and restore the car body grid map that has determined the car body boundary frame, so that the car body outline maintains the original tilt angle with the coordinate axis of the coordinate system.

[0052] S407: Obtain cargo point cloud data within the cargo box boundary in the cargo box grid map, calculate the cargo height in each grid, and then select the center point coordinates of the grid with the lowest cargo height as the optimal unloading point coordinates.

[0053] S408: Transform the coordinates of the optimal unloading point to the coordinate system of the unloader to obtain the coordinates of the unloader's unloading point.

[0054] Compared with existing technologies, the ship hold unloading and grasping control method based on laser point cloud recognition in this invention has the following advantages:

[0055] This invention uses ship point cloud data to identify the hold and cargo in real time and calculate the optimal coordinates of the cargo to be grasped. It also generates the unloading point coordinates of the unloader based on the target truck's location. The point cloud data accurately identifies and locates the hold and cargo, allowing for the calculation of the optimal cargo-grabbing coordinates. This enables faster planning of the unloader's grasping path and coordinates, automating cargo grasping and improving efficiency. Similarly, the point cloud data accurately identifies the truck and its compartment, quickly and accurately determining the unloading point coordinates, further improving unloading accuracy and efficiency.

[0056] This invention, based on calculating the optimal coordinates of the cargo's grabbing point, further transforms these coordinates into the coordinate system of the unloading machine to obtain the unloading machine's grabbing point coordinates. This coordinate system transformation allows the unloading machine to locate the cargo more quickly and accurately, reducing the time spent searching for and locating the cargo, thereby improving the efficiency of cargo grabbing and unloading from the ship's hold. Simultaneously, transforming the optimal grabbing point coordinates to the unloading machine's coordinate system ensures the consistency and compatibility of the grabbing point coordinate data, making the entire cargo grabbing and unloading control process smoother. Furthermore, precise coordinate transformation can avoid collisions and accidents during the grabbing process, improving the safety of the grabbing operation. The combination of coordinate transformation and automation enables a higher degree of automation and intelligence in cargo grabbing from the ship's hold, thereby reducing manual intervention and further improving the accuracy and efficiency of cargo grabbing and unloading from the hold. Attached Figure Description

[0057] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0058] Figure 1 The flowchart shows a method for controlling unloading and grasping materials in a ship's hold based on laser point cloud recognition.

[0059] Figure 2 A schematic diagram of point clouds representing ships;

[0060] Figure 3 The coordinate system for the laser point cloud of the ship's hull;

[0061] Figure 4 A flowchart for determining the optimal coordinates of the captured point;

[0062] Figure 5 A schematic diagram showing the segmentation of the ship's planar outline;

[0063] Figure 6 This is a schematic diagram of a two-dimensional rasterization.

[0064] Figure 7 This is a schematic diagram illustrating the effect of binarization of an image;

[0065] Figure 8 A schematic diagram for identifying ship cabins;

[0066] Figure 9 This is a schematic diagram of the ship's cabin outline;

[0067] Figure 10 A schematic diagram of the ship's cabin outline;

[0068] Figure 11 This is a schematic diagram of the rotated hull grid map.

[0069] Figure 12 and Figure 13 This is a schematic diagram of the ship's cabin boundary frame;

[0070] Figure 14 A schematic diagram of cargo point cloud;

[0071] Figure 15 This is a schematic diagram of the installation of video surveillance for a ship unloader. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0073] It should be noted that similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the figures, or the orientation or positional relationship commonly used when the product is in use. They are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance. In addition, the terms "horizontal," "vertical," etc., do not mean that the component is required to be absolutely horizontal or suspended, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted. In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0074] The following detailed explanation illustrates the specific implementation methods:

[0075] Example:

[0076] This embodiment discloses a method for controlling unloading and grasping materials in a ship's hold based on laser point cloud recognition.

[0077] like Figure 1As shown, the ship hold unloading and grasping control method based on laser point cloud recognition includes:

[0078] S1: (Using lidar) Real-time acquisition of point cloud data of the target vessel;

[0079] S2: Identify the ship's hold and cargo in real time based on the ship's point cloud data, and calculate the optimal coordinates of the cargo in the hold to be grabbed;

[0080] S3: Transform the coordinates of the best grab point of the cargo to the coordinate system of the unloader to obtain the coordinates of the grab point of the unloader;

[0081] S4: Generate the unloading point coordinates of the unloader in real time based on the location of the target truck used to load the cargo;

[0082] S5: Based on the coordinates of the grabbing point and unloading point of the unloader, control the unloader in real time to perform cargo grabbing of the target vessel and cargo loading of the target truck.

[0083] In this embodiment, the attitude information of each motion unit of the ship unloader is also acquired in real time, and the working attitude of the ship unloader is displayed in real time through a 3D model, so that managers can easily check the working status of the ship unloader.

[0084] This invention uses ship point cloud data to identify the hold and cargo in real time and calculate the optimal coordinates of the cargo to be grasped. It also generates the unloading point coordinates of the unloader based on the target truck's location. The point cloud data accurately identifies and locates the hold and cargo, allowing for the calculation of the optimal cargo-grabbing coordinates. This enables faster planning of the unloader's grasping path and coordinates, automating cargo grasping and improving efficiency. Similarly, the point cloud data accurately identifies the truck and its compartment, quickly and accurately determining the unloading point coordinates, further improving unloading accuracy and efficiency.

[0085] This invention, based on calculating the optimal coordinates of the cargo's grabbing point, further transforms these coordinates into the coordinate system of the unloading machine to obtain the unloading machine's grabbing point coordinates. This coordinate system transformation allows the unloading machine to locate the cargo more quickly and accurately, reducing the time spent searching for and locating the cargo, thereby improving the efficiency of cargo grabbing and unloading from the ship's hold. Simultaneously, transforming the optimal grabbing point coordinates to the unloading machine's coordinate system ensures the consistency and compatibility of the grabbing point coordinate data, making the entire cargo grabbing and unloading control process smoother. Furthermore, precise coordinate transformation can avoid collisions and accidents during the grabbing process, improving the safety of the grabbing operation. The combination of coordinate transformation and automation enables a higher degree of automation and intelligence in cargo grabbing from the ship's hold, thereby reducing manual intervention and further improving the accuracy and efficiency of cargo grabbing and unloading from the hold.

[0086] like Figure 2As shown, lidar may detect other objects outside the hull, which can interfere with subsequent steps. Therefore, the ship point cloud data needs to be preprocessed to remove other objects outside the hull from the lidar point cloud. First, the coordinate system of the lidar point cloud is determined, such as... Figure 3 As shown, the plane formed by the x-axis and z-axis is almost parallel to the ground, and this coordinate axis is fixed. Based on the approximate location of the ship, only points with x-axis coordinates between [-50000, 50000], y-axis coordinates between [-1500, -15000], and z-axis coordinates between [0, 30000] are retained, and all other points are removed.

[0087] Combination Figure 4 As shown, the present invention preprocesses ship point cloud data through the following steps:

[0088] S2021: Downsample the ship point cloud data to obtain downsampled point cloud data;

[0089] In this embodiment, the original ship point cloud data is downsampled using VoxelGrid (voxelized grid filter) provided by the pcl library to reduce the number of point clouds while preserving the shape features of the point clouds.

[0090] S2022: Remove all point cloud data except for the hull body from the downsampled point cloud data to obtain the hull point cloud data;

[0091] In this embodiment, based on the location characteristics of the lidar and the characteristics of the ship, the point cloud on land (i.e., z<0) and the point cloud outside the scene (i.e., x<-100000 or x>100000) can be removed, such as... Figure 5 As shown.

[0092] S2023: Perform two-dimensional gridding processing on the hull point cloud data to obtain the hull point cloud grid image;

[0093] In this embodiment, the original point cloud retains complete information, but the computational load is very high, making it unsuitable for directly calculating the optimal picking point. Therefore, the original point cloud is processed into a two-dimensional rasterization. Rasterization of the laser point cloud involves processing the area scanned by the lidar using a grid, with each grid representing a small area in space containing a portion of the point cloud. Two-dimensional rasterization projects the three-dimensional point cloud onto a plane.

[0094] The remaining point cloud is scaled down by a factor of 100 and projected onto the xz plane to form a rasterized map (i.e., the subsequent hull grid map). The highest point of the projected point cloud (i.e., the maximum value on the y-axis) is stored within each grid cell. An xz raster plane is constructed on the plane formed by the x and z axes. A schematic diagram of the rasterization is shown below. Figure 6As shown, the size is 300×1000, and each pixel represents a grid. If a point cloud exists within a grid, the pixel is black; if no point cloud exists within a grid, the pixel is white. Figure 6 Similar to the projection of the ship's hull onto the plane formed by the x-axis and z-axis.

[0095] S2024: Perform image binarization on the hull point cloud grid image to obtain the hull grid map.

[0096] In this embodiment, a binarized image of the same size as the grayscale image is first created, with a default pixel value of 255 (white). Based on the values ​​in the grid map, the pixel values ​​corresponding to grid cells with values ​​less than a certain threshold are set to 0 (black). Then, the pixels in the resulting binarized image are projected onto the u and v coordinate axes to obtain several line segments. Finally, based on the actual specifications of the hull, the location of the hull is determined by identifying line segments of corresponding lengths on the two coordinate axes, thus obtaining a hull grid map containing only the hull, such as... Figure 7 As shown.

[0097] The formula for creating a grayscale image is described as follows:

[0098]

[0099] This invention preprocesses ship point cloud data, removing point cloud data irrelevant to the ship hull, and then performs 2D gridding and binarization to obtain a ship hull grid map. The preprocessing step greatly simplifies the original point cloud data, retaining only data relevant to the ship hull, reducing the amount of data required for subsequent processing and improving processing speed. Simultaneously, removing irrelevant point cloud data reduces interference with ship hull identification and positioning, improving the accuracy of subsequent processing. Furthermore, 2D gridding and binarization convert the point cloud data into a more easily processed format, facilitating subsequent tasks such as ship hull identification and cargo positioning. In addition, the grid map provides a more intuitive display of the ship's shape and structure, facilitating visual analysis and monitoring, and significantly reduces the amount of data requiring computation, improving computational efficiency and enabling real-time processing.

[0100] Combination Figure 8 As shown, the optimal coordinates of the cargo's grabbing point are calculated using the following steps:

[0101] S201: Determine the coordinate system of the ship point cloud data;

[0102] S202: Preprocess the ship point cloud data, remove point cloud data that is not related to the ship hull, and realize two-dimensional gridding and binarization to obtain the corresponding ship hull grid map.

[0103] S203: Extract the hull outline from the hull grid map, and extract lines on the hull outline to obtain the corresponding hull outline lines.

[0104] In this embodiment, for the ship hull grid map, the `findContours` function provided by the OpenCV library can be used to obtain the outline of the ship hull, ignoring pixels inside the ship's interior. The result of extracting the ship hull outline is as follows. Figure 9 As shown.

[0105] S204: Calculate the tilt angle of the hull outline relative to the coordinate system of the ship point cloud data, and rotate the hull grid map according to the tilt angle so that the hull outline is parallel to the coordinate axis of the coordinate system.

[0106] In this embodiment, considering the complex pixel distribution of the original binarized image, the extracted lines are rather messy, which is not conducive to subsequent line processing. This invention utilizes the HoughLinesP function provided by the OpenCV library to extract lines from the ship's hull outline, achieving the following results: Figure 10 As shown, the lines are concentrated on the edge of the cabin, and the tilt angle theta of the ship's hull relative to the coordinate axis can be obtained based on the slope of the extracted lines.

[0107] Based on the obtained tilt angle, the warpAffine function provided by the OpenCV library is used to rotate the hull counterclockwise by theta degrees, making the hull and cabins parallel to the coordinate axes, which facilitates subsequent cabin recognition. The effect after rotation is as follows. Figure 11 As shown.

[0108] S205: Perform a scan line scan on the rotated hull grid map to obtain the corresponding cabin boundary box;

[0109] In this embodiment, for the rotated image, the boundaries of the cabin are locked sequentially from top to bottom and from left to right using scan lines. First, the hull grid map is scanned from top to bottom to determine the upper and lower boundaries of the cabin. The upper boundary of the cabin is determined by the fact that the upper part of the cabin is almost entirely black pixels and the lower part is almost entirely white pixels. The lower boundary of the cabin is determined by the fact that the upper and lower parts of the cabin are almost entirely white pixels and the boundary line is mostly black pixels. Then, the hull grid map is scanned from left to right to determine the left and right boundaries of the cabin. The left boundary of the cabin is determined by the fact that the left side is almost entirely black pixels and the right side is almost entirely white pixels. The right boundary of the cabin is determined by the fact that the left side is almost entirely white pixels and the right side is almost entirely black pixels.

[0110] The location of the ship's cabin can be determined based on the identified boundary positions. Finally, the ship's hull is rotated clockwise by theta degrees to obtain the pixel coordinates of the cabin in the original binarized image. Figure 12 and Figure 13 As shown, the cabin is represented by a rectangle.

[0111] S206: Rotate and restore the hull grid map that has determined the cabin boundary frame so that the hull outline maintains the original tilt angle with the coordinate axis of the coordinate system.

[0112] In this embodiment, the ship's hold is identified in the binarized image and represented as a rectangular frame. The coordinates of the ship's hold in the raster map can be determined by the pixel coordinates of the four vertices of the rectangular frame, and then restored to the world coordinate system to obtain the point cloud of the cargo inside the hold. Figure 14 As shown, the point cloud within the box represents the cargo.

[0113] S207: Obtain cargo point cloud data within the ship compartment boundary box in the hull grid map, calculate the cargo height in each grid, and then select the center point coordinates of the grid with the highest cargo height as the optimal capture point coordinates.

[0114] In this embodiment, each grid cell stores cargo height information. Therefore, it is only necessary to compare the cargo heights within all grid cells in the ship's hold area and select the grid cell with the highest height. The optimal grab point coordinates along the x and z axes are the center point of that grid cell, and the optimal grab point coordinates along the y axis are the cargo height information of that grid cell. Within the ship's hold area of ​​the grid map, the highest point P is found, with grid map coordinates of (x, z) and a maximum value of y. Since the previous steps reduced the remaining point cloud by a factor of 100, the coordinates of the point restored to the world coordinate system are (100x, y, 100z), which are the optimal grab point coordinates.

[0115] This invention first extracts the hull outline and hull outline lines, then calculates the tilt angle of the hull outline lines relative to the coordinate system of the ship's point cloud data, and rotates the hull grid map according to the tilt angle, making the hull outline lines parallel to the coordinate axes of the coordinate system. Finally, it performs scan line scanning on the rotated hull grid map to obtain the cabin boundary boxes. By extracting the hull outline and outline lines, the hull boundary can be located more accurately, thus improving the accuracy of the cabin boundary boxes. Simultaneously, rotating the hull grid map aligns the hull outline lines with the coordinate axes, simplifying subsequent scan line and boundary box calculations. Furthermore, using the scan line scanning method allows for rapid determination of the cabin boundary boxes, and accurate cabin boundary box data can be interfaced with other ship control systems or platforms, improving system compatibility and integration capabilities. In addition, the cabin boundary boxes obtained through scan line scanning can more clearly display the structure and shape of the cabins, providing basic data and information for subsequent tasks such as cabin identification and cargo positioning, facilitating the expansion of other functions and applications.

[0116] In the specific implementation process, the coordinates of the optimal grab point of the cargo are transformed to the coordinate system of the unloader through the following steps:

[0117] S301: A photosensitive reflective strip is installed at the end of the gripper of the ship unloader. The position A of the photosensitive reflective strip in the coordinate system of the ship unloader is calculated using the ship unloader's attitude sensor. i ;

[0118] S302: Using lidar to acquire ship point cloud data, identify the photosensitive reflective strip set at the end of the unloader's gripper claws, and calculate the position B of the photosensitive reflective strip in the lidar coordinate system. i ;

[0119] S303: Obtain multiple sets of position A i and position B i Calculate the coordinate transformation relationship T;

[0120] The formula is described as follows:

[0121] A i *T=B i (i = 0, 1, 2, ...);

[0122] S304: Transform the coordinates of the optimal grab point to the coordinate system of the ship unloader through coordinate transformation relationship T;

[0123] The formula is described as follows:

[0124] B z / T=A z ;

[0125] In the formula: B z Indicates the optimal coordinates of the captured point (x) B ,y B A z B z Corresponding grab point coordinates (x A ,y A ).

[0126] Specifically, through multiple sets of positions A i and position B i The following calibration algorithm is constructed:

[0127]

[0128] A*T = B;

[0129] In the formula: A and B represent multiple sets of positions A i and position B i The matrix formed; i represents the group number; T represents the coordinate transformation relationship to be calibrated;

[0130] Develop and optimize performance metrics (performance metrics are used to improve the accuracy of the gripping point coordinates in the global coordinate system):

[0131]

[0132] The coordinate transformation relationship T is calculated using the following formula:

[0133] T = (A T′ A) -1 A T′ B;

[0134] In the formula: T′ represents a multi-point exponential relationship, that is, multiple positions A i and position B i Having the same or similar exponential relationship is an existing concept, but in this invention it refers to a conversion formula between real-time coordinates and global coordinates.

[0135] In the specific implementation process, the corresponding load capacity is obtained through the basic information of the target truck. The number of times the unloader grabs cargo is calculated based on the load capacity of the target truck and the weight of the cargo fully loaded in the unloader's grab bucket. Then, the loaded amount is judged based on the number of times the target truck has been loaded to determine whether it is greater than the loading threshold. If not, the unloading point coordinates of the unloader are generated based on the point cloud data of the target truck. If so, cargo loading is not performed on the target truck.

[0136] Specifically, the coordinates of the target truck's unloading point are calculated using the following steps:

[0137] S401: Determine the coordinate system of the target truck point cloud data;

[0138] S402: Preprocess the point cloud data of the target truck, remove point cloud data that is irrelevant to the truck body, and realize two-dimensional gridding and binarization to obtain the corresponding truck body grid map.

[0139] S403: Extract the vehicle outline from the vehicle grille map, and extract lines on the vehicle outline to obtain the corresponding vehicle outline line;

[0140] S404: Calculate the tilt angle of the carriage outline relative to the coordinate system of the point cloud data, and rotate the carriage grille map according to the tilt angle so that the carriage outline is parallel to the coordinate axis of the coordinate system.

[0141] S405: Perform a scan line scan on the rotated carriage grille map to obtain the corresponding carriage boundary box;

[0142] S406: Rotate and restore the car body grid map that has determined the car body boundary frame, so that the car body outline maintains the original tilt angle with the coordinate axis of the coordinate system.

[0143] S407: Obtain cargo point cloud data within the cargo box boundary in the cargo box grid map, calculate the cargo height in each grid, and then select the center point coordinates of the grid with the lowest cargo height as the optimal unloading point coordinates.

[0144] S408: Transform the coordinates of the optimal unloading point to the coordinate system of the unloader to obtain the coordinates of the unloader's unloading point.

[0145] In the specific implementation process, the truck point cloud data of the target truck is obtained, and then the corresponding predicted load capacity is calculated based on the truck point cloud data. It is then determined whether the predicted load capacity is greater than the loading threshold. If so, the position coordinates of the target truck are used as the unloading point coordinates of the unloader; otherwise, cargo loading is not performed on the target truck.

[0146] In this embodiment, the loading threshold is used to determine whether the truck has sufficient loading capacity. The specific value can be set as needed, such as half of the truck's maximum capacity.

[0147] This invention acquires point cloud data of target trucks using lidar and calculates their predicted load capacity based on this data. This allows for the determination of whether a target truck has sufficient loading capacity, and if so, it is included in the cargo loading area, thereby improving the effectiveness of unloading and grabbing from the ship's hold. Simultaneously, the location of the target truck automatically determines the coordinates of the unloading point on the ship unloader, enabling rapid and accurate determination of the unloading location and further improving the unloading efficiency of the ship's hold.

[0148] In the specific implementation process, the location coordinates of the target truck are used as the unloading point coordinates of the unloading machine. Based on the predicted loading capacity of the target truck and the weight of the fully loaded cargo of the unloading machine, the number of times the unloading machine grabs cargo and the amount of cargo grabbed in a single operation are calculated. Then, based on the number of times the unloading machine grabs cargo and the amount of cargo grabbed in a single operation, the unloading machine is controlled in real time to perform cargo grabbing of the target ship and cargo loading of the target truck.

[0149] In this embodiment, the number of times the unloader grabs cargo and the amount of cargo grabbed per grab are calculated using the formula: "Predicted load capacity / Fully loaded cargo weight of the unloader = Number of cargo grabs (integer)". For example, if the predicted load capacity is 22 tons and the fully loaded cargo weight of the unloader is 5 tons, then the unloader will grab cargo 5 times, and the amounts of cargo grabbed per grab will be 5 tons, 5 tons, 5 tons, 5 tons, and 2 tons, respectively.

[0150] In other preferred embodiments, the load capacity information of the target truck can also be obtained through the truck's license plate (at which time the truck is considered empty), and the number of tasks and the weight of the unloader to grab for each task can be determined by the weight of the cargo fully loaded.

[0151] This invention calculates the number of times the unloader grabs cargo and the amount of cargo grabbed per grab based on the predicted load capacity of the target truck and the weight of the cargo fully loaded by the unloader. This allows the unloader to dynamically adjust the amount of cargo grabbed each time based on the predicted load capacity of the target truck, thereby completing the cargo loading as quickly as possible while meeting the loading capacity of the target truck, thus improving the cargo loading efficiency of unloading cargo from the ship's hold.

[0152] In the specific implementation process, the predicted load capacity of the target truck is calculated through the following steps:

[0153] S501: Preprocess the truck point cloud data of the target truck;

[0154] In this embodiment, preprocessing includes noise reduction, filling in missing data, and correcting geometric errors to improve data quality.

[0155] S502: Segment the cargo box and cargo portion from the preprocessed truck point cloud data to obtain cargo point cloud data;

[0156] In this embodiment, the truck and cargo parts can be segmented from point cloud data based on the size and shape of the truck by setting a threshold and using plane fitting.

[0157] S503: Extract cargo point cloud features from cargo point cloud data;

[0158] In this embodiment, the cargo point cloud features include the cargo's height, width, length, volume, etc., which can reflect the cargo's loading capacity.

[0159] S504: Input the cargo point cloud features into the trained cargo prediction model and output the corresponding predicted load capacity.

[0160] The cargo prediction model is built on a machine learning model or a deep learning model and is pre-trained using cargo point cloud feature samples and known actual load capacity.

[0161] In this embodiment, the cargo prediction model is built based on the BP neural network model, with cargo point cloud features as input and predicted loadable quantity as output.

[0162] This invention segments cargo boxes and cargo portions from preprocessed truck point cloud data to obtain cargo point cloud data. It then extracts cargo point cloud features from this data and constructs a cargo prediction model based on machine learning or deep learning models. The machine learning and deep learning models automatically extract useful features from the cargo point cloud data without human intervention, improving efficiency and enabling the extraction of complex features imperceptible to the human eye. Furthermore, the deep learning and machine learning modeling methods automatically explore and extract various cargo features such as shape, size, arrangement, and density. The diversity and completeness of these features contribute to improved prediction accuracy and stability. Moreover, machine learning and deep learning models can extract more complex nonlinear feature relationships from the cargo point cloud data, thereby improving the accuracy of truck load prediction.

[0163] In practical implementation, this invention also includes a safety mechanism for the ship unloader:

[0164] 1) Collision Avoidance: When planning the path of the ship unloader, trucks, ships, dikes, and water surfaces are taken into account as key collision avoidance targets. During the path tracking of the ship unloader, real-time obstacle detection is performed based on 3D point cloud data to predict collisions, pause the movement of the ship unloader in advance, and issue an alarm.

[0165] 2) Warning: When the ship unloader is in operation, there will be light prompts. A flashing green light indicates "The ship unloader is in operation, please do not approach", a flashing yellow light indicates "The ship unloader has encountered an obstacle and has stopped operation, requesting assistance for inspection", and a flashing red light indicates "Emergency stop operation".

[0166] 3) Emergency Braking: Dispatch room personnel can remotely apply the emergency brake to the ship unloader via a visual interface. The control tower where the ship unloader is located is equipped with multiple physical emergency brake buttons, which can be used to apply the emergency brake to the ship unloader.

[0167] In specific implementation, the present invention also includes the following control and execution steps for the ship unloader:

[0168] 1) Ship unloader location -> cargo grabbing point: The scheduling system plans the optimal movement path for the ship unloader based on the current coordinates of the ship unloader and the cargo grabbing point, as well as the 3D point cloud. The ship unloader tracks and stops obstacles in real time according to the path and the 3D point cloud data until it reaches the cargo grabbing point.

[0169] 2) Unloading machine grabbing cargo: The unloading machine grabs cargo by controlling the take-up and release lines.

[0170] 3) Ship unloader location -> cargo release point: The scheduling system plans the optimal movement path for the ship unloader based on the current coordinates of the ship unloader, the coordinates of the cargo release point, and the 3D point cloud. The ship unloader tracks and stops obstacles in real time according to the path and the 3D point cloud data until it reaches the cargo release point.

[0171] 4) Unloading cargo by ship unloader: The ship unloader unloads cargo by controlling the take-up and release lines.

[0172] 5) The ship unloader returns to the standby point or continues to perform the next loading and unloading task: The scheduling system plans the optimal movement path of the ship unloader based on the current coordinates of the ship unloader, the coordinates of the standby point / grabbing point, and the 3D point cloud. The ship unloader tracks and stops obstacles in real time according to the path and 3D point cloud data until it reaches the target point.

[0173] The entire crane's motion control is based on a PLC (Siemens), with a frequency converter as the execution unit, plus various sensors and fault protection.

[0174] In the specific implementation process, the monitoring camera installed on the ship unloader can be upgraded to a binocular vision 3D model, and then the path planning function can be realized based on the established on-site 3D model and the start of the hook.

[0175] The camera installation location is as follows: Figure 15 As shown: Camera 1 monitors the cargo and spreading equipment; Camera 2, located in the control room, monitors the operational status of various mechanisms; Camera 3 monitors the luffing and power-collecting mechanisms; Camera 4 assists Camera 1 in monitoring the cargo and ship; Camera 5 monitors the safety and direction of the slewing mechanism; Camera 6 monitors the driver's cab; Cameras 7 and 8 provide overall monitoring. Monitoring utilizes a dedicated fiber optic line to minimize latency between actual operation and video feed.

[0176] In specific implementation, the present invention also discloses the following personnel safety protection design:

[0177] During normal operation, no personnel are allowed to enter the unloader's operating area. The safety protection mechanism includes three modules: safety helmet recognition, personnel cleaning recognition, and hazardous behavior recognition. Binocular cameras are installed at selected points in each berth. Image recognition technology identifies personnel on the ground near the unloader and personnel cleaning the ship's hold. The identification results are fed back to the unloader's control system to ensure safe production during automated operation. When personnel are detected entering the operating area, the unloader automatically pauses its operation and issues an audible and visual alarm. After inspection personnel persuade the personnel to leave, the control system issues a command to resume operation.

[0178] In the latter stages of unloading, as cranes and other equipment are unable or have difficulty retrieving cargo from the hold, cleaning personnel are required to enter the hold for cleaning. During autonomous operation, the unloader must give way to these personnel to ensure safe operation. To guarantee the safety of the cleaning personnel inside the hold, their location needs to be identified. The unloader is equipped with binocular cameras to collect image data from inside the hold. This data is then transmitted to an AI analysis server. Using AI-powered image recognition technology, the server automatically identifies the number and location of the cleaning personnel inside the hold and transmits the results to the unloader's control system. Furthermore, the identified number is compared to the recorded number of cleaning personnel working on the ship. If a discrepancy occurs, a warning is issued and transmitted to the unloader's control system. A safety officer then confirms safety on-site before proceeding with the unloading operation.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A method for controlling unloading and grasping materials in a ship's hold based on laser point cloud recognition, characterized in that, include: S1: Acquire point cloud data of the target vessel in real time; S2: Identify the ship's hold and cargo in real time based on the ship's point cloud data, and calculate the optimal coordinates of the cargo in the hold to be grabbed; In step S2, the optimal coordinates of the cargo's grabbing point are calculated using the following steps: S201: Determine the coordinate system of the ship point cloud data; S202: Preprocess the ship point cloud data, remove point cloud data that is not related to the ship hull, and realize two-dimensional gridding and binarization to obtain the corresponding ship hull grid map. S203: Extract the hull outline from the hull grid map, and extract lines on the hull outline to obtain the corresponding hull outline lines. S204: Calculate the tilt angle of the hull outline relative to the coordinate system of the point cloud data, and rotate the hull grid map according to the tilt angle so that the hull outline is parallel to the coordinate axis of the coordinate system. S205: Perform a scan line scan on the rotated hull grid map to obtain the corresponding cabin boundary box; S206: Rotate and restore the hull grid map that has determined the cabin boundary frame so that the hull outline maintains the original tilt angle with the coordinate axis of the coordinate system. S207: Obtain cargo point cloud data within the ship compartment boundary box in the hull grid map, calculate the cargo height in each grid, and then select the center point coordinates of the grid with the highest cargo height as the best capture point coordinates. S3: Transform the coordinates of the best grab point of the cargo to the coordinate system of the unloader to obtain the coordinates of the grab point of the unloader; In step S3, the coordinates of the optimal grab point of the cargo are transformed to the coordinate system of the unloader through the following steps: S301: A photosensitive reflective strip is installed at the end of the gripper of the ship unloader. The position of the photosensitive reflective strip in the coordinate system of the ship unloader is calculated by the ship unloader's attitude sensor. ; S302: Using lidar to acquire ship point cloud data, identify the photosensitive reflective strip set at the end of the unloader's gripper claws and calculate the position of the photosensitive reflective strip in the lidar coordinate system. ; S303: Obtain multiple sets of locations and location Calculate coordinate transformation relationship ; The formula is described as follows: ）； S304: Through coordinate transformation relationships Transform the coordinates of the optimal grab point to the coordinate system of the ship unloader; The formula is described as follows: ； In the formula: Indicates the coordinates of the best point to be captured. ; express Corresponding grab point coordinates ; S4: Generate the unloading point coordinates of the unloader in real time based on the location of the target truck used to load the cargo; In step S4, the corresponding loadable capacity is obtained through the basic information of the target truck. The number of times the unloader grabs cargo is calculated based on the loadable capacity of the target truck and the weight of the cargo fully loaded in the unloader's grab bucket. Then, it is determined whether the loaded capacity is greater than the loading threshold based on the number of times the target truck has been loaded. If not, the unloading point coordinates of the unloader are generated based on the point cloud data of the target truck. If so, cargo loading is not performed on the target truck. In step S4, the coordinates of the unloading point of the target truck are calculated through the following steps: S401: Determine the coordinate system of the target truck point cloud data; S402: Preprocess the point cloud data of the target truck, remove point cloud data that is irrelevant to the truck body, and realize two-dimensional gridding and binarization to obtain the corresponding truck body grid map. S403: Extract the vehicle outline from the vehicle grille map, and extract lines on the vehicle outline to obtain the corresponding vehicle outline line; S404: Calculate the tilt angle of the carriage outline relative to the coordinate system of the point cloud data, and rotate the carriage grille map according to the tilt angle so that the carriage outline is parallel to the coordinate axis of the coordinate system. S405: Perform a scan line scan on the rotated carriage grille map to obtain the corresponding carriage boundary box; S406: Rotate and restore the car body grid map that has determined the car body boundary frame, so that the car body outline maintains the original tilt angle with the coordinate axis of the coordinate system. S407: Obtain cargo point cloud data within the cargo box boundary in the cargo box grid map, calculate the cargo height in each grid, and then select the center point coordinates of the grid with the lowest cargo height as the optimal unloading point coordinates. S408: Transform the coordinates of the optimal unloading point to the coordinate system of the unloader to obtain the coordinates of the unloader's unloading point; S5: Based on the coordinates of the grabbing point and unloading point of the unloader, control the unloader in real time to perform cargo grabbing of the target vessel and cargo loading of the target truck.

2. The ship hold unloading and grasping control method based on laser point cloud recognition as described in claim 1, characterized in that: In step S202, the ship point cloud data is preprocessed through the following steps: S2021: Downsample the ship point cloud data to obtain downsampled point cloud data; S2022: Remove all point cloud data except for the hull body from the downsampled point cloud data to obtain the hull point cloud data; S2023: Perform two-dimensional gridding processing on the hull point cloud data to obtain the hull point cloud grid image; S2024: Perform image binarization on the hull point cloud grid image to obtain the hull grid map.

3. The ship hold unloading and grasping control method based on laser point cloud recognition as described in claim 1, characterized in that: In step S205, the ship's grid map is first scanned from top to bottom to determine the upper and lower boundaries of the cabin. The upper boundary of the cabin is determined by black pixels above and white pixels below, and the lower boundary of the cabin is determined by white pixels above and below and black pixels above the boundary line. Then, the ship's grid map is scanned from left to right to determine the left and right boundaries of the cabin. The left boundary of the cabin is determined by black pixels on the left and white pixels on the right, and the right boundary of the cabin is determined by white pixels on the left and black pixels on the right.

4. The ship compartment unloading and grasping control method based on laser point cloud recognition as described in claim 1, characterized in that: In step S303, multiple sets of positions are used. and location The following calibration algorithm is constructed: ； ； In the formula: and Indicates multiple sets of positions and location The matrix formed; Indicates the group number; This indicates the coordinate transformation relationship to be calibrated; The coordinate transformation relationship is calculated using the following formula. : ； In the formula: This indicates a multi-point exponential relationship.

Images