A method for identifying the hull structure of bulk carriers based on point cloud deep learning

Abstract

The present invention provides a method for identifying the hull structure of bulk cargo vessels based on point cloud deep learning. Specifically, lidar is used to scan the point cloud of loading and unloading operations at the port, and the onshore part is filtered out to obtain the filtered point cloud; a labeling specification for object detection of the point cloud of the vessel is formulated; the point cloud is input into the point cloud object detection model for training until the loss function of the point cloud object detection model reaches the convergence condition; the point cloud object detection model identifies the point cloud of all clearly visible vessels in the input point cloud and predicts the category of each vessel; a labeling specification for component segmentation of the point cloud of bulk cargo vessels is formulated; the point cloud of the bulk cargo vessel within the 3D box is labeled for component segmentation, and for the extracted point cloud block containing the vessel, sampling and coordinate normalization are performed using a strategy; then the point cloud block is input into the point cloud component segmentation model for training. When the convergence condition is reached, the point cloud component segmentation model predicts the hull structure of the bulk cargo vessel.

Description

Technical Field

[0001] The present invention relates to the technical field of automated operation of bulk goods in ports, and particularly to a method for identifying the hull structure of bulk cargo ships based on point cloud deep learning. Background Art

[0002] With the continuous development of the unmanned level of ports, the degree of automated operation in ports has been continuously improved. When loading or unloading bulk materials onto or from ships, lidar is usually used to scan the port bulk cargo operation site in real time, and the point cloud of the port bulk cargo operation is output. For the point cloud of port bulk cargo operation, using traditional filtering methods for processing has low accuracy; using image deep learning methods requires converting the three-dimensional data of the point cloud into two-dimensional data of an image, resulting in a large amount of information loss and low accuracy. Currently, deep learning methods for point clouds are mainly applied in the fields of robotics and fusion perception. Universities, research institutes, and large technology companies are all actively promoting the research and application of point cloud deep learning. Deep learning methods for point clouds are a technological trend in the port automation industry. The object detection method for point clouds has a wide range of applications in the field of intelligent driving, being real-time and efficient. The research and application of the point cloud component segmentation method are in the stage of development and exploration and can handle non-complex business scenarios.

[0003] Identifying the hull structure of ships based on point cloud deep learning is beneficial for identifying different hull structures and is beneficial for improving operation efficiency and safety. Therefore, it is very necessary to provide a method for identifying the hull structure of bulk cargo ships based on point cloud deep learning. Summary of the Invention

[0004] In view of this, the present invention proposes a method for identifying the hull structure of bulk cargo ships based on point cloud deep learning for specific bulk cargo ships in ports.

[0005] The technical solution of the present invention is realized as follows: The present invention provides a method for identifying the hull structure of bulk cargo ships based on point cloud deep learning, including the following steps:

[0006] S1: Use lidar to scan to obtain the point cloud of port loading and unloading operations, filter out the part on the shore, and obtain the filtered point cloud;

[0007] S2: According to the operation scenario of port bulk cargo, formulate the annotation specification for object detection of the point cloud of the ship;

[0008] S3: Input the point cloud of the ship with object detection annotations in step S2 into the point cloud object detection model for training until the loss function of the point cloud object detection model reaches the convergence condition;

[0009] S4: Input the point cloud filtered in step S1 into the point cloud object detection model trained in step S3. The point cloud object detection model marks each ship with a 3D box according to the input point cloud of the clearly visible ships, and obtains the position of the 3D box and the category of the corresponding ship.

[0010] S5: According to the relative position between the 3D box and the bulk cargo handling machinery, and the category of the ship corresponding to the 3D box, select the 3D box containing the point cloud of the bulk cargo ship with the highest priority.

[0011] S6: Formulate the annotation specification for point cloud component segmentation of the bulk cargo ship according to the port bulk cargo handling scenario and the category of the bulk cargo ship; use the position information and category information of the 3D box obtained in step S5 to perform point cloud component segmentation annotation on the point cloud within the range of the 3D box.

[0012] S7: Use a strategy to sample and normalize the coordinates of the point cloud blocks of the bulk cargo ship after point cloud component segmentation annotation.

[0013] S8: Input the point cloud blocks of the bulk cargo ship after sampling and coordinate normalization in step S7 into the point cloud component segmentation model for training until the loss function of the model converges.

[0014] S9: Use the trained point cloud component segmentation model to segment the three-dimensional structure of the bulk cargo ship from the point cloud processed in S7; according to the normalization parameters of each point cloud block retained in step S7, restore the prediction result of the model to the size of the point cloud block containing the ship before normalization.

[0015] Based on the above technical solutions, preferably, the annotation specification for the object detection of the point cloud of the ship described in step S2 is, on the basis of step S1, perform annotation on the object detection of the point cloud of the clearly visible ship, construct a 3D box (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax) with the smallest volume to enclose the ship, and mark the category of this 3D box. (Xmin, Ymin, Zmin) and (Xmax, Ymax, Zmax) are respectively the two vertices closest and farthest from the origin of the 3D box; one 3D box contains only one ship. If there is more than one ship in the port bulk cargo handling scenario, establish 3D boxes corresponding to different ships respectively, and there is no overlapping part between any two adjacent 3D boxes; after obtaining the 3D box, perform feature extraction on the point cloud of the ship within the range of the 3D box.

[0016] Preferably, the point cloud filtered in step S1 is input into the point cloud object detection model trained in step S3. The point cloud object detection model marks each ship with a 3D box according to the point cloud of the clearly visible ship, obtains the position of the 3D box and the category of the corresponding ship. That is, the point cloud object detection model predicts the position information of the 3D boxes containing each clearly visible ship and the category of the corresponding ship. The position information is the six vertex parameters Xmin, Xmax, Ymin, Ymax, Zmin, and Zmax. The point cloud block containing a single bulk carrier is extracted from the point cloud filtered in step S1 using the six vertex parameters and the ship category. The category of the ship corresponding to the predicted 3D box is one of an uncovered bulk carrier, a flat-covered bulk carrier, a vertical-covered bulk carrier, or a non-bulk carrier.

[0017] Further preferably, in step S5, according to the relative position between the 3D box and the bulk handling machinery, and the category of the ship corresponding to the 3D box, the 3D box containing the point cloud of the bulk carrier with the highest priority is selected. In the bulk handling scenario, the bulk carrier with the highest priority is selected, and according to the relative position between the 3D box and the handling machinery and the category of the 3D box, the 3D box containing the ship under operation is determined.

[0018] Even more preferably, the selection of the bulk carrier with the highest priority is as follows: The bulk carriers in the port include the bulk carriers entering the port, the bulk carriers leaving the port, the ships queuing for operation, and the bulk carriers under operation; the relative positions between different bulk carriers are not fixed; the automated bulk handling in the port focuses on the bulk carriers under operation and sets the priority of the bulk carriers under operation as the highest.

[0019] Still further preferably, in step S6, according to the port bulk handling scenario and the category of the bulk carrier, the annotation specification for the point cloud component segmentation of the bulk carrier is formulated. That is, the point cloud of an uncovered bulk carrier, a flat-covered bulk carrier, or a vertical-covered bulk carrier is respectively annotated as a bow subcategory, a hull subcategory, a bulkhead / cover subcategory, a material subcategory, a stern subcategory, a water surface subcategory, a grab subcategory, and a noise subcategory.

[0020] Preferably, in step S7, for the point cloud block of the bulk carrier after point cloud component segmentation annotation, a strategy is used for sampling and coordinate normalization. That is, when the number of points in the point cloud block is more than the specified first preset number of points, voxelization sampling is adopted; when the number of points in the point cloud block is less than the specified first preset number of points, random repeated sampling is performed from the points in the point cloud block, and the repeated sampling points are added to the point cloud block. The number of repeated sampling points is equal to the difference between the number of points in the point cloud block before using the strategy and the specified first preset number of points.

[0021] By the above strategies, it is ensured that the number of points in the point cloud input to the component segmentation model is the same, and the coordinates of each point cloud block in each 3D box are normalized, and the normalization parameters of each point cloud block are retained.

[0022] More preferably, for the point cloud component segmentation model described in step S8, its constraint conditions are as follows: 1) When the point cloud component segmentation model is trained and inferred, the number of points in each point cloud block input to the point cloud component segmentation model must be consistent; 2) The input data is the three-dimensional coordinates of each point in the point cloud block and its corresponding category attribute of the bulk carrier; 3) The point cloud component segmentation model is only used for bulk carriers; 4) Even if the point cloud of the ship is incomplete, the point cloud block containing this ship is used as input; 5) Only the point cloud of the bulk carrier is segmented, and the point cloud component segmentation model of this solution is not applicable to non-bulk carriers.

[0023] A method for identifying the hull structure of a bulk carrier based on point cloud deep learning provided by the present invention has the following beneficial effects compared with the prior art:

[0024] (1) In view of the problem of insufficient accuracy in identifying bulk carriers based on image deep learning, this solution proposes a method for identifying the hull structure of a bulk carrier based on point cloud deep learning. This method obtains a 3D box containing a bulk carrier through a point cloud object detection model, filters out the 3D boxes containing working bulk carriers based on the relative position between the 3D box and the port operation machinery and the category of the 3D box. For the filtered 3D boxes, the point cloud blocks of the working bulk carriers are extracted from the entire point cloud, and the three-dimensional structure of the bulk carrier is segmented from this point cloud block using a point cloud component segmentation model; it can be applied to the identification in the real-time automated operation scenario of port bulk cargo. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings without creative efforts based on these drawings.

[0026] Figure 1 It is a flowchart of the training mode of a method for identifying the hull structure of a bulk carrier based on point cloud deep learning according to the present invention;

[0027] Figure 2 It is a flowchart of the inference mode of a method for identifying the hull structure of a bulk carrier based on point cloud deep learning according to the present invention;

[0028] Figure 3Prediction effect diagram of the hull structure recognition method for bulk cargo ships based on point cloud deep learning of the present invention for flat cover bulk cargo ships;

[0029] Figure 4 Prediction effect diagram of the hull structure recognition method for bulk cargo ships based on point cloud deep learning of the present invention for vertical cover bulk cargo ships;

[0030] Figure 5 Prediction effect diagram of the hull structure recognition method for bulk cargo ships based on point cloud deep learning of the present invention for non-cover bulk cargo ships. Detailed implementation manners

[0031] Next, in combination with the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0032] As Figure 1 shown, the present invention provides a method for identifying the hull structure of bulk cargo ships based on point cloud deep learning, including the following steps:

[0033] S1: Use lidar to scan to obtain the point cloud of port loading and unloading operations, filter out the part on the shore, and obtain the filtered point cloud;

[0034] Filtering out the point cloud on the shore can reduce unnecessary computational workload and narrow the candidate area range during subsequent point cloud data processing.

[0035] It should be noted that the above-mentioned filtering out of the point cloud on the shore is based on the coordinate values in the port bulk cargo operation coordinate system. Let the boundary line between the port shore and the water surface be the X-axis direction. The definition of the Y-axis direction is perpendicular to the X-axis direction and in the same plane as the water surface. The positive direction of the Y-axis refers to the direction from the shore to the water surface. The positive direction of the Z-axis is perpendicular to the water surface and upward. Eliminating the points on the shore means eliminating the points with a Y-axis coordinate less than 0.

[0036] S2: According to the operation scenarios of port bulk cargo, formulate the annotation specifications for the object detection of the point cloud of ships.

[0037] The annotation specification for the object detection of the point cloud of the vessel formulated herein is to perform the annotation of the object detection of the point cloud for the clearly visible vessels on the basis of filtering out the point cloud of the onshore part in step S1, construct a 3D box (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax) with the smallest volume to enclose the vessel, and mark the category of this 3D box. (Xmin, Ymin, Zmin) and (Xmax, Ymax, Zmax) are the two vertices that are closest and farthest from the origin of the 3D box respectively; one 3D box contains only one vessel. In the operation scenario of port bulk cargo, if there is more than one vessel, 3D boxes shall be established respectively for different vessels, and there is no overlapping part between any two adjacent 3D boxes.

[0038] The 3D boxes here have four different categories, namely, bulk cargo vessels without cover plates, bulk cargo vessels with flat cover plates, bulk cargo vessels with vertical cover plates, and non-bulk cargo vessels. For the three categories of bulk cargo vessels, namely, bulk cargo vessels without cover plates, bulk cargo vessels with flat cover plates, and bulk cargo vessels with vertical cover plates, it is easy to distinguish due to the large differences in the hatch openings of the vessels. Non-bulk cargo vessels refer to tugboats, patrol boats, passenger ships, refueling ships, etc. It should be noted that the farther the distance from the lidar, the more incomplete the vessel is and the smaller the point density is. Therefore, the point cloud output by the lidar will have incomplete and unclear vessels. These vessels are far from the operating machinery and do not need to be annotated for object detection of the point cloud.

[0039] S3: Input the point cloud of the vessels that have been annotated for object detection in step S2 into the point cloud object detection model for training until the loss function of the point cloud object detection model reaches the convergence condition.

[0040] The point cloud of the labeled vessels for which object detection has been performed is input into a point cloud object detection model for training. As a specific implementation of this solution, the neural network CenterPoint can be used to perform feature extraction and training on the point cloud input of the labeled vessels for which object detection has been performed. The neural network CenterPoint is a method for object detection using point cloud deep learning. The algorithm processing flow is to extract the map view feature representation from the point cloud using a 3D Backbone, then use a 2D CNN architecture detection head to find the object center, and use the center feature to regress to the complete 3D bounding box. The box prediction is used to extract the point features at the 3D center of each face of the estimated 3D bounding box, and these point features will be passed into the MLP to predict the IoU-guided confidence score and box regression optimization. This algorithm is a mature algorithm, and those skilled in the art can easily obtain the relevant content of this algorithm, so it will not be elaborated here. Of course, there are many methods for the point cloud object detection model. The content illustrated by using the neural network CenterPoint above is not regarded as a limitation to this solution. Those skilled in the art can choose other well-known point cloud object detection models in this field according to needs to implement the training of object detection based on point cloud.

[0041] S4: Input the point cloud filtered in step S1 into the point cloud object detection model trained in step S3. The point cloud object detection model marks each vessel with a 3D box according to the input point cloud of the clearly visible vessels, and obtains the position of the 3D box and the category of the corresponding vessel.

[0042] Specifically, the point cloud object detection model trained in the previous step is used to predict the position information of the 3D box, that is, the six vertex parameters Xmin, Xmax, Ymin, Ymax, Zmin, and Zmax. Using the six vertex parameters, the point cloud block containing a single bulk cargo vessel is extracted from the point cloud filtered in step S1; the vessel category corresponding to the predicted 3D box is one of the uncovered bulk cargo vessel, flat-covered bulk cargo vessel, vertical-covered bulk cargo vessel, or non-bulk cargo vessel. The boundary of the point cloud block is the six vertex parameters. The six vertices of the 3D box determine the length, width, and height of the 3D box. Each point cloud is three-dimensional data and has the coordinates of the three axes X, Y, and Z. Each vessel's point cloud is divided separately with a 3D box to obtain the point cloud blocks corresponding to several 3D boxes and the categories of the 3D boxes.

[0043] S5: According to the relative position between the 3D box and the bulk cargo handling machinery, and the category of the vessel corresponding to the 3D box, take the 3D box containing the point cloud of the bulk cargo vessel with the highest priority.

[0044] The specific content of this step is to select the bulk cargo ship with the highest priority in the bulk cargo operation scenario. The bulk cargo ships in the port include those entering the port, those leaving the port, those queuing for operation, and those in operation; the relative positions of different bulk cargo ships are not fixed; the automated bulk cargo operation in the port focuses on the bulk cargo ship in operation and sets the priority of the bulk cargo ship in operation as the highest. The 3D box of the point cloud containing the bulk cargo ship with the highest priority can also be understood as the corresponding priority of the 3D box.

[0045] S6: According to the port bulk cargo operation scenario and the category of bulk cargo ships, formulate the annotation specification for the point cloud component segmentation of bulk cargo ships; use the position information and category information of the 3D box obtained in step S5 to perform point cloud component segmentation annotation on the point cloud within the range of the 3D box.

[0046] For the bulk cargo ships without covers, flat - covered bulk cargo ships, and vertical - covered bulk cargo ships, there are significant differences in the component segmentation annotation rules, especially for the hatch part. Step S6 is to separately annotate the point clouds of the bulk cargo ships without covers, flat - covered bulk cargo ships, or vertical - covered bulk cargo ships: the bulk cargo ship without covers is annotated as bow sub - category, hull sub - category, partition sub - category, material sub - category, stern sub - category, water surface sub - category, grab sub - category, and noise sub - category; for the vertical - covered bulk cargo ship, it is annotated as bow sub - category, hull sub - category, cover sub - category, material sub - category, stern sub - category, water surface sub - category, grab sub - category, and noise sub - category; for the flat - covered bulk cargo ship, it is annotated as bow sub - category, hull sub - category, cover sub - category, material sub - category, stern sub - category, water surface sub - category, grab sub - category, and noise sub - category.

[0047] The point cloud blocks of the bulk cargo ship extracted from the point cloud of the bulk cargo ship within the 3D box do not exceed the boundary of the 3D box. In addition, within the 3D box, there are not only bulk cargo ships but also water surfaces and suspended noise points. These noise points may be the chutes of materials above the hatch or floating dust, etc. The point cloud component segmentation annotation is performed not only on the bulk cargo ship in the 3D box but also on the water surface and noise points to better segment the three - dimensional structure of the bulk cargo ship hull.

[0048] For the ship without a cover, there is no cover set, only the corresponding partition sub - category, while for the ship with a cover, the cover sub - category is marked as a vertical cover or a flat cover accordingly.

[0049] S7: For the extracted point cloud blocks containing ships, use a strategy for sampling and coordinate normalization.

[0050] Specifically, when the number of points in the point cloud block is more than the specified first preset number of points, voxelization sampling is adopted; when the number of points in the point cloud block is less than the specified first preset number of points, random repeated sampling is performed from the points in the point cloud block, and the repeated sampling points are added to the point cloud block. The number of repeated sampling points is equal to the difference between the number of points in the point cloud block before using the strategy and the specified first preset number of points.

[0051] By the above strategy, it is ensured that the number of points input into the component segmentation model is the same, and the coordinates of each point cloud block in each 3D box are normalized, and the normalization parameters of each point cloud block are retained.

[0052] S8: Input the point cloud block containing the ship after sampling and coordinate normalization in step S7 into the point cloud component segmentation model for training until the loss function of the model converges.

[0053] More preferably, for the point cloud component segmentation model described in step S8, the constraint conditions are as follows: 1) When the point cloud component segmentation model is trained and inferred, the number of points in each point cloud block input into the point cloud component segmentation model must be consistent; 2) The input data is the three-dimensional coordinates of each point in the point cloud block and its corresponding category attribute of the bulk carrier; 3) The point cloud component segmentation model is only used for bulk carriers; 4) Even if the point cloud of the ship is incomplete, the point cloud block containing this ship is used as input; 5) Only the point cloud component segmentation of bulk carriers is performed, and non-bulk carriers are not applicable to the point cloud component segmentation model of this solution.

[0054] The point cloud of the labeled ship with point cloud component segmentation is input into the point cloud component segmentation model for training. As a specific implementation of this solution, the neural network RandLA-Net can be used to extract features and train the point cloud input of the labeled ship with point cloud component segmentation. The neural network RandLA-Net is a method for component segmentation using point cloud deep learning. The algorithm processing flow is to use random sampling to quickly process large-scale point clouds, aggregate the domain features of points with the Local Spatial Encoding module; downsample with the Dilated Residual Block module to expand the receptive field of the network; then upsample with the Feature Decoder module, and fuse the feature information of the same dimension as above with the Multi-Layer Perceptron; connect 3 Full Connection layers, and finally output the category of each point in the sampled point cloud. This algorithm is a mature algorithm, and those skilled in the art can easily obtain the relevant content of this algorithm, which will not be elaborated here. Of course, there are many methods for the point cloud component segmentation model. The content exemplified by using the neural network RandLA-Net above is not regarded as a limitation to this solution. Those skilled in the art can choose other well-known point cloud component segmentation models in the art according to needs to implement the training of component segmentation based on point cloud.

[0055] S9: Use the trained point cloud component segmentation model to segment the three-dimensional structure of the bulk carrier ship from the point cloud processed in S7; according to the normalization parameters of each point cloud block retained in step S7, restore the prediction result of the model to the size of the point cloud block containing the ship before normalization.

[0056] Here, the point cloud block needs to be first input into the point cloud component segmentation model for training until the loss function of the point cloud component segmentation model reaches the convergence condition; then re-input the point cloud block to be segmented into the trained point cloud component segmentation model, and the hull structure of the bulk carrier ship is predicted by the point cloud component segmentation model. The schematic diagrams of the hull structures of flat cover bulk carrier ships, vertical cover bulk carrier ships, and non-cover bulk carrier ships predicted by the cloud component segmentation model are respectively as Figure 3 , Figure 4 and Figure 5 shown.

[0057] When running for the first time, adopt the Figure 1 process. After the model training is completed, the Figure 2 process can be adopted.

[0058] The core idea of this solution is: First, after the point cloud is filtered in step S1, the point cloud object detection model is used to identify all 3D boxes containing a single visible ship. Then, 3D boxes containing one or more operating bulk carriers are screened out. Each 3D box contains a bulk carrier. Based on the information of the 3D boxes, point cloud blocks are extracted from the entire point cloud, and then the point cloud blocks are segmented into components to segment the 3D hull structure of the operating ship.

[0059] Alternatively, a second approach can be adopted: directly segment the point cloud filtered in step S1 to identify all visible bulk carrier 3D structures. This approach is very expensive in terms of point cloud data annotation. Component segmentation is more difficult, time-consuming, and costly than object detection, and its accuracy is lower. Furthermore, component segmentation models are more computationally intensive and require longer inference time than object detection models.

[0060] Typically, lidar scans focus on operating bulk carriers. Therefore, the portion of the point cloud depicting operating bulk carriers is the clearest and has the highest point cloud density. The further away a non-operating bulk carrier is from the lidar, the less clear its point cloud becomes and the lower its point cloud density. For complex bulk carrier operations, such as multiple bulk carriers in close proximity, the first approach provides better prediction results than the second.

[0061] Bulk carriers have diverse hull shapes, and traditional filtering algorithms are overly simplistic and cannot handle some unusual shapes, such as those with hatches. If multiple ships are located close together at a port, traditional filtering algorithms cannot filter out the queued vessels. In such cases, either manual bulk handling is required or the distance between bulk carriers is kept greater to allow traditional filtering algorithms to filter out the queued vessels. Pedestrians standing on either side of a ship can hinder traditional filtering algorithms.

[0062] For uncovered bulk carriers, the closer the hold is to being full, the less distinct the hatch opening becomes, making it more difficult to locate. When the hold is empty or low on material, the hatch opening's height difference can be exploited to easily locate the hatch opening, facilitating automated positioning of material handling and discharge.

[0063] For bulk carriers with vertical and flat covers, there are protrusions in the hull part between the two hatches that are higher than the plane height of the hatches, such as cranes, electromechanical boxes, etc., which hinder the traditional filtering algorithm from identifying bulk carriers with covers.

[0064] Bulk carriers have a pitch angle relative to the water surface, with a significant difference in cargo volume between the bow and stern, which can easily cause the hull plane to be non-parallel to the water surface, rendering traditional filtering algorithms ineffective.

[0065] The recognition pass rate of bulk cargo ships with automated operations adopting this solution is significantly higher than that of traditional filtering algorithms. The test method for automated operations is that if the error between the hatch rectangle obtained from the output result of this solution and the manually fine-labeled hatch rectangle does not exceed 0.5 meters, the automated operation is considered to pass.

[0066] The above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for identifying the hull structure of a bulk carrier based on point cloud deep learning, characterized in that, It includes the following steps: S1: Use lidar to scan to obtain the point cloud of the ship loading and unloading operations in the port, filter out the part on the shore, and obtain the filtered point cloud; S2: According to the operation scenarios of bulk cargo in the port, formulate the annotation specifications for the object detection of the point cloud of the ship; S3: Input the point cloud of the ship with object detection annotations in step S2 into the point cloud object detection model for training until the loss function of the point cloud object detection model reaches the convergence condition; S4: Input the filtered point cloud in step S1 into the point cloud object detection model trained in step S3. The point cloud object detection model marks each ship with a 3D box according to the clearly visible point cloud of the ship, and obtains the position of the 3D box and the corresponding category of the ship; S5: According to the relative position between the 3D box and the bulk cargo handling machinery, and the category of the ship corresponding to the 3D box, select the 3D box containing the point cloud of the bulk cargo ship with the highest priority; S6: According to the port bulk cargo operation scenario and the category of the bulk cargo ship, formulate the annotation specifications for the point cloud component segmentation of the bulk cargo ship; use the position information and category information of the 3D box obtained in step S5 to perform point cloud component segmentation annotation on the point cloud within the range of the 3D box; S7: Use a strategy to sample and normalize the coordinates of the point cloud block of the bulk cargo ship after point cloud component segmentation annotation; S8: Input the point cloud block of the bulk cargo ship after sampling and coordinate normalization in step S7 into the point cloud component segmentation model for training until the loss function of the model converges; S9: Use the trained point cloud component segmentation model to segment the three-dimensional structure of the bulk cargo ship from the point cloud processed in S7; according to the normalization parameters of each point cloud block retained in step S7, restore the prediction result of the model to the size of the point cloud block containing the ship before normalization.

2. The hull structure recognition method of a bulk carrier based on point cloud deep learning according to claim 1, wherein The annotation specifications for the object detection of the point cloud of the ship described in step S2 are, on the basis of step S1, to perform object detection annotation on the clearly visible point cloud of the ship, construct a 3D box with the smallest volume (Xmin, Xmax, Ymin, Ymax, Zmin, Zmax) to enclose the ship, and mark the category of this 3D box. (Xmin, Ymin, Zmin) and (Xmax, Ymax, Zmax) are respectively the two vertices with the closest and farthest distances from the origin of the 3D box; one 3D box contains only one ship. If there is more than one ship in the operation scenario of port bulk cargo, 3D boxes are established corresponding to different ships respectively, and there is no overlapping part between any two adjacent 3D boxes; after obtaining the 3D box, perform feature extraction on the point cloud of the ship within the range of the 3D box.

3. The method for identifying the hull structure of a bulk carrier based on point cloud deep learning according to claim 2, wherein In step S4, the point cloud filtered in step S1 is input into the point cloud target detection model trained in step S3. The point cloud target detection model marks each ship with a 3D box based on the input point cloud of clearly visible ships, and obtains the position of the 3D box and the corresponding ship category. The point cloud target detection model is used to predict the position information of the 3D box containing each clearly visible ship and the corresponding ship category. The position information is six vertex parameters Xmin, Xmax, Ymin, Ymax, Zmin and Zmax. The six vertex parameters and the ship category are used to extract a point cloud block containing a single bulk carrier from the point cloud filtered in step S1. The category of the ship corresponding to the predicted 3D box is one of a bulk carrier without a cover, a bulk carrier with a flat cover, a bulk carrier with a vertical cover, or a non-bulk carrier.

4. The method for identifying the hull structure of a bulk carrier based on point cloud deep learning according to claim 3, wherein, In step S5, the 3D frame containing the point cloud of the bulk carrier vessel with the highest priority is selected based on the relative position of the 3D frame and the bulk cargo operation machinery, as well as the category of the vessel corresponding to the 3D frame. In the bulk cargo operation scene, the highest priority bulk carrier vessel is selected, and the 3D frame containing the operating vessel is determined based on the relative position of the 3D frame and the operation machinery, as well as the category of the 3D frame.

5. The hull structure recognition method of a bulk carrier based on point cloud deep learning according to claim 4, wherein The bulk carriers with the highest priority are selected as follows: the bulk carriers in the port include bulk carriers entering the port, bulk carriers leaving the port, ships waiting in line for operation, and bulk carriers currently operating; the relative positions of different bulk carriers are not fixed; the bulk cargo automation operation in the port focuses on the bulk carriers currently operating, and sets the priority of the bulk carriers currently operating to the highest level.

6. The method for identifying the hull structure of a bulk carrier based on point cloud deep learning according to claim 5, wherein In step S6, based on the port bulk cargo operation scene and the category of bulk carriers, a labeling specification for the segmentation of point cloud components of bulk carriers is formulated, and the point clouds of bulk carriers without cover, bulk carriers with flat cover, or bulk carriers with vertical cover are respectively labeled as bow subcategory, hull subcategory, bulkhead / cover subcategory, material subcategory, stern subcategory, water surface subcategory, grab subcategory, and noise subcategory.

7. A method for identifying the hull structure of a bulk carrier based on point cloud deep learning according to claim 6, characterized in that, In step S7, the point cloud block of the bulk carrier after segmentation and annotation of the point cloud component is sampled and coordinate normalized using a strategy, which is that when the number of points in the point cloud block is more than a specified first preset number of points, voxel sampling is used; when the number of points in the point cloud block is less than the specified first preset number of points, random repeated sampling is performed from the points in the point cloud block, and the repeated sampling points are added to the point cloud block, and the number of repeated sampling points is equal to the difference between the number of points in the point cloud block before the strategy is used and the specified first preset number of points; The above strategy ensures that the number of points in the point cloud input to the component segmentation model is the same, and the coordinates of each point cloud block in each 3D frame are normalized to retain the normalized parameters of each point cloud block.

8. A method for identifying the hull structure of a bulk carrier based on point cloud deep learning according to claim 7, characterized in that, The constraints of the point cloud component segmentation model described in step S8 are as follows: 1) When training and inferring the point cloud component segmentation model, the number of points in each point cloud block input into the point cloud component segmentation model must be consistent; 2) The input data is the three-dimensional coordinates of each point in the point cloud block and its corresponding category attribute of the bulk carrier; 3) The point cloud component segmentation model is only used for bulk carriers; 4) Even if the point cloud of the ship is incomplete, the point cloud block containing this ship is used as input; 5) Point cloud component segmentation is only performed on bulk carriers, and non-bulk carriers are not applicable to the point cloud component segmentation model of this solution.

Images