Ship modeling and contour measurement system and method based on image-laser point cloud fusion data

Through the ship modeling system of image-laser point cloud fusion data, combined with semantic segmentation and 3D point cloud matching algorithm, the problems of ship modeling time and low measurement efficiency in the existing technology are solved, and the rapidity in harsh environments is achieved , automated three-dimensional modeling and measurement improves the safety and efficiency of ship loading operations.

CN119027482APending Publication Date: 2024-11-26QINHUANGDAO PORT

Patent Information

Application Number
CN202410941952.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing ship modeling and measurement methods are single and time-consuming, making it difficult to achieve rapid three-dimensional modeling and automated measurement of large ships. Especially in harsh bulk cargo dock environments, it cannot meet the dynamic obstacle avoidance and full automation requirements of ship loading operations.

Method used

Adopt ship modeling and contour measurement system based on image-laser point cloud fusion data, through visual-lidar fusion acquisition equipment, ship 3D modeling module, ship 3D contour measurement module, model 3D visualization/model storage/data intercalation module With similar ship-type data rapid measurement module, it combines semantic segmentation, point cloud gridding and 3D point cloud matching algorithms to achieve automated measurement and rapid modeling.

Benefits of technology

It significantly improves the authenticity and measurement speed of ship modeling, improves measurement accuracy, and can maintain high efficiency in surge environments, realizes automated measurement of hatch and hull attachment facilities, and meets the safety and efficiency needs of ship loading operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027482A_ABST
    Figure CN119027482A_ABST
Patent Text Reader

Abstract

The invention relates to a rapid ship three-dimensional modeling and contour measurement system and method, and the method comprises the steps: erecting a vision-laser radar fusion collection device at a cantilever crane near a cab of a port operation machine, enabling the port operation machine to move along a track, and carrying out the collection of the three-dimensional color point cloud and image data of a bulk cargo ship; through ship three-dimensional modeling, ship three-dimensional profile measurement, model three-dimensional visualization / model storage / data mutual correction and similar ship type data rapid measurement, bulk cargo ship 3D model reconstruction and key part automatic measurement are realized. The ship surface hatch linear fitting and distance measurement are combined to complete hatch automatic measurement and ship key part size measurement, the reality sense of ship modeling and the measurement speed can be greatly improved, meanwhile, the method is not affected by surges, and the measurement precision is high.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of visual measurement, and in particular to a ship modeling and contour measurement system and method based on image-laser point cloud fusion data. Background Art

[0002] At present, in the process of bulk cargo loading and unloading, the rapid acquisition and measurement of three-dimensional information is the key to supporting the automated and safe operation during bulk cargo loading operations. However, due to the poor operating environment of bulk cargo terminals, the fluidity of bulk cargo materials, and the random uncertainty of waves and surges, the hull position of large bulk carriers will undergo complex changes during operation, so it is very difficult to quickly model and measure large ships in three dimensions. The existing ship modeling methods are relatively simple, and only use laser radar scanning technology to achieve it. It often requires complex hull scanning and offline manual operation to complete the three-dimensional measurement of the ship. It often takes dozens of minutes or even longer from ship point cloud scanning to complete the measurement, which is inefficient. It is not only difficult to achieve automated three-dimensional measurement of the three-dimensional contours of hatches and ship ancillary facilities, but also difficult to meet the needs of fully automated loading operations. For example, Goldan in the Netherlands uses photogrammetry technology to perform 3D measurement of hull plates. This method has a large measurement range, but requires a large number of target points to be placed on the hull in advance, and has extremely high requirements for ambient lighting. Hiekata et al. from the University of Tokyo in Japan use a laser tracker to measure the point cloud data of a curved plate. During the measurement process, they need to constantly operate the target ball to guide the laser tracker to measure, which has low measurement efficiency and high labor intensity. The Geomagic Control X 3D detection platform in Germany can achieve surface analysis, geometric detection of ship parts, and dent detection of decks by reconstructing and accurately adjusting complex point cloud models, but the measurement time is relatively long. Harbin Institute of Technology has achieved ship contour measurement based on the Livox horizon laser radar, but can only measure the length, width and hatch data of the ship.

[0003] In summary, the current ship modeling and measurement methods are relatively simple, and the calculation of ship shape and volume usually requires manual offline measurement, which is time-consuming and leads to the problem that dynamic obstacle avoidance and fully automated safe operation cannot be performed during the loading process. In order to obtain the three-dimensional contour information of the ship in a timely manner, ensure safe obstacle avoidance during the operation of the ship loader, and improve the efficiency and accuracy of remote operation of loading, it is urgent to propose a fast three-dimensional modeling and measurement method. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a ship modeling and contour measurement system and method based on image-laser point cloud fusion data. By utilizing 3D laser point cloud and image data to realize 3D model reconstruction of bulk carriers and automatic measurement of key parts, the realism of ship modeling and the speed of measurement can be greatly improved, and it is not affected by surges and has high measurement accuracy.

[0005] The present invention solves the technical problem by adopting the following technical solutions:

[0006] The ship modeling and contour measurement system based on image-laser point cloud fusion data includes a vision-laser radar fusion acquisition device, a ship three-dimensional modeling module, a ship three-dimensional contour measurement module, a model three-dimensional visualization / model storage / data mutual calibration module and a similar ship type data rapid measurement module, wherein the vision-laser radar fusion acquisition device, the ship three-dimensional modeling module, the ship three-dimensional contour measurement module, the model three-dimensional visualization / model storage / data mutual calibration module and the similar ship type data rapid measurement module are connected in sequence, the vision-laser radar fusion acquisition device is used to collect three-dimensional color point cloud and image data of bulk carriers, the ship three-dimensional modeling module is used to model according to the three-dimensional color point cloud and image data of bulk carriers, the ship three-dimensional contour measurement module is used to measure the length and width of bulk carriers and the contour data of key parts according to the three-dimensional color point cloud and image data of bulk carriers, the model three-dimensional visualization / model storage / data mutual calibration module is used to display and store the three-dimensional color point cloud of bulk carriers, and the similar ship type data rapid measurement module uses a 3D point cloud matching algorithm to perform data alignment and rapid measurement on bulk carriers.

[0007] Moreover, the vision-lidar fusion acquisition equipment is installed on the arm near the driver's cab of the port operation machinery.

[0008] Moreover, the key parts contour data measured by the ship three-dimensional contour measurement module include the positions and external dimensions of hatches, hatch covers, superstructures and other auxiliary equipment and facilities of the ship.

[0009] A ship modeling and contour measurement system measurement method based on image-laser point cloud fusion data, comprising the following steps:

[0010] Step 1: After the ship docks at the coast, the vision-lidar fusion acquisition device is controlled to move along the track from the leftmost end of the ship entering the image field to the rightmost end to complete the three-dimensional scanning of the ship and obtain the three-dimensional color point cloud and image data of the ship;

[0011] Step 2: The ship 3D modeling module performs 3D modeling based on the 3D color point cloud and image data of the ship and based on downsampling and point cloud meshing algorithms;

[0012] Step 3: The ship 3D profile measurement module is used to measure the length and width of the bulk carrier, as well as the profile data of key parts according to the 3D color point cloud and image data of the bulk carrier;

[0013] Step 4: The model 3D visualization / model storage / data mutual verification module is used to display and store the 3D color point cloud of the bulk carrier;

[0014] Step 5: The similar ship type data rapid measurement module uses a 3D point cloud matching algorithm to perform data alignment and rapid measurement on bulk carriers.

[0015] Furthermore, the step 2 comprises the following steps:

[0016] Step 2.1, transforming the collected complete three-dimensional color point cloud of the ship into an orthographic projection point cloud;

[0017] Step 2.2: Use the semantic segmentation network to segment the hull, hatch and coast of the ortho-projected point cloud data, complete the separation of the ship and the shore, and project the image semantic segmentation results onto the 3D point cloud to obtain the point cloud contour of the ship surface;

[0018] Step 2.3: Mesh the obtained ship point cloud to complete the three-dimensional model of the ship.

[0019] Moreover, the specific implementation method of step 3 is:

[0020] Step 3.1: For the hull, directly input the point cloud contour of the ship surface obtained by semantic segmentation (not mentioned above), and use the minimum circumscribed rectangle method to calculate the length and width of the ship;

[0021] Step 3.2, for hatches, the number of hatches obtained by semantic segmentation is counted, and a straight line is fitted to the point cloud data of each hatch, and then the length and width of the hatch are measured based on the distance between the fitted lines on the opposite sides of the hatch;

[0022] Step 3.3: For other ancillary facilities of the ship, the point cloud on the surface of the ship is used as the boundary, and the point cloud on the upper part is clustered and the outer contour bounding box is calculated to achieve the contour measurement of the ancillary facilities.

[0023] Moreover, the specific implementation method of step 4 is: identifying the ship type through the YOLOv5 target detection network, calling the stored three-dimensional model of the ship of the current operating model, and calling the ship size data of the corresponding model in the database according to the ship model to realize the evaluation of the ship measurement accuracy, and at the same time realize the storage of the reconstructed ship three-dimensional model data.

[0024] Moreover, the specific implementation method of step 5 is: the next time the ship enters the port, the scanned visual-laser fusion point cloud data is quickly compared with the data stored in the database. If the same data exists, the data alignment and rapid measurement of the current operating ship are completed based on the 3D point cloud matching algorithm, and the stored model is automatically called to avoid repeated measurements.

[0025] The advantages and positive effects of the present invention are:

[0026] The present invention sets up the vision-lidar fusion acquisition equipment on the arm near the driver's cab of the port operation machinery, moves the port operation machinery along the track to collect the 3D color point cloud and image data of the bulk carrier, and then realizes the reconstruction of the 3D model of the bulk carrier and the automatic measurement of key parts through ship 3D modeling, ship 3D contour measurement, model 3D visualization / model storage / data mutual verification, and rapid measurement of similar ship type data. The present invention completes the separation of shore and ship and the segmentation of key parts of the ship based on semantic segmentation, and completes the automatic measurement of the hatch and the size measurement of the key parts of the ship in combination with the ship surface hatch linear fitting and ranging. The present invention can greatly improve the realism of ship modeling and the speed of measurement, and is not affected by surge and has high measurement accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the system installation of the present invention;

[0028] Figure 2 is a flow chart of the present invention;

[0029] Figure 3 Schematic diagram of the semantic segmentation network principle of the present invention.

[0030] Description of labels:

[0031] 1-Vision-LiDAR fusion acquisition equipment, 2-Pre-scan the ship. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below with reference to the accompanying drawings.

[0033] The ship modeling and contour measurement system based on image-laser point cloud fusion data includes a vision-laser radar fusion acquisition device, a ship three-dimensional modeling module, a ship three-dimensional contour measurement module, a model three-dimensional visualization / model storage / data mutual calibration module and a similar ship type data rapid measurement module, wherein the vision-laser radar fusion acquisition device, the ship three-dimensional modeling module, the ship three-dimensional contour measurement module, the model three-dimensional visualization / model storage / data mutual calibration module and the similar ship type data rapid measurement module are connected in sequence, the vision-laser radar fusion acquisition device is used to collect three-dimensional color point cloud and image data of bulk carriers, the ship three-dimensional modeling module is used to model according to the three-dimensional color point cloud and image data of bulk carriers, the ship three-dimensional contour measurement module is used to measure the length and width of bulk carriers and the contour data of key parts according to the three-dimensional color point cloud and image data of bulk carriers, the model three-dimensional visualization / model storage / data mutual calibration module is used to display and store the three-dimensional color point cloud of bulk carriers, and the similar ship type data rapid measurement module uses a 3D point cloud matching algorithm to perform data alignment and rapid measurement on bulk carriers.

[0034] like Figure 1 As shown, the vision-laser radar fusion acquisition device 1 is installed at the arm near the driver's cab of the port operation machinery. By adjusting the pan / tilt, the collected point cloud and image include the complete image of the pre-scanned ship 2, and the three-dimensional color point cloud and image data of the bulk carrier are collected through the movement of the loader. In addition, the vision-laser fusion acquisition device is connected to the industrial computer through 2 Gigabit network cables. The industrial computer is installed in the driver's cab for data processing and storage. The operator can remotely connect to the industrial computer in the monitoring center and install the client measurement software system to perform real-time three-dimensional scanning, model display, browsing and three-dimensional measurement tasks of the ship.

[0035] The key parts contour data measured by the ship's 3D contour measurement module include the location and dimensions of hatches, hatch covers, superstructures and other ship ancillary equipment and facilities.

[0036] A measurement method for ship modeling and contour measurement system based on image-laser point cloud fusion data, such as Figure 2 As shown, the following steps are included:

[0037] Step 1: After the ship docks at the coast, control the vision-lidar fusion acquisition equipment to move along the track from the leftmost end of the ship entering the image field to the rightmost end to complete the three-dimensional scanning of the ship and obtain the three-dimensional color point cloud and image data of the ship.

[0038] Step 2: The ship 3D modeling module performs 3D modeling based on the 3D color point cloud and image data of the ship and based on downsampling and point cloud meshing algorithms.

[0039] Step 2 includes the following steps:

[0040] Step 2.1: transform the collected complete three-dimensional color point cloud of the ship into an orthographic projection point cloud.

[0041] Step 2.2: Utilization Figure 3 The semantic segmentation network shown in the figure uses the semantic segmentation network to segment the hull, hatch and coast of the point cloud data, completes the separation of shore and ship, and projects the image semantic segmentation result onto the 3D point cloud to obtain the point cloud contour of the ship surface.

[0042] The black dotted lines in the semantic segmentation network represent convolutions, and the black solid lines represent upsampling, downsampling, and bilateral fusion operations. The backbone network mainly uses ResNet. After conv3, that is, after processing to 1 / 8 of the original image size, a depth branch is added. This branch has a higher number of channels and a smaller size than the main branch. It is used to increase the receptive field and extract semantic information. A bilateral fusion structure is added at each end of each branch to fuse high-resolution feature maps with low-resolution, high-semantic feature maps. The semantic segmentation network has the following two structures:

[0043] (1) Bilateral fusion structure

[0044] The structure mainly includes downsampling of high-resolution images, upsampling of low-resolution images, image merging after sampling, and RELU function activation operations. Bilateral fusion includes fusing high-resolution branches to low-resolution branches (high-to-low fusion), and fusing low-resolution branches to high-resolution branches (low-to-high fusion). For high-to-low fusion, the high-resolution feature map is downsampled by a sequence of 3×3 convolutions with a stride of 2. For low-resolution to high-resolution, the low-resolution feature map is first compressed with a 1×1 convolution and then upsampled with bilinear interpolation.

[0045] (2) DAPPM module

[0046] The DAPPM module can be seen as a combination of deep feature aggregation and pyramid pooling. Different pooling kernels and increasing stride steps are used to generate feature maps with input image resolutions of 1 / 128, 1 / 256, and 1 / 512. The input 1 / 64 resolution feature map and the image-level information generated by global average pooling are also utilized.

[0047] Step 2.3: Mesh the obtained ship point cloud to complete the three-dimensional model of the ship. The specific steps are as follows:

[0048] First, for the data collected by the vision-lidar fusion acquisition device, use the PCLVoxelGrid class to create a series of three-dimensional voxel grids through the input point cloud data, and then replace the voxel center point with the point closest to the voxel center in the original point cloud data in this voxel grid to improve the expression accuracy of the point cloud data. Next, perform the following steps:

[0049] 1) Make a bounding box of the point cloud, that is, a large cube, and then divide the cube into several small cubes, and each cube is a voxel.

[0050] 2) Consider whether the voxel needs to be updated, which is determined by the closest distance of the voxel's center of mass position. If the distance is the shortest, the voxel will replace the original voxel and become the new voxel.

[0051] 3) After looping through all the cubes, the simplification of the point cloud data is completed.

[0052] Furthermore, the Greedy Projection triangulation algorithm in the PCL library is called, a triangle sample is selected to make an initial surface, the surface boundary is continuously expanded to obtain a complete triangular mesh surface, and finally the topological relationship of the three-dimensional point cloud data points is determined according to the connection relationship of the projection points to obtain the final three-dimensional model.

[0053] Step 3: The ship 3D profile measurement module is used to measure the length and width of the bulk carrier, as well as the profile data of key parts based on the 3D color point cloud and image data of the bulk carrier.

[0054] The specific implementation method of step 3 is:

[0055] Step 3.1: For the hull, directly input the point cloud contour of the ship surface obtained by segmentation, and use the minimum circumscribed rectangle method to calculate the length and width of the ship;

[0056] Step 3.2, for the hatches, the number of hatches obtained by semantic segmentation is counted, and the point cloud data of each hatch is fitted with a line using the least squares method, and then the length and width of the hatch are measured based on the distance between the fitted lines on the opposite sides of the hatch;

[0057] Step 3.3: For other ancillary facilities of the ship, the point cloud on the surface of the ship is used as the boundary, and the KMeans algorithm is used to cluster the point cloud on the upper part and calculate the minimum contour bounding box, so as to realize the contour measurement of the ancillary facilities.

[0058] Step 4: The model 3D visualization / model storage / data mutual verification module is used to display and store the 3D color point cloud of the bulk carrier.

[0059] The specific implementation method of step 4 is: identify the ship type through the YOLOv5 target detection network, call the stored three-dimensional model of the current operating model ship, and call the ship size data of the corresponding model in the database according to the ship model to evaluate the ship measurement accuracy, and at the same time realize the storage of the reconstructed ship three-dimensional model data.

[0060] Step 5: The similar ship type data rapid measurement module uses a 3D point cloud matching algorithm to perform data alignment and rapid measurement on bulk carriers.

[0061] The specific implementation method of step 5 is as follows: the next time a ship enters the port, the scanned visual-laser fusion point cloud data is quickly compared with the data stored in the database. If the same data exists, the data alignment and rapid measurement of the current operating ship are completed based on the 3D point cloud matching algorithm, and the stored model is automatically called to avoid repeated measurements.

[0062] It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation manner. Any other implementation manners derived by those skilled in the art based on the technical solution of the present invention also fall within the scope of protection of the present invention.

Claims

1. Ship modeling and contour measurement system based on image-laser point cloud fusion data, characterized by: It includes vision-lidar fusion acquisition equipment, ship 3D modeling module, ship 3D contour measurement module, model 3D visualization / model storage / data mutual calibration module and similar ship type data rapid measurement module, wherein the vision-lidar fusion acquisition equipment, ship 3D modeling module, ship 3D contour measurement module, model 3D visualization / model storage / data mutual calibration module and similar ship type data rapid measurement module are connected in sequence, the vision-lidar fusion acquisition equipment is used to collect 3D color point cloud and image data of bulk carriers, the ship 3D modeling module is used to model according to the 3D color point cloud and image data of bulk carriers, the ship 3D contour measurement module is used to measure the length and width of bulk carriers and contour data of key parts according to the 3D color point cloud and image data of bulk carriers, the model 3D visualization / model storage / data mutual calibration module is used to display and store the 3D color point cloud of bulk carriers, and the similar ship type data rapid measurement module adopts 3D point cloud matching algorithm to align and rapidly measure the data of bulk carriers.

2. The ship modeling and contour measurement system based on image-laser point cloud fusion data according to claim 1 is characterized in that: The vision-lidar fusion acquisition equipment is installed on the arm near the driver's cab of the port operation machinery.

3. The ship modeling and contour measurement system based on image-laser point cloud fusion data according to claim 1 is characterized in that: The key parts contour data measured by the ship three-dimensional contour measurement module include the positions and external dimensions of hatches, hatch covers, superstructures and other ancillary equipment and facilities of the ship.

4. A measurement method for a ship modeling and contour measurement system based on image-laser point cloud fusion data according to any one of claims 1 to 3, characterized in that: The following steps are involved: Step 1: After the ship docks at the coast, the vision-lidar fusion acquisition device is controlled to move along the track from the leftmost end of the ship entering the image field to the rightmost end to complete the three-dimensional scanning of the ship and obtain the three-dimensional color point cloud and image data of the ship; Step 2: The ship 3D modeling module performs 3D modeling based on the 3D color point cloud and image data of the ship and based on downsampling and point cloud meshing algorithms; Step 3: The ship 3D profile measurement module is used to measure the length and width of the bulk carrier, as well as the profile data of key parts according to the 3D color point cloud and image data of the bulk carrier; Step 4: The model 3D visualization / model storage / data mutual verification module is used to display and store the 3D color point cloud of the bulk carrier; Step 5: The similar ship type data rapid measurement module uses a 3D point cloud matching algorithm to perform data alignment and rapid measurement on bulk carriers.

5. The measurement method of the ship modeling and contour measurement system based on image-laser point cloud fusion data according to claim 4 is characterized in that: The step 2 comprises the following steps: Step 2.1, transforming the collected complete three-dimensional color point cloud of the ship into an orthographic projection point cloud; Step 2.2: Use the semantic segmentation network to segment the hull, hatch and coast of the ortho-projected point cloud data, complete the separation of the ship and the shore, and project the image semantic segmentation results onto the 3D point cloud to obtain the point cloud contour of the ship surface; Step 2.3: Mesh the obtained ship point cloud to complete the three-dimensional model of the ship.

6. The measurement method of the ship modeling and contour measurement system based on image-laser point cloud fusion data according to claim 4 is characterized in that: The specific implementation method of step 3 is: Step 3.1: For the hull, directly input the point cloud contour of the ship surface obtained by semantic segmentation, and use the minimum circumscribed rectangle method to calculate the length and width of the ship; Step 3.2, for hatches, the number of hatches obtained by semantic segmentation is counted, and a straight line is fitted to the point cloud data of each hatch, and then the length and width of the hatch are measured based on the distance between the fitted lines on the opposite sides of the hatch; Step 3.3: For other ancillary facilities of the ship, the point cloud on the surface of the ship is used as the boundary, and the point cloud on the upper part is clustered and the outer contour bounding box is calculated to achieve the contour measurement of the ancillary facilities.

7. The measurement method of the ship modeling and contour measurement system based on image-laser point cloud fusion data according to claim 4 is characterized in that: The specific implementation method of step 4 is: identify the ship type through the YOLOv5 target detection network, call the stored three-dimensional model of the current operating model ship, and call the ship size data of the corresponding model in the database according to the ship model to evaluate the ship measurement accuracy, and at the same time realize the storage of the reconstructed ship three-dimensional model data.

8. The measurement method of the ship modeling and contour measurement system based on image-laser point cloud fusion data according to claim 4 is characterized in that: The specific implementation method of step 5 is as follows: the next time the ship enters the port, the scanned visual-laser fusion point cloud data is quickly compared with the data stored in the database. If the same data exists, the data alignment and rapid measurement of the current operating ship are completed based on the 3D point cloud matching algorithm, and the stored model is automatically called to avoid repeated measurements.

Citation Information

Patent Citations

  • Method for establishing ship model based on fusion of video image and laser radar point cloud

    CN114187409A

  • Ship size measurement method based on laser radar three-dimensional point cloud algorithm

    CN117706577A

  • Automatic ship loading control system

    CN117806223A

  • Multi-lidar based unloader automatic operation device and method

    KR102684207B1

Cited By

  • Bulk cargo ship hatch measuring method and system based on ship map and local point cloud

    CN122486472A