Bulk carrier image point cloud fusion scanning segmentation method

Abstract

The present application relates to the technical field of port operation, and discloses a bulk carrier image point cloud fusion scanning segmentation method, which is characterized by being realized based on a laser radar and a camera arranged on a port gantry crane. The method comprises the following steps: S1, at the same time, the laser radar scans a bulk carrier to obtain point cloud data, and the camera photographs the bulk carrier to obtain image data; S2, regions are divided in the point cloud data, and each region is labeled; regions are divided in the image data, and each region is labeled. The bulk carrier image point cloud fusion scanning segmentation method sets a multi-modal fusion deep learning model to realize the fusion region division of the point cloud data and the image data, and forms a label. Compared with the point cloud data, the method has higher accuracy, and provides a basis for the positioning of the hatch and the judgment of the internal material height information of the hatch.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of port operation, and particularly relates to a bulk carrier image point cloud fusion scanning segmentation method. BACKGROUND

[0002] Multi-layer light lidar is a rotating ranging radar that can simultaneously emit and receive multiple laser beams, can identify the height information of an object and obtain a 3D scanning map of the surrounding environment. The high-frequency electromagnetic waves emitted by the lidar can achieve all-weather anti-interference spatial information sensing capability of dozens of meters under certain emission power conditions.

[0003] A three-dimensional vehicle and ship scheduling method and device based on a laser scanning device are disclosed in CN116050590A, which realizes the scheduling operation of vehicles and ships by scanning the ship body through a laser radar, but the data obtained by the radar has some original problems, such as the rapid decrease in the spatial density of point clouds when the distance increases, which leads to poor detection ability of the edges of the radar and easy deviation in the detection of the attitude of the ship body and the hatch, which will directly affect the range detection of the hatch operation. SUMMARY

[0004] Therefore, the present application provides a bulk carrier image point cloud fusion scanning segmentation method to solve the problems of poor video data precision and the inability of point cloud information to effectively judge certain categories or abnormal conditions.

[0005] The technical solution of the present application is as follows: The present application provides a bulk carrier image point cloud fusion scanning segmentation method, which is realized based on a laser radar and a camera arranged on a port gantry. The method comprises the following steps:

[0006] S1. At the same time, the laser radar scans the bulk carrier to obtain point cloud data, and the camera photographs the bulk carrier to obtain image data;

[0007] S2. Regions are divided in the point cloud data, and each region is labeled, and regions are divided in the image data, and each region is labeled;

[0008] S3. The coordinate systems of the laser radar and the camera are converted so as to be in the same coordinate system;

[0009] S4. A registration algorithm is used to align the point cloud data and the image data to form training data;

[0010] S5. A multi-modal fusion deep learning model is designed, and the training data is used to train the model to obtain an optimal model;

[0011] S6. In the actual operation, the point cloud data and image data collected in real time in step S1 are input into the trained multimodal fusion deep learning model to obtain point cloud fusion data with predicted category point labels.

[0012] S7. Post-process the point cloud fusion data to obtain a visualized point cloud image.

[0013] Based on the above technical solutions, preferably, step S4 includes the following sub-steps:

[0014] S41. Extract feature points from point cloud data and image data, and use a matching algorithm to pair feature points;

[0015] S42. Calculate the preliminary transformation matrix between the point cloud data and the image data using paired feature points;

[0016] S43. Optimize the initial transformation matrix using an optimization algorithm;

[0017] S44. Apply the optimized transformation matrix to the point cloud data and align it to the image data to obtain aligned point cloud data and image data. Use the aligned point cloud data and image data as training data.

[0018] Based on the above technical solutions, preferably, before step S4, the internal and external parameters of the camera are calibrated.

[0019] More preferably, the matching algorithm in step S41 is:

[0020] D(x) = d1, d2, ..., d 128

[0021]

[0022]

[0023]

[0024] Where D(x) is the SIFT feature, M is the set of paired feature points, and v i It is the i-th element in the 128-dimensional descriptor of the center pixel, m i and It is a set of matching pairs, d i It is a SIFT descriptor in a 2D image. It is a SIFT descriptor in 3D point clouds.

[0025] Further preferably, the step S42 finds the best matching pairs between the point cloud data and the image data by a RANSAC algorithm, and obtains a preliminary transformation matrix corresponding to the best matching pairs, wherein the RANSAC algorithm is as follows:

[0026]

[0027]

[0028] wherein Θ is a set of all possible transformation matrices, t is a distance error threshold, w is a set of inlier pairs, n represents the number of corresponding point pairs, w i represents the i-th inlier pair in the set of inlier pairs, Θ is a set of all possible transformation matrices, P(m i |θ) is the probability of the matching pair m i under the given transformation matrix θ.

[0029] Further preferably, the optimization algorithm in the step S43 is as follows:

[0030]

[0031] wherein pi and qi represent the 3D points and 2D points in the corresponding point pairs respectively, h(p i ) is the result of transforming the i-th point in the 3D point cloud by the transformation matrix H, is the sum of squares of the Euclidean distances between the corresponding points.

[0032] On the basis of the above technical solution, preferably, the method further comprises a step S8 of selectively verifying the effectiveness of the multi-modal fusion deep learning model before the step S6.

[0033] Further preferably, the step S8 comprises the following sub-steps:

[0034] S81, selecting a plurality of feature points as a contrast feature group;

[0035] S82, obtaining the coordinates of the contrast feature group of the bulk carrier by shooting images to form an image contrast group, and obtaining the coordinates of the contrast feature group of the bulk carrier by point cloud data to form a point cloud contrast group;

[0036] S83, inputting the same image data and point cloud data into the multi-modal fusion deep learning model to obtain point cloud fusion data with corresponding labels of the contrast feature group;

[0037] S84, comparing the contrast feature group in the point cloud fusion data with corresponding labels of the contrast feature group with the image contrast group and the point cloud contrast group respectively to evaluate the accuracy of the model.

[0038] Preferably, the post-processing in step S7 includes threshold segmentation, smoothing processing and data visualization.

[0039] Further preferably, in step S84, if the model accuracy evaluation is unqualified, the model is optimized by modifying the model, increasing the training samples or modifying the hyperparameters.

[0040] The bulk cargo ship image point cloud fusion scanning segmentation method has the following beneficial effects compared with the prior art:

[0041] (1) By setting a multi-modal fusion deep learning model, the fusion area of point cloud data and image data is divided, and a label is formed, which has higher accuracy than point cloud data, especially for locating the hatch and judging the height information of the materials inside the hatch, and the collection of information such as the opening and closing state, specific position and material height of the hatch of the bulk cargo ship can provide information support for high-automation unloading operations and improve unloading efficiency.

[0042] (2) The post-processing of the model output data meets the visualization needs of port command and dispatch, which is beneficial to unloading planning and ship scheduling, and especially provides convenience for the operation of the gantry crane driver.

[0043] (3) The effectiveness of the model is verified before operation, which can avoid the failure of the model to provide information support in different environments. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1 The step schematic diagram of the bulk cargo ship image point cloud fusion scanning segmentation method of the present application;

[0046] Figure 2 The laser radar and camera installation position and bulk cargo ship structure schematic diagram of the bulk cargo ship image point cloud fusion scanning segmentation method of the present application;

[0047] Figures 3-4 The image data and point cloud data four corner points schematic diagram of the bulk cargo ship image point cloud fusion scanning segmentation method of the present application;

[0048] Figures 5-6 The model output post-processing visualization schematic diagram of the bulk cargo ship image point cloud fusion scanning segmentation method of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be clearly and completely described in combination with the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0050] In the actual unloading process of a port bulk cargo ship, the ship is needed to be grabbed by a portal crane through a hatch of the ship. When the ship is grabbed, the opening and closing state, the specific position and the material height of the hatch of the ship are needed to be collected to provide information support for highly automated unloading operation and improve unloading efficiency. Therefore, the method is proposed.

[0051] As shown in Figures 1-6 The bulk cargo ship image point cloud fusion scanning and segmentation method of the present application is realized based on a laser radar and a camera arranged on a port portal crane. Specifically, the laser radar and the camera can be arranged as a module. The module further includes a holder. The laser radar needs to be arranged as a rotatable component to better perform scanning operation on the ship body. In addition, a panoramic eagle camera is needed to be selected for the camera. In order to process the data collected by the laser radar and the camera, a corresponding computing device is needed to be provided. The module is installed at a position where the entire hatch can be seen. The camera can be installed in front of the driver's room, on the mechanism or on the wharf stand. In the embodiment, the camera is arranged below the elephant trunk beam of the portal crane, so that it has a better global view. The method includes steps S1-S8.

[0052] Step S1: At the same time, the laser radar scans the bulk cargo ship to obtain point cloud data, and the camera photographs the bulk cargo ship to obtain image data.

[0053] In this step, the collected point cloud data and image data both have time attributes and need to be obtained by scanning or photographing at the same time. Otherwise, problems such as subsequent data misalignment may occur.

[0054] Step S2: Regions are divided in the point cloud data, and each region is labeled. Regions are divided in the image data, and each region is labeled.

[0055] In the collected point cloud data and image data, the features of the bulk cargo ship, such as the bow, the body, the stern, the cover plate, the partition plate and the material, are included. In the point cloud data and the image data, different feature categories need to be classified and labeled. For the point cloud data, the regions of the bow, the body, the stern, the cover plate, the partition plate and the material can be divided and labeled respectively. For the image data, only partial features need to be classified and labeled, such as the hatchway and the cover plate state, for subsequent alignment of the point cloud data and the image data.

[0056] It should be noted that this step is a preparation step for training data, and the point cloud data and the image data collected at the same time are taken as a group. At least six thousand groups of data are needed for subsequent model training.

[0057] Step S3: Convert the coordinate systems of the laser radar and the camera to the same coordinate system.

[0058] Before the coordinate system conversion, the camera internal and external parameters need to be calibrated, including the focal length, distortion coefficient, rotation and translation information of the camera, which can be directly estimated by using camera calibration technology.

[0059] This step converts the coordinate systems of the camera and the laser radar to the same coordinate system, which is convenient for subsequent registration. In addition, the height of the bulk cargo ship features such as the hatchway can be measured by the laser radar to determine the shooting angle of the camera and the position of the hatchway.

[0060] Step S4: Align the point cloud data and the image data using a registration algorithm to form training data.

[0061] Specifically, the method used is a point cloud and image instance segmentation registration method. The feature points in the camera and the point cloud data are matched to calculate the relative pose information, and then they are aligned through transformation. The segmentation here refers to the region division in the previous labeling process.

[0062] This step can be specifically divided into steps S41-S44.

[0063] Step S41: Extract feature points from the point cloud data and the image data, and use a matching algorithm to pair the feature points.

[0064] The image data is regarded as a 2D image, and the point cloud data is regarded as 3D point cloud data. Feature points are extracted from the 2D image and the 3D point cloud data, such as the hatch corner point, the SIFT feature or the cabin. Then, a matching algorithm (such as nearest neighbor matching or FLANN matching) is used to pair the 2D image feature points and the 3D point cloud feature points.

[0065] The matching algorithm in this step is:

[0066] D(x) = d1, d2,..., d 128

[0067]

[0068]

[0069]

[0070] where D(x) is the SIFT feature, M is the set of matched feature points, v i is the i-th element of the 128-dimensional descriptor of the center pixel, mi and is a set of matched pairs, d i is the SIFT descriptor in 2D image, is the SIFT descriptor in 3D point cloud.

[0071] Step S42: Calculate the initial transformation matrix between the point cloud data and the image data using the matched feature points.

[0072] Calculate the initial transformation matrix between the 2D image and the 3D point cloud using the matched feature points, for example, RANSAC algorithm or ICP algorithm, RANSAC algorithm is a robust estimation method based on random sampling, used to find the best matched pairs.

[0073] Find the best matched pairs between the point cloud data and the image data by RANSAC algorithm, get the corresponding initial transformation matrix by the best matched pairs, the RANSAC algorithm is:

[0074]

[0075]

[0076] where Θ is the set of all possible transformation matrices, t is the distance error threshold, w is the set of inlier pairs, n represents the number of corresponding point pairs, w i represents the i-th inlier pair in the inlier pair set, θ is the set of all possible transformation matrices, P(m i |θ) is the probability of the matched pair m i under the given transformation matrix θ, find the transformation matrix with the most inlier pairs by this function, get the maximum consensus set.

[0077] Step S43: Optimize the initial transformation matrix using optimization algorithm.

[0078] The transformation matrix is further optimized using an optimization algorithm (e.g. least squares or non-linear optimization) to minimize the distance error between the 2D image and the 3D point cloud, with the 3D transformation into the spatial coordinate system, and the camera is tilted, in this step, an evaluation function is usually needed to measure the goodness of the optimization result, for example, root mean square error (RMSE), this paper uses non-linear optimization to better fit the error caused by angle inconsistency.

[0079] The optimization algorithm in this step is:

[0080]

[0081] where p i and q i represent the 3D points and 2D points in the corresponding point pair respectively, h(p x ) is the result of transforming the i-th point in the 3D point cloud by the transformation matrix H, y is the sum of the squares of the Euclidean distances between the corresponding points, and the function is used to find the minimum error re-projection matrix.

[0082] Step S44: Apply the optimized transformation matrix to the point cloud data to align it to the image data, obtain the aligned point cloud data and image data, and use the aligned point cloud data and image data as training data.

[0083] The specific formula is:

[0084]

[0085] where O is the origin of the coordinate system, XYZ axes form the world coordinate system, and uv is the image coordinate system in pixels, if the coordinates of the object point P in the world coordinate system are (x, y, z), the corresponding image point p in the image coordinate system is (u, v), f x and f y represent the horizontal and vertical focal lengths of the camera respectively, c x and c y represent the horizontal and vertical coordinates of the image center, r 11 to r 33 represent the 9 elements of the rotation matrix, t x , t y and t z represent the 3 elements of the translation vector, T 1，1 to T 3,4 represent the combination of translation and rotation elements in the camera extrinsic parameters, this formula can be regarded as a transformation matrix from the camera coordinate system to the image coordinate system.

[0086] where r 11 to r 33can be extracted from the transformation matrix H, i.e. H = R + t, thus, after knowing the transformation matrix H, the point p in the 3D point cloud can be transformed to the point q on the 2D image directly according to the above formula.

[0087] In this step, the region annotation in the point cloud data will be mapped to the image data.

[0088] Step S5: Design a multi-modal fusion deep learning model and train it using the training data to obtain the optimal model.

[0089] The deep neural network can use a convolutional neural network (CNN) and can be implemented using a variational autoencoder (VAE) architecture. In the CNN architecture, some convolutional layers and pooling layers can be used to extract features from the camera data and the point cloud data and fuse them together to generate aligned 3D point clouds and images. In the VAE architecture, the camera data and the point cloud data can be encoded into the distribution of the latent space, and their encodings can be fused to generate the fused 3D point cloud and image.

[0090] When designing a neural network model for image and point cloud fusion, the differences in feature representation between point clouds and images need to be considered, as well as how to combine the features of the two. Therefore, the multi-modal fusion deep learning model needs to be designed and optimized as follows.

[0091] Multi-branch structure: Since point clouds and images have different feature representation methods, it is common to design multiple branches to extract features from both. These branches can share some convolutional layers to reduce the number of parameters.

[0092] Fusion layer design: Fusing features from different branches is a key step, and the design of the fusion layer needs to consider the differences in feature representation between point clouds and images and how to fuse them. Common fusion methods include modal weighting and attention mechanisms.

[0093] Multi-level feature fusion: For the feature representation of point clouds and images, multi-level features can be used for fusion to obtain more rich information. For example, lower-level features can be used for local feature representation, while higher-level features can be used for global feature representation.

[0094] Preservation of spatial information: Point clouds and images both have spatial information, so the model structure design needs to consider how to preserve this information. Common methods include using 3D convolution, dilated convolution, etc.

[0095] Selection of optimization algorithm: In model training, a suitable optimization algorithm needs to be selected for parameter update. For point cloud and image fusion models, common optimization algorithms include SGD, Adam, etc. At the same time, attention should be paid to the setting of learning rate, the use of regularization, etc. during training to avoid overfitting and underfitting.

[0096] In the training process, at least six thousand sets of training data are used as the training set, and the training data in the training set are divided into multiple single input sets according to the parameters of the computing device. In one training process, the single input sets are input into the model in turn until the training set is completely input, which is one iteration training. The optimal model is saved as the model for actual use after repeating the iteration training for more than 200 times.

[0097] Step S6: In the actual operation process, the point cloud data and image data collected in real time in step S1 are input into the trained multi-modal fusion deep learning model to obtain point cloud fusion data with predicted category point labels.

[0098] The point cloud data and image data collected in real time in step S1 are different from the point cloud data and image data of the training data.

[0099] In the deep neural network training process, a training set with real alignment data is usually used for supervised learning. During the training process, the network gradually learns how to align the camera data and the point cloud data, and generates the fused 3D point cloud and image. In the test, different image data and point cloud data from the training set are input, and the network outputs the fused 3D point cloud and image. After multiple iteration training, the neural network weight parameters after feature extraction are obtained. Through the weight parameters, category prediction and region labeling are completed to form the predicted category point labels, so that the output point cloud fusion data has region division category labels.

[0100] Step S7: Post-processing the point cloud fusion data to obtain a visual point cloud image.

[0101] The post-processing includes threshold segmentation, smoothing processing and data visualization. The point cloud fusion data is segmented by region division labeling and threshold segmentation, and different colors are used to represent the regions of hatch cover, bow, stern, material, hatch cover plate, etc. The segmented regions have obvious burrs. At this time, the burrs are removed through smoothing processing, and the visual point cloud image is obtained through visualization processing. In the point cloud image, there is a kind of region division of the ship body.

[0102] Step S8: Selectively verify the effectiveness of the multi-modal fusion deep learning model before step S6.

[0103] Due to the size difference of bulk carriers and other reasons, in some cases, the effectiveness of the multimodal fusion deep learning model needs to be verified, and the verification method includes steps S81-S84.

[0104] Step S81: Select a plurality of feature points as a contrast feature group.

[0105] In the present embodiment, four corner points of a hatch opening are selected as feature points to form a contrast feature group.

[0106] Step S82: Obtain the coordinates of the contrast feature group of the bulk carrier through the image to form an image contrast group, and obtain the coordinates of the contrast feature group of the bulk carrier through the point cloud data to form a point cloud contrast group.

[0107] The spatial coordinates of the four corner points are calculated in the photographed image through the image algorithm, and the spatial coordinates are taken as the image contrast group. Similarly, the point cloud algorithm is used to process the point cloud data to obtain the spatial coordinates of the four corner points of the same hatch opening, and the spatial coordinates are taken as the point cloud contrast group.

[0108] Step S83: Input the same image data and point cloud data into the multimodal fusion deep learning model to obtain point cloud fusion data with corresponding labels of the contrast feature group.

[0109] The four corner points of the same hatch opening are input into the model, and the region of the hatch opening is divided in the point cloud fusion data, and the spatial coordinates of the four corner points are calculated.

[0110] Step S84: Compare the contrast feature group in the point cloud fusion data with corresponding labels of the contrast feature group with the image contrast group and the point cloud contrast group respectively to evaluate the accuracy of the model.

[0111] The spatial coordinates output by the model are compared with the point cloud contrast group and the image contrast group respectively to evaluate the accuracy of the hatch opening position. In the present embodiment, the average corner point distance is used to measure the effectiveness of the model.

[0112] The average corner point distance refers to the average value of the distance between any two adjacent corner points in a plane figure. Let the plane figure have n corner points, l i represents the length of the ith edge, s i represents the sine value of the ith interior angle, and the formula of the average corner point distance is:

[0113]

[0114] wherein, The length of the opposite side of the i-th inner angle is used as one of the model accuracy indicators by comparing the size of the average angle point. In addition, the accuracy of the angle point distance can be used as an indicator.

[0115] In this embodiment, the ship hatch is required to achieve a segmentation accuracy of 0.2 m under the condition of 95% scene. If the model accuracy evaluation is not qualified, the model needs to be optimized and improved by modifying the model, increasing the training samples or modifying the hyperparameters, etc. to meet the needs of actual production operations.

[0116] In addition, in one specific embodiment, the door machine material grabbing frequency is two minutes, and the scanning of the bulk cargo ship can be selected every 30-120 seconds to meet the attitude detection and material level detection of the bulk cargo ship during the material grabbing process.

[0117] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A bulk carrier image point cloud fusion scanning segmentation method, characterized in that, The laser radar and the camera are arranged on a port door machine, and the specific steps are as follows: S1. At the same time, the laser radar scans the bulk carrier to obtain point cloud data, and the camera photographs the bulk carrier to obtain image data; S2. After dividing the regions in the point cloud data, the ship head, ship body, ship tail, cover plate, partition plate and material are labeled, and after dividing the regions in the image data, the hatch and cover plate state are labeled; S3. The coordinate systems of the laser radar and the camera are converted to be in the same coordinate system; S4. The labeled point cloud data and the labeled image data are aligned to form training data; The step S4 includes the following sub-steps: S41. Feature points are extracted from the labeled point cloud data and the labeled image data, and a matching algorithm is used for feature point pairing; S42. The paired feature points are used to calculate the preliminary transformation matrix between the labeled point cloud data and the labeled image data; S43. An optimization algorithm is used to optimize the preliminary transformation matrix; S44. The optimized transformation matrix is applied to the labeled point cloud data, and the labeled point cloud data is aligned to the image data to obtain aligned point cloud data and image data, and the aligned point cloud data and image data are used as training data; The matching algorithm in step S41 is: ； where D(x) is the SIFT feature, M is the set of matched feature points, is the i-th element in the 128-dimensional descriptor of the center pixel, m i and is a set of matched pairs, is the SIFT descriptor in 2D image, is the SIFT descriptor in 3D point cloud; In step S42, the best matching pair between the labeled point cloud data and the labeled image data is found first, and then the corresponding preliminary transformation matrix is obtained through the best matching pair, which specifically includes: w= ； ； where A = (a, b, c, d, e, f)T , and A < t, Θ is the set of all possible transformation matrices, t is the distance error threshold, w is the set of inlier point pairs, and n denotes the number of corresponding point pairs, is the set of all possible transformation matrices, is the probability of matching the pair under the given transformation matrix . S5. A multi-modal fusion deep learning model is designed, and the training data is used to train the model to obtain an optimal model; S6. In the actual operation process, the point cloud data and image data collected in real time in step S1 are input into the trained multi-modal fusion deep learning model to obtain point cloud fusion data with predicted category point labels; S7. The point cloud fusion data is post-processed to obtain a visual point cloud image; Before step S6, it further includes: selecting a plurality of feature points as a comparison feature group; obtaining the coordinates of the comparison feature group of the bulk carrier through the image to form an image comparison group, and obtaining the coordinates of the comparison feature group of the bulk carrier through the point cloud data to form a point cloud comparison group; inputting the same image data and point cloud data into the multi-modal fusion deep learning model to obtain point cloud fusion data with corresponding labels of the comparison feature group; and comparing the positions of the comparison feature group in the point cloud fusion data with corresponding labels of the comparison feature group, the image comparison group and the point cloud comparison group, respectively, to evaluate the accuracy of the model.

2. The bulk carrier image point cloud fusion scan segmentation method of claim 1, wherein, Before step S4, the internal and external parameters of the camera are calibrated.

3. The bulk carrier image point cloud fusion scan segmentation method of claim 1, wherein, The post-processing in step S7 includes threshold segmentation, smoothing processing and data visualization.

4. The bulk carrier image point cloud fusion scan segmentation method of claim 1, wherein, If the model accuracy evaluation is unqualified, the model is optimized by modifying the model, increasing training samples or modifying hyperparameters.

Images