Ship point cloud completion method and system based on multi-modal data fusion

Abstract

The invention discloses a ship point cloud completion method and system based on multi-modal data fusion. The method comprises the following steps: mapping a two-dimensional ship image into a reconstruction point cloud with a complete global structure through a coding-conversion-decoding network architecture; performing coarse registration and fine registration on the original ship point cloud and the reconstructed point cloud to obtain a target registration point cloud; sampling the target registration point cloud by adopting a Poisson disk to obtain uniformly distributed registration point clouds; dividing the uniformly distributed registration point clouds into high-precision point clouds and low-precision point clouds according to chamfering distances; extracting features of the two-dimensional ship image, the high-precision point cloud and the low-precision point cloud and carrying out feature fusion to obtain fused features; according to the fusion features, the high-precision point clouds and the low-precision point clouds are optimized, complemented and combined, and ship complemented point clouds are obtained. According to the method, multi-modal data fusion and offset prediction and movement based on fusion features are adopted, ship point cloud completion is achieved, and the integrity and precision of ship point cloud data are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional point cloud processing. Specifically, it relates to a method and system for ship point cloud completion based on multi-modal data fusion. Background Art

[0002] Three-dimensional point cloud data has important application value in the ship field. However, the actually collected ship point clouds are often incomplete due to problems such as occlusion, noise, and uneven density, which affect subsequent modeling and analysis.

[0003] There are many deficiencies in the existing technologies for three-dimensional point cloud restoration and refinement. First, the problem of insufficient information is particularly prominent. A single modality (relying only on point cloud data) is difficult to effectively restore large-scale missing structures, resulting in obvious deficiencies in the overall shape and detail performance of the generated point cloud. Second, the difficulty of cross-modal fusion is also an urgent problem to be solved. There is a lack of effective methods for feature alignment and joint optimization between images and point clouds, making it difficult for data of different modalities to fully complement each other, thus affecting the accuracy and integrity of point cloud restoration. In addition, the current global refinement strategy often fails to retain the details of local high-precision regions when processing point clouds, resulting in detail loss and affecting the quality and application effect of the final model.

[0004] In terms of the limitations of the existing technologies, geometric-based methods (such as Laplacian smoothing) can fill small holes to a certain extent, but their applicable range is limited and they cannot handle the restoration tasks of large-scale missing regions. Alignment-based methods (such as three-dimensional template matching) face the problems of high computational cost and high sensitivity to noise, which limit their popularization and application in practical applications. In addition, single-modal deep learning models (such as PCN, TopNet) have a significant decline in the generation effect when the input data is severely missing, and it is difficult to meet the requirements of high-quality point cloud generation.

[0005] Therefore, there is an urgent need for a multi-modal data fusion method to improve the overall accuracy and robustness of ship point cloud completion. Summary of the Invention

[0006] An embodiment of the present invention provides a method and system for ship point cloud completion based on multi-modal data fusion to solve the problems of weak global structure inference ability and low cross-modal fusion efficiency in the existing technologies for ship point cloud completion using geometric filling, template matching, and single-modal deep learning models.

[0007] To achieve the above object, on the one hand, the present invention provides a ship point cloud completion method based on multi-modal data fusion, and the method includes: S1. Mapping a two-dimensional ship image into a reconstructed point cloud with a complete global structure through an encoding-conversion-decoding network architecture; S2. Coarsely registering and finely registering the original ship point cloud with the reconstructed point cloud to obtain a target registered point cloud; S3. Using Poisson disk sampling for the target registered point cloud to obtain a uniformly distributed registered point cloud; S4. Dividing the uniformly distributed registered point cloud into a high-precision point cloud and a low-precision point cloud according to the chamfer distance; S5. Extracting features from the two-dimensional ship image, the high-precision point cloud and the low-precision point cloud, and performing feature fusion to obtain a fused feature; S6. Optimally completing and merging the high-precision point cloud and the low-precision point cloud respectively according to the fused feature to obtain a ship-completed point cloud.

[0008] Optionally, S1 includes: S11. Extracting multi-scale features from the two-dimensional ship image through multi-layer convolution operations in the encoding stage; S12. Converting the extracted multi-scale features into a three-dimensional initial point cloud through a fully connected layer or a reshaping operation in the conversion stage; S13. Outputting the reconstructed point cloud with a complete global structure through multi-layer transposed convolution operations in the decoding stage.

[0009] Optionally, S2 includes: S21. Coarsely registering the original ship point cloud with the reconstructed point cloud to obtain an initial registered point cloud; S22. Finely registering the original ship point cloud with the initial registered point cloud to obtain a target registered point cloud.

[0010] Optionally, S21 includes: S211. Extracting key feature points from the original ship point cloud and the reconstructed point cloud, and calculating descriptors for each feature point; S212. Screening out a set of matching feature point pairs according to the similarity of the descriptors of all feature points of the two point clouds; S213. Estimating rigid transformation parameters according to the screened set of matching feature point pairs; S214. Transforming the reconstructed point cloud according to the estimated rigid transformation parameters to obtain the initial registered point cloud.

[0011] Optionally, S22 includes: S221. Searching for the nearest point in the initial registered point cloud for each point in the original ship point cloud to form a current point pair set; S222. Re-estimating the rigid transformation parameters according to the current point pair set; S223. Updating the initial registered point cloud according to the re-estimated rigid transformation parameters; and calculating the error between the updated initial registered point cloud and the original ship point cloud; S224. Repeating S221~S223 until the error is lower than a preset threshold or the maximum number of iterations is reached, stopping the update, and taking the last updated initial registered point cloud as the target registered point cloud.

[0012] Optionally, S4 includes: S41. For each point in the uniformly distributed registered point cloud, calculate its nearest neighbor distance to the original ship point cloud; S42. Randomly divide the uniformly distributed registered point cloud into two point sets and calculate the chamfer distance between the two point sets; S43. Divide the uniformly distributed registered point cloud into a high-precision point cloud and a low-precision point cloud according to the nearest neighbor distance and chamfer distance of each point in the uniformly distributed registered point cloud.

[0013] Optionally, S5 includes: S51. Use a lightweight convolutional neural network to extract semantic features from the two-dimensional ship image and map them to the point cloud resolution to obtain point-by-point image features; S52. Use the FPFH descriptor to extract local geometric features from the high-precision point cloud and the low-precision point cloud to obtain point cloud features; S53. Directly splice the point-by-point image features and the point cloud features along the channel dimension and simplify the feature dimension to obtain the fusion features.

[0014] Optionally, S6 includes: S61. Calculate the average distance between points in the uniformly distributed registered point cloud and set the constraints for the high-precision point cloud and the constraints for the low-precision point cloud according to the average distance between points; S62. Use the fusion features to predict the high-precision offset for the high-precision point cloud; use the fusion features and a 3-layer MLP to predict the low-precision offset for the low-precision point cloud; S63. Update the high-precision point cloud according to the high-precision offset and the constraints of the high-precision point cloud; update the low-precision point cloud according to the low-precision offset and the constraints of the low-precision point cloud; S64. Merge the updated high-precision point cloud and the updated low-precision point cloud to obtain the ship-completed point cloud.

[0015] On the other hand, the present invention provides a ship point cloud completion system based on multi-modal data fusion. The system includes: a point cloud reconstruction unit for mapping a two-dimensional ship image into a reconstructed point cloud with a complete global structure through an encoding-conversion-decoding network architecture; a point cloud registration unit for performing rough registration and fine registration on the original ship point cloud and the reconstructed point cloud to obtain a target registered point cloud; a point cloud optimization unit for using Poisson disk sampling on the target registered point cloud to obtain a uniformly distributed registered point cloud; a point cloud division unit for dividing the uniformly distributed registered point cloud into a high-precision point cloud and a low-precision point cloud according to the chamfer distance; a feature fusion unit for extracting features from the two-dimensional ship image, the high-precision point cloud, and the low-precision point cloud and performing feature fusion to obtain fusion features; a point cloud completion unit for respectively optimizing and completing the high-precision point cloud and the low-precision point cloud according to the fusion features and merging them to obtain the ship-completed point cloud.

[0016] Optionally, the point cloud reconstruction unit includes: an encoding subunit, configured to extract multi-scale features from the two-dimensional ship image through multi-layer convolutional operations during the encoding phase; a conversion subunit, configured to convert the extracted multi-scale features into a three-dimensional initial point cloud through a fully connected layer or a reshaping operation during the conversion phase; and a decoding subunit, configured to output the reconstructed point cloud with a complete global structure through multi-layer deconvolutional operations during the decoding phase.

[0017] Advantages of the present invention:

[0018] The present invention provides a ship point cloud completion method and system based on multi-modal data fusion. Among them, the method effectively compensates for the deficiencies of a single modality in structural inference and detail restoration through cross-modal fusion of two-dimensional ship images and original ship point cloud data; uses a strategy combining coarse registration and fine registration to achieve precise alignment in space between the reconstructed point cloud and the original ship point cloud, ensuring data consistency; adopts Poisson disk sampling technology to solve the problem of uneven point cloud density and improve the uniformity of point cloud distribution; differentiates local details and global structures through chamfer distance adaptive partitioning, which divides the point cloud into high-precision and low-precision regions; multi-modal feature fusion and offset prediction and movement based on the fused features achieve effective completion of point cloud data in high-precision and low-precision regions, significantly improving the integrity and accuracy of ship point cloud data. Description of the Drawings

[0019] Figure 1 is a flowchart of a ship point cloud completion method based on multi-modal data fusion provided by an embodiment of the present invention;

[0020] Figure 2 is a flowchart of mapping a two-dimensional ship image to a reconstructed point cloud provided by an embodiment of the present invention;

[0021] Figure 3 is a flowchart of registering the original ship point cloud and the reconstructed point cloud provided by an embodiment of the present invention;

[0022] Figure 4 is a flowchart of coarse registration of the original ship point cloud and the reconstructed point cloud provided by an embodiment of the present invention;

[0023] Figure 5 is a flowchart of fine registration of the original ship point cloud and the initially registered point cloud provided by an embodiment of the present invention;

[0024] Figure 6 is a flowchart of dividing the uniformly distributed registered point cloud provided by an embodiment of the present invention;

[0025] Figure 7 is a flowchart of feature fusion provided by an embodiment of the present invention;

[0026] Figure 8 It is a flowchart of point cloud completion provided by an embodiment of the present invention;

[0027] Figure 9 It is a schematic structural diagram of a ship point cloud completion system based on multi-modal data fusion provided by an embodiment of the present invention;

[0028] Figure 10 It is a schematic structural diagram of a point cloud reconstruction unit provided by an embodiment of the present invention;

[0029] Figure 11 It is a schematic structural diagram of a point cloud registration unit provided by an embodiment of the present invention;

[0030] Figure 12 It is a schematic structural diagram of a rough point cloud registration sub-unit provided by an embodiment of the present invention;

[0031] Figure 13 It is a schematic structural diagram of a fine point cloud registration sub-unit provided by an embodiment of the present invention;

[0032] Figure 14 It is a schematic structural diagram of a point cloud division unit provided by an embodiment of the present invention;

[0033] Figure 15 It is a schematic structural diagram of a feature fusion unit provided by an embodiment of the present invention;

[0034] Figure 16 It is a schematic structural diagram of a point cloud completion unit provided by an embodiment of the present invention. Detailed implementation manners

[0035] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0036] Figure 1 It is a flowchart of a method for ship point cloud completion based on multi-modal data fusion provided by an embodiment of the present invention. As Figure 1 shown, the method includes:

[0037] S1. Map a two-dimensional ship image into a reconstructed point cloud with a complete global structure through an encoding-conversion-decoding network architecture;

[0038] Figure 2 It is a flowchart of mapping a two-dimensional ship image into a reconstructed point cloud provided by an embodiment of the present invention. As Figure 2 shown, the S1 includes:

[0039] S11. Extract multi-scale features from the two-dimensional ship image through multi-layer convolution operations during the encoding stage;

[0040] In an alternative embodiment, during the encoding stage, after preprocessing (such as normalization, denoising, contrast enhancement, etc.) the two-dimensional ship image (single-view image), it is fed into the encoder. The encoder adopts a 7-layer convolution structure, gradually compressing the spatial dimension and extracting multi-scale features. The first convolution layer uses a 3×3 convolution kernel, and the number of channels is set to 16, which is used to capture low-level features such as edges and textures of the image; then gradually increase the convolution kernel size and the number of channels (such as from 3×3 → 5×5, and the number of channels from 32, 64 to 128, 256) to extract multi-scale and multi-level feature information. After each layer of convolution, the model's expressive ability is enhanced through a non-linear activation function (such as ReLU), and at the same time, the feature map size is gradually reduced through stride adjustment, forming a high-dimensional and low-resolution latent feature representation. This process compresses the input image from the original RGB pixel space into a high-dimensional feature containing object geometry and semantic information.

[0041] For example:

[0042] Layer 1: 3×3 convolution, output channels 16, stride 2, activation function ReLU;

[0043] Layers 2 - 4: 3×3 convolution, the number of channels is 32, 64, 128 in sequence, stride 2, activation function ReLU;

[0044] Layers 5 - 7: 5×5 convolution, the number of channels 256, stride 1, activation function ReLU; the output feature map size is 5×5×256.

[0045] S12. Convert the extracted multi-scale features into a three-dimensional initial point cloud through a fully connected layer or a reshaping operation during the conversion stage;

[0046] In an alternative embodiment, during the conversion stage, the high-dimensional features output by the encoder (such as 5×5, 256) are converted into a three-dimensional initial point cloud (7×7×512) through a fully connected layer or a reshaping operation. This step essentially aligns the two-dimensional ship image features with the three-dimensional point cloud space, and maps the compressed features to the preset three-dimensional point cloud dimension through a linear transformation. The 7×7×512 dimension here means generating the three-dimensional coordinates of 512 points, and each point is distributed in a 7×7 grid, providing a basic structure for subsequent refinement.

[0047] S13. Output the reconstructed point cloud with a complete global structure from the three-dimensional initial point cloud through multi-layer deconvolution operations during the decoding stage.

[0048] In an optional embodiment, in the decoding stage, the three-dimensional initial point cloud is gradually restored in spatial resolution and the details of the point cloud are refined through three layers of deconvolution operations (such as 5×5, 256→5×5, 128→3×3, 64). The deconvolution layer enlarges the size of the feature map through upsampling and combines the skip connection information in the encoding stage to effectively compensate for the detailed information that may be lost during the deconvolution process. Finally, the network compresses the number of channels to 3 through a 3×3, 3 convolutional layer and directly outputs the XYZ coordinates of each point, thereby forming a reconstructed point cloud with a complete global structure. Throughout the process, the network adopts an end-to-end training strategy and uses the chamfer distance as the main loss function to ensure that the reconstructed point cloud is highly consistent with the real point cloud in terms of global structure and local details.

[0049] The advantage of this method is that it directly establishes the mapping relationship from the image to the point cloud, avoiding the multi-view dependence or intermediate steps of depth estimation in traditional methods. However, its limitation is that the single-view input results in incomplete reconstruction of occluded areas, and the point cloud density is limited by the preset output dimension. By adjusting the number of deconvolution layers and the number of channels, a balance can be achieved between computational efficiency and reconstruction accuracy.

[0050] S2. Coarsely register and finely register the original ship point cloud with the reconstructed point cloud to obtain the target registered point cloud;

[0051] Figure 3 is the flowchart of registering the original ship point cloud with the reconstructed point cloud provided by the embodiment of the present invention; as Figure 3 shown, the S2 includes:

[0052] S21. Coarsely register the original ship point cloud with the reconstructed point cloud to obtain the initial registered point cloud;

[0053] Figure 4 is the flowchart of coarsely registering the original ship point cloud with the reconstructed point cloud provided by the embodiment of the present invention; as Figure 4 shown, the S21 includes:

[0054] S211. Extract key feature points from the original ship point cloud and the reconstructed point cloud, and calculate descriptors for each feature point;

[0055] Extract key feature points with representativeness and rich local geometric information from the original ship point cloud and the reconstructed point cloud . Usually, these feature points are located at the edges, corners or regions with significant local curvature changes of the point cloud, and can fully reflect the geometric structure of the object.

[0056] Generate a high-dimensional descriptor for each extracted feature point to capture the local geometric information around it.

[0057] For example, using the FPFH (Fast Point Feature Histograms) descriptor:

[0058] Neighborhood determination: With each feature point as the center, collect neighborhood points within a certain radius;

[0059] Normal vector calculation: Calculate the normal vector of each point within the neighborhood, providing a basis for subsequent description construction;

[0060] Histogram construction: Statistically analyze information such as the angles and distances between points within the neighborhood and the center point, and construct a histogram as the feature description;

[0061] Normalization processing: Normalize the histogram data to reduce the impact brought by scale differences;

[0062] Through this process, each feature point is attached with a descriptor vector, representing its local geometric structure.

[0063] S212. According to the similarity of the descriptors of all feature points of the two point clouds, screen out the set of matching feature point pairs;

[0064] For all feature points in the original ship point cloud and the reconstructed point cloud, calculate the similarity between their respective descriptors. Usually, the Euclidean distance or cosine similarity is used to measure the similarity between two descriptors. Screen out the feature point pairs with a similarity higher than 0.8.

[0065] For example: If the similarity between the descriptor of the current feature point in the original ship point cloud and the descriptor of one of the feature points in the reconstructed point cloud is higher than 0.8, then the two feature points are used as a matching feature point pair; Use the above method to calculate the similarity of the descriptors of all feature points of the two point clouds, and screen out the set of matching feature point pairs.

[0066] S213. According to the set of matching feature point pairs screened out, estimate the rigid transformation parameters;

[0067] The rigid transformation parameters include: rotation matrix R and translation vector t; Set the set of matching point pairs as , where is the feature point in the original ship point cloud, is the matching feature point in the corresponding reconstructed point cloud, and K is the number of matching feature point pairs.

[0068] According to the set of matching feature point pairs screened out, it is necessary to find a rotation matrix R and a translation vector t, that is, transform each point in Q to , making it as close as possible to the corresponding . Expressed in mathematical form as:

[0069]

[0070] To measure the alignment degree between point pairs, the square of the Euclidean distance is usually used as the error metric, and then the least squares optimization objective is formed:

[0071]

[0072] The physical meaning of this objective function is: to find the rigid transformation (rotation + translation) that minimizes the sum of the distances of all matching feature point pairs

[0073] The SVD (Singular Value Decomposition) is usually used to solve for the optimal R and t. The solution process is as follows:

[0074] (1) First, calculate the centroids of the matching feature point pairs of the original ship point cloud P and the reconstructed point cloud Q in their respective coordinate systems:

[0075]

[0076] Centroid and respectively represent the average positions of the two groups of point clouds in space.

[0077] (2) Centralize all matching feature point pairs with respect to their respective centroids to obtain new point coordinates:

[0078]

[0079] (3) Define the covariance matrix H:

[0080]

[0081] where and are both 3×1 column vectors. Therefore, is a 3×3 matrix, and the final 3×3 covariance matrix H is obtained after summing all matching feature point pairs.

[0082] Next, perform the singular value (SVD) decomposition on H:

[0083]

[0084] where U and V are 3×3 orthogonal matrices, and Σ is a diagonal singular value matrix (the diagonal elements are non-negative real numbers and are sorted from largest to smallest).

[0085] (4) According to the SVD result, the optimal rotation matrix can be obtained:

[0086]

[0087] ​If the determinant of the matrix \(R\), \(\det(R)<0\), then correct \(V\) or \(U\) to ensure that \(R\) is an orthogonal matrix.

[0088] (5) Calculate the translation vector through the centroid relationship:

[0089]

[0090] Recall the objective function to be minimized:

[0091]

[0092] Through the above steps, it can be proven that the \(R\) and \(t\) obtained by SVD are exactly the solutions that minimize the objective function.

[0093] S214. Transform the reconstructed point cloud according to the estimated rigid transformation parameters to obtain the initial registered point cloud.

[0094] After obtaining \(R\) and \(t\), all points in the reconstructed point cloud \(Q\) can be transformed to obtain the initial registered point cloud:

[0095]

[0096] Initial registered point cloud Is initially aligned with the original ship point cloud \(P\), thus completing the rough registration step.

[0097] S22. Perform fine registration on the original ship point cloud and the initial registered point cloud to obtain the target registered point cloud.

[0098] Figure 5 Is the flowchart of the fine registration of the original ship point cloud and the initial registered point cloud provided by the embodiment of the present invention; as Figure 5 Shown, the S22 includes:

[0099] S221. For each point in the original ship point cloud, search for the nearest point in the initial registered point cloud to form the current point pair set;

[0100] For each , search for the point in that is the closest to to form the current point pair set .

[0101] S222. Re - estimate the rigid transformation parameters according to the current point pair set;

[0102] Use a method similar to S213 to re - estimate the rigid transformation parameters, including: rotation matrix \(R\) and translation vector \(t\).

[0103] S223. Update the initial registered point cloud according to the re - estimated rigid transformation parameters; and calculate the error between the updated initial registered point cloud and the original ship point cloud.

[0104] The error is calculated according to the following formula:

[0105]

[0106] where E is the error, is the original ship point cloud, is the updated initial registered point cloud, and N is the number of points in the original ship point cloud or the initial registered point cloud.

[0107] S224. Repeat S221 - S223 until the error is lower than the preset threshold or the maximum number of iterations is reached, then stop the update and use the last updated initial registered point cloud as the target registered point cloud.

[0108] Judge whether to continue the iteration according to the calculated error. If E≥ and the number of iterations has not reached the maximum number of iterations, then use the updated initial registered point cloud as the new initial registered point cloud, return to step S221, and continue the nearest - neighbor matching, re - estimate the rigid transformation parameters, update the initial registered point cloud and calculate the error; when the error E is lower than the preset threshold , or the maximum number of iterations is reached, it is considered to have converged and the registration process ends. At this time, the initial registered point cloud obtained by the last update is used as the target registered point cloud , that is, it is considered to be close enough to the original ship point cloud P, achieving a good registration effect.

[0109] S3. Use Poisson disk sampling on the target registered point cloud to obtain a uniformly distributed registered point cloud;

[0110] In the target registered point cloud, the density is uneven. Therefore, it is necessary to perform uniform sampling to ensure the uniformity of the point cloud distribution. Here, we use Poisson disk sampling, and its principle is as follows:

[0111] Poisson disk sampling is a method for generating a uniformly distributed point cloud. Its main purpose is to generate multiple points in a given sampling area and ensure that the distance between these points meets a certain minimum value. This method combines random sampling and distance constraints to avoid over - aggregation of points, thus achieving a high - quality scattered point cloud.

[0112] In Poisson disk sampling, first set a minimum distance r, which is used to ensure the distance between each newly generated point and the existing points. The specific sampling process is as follows:

[0113] (1)Random initialization: Randomly select a point from the target registration point cloud and add it to the sampling set S and the active list A.

[0114] (2)Iterative sampling:

[0115] When the active list A is not empty, repeat the following operations:

[0116] Randomly select a point from the active list .

[0117] Randomly generate k candidate points (for example: 30) within a region centered at with a radius range of r; The generation of candidate points can be expressed as:

[0118] For each candidate point :

[0119] Traverse the sampling set S and calculate the minimum distance of (s ∈ S). If all distances satisfy , then add to S and A.

[0120] If no new points are successfully added to the k candidate points of , then remove from the active list A.

[0121] (3)Termination and output: When the active list A is empty, the sampling ends, and the output set S is the uniformly distributed registration point cloud.

[0122] Through step S3, after Poisson disk sampling, the target registration point cloud can be transformed into a uniformly distributed registration point cloud. This process includes random initialization, generation of candidate points, distance verification, update of the active list, and iteration termination, ensuring that the sampled point cloud not only maintains the overall structural information but also eliminates the problem of uneven local density, providing high-quality input data for subsequent point cloud refinement, reconstruction, or other processing steps.

[0123] S4. Divide the uniformly distributed registration point cloud into a high-precision point cloud and a low-precision point cloud according to the chamfer distance;

[0124] In an alternative embodiment, Figure 6 is the flowchart for dividing the uniformly distributed registration point cloud provided by the embodiments of the present invention; As Figure 6 shown, the S4 includes:

[0125] S41. For each point in the uniformly distributed registration point cloud, calculate its nearest neighbor distance to the original ship point cloud;​

[0126] Specifically, its formula is:

[0127]

[0128] Wherein, is the i-th point in the uniformly distributed registered point cloud, is the i-th point in the original ship point cloud, is the nearest neighbor distance from the i-th point in the uniformly distributed registered point cloud to the original ship point cloud.

[0129] S42. Randomly divide the uniformly distributed registered point cloud into two point sets, and calculate the chamfer distance between the two point sets;

[0130] Specifically, randomly divide the uniformly distributed registered point cloud into two point sets and , each point set contains half of the points in the uniformly distributed registered point cloud data, and calculate the chamfer distance between these two point sets and . For each point in , find its nearest neighbor point in to obtain ; for each point in , find its nearest neighbor point in to obtain .

[0131] The specific formula is:

[0132]

[0133]

[0134]

[0135] Wherein, is the chamfer distance between the two point sets and .

[0136] S43. Divide the uniformly distributed registered point cloud into a high-precision point cloud and a low-precision point cloud according to the nearest neighbor distance and chamfer distance of each point in the uniformly distributed registered point cloud.

[0137] In an optional embodiment, for each point in the uniformly distributed registered point cloud, check its nearest neighbor distance to the original ship point cloud (obtained by S41):

[0138] If , then Incorporate into the high-precision point cloud set ;

[0139] If , then incorporate into the low-precision point cloud set .

[0140] S5. Extract features from the two-dimensional ship image, high-precision point cloud, and low-precision point cloud, and perform feature fusion to obtain the fused features;

[0141] Figure 7 is the flowchart of feature fusion provided by the embodiment of the present invention. As Figure 7 shown, the S5 includes:

[0142] S51. Use a lightweight convolutional neural network to extract semantic features from the two-dimensional ship image and map them to the point cloud resolution to obtain per-point image features;

[0143] Specifically, use the lightweight convolutional neural network MobileNetV2 to extract semantic features from the two-dimensional ship image , and map the extracted semantic features to the point cloud resolution through bilinear interpolation to obtain per-point image features .

[0144] S52. Use the FPFH descriptor to extract local geometric features from the high-precision point cloud and the low-precision point cloud to obtain point cloud features;

[0145] For the high-precision point cloud and the low-precision point cloud extract local geometric features respectively: calculate the FPFH (Fast Point Feature Histogram) descriptor for each point, capture local information such as normal vectors and curvatures, and output geometric features .

[0146] S53. Directly splice the per-point image features and the point cloud features along the channel dimension, and simplify the feature dimension to obtain the fused features.

[0147] Directly splice the per-point image features and the point cloud features along the channel dimension:

[0148]

[0149] Simplify the feature dimension through two layers of MLP (256→128): .

[0150] S6. Optimize and complement the high-precision point cloud and the low-precision point cloud respectively according to the fused features, and merge them to obtain the ship-complemented point cloud.

[0151] Figure 8It is the flowchart of point cloud completion provided by the embodiments of the present invention; as Figure 8 shown, the S6 includes:

[0152] S61. Calculate the average distance between points of the uniformly distributed registered point cloud, and set the constraints for the high-precision point cloud and the constraints for the low-precision point cloud according to the average distance between points;

[0153] For the uniformly distributed registered point cloud containing N points, its average distance between points is calculated according to the following formula:

[0154]

[0155] where is the Euclidean distance from point to its nearest neighbor point.

[0156] The constraint for the high-precision point cloud is 0.5 times the average distance between points, and the constraint for the low-precision point cloud is 2 times the average distance between points.

[0157] S62. Use the fusion features to predict the high-precision offset for the high-precision point cloud; use the fusion features and predict the low-precision offset through a 3-layer MLP for the low-precision point cloud;

[0158] Specifically, for the high-precision point cloud, the dual-branch network uses the fusion features to accurately predict the tiny offset (i.e., the high-precision offset Δhigh∈R N high ×3 ), to ensure that the details are retained. For the low-precision point cloud, the fusion features are used and processed through a three-layer MLP (128→64→3) to predict an offset allowing a larger range (i.e., the low-precision offset Δlow∈R N low ×3 ). The application of the fusion features in this process enables the model to identify and adjust the key points in the low-precision area, fill in the missing geometric structures, and thus improve the coherence of the overall point cloud.

[0159] S63. Update the high-precision point cloud according to the high-precision offset and the constraints of the high-precision point cloud; update the low-precision point cloud according to the low-precision offset and the constraints of the low-precision point cloud;

[0160] According to the high-precision offset Δ high ∈R N high ×3 and the constraint of the high-precision point cloud ||Δ high ||2≤0.5 update the high-precision point cloud; according to the low-precision offset Δ low ∈R N low×3 and the constraint of the low-precision point cloud ||Δ low ||2 ≤ 2 Update the low-precision point cloud.

[0161] S64. Merge the updated high-precision point cloud and the updated low-precision point cloud to obtain the ship-completed point cloud.

[0162] Figure 9 is a schematic structural diagram of a ship point cloud completion system based on multi-modal data fusion provided by an embodiment of the present invention. As Figure 9 shown, the system includes:

[0163] A point cloud reconstruction unit 201, configured to map a two-dimensional ship image into a reconstructed point cloud with a complete global structure through an encoding-conversion-decoding network architecture;

[0164] Figure 10 is a schematic structural diagram of the point cloud reconstruction unit provided by an embodiment of the present invention; as Figure 10 shown, the point cloud reconstruction unit 201 includes:

[0165] An encoding subunit 2011, configured to extract multi-scale features from a two-dimensional ship image through multi-layer convolution operations during the encoding stage;

[0166] A conversion subunit 2012, configured to convert the extracted multi-scale features into a three-dimensional initial point cloud through a fully connected layer or a reshaping operation during the conversion stage;

[0167] A decoding subunit 2013, configured to output the reconstructed point cloud with a complete global structure from the three-dimensional initial point cloud through multi-layer transposed convolution operations during the decoding stage.

[0168] A point cloud registration unit 202, configured to perform coarse registration and fine registration on the original ship point cloud and the reconstructed point cloud to obtain a target registered point cloud;

[0169] Figure 11 is a schematic structural diagram of the point cloud registration unit provided by an embodiment of the present invention; as Figure 11 shown, the point cloud registration unit 202 includes:

[0170] A point cloud coarse registration subunit 2021, configured to perform coarse registration on the original ship point cloud and the reconstructed point cloud to obtain an initial registered point cloud;

[0171] A point cloud fine registration subunit 2022, configured to perform fine registration on the original ship point cloud and the initial registered point cloud to obtain a target registered point cloud.

[0172] Figure 12 is a schematic structural diagram of the point cloud coarse registration subunit provided by an embodiment of the present invention; as Figure 12As shown, the rough point cloud registration subunit 2021 includes:

[0173] An extraction module 20211, configured to extract key feature points from the original ship point cloud and the reconstructed point cloud, and calculate descriptors for each feature point;

[0174] A screening module 20212, configured to screen out a set of matching feature point pairs according to the similarity of the descriptors of all feature points of the two point clouds;

[0175] A first estimation module 20213, configured to estimate rigid transformation parameters according to the screened set of matching feature point pairs;

[0176] A transformation module 20214, configured to transform the reconstructed point cloud according to the estimated rigid transformation parameters to obtain the initial registered point cloud.

[0177] Figure 13 is a schematic structural diagram of the fine point cloud registration subunit provided by an embodiment of the present invention; as Figure 13 shown, the fine point cloud registration subunit 2022 includes:

[0178] A search and construction module 20221, configured to search for the nearest point in the initial registered point cloud for each point in the original ship point cloud to form a current point pair set;

[0179] A second estimation module 20222, configured to re-estimate the rigid transformation parameters according to the current point pair set;

[0180] An update calculation module 20223, configured to update the initial registered point cloud according to the re-estimated rigid transformation parameters; and calculate the error between the updated initial registered point cloud and the original ship point cloud;

[0181] A repeated iteration module 20224, configured to repeat the search and construction module, the second estimation module, and the update calculation module until the error is lower than a preset threshold or the maximum number of iterations is reached, stop the update, and use the finally updated initial registered point cloud as the target registered point cloud.

[0182] A point cloud optimization unit 203, configured to obtain a uniformly distributed registered point cloud by performing Poisson disk sampling on the target registered point cloud;

[0183] A point cloud division unit 204, configured to divide the uniformly distributed registered point cloud into a high-precision point cloud and a low-precision point cloud according to the chamfer distance;

[0184] Figure 14 is a schematic structural diagram of the point cloud division unit provided by an embodiment of the present invention; as Figure 14 shown, the point cloud division unit 204 includes:

[0185] The first calculation subunit 2041 is configured to calculate the nearest neighbor distance from each point in the uniformly distributed registered point cloud to the original ship point cloud;

[0186] The second calculation subunit 2042 is configured to randomly divide the uniformly distributed registered point cloud into two point sets and calculate the chamfer distance between the two point sets;

[0187] The division subunit 2043 is configured to divide the uniformly distributed registered point cloud into a high-precision point cloud and a low-precision point cloud according to the nearest neighbor distance and the chamfer distance of each point in the uniformly distributed registered point cloud.

[0188] The feature fusion unit 205 is configured to extract features from the two-dimensional ship image, the high-precision point cloud, and the low-precision point cloud, and perform feature fusion to obtain a fused feature;

[0189] Figure 15 It is a schematic structural diagram of the feature fusion unit provided by an embodiment of the present invention; as Figure 15 shown, the feature fusion unit 205 includes:

[0190] The image feature extraction subunit 2051 is configured to use a lightweight convolutional neural network to extract semantic features from the two-dimensional ship image and map them to the point cloud resolution to obtain point-by-point image features;

[0191] The point cloud feature extraction subunit 2052 is configured to use the FPFH descriptor to extract local geometric features from the high-precision point cloud and the low-precision point cloud to obtain point cloud features;

[0192] The feature fusion subunit 2053 is configured to directly splice the point-by-point image features and the point cloud features along the channel dimension and simplify the feature dimension to obtain the fused feature.

[0193] The point cloud completion unit 206 is configured to optimize and complete the high-precision point cloud and the low-precision point cloud respectively according to the fused feature, and merge them to obtain a ship-completed point cloud.

[0194] Figure 16 It is a schematic structural diagram of the point cloud completion unit provided by an embodiment of the present invention; as Figure 16 shown, the point cloud completion unit 206 includes:

[0195] The constraint calculation subunit 2061 is configured to calculate the average distance between points in the uniformly distributed registered point cloud, and set constraints for the high-precision point cloud and constraints for the low-precision point cloud according to the average distance between points;

[0196] The offset prediction subunit 2062 is configured to predict a high-precision offset for the high-precision point cloud using the fused feature; predict a low-precision offset for the low-precision point cloud using the fused feature and through a 3-layer MLP;

[0197] A point cloud update subunit 2063 is configured to update the high-precision point cloud according to the high-precision offset and the constraints of the high-precision point cloud; and update the low-precision point cloud according to the low-precision offset and the constraints of the low-precision point cloud.

[0198] A merging subunit 2064 is configured to merge the updated high-precision point cloud and the updated low-precision point cloud to obtain the ship-completed point cloud.

[0199] The system of the present application corresponds to the above method, and the specific implementation manners of the system will not be repeated here.

[0200] Advantages of the present invention:

[0201] The present invention provides a ship point cloud completion method and system based on multi-modal data fusion. Among them, the method effectively makes up for the deficiencies of a single modality in structural inference and detail restoration through the cross-modal fusion of two-dimensional ship images and original ship point cloud data; uses a strategy combining coarse registration and fine registration to achieve precise alignment in space between the reconstructed point cloud and the original ship point cloud, ensuring data consistency; adopts Poisson disk sampling technology to solve the problem of uneven point cloud density and improve the uniformity of point cloud distribution; divides the point cloud into high-precision and low-precision regions through chamfer distance adaptive partitioning, realizing differential processing of local details and global structures; multi-modal feature fusion and offset prediction and movement based on the fused features achieve effective completion of the high-precision and low-precision region point cloud data, significantly improving the integrity and accuracy of the ship point cloud data.

[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A ship point cloud completion method based on multi-modal data fusion, characterized in that Including: S1. Map a two-dimensional ship image into a reconstructed point cloud with a complete global structure through an encoding-conversion-decoding network architecture; S2. Coarsely register and finely register the original ship point cloud with the reconstructed point cloud to obtain a target registered point cloud; S3. Use Poisson disk sampling on the target registered point cloud to obtain a uniformly distributed registered point cloud; S4. Divide the uniformly distributed registered point cloud into a high-precision point cloud and a low-precision point cloud according to the chamfer distance; S5. Extract features from the two-dimensional ship image, the high-precision point cloud, and the low-precision point cloud, and perform feature fusion to obtain a fused feature; S6. Optimize and complement the high-precision point cloud and the low-precision point cloud respectively according to the fused feature, and merge them to obtain a ship-complemented point cloud.

2. The method according to claim 1, characterized in that, The S1 includes: S11. Extract multi-scale features from the two-dimensional ship image through multi-layer convolution operations during the encoding stage; S12. Convert the extracted multi-scale features into a three-dimensional initial point cloud through a fully connected layer or a reshaping operation during the conversion stage; S13. Output the reconstructed point cloud with a complete global structure from the three-dimensional initial point cloud through multi-layer transposed convolution operations during the decoding stage.

3. The method according to claim 1, wherein The S2 includes: S21. Coarsely register the original ship point cloud with the reconstructed point cloud to obtain an initial registered point cloud; S22. Finely register the original ship point cloud with the initial registered point cloud to obtain a target registered point cloud.

4. The method according to claim 3, wherein The S21 includes: S211. Extract key feature points from the original ship point cloud and the reconstructed point cloud, and calculate descriptors for each feature point; S212. Screen out a set of matching feature point pairs according to the similarity of the descriptors of all feature points in the two point clouds; S213. Estimate the rigid transformation parameters according to the screened set of matching feature point pairs; S214. Transform the reconstructed point cloud according to the estimated rigid transformation parameters to obtain the initial registered point cloud.

5. The method according to claim 4, wherein The S22 includes: S221. Search for the nearest point in the initial registered point cloud for each point in the original ship point cloud to form a current point pair set; S222. Re-estimate the rigid transformation parameters according to the current point pair set; S223. Update the initial registered point cloud according to the re-estimated rigid transformation parameters; and calculate the error between the updated initial registered point cloud and the original ship point cloud; S224. Repeat S221~S223 until the error is lower than the preset threshold or the maximum number of iterations is reached, stop the update, and use the last updated initial registered point cloud as the target registered point cloud.

6. The method according to claim 1, wherein The S4 includes: S41. Calculate the nearest neighbor distance of each point in the uniformly distributed registered point cloud to the original ship point cloud; S42. Randomly divide the uniformly distributed registered point cloud into two point sets, and calculate the chamfer distance between the two point sets; S43. Divide the uniformly distributed registered point cloud into a high-precision point cloud and a low-precision point cloud according to the nearest neighbor distance and the chamfer distance of each point in the uniformly distributed registered point cloud.

7. The method according to claim 1, characterized in that, The S5 includes: S51. Use a lightweight convolutional neural network to extract semantic features from the two-dimensional ship image and map them to the point cloud resolution to obtain point-by-point image features; S52. Extract local geometric features from the high-precision point cloud and the low-precision point cloud using the FPFH descriptor to obtain point cloud features; S53. Directly splice the per-point image features and the point cloud features along the channel dimension, and simplify the feature dimension to obtain the fusion features.

8. The method according to claim 1, characterized in that, The said S6 includes: S61. Calculate the average inter-point distance of the uniformly distributed registered point cloud, and set the constraints for the high-precision point cloud and the constraints for the low-precision point cloud according to the average inter-point distance; S62. Use the fusion features to predict the high-precision offset for the high-precision point cloud; use the fusion features and a 3-layer MLP to predict the low-precision offset for the low-precision point cloud; S63. Update the high-precision point cloud according to the high-precision offset and the constraints of the high-precision point cloud; update the low-precision point cloud according to the low-precision offset and the constraints of the low-precision point cloud; S64. Merge the updated high-precision point cloud and the updated low-precision point cloud to obtain the ship-completed point cloud.

9. A ship point cloud completion system based on multi-modal data fusion, characterized in that, Include: A point cloud reconstruction unit, which is used to map a two-dimensional ship image into a reconstructed point cloud with a complete global structure through an encoding-conversion-decoding network architecture; A point cloud registration unit, which is used to perform rough registration and fine registration on the original ship point cloud and the reconstructed point cloud to obtain a target registered point cloud; A point cloud optimization unit, which is used to obtain a uniformly distributed registered point cloud by performing Poisson disk sampling on the target registered point cloud; A point cloud division unit, which is used to divide the uniformly distributed registered point cloud into a high-precision point cloud and a low-precision point cloud according to the chamfer distance; A feature fusion unit, which is used to extract features from the two-dimensional ship image, the high-precision point cloud and the low-precision point cloud, and perform feature fusion to obtain fusion features; A point cloud completion unit, which is used to optimize and complete the high-precision point cloud and the low-precision point cloud respectively according to the fusion features, and perform merging to obtain the ship-completed point cloud.

10. The system according to claim 9, characterized in that, The said point cloud reconstruction unit includes: An encoding sub-unit, which is used to extract multi-scale features from the two-dimensional ship image through multi-layer convolution operations during the encoding stage; A conversion sub-unit, which is used to convert the extracted multi-scale features into a three-dimensional initial point cloud through a fully connected layer or a reshaping operation during the conversion stage; A decoding sub-unit, which is used to output the reconstructed point cloud with a complete global structure from the three-dimensional initial point cloud through multi-layer transposed convolution operations during the decoding stage.

Images