A ship point cloud completion method and system based on multimodal data fusion

Abstract

The present invention discloses a method and system for completing a ship point cloud based on multimodal data fusion. The method comprises: mapping a two-dimensional ship image into a globally complete reconstructed point cloud through an encoding-conversion-decoding network architecture; performing coarse and fine registration of the original ship point cloud and the reconstructed point cloud to obtain a target registration point cloud; sampling the target registration point cloud using Poisson disks to obtain a uniformly distributed registration point cloud; dividing the uniformly distributed registration point cloud into a high-precision point cloud and a low-precision point cloud based on the chamfer distance; extracting features from the two-dimensional ship image, the high-precision point cloud, and the low-precision point cloud and performing feature fusion to obtain fused features; optimizing and completing the high-precision point cloud and the low-precision point cloud based on the fused features, and merging them to obtain a completed ship point cloud. This method utilizes multimodal data fusion and offset prediction and movement based on the fused features to achieve ship point cloud completion, significantly improving the integrity and accuracy of the ship point cloud data.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional point cloud processing, and in particular to a ship point cloud completion method and system based on multimodal data fusion. Background Art

[0002] Three-dimensional point cloud data has important application value in the marine field. However, the actual collected ship point clouds are often incomplete due to occlusion, noise, and uneven density, which affects subsequent modeling and analysis.

[0003] Existing technologies have many defects in the restoration and refinement of three-dimensional point clouds. First, the problem of insufficient information is particularly prominent. A single modality (relying only on point cloud data) cannot effectively restore a large range of missing structures, resulting in obvious deficiencies in the overall shape and detail performance of the generated point cloud. Secondly, the difficulty of cross-modal fusion is also a problem that needs to be solved urgently. The lack of effective methods for feature alignment and joint optimization of images and point clouds makes it difficult for data from different modalities to fully complement each other, thus affecting the accuracy and completeness of point cloud restoration. In addition, the current global refinement strategy often cannot retain the details of local high-precision areas when processing point clouds, resulting in loss of details, affecting the quality of the final model and the application effect.

[0004] Regarding the limitations of existing technologies, while geometric-based methods (such as Laplace smoothing) can fill small holes to a certain extent, their applicability is limited and they cannot address the task of restoring large missing areas. Alignment-based methods (such as 3D template matching) face high computational costs and high sensitivity to noise, limiting their widespread adoption in practical applications. Furthermore, single-modal deep learning models (such as PCN and TopNet) experience significant performance degradation when input data is severely missing, making them difficult to meet the requirements for high-quality point cloud generation.

[0005] Therefore, a multimodal data fusion method is urgently needed to improve the overall accuracy and robustness of ship point cloud completion. Summary of the Invention

[0006] In an embodiment of the present invention, a method and system for ship point cloud completion based on multimodal data fusion are provided to solve the problems in the prior art of ship point cloud completion using geometric filling, template matching and single-modal deep learning models, which have weak global structure inference capabilities and low cross-modal fusion efficiency.

[0007] To achieve the above-mentioned purpose, on the one hand, the present invention provides a ship point cloud completion method based on multimodal data fusion, which includes: S1, mapping the two-dimensional ship image into a reconstructed point cloud with complete global structure through an encoding-conversion-decoding network architecture; S2, coarsely aligning and finely aligning the original ship point cloud with the reconstructed point cloud to obtain a target registration point cloud; S3, sampling the target registration point cloud using Poisson disks to obtain a uniformly distributed registration point cloud; S4, dividing the uniformly distributed registration point cloud into a high-precision point cloud and a low-precision point cloud according to the chamfer distance; S5, extracting features from the two-dimensional ship image, the high-precision point cloud and the low-precision point cloud, and performing feature fusion to obtain a fusion feature; S6, optimizing and completing the high-precision point cloud and the low-precision point cloud according to the fusion feature, and merging them to obtain a ship completion point cloud.

[0008] Optionally, the S1 includes: S11, extracting multi-scale features from the two-dimensional ship image through multi-layer convolution operations in the encoding stage; S12, converting the extracted multi-scale features into a three-dimensional initial point cloud through a fully connected layer or a reshaping operation in the conversion stage; S13, outputting the reconstructed point cloud with complete global structure through multi-layer deconvolution operations on the three-dimensional initial point cloud in the decoding stage.

[0009] Optionally, the S2 includes: S21, coarsely registering the original ship point cloud with the reconstructed point cloud to obtain an initial registered point cloud; S22, finely registering the original ship point cloud with the initial registered point cloud to obtain a target registered point cloud.

[0010] Optionally, the S21 includes: S211, extracting key feature points from the original ship point cloud and the reconstructed point cloud, and calculating a descriptor for each feature point; S212, screening out a set of matching feature point pairs based on the similarity of the descriptors of all feature points of the two point clouds; S213, estimating rigid transformation parameters based on the screened set of matching feature point pairs; S214, transforming the reconstructed point cloud based on the estimated rigid transformation parameters to obtain the initial registration point cloud.

[0011] Optionally, the S22 includes: S221, for each point in the original ship point cloud, searching for the nearest point in the initial registration point cloud to form a current point pair set; S222, based on the current point pair set, re-estimating the rigid transformation parameters; S223, updating the initial registration point cloud based on the re-estimated rigid transformation parameters; and calculating the error between the updated initial registration point cloud and the original ship point cloud; S224, repeating S221 to S223 until the error is lower than a preset threshold or reaches the maximum number of iterations, stopping the update, and using the last updated initial registration point cloud as the target registration point cloud.

[0012] Optionally, the S4 includes: S41. For each point in the uniformly distributed registration point cloud, calculating its nearest neighbor distance to the original ship point cloud; S42. Randomly dividing the uniformly distributed registration point cloud into two point sets, and calculating the chamfer distance of the two point sets; S43. According to the nearest neighbor distance and chamfer distance of each point in the uniformly distributed registration point cloud, dividing the uniformly distributed registration point cloud into a high-precision point cloud and a low-precision point cloud.

[0013] Optionally, the S5 includes: S51, using a lightweight convolutional neural network to extract semantic features from the two-dimensional ship image and mapping them to the point cloud resolution to obtain point-by-point image features; S52, using the FPFH descriptor to extract local geometric features from the high-precision point cloud and the low-precision point cloud to obtain point cloud features; S53, directly splicing the point-by-point image features and the point cloud features along the channel dimension, and simplifying the feature dimension to obtain the fusion feature.

[0014] Optionally, the S6 includes: S61, calculating the average distance between points of the uniformly distributed registration point cloud, and setting constraints for the high-precision point cloud and the low-precision point cloud based on the average distance between points; S62, using fusion features to predict high-precision offsets for the high-precision point cloud; using fusion features to predict low-precision offsets for the low-precision point cloud through a 3-layer MLP; S63, updating the high-precision point cloud based on the high-precision offsets and the constraints of the high-precision point cloud; updating the low-precision point cloud based on the low-precision offsets and the constraints of the low-precision point cloud; S64, merging the updated high-precision point cloud and the updated low-precision point cloud to obtain the ship completion point cloud.

[0015] On the other hand, the present invention provides a ship point cloud completion system based on multimodal data fusion, which includes: a point cloud reconstruction unit, which is used to map the two-dimensional ship image into a reconstructed point cloud with complete global structure through an encoding-conversion-decoding network architecture; a point cloud registration unit, which is used to perform coarse registration and fine registration on the original ship point cloud and the reconstructed point cloud to obtain a target registration point cloud; a point cloud optimization unit, which is used to obtain a uniformly distributed registration point cloud using Poisson disk sampling on the target registration point cloud; a point cloud division unit, which is used to divide the uniformly distributed registration point cloud into a high-precision point cloud and a low-precision point cloud according to the chamfer distance; a feature fusion unit, which is used to extract features from the two-dimensional ship image, the high-precision point cloud and the low-precision point cloud, and perform feature fusion to obtain a fusion feature; a point cloud completion unit, which is used to optimize and complete the high-precision point cloud and the low-precision point cloud respectively according to the fusion feature, and merge them to obtain a ship completion point cloud.

[0016] Optionally, the point cloud reconstruction unit includes: an encoding subunit, used to extract multi-scale features from the two-dimensional ship image through multi-layer convolution operations in the encoding stage; a conversion subunit, used to convert the extracted multi-scale features into a three-dimensional initial point cloud through a fully connected layer or a reshaping operation in the conversion stage; and a decoding subunit, used to output the reconstructed point cloud with complete global structure through multi-layer deconvolution operations on the three-dimensional initial point cloud in the decoding stage.

[0017] Beneficial effects of the present invention:

[0018] The present invention provides a ship point cloud completion method and system based on multimodal data fusion, wherein the method effectively compensates for the deficiencies of a single modality in structural inference and detail recovery through cross-modal fusion of two-dimensional ship images and original ship point cloud data; utilizes a strategy combining coarse registration and fine registration to achieve precise spatial alignment of the reconstructed point cloud and the original ship point cloud, thereby ensuring data consistency; adopts Poisson disk sampling technology to solve the problem of uneven point cloud density and improve the uniformity of point cloud distribution; through adaptive partitioning of chamfer distance, the point cloud is divided into high-precision and low-precision areas, thereby achieving differentiated processing of local details and global structures; multimodal feature fusion and offset prediction and movement based on fused features achieve effective completion of point cloud data in high-precision and low-precision areas, thereby significantly improving the integrity and accuracy of ship point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of a ship point cloud completion method based on multimodal data fusion provided by an embodiment of the present invention;

[0020] Figure 2 This is a flowchart of mapping a two-dimensional ship image into a reconstructed point cloud provided by an embodiment of the present invention;

[0021] Figure 3 This is a flow chart of registering the original ship point cloud with the reconstructed point cloud provided by an embodiment of the present invention;

[0022] Figure 4 This is a flow chart of roughly registering the original ship point cloud with the reconstructed point cloud provided by an embodiment of the present invention;

[0023] Figure 5 This is a flow chart of finely registering the original ship point cloud with the initial registration point cloud provided by an embodiment of the present invention;

[0024] Figure 6 This is a flow chart of dividing a uniformly distributed registration point cloud provided by an embodiment of the present invention;

[0025] Figure 7 This is a flow chart of feature fusion provided by an embodiment of the present invention;

[0026] Figure 8 This is a flowchart of point cloud completion provided by an embodiment of the present invention;

[0027] Figure 9 1 is a structural diagram of a ship point cloud completion system based on multimodal data fusion provided by an embodiment of the present invention;

[0028] Figure 10 is a structural diagram of a point cloud reconstruction unit provided by an embodiment of the present invention;

[0029] Figure 11 is a structural diagram of a point cloud registration unit provided by an embodiment of the present invention;

[0030] Figure 12 is a structural diagram of a point cloud coarse registration subunit provided by an embodiment of the present invention;

[0031] Figure 13 is a structural diagram of a point cloud precise registration subunit provided by an embodiment of the present invention;

[0032] Figure 14 is a structural diagram of a point cloud segmentation unit provided in an embodiment of the present invention;

[0033] Figure 15 is a structural diagram of a feature fusion unit provided in an embodiment of the present invention;

[0034] Figure 16 4 is a schematic structural diagram of a point cloud completion unit provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0036] Figure 1 This is a flow chart of a ship point cloud completion method based on multimodal data fusion provided by an embodiment of the present invention. Figure 1 As shown, the method includes:

[0037] S1, through the encoding-transformation-decoding network architecture, the 2D ship image is mapped into a reconstructed point cloud with complete global structure;

[0038] Figure 2 FIG. 1 is a flowchart of mapping a two-dimensional ship image into a reconstructed point cloud according to an embodiment of the present invention; FIG. Figure 2 As shown, the S1 includes:

[0039] S11, in the encoding stage, extracting multi-scale features from the two-dimensional ship image through multi-layer convolution operations;

[0040] In one optional embodiment, during the encoding stage, the two-dimensional ship image (single-view image) undergoes preprocessing (such as normalization, denoising, and contrast enhancement) before being fed into the encoder. The encoder employs a seven-layer convolutional architecture to progressively compress spatial dimensions and extract multi-scale features. The first convolutional layer uses a 3×3 kernel with 16 channels to capture low-level image features such as edges and textures. Subsequently, the kernel size and number of channels are gradually increased (e.g., from 3×3 to 5×5, and from 32 and 64 to 128 and 256 channels) to extract multi-scale and multi-layered feature information. After each convolutional layer, a nonlinear activation function (such as ReLU) is applied to enhance the model's expressive power. Meanwhile, the feature map size is progressively reduced through step size adjustment to form a high-dimensional, low-resolution latent feature representation. This process compresses the input image from its original RGB pixel space into high-dimensional features that contain both geometric and semantic information about the object.

[0041] For example:

[0042] Layer 1: 3×3 convolution, 16 output channels, stride 2, activation function ReLU;

[0043] Layers 2-4: 3×3 convolutions, with 32, 64, and 128 channels, stride 2, and ReLU activation function.

[0044] Layers 5-7: 5×5 convolution, 256 channels, stride 1, ReLU activation function; output feature map size 5×5×256.

[0045] S12, in the conversion stage, the extracted multi-scale features are converted into a three-dimensional initial point cloud through a fully connected layer or a reshaping operation;

[0046] In an optional embodiment, during the conversion phase, the high-dimensional features (e.g., 5×5, 256) output by the encoder are converted into a 3D initial point cloud (7×7×512) via a fully connected layer or reshape operation. This step essentially performs modal alignment between the 2D ship image features and the 3D point cloud space, mapping the compressed features to the pre-set 3D point cloud dimensions via a linear transformation. The 7×7×512 dimensions here generate 512 3D coordinates, each distributed in a 7×7 grid, providing the underlying structure for subsequent refinement.

[0047] S13. In the decoding stage, the three-dimensional initial point cloud is subjected to multi-layer deconvolution operations to output the reconstructed point cloud with a complete global structure.

[0048] In one optional embodiment, during the decoding phase, the initial 3D point cloud is processed through three layers of deconvolution (e.g., 5×5, 256 → 5×5, 128 → 3×3, 64) to gradually restore spatial resolution and refine point cloud details. The deconvolution layer expands the feature map size through upsampling and, combined with the skip connection information from the encoding phase, effectively compensates for details that may have been lost during the deconvolution process. Finally, the network compresses the number of channels to 3 through a 3×3, 3 convolution layer, directly outputting the X, Y, and Z coordinates of each point, thereby forming a reconstructed point cloud with a complete global structure. Throughout this process, the network employs an end-to-end training strategy, using chamfer distance as the primary loss function, to ensure that the reconstructed point cloud is highly consistent with the ground truth in both global structure and local details.

[0049] The advantage of this method is that it directly maps images to point clouds, avoiding the multi-view dependency or intermediate depth estimation steps required in traditional methods. However, its limitations include incomplete reconstruction of occluded areas due to single-view input, and point cloud density is limited by the preset output dimensions. By adjusting the number of deconvolution layers and channels, a balance can be achieved between computational efficiency and reconstruction accuracy.

[0050] S2. performing coarse registration and fine registration on the original ship point cloud and the reconstructed point cloud to obtain a target registered point cloud;

[0051] Figure 3 FIG. 1 is a flow chart of registering the original ship point cloud with the reconstructed point cloud provided by an embodiment of the present invention; Figure 3 As shown, the S2 includes:

[0052] S21, coarsely registering the original ship point cloud with the reconstructed point cloud to obtain an initial registered point cloud;

[0053] Figure 4 FIG. 1 is a flow chart of roughly registering the original ship point cloud with the reconstructed point cloud provided by an embodiment of the present invention; Figure 4 As shown, the S21 includes:

[0054] S211, extracting key feature points from the original ship point cloud and the reconstructed point cloud, and calculating a descriptor for each feature point;

[0055] From the original ship point cloud and reconstructed point cloud The key feature points with representative and rich local geometric information are extracted from the point cloud. Usually, these feature points are located at the edge, corner or the area with significant local curvature change of the point cloud, which can fully reflect the geometric structure of the object.

[0056] A high-dimensional descriptor is generated for each extracted feature point to capture the local geometric information around it.

[0057] For example, using the FPFH (Fast Point Feature Histograms) descriptor:

[0058] Neighborhood determination: taking each feature point as the center, collect neighborhood points within a certain radius;

[0059] Normal vector calculation: Calculate the normal vector of each point in the neighborhood to provide a basis for subsequent description construction;

[0060] Histogram construction: Count the angles, distances, and other information between points in the neighborhood and the center point, and construct a histogram as a feature description;

[0061] Normalization: Normalize the histogram data to reduce the impact of scale differences;

[0062] Through this process, each feature point is accompanied by a descriptor vector that represents its local geometric structure.

[0063] S212, screening out a set of matching feature point pairs based on the similarity of the descriptors of all feature points of the two point clouds;

[0064] For all feature points in the original ship point cloud and the reconstructed point cloud, calculate the similarity between their respective descriptors. Euclidean distance or cosine similarity is commonly used to measure the similarity between two descriptors. Select feature point pairs with a similarity greater than 0.8.

[0065] For example, if the similarity between the descriptor of the current feature point in the original ship point cloud and the descriptor of one of the feature points in the reconstructed point cloud is higher than 0.8, the two feature points are taken as a matching feature point pair. The similarity of the descriptors of all feature points of the two point clouds is calculated using the above method to filter out the set of matching feature point pairs.

[0066] S213, estimating rigid transformation parameters based on the screened matching feature point pair set;

[0067] The rigid transformation parameters include: rotation matrix R and translation vector t; setting the matching point pair set to ,in is the feature point in the original ship point cloud, is the feature point matched in the corresponding reconstructed point cloud, and K is the number of matching feature point pairs.

[0068] According to the selected matching feature point pairs, a rotation matrix R and a translation vector t need to be found, that is, each point in Q Transformed into , so that it matches the corresponding As close as possible. Expressed in mathematical form:

[0069]

[0070] In order to measure the degree of alignment between point pairs, the square of the Euclidean distance is usually used as the error metric, thereby forming the least squares optimization objective:

[0071]

[0072] The physical meaning of the objective function is to find the point pairs that can make all matching feature points The rigid transformation (rotation + translation) that minimizes the sum of the distances.

[0073] SVD (singular value decomposition) is usually used to solve the optimal R and t. The solution process is:

[0074] (1) First, calculate the centroid of the matching feature point pairs of the original ship point cloud P and the reconstructed point cloud Q in their respective coordinate systems:

[0075]

[0076] Center of mass and They represent the average positions of the two groups of point clouds in space.

[0077] (2) Center all matching feature point pairs relative to their respective centroids to obtain new point coordinates:

[0078]

[0079] (3) Define the covariance matrix H:

[0080]

[0081] in and are all 3×1 column vectors, so It is a 3×3 matrix, and the final 3×3 covariance matrix H is obtained by summing all matching feature point pairs.

[0082] Next, perform singular value decomposition (SVD) on H:

[0083]

[0084] Where U and V are 3×3 orthogonal matrices, and Σ is a diagonal singular value matrix (diagonal elements are non-negative real numbers and are sorted from large to small).

[0085] (4) According to the SVD results, the optimal rotation matrix can be obtained:

[0086]

[0087] If it is detected that the determinant det(R)<0, V or U is modified to ensure that R is an orthogonal matrix.

[0088] (5) Calculate the translation vector through the center of mass relationship:

[0089]

[0090] Recall the objective function to be minimized:

[0091]

[0092] Through the above steps, it can be proved that R and t obtained by SVD are the solutions that minimize the objective function.

[0093] S214 : transforming the reconstructed point cloud according to the estimated rigid transformation parameters to obtain the initial registration point cloud.

[0094] After obtaining R and t, all points in the reconstructed point cloud Q can be transformed to obtain the initial registration point cloud:

[0095]

[0096] Initial registration of point clouds It is preliminarily aligned with the original ship point cloud P, thus completing the coarse registration step.

[0097] S22: Finely register the original ship point cloud with the initial registration point cloud to obtain a target registration point cloud.

[0098] Figure 5 FIG. 1 is a flow chart of precisely registering the original ship point cloud with the initial registration point cloud provided by an embodiment of the present invention; Figure 5 As shown, the S22 includes:

[0099] S221. For each point in the original ship point cloud, search for the nearest point in the initial registration point cloud to form a current point pair set;

[0100] For each ,exist Search distance The nearest point , forming the current point pair set .

[0101] S222, re-estimating the rigid transformation parameters according to the current point pair set;

[0102] A method similar to S213 is used to re-estimate the rigid transformation parameters, including: the rotation matrix R and the translation vector t.

[0103] S223, updating the initial registration point cloud according to the re-estimated rigid transformation parameters; and calculating the error between the updated initial registration point cloud and the original ship point cloud;

[0104] The error is calculated according to the following formula:

[0105]

[0106] Where E is the error, is the original ship point cloud, is the updated initial registration point cloud, and N is the number of points in the original ship point cloud or the initial registration point cloud.

[0107] S224, repeat S221 to S223 until the error is lower than the preset threshold or the maximum number of iterations is reached, stop updating, and use the last updated initial registration point cloud as the target registration point cloud.

[0108] Determine whether to continue iterating based on the calculated error. If E≥ If the number of iterations does not reach the maximum number of iterations, the updated initial registration point cloud is used as the new initial registration point cloud, and the process returns to step S221 to continue nearest neighbor matching, re-estimate rigid transformation parameters, update the initial registration point cloud and error calculation; when the error E is lower than the preset threshold , or when the maximum number of iterations is reached, it is considered to have converged and the registration process ends. At this point, the initial registration point cloud obtained by the last update is used as the target registration point cloud , that is, it is considered to be close enough to the original ship point cloud P, achieving a good registration effect.

[0109] S3, the target registration point cloud is sampled using Poisson disk to obtain a uniformly distributed registration point cloud;

[0110] In the target registration point cloud, the density may be uneven. Therefore, it is necessary to sample it uniformly to ensure the uniformity of the point cloud distribution. Here we use Poisson Disk Sampling, the principle of which is as follows:

[0111] Poisson disk sampling is a method for generating uniformly distributed point clouds. Its main goal is to generate multiple points within a given sampling area while ensuring that the distance between these points meets a certain minimum value. This method avoids excessive point aggregation through a combination of random sampling and distance constraints, thereby achieving a high-quality dispersed point cloud.

[0112] In Poisson disk sampling, a minimum distance r is first set to ensure that each new point generated is close to the existing points. The specific sampling process is as follows:

[0113] (1) Random initialization: from the target registration point cloud Randomly select a point , add it to the sampling set S and active list A.

[0114] (2) Iterative sampling:

[0115] When the active list A is not empty, repeat the following operations:

[0116] Randomly select a point in the active list .

[0117] In As the center, k candidate points (for example, 30) are randomly generated in an area with a radius of r. The generation of candidate points can be expressed as:

[0118] For each candidate point :

[0119] Traverse the sampling set S and calculate (s∈S). If all distances satisfy , then Join S and A.

[0120] If If none of the k candidate points are successfully added, they are removed from the active list A. .

[0121] (3) Termination and output: When the active list A is empty, sampling ends and the output set S is the uniformly distributed registration point cloud.

[0122] Through the S3 step, the target registration point cloud can be converted into a uniformly distributed registration point cloud after Poisson disk sampling. This process includes random initialization, candidate point generation, distance verification, active list update, and iteration termination. It ensures that the sampled point cloud maintains overall structural information while eliminating local density unevenness, providing high-quality input data for subsequent point cloud refinement, reconstruction, or other processing steps.

[0123] S4, dividing the evenly distributed registration point cloud into high-precision point cloud and low-precision point cloud according to the chamfer distance;

[0124] In an optional embodiment, Figure 6 This is a flow chart of dividing the evenly distributed registration point cloud provided by an embodiment of the present invention; Figure 6 As shown, the S4 includes:

[0125] S41. For each point in the uniformly distributed registration point cloud , calculate its nearest neighbor distance to the original ship point cloud;

[0126] Specifically, the formula is:

[0127]

[0128] in, To uniformly distribute the i-th point in the registration point cloud, is the i-th point in the original ship point cloud, is the nearest neighbor distance between the i-th point in the uniformly distributed registration point cloud and the original ship point cloud.

[0129] S42, randomly dividing the uniformly distributed registration point cloud into two point sets, and calculating the chamfer distance between the two point sets;

[0130] Specifically, the uniformly distributed registration point cloud is randomly divided into two point sets and , each point set contains half of the points in the uniformly distributed registration point cloud data, and the two point sets are calculated and The chamfer distance between Each point in ,exist Find its nearest neighbor point in ;for Each point in ,exist Find its nearest neighbor point in .

[0131] The specific formula is:

[0132]

[0133]

[0134]

[0135] in, For two point sets and The chamfer distance between them.

[0136] S43. Divide the evenly distributed registration point cloud into a high-precision point cloud and a low-precision point cloud according to the nearest neighbor distance and the chamfer distance of each point in the evenly distributed registration point cloud.

[0137] In an optional embodiment, for each point in the uniformly distributed registration point cloud , check its nearest neighbor distance to the original ship point cloud (obtained by S41):

[0138] if , then Classify into high-precision point cloud collection ;

[0139] if , then Divide into low-precision point cloud collection .

[0140] S5, extracting features from the two-dimensional ship image, the high-precision point cloud, and the low-precision point cloud, and performing feature fusion to obtain fused features;

[0141] Figure 7 This is a flow chart of feature fusion provided by an embodiment of the present invention. Figure 7 As shown, the S5 includes:

[0142] S51. Extract semantic features from 2D ship images using a lightweight convolutional neural network and map them to point cloud resolution to obtain point-by-point image features.

[0143] Specifically, a lightweight convolutional neural network MobileNetV2 is used to extract semantic features from two-dimensional ship images. , the extracted semantic features are mapped to the point cloud resolution through bilinear interpolation to obtain point-by-point image features .

[0144] S52, extracting local geometric features from the high-precision point cloud and the low-precision point cloud using the FPFH descriptor to obtain point cloud features;

[0145] For high-precision point clouds and low-precision point clouds Extract local geometric features separately: calculate the FPFH (Fast Point Feature Histogram) descriptor of each point, capture local information such as normal vector and curvature, and output geometric features .

[0146] S53. Directly concatenate the point-by-point image features and the point cloud features along the channel dimension, and simplify the feature dimension to obtain the fused features.

[0147] Directly concatenate the point-by-point image features and point cloud features along the channel dimension:

[0148]

[0149] Reduce the feature dimension by two layers of MLP (256→128): .

[0150] S6. Optimize and complete the high-precision point cloud and the low-precision point cloud according to the fusion features, and merge them to obtain the completed point cloud of the ship.

[0151] Figure 8This is a flowchart of point cloud completion provided by an embodiment of the present invention; Figure 8 As shown, the S6 includes:

[0152] S61, calculating the average distance between points of the evenly distributed registration point cloud, and setting the constraints of the high-precision point cloud and the low-precision point cloud according to the average distance between the points;

[0153] For a uniformly distributed registration point cloud containing N points, the average distance between points is calculated according to the following formula:

[0154]

[0155] in, for point The Euclidean distance to its nearest neighbor.

[0156] The constraint for high-precision point clouds is 0.5 times the average distance between points, and the constraint for low-precision point clouds is 2 times the average distance between points.

[0157] S62: For high-precision point clouds, fusion features are used to predict high-precision offsets; for low-precision point clouds, fusion features are used to predict low-precision offsets through a 3-layer MLP.

[0158] Specifically, for high-precision point clouds, the dual-branch network uses fusion features to accurately predict small offsets (i.e., high-precision offsets Δhigh∈R N high ×3 ), to ensure that details are preserved. For low-precision point clouds, the fusion features are used and processed through three layers of MLP (128→64→3) to predict the offset that allows a wider range (i.e., the low-precision offset Δlow∈R N low ×3 The application of fusion features in this process enables the model to identify and adjust key points in low-precision areas, fill in missing geometric structures, and thus improve the coherence of the overall point cloud.

[0159] S63, updating the high-precision point cloud according to the high-precision offset and the constraints of the high-precision point cloud; updating the low-precision point cloud according to the low-precision offset and the constraints of the low-precision point cloud;

[0160] According to the high-precision offset Δ high ∈R N high ×3 And the constraints of high-precision point cloud ||Δ high ||2≤0.5 Update the high-precision point cloud; according to the low-precision offset Δ low ∈R N low×3 And the constraints of low-precision point cloud ||Δ low ||2≤2 Update low-precision point cloud.

[0161] S64: Merge the updated high-precision point cloud and the updated low-precision point cloud to obtain the ship completed point cloud.

[0162] Figure 9 is a structural diagram of a ship point cloud completion system based on multimodal data fusion provided by an embodiment of the present invention, such as Figure 9 As shown, the system includes:

[0163] The point cloud reconstruction unit 201 is used to map the two-dimensional ship image into a reconstructed point cloud with a complete global structure through an encoding-transforming-decoding network architecture;

[0164] Figure 10 is a schematic structural diagram of a point cloud reconstruction unit provided by an embodiment of the present invention; Figure 10 As shown, the point cloud reconstruction unit 201 includes:

[0165] The encoding subunit 2011 is used to extract multi-scale features from the two-dimensional ship image through multi-layer convolution operations during the encoding stage;

[0166] The conversion subunit 2012 is used to convert the extracted multi-scale features into a three-dimensional initial point cloud through a fully connected layer or a reshaping operation in the conversion stage;

[0167] The decoding subunit 2013 is used to perform multi-layer deconvolution operations on the 3D initial point cloud during the decoding phase to output the reconstructed point cloud with a complete global structure.

[0168] The point cloud registration unit 202 is used to perform coarse registration and fine registration on the original ship point cloud and the reconstructed point cloud to obtain a target registration point cloud;

[0169] Figure 11 is a schematic structural diagram of a point cloud registration unit provided by an embodiment of the present invention; Figure 11 As shown, the point cloud registration unit 202 includes:

[0170] The point cloud coarse registration subunit 2021 is used to coarsely register the original ship point cloud with the reconstructed point cloud to obtain an initial registered point cloud;

[0171] The point cloud fine registration subunit 2022 is configured to perform fine registration on the original ship point cloud and the initial registration point cloud to obtain a target registration point cloud.

[0172] Figure 12 is a schematic structural diagram of a point cloud coarse registration subunit provided in an embodiment of the present invention; Figure 12As shown, the point cloud coarse registration subunit 2021 includes:

[0173] An extraction module 20211 is used to extract key feature points from the original ship point cloud and the reconstructed point cloud, and calculate a descriptor for each feature point;

[0174] The screening module 20212 is used to screen out a set of matching feature point pairs based on the similarity of the descriptors of all feature points of the two point clouds;

[0175] A first estimation module 20213 is configured to estimate rigid transformation parameters based on the selected matching feature point pair set;

[0176] The transformation module 20214 is used to transform the reconstructed point cloud according to the estimated rigid transformation parameters to obtain the initial registration point cloud.

[0177] Figure 13 is a structural diagram of a point cloud precise registration subunit provided by an embodiment of the present invention; Figure 13 As shown, the point cloud fine registration subunit 2022 includes:

[0178] A search and construction module 20221 is configured to search for the nearest point in the initial registration point cloud for each point in the original ship point cloud to form a current point pair set;

[0179] A second estimation module 20222 is configured to re-estimate the rigid transformation parameters based on the current point pair set;

[0180] An update calculation module 20223 is configured to update the initial registration point cloud according to the re-estimated rigid transformation parameters; and calculate the error between the updated initial registration point cloud and the original ship point cloud;

[0181] The iterative module 20224 is used to repeatedly search the construction module, the second estimation module, and the update calculation module until the error is lower than a preset threshold or the maximum number of iterations is reached, stop updating, and use the last updated initial registration point cloud as the target registration point cloud.

[0182] The point cloud optimization unit 203 is used to sample the target registration point cloud using Poisson disk to obtain a uniformly distributed registration point cloud;

[0183] A point cloud division unit 204 is configured to divide the uniformly distributed registration point cloud into a high-precision point cloud and a low-precision point cloud according to the chamfer distance;

[0184] Figure 14 Schematic diagram of the structure of the point cloud segmentation unit provided by the embodiment of the present invention; Figure 14 As shown, the point cloud division unit 204 includes:

[0185] The first calculation subunit 2041 is configured to calculate, for each point in the uniformly distributed registration point cloud, its nearest neighbor distance to the original ship point cloud;

[0186] The second calculation subunit 2042 is used to randomly divide the uniformly distributed registration point cloud into two point sets and calculate the chamfer distance between the two point sets;

[0187] The division subunit 2043 is configured to divide the uniformly distributed registered point cloud into a high-precision point cloud and a low-precision point cloud according to the nearest neighbor distance and the chamfer distance of each point in the uniformly distributed registered point cloud.

[0188] The feature fusion unit 205 is used to extract features from the two-dimensional ship image, the high-precision point cloud and the low-precision point cloud, and perform feature fusion to obtain fused features;

[0189] Figure 15 : is a structural diagram of a feature fusion unit provided in an embodiment of the present invention; Figure 15 As shown, the feature fusion unit 205 includes:

[0190] The image feature extraction subunit 2051 is used to extract semantic features from the two-dimensional ship image using a lightweight convolutional neural network and map them to point cloud resolution to obtain point-by-point image features;

[0191] The point cloud feature extraction subunit 2052 is used to extract local geometric features of the high-precision point cloud and the low-precision point cloud using the FPFH descriptor to obtain point cloud features;

[0192] The feature fusion subunit 2053 is used to directly splice the point-by-point image features and the point cloud features along the channel dimension, and simplify the feature dimension to obtain the fused feature.

[0193] The point cloud completion unit 206 is used to optimize and complete the high-precision point cloud and the low-precision point cloud respectively according to the fusion features, and merge them to obtain the completed point cloud of the ship.

[0194] Figure 16 is a schematic structural diagram of a point cloud completion unit provided by an embodiment of the present invention; Figure 16 As shown, the point cloud completion unit 206 includes:

[0195] The constraint calculation subunit 2061 is used to calculate the average distance between points of the uniformly distributed registration point cloud, and set the constraints of the high-precision point cloud and the low-precision point cloud according to the average distance between points;

[0196] The offset prediction subunit 2062 is used to predict the high-precision offset of the high-precision point cloud using the fusion feature; and predict the low-precision offset of the low-precision point cloud using the fusion feature and a 3-layer MLP;

[0197] The point cloud updating subunit 2063 is configured to update the high-precision point cloud according to the high-precision offset and the constraints of the high-precision point cloud; and to update the low-precision point cloud according to the low-precision offset and the constraints of the low-precision point cloud;

[0198] The merging subunit 2064 is configured to merge the updated high-precision point cloud and the updated low-precision point cloud to obtain the completed point cloud of the ship.

[0199] The system of the present application corresponds to the above-mentioned method, and the specific implementation of the system will not be repeated here.

[0200] Beneficial effects of the present invention:

[0201] The present invention provides a ship point cloud completion method and system based on multimodal data fusion, wherein the method effectively compensates for the deficiencies of a single modality in structural inference and detail recovery through cross-modal fusion of two-dimensional ship images and original ship point cloud data; utilizes a strategy combining coarse registration and fine registration to achieve precise spatial alignment of the reconstructed point cloud and the original ship point cloud, thereby ensuring data consistency; adopts Poisson disk sampling technology to solve the problem of uneven point cloud density and improve the uniformity of point cloud distribution; through adaptive partitioning of chamfer distance, the point cloud is divided into high-precision and low-precision areas, thereby achieving differentiated processing of local details and global structures; multimodal feature fusion and offset prediction and movement based on fused features achieve effective completion of point cloud data in high-precision and low-precision areas, thereby significantly improving the integrity and accuracy of ship point cloud data.

[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A ship point cloud completion method based on multimodal data fusion, characterized in that: include: S1, through the encoding-transformation-decoding network architecture, the 2D ship image is mapped into a reconstructed point cloud with complete global structure; S2. performing coarse registration and fine registration on the original ship point cloud and the reconstructed point cloud to obtain a target registered point cloud; S3, the target registration point cloud is sampled using Poisson disk to obtain a uniformly distributed registration point cloud; S4, dividing the evenly distributed registration point cloud into high-precision point cloud and low-precision point cloud according to the chamfer distance; S5, extracting features from the two-dimensional ship image, the high-precision point cloud, and the low-precision point cloud, and performing feature fusion to obtain fused features; S6. Optimize and complete the high-precision point cloud and the low-precision point cloud according to the fusion features, and merge them to obtain the completed point cloud of the ship; The S4 includes: S41. For each point in the uniformly distributed registration point cloud, calculate its nearest neighbor distance to the original ship point cloud; S42, randomly dividing the uniformly distributed registration point cloud into two point sets, and calculating the chamfer distance between the two point sets; S43, dividing the evenly distributed registration point cloud into a high-precision point cloud and a low-precision point cloud according to the nearest neighbor distance and the chamfer distance of each point in the evenly distributed registration point cloud; The S5 includes: S51. Extract semantic features from 2D ship images using a lightweight convolutional neural network and map them to point cloud resolution to obtain point-by-point image features. S52, extracting local geometric features from the high-precision point cloud and the low-precision point cloud using the FPFH descriptor to obtain point cloud features; S53. Directly concatenate the point-by-point image features and the point cloud features along the channel dimension, and simplify the feature dimension to obtain the fused features.

2. The method according to claim 1, characterized in that Said S1 comprises: S11, in the encoding stage, extracting multi-scale features from the two-dimensional ship image through multi-layer convolution operations; S12, in the conversion stage, the extracted multi-scale features are converted into a three-dimensional initial point cloud through a fully connected layer or a reshaping operation; S13. In the decoding stage, the three-dimensional initial point cloud is subjected to multi-layer deconvolution operations to output the reconstructed point cloud with a complete global structure.

3. The method according to claim 1, characterized in that The S2 includes: S21, coarsely registering the original ship point cloud with the reconstructed point cloud to obtain an initial registered point cloud; S22: Finely register the original ship point cloud with the initial registration point cloud to obtain a target registration point cloud.

4. The method according to claim 3, characterized in that The S21 includes: S211, extracting key feature points from the original ship point cloud and the reconstructed point cloud, and calculating a descriptor for each feature point; S212, screening out a set of matching feature point pairs based on the similarity of the descriptors of all feature points of the two point clouds; S213, estimating rigid transformation parameters based on the screened matching feature point pair set; S214 : transforming the reconstructed point cloud according to the estimated rigid transformation parameters to obtain the initial registration point cloud.

5. The method according to claim 4, characterized in that The S22 includes: S221. For each point in the original ship point cloud, search for the nearest point in the initial registration point cloud to form a current point pair set; S222, re-estimating the rigid transformation parameters according to the current point pair set; S223, updating the initial registration point cloud according to the re-estimated rigid transformation parameters; and calculating the error between the updated initial registration point cloud and the original ship point cloud; S224, repeat S221 to S223 until the error is lower than the preset threshold or the maximum number of iterations is reached, stop updating, and use the last updated initial registration point cloud as the target registration point cloud.

6. The method according to claim 1, characterized in that The S6 includes: S61, calculating the average distance between points of the evenly distributed registration point cloud, and setting the constraints of the high-precision point cloud and the low-precision point cloud according to the average distance between the points; S62: For high-precision point clouds, fusion features are used to predict high-precision offsets; for low-precision point clouds, fusion features are used to predict low-precision offsets through a 3-layer MLP. S63, updating the high-precision point cloud according to the high-precision offset and the constraints of the high-precision point cloud; updating the low-precision point cloud according to the low-precision offset and the constraints of the low-precision point cloud; S64: Merge the updated high-precision point cloud and the updated low-precision point cloud to obtain the ship completed point cloud.

7. A ship point cloud completion system based on multimodal data fusion, characterized in that: include: A point cloud reconstruction unit is used to map the two-dimensional ship image into a reconstructed point cloud with complete global structure through an encoding-transformation-decoding network architecture; a point cloud registration unit, configured to perform coarse registration and fine registration on the original ship point cloud and the reconstructed point cloud to obtain a target registration point cloud; The point cloud optimization unit is used to obtain a uniformly distributed registration point cloud by sampling the target registration point cloud using a Poisson disk; A point cloud division unit is used to divide the evenly distributed registration point cloud into high-precision point cloud and low-precision point cloud according to the chamfer distance; A feature fusion unit is used to extract features from the two-dimensional ship image, high-precision point cloud and low-precision point cloud, and perform feature fusion to obtain fused features; The point cloud completion unit is used to optimize and complete the high-precision point cloud and the low-precision point cloud according to the fusion features, and merge them to obtain the completed point cloud of the ship; The point cloud division unit includes: A first calculation subunit is configured to calculate, for each point in the uniformly distributed registration point cloud, a nearest neighbor distance from the point to the original ship point cloud; The second computing subunit is used to randomly divide the uniformly distributed registration point cloud into two point sets and calculate the chamfer distance between the two point sets; A division subunit is used to divide the uniform distribution registration point cloud into a high-precision point cloud and a low-precision point cloud according to the nearest neighbor distance and chamfer distance of each point in the uniform distribution registration point cloud; The feature fusion unit includes: The image feature extraction subunit is used to extract semantic features from two-dimensional ship images using a lightweight convolutional neural network and map them to point cloud resolution to obtain point-by-point image features; The point cloud feature extraction subunit is used to extract local geometric features of high-precision point clouds and low-precision point clouds using FPFH descriptors to obtain point cloud features; The feature fusion subunit is used to directly splice the point-by-point image features and the point cloud features along the channel dimension, and simplify the feature dimension to obtain the fused features.

8. The system according to claim 7, characterized in that The point cloud reconstruction unit includes: The encoding subunit is used to extract multi-scale features from the two-dimensional ship image through multi-layer convolution operations during the encoding stage; The conversion subunit is used to convert the extracted multi-scale features into a three-dimensional initial point cloud through a fully connected layer or a reshaping operation in the conversion stage; The decoding subunit is used to output the reconstructed point cloud with complete global structure through multi-layer deconvolution operation on the three-dimensional initial point cloud in the decoding stage.

Images