A 3D point cloud data completion method based on multimodal joint perception

Abstract

A three-dimensional point cloud data completion method based on multimodal joint perception belongs to the field of point cloud completion technology. The method is as follows: using the CLIP visual large model to extract image features; extracting point cloud features; comprehensively using image features and point cloud features to generate shape latent codes to guide the diffusion model to generate the initial point cloud. After splicing the initial point cloud and the incomplete input point cloud, the farthest point sampling is used to obtain the process point cloud; based on non-local similarity, the process point cloud coordinate offset is generated; the offset is adjusted using a mask mechanism, and the high-quality points and low-quality points in the process point cloud are processed to obtain a completed high-quality point cloud. The present invention proposes to complete cross-modal shape fusion based on a diffusion model, and at the same time, use the image features extracted by the visual large model and the point cloud features to generate a shape latent code to guide the diffusion model, thereby realizing the transfer from the image domain to the point cloud domain; based on non-local similarity, a mask mechanism is used to generate a coordinate offset, thereby realizing cross-level refinement and rough output, and generating a higher quality point cloud.

Description

Technical Field

[0001] The invention relates to a three-dimensional point cloud data completion method based on multi-modal joint perception, and belongs to the technical field of point cloud completion. Background Art

[0002] With the rapid development of scanning equipment and 3D reconstruction technology, point cloud, as the simplest and most flexible data structure, has been widely used in many fields such as robot navigation, virtual / augmented reality, and autonomous driving. In these application scenarios, it is very necessary to obtain complete point cloud data. However, due to line of sight obstruction or long acquisition distance, the obtained point cloud data always has incomplete content. Point cloud completion technology can effectively infer the missing point cloud information, thereby realizing the reconstruction or supplement of the complete point cloud.

[0003] Traditional point cloud completion techniques only analyze and predict single-modal point cloud data for shape recovery. However, problems such as the low scanning resolution of 3D sensors usually result in the inherent sparsity of the captured point cloud, and the available information of the point cloud data is limited. There is a large uncertainty in inferring the missing parts using only single-modal point clouds, so single-modal point cloud completion is challenging. It is worth noting that existing technologies have been able to map the experience learned from 2D images to the understanding of the 3D world, and have successfully achieved the inference and reconstruction of the shape of 3D objects from 2D image experience. Therefore, it is reasonable to assume that point cloud completion technology can benefit from 2D images. At the same time, with the reduction of hardware costs, cross-sensor settings such as 3D scanners with RGB cameras and depth cameras are becoming more and more common. The point cloud data obtained is accompanied by a one-to-one corresponding 2D image, which is also the advantage of the joint image modality to complete the point cloud completion task. So far, there are relatively few technologies that use data from additional modalities to supplement the missing information of point clouds for enhancement, and the quality of the completed point cloud still needs to be further improved. Summary of the invention

[0004] In order to solve the problems existing in the background technology, the present invention provides a three-dimensional point cloud data completion method based on multimodal joint perception.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions:

[0006] Solution 1:

[0007] A three-dimensional point cloud data completion method based on multimodal joint perception, the method comprising the following steps:

[0008] S1: Coarse-grained complete model generation;

[0009] S101: Use the pre-trained CLIP visual model or ResNet18 to extract image features of a single view;

[0010] S102: Extract point cloud features using PointNet;

[0011] S103: comprehensively utilizing the image features obtained in S101 and the point cloud features extracted in S102 to generate a shape latent code, and guide the diffusion model to perform shaping to generate a complete but rough initial point cloud.

[0012] S2: Fine-grained texture optimization.

[0013] S201: splicing the initial point cloud obtained in S1 with the incomplete input point cloud to obtain an intermediate point cloud;

[0014] S202: Perform farthest point sampling on the intermediate point cloud obtained in S201 to obtain a process point cloud;

[0015] S20201: Input the intermediate point cloud obtained in S201, select the point farthest from the center of gravity of the point cloud as the starting point P0, and obtain the sampling point set {P0};

[0016] S20202: Calculate the Euclidean distances from all points except the starting point P0 to the starting point P0, select the point corresponding to the maximum value of the Euclidean distance value as the first far point P1, and update the sampling point set {P0, P1};

[0017] S20203: Calculate the Euclidean distance from one of the unsampled points to all the sampled points, and select the closest distance;

[0018] S20204: Calculate all unsampled points in sequence according to S20203 to obtain a collection of the shortest distances;

[0019] S20205: Select the point corresponding to the maximum value in the closest distance set as the second farthest point P2, and update the sampling point set {P0, P1, P2};

[0020] S20206: Repeat S20203-S20205 until N' target points are sampled to obtain the final sampling point set {P0, P1, ..., P N’-1 , P N’}, that is: process point cloud.

[0021] S203: Based on non-local similarity, the geometric relationship and feature distance between the process point cloud obtained in S202 and the incomplete input point cloud are calculated to generate a coordinate offset of the process point cloud obtained in S202;

[0022] S20301: input the incomplete input point cloud and the process point cloud obtained in S202 into the same dynamic graph convolutional neural network to obtain feature f1 and feature f2 respectively;

[0023] S20302: Calculate the feature similarity between feature f1 and feature f2 by dot product, and calculate the coordinate distance between the incomplete input point cloud and the process point cloud by Euclidean distance;

[0024] S20303: combining feature similarity and coordinate distance to obtain a distance matrix;

[0025] S20304: Based on non-local similarity, for each point in the process point cloud, take the first k points corresponding to the closest distances in the entire distance matrix, find the k features corresponding to the k points in the feature f1, perform maximum pooling and average pooling on the k features respectively, and obtain the maximum value feature and average value feature of the k features;

[0026] S20305: splicing the maximum value feature and the average value feature of the k features with the feature f2 at the corresponding position of the process point cloud and the coordinates of the corresponding position of the process point cloud to obtain a spliced ​​feature;

[0027] S20306: Process all points in the process point cloud by concatenating features to obtain feature f;

[0028] S20307: Input feature f into the MLP and output the coordinate offset value of each point in the process point cloud.

[0029] S204: Use the mask mechanism to adjust the coordinate offset, use different offset strategies to process the high-quality points from the incomplete input point cloud obtained in S202 and the low-quality points from the initial point cloud obtained in S1, refine the rough output of the shaping network, and obtain a completed high-quality complete point cloud.

[0030] S20401: setting the mask values ​​of the high-quality points from the incomplete input point cloud in the process point cloud obtained in S202 to 0, covering the generated coordinate offset, and limiting the movement of the points in the fine part;

[0031] S20402: setting the mask value of the low-quality point from the initial point cloud obtained in S1 in the process point cloud obtained in S202 to 1, and performing the offset normally;

[0032] S20403: multiply the mask value of S20401 and the mask value of S20402 by the coordinate offset respectively to obtain a new coordinate offset value;

[0033] S20404: Use the coordinates of the process point cloud obtained in S202 and the new coordinate offset value obtained in S20403 to output a completed high-quality complete point cloud.

[0034] Option 2:

[0035] A three-dimensional point cloud data completion method based on multimodal joint perception, the method comprising the following steps:

[0036] S1: Coarse-grained complete model generation;

[0037] S101: Use the pre-trained CLIP visual model or ResNet18 to extract image features of a single view;

[0038] S102: uniformly sample N points from the surface of a cube or a sphere to initialize a 3D voxel point set containing N points;

[0039] S103: The image features obtained in S101 are copied N times and then spliced ​​to the coordinates of the 3D voxel point set to obtain the feature f';

[0040] S104: Input the feature f' into the MLP to generate a complete but rough initial point cloud.

[0041] S2: Fine-grained texture optimization.

[0042] S201: splicing the initial point cloud obtained in S1 with the incomplete input point cloud to obtain an intermediate point cloud;

[0043] S202: Perform farthest point sampling on the intermediate point cloud obtained in S201 to obtain a process point cloud;

[0044] S20201: Input the intermediate point cloud obtained in S201, select the point farthest from the center of gravity of the point cloud as the starting point P0, and obtain the sampling point set {P0};

[0045] S20202: Calculate the Euclidean distances from all points except the starting point P0 to the starting point P0, select the point corresponding to the maximum value of the Euclidean distance value as the first far point P1, and update the sampling point set {P0, P1};

[0046] S20203: Calculate the Euclidean distance from one of the unsampled points to all the sampled points, and select the closest distance;

[0047] S20204: Calculate all unsampled points in sequence according to S20203 to obtain a collection of the shortest distances;

[0048] S20205: Select the point corresponding to the maximum value in the closest distance set as the second farthest point P2, and update the sampling point set {P0, P1, P2};

[0049] S20206: Repeat S20203-S20205 until N' target points are sampled to obtain the final sampling point set {P0, P1, ..., P N’-1 , P N’}, that is: process point cloud.

[0050] S203: Based on non-local similarity, the geometric relationship and feature distance between the process point cloud obtained in S202 and the incomplete input point cloud are calculated to generate a coordinate offset of the process point cloud obtained in S202;

[0051] S20301: input the incomplete input point cloud and the process point cloud obtained in S202 into the same dynamic graph convolutional neural network to obtain feature f1 and feature f2 respectively;

[0052] S20302: Calculate the feature similarity between feature f1 and feature f2 by dot product, and calculate the coordinate distance between the incomplete input point cloud and the process point cloud by Euclidean distance;

[0053] S20303: combining feature similarity and coordinate distance to obtain a distance matrix;

[0054] S20304: Based on non-local similarity, for each point in the process point cloud, take the first k points corresponding to the closest distances in the entire distance matrix, find the k features corresponding to the k points in the feature f1, perform maximum pooling and average pooling on the k features respectively, and obtain the maximum value feature and average value feature of the k features;

[0055] S20305: splicing the maximum value feature and the average value feature of the k features with the feature f2 at the corresponding position of the process point cloud and the coordinates of the corresponding position of the process point cloud to obtain a spliced ​​feature;

[0056] S20306: Process all points in the process point cloud by concatenating features to obtain feature f;

[0057] S20307: Input feature f into the MLP and output the coordinate offset value of each point in the process point cloud.

[0058] S204: Use the mask mechanism to adjust the coordinate offset, use different offset strategies to process the high-quality points from the incomplete input point cloud obtained in S202 and the low-quality points from the initial point cloud obtained in S1, refine the rough output of the shaping network, and obtain a completed high-quality complete point cloud.

[0059] S20401: setting the mask values ​​of the high-quality points from the incomplete input point cloud in the process point cloud obtained in S202 to 0, covering the generated coordinate offset, and limiting the movement of the points in the fine part;

[0060] S20402: setting the mask value of the low-quality point from the initial point cloud obtained in S1 in the process point cloud obtained in S202 to 1, and performing the offset normally;

[0061] S20403: multiply the mask value of S20401 and the mask value of S20402 by the coordinate offset respectively to obtain a new coordinate offset value;

[0062] S20404: Use the coordinates of the process point cloud obtained in S202 and the new coordinate offset value obtained in S20403 to output a completed high-quality complete point cloud.

[0063] Option 3:

[0064] A three-dimensional point cloud data completion method based on multimodal joint perception, the method comprising the following steps:

[0065] S1: Coarse-grained complete model generation;

[0066] S101: Use the pre-trained CLIP visual model or ResNet18 to extract image features of a single view;

[0067] S102: Add Gaussian noise to the incomplete input point cloud to initialize a 3D point set containing N points;

[0068] S103: The image features obtained in S101 are copied N times and then spliced ​​to the coordinates of the 3D point set to obtain the feature f';

[0069] S104: Input the feature f' into the MLP to generate a complete but rough initial point cloud.

[0070] S2: Fine-grained texture optimization.

[0071] S201: splicing the initial point cloud obtained in S1 with the incomplete input point cloud to obtain an intermediate point cloud;

[0072] S202: Perform farthest point sampling on the intermediate point cloud obtained in S201 to obtain a process point cloud;

[0073] S20201: Input the intermediate point cloud obtained in S201, select the point farthest from the center of gravity of the point cloud as the starting point P0, and obtain the sampling point set {P0};

[0074] S20202: Calculate the Euclidean distances from all points except the starting point P0 to the starting point P0, select the point corresponding to the maximum value of the Euclidean distance value as the first far point P1, and update the sampling point set {P0, P1};

[0075] S20203: Calculate the Euclidean distance from one of the unsampled points to all the sampled points, and select the closest distance;

[0076] S20204: Calculate all unsampled points in sequence according to S20203 to obtain a collection of the shortest distances;

[0077] S20205: Select the point corresponding to the maximum value in the closest distance collection as the second farthest point P2, and update the sampling point set {P0, P1, P2};

[0078] S20206: Repeat S20203-S20205 until N' target points are sampled to obtain the final sampling point set {P0, P1, ..., P N’-1 , P N’}, that is: process point cloud.

[0079] S203: Based on non-local similarity, the geometric relationship and feature distance between the process point cloud obtained in S202 and the incomplete input point cloud are calculated to generate a coordinate offset of the process point cloud obtained in S202;

[0080] S20301: input the incomplete input point cloud and the process point cloud obtained in S202 into the same dynamic graph convolutional neural network to obtain feature f1 and feature f2 respectively;

[0081] S20302: Calculate the feature similarity between feature f1 and feature f2 by dot product, and calculate the coordinate distance between the incomplete input point cloud and the process point cloud by Euclidean distance;

[0082] S20303: combining feature similarity and coordinate distance to obtain a distance matrix;

[0083] S20304: Based on non-local similarity, for each point in the process point cloud, take the first k points corresponding to the closest distances in the entire distance matrix, find the k features corresponding to the k points in the feature f1, perform maximum pooling and average pooling on the k features respectively, and obtain the maximum value feature and average value feature of the k features;

[0084] S20305: splicing the maximum value feature and the average value feature of the k features with the feature f2 at the corresponding position of the process point cloud and the coordinates of the corresponding position of the process point cloud to obtain a spliced ​​feature;

[0085] S20306: Process all points in the process point cloud by concatenating features to obtain feature f;

[0086] S20307: Input feature f into the MLP and output the coordinate offset value of each point in the process point cloud.

[0087] S204: Use the mask mechanism to adjust the coordinate offset, use different offset strategies to process the high-quality points from the incomplete input point cloud obtained in S202 and the low-quality points from the initial point cloud obtained in S1, refine the rough output of the shaping network, and obtain a completed high-quality complete point cloud.

[0088] S20401: setting the mask values ​​of the high-quality points from the incomplete input point cloud in the process point cloud obtained in S202 to 0, covering the generated coordinate offset, and limiting the movement of the points in the fine part;

[0089] S20402: setting the mask value of the low-quality point from the initial point cloud obtained in S1 in the process point cloud obtained in S202 to 1, and performing the offset normally;

[0090] S20403: multiply the mask value of S20401 and the mask value of S20402 by the coordinate offset respectively to obtain a new coordinate offset value;

[0091] S20404: Use the coordinates of the process point cloud obtained in S202 and the new coordinate offset value obtained in S20403 to output a completed high-quality complete point cloud.

[0092] Compared with the prior art, the present invention has the following beneficial effects:

[0093] 1. Different from the previous cross-modal fusion technology, this invention proposes to complete cross-modal shape fusion based on diffusion model, and simultaneously uses the image features and point cloud features extracted by the visual large model to generate shape latent codes to guide the diffusion model, thus realizing the transfer from image domain to point cloud domain;

[0094] 2. Different from the previous method of calculating local similarity to refine the rough point cloud, the present invention generates coordinate offsets based on non-local similarity using a mask mechanism, achieves cross-level refinement of the rough output, and generates a higher quality point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 is a flow chart of the present invention;

[0096] Figure 2 It is a schematic diagram of the directed process of the point cloud diffusion model;

[0097] Figure 3 It is a schematic diagram of the coordinate shift network structure generated based on non-local similarity;

[0098] Figure 4 It is a schematic diagram of edge convolution;

[0099] Figure 5 It is a schematic diagram of the mask mechanism. DETAILED DESCRIPTION

[0100] The technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0101] Embodiment 1:

[0102] A three-dimensional point cloud data completion method based on multimodal joint perception is a three-dimensional point cloud completion technology that combines image modality application visual large model technology, diffusion model technology and non-local similarity calculation technology. The method comprises the following steps:

[0103] S1: Coarse-grained complete model generation;

[0104] S101: Use the pre-trained CLIP visual model or ResNet18 to extract image features of a single view;

[0105] S102: Extract point cloud features using PointNet;

[0106] Both image features and point cloud features are global features. The point cloud features extracted by dynamic graph convolutional neural network reflect local detail information.

[0107] S103: Comprehensively utilize the image features obtained in S101 and the point cloud features extracted in S102 to generate shape latent codes, and guide the diffusion model to shape and generate a complete but relatively rough initial point cloud. The directed process of the diffusion model is shown in the attached figure. Figure 2 shown.

[0108] S2: Fine-grained texture optimization.

[0109] S201: splicing the initial point cloud obtained in S1 with the incomplete input point cloud with high-quality detail points, retaining as many local details of the shape as possible, and obtaining an intermediate point cloud;

[0110] S202: Perform farthest point sampling on the intermediate point cloud obtained in S201 to obtain a process point cloud;

[0111] S20201: Input the intermediate point cloud obtained in S201, select the point farthest from the center of gravity of the point cloud as the starting point P0, and obtain the sampling point set {P0};

[0112] S20202: Calculate the Euclidean distances from all points except the starting point P0 to the starting point P0, select the point corresponding to the maximum value of the Euclidean distance value as the first far point P1, and update the sampling point set {P0, P1};

[0113] S20203: Calculate the Euclidean distance from one of the unsampled points to all the sampled points, and select the closest distance;

[0114] S20204: Calculate all unsampled points in sequence according to S20203 to obtain a collection of the shortest distances;

[0115] S20205: Select the point corresponding to the maximum value in the closest distance collection as the second farthest point P2, and update the sampling point set {P0, P1, P2};

[0116] S20206: Repeat S20203-S20205 until N' target points are sampled to obtain the final sampling point set {P0, P1, ..., P N’-1 , P N’}, that is: process point cloud.

[0117] For example: there are 10 points in the point cloud.

[0118] Step 1: Select the point farthest from the center of gravity of the point cloud as the starting point P0, and obtain the sampling point set {P0};

[0119] Step 2: Calculate the Euclidean distances from the remaining 9 points to the starting point P0, select the point corresponding to the maximum value of the Euclidean distance value as the first far point P1, and update the sampling point set {P0, P1};

[0120] Step 3: For ease of understanding, name the 8 points other than the starting point P0 and the first far point P1 as A, B, C, D, E, F, G, and H respectively;

[0121] Step 4: Calculate the Euclidean distance from point A to the starting point P0, and calculate the Euclidean distance from point A to the first farthest point P1. Compare the two Euclidean distances of point A, and select the smaller of the two Euclidean distances as the closest distance;

[0122] Step 5: Calculate the Euclidean distance from point B to the starting point P0, and calculate the Euclidean distance from point B to the first farthest point P1. Compare the two Euclidean distances of point B and select the closest distance;

[0123] Step 6: Calculate points C, D, E, F, G, and H in turn according to steps 4 and 5, and select the closest distance;

[0124] Step 7: Integrate the 8 closest distances of the above 8 points to obtain the closest distance collection;

[0125] Step 8: Select the point corresponding to the maximum value in the closest distance set, such as point A, record it as the second farthest point P2, and update the sampling point set {P0, P1, P2};

[0126] Step 9: Calculate the Euclidean distance from point B to the starting point P0, calculate the Euclidean distance from point B to the first farthest point P1, and calculate the Euclidean distance from point B to the second farthest point P2. Compare the three Euclidean distances of point B and select the closest distance;

[0127] Step 10: Calculate the Euclidean distance from point C to the starting point P0, calculate the Euclidean distance from point C to the first farthest point P1, and calculate the Euclidean distance from point C to the second farthest point P2. Compare the three Euclidean distances of point C and select the closest distance;

[0128] Step 11: Calculate point D, point E, point F, point G, and point H in turn according to step 9 and step 10, and select the shortest distance;

[0129] Step 12: Integrate the 7 closest distances of the above 7 points to obtain the closest distance collection;

[0130] Step 13: Select the point corresponding to the maximum value in the closest distance set, such as point B, record it as the third farthest point P3, and update the sampling point set {P0, P1, P2, P3};

[0131] The remaining unsampled points are operated according to the above steps until N' target points are sampled and the final sampling point set {P0, P1, ..., P N’-1 , P N’}, that is: process point cloud.

[0132] S203: Based on non-local similarity, the geometric relationship and feature distance between the process point cloud obtained in S202 and the incomplete input point cloud are calculated to generate the coordinate offset of the process point cloud obtained in S202. The network structure is as shown in the attached figure. Figure 3 As shown;

[0133] S20301: Input the incomplete input point cloud and the process point cloud obtained in S202 into the same dynamic graph convolutional neural network to obtain features f1 and f2. The core technology of the dynamic graph convolutional neural network, edge convolution, is shown in the attached figure. Figure 4 As shown;

[0134] S20302: Calculate the feature similarity between feature f1 and feature f2 by dot product, and calculate the coordinate distance between the incomplete input point cloud and the process point cloud by Euclidean distance;

[0135] S20303: combining feature similarity and coordinate distance to obtain a distance matrix;

[0136] S20304: Based on non-local similarity, for each point in the process point cloud, take the first k points corresponding to the closest distances in the entire distance matrix, find the k features corresponding to the k points in the feature f1, perform maximum pooling and average pooling on the k features respectively, and obtain the maximum value feature and average value feature of the k features;

[0137] S20305: splicing the maximum value feature and the average value feature of the k features with the feature f2 at the corresponding position of the process point cloud and the coordinates of the corresponding position of the process point cloud to obtain a spliced ​​feature;

[0138] S20306: Process all points in the process point cloud by concatenating features to obtain feature f;

[0139] S20307: Input feature f into the MLP and output the coordinate offset value of each point in the process point cloud.

[0140] S204: Use the mask mechanism to adjust the coordinate offset in different degrees in different layers, use different offset strategies to process the high-quality points from the incomplete input point cloud obtained in S202 and the low-quality points from the initial point cloud obtained in S1, refine the rough output of the shaping network, and finally obtain a high-quality complete point cloud that is completed from coarse to fine and from the whole to the part. The mask mechanism is as shown in the attached figure. Figure 5 shown.

[0141] S20401: setting the mask values ​​of the high-quality points from the incomplete input point cloud in the process point cloud obtained in S202 to 0, covering the generated coordinate offset, and limiting the movement of the points in the fine part;

[0142] S20402: setting the mask value of the low-quality point from the initial point cloud obtained in S1 in the process point cloud obtained in S202 to 1, and performing the offset normally;

[0143] S20403: multiply the mask value of S20401 and the mask value of S20402 by the coordinate offset respectively to obtain a new coordinate offset value;

[0144] S20404: Use the coordinates of the process point cloud obtained in S202 and the new coordinate offset value obtained in S20403 to output a completed high-quality complete point cloud.

[0145] Example 1 of the present invention was tested on the ShapeNet-ViPC dataset and compared with 11 existing point cloud completion technologies, including point-based AtlasNet and PCN, folding-based FoldingNet and TopNet, convolution-based GRNet, GAN-based PF-Net, Transformer-based PoinTr, SDT and Seedformer, and cross-modal methods ViPC and CSDN.

[0146] Chamfer distance and F-Score are selected as evaluation indicators. The calculation method is shown in formula (1) and formula (2). Lower chamfer distance and higher F-Score correspond to better completion quality. The CD value unit is 10 -3 , the threshold of F-score is 0.001.

[0147] The results are shown in Table 1 and Table 2. It can be seen that the method of the present invention achieves the best completion performance compared with other methods.

[0148]

[0149]

[0150] Where:

[0151] CD represents the chamfer distance;

[0152] S1 represents the complete point cloud;

[0153] S2 represents the ground truth reference point cloud, which is a reference standard for evaluating model performance. The more similar the complete point cloud generated by the present invention is to the ground truth reference point cloud, the better.

[0154] x represents a point in the point set S1;

[0155] y represents a point in the point set S2;

[0156] P represents precision;

[0157] d represents the distance threshold, which controls the strictness of the F-Score;

[0158] P(d) represents the precision calculation function, which calculates the percentage of reconstructed points within a certain distance from the ground truth reference point, representing the accuracy of reconstruction;

[0159] R represents recall;

[0160] R(d) represents the recall calculation function, which calculates the percentage of ground truth reference points to the reconstructed points within a certain distance, indicating the completeness of the reconstruction;

[0161] F-Score is the harmonic mean between precision and recall, which is used to evaluate the surface distance of an object and reflects the percentage of correctly reconstructed points.

[0162] Table 1 Chamfer distance

[0163]

[0164]

[0165] Table 2 F-score@0.001

[0166]

[0167]

[0168] Embodiment 2:

[0169] The difference between this embodiment and embodiment 1 is that:

[0170] The S1 comprises the following steps:

[0171] S101: Use the pre-trained CLIP visual model or ResNet18 to extract image features of a single view;

[0172] S102: uniformly sample N points from the surface of a cube or a sphere to initialize a 3D voxel point set containing N points;

[0173] S103: The image features obtained in S101 are copied N times and then spliced ​​to the coordinates of the 3D voxel point set to obtain the feature f';

[0174] S104: Input the feature f' into the MLP to generate a complete but relatively rough initial point cloud.

[0175] Embodiment 3:

[0176] The difference between this embodiment and embodiment 1 is that:

[0177] The S1 comprises the following steps:

[0178] S101: Use the pre-trained CLIP visual model or ResNet18 to extract image features of a single view;

[0179] S102: Add Gaussian noise to the incomplete input point cloud to initialize a 3D point set containing N points;

[0180] S103: The image features obtained in S101 are copied N times and then spliced ​​to the coordinates of the 3D point set to obtain the feature f';

[0181] S104: Input the feature f' into the MLP to generate a complete but relatively rough initial point cloud.

[0182] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations within the meaning and range of equivalents of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0183] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A three-dimensional point cloud data completion method based on multimodal joint perception, characterized by: The method comprises the following steps: S1: Coarse-grained complete model generation; S2: fine-grained texture optimization; The S2 comprises the following steps: S201: splicing the initial point cloud obtained in S1 with the incomplete input point cloud to obtain an intermediate point cloud; S202: Perform farthest point sampling on the intermediate point cloud obtained in S201 to obtain a process point cloud; S203: Based on non-local similarity, the geometric relationship and feature distance between the process point cloud obtained in S202 and the incomplete input point cloud are calculated to generate a coordinate offset of the process point cloud obtained in S202; S204: Use the mask mechanism to adjust the coordinate offset, use different offset strategies to process the high-quality points from the incomplete input point cloud obtained in S202 and the low-quality points from the initial point cloud obtained in S1, refine the rough output of the shaping network, and obtain a completed high-quality complete point cloud.

2. The method for completing three-dimensional point cloud data based on multimodal joint perception according to claim 1, characterized in that: The S1 comprises the following steps: S101: Use the pre-trained CLIP visual model or ResNet18 to extract image features of a single view; S102: Extract point cloud features using PointNet; S103: comprehensively utilizing the image features obtained in S101 and the point cloud features extracted in S102 to generate a shape latent code, and guide the diffusion model to perform shaping to generate a complete but rough initial point cloud.

3. The method for completing three-dimensional point cloud data based on multimodal joint perception according to claim 1, characterized in that: The S1 comprises the following steps: S101: Use the pre-trained CLIP visual model or ResNet18 to extract image features of a single view; S102: uniformly sample N points from the surface of a cube or a sphere to initialize a 3D voxel point set containing N points; S103: The image features obtained in S101 are copied N times and then spliced ​​to the coordinates of the 3D voxel point set to obtain the feature f'; S104: Input the feature f' into the MLP to generate a complete but rough initial point cloud.

4. The method for completing three-dimensional point cloud data based on multimodal joint perception according to claim 1, characterized in that: The S1 comprises the following steps: S101: Use the pre-trained CLIP visual model or ResNet18 to extract image features of a single view; S102: Add Gaussian noise to the incomplete input point cloud to initialize a 3D point set containing N points; S103: The image features obtained in S101 are copied N times and then spliced ​​to the coordinates of the 3D point set to obtain the feature f'; S104: Input the feature f' into the MLP to generate a complete but rough initial point cloud.

5. The method for completing three-dimensional point cloud data based on multimodal joint perception according to claim 4, characterized in that: The S202 comprises the following steps: S20201: Input the intermediate point cloud obtained in S201, select the point farthest from the center of gravity of the point cloud as the starting point P0, and obtain the sampling point set {P0}; S20202: Calculate the Euclidean distances from all points except the starting point P0 to the starting point P0, select the point corresponding to the maximum value of the Euclidean distance value as the first far point P1, and update the sampling point set {P0, P1}; S20203: Calculate the Euclidean distance from one of the unsampled points to all the sampled points, and select the closest distance; S20204: Calculate all unsampled points in sequence according to S20203 to obtain a collection of the shortest distances; S20205: Select the point corresponding to the maximum value in the closest distance collection as the second farthest point P2, and update the sampling point set {P0, P1, P2}; S20206: Repeat S20203-S20205 until N' target points are sampled to obtain the final sampling point set {P0, P1, ..., P N’-1 , P N’ }, that is: process point cloud.

6. The method for completing three-dimensional point cloud data based on multimodal joint perception according to claim 5, characterized in that: The S203 comprises the following steps: S20301: input the incomplete input point cloud and the process point cloud obtained in S202 into the same dynamic graph convolutional neural network to obtain feature f1 and feature f2 respectively; S20302: Calculate the feature similarity between feature f1 and feature f2 by dot product, and calculate the coordinate distance between the incomplete input point cloud and the process point cloud by Euclidean distance; S20303: combining feature similarity and coordinate distance to obtain a distance matrix; S20304: Based on non-local similarity, for each point in the process point cloud, take the first k points corresponding to the closest distances in the entire distance matrix, find the k features corresponding to the k points in the feature f1, perform maximum pooling and average pooling on the k features respectively, and obtain the maximum value feature and average value feature of the k features; S20305: splicing the maximum value feature and the average value feature of the k features with the feature f2 at the corresponding position of the process point cloud and the coordinates of the corresponding position of the process point cloud to obtain a spliced ​​feature; S20306: Process all points in the process point cloud by concatenating features to obtain feature f; S20307: Input feature f into the MLP and output the coordinate offset value of each point in the process point cloud.

7. The method for completing three-dimensional point cloud data based on multimodal joint perception according to claim 5, characterized in that: The S204 comprises the following steps: S20401: setting the mask values ​​of the high-quality points from the incomplete input point cloud in the process point cloud obtained in S202 to 0, covering the generated coordinate offset, and limiting the movement of the points in the fine part; S20402: setting the mask value of the low-quality point from the initial point cloud obtained in S1 in the process point cloud obtained in S202 to 1, and performing the offset normally; S20403: multiply the mask value of S20401 and the mask value of S20402 by the coordinate offset respectively to obtain a new coordinate offset value; S20404: Use the coordinates of the process point cloud obtained in S202 and the new coordinate offset value obtained in S20403 to output a completed high-quality complete point cloud.

Images