Bidirectional cross-modal image-guided point cloud inpainting method with multi-scale progressive refinement

Abstract

The present invention discloses a bidirectional cross-modal image-guided point cloud restoration method with multi-scale progressive refinement, which relates to the field of computer vision technology for artificial intelligence drones or embodied robots, including: converting an incomplete point cloud into a depth map; based on a bidirectional interactive compensation module, compensating for the missing geometric depth information in the view image modality according to the depth map, and compensating for the missing texture and color information in the point cloud modality using the view image modality, to obtain a final enhanced feature representation; decoding the final enhanced feature representation based on a coarse point cloud generation module to generate a coarse point cloud; using a point cloud refinement module to refine the coarse point cloud, and guiding the fusion of global shape and local structure information through dual-stream features to gradually generate a predicted refined point cloud. The present invention bridges the information gap between point cloud and image modalities, enhances the accuracy of generating the overall shape and local details of the point cloud, and can reasonably generate a complete point cloud with a fine-grained semantic structure.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology for artificial intelligence drones or embodied robots, and in particular to a bidirectional cross-modal image-guided point cloud restoration method with multi-scale progressive refinement. Background Art

[0002] During flight, drones require real-time three-dimensional information about their surroundings to construct an environmental model and perform autonomous navigation. This method uses sensors such as cameras and lidar onboard drones to acquire image and point cloud data. Using a bidirectional interaction compensation module, the texture and color information in the images is fused with the geometric depth information in the point cloud to generate a point cloud model of the environment. Embodied robots, when interacting with humans or the environment, require accurate spatial perception of their surroundings. This method generates detailed, high-quality point cloud models, enabling them to more accurately identify and understand information such as the shape, position, and size of surrounding objects. Embodied robots need to understand their surroundings to make informed decisions and actions. By fusing image and point cloud data, this method creates a rich global representation of semantic information. During autonomous navigation, embodied robots need to perceive their surroundings in real time and plan paths to avoid obstacles. The point cloud model generated by this method provides the robot with three-dimensional spatial information.

[0003] Point cloud repair technology aims to restore missing or sparse point cloud data caused by occlusion, sensor noise or resolution limitations. With the widespread application of 3D scanning technology, the demand for point cloud data in autonomous driving, robotic vision, 3D reconstruction, virtual reality and other fields is increasing. However, the inevitable missing problems in the point cloud acquisition process affect the integrity and accuracy of the data. Therefore, point cloud repair has become a key technology to improve data quality and application performance. Point cloud repair can not only improve the integrity and accuracy of the data and enhance the 3D modeling and environmental perception capabilities, but also has important significance for improving the safety of autonomous driving systems, optimizing the immersion in virtual reality, and supporting large-scale data set processing. By repairing missing point cloud data, the accuracy and reliability of subsequent applications can be guaranteed, especially in complex environments. In the future, point cloud repair will be combined with emerging technologies such as deep learning to continuously improve the repair accuracy and computational efficiency, and promote the further development of computer vision, artificial intelligence and robotics. However, there are still some problems with point cloud repair:

[0004] (1) Traditional point cloud restoration methods (such as algorithms based on interpolation, surface fitting, or neighborhood reconstruction) mainly rely on the local structure or spatial neighborhood information of the point cloud. Although these methods are simple and computationally efficient, they often cannot effectively restore details in complex scenes, especially when there is a large amount of missing data or sparse areas, the restoration effect is poor.

[0005] (2) Traditional methods mostly rely on fixed-scale restoration strategies, which may perform well at a certain scale but cannot effectively handle details at different levels. Especially in the case of complex geometric structures or large-scale missing information, conventional methods may not be able to restore global and local details at the same time.

[0006] (3) Traditional restoration methods may cause loss of details or generate unnatural restoration areas when processing large missing areas. For example, simple interpolation methods may make the restoration area too smooth, losing the details and structure of the real scene. Summary of the Invention

[0007] The present invention provides a bidirectional cross-modal image-guided point cloud restoration method with multi-scale progressive refinement. The method aims to significantly improve the restoration effect by utilizing bidirectional cross-modal and image information to guide point cloud restoration, and by utilizing a multi-scale progressive refinement strategy. In particular, when facing large-scale missing objects, complex scenes, or dynamic targets, the restoration results are more refined and natural.

[0008] In order to solve the above problems, the technical solutions adopted by the present invention are as follows:

[0009] The present invention provides a bidirectional cross-modal image-guided point cloud restoration method with multi-scale progressive refinement, comprising the following steps:

[0010] Step 1: Get the view image and incomplete point clouds , the incomplete point cloud Obtain the depth map through perspective projection transformation;

[0011] Step 2: Based on the bidirectional interactive compensation module with multi-scale progressive refinement, the depth map is used to compensate for the missing geometric depth information in the view image modality, and the view image modality is used to compensate for the missing texture and color information in the point cloud modality to obtain the final enhanced feature representation. ;

[0012] Step 3: Decode the final enhanced feature representation based on the coarse point cloud generation module , generating a coarse point cloud representing the approximate shape of the completed object ;

[0013] Step 4: Use the point cloud refinement module to refine the coarse point cloud Perform refinement processing, guide the fusion of global shape and local structure information through dual-stream features, and gradually generate predicted refined point clouds .

[0014] As a further description of the above technical solution, in step 1, a set of points needs to be projected onto six orthogonal planes to generate a sparse depth image; for each point in the point cloud, perspective projection is used to derive its 2D coordinates on the image plane; the coordinates are discretized, and the depth value of each image position is determined by the weighted average of nearby pixels.

[0015] As a further description of the above technical solution, in step 2, the bidirectional interactive compensation module includes a point encoder, a depth encoder, and an image encoder pre-trained on the ImageNet image dataset; wherein the point encoder consists of three layers of collective abstraction layers, which extract point cloud features from point clouds of different levels and scales through hierarchical downsampling ; The depth encoder uses the ResNet-18 model to extract depth map features from the depth map ; The image encoder is used to extract the view image features from the view image .

[0016] As a further description of the above technical solution, the bidirectional interaction compensation module includes two cross-modal Transformers; the first cross-modal Transformer is based on the depth map feature. and view image features The semantic association is established between them, and the specific implementation is expressed by the following formula:

[0017]

[0018]

[0019]

[0020]

[0021] in: Indicates that by and The fused features obtained by concatenation; Represents positional embedding; Represents the projected viewpoint; , , represents the linear layer; , , Represents the result after linear transformation, which is used for the operation of the self-attention mechanism; Representation layer normalization; represents a feed-forward network consisting of two linear layers;

[0022] The second cross-modal Transformer enhances view feature representation Point cloud features The semantic association is established between them, and the specific implementation formula is as follows:

[0023]

[0024]

[0025]

[0026]

[0027]

[0028]

[0029] in: Represents enhanced view features Point cloud features The fused features obtained by concatenation; Represents the final enhanced point cloud feature representation; and Represent the fusion features after primary and secondary normalization respectively; , , represents the linear layer; , , , , , Represents the result after linear transformation and is used for the operation of the self-attention mechanism.

[0030] As a further description of the above technical solution, the point cloud feature representation is enhanced With the initial point cloud features Connect together to get the final enhanced feature representation , and finally enhanced feature representation As the input of the coarse point cloud generation module and the point cloud refinement module.

[0031] As a further description of the above technical solution, in step 3, the coarse point cloud generation module uses transposed convolution to extract the final enhanced feature representation Decode the features of each point.

[0032] As a further description of the above technical solution, the coarse point cloud generation module adopts a two-branch approach. One branch uses the NodeShuffle module to generate new points from the latent space by combining the spatial information provided by adjacent points. The other branch uses the self-attention mechanism to simulate the long-range dependency between single-point features and capture the correlation between points to generate new points.

[0033] As a further description of the above technical solution, the new points of the two branches are combined together by element-by-element addition to obtain a newly generated single point, and the newly generated single point is combined with the final enhanced feature representation After connection, input into the multi-layer perceptron to generate a transition coarse point cloud , the transition coarse point cloud With incomplete point cloud Connect and use the farthest point sampling algorithm to get the coarse point cloud .

[0034] As a further description of the above technical solution, the point cloud refinement module includes a global shape prediction submodule, a local structure optimization submodule, a DFGF submodule and an upsampling submodule; the global shape prediction submodule is used to represent the point cloud according to the feature and coarse point cloud Generate feature representations that represent global shapes ; The local structure optimization submodule is used to optimize the incomplete point cloud based on edge convolution and farthest point sampling Iteratively aggregate and refine the local features to obtain a feature representation suitable for enhancing the local structure ;DFGF submodule is used to represent the and Get local structural features and global shape features ; The upsampling submodule is used to rely on local structural features and global shape features Fusion feature representation , generate coordinate offsets through MLP and reshaping operations, and use coordinate offsets to correct the coarse point cloud Make fine adjustments and finally get a complete point cloud .

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] (1) A bidirectional interactive compensation module is introduced to effectively address the cross-modal data challenge in image-guided point cloud completion. By enhancing feature representation, the module bridges the information gap between point cloud and image modalities.

[0037] (2) A point cloud refinement module is proposed, which uses a dual-stream structure to optimize the point cloud generation process, significantly enhancing the accuracy of the overall shape and local detail generation of the point cloud, and obtaining a more accurate and complete point cloud.

[0038] (3) The proposed bidirectional cross-modal image-guided point cloud restoration method with multi-scale progressive refinement successfully alleviates the sensitivity of previous models to the input view, can reasonably generate complete point clouds with fine-grained semantic structures, and achieves the best performance on benchmark datasets.

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, embodiments of the present invention are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 It is the overall framework diagram of the method of the present invention;

[0042] Figure 2 It is a two-way interactive compensation module diagram of the method of the present invention;

[0043] Figure 3 It is a rough point cloud generation module diagram of the method of the present invention;

[0044] Figure 4 It is a point cloud refinement module diagram of the method of the present invention;

[0045] Figure 5 It is the experimental result figure of the method of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0047] The embodiment of the present invention provides a bidirectional cross-modal image-guided point cloud restoration network BiMPR-Net with multi-scale progressive refinement. Figure 1 The BiMPR-Net model consists of a bidirectional interactive compensation module, a coarse point cloud generation module (CPGM), and a point cloud refinement module. The entire completion process follows a strategy of gradually generating a predicted point cloud from coarse to fine.

[0048] The embodiment of the present invention provides a bidirectional cross-modal image-guided point cloud restoration method with multi-scale progressive refinement, based on Figure 1 The BiMPR-Net network implementation shown includes the following steps:

[0049] Step 1: Get the view image and incomplete point clouds , the incomplete point cloud The depth map is obtained through perspective projection transformation.

[0050] In this implementation, a set of points is projected onto six orthogonal planes to generate a sparse depth image. For each point in the point cloud, perspective projection is used to derive its 2D coordinates on the image plane. The coordinates are discretized, and the depth value at each image location is determined by taking a weighted average of nearby pixels, which helps mitigate noise by averaging neighboring pixel values.

[0051] Step 2: Use a bidirectional interactive compensation module to compensate for the missing information in the point cloud modality and the depth map modality. The depth map obtained by point cloud projection is used to compensate for the missing geometric depth information in the view image modality. Conversely, the view image modality is used to compensate for the missing texture and color information in the point cloud modality. This process effectively fuses cross-modal data by enhancing feature representation, creating a more comprehensive and richer global representation of semantic information.

[0052] In this embodiment, the two-way interactive compensation module is as follows: Figure 1 The left part and Figure 2 As shown, it includes three encoders for extracting key feature information from different modal data, namely a point encoder for processing incomplete point cloud data, a depth encoder for processing depth maps, and an image encoder for processing view images.

[0053] Among them, the point cloud encoder is inspired by PointNet++ and consists of three layers of group sampling and point cloud network. It extracts key information from point clouds of different levels and scales through layered downsampling to obtain point cloud features. .

[0054] The depth encoder uses the classic ResNet-18 model. The number of input channels of the first convolutional layer is adjusted to 1, and the final depth map feature with a dimension of 128 is obtained. .

[0055] Similarly, the image encoder uses ResNet-18 as the backbone to extract perspective image features from the view image , the dimension is also 128.

[0056] To address the potential limitations of model training and generalization performance due to limited data, the present invention pre-trains the image encoder on the ImageNet image dataset to transfer visual modality knowledge from the large-scale dataset to the BiMPR-Net model proposed in this invention, thereby enhancing the generalization performance of the image encoder and making it better at capturing the precise semantic information in the feature representation of viewpoint images.

[0057] This paper uses cross-modal Transformer to extract features from depth map and view image features The semantic association is established between the two points, and the depth map obtained by point cloud projection is used to make up for the lack of spatial position and distance information in the single-view image. The specific implementation of this cross-modal Transformer is expressed by the following formula:

[0058] (1)

[0059] (2)

[0060] (3)

[0061] (4)

[0062] in: Indicates that by and The fused features obtained by concatenation; Represents positional embedding; Represents the projected viewpoint; , , represents the linear layer; , , Represents the result after linear transformation, which is used for the operation of the self-attention mechanism; Representation layer normalization; Represents a feedforward network consisting of two linear layers. This paper uses self-attention to learn fusion features In this process, the present invention conveys information about the projection viewpoint to the model through position encoding, helping the model understand the spatial relationship between features and viewpoint differences. By obtaining attention weights, the model selectively focuses on the fused features. different parts of the fusion feature to obtain more discriminative In order to further enhance the feature representation capability, this paper introduces a feedforward network to enable the model to learn more complex and abstract feature representations. . It can be considered as an enhanced view feature representation after compensating the depth image information.

[0063] Then, the enhanced view feature representation obtained after the first cross-modal Transformer is , fed to the second cross-modal Transformer to be combined with the point cloud features Establish semantic associations. This interactive process compensates for the missing appearance and visual attribute information in incomplete point clouds. Considering the increased complexity of dimensionality and feature interactions, an additional self-attention layer is introduced here compared to the previous cross-modal Transformer. This enhancement aims to better learn the correlation between cross-modal features. Since point cloud features inherently contain position information, this paper omits position embedding in this context. The formula is as follows:

[0064] (5)

[0065] (6)

[0066] (7)

[0067] (8)

[0068] (9)

[0069] (10)

[0070] in: Represents enhanced view features Point cloud features The fused features obtained by concatenation; and Represent the fusion features after primary and secondary normalization respectively; , , represents the linear layer; , , , , , Represents the result after linear transformation, which is used for the operation of the self-attention mechanism; Represents a feed-forward network consisting of two linear layers.

[0071] After passing through the second cross-modal Transformer, the enhanced point cloud feature representation is obtained .

[0072] Finally, the present invention will enhance the point cloud feature representation With the initial point cloud features Connect them to get the final enhanced feature representation of the model , which is the global representation. This feature represents It is achieved by seamlessly integrating the initial point cloud features with features supplemented and enriched through cross-modal interaction, model learning, so as to more accurately capture and express the essential features and key information in the point cloud data.

[0073] Step 3: Decode the final enhanced feature representation based on the coarse point cloud generation module (CPGM) , to generate a coarse point cloud representing the approximate shape of the completed object ( = 1024).

[0074] like Figure 1 The middle part and Figure 3 As shown, unlike the fully connected decoder that focuses on capturing global geometry or the folding-based decoder that helps to represent local geometry, the embodiment of the present invention initially adopts transposed convolution (deconvolution) to extract the final enhanced feature representation. Decode the features of each point .

[0075] For each point feature , the present invention combines it with repeated Concatenate and then integrate through convolutional layers. Feature representation Connecting with the intermediate feature maps, the feature representation of each point is enriched by merging global shape information with local details. This fusion improves the accuracy and structural integrity of the generated coarse point cloud, thereby promoting more effective feature learning. As a result, the generated point cloud not only preserves the overall shape but also captures detailed information. Furthermore, this feature fusion guides the decoding process, enabling the decoder to simultaneously focus on local features and the broader object structure. This dual attention helps the model generate fine-grained point clouds that are consistent with the overall object shape, ensuring a balanced representation of detail and structure. This approach results in more accurate and well-structured point clouds.

[0076] To increase the number of single-point features used to generate new points, this paper employs a two-branch approach. One branch utilizes NodeShuffle (a module for point cloud upsampling whose core concept is to use a graph convolutional network (GCN) to encode the spatial information of a point's neighborhood to generate new points) to generate new points from the latent space by combining the spatial information provided by adjacent points. The other branch employs a self-attention mechanism to effectively model long-range dependencies between single-point features, capturing the correlations between points to generate new points.

[0077] The advantages of the two branches are combined through element-by-element addition, that is, the new points of the two branches are combined to obtain a newly generated single point, which promotes the integration of local spatial information and global correlation learning, thereby promoting the generation of new point features.

[0078] Subsequently, the newly generated single point and the final enhanced feature representation Connect them and input into MLP (Multi-layer Perceptron) to generate transition coarse point cloud .

[0079] In order to further preserve the structural information of the input point cloud, the transition coarse point cloud With incomplete point cloud Connect them and use the farthest point sampling algorithm to get the final size of Coarse point cloud .

[0080] Step 4: Use the point cloud refinement module to refine the coarse point cloud generated in step 3. Refinement processing is performed, and the fusion of global shape and local structure information is guided by dual-stream features to gradually generate predicted refined point clouds to improve the quality and accuracy of the final completed point cloud.

[0081] like Figure 1 The right part and Figure 4 As shown in Figure 3, the point cloud refinement module consists of four key sub-modules: a global shape prediction sub-module, a local structure optimization sub-module, two stacked DFGF (two-stream feature guided fusion) sub-modules, and an upsampling sub-module.

[0082] The input of the global shape prediction submodule is represented by the enhanced feature and coarse point cloud Composition. Using MLP (Multi-layer Perceptron) from Then the enhanced feature representation is The input is fed into another MLP to reduce the feature dimension and capture the most important feature information. The outputs of these two MLPs are then concatenated to obtain a feature representation representing the global shape. .

[0083] The local structure optimization submodule is based on incomplete point cloud As input, the incomplete point cloud is processed through two layers of EdgeConv (edge ​​convolution) and one layer of FPS (furthest point sampling). Iteratively aggregate and refine the local features to obtain a feature representation suitable for enhancing the local structure .

[0084] Feature Representation and All of them are input into the DFGF submodule to promote the interaction and fusion of global and local information.

[0085] like Figure 4 As shown in Figure (e): First, the present invention adopts the self-attention mechanism to learn the context information in the global shape and local structure feature representation, and the output is represented as and ; Then, group convolution is used to Perform position encoding, and the results before and after convolution are directly added as the query in the cross attention mechanism ( ) is the source, and As the key in the cross attention mechanism ( ) and value ( ). The model uses a cross-attention mechanism to achieve guided fusion of dual-stream features and semantic correlation. By utilizing global shape information as , local shape information as , to achieve guided fusion of dual-stream features. Structural information as and This process guides the model to focus on local information related to the global shape, thereby extracting more relevant and meaningful features from the local structure. Further processing involves By enhancing the cross attention and fusing the local structured features of the global information, we can obtain Finally, self-attention is used again to learn feature representation relationship, and through Further enhance the output to obtain the final global shape feature representation, denoted as .

[0086] The local structural features output by the second DFGF submodule and global shape features Connect them to form a fusion feature representation , which can capture both global shape and local details.

[0087] The upsampling submodule relies on the feature information of global shape and local structure, that is, the fusion feature representation , a set of coordinate offsets are generated through MLP and reshaping operations, and the coordinate offsets are used to Make fine adjustments to get a finer and more accurate complete point cloud , the entire repair process is completed.

[0088] The method of the present invention is different from the traditional shape completion method that directly infers the complete shape from the incomplete input point cloud. The method of the present invention uses the additional modal information provided by the view image to help obtain a more accurate shape of the incomplete point cloud. Specifically, given an incomplete point cloud and any view image As input,

[0089] Leverage information from both modalities to infer missing parts of an incomplete point cloud and generate a complete point cloud It is worth noting that the number of points in the input and output point clouds of the present invention is the same.

[0090] A large number of quantitative and qualitative experiments on benchmark datasets show that the BiMPR-Net model achieves state-of-the-art performance. Figure 5 As shown, the excellence and necessity of the present invention are proved.

[0091] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A bidirectional cross-modal image-guided point cloud restoration method with multi-scale progressive refinement, characterized in that The following steps are involved: Step 1: Get view image I in and incomplete point cloud P in , the incomplete point cloud P in Obtain the depth map through perspective projection transformation; Step 2: Based on a bidirectional interactive compensation module with multi-scale progressive refinement, the depth map is used to compensate for the missing geometric depth information in the view image modality, and the view image modality is used to compensate for the missing texture and color information in the point cloud modality, to obtain the final enhanced feature representation F. Step 3: Decode the final enhanced feature representation F based on the coarse point cloud generation module to generate a coarse point cloud P representing the approximate shape of the completed object. co ; Step 4: Use the point cloud refinement module to refine the coarse point cloud P co Perform refinement processing, guide the fusion of global shape and local structure information through dual-stream features, and gradually generate the predicted refined point cloud P out ; In step 2, the bidirectional interaction compensation module includes a point encoder, a depth encoder, and an image encoder pre-trained on the ImageNet image dataset; the point encoder consists of three layers of set abstraction layers, which extract point cloud features F from point clouds of different levels and scales through hierarchical downsampling. p ; The depth encoder uses the ResNet-18 model to extract the depth map feature F from the depth map d ; The image encoder is used to extract the view image feature F from the view image v ; The bidirectional interaction compensation module includes two cross-modal Transformers; the first cross-modal Transformer is based on the depth map feature F d and view image features F v The semantic association is established between them, and the specific implementation is expressed by the following formula: F tf1 =Concat(F d ,F v ) Q1=α1(F tf1 )+PE(PV),K1=β1(F tf1 )+PE(PV),V1=γ1(F tf1 ) F′ tf1 =LN(F tf1 +SelfAttn(Q1,K1,V1)) F″ tf1 =F′ tf1 +FFN(F′ tf1 ) Among them: F tf1 It means that by d and F v The fused features obtained by connecting them; PE represents position embedding; PV represents projected viewpoint; α1, β1, γ1 represent linear layers; Q1, K1, V1 represent the results after linear transformation, which are used for the operation of the self-attention mechanism; LN represents layer normalization; FFN represents a feedforward network consisting of two linear layers; The second cross-modal Transformer enhances the view feature representation F″ tf1 With point cloud feature F p The semantic association is established between them, and the specific implementation formula is as follows: F tf2 =Concat(F p ,F″ tf1 ) Q2=α2(F tf2 ),K2=β2(F tf2 ),V2=γ2(F tf2 ) <h2 style=";text-align:left;direction:ltr">F′<h2 style=";text-align:left;direction:ltr"> tf2 <h2 style=";text-align:left;direction:ltr"> (LN(F<h2 style=";text-align:left;direction:ltr"> tf2 <h2 style=";text-align:left;direction:ltr"> +SelfAttn(Q2,K2,V2)) Q3=K3=V3=F′ tf2 F″ tf2 =LN(F′ tf2 +SelfAttn(Q3,K3,V3)) F″′ tf2 =F″ tf2 +FFN(F″ tf2 ) Among them: F tf2 Indicates the enhanced view feature F″ tf1 With point cloud feature F p The fusion feature obtained by connection; F″′ tf2 represents the final enhanced point cloud feature representation; F′ tf2 and F″ tf2 They represent the fused features after primary and secondary normalization respectively; α2, β2, γ2 represent linear layers; Q2, K2, V2, Q3, K3, V3 represent the results after linear transformation, which are used for the operation of the self-attention mechanism; FFN represents a feedforward network consisting of two linear layers.

2. The bidirectional cross-modal image-guided point cloud restoration method according to claim 1, characterized in that: In step 1, a set of points is projected onto six orthogonal planes to generate a sparse depth image. For each point in the point cloud, perspective projection is used to derive its 2D coordinates on the image plane. The coordinates are discretized and the depth value of each image position is determined by the weighted average of nearby pixels.

3. The bidirectional cross-modal image-guided point cloud restoration method according to claim 1, characterized in that: Enhanced point cloud feature representation With the initial point cloud feature F p The final enhanced feature representation F is concatenated and used as the input of the coarse point cloud generation module and the point cloud refinement module.

4. The bidirectional cross-modal image-guided point cloud restoration method according to claim 3, characterized in that: In step 3, the coarse point cloud generation module uses transposed convolution to decode the features of each point from the final enhanced feature representation F.

5. The bidirectional cross-modal image-guided point cloud restoration method according to claim 4, characterized in that: The coarse point cloud generation module adopts a two-branch approach. One branch uses the NodeShuffle module to generate new points from the latent space by combining the spatial information provided by adjacent points. The other branch uses the self-attention mechanism to simulate the long-range dependency between single-point features and capture the correlation between points to generate new points.

6. The bidirectional cross-modal image-guided point cloud restoration method according to claim 5, characterized in that: The new points of the two branches are combined by element-by-element addition to obtain a newly generated single point. The newly generated single point is connected with the final enhanced feature representation F and then input into the multi-layer perceptron to generate a transition coarse point cloud P. tc , the transition coarse point cloud P tc With incomplete point cloud P in Connect and use the farthest point sampling algorithm to get the rough point cloud P co .

7. The bidirectional cross-modal image-guided point cloud restoration method according to claim 6, characterized in that: The point cloud refinement module includes a global shape prediction submodule, a local structure optimization submodule, a DFGF submodule and an upsampling submodule; the global shape prediction submodule is used to represent the final enhanced feature F and the coarse point cloud P co Generate a feature representation F that represents the global shape gl ; The local structure optimization submodule is used to optimize the incomplete point cloud P based on edge convolution and farthest point sampling. in Iteratively aggregate and refine the local features of the lo DFGF submodule is used to represent F according to the feature gl and F lo Get the local structural feature F″ lo and global shape feature F″ gl ; The upsampling submodule is used to rely on the local structure feature F″ lo and global shape feature F″ gl The fusion feature representation F″ is generated by MLP and reshaping operations to generate coordinate offsets, and the coordinate offsets are used to represent the coarse point cloud P co Make fine adjustments and finally get the complete point cloud P out .

Images