An explicit geometry and agent token guided cross-modal point cloud completion method

Abstract

The application discloses a kind of explicit geometry and agent token guiding cross-modal point cloud completion method, this method first extracts two-dimensional image feature map and dense three-dimensional point cloud feature, and using camera parameter and depth buffer technology, generate the geometric guide mask containing visibility determination;Subsequently, through the geometric guide cross attention module, under the constraint of physical projection, the cross-modal features of known area are fused, and the original geometric information of the occluded point is retained;Introduce the learning agent token mechanism, so that it actively queries image foreground features to excavate missing visual clues;Finally, the fused features and updated agent tokens are spliced, and the complete three-dimensional point cloud is generated through the decoder.The application effectively solves the misplacement and semantic confusion problem in cross-modal feature fusion by introducing explicit physical constraints and active feature excavation mechanism, and can achieve high-precision, structure-complete point cloud completion under the condition of occlusion and large-area loss.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and 3D data processing, and specifically relates to a cross-modal point cloud completion method guided by explicit geometry and proxy tokens. Background Technology

[0002] Point cloud completion, a fundamental core technology in the field of 3D computer vision, primarily functions to recover complete, continuous, and high-precision 3D geometric shapes from incomplete or sparse local point cloud data acquired by sensors. This technology is widely used in various fields, including autonomous driving environmental perception, robotic grasping operations, augmented reality 3D modeling, reverse engineering of industrial products, and digital reconstruction of cultural relics. In actual operation, when data is collected by LiDAR and various depth cameras used in autonomous driving systems, the point clouds of target objects acquired by sensors often suffer from large-area incompleteness, local sparseness, and topological breaks due to objective physical factors such as limited scanning angles, object occlusion, and the optical reflection characteristics of object surfaces. Directly using these incomplete point clouds for downstream tasks such as obstacle detection, vehicle pose calculation, and robot trajectory planning significantly reduces the accuracy of subsequent perception and operation processes. In extreme cases, it can even lead to perception failures and mechanical operational errors, posing safety hazards. Therefore, achieving high-precision completion of incomplete 3D point clouds is a necessary prerequisite for ensuring the stable and reliable operation of various 3D perception systems.

[0003] Early research solutions for point cloud completion were all single-modal processing schemes, relying solely on the 3D geometric features carried by the incomplete point cloud to complete the shape inference. However, single-modal data itself has inherent information limitations. When the missing area of ​​the point cloud accounts for a large proportion, the target object has a complex structure, and the local geometric details are relatively fine, it is difficult to distinguish the symmetrical structure of the object from the subtle concave and convex textures based solely on 3D coordinate information. The resulting completion models generally suffer from problems such as blurred edges, dense surface noise, distorted topology, and inaccurate shape inference of missing areas, making them unsuitable for application scenarios with high requirements for geometric fidelity, such as autonomous driving and industrial inspection.

[0004] To address the lack of information in single-modal point clouds, the industry has gradually developed view-guided cross-modal point cloud completion technology. This type of method simultaneously inputs a 2D RGB image taken from the same perspective as the incomplete point cloud as auxiliary information to participate in the completion process. 2D images possess a regular pixel grid structure, providing high-resolution texture information, clear object boundaries, and global semantic priors to fill missing point cloud data, effectively assisting the network in reasoning about the geometric structure of the missing 3D regions. However, 2D images are two-dimensional regular discrete pixel sets, while 3D point clouds are three-dimensional unordered discrete point sets. A significant spatial dimensional gap and modal representation differences exist between the two. Achieving accurate spatial alignment between pixels and 3D points, and interference-free feature fusion, has long been a core technical challenge that view-guided cross-modal point cloud completion tasks have struggled to overcome.

[0005] To address this core challenge, existing research has proposed several cross-modal feature fusion schemes, among which ViPC, XMFnet, and EGIInet are representative technologies. ViPC, as an early representative scheme for vision-guided point cloud completion, had its related results published at the 2021 IEEE Conference on Computer Vision and Pattern Recognition. The scheme is divided into three stages: modality conversion, local fineness filtering, and differential refinement generation. In the modality conversion stage, a coarse 3D point cloud is directly reconstructed from a single 2D image. The reconstructed coarse point cloud is then directly stitched and aligned with the original incomplete point cloud. Subsequently, a filter is used to distinguish between the fine regions of the original point cloud and the coarse regions of the image reconstruction. Finally, a dynamic offset predictor is used to perform differential refinement processing on the two types of regions to output a complete 3D point cloud. This scheme establishes a technical framework for 2D image-assisted 3D point cloud completion for the first time. However, the underlying scheme has unavoidable defects. Solving the 3D spatial coordinates from a single 2D image is an ill-conditioned inverse problem at the mathematical level. The process of reconstructing coarse point clouds based on images is prone to introducing a large amount of spatial noise. At the same time, the scheme only uses a simple feature stitching method to complete cross-modal shallow fusion, which cannot achieve accurate spatial registration at the microscale of pixels and 3D points. The noise generated in the previous image reconstruction stage will continue to propagate to the next stage. The final output complete point cloud surface has a large number of spikes and outliers, making it difficult to restore the high-frequency geometric details and local topological structure of complex targets. The edge distortion and blurring problems of the completed model are very prominent.

[0006] To avoid the noise errors introduced by image reconstruction in the ViPC scheme, the industry has proposed the XMFnet scheme. This scheme abandons the intermediate step of explicitly reconstructing 3D point clouds and directly constructs a cross-modal fusion architecture in the network's latent feature space. The core relies on the standard cross-attention mechanism to establish the correlation mapping between image features and point cloud features, thereby achieving complementary fusion of multimodal information in the latent space. Although XMFnet alleviates the problem of noise introduced by image reconstruction to a certain extent, the scheme itself has a core design flaw. The standard cross-attention mechanism does not introduce any physical imaging prior constraints. The network blindly searches the matching relationship between pixels and 3D points globally based solely on the statistical regularity of the training data, completely ignoring the imaging perspective regularity corresponding to the camera's intrinsic and extrinsic parameters and the occlusion relationship in 3D space.

[0007] To address the blind attention matching limitation of XMFnet, the existing technology EGIInet solution has emerged. This solution introduces a channel feature transfer loss based on the Gram matrix. By constraining the similarity between the global statistical distribution of image features and point cloud feature channels, it constructs an implicit soft constraint to indirectly regulate the cross-modal feature interaction process, thereby reducing the probability of irrelevant pixels being incorrectly associated with 3D points. However, this solution only applies soft alignment constraints at the feature distribution level and does not introduce physical rules related to real 3D projection. The constraint strength is weak and it lacks precise spatial positioning capabilities. Essentially, it is still a solution that relies on latent space data to drive optimization and cannot fundamentally solve the feature mismatch problem caused by perspective occlusion.

[0008] A summary of the existing mainstream solutions reveals that current view-guided cross-modal point cloud completion technology still suffers from several inherent limitations that are difficult to overcome. Early fusion solutions relied on ill-conditioned single... Figure 3 3D reconstruction combined with shallow feature stitching easily introduces and amplifies noise. Mainstream implicit attention fusion schemes lack physical constraints that conform to the laws of optical projection, relying entirely on pure data-driven, indiscriminate global search, ignoring physical perspective rules and 3D occlusion relationships, frequently resulting in problems such as feature penetration and semantic confusion. For regions where point clouds are completely missing, existing schemes rely solely on the natural generalization of global features to generate points, lacking an active feature mining mechanism for mismatched foreground regions in the image. This fails to fully utilize visual cues remaining in the image, significantly limiting the performance ceiling of the model in scenarios involving large-area missing samples and fine topological structure restoration. In addition, most existing technologies only use a single 3D reconstruction loss to complete network optimization, lacking explicit supervision methods for cross-modal feature alignment. This makes it difficult to simultaneously ensure the matching accuracy of pixels and 3D points as well as the accuracy of 3D geometric shape restoration. Some schemes also rely on computationally cumbersome and error-prone differentiable rendering modules to achieve image supervision, further increasing the algorithm's runtime overhead. These multiple bottlenecks collectively restrict the practical application of cross-modal point cloud completion technology in high-precision, complex occlusion real-world scenarios such as autonomous driving and industrial 3D reconstruction. Summary of the Invention

[0009] The main objective of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide an explicit geometry and surrogate token-guided cross-modal point cloud completion method. This invention aims to solve the problems of misalignment and semantic confusion in cross-modal feature fusion by introducing physical projection constraints, geometric mask attention, surrogate token inference mechanism and cross-modal contrast alignment strategy. Thus, even when the input data is incomplete or occluded, it can still generate a high-precision, structurally complete and sharp-detailed 3D point cloud model.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] This invention provides a cross-modal point cloud completion method based on explicit geometry and proxy token guidance, comprising the following steps:

[0012] S1. Map heterogeneous two-dimensional image data and three-dimensional point cloud data to a high-dimensional semantic feature space respectively, and output two-dimensional image feature map and dense three-dimensional point cloud feature after purification by masking.

[0013] S2. Using the camera intrinsic and extrinsic parameter matrices, N 3D points are projected onto a 2D plane with the same resolution as the original image. The occlusion and visibility determination are completed by combining the depth buffer array, and the visibility state value is generated. Then, a spatial mapping matrix is ​​constructed through cross-resolution coordinate mapping. Based on the spatial mapping matrix and the visibility state value, a 2D overlay mask and a geometric guide mask are generated.

[0014] S3. Convert the geometrically guided mask into a penalty matrix, introduce the geometrically guided cross-attention module to complete the cross-modal feature fusion of the known visible area, and retain the original geometric features of the occluded 3D points through residual connection;

[0015] S4. Invert the two-dimensional overlay mask and perform a logical intersection operation with the feature-level foreground mask to obtain a missing region index matrix representing the area in the foreground of the image that is not covered by visible three-dimensional points. Based on the missing region index matrix, initialize a learnable proxy token carrying a two-dimensional position code, and use a proxy cross-attention layer to mine visual features corresponding to the missing region from the foreground features of the image to update the proxy token representation.

[0016] S5. The fused features output by the geometry-guided cross-attention module are concatenated with the updated proxy token features to obtain joint features. The joint features are then fed into a feature decoder composed of a multi-layer MLP to decode and output a complete 3D point cloud.

[0017] S6. During the training phase, a total loss function is constructed, which is a weighted sum of geometric reconstruction loss and cross-modal feature alignment loss. The total loss is used to perform end-to-end joint training on the entire network to complete the optimization of model parameters.

[0018] As a preferred technical solution, in step S1, the output of the two-dimensional image feature map is specifically as follows:

[0019] A pre-trained two-dimensional convolutional network is used to extract feature maps from the original image;

[0020] The pre-computed high-resolution image foreground mask is downsampled to the same size as the original image feature map by max pooling to obtain the feature-level foreground mask;

[0021] The Hadamard product is performed between the feature-level foreground mask and the original image feature map, and the activation values ​​of the background region features are reduced to zero to obtain the purified two-dimensional image feature map.

[0022] As a preferred technical solution, the dense 3D point cloud features in step S1 are specifically as follows:

[0023] Using the PointNet++ backbone network, the point cloud is downsampled and multi-scale local geometric features are aggregated through the abstraction layer, and then multi-layer features are upsampled and fused through the feature propagation layer to output dense 3D point cloud features point by point.

[0024] As a preferred technical solution, in step S2, the visibility state value is generated, specifically as follows:

[0025] Based on the two-dimensional plane obtained by projection, the continuous two-dimensional projected coordinates and actual depth values ​​of each three-dimensional point are obtained.

[0026] When multiple 3D points are projected onto the same pixel location, resulting in occlusion competition, the actual depth values ​​of all participating points are extracted, and only the smallest value is retained to construct a depth buffer array.

[0027] The actual depth value of each 3D point is compared with the nearest depth value recorded by the depth buffer array at its projected pixel position to generate a visibility status value. If the projection of the current 3D point is within the effective range of the image, the depth value is valid, and the actual depth value of the current 3D point is not greater than the sum of the corresponding nearest depth value and the preset depth tolerance, then the visibility status value is set to 1, indicating that the point is the foremost point at the projected pixel position and is not occluded; otherwise, the visibility status value is set to 0.

[0028] As a preferred technical solution, the process of generating the two-dimensional overlay mask in step S2 is as follows:

[0029] Divide the original image projection coordinates by the image downsampling step size to convert them to a low-dimensional feature map coordinate system;

[0030] A spatial mapping matrix is ​​constructed based on the scaled coordinates, and the feature map mesh corresponding to each 3D point is marked.

[0031] The mapping matrix is ​​multiplied by the visibility vector and then binarized to obtain a two-dimensional coverage mask. A mask value of 1 indicates that the corresponding feature grid has visible 3D point projections, while a value of 0 indicates that the grid has no effective 3D point coverage.

[0032] As a preferred technical solution, the process of generating the geometric guide mask in step S2 is as follows:

[0033] The spatial mapping matrix is ​​transposed and multiplied element-wise with the visibility vector to generate a binary geometric guidance mask. This geometric guidance mask acts as a physical switch for cross-modal feature interaction. The mask value is 1 only when the 3D point is visible and its projection matches the corresponding image grid, allowing feature interaction between the two. In other cases, the mask value is 0, directly blocking invalid or misaligned cross-modal feature matching, providing a hard physical constraint for subsequent geometric guidance cross-attention.

[0034] As a preferred technical solution, in step S3, the calculation process of the geometry-guided cross-attention module includes:

[0035] In the geometric guide mask, the positions with a value of 0 are assigned a maximum negative value to generate a penalty matrix, and the positions with a value of 1 are set to have a penalty value of 0.

[0036] The query matrix Q is obtained by linear projection of the point cloud features, and the key matrix K and value matrix V are obtained by linear projection of the image features.

[0037] Multiply Q by the transpose of K, divide by the square root of the dimension, and then stack the penalty matrix. After Softmax normalization, the attention weights are obtained. The attention weights are multiplied by V to obtain the cross-modal fusion features.

[0038] The module output features are obtained by adding the dense features of the original point cloud and the residual of the fused features.

[0039] As a preferred technical solution, the calculation formula for generating the missing region index matrix in step S4 is as follows:

[0040] ;

[0041] in, For two-dimensional overlay mask, Foreground mask at the feature level, This represents the element-wise Hadamard product; A grid with a value of 1 represents a topologically missing region that belongs to the foreground of an object and is not covered by visible 3D points.

[0042] As a preferred technical solution, step S4, initializing and updating the proxy token, includes:

[0043] Count the number of grids with a value of 1 in the missing region index matrix, K, and copy the globally shared learnable basic token to obtain K initial tokens;

[0044] Extract the two-dimensional coordinates of each missing grid to generate a position code and overlay it onto the corresponding token;

[0045] Using the initial proxy token as the query vector, the purified two-dimensional image features are used as key and value vectors. The proxy cross-attention calculation is performed in combination with the foreground penalty matrix, and the original token is superimposed to obtain the updated proxy token features.

[0046] As a preferred technical solution, in step S6, the geometric reconstruction loss is constructed using L2 chamfer distance to constrain the spatial geometric fit between the decoder output complete point cloud and the real standard point cloud;

[0047] The cross-modal feature alignment loss is constructed based on the InfoNCE contrastive learning architecture. It uses the visibility identifier of 3D points as weights to bring the similarity between the visible 3D point features and the mesh features of the matching image closer, while rejecting irrelevant background pixel features.

[0048] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0049] (1) This invention generates a geometric guidance matrix that includes depth conflict determination, a two-dimensional overlay mask, and a point-to-pixel admission switch by establishing an explicit physical projection mapping. Unlike existing technologies that mainly rely on learned cross-attention weights to find modal correspondences, this invention introduces an explicit geometric mask generation module in the GMCA module to construct a physical constraint mask with cross-modal coordinate alignment. Combined with the residual protection mechanism in the GMCA module, it achieves accurate absorption of visible texture and lossless preservation of the original geometric features of occluded areas. This "physical prior guidance" approach forcibly cuts off the feature interaction between the occluded point and the background area from the underlying mathematical logic, solving the problem of cross-modal feature contamination.

[0050] (2) This invention innovatively introduces a learnable proxy tokens mechanism for completely missing geometric regions in point clouds. Unlike existing technologies that only "generalize" complete shapes by upsampling and branching features of incomplete point clouds, this invention specifically generates independent tokens with two-dimensional spatial location codes for topological blind areas and uses them as active query vectors to retrieve structural symmetry clues in the global image. This asymmetric interactive design of "locating gaps and completing them specifically" helps the model utilize visual clues in the foreground of the image that are not covered by the existing point cloud, improves the inference and recovery capabilities of large-area missing structures, and reduces topological breaks in the completion results of extremely incomplete samples.

[0051] (3) In the training phase, this invention introduces a cross-modal feature alignment loss based on visibility state values, which ensures that the visible 3D point features and the image mesh features whose projections match remain consistent in the latent feature space, and suppresses mismatches between them and image features in the background region or incorrect locations. Thus, explicit supervision can be applied to the cross-modal feature fusion process without relying on a complex differentiable rendering module, improving the spatial anchoring accuracy of point cloud features and image features, and further enhancing the geometric consistency and detail recovery capability of the completed point cloud. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart of a cross-modal point cloud completion method guided by explicit geometry and proxy tokens according to an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of a feature encoder according to an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of geometrically guided cross-attention in an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of an asymmetric proxy token according to an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram of the decoder according to an embodiment of the present invention. Detailed Implementation

[0058] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0059] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0060] like Figure 1 As shown, the embodiment provides a cross-modal point cloud completion method based on explicit geometry and proxy token guidance. It is a complete technical solution for cross-modal 3D point cloud reconstruction and enhancement based on explicit physical masks, controlled cross-attention, and asymmetric proxy completion, specifically including the following steps:

[0061] S1. Map heterogeneous two-dimensional image data and three-dimensional point cloud data to a high-dimensional semantic feature space respectively, and output two-dimensional image feature map and dense three-dimensional point cloud feature after purification by masking.

[0062] like Figure 2 As shown, step S1 specifically includes:

[0063] S11. Local perception and foreground refinement encoding of two-dimensional image features, specifically:

[0064] The resolution of a single received image is A two-dimensional RGB image is obtained, and image features are extracted using a pre-trained two-dimensional feature extraction network. The cumulative downsampling step size of this feature extraction network is set to r. Due to the step size r, the spatial resolution of the output initial feature map is reduced to [value missing]. (in ).

[0065] To eliminate the interference of background noise on 3D reconstruction, a foreground purification mechanism is introduced. This involves obtaining a pre-computed high-resolution image foreground mask. The feature-level foreground mask is obtained by downsampling it to a size perfectly aligned with the feature map using the max pooling operator. Then, the mask was... The Hadamard product (element-wise multiplication) is performed with the initial image features. This physically forces the feature activation values ​​of the background region to zero, ultimately resulting in a purified two-dimensional image feature map. .

[0066] S12. Geometric extraction and point-by-point encoding of 3D point cloud features, specifically:

[0067] The system synchronously receives an initial 3D residual point cloud containing N points and uses PointNet++ as the backbone network for 3D geometric feature extraction. To fully capture the multi-scale spatial information of the point cloud, the network performs downsampling and feature aggregation on the point cloud through an abstraction layer to extract local geometric features. Subsequently, upsampling and feature fusion are performed through a feature propagation layer to output dense 3D point cloud features point by point. .

[0068] S2. Using the camera's intrinsic and extrinsic parameter matrices, N 3D points are projected onto a 2D plane with the same resolution as the original image. An occlusion and visibility determination are completed using a depth buffer array, generating a visibility state value. Then, a spatial mapping matrix is ​​constructed through cross-resolution coordinate mapping. Based on the spatial mapping matrix and the visibility state value, a 2D overlay mask and a geometric guide mask are generated. This step aims to provide accurate spatial and visibility priors for feature fusion by simulating real physical projection laws and combining the downsampling characteristics of the encoder.

[0069] Further, step S2 includes:

[0070] S21. Depth conflict resolution and visibility determination based on high-resolution physical projection;

[0071] To address the self-occlusion problem of 3D point clouds during projection, camera intrinsic and extrinsic parameter matrices are used to project N 3D points onto a 2D plane with the same resolution as the original image, obtaining continuous 2D projected coordinates for each point. and actual depth value When multiple points projected onto the same pixel location constitute occlusion contention, the depth values ​​of all participating points are extracted, and only the smallest value is retained to construct a depth buffer array. Then, the actual depth of each point is compared with the nearest depth recorded on its projected coordinates to generate a visibility state value. :

[0072]

[0073] A value of 1 indicates that the point is at the very front and is not obscured.

[0074] S22, Two-dimensional overlay mask based on coordinate scaling and spatial mapping The generation process is as follows:

[0075] To determine which meshes on the dimensionality-reduced feature map are covered by 3D points, cross-resolution coordinate mapping must be performed. The original projected coordinates obtained in step S1 are then used... Perform proportional scaling, that is, divide both the horizontal and vertical components by the step size r to obtain the scaled mapping coordinates in the feature map coordinate system. :

[0076] Based on the scaling coordinates, construct a space with a size of spatial mapping matrix If the scaled coordinates of the i-th 3D point fall within the physical boundary represented by the j-th mesh, then the corresponding element value in the matrix is... If it does not fall in, then Subsequently, the system will use this matrix. With visibility column vector Multiply and apply the binarization decision function. The generated size is Two-dimensional overlay mask matrix :

[0077]

[0078] This formula excludes occluded points. If at least one visible point is mapped within the mesh, the output is 1, indicating that the mesh area is covered by visible geometry; otherwise, the output is 0, indicating that the area is blank.

[0079] S23. Geometric Guiding Mask Based on Visibility and Spatial Mapping generate;

[0080] Further spatial mapping matrix Transpose (size becomes) ), and visibility column vector Perform element-wise multiplication with row alignment to generate a geometric guide mask. :

[0081]

[0082] This matrix constitutes a one-to-one physical admission switch: if and only if the i-th point falls into the j-th grid ( And not obscured ( When the switch is set to 1, interaction is allowed; otherwise, interaction is disabled.

[0083] S3. Convert the geometrically guided mask into a penalty matrix, introduce the geometrically guided cross-attention module to complete the cross-modal feature fusion of the known visible area, and retain the original geometric features of the occluded 3D points through residual connection;

[0084] like Figure 3As shown, the geometry-guided cross-attention module is used to achieve controlled interaction between 3D point cloud features and 2D image features under explicit geometric constraints. This module takes the 3D point cloud features obtained in step S1 and the refined 2D image features as input, and uses the geometry-guided mask obtained in step S2 as the physical constraint for cross-modal feature interaction. By converting the geometry-guided mask into a geometry penalty matrix, the module can suppress erroneous attention responses between occluded points, misaligned points, and irrelevant image meshes; simultaneously, it preserves the original geometric structure of the 3D point cloud itself through residual connections.

[0085] Furthermore, step S3 specifically includes:

[0086] S31. Generation of the geometric penalty matrix;

[0087] Based on the geometric guide mask obtained in step S2 Constructing the geometric penalty matrix For valid point-image grid matching positions with a value of 1 in the geometric guide mask, the corresponding penalty value is set to 0; for invalid matching positions with a value of 0 in the geometric guide mask, the corresponding penalty value is set to a maximum negative value to suppress cross-modal feature interactions between occluded points, misaligned points and irrelevant image grids.

[0088] S32, Cross-attention fusion under geometric constraints;

[0089] The 3D point cloud features obtained in step S1 are used as query feature inputs, and a query matrix is ​​obtained through linear mapping. The purified two-dimensional image features are used as key and value features as inputs, and a key matrix is ​​obtained through linear mapping. Sum matrix Then, the query matrix will be used. AND key matrix The similarity results are scaled according to the feature dimension and a geometric penalty matrix is ​​superimposed. The attention weights A are obtained by Softmax normalization, and then the attention weights are used... Log-value matrix Weighted summation is performed to obtain cross-modal fusion features. :

[0090]

[0091]

[0092] in, Represents the feature dimension. Due to the geometric penalty matrix The constraint causes the attention weights corresponding to occluded or misaligned points to approach 0, thereby preventing invalid image features from contaminating the 3D point cloud features.

[0093] S33, Residual connection output;

[0094] To prevent occluded 3D points or surface points facing away from the camera from losing their original geometry, residual connections are introduced. The input 3D point cloud features are added to the cross-modal fusion features to obtain the output features of the geometry-guided cross-attention module.

[0095]

[0096] For visible and projected 3D points, the module can absorb effective visual features from the corresponding image grid; for occluded or non-effectively matched 3D points, the residual connection can preserve their original 3D geometric features because their cross-modal fusion features are suppressed by the geometric penalty matrix.

[0097] S4. After inverting the two-dimensional coverage mask, perform a logical intersection operation with the feature-level foreground mask to obtain a missing region index matrix representing the area in the image foreground that is not covered by visible three-dimensional points; initialize a learnable proxy token carrying a two-dimensional position code based on the missing region index matrix, and use a proxy cross-attention layer to mine visual features corresponding to the missing region from the image foreground features to update the proxy token representation.

[0098] like Figure 4 As shown, asymmetric surrogate token completion aims to address the topological missingness problem of incomplete cloud in its initial state. It utilizes learnable surrogate tokens as spatial placeholders and, through their cross-modal interaction with image foreground features, combines this with the 3D structure generation capabilities of the subsequent decoder to achieve prediction and topological completion of unknown geometric regions. The specific implementation steps are as follows:

[0099] S41. Generation of the missing region index matrix;

[0100] The feature-level foreground mask obtained during the feature encoding stage is called. and the two-dimensional overlay mask obtained during the mask generation stage. The missing region index matrix is ​​generated by inverting the two-dimensional overlay mask and performing element-wise logical intersection operation with the feature-level foreground mask. :

[0101]

[0102] This intersection operation physically eliminates the interference of background noise and rigidly avoids areas already covered by 3D point clouds, thus accurately locking the topological blind zone that "belongs to the foreground of the target object and lacks 3D entity mapping".

[0103] S42, based on Driver-driven learnable token initialization and position encoding;

[0104] generated For hard-space guidance, perform asymmetric token initialization. Statistics The number of grid cells with a median value of 1 is denoted as K. A globally shared, learnable parameter vector is maintained in the network. And replicate it K times. Then, rigorous extraction... For the corresponding two-dimensional spatial grid coordinates j, generate the corresponding two-dimensional spatial position encoding (PE) and inject it into the copied parameters:

[0105]

[0106] The resulting initial proxy token Semantically, these are learnable parameters waiting to be updated, but spatially they have already been... It gives a clear topological gap orientation.

[0107] S43. Proactive cross-modal semantic interaction based on proxy tokens;

[0108] To enable proxy tokens in a blank semantic state to predict 3D topology, a proxy cross-attention layer was constructed.

[0109] In this layer, the initial proxy token As the active query vector, it is linearly projected as Simultaneously, two-dimensional image feature maps As a global knowledge base (Key-Value), it is linearly projected into a key matrix. AND-value matrix At the same time, utilizing Construct the foreground penalty matrix Set the penalty value for the background area to Then, controlled attentional interaction is performed:

[0110]

[0111] Under this interaction mechanism, The missing information determines "where the system initiates queries," while complete image features provide the "answer." The proxy token uses its own missing location encoding as a clue to search for visual patterns and structural symmetry within the global image features. This asymmetric "active query" mechanism endows the proxy token with powerful geometric inference capabilities.

[0112] S5. The fused features output by the geometry-guided cross-attention module are concatenated with the updated proxy token features to obtain joint features. The joint features are then fed into a feature decoder composed of a multi-layer MLP to decode and output a complete 3D point cloud.

[0113] Furthermore, such as Figure 5 As shown, the decoder adopts a multi-layer MLP structure. First, it integrates the output features of the geometry-guided cross-attention module and the asymmetric proxy token completion module. and spliced ​​together ,Will By reducing the feature dimension through a multi-layer MLP, a complete point cloud is finally output.

[0114] S6. During the training phase, a total loss function is constructed, which is a weighted sum of geometric reconstruction loss and cross-modal feature alignment loss. The total loss is used to perform end-to-end joint training on the entire network to complete the optimization of model parameters.

[0115] The network training in this invention is driven end-to-end by a combination of 3D geometric reconstruction loss and cross-modal feature alignment loss. Through dual constraints, the model is ensured to achieve high-dimensional consistency alignment between point cloud features and image features while generating accurate geometric topology.

[0116] (1) Geometric reconstruction loss );

[0117] The loss function aims to constrain the complete 3D point cloud generated by the decoder. Closely matches the actual global target shape The system uses L2 chamfer distance as the metric.

[0118]

[0119] (2) Cross-modal feature alignment loss );

[0120] To explicitly supervise the feature fusion process of the GMCA module, the system introduces a contrastive learning loss based on the InfoNCE architecture within the latent feature space. This loss is calculated using visibility vectors. The guidance forces the 3D point features 2D feature patches at their correct projection locations This achieves semantic alignment while excluding irrelevant background or incorrectly positioned features. Its mathematical expression is as follows:

[0121]

[0122] The total loss function is constructed using a weighted summation method to achieve end-to-end joint optimization:

[0123]

[0124] This invention directly injects camera intrinsic and extrinsic parameters and Z-buffer depth testing as hard physical masks into the cross-attention operator. By forcing the cross-modal feature flow to strictly follow the true optical projection path, this invention breaks the blindness of traditional implicit global search and fundamentally eliminates feature penetration and semantic confusion caused by the lack of physical constraints. The introduction of this explicit physical rule enables the network to achieve absolutely accurate spatial anchoring at the point-pixel microscale, avoiding the noise introduced by early methods that rely on ill-conditioned single-image reconstruction, and significantly improving the feature fusion accuracy of the system in complex occlusion scenarios.

[0125] Meanwhile, to enhance the model's ability to infer and recover large-area missing structures, this invention innovatively introduces an asymmetric proxy token mechanism. For completely missing regions in the point cloud, this mechanism can adaptively and proactively mine missing visual cues from foreground areas of objects not covered by known residual point clouds, changing the passive situation of existing methods that can only rely on global features for natural generalization. Furthermore, this invention combines a visibility-based cross-modal contrastive learning strategy, using the projection relationship between 3D points and corresponding 2D image grids to impose explicit constraints on the feature alignment process. This eliminates the need for 2D reconstruction labels or 2D differentiable rendering modules, thereby reducing the impact of training complexity and rendering errors on model optimization and improving the model's ability to perceive and repair fine topological structures. In summary, the goal of this invention is to overcome the bottlenecks of existing cross-modal point cloud completion technologies in terms of physical rule fusion, missing region inference, and precise feature anchoring. By constructing an integrated completion system with explicit physical constraints, active visual mining, and pure 3D supervised alignment, it achieves higher quality and more physically logical completion effects in typical tasks such as autonomous driving target completion, robot 3D scene perception, and augmented reality (AR), providing an advanced and reliable solution for complex 3D vision generation and repair tasks.

[0126] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A cross-modal point cloud completion method based on explicit geometry and proxy token guidance, characterized in that, Includes the following steps: S1. Map heterogeneous two-dimensional image data and three-dimensional point cloud data to a high-dimensional semantic feature space respectively, and output two-dimensional image feature map and dense three-dimensional point cloud feature after purification by masking. S2. Using the camera intrinsic and extrinsic parameter matrices, project N three-dimensional points onto a two-dimensional plane with the same resolution as the original image. Combine this with a depth buffer array to complete occlusion and visibility determination and generate visibility state values. Then, a spatial mapping matrix is ​​constructed through cross-resolution coordinate mapping, and a two-dimensional overlay mask and a geometric guidance mask are generated based on the spatial mapping matrix and the visibility state value; S3. Convert the geometrically guided mask into a penalty matrix, introduce the geometrically guided cross-attention module to complete the cross-modal feature fusion of the known visible area, and retain the original geometric features of the occluded 3D points through residual connection; S4. Invert the two-dimensional covering mask and perform a logical intersection operation with the feature-level foreground mask to obtain the missing region index matrix representing the area in the foreground of the image that is not covered by visible three-dimensional points. A learnable proxy token carrying a two-dimensional positional code is initialized based on the missing region index matrix. A proxy cross-attention layer is used to mine visual features corresponding to the missing region from the image foreground features and update the proxy token representation. S5. The fused features output by the geometry-guided cross-attention module are concatenated with the updated proxy token features to obtain joint features. The joint features are then fed into a feature decoder composed of a multi-layer MLP to decode and output a complete 3D point cloud. S6. During the training phase, a total loss function is constructed, which is a weighted sum of geometric reconstruction loss and cross-modal feature alignment loss. The total loss is used to perform end-to-end joint training on the entire network to complete the optimization of model parameters.

2. The cross-modal point cloud completion method based on explicit geometry and proxy token guidance according to claim 1, characterized in that, In step S1, the output of the two-dimensional image feature map is specifically as follows: A pre-trained two-dimensional convolutional network is used to extract feature maps from the original image; The pre-computed high-resolution image foreground mask is downsampled to the same size as the original image feature map by max pooling to obtain the feature-level foreground mask; The Hadamard product is performed between the feature-level foreground mask and the original image feature map, and the activation values ​​of the background region features are reduced to zero to obtain the purified two-dimensional image feature map.

3. The cross-modal point cloud completion method based on explicit geometry and proxy token guidance according to claim 1, characterized in that, The dense 3D point cloud features in step S1 are specifically as follows: Using the PointNet++ backbone network, the point cloud is downsampled and multi-scale local geometric features are aggregated through the abstraction layer, and then multi-layer features are upsampled and fused through the feature propagation layer to output dense 3D point cloud features point by point.

4. The cross-modal point cloud completion method based on explicit geometry and proxy token guidance according to claim 1, characterized in that, In step S2, the visibility state value is generated, specifically as follows: Based on the two-dimensional plane obtained by projection, the continuous two-dimensional projected coordinates and actual depth values ​​of each three-dimensional point are obtained. When multiple 3D points are projected onto the same pixel location, resulting in occlusion competition, the actual depth values ​​of all participating points are extracted, and only the smallest value is retained to construct a depth buffer array. The actual depth value of each 3D point is compared with the nearest depth value recorded by the depth buffer array at its projected pixel position to generate a visibility status value. If the projection of the current 3D point is within the effective range of the image, the depth value is valid, and the actual depth value of the current 3D point is not greater than the sum of the corresponding nearest depth value and the preset depth tolerance, then the visibility status value is set to 1, indicating that the point is the foremost point at the projected pixel position and is not occluded; otherwise, the visibility status value is set to 0.

5. The cross-modal point cloud completion method based on explicit geometry and proxy token guidance according to claim 4, characterized in that, In step S2, the process of generating the two-dimensional overlay mask is as follows: Divide the original image projection coordinates by the image downsampling step size to convert them to a low-dimensional feature map coordinate system; A spatial mapping matrix is ​​constructed based on the scaled coordinates, and the feature map mesh corresponding to each 3D point is marked. The mapping matrix is ​​multiplied by the visibility vector and then binarized to obtain a two-dimensional coverage mask. A mask value of 1 indicates that the corresponding feature grid has visible 3D point projections, while a value of 0 indicates that the grid has no effective 3D point coverage.

6. The cross-modal point cloud completion method based on explicit geometry and proxy token guidance according to claim 4, characterized in that, In step S2, the process of generating the geometric guide mask is as follows: The spatial mapping matrix is ​​transposed and multiplied element-wise with the visibility vector to generate a binary geometric guidance mask. This geometric guidance mask acts as a physical switch for cross-modal feature interaction. The mask value is 1 only when the 3D point is visible and its projection matches the corresponding image grid, allowing feature interaction between the two. In other cases, the mask value is 0, directly blocking invalid or misaligned cross-modal feature matching, providing a hard physical constraint for subsequent geometric guidance cross-attention.

7. The cross-modal point cloud completion method based on explicit geometry and proxy token guidance according to claim 1, characterized in that, In step S3, the calculation process of the geometry-guided cross-attention module includes: In the geometric guide mask, the positions with a value of 0 are assigned a maximum negative value to generate a penalty matrix, and the positions with a value of 1 are set to have a penalty value of 0. The query matrix Q is obtained by linear projection of the point cloud features, and the key matrix K and value matrix V are obtained by linear projection of the image features. The attention weights are obtained by transposing Q and K, dividing them by the square root of their dimensions, and then superimposing them with a penalty matrix. After Softmax normalization, the attention weights are obtained. The attention weights are multiplied by V to obtain the cross-modal fusion features. The module output features are obtained by adding the dense features of the original point cloud and the residual of the fused features.

8. The cross-modal point cloud completion method based on explicit geometry and proxy token guidance according to claim 1, characterized in that, Step S4: The formula for calculating the missing region index matrix is ​​as follows: ； in, For two-dimensional overlay mask, Foreground mask at the feature level, This represents the element-wise Hadamard product; A grid with a value of 1 represents a topologically missing region that belongs to the foreground of an object and is not covered by visible 3D points.

9. The cross-modal point cloud completion method based on explicit geometry and proxy token guidance according to claim 1, characterized in that, Step S4, initializing and updating the proxy token, includes the following steps: Count the number of grids with a value of 1 in the missing region index matrix, K, and copy the globally shared learnable basic token to obtain K initial tokens; Extract the two-dimensional coordinates of each missing grid to generate a position code and overlay it onto the corresponding token; Using the initial proxy token as the query vector, the purified two-dimensional image features are used as key and value vectors. The proxy cross-attention calculation is performed in combination with the foreground penalty matrix, and the original token is superimposed to obtain the updated proxy token features.

10. The cross-modal point cloud completion method based on explicit geometry and proxy token guidance according to claim 1, characterized in that, In step S6, the geometric reconstruction loss is constructed using L2 chamfer distance to constrain the spatial geometric fit between the decoder output complete point cloud and the real standard point cloud; The cross-modal feature alignment loss is constructed based on the InfoNCE contrastive learning architecture. It uses the visibility identifier of 3D points as weights to bring the similarity between the visible 3D point features and the mesh features of the matching image closer, while rejecting irrelevant background pixel features.

Images