Point cloud completion method and device based on multi-object guidance and depth order constraint
Patent Information
- Application Number
- CN202611467739.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-21
- Publication Date
- 2026-10-27
AI Technical Summary
[0005]然而，现有三维点云补全方法多聚焦于单个物体的几何特征提取，通常仅依据单一残缺点云推断完整结构，对物体与场景以及多个物体之间的语义联系和空间约束利用不足，容易使补全结果在语义类别、相对布局或物理位置方面与实际场景不一致
[0055]（1）本发明利用多个物体提供的相对位置、视觉相似度等多维度场景上下文特征，为目标物体缺失结构推断提供附加信息，引导模型生成更加符合空间物理规律与语义合理性的物体完整形状；
Smart Images
Figure CN122989453A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of point cloud processing technology, and in particular to a point cloud completion method and apparatus based on multi-object guidance and depth order constraints. Background Technology
[0002] Point clouds are collections of discrete points in three-dimensional space used to represent the geometry of objects. As the raw output of sensing devices such as LiDAR, point clouds are simple in structure and computationally efficient, forming a key data foundation for scientific research in the field of 3D vision and are widely used in autonomous driving, embodied intelligent robots, and other fields. However, sensing devices such as LiDAR are often affected by factors such as limited viewing angle, object occlusion, and insufficient sensor resolution, resulting in point clouds with local missing, sparse, broken, or even large invisible areas. This incompleteness of shape means that the system can only perceive local geometric information, affecting its accurate understanding of the overall structure of 3D targets and posing a fundamental challenge to downstream applications.
[0003] 3D point cloud completion is a research direction proposed to solve this problem. The core goal of 3D point cloud completion is to learn the structural rules of an object from its incomplete point cloud, infer and generate the missing geometric structure, and thus restore an accurate and reasonable complete shape.
[0004] In the real world, the geometric structure of objects is easily influenced by semantic relationships and spatial constraints within a scene. For example, if an object is located above a table, its bottom is usually flat; if it is adjacent to multiple chair-like structures, it is more likely to be a table structure itself; and in a street scene, a missing shape appearing next to a pedestrian point cloud is very likely still a pedestrian. Contextual information such as the symbiotic relationships, relative layouts, and physical constraints between objects in these scenes provides important priors for inferring the missing structure of an object.
[0005] However, existing 3D point cloud completion methods mostly focus on extracting the geometric features of individual objects, typically inferring the complete structure based solely on a single incomplete point cloud. They fail to adequately utilize the semantic relationships and spatial constraints between objects and the scene, as well as between multiple objects, easily leading to inconsistencies between the completion results and the actual scene in terms of semantic category, relative layout, or physical location. A few existing methods that utilize inter-object information typically treat paired objects as processing units and complete the data sequentially according to a predetermined object order. This makes it difficult to simultaneously adapt to multiple objects of variable number and disordered arrangement within the same scene, and also fails to fully integrate the complementary information provided by the point cloud and the image from the same viewpoint. Furthermore, these methods often lack constraints directly derived from the observed geometry, making it difficult to promptly suppress results that do not conform to the observation depth order, such as generating points located before actual visible occlusion surfaces. Therefore, how to jointly utilize the cross-modal scene context of multiple objects within the same frame, without relying on a fixed number of objects and input order, and to constrain the generation of missing structures with real observation depth constraints, is a problem that needs to be solved in 3D point cloud completion. Summary of the Invention
[0006] To address the above issues, this invention proposes a point cloud completion method and apparatus based on multi-object guidance and depth order constraints. The method extracts geometric, semantic, spatial, and textural features of the target object through a cross-modal object representation module and fuses them into object features. An inter-object context feature construction module acquires the inter-object context features of different target objects. A scene context-aware interaction module fuses these object features to obtain a contextual condition vector. This allows the 3D point cloud completion network to no longer rely solely on a single incomplete point cloud but instead obtains prior information on the semantic associations, spatial layouts, and physical relationships of other objects in the same frame, ensuring the completion result is consistent with the actual scene in terms of semantic category, relative layout, and physical location. Multiple target objects are input as an unordered multi-object set without pre-setting the number or input order of objects. Simultaneously utilizing cross-modal information from point clouds and images avoids dependence on object order and fully leverages the complementary information between point clouds and RGB images, improving feature richness. An observation depth order loss is added to the total loss function, directly obtaining a hard constraint on depth order from the actual observed geometry. This promptly suppresses the phenomenon of completion points penetrating visible occluded objects, ensuring the completion result conforms to spatial physical laws and semantic rationality.
[0007] On the one hand, the point cloud completion method based on multi-object guidance and depth order constraints has the following specific steps:
[0008] S1, use a 3D sensor to acquire scene point cloud, use an image sensor to acquire scene RGB, and acquire calibration parameters of the 3D sensor and the image sensor;
[0009] S2, input the scene point cloud into the trained point cloud segmentation model to obtain the object residual point cloud of each target object; input the scene RGB image into the trained image segmentation model to obtain the two-dimensional instance mask and object RGB image of each target object;
[0010] S3, construct and train a 3D point cloud completion network to obtain a trained 3D point cloud completion network; the 3D point cloud completion network includes a cross-modal object representation module, an inter-object context feature construction module, a scene context-aware interaction module, a conditional mapping network, and a shape completion network; the cross-modal object representation module extracts the geometric, semantic, and spatial features of the object's residual point cloud, extracts the texture features of the object's RGB image, and fuses the geometric, semantic, spatial, and texture features into object features; the inter-object context feature construction module obtains the inter-object context features of different target objects; the scene context-aware interaction module fuses the object features based on the inter-object context features to obtain a context condition vector; the conditional mapping network maps the context condition vector into modulation parameters; the shape completion network completes the object's residual point cloud based on the modulation parameters to obtain an initial complete point cloud; the initial complete point cloud and the object's residual point cloud are merged into a final complete point cloud;
[0011] The total loss function of the 3D point cloud completion network includes the observation depth order loss constrained by the before and after occlusion relationship, and the depth order loss is constructed based on calibration parameters and two-dimensional instance masks;
[0012] S4, use the trained 3D point cloud completion network to complete the 3D point cloud.
[0013] Preferably, the shape completion network includes an encoder and a decoder; the encoder includes a first set abstraction layer, a first point Transformer layer, a second set abstraction layer, a second point Transformer layer, and a third set abstraction layer connected in sequence; the decoder includes a first transposed convolution, a first multilayer perceptron, a second multilayer perceptron, a second transposed convolution, a third multilayer perceptron, and a fourth multilayer perceptron.
[0014] The object defect cloud input to the encoder is processed layer by layer to extract features, resulting in global shape features. The global shape features are then subjected to conditional affine modulation to obtain the first modulation feature. The first modulation feature is then decoded by the first transposed convolution and the first multilayer perceptron to obtain a coarse point cloud. The coarse point cloud is concatenated with the global shape features to obtain the first concatenated feature. The first concatenated feature is then processed by the second multilayer perceptron to obtain the first coarse point cloud feature. The first coarse point cloud feature is then processed by the second transposed convolution and the third multilayer perceptron to output upsampled features. These upsampled features are then concatenated with the global shape features to obtain the second concatenated feature. The second concatenated feature is then subjected to conditional affine modulation and decoded by the fourth multilayer perceptron to obtain the initial complete point cloud.
[0015] Preferably, the conditional affine modulation is expressed as:
[0016] ;
[0017] in, Indicates the geometric features after modulation. Indicates the first One object, Indicates the first layer; Indicates the geometric features before modulation; Indicates the scaling parameter; This indicates element-wise multiplication; This represents the bias parameter.
[0018] Preferably, the contextual features between objects include one or more of the following: the center position of the defect clouds of the two objects, the center distance, the relative orientation, the logarithmic scale ratio, the occlusion order, the contact relationship, the support relationship, and the visual representation similarity.
[0019] Preferably, the cross-modal object representation module specifically comprises:
[0020] The object defect cloud is processed using a trained point cloud encoder, and the global geometric feature vector is extracted as geometric features. The object defect cloud is projected onto the RGB image feature plane of the same acquisition viewpoint using calibration parameters. The visual features at the corresponding positions are obtained by bilinear interpolation and pooling to obtain semantic features. The 3D axial bounding box parameters of the object defect cloud are used as spatial features. The object RGB image is processed using a trained visual encoder, and the global texture feature vector is extracted as texture features.
[0021] Preferably, the fusion of object features based on inter-object contextual features to obtain a contextual condition vector is as follows:
[0022] The interaction gating coefficient is calculated based on object characteristics and expressed as follows:
[0023] ;
[0024] in, Indicates the interaction gating coefficient; Representation Object For the target object Context contribution weight; Indicates an interactive gating coefficient encoder; Represents the target object Object characteristics; Representation Object Object characteristics; Represents the contextual characteristics between objects; Indicates feature concatenation operation;
[0025] By aggregating the representations of other objects using interaction gating coefficients, we obtain context-aggregated features, represented as:
[0026] ;
[0027] in, Represents the target object Contextual aggregation representation; Represents an object feature mapping function; This indicates an extremely small positive number that prevents the denominator from being zero;
[0028] By performing a residual concatenation between the context aggregation features and the object features, we obtain the context condition vector, which is represented as follows:
[0029] ;
[0030] in, Represents the context condition vector; Representation layer normalization; This represents a context-aware set interaction encoder.
[0031] Preferably, the total loss function of the 3D point cloud completion network includes contour consistency loss, visible structure consistency loss, and observation depth order loss; the total loss function is expressed as:
[0032] ;
[0033] ;
[0034] in, Represents the total loss function; This represents the basic completion loss of the shape completion network; The weight representing the loss due to observation depth order; This indicates the loss of observation depth order; The weights representing the contour consistency loss; This indicates the loss of contour consistency. The weights representing the visible structural consistency loss; This represents the visible structural consistency loss.
[0035] Preferably, the observation depth order loss is specifically implemented as follows:
[0036] Using the extrinsic and intrinsic parameters from the scene coordinate system to the camera coordinate system, candidate completion points are projected onto an image plane with the same acquisition viewpoint, and the camera depth is obtained.
[0037] ;
[0038] ;
[0039] in, Indicates projected pixels; Represents perspective projection; Indicates camera intrinsic parameters; This represents the extrinsic parameters from the scene coordinate system to the camera coordinate system; Indicates candidate completion points homogeneous coordinates; This indicates the forward depth of the candidate completion point from the camera; Indicates taking Axial components;
[0040] Other objects The actual lidar observation points are projected onto the image plane; when candidate completion points... Projected pixels Located in the object Within a two-dimensional instance mask, and within its neighborhood, there are objects. When determining the actual observation point, the forward depth of the observation point closest to the camera in the neighborhood is taken as the depth of the visible surface of that pixel. Candidate points that meet these conditions constitute a valid set of candidate completion points. Construct the depth-order violation quantity, represented as:
[0041] ;
[0042] in, This indicates a depth order violation. Indicates the depth of the visible surface; Indicates depth interval error; This means that when the value inside the parentheses is greater than zero, the value is itself; when the value inside the parentheses is less than or equal to zero, the value is zero.
[0043] The observation depth order loss is constructed and expressed as:
[0044] ;
[0045] in, This indicates the loss of observation depth order; Represents the set of valid candidate completion points The cardinality of the set.
[0046] Preferably, the visible structural consistency loss is expressed as:
[0047] ;
[0048] in, Clouds representing the defects of an object The point in the middle; Indicates the prediction of the complete point cloud The point in the middle; The number of points representing the defects of an object; This indicates taking the minimum value; express distance.
[0049] On the other hand, the point cloud completion device based on multi-object guidance and depth order constraints includes the following:
[0050] The data acquisition module is used to acquire scene point clouds using a 3D sensor, acquire scene RGB using an image sensor, and acquire calibration parameters of the 3D sensor and the image sensor.
[0051] The data processing module is used to input the scene point cloud into the trained point cloud segmentation model to obtain the object residual point cloud of each target object; and to input the scene RGB image into the trained image segmentation model to obtain the two-dimensional instance mask and object RGB image of each target object.
[0052] A network construction and training module is used to construct and train a 3D point cloud completion network to obtain a trained 3D point cloud completion network. The 3D point cloud completion network includes a cross-modal object representation module, an inter-object context feature construction module, a scene context-aware interaction module, a conditional mapping network, and a shape completion network. The cross-modal object representation module extracts the geometric, semantic, and spatial features of the object's residual point cloud, extracts the texture features of the object's RGB image, and fuses the geometric, semantic, spatial, and texture features into object features. The inter-object context feature construction module obtains the inter-object context features of different target objects. The scene context-aware interaction module fuses object features based on the inter-object context features to obtain a context condition vector. The conditional mapping network maps the context condition vector into modulation parameters. The shape completion network completes the object's residual point cloud based on the modulation parameters to obtain an initial complete point cloud. The initial complete point cloud and the object's residual point cloud are then merged into a final complete point cloud.
[0053] The total loss function of the 3D point cloud completion network includes the observation depth order loss constrained by the front and back occlusion relationship, and the depth order loss is constructed based on calibration parameters and two-dimensional instance masks; the 3D point cloud completion module is used to perform 3D point cloud completion using the trained 3D point cloud completion network.
[0054] Compared with the prior art, the present invention has the following beneficial effects:
[0055] (1) This invention utilizes the multi-dimensional scene context features such as relative position and visual similarity provided by multiple objects to provide additional information for inferring the missing structure of the target object, and guides the model to generate a more complete shape of the object that conforms to the laws of spatial physics and semantic rationality.
[0056] (2) The present invention uses a scene context-aware interactive encoder to process a variable number of unordered object sets, and adaptively generates interactive gating based on the target object, other objects and their context features, while maintaining the object arrangement and permutation variability and suppressing interference from irrelevant objects;
[0057] (3) The present invention uses the actual three-dimensional sensor observation depth of other objects as a basis to directly form the lower bound of the effective candidate completion points, so that the candidate completion points are consistent with the front and back order of the real visible surfaces in the scene, and at the same time reduce the noise points in the completion results;
[0058] (4) The training process of the present invention does not require complete point cloud supervision, which alleviates the problem of insufficient three-dimensional complete shape data, gets rid of the dependence on real complete point cloud, and reduces the difficulty of data acquisition. Attached Figure Description
[0059] The present invention will now be described in further detail with reference to the accompanying drawings;
[0060] Figure 1 This is a step diagram of the point cloud completion method based on multi-object guidance and depth order constraints according to an embodiment of the present invention;
[0061] Figure 2 This is a flowchart illustrating the point cloud completion method based on multi-object guidance and depth order constraints according to an embodiment of the present invention.
[0062] Figure 3 This is a shape completion network structure diagram of the point cloud completion method based on multi-object guidance and depth order constraints according to an embodiment of the present invention.
[0063] Figure 4 This is a schematic diagram of the pickup truck completion result of the point cloud completion method based on multi-object guidance and depth order constraint according to an embodiment of the present invention; wherein, the left image is the incomplete point cloud; the middle image is the completion result from the first view; and the right image is the completion result from the second view.
[0064] Figure 5 This is a schematic diagram of the car completion result of the point cloud completion method based on multi-object guidance and depth order constraint according to an embodiment of the present invention; wherein, the left image is the residual point cloud; the middle image is the completion result from the first view; and the right image is the completion result from the second view.
[0065] Figure 6 This is a schematic diagram of the SUV completion result of the point cloud completion method based on multi-object guidance and depth order constraints according to an embodiment of the present invention; wherein, the left image is the residual point cloud; the middle image is the completion result from the first view; and the right image is the completion result from the second view.
[0066] Figure 7 This is a structural block diagram of a point cloud completion device based on multi-object guidance and depth order constraints according to an embodiment of the present invention. Detailed Implementation
[0067] The present invention will be further described below through specific embodiments.
[0068] like Figure 1 and Figure 2 As shown, the point cloud completion method based on multi-object guidance and depth order constraints has the following specific steps:
[0069] S1: Use a 3D sensor to acquire scene point clouds, use an image sensor to acquire scene RGB, and acquire calibration parameters for the 3D sensor and the image sensor.
[0070] S2, input the scene point cloud into the trained point cloud segmentation model to obtain the object residual point cloud of each target object; input the scene RGB image into the trained image segmentation model to obtain the two-dimensional instance mask and object RGB image of each target object.
[0071] S3, construct and train a 3D point cloud completion network to obtain a trained 3D point cloud completion network.
[0072] S4, use the trained 3D point cloud completion network to complete the 3D point cloud.
[0073] In this embodiment, scene point clouds, scene RGB images, and camera and LiDAR calibration parameters from the KITTI dataset are used. Scene point clouds are read frame by frame. Scene RGB image Camera internal parameters And the calibration extrinsic parameters of lidar to the camera coordinate system Taking a vehicle target as an example, the scene point cloud is... Input the trained 3D instance segmentation model Point-SAM, extract Defect cloud of a target object The scene image is input into the trained 2D instance segmentation model SAM3 to obtain the 2D instance mask and RGB image region of the corresponding target object. Centering and scale normalization are performed on each target defect cloud, and the inverse transform is saved. The object defect cloud is then upsampled or downsampled to obtain a result containing... The defect cloud of each point; by scaling the RGB image, we obtain... Image size. In this embodiment, the residual cloud is sampled at 2048 points, and the target object image is scaled to 224×224 pixels.
[0074] A 3D point cloud completion network was constructed and trained to obtain a trained 3D point cloud completion model. The 3D point cloud completion model includes a cross-modal object representation module, an inter-object context feature construction module, a scene context-aware interactive encoding module, a conditional mapping network, and a shape completion network. The loss functions used in the training process of the 3D point cloud completion model include observation depth order loss, contour consistency loss, and visible structure consistency loss.
[0075] Cross-modal object representation module. Treats each target object in the scene as a unit of object representation within an unordered set of multiple objects. For target objects... Construct object characteristics The object characteristics consist of the following information:
[0076] ;
[0077] in, Represents the object representation encoder; Indicates feature concatenation operation; The geometric features are represented by the following method: by using a trained point cloud encoder to process the object's residual point cloud, a global geometric feature vector is extracted as the geometric features. The texture features are represented by the following method: by processing the RGB image of the object with a trained visual encoder, a global texture feature vector is extracted as the texture features. The semantic features are obtained by projecting the point cloud of the target object onto the RGB image feature plane of the same acquisition viewpoint using calibration parameters, and then extracting the features at the corresponding positions from the texture features through bilinear interpolation and performing pooling. For spatial features, it is represented as the 3D axial bounding box (AABB) parameters of the object's defect cloud, including center coordinates and dimensions.
[0078] In a specific embodiment, the object representation encoder uses two fully connected layers. The input dimension of the first fully connected layer is equal to the total dimension of the spliced features, and the output dimension is 1024. Then, it passes through the LayerNorm layer, the GELU activation layer, the Dropout layer, and the second fully connected layer in sequence to output 512-dimensional object features. Extracted using a trained PointNet++ encoder; Extracted using a DINOv2 encoder.
[0079] Inter-object context feature building module. For the target object Other objects Contextual features between objects are constructed based on defect cloud information and visual representation in a unified scene coordinate system. The features consist of one or more of the following: center distance, relative orientation, observation scale ratio, contact relationship, support relationship, and visual representation similarity. One specific implementation is as follows:
[0080] ;
[0081] in, Representation Object The observation center relative to the object The distance between observation centers; This indicates the occlusion order obtained from the two-dimensional instance mask and the corresponding point cloud observation depth; This represents the cosine similarity between texture features of objects. The features are calculated for all target objects to form a set of contextual features between objects. .
[0082] Scene context-aware interaction coding module. For the target object. Other objects The interaction gating coefficient is calculated based on the characteristics of the target object, the characteristics of other objects, and the contextual characteristics between objects.
[0083] ;
[0084] in, Representation Object For the target object Context contribution weight; Represents the Sigmoid function; The interaction gating coefficient encoder, in this embodiment, employs a three-layer multilayer perceptron. The input is a concatenated vector of target object features, other object features, and inter-object context features. The two hidden layers have dimensions of 256 and 64 respectively, and GELU activation is used in the hidden layers. A Dropout layer is applied after the first hidden layer, outputting a one-dimensional gating value. Subsequently, the representations of other objects are weighted and aggregated based on the interaction gating coefficient.
[0085] ;
[0086] in, Represents the target object Contextual aggregation representation; In this embodiment, the object feature mapping function is represented using two fully connected layers. Pick This is used to prevent the denominator from being zero.
[0087] Furthermore, residual fusion is performed between the target object's own representation and the context aggregation representation to obtain the scene context condition vector. :
[0088] ;
[0089] in, Representation layer normalization; The context-aware ensemble interactive encoder, in this embodiment, employs a two-layer multilayer perceptron. It concatenates the 512-dimensional features of the target object itself with a 512-dimensional aggregated context representation, then sequentially passes this concatenation through a fully connected layer, a GELU activation layer, a Dropout layer, and another fully connected layer, outputting a 512-dimensional fused feature. All target objects share the parameters of the scene context-aware interactive encoder.
[0090] target object The flawed cloud Input shape completion network encoder to obtain the first Layer geometric features Using conditional mapping networks scene context condition vector Convert to modulation parameters for the corresponding level:
[0091] ;
[0092] In this embodiment, the conditional mapping network A two-layer multilayer perceptron is used. First, the 512-dimensional scene context condition vector is mapped to a 256-dimensional hidden feature. After GELU activation, the output vector has a dimension twice the number of geometric feature channels of the corresponding layer, and is split into scaling parameters and bias parameters along the channel dimension.
[0093] Subsequently, conditional affine modulation is applied to the intermediate geometric features in the shape completion network:
[0094] ;
[0095] in, and These represent the geometric features before and after modulation, respectively; and They represent the first The scaling and bias parameters of the layer. The modulation can be used in the encoding stage, the decoding stage, or at least one of these stages, so that scene context information continues to act from the global shape layer to the local geometry generation.
[0096] The shape completion network consists of an encoder and a decoder. Figure 3 The structure of the shape completion network is illustrated. The encoder consists of three ensemble abstraction layers and two point Transformer layers, outputting global shape features. In the decoder, the global shape features are conditionally affine modulated using modulation parameters to obtain the first modulated feature. The first modulated feature is decoded into a coarse point cloud containing 512 points using a first transposed convolutional layer and a first multilayer perceptron. The global feature vector is copied 512 times and concatenated with the coarse point cloud to obtain the first concatenated feature, which is then processed by a second multilayer perceptron to obtain the first coarse point cloud feature. The second transposed convolutional layer and a third multilayer perceptron are used to upsample the data by 16 times, and the data is concatenated with the global feature vector copied 8192 times to obtain the second concatenated feature. The second concatenated feature is conditionally affine modulated, and finally decoded by a fourth multilayer perceptron to obtain the initial complete point cloud containing 8192 points. .
[0097] The remaining cloud The points in the set are used as observation anchor points. The initial complete point cloud output from the shape completion network. Remove points that overlap with or are close to the observation anchor point, and sample from the remaining points. The set of candidate completion points is obtained from each point. ,in , This represents the preset number of points in the final complete point cloud. In this embodiment, The observation anchor points and candidate completion points are merged to form a complete point cloud. .
[0098] ;
[0099] Using extrinsic parameters from scene coordinate system to camera coordinate system and camera internal reference Complete the candidate points Project the image onto the same acquisition viewpoint and obtain the camera depth:
[0100] ;
[0101] in, Candidate completion points homogeneous coordinates; Represents perspective projection; Indicates projected pixels; This represents the forward depth of the candidate completion point from the camera; the smaller the value, the closer the position is to the camera.
[0102] For other objects The actual lidar observation points are projected onto the image plane according to the same calibration relationship. When candidate completion points... Projected pixels Located in the object Within a two-dimensional instance mask, and within a preset neighborhood of that pixel, there are objects belonging to the object. When determining the actual LiDAR observation points, the forward depth of the observation point closest to the camera is taken, representing the depth of the visible object surface displayed by that pixel, and denoted as... Candidate completion points that meet the above conditions constitute the set of valid candidate completion points. .
[0103] Due to the object The 2D instance mask is obtained by segmenting its RGB image, and the actual visible object is the pixel area covered by the mask. Therefore, if the target object If the candidate completion point is projected onto the mask region, then that point should be... Occlusion, meaning: the depth of a valid candidate completion point in the region should be greater than that of the object. The surface depth, from which the depth order violation is defined:
[0104] ;
[0105] in, This is for depth interval error, used to accommodate sensor noise and calibration error; This indicates the value inside the parentheses. When it is greater than zero, the value is taken as this value. The value is zero when the value within the parentheses is less than or equal to zero. When the candidate completion points satisfy the stated depth order, When the candidate completion point is located before or at a small interval from the actual observed surface, .
[0106] The average of the squared depth order violations of all valid candidate completion points yields the observed depth order loss:
[0107] ;
[0108] in, Represents the target object The number of valid candidate completion points; when the total number of valid candidate completion points in the training batch is zero. Take zero.
[0109] The total loss of the unified network is:
[0110] ;
[0111] in, The basic completion loss for the shape completion network includes contour consistency loss. Consistency loss with visible structure :
[0112] ;
[0113] in, The contour consistency loss is calculated as follows: using camera intrinsic and extrinsic parameters, the predicted initial complete point cloud is rendered into a predicted contour map through differentiable projection, and the pixel-level difference between the predicted contour map and the corresponding 2D instance mask is calculated. loss.
[0114] To constrain the initial complete point cloud to contain and preserve the visible structure of the original residual point cloud, a unidirectional chamfer distance from the residual point cloud to the initial complete point cloud is used:
[0115] ;
[0116] in, and They represent the incomplete clouds respectively. With prediction of complete point cloud The point in the middle, This represents the number of points in the residual cloud. This indicates taking the minimum value. express distance.
[0117] The gradient of the total loss simultaneously updates the cross-modal object representation module, the inter-object context feature construction module, the scene context-aware interaction encoding module, and the shape completion network. The trained network outputs a complete point cloud through a single forward computation during the inference phase.
[0118] This embodiment uses complete scene frames as batch processing units, allowing objects within the same frame to participate in scene context encoding and observation depth order constraints. The unified network employs the AdamW optimizer, with weight decay at a certain value. Each batch of input consists of two complete scene frames. If the number of targets in each frame exceeds eight, eight targets are randomly selected. The initial learning rate for the scene context-related modules is set to... Shape completion network The first 10 epochs It will increase linearly from 0 to 1, and then remain at 1. and All values are set to 1. A total of 200 epochs are trained. Figure 4 , Figure 5 and Figure 6 The visualization results of point cloud completion on different vehicle targets in this embodiment are shown.
[0119] like Figure 7 As shown, the present invention also discloses a point cloud completion device based on multi-object guidance and depth order constraints, comprising:
[0120] The data acquisition module 701 is used to acquire scene point clouds using a 3D sensor, acquire scene RGB using an image sensor, and acquire calibration parameters of the 3D sensor and the image sensor.
[0121] The data processing module 702 is used to input the scene point cloud into a trained point cloud segmentation model to obtain the object residual point cloud of each target object; and to input the scene RGB image into a trained image segmentation model to obtain the two-dimensional instance mask and object RGB image of each target object.
[0122] The network construction and training module 703 is used to construct and train a 3D point cloud completion network to obtain a trained 3D point cloud completion network. The 3D point cloud completion network includes a cross-modal object representation module, an inter-object context feature construction module, a scene context-aware interaction module, a conditional mapping network, and a shape completion network. The cross-modal object representation module extracts the geometric, semantic, and spatial features of the object's residual point cloud, extracts the texture features of the object's RGB image, and fuses the geometric, semantic, spatial, and texture features into object features. The inter-object context feature construction module obtains the inter-object context features of different target objects. The scene context-aware interaction module fuses object features based on the inter-object context features to obtain a context condition vector. The conditional mapping network maps the context condition vector into modulation parameters. The shape completion network completes the object's residual point cloud based on the modulation parameters to obtain an initial complete point cloud. The initial complete point cloud and the object's residual point cloud are then merged into a final complete point cloud.
[0123] The total loss function of the 3D point cloud completion network includes the observation depth order loss constrained by the before and after occlusion relationship, and the depth order loss is constructed based on calibration parameters and two-dimensional instance masks;
[0124] The 3D point cloud completion module 704 is used to complete 3D point clouds using a trained 3D point cloud completion network.
[0125] The specific implementation of the point cloud completion device based on multi-object guidance and depth order constraints is the same as that of the point cloud completion method based on multi-object guidance and depth order constraints, and will not be described again in this embodiment.
[0126] The above are merely specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial modifications made to the present invention using this concept shall be considered as infringing upon the protection scope of the present invention.
Claims
1. A point cloud completion method based on multi-object guidance and depth order constraints, characterized in that, Includes the following steps: S1, use a 3D sensor to acquire scene point cloud, use an image sensor to acquire scene RGB, and acquire calibration parameters of the 3D sensor and the image sensor; S2, input the scene point cloud into the trained point cloud segmentation model to obtain the object residual point cloud of each target object; The scene RGB image is input into the trained image segmentation model to obtain the two-dimensional instance mask and object RGB image of each target object; S3, construct and train a 3D point cloud completion network to obtain a trained 3D point cloud completion network; the 3D point cloud completion network includes a cross-modal object representation module, an inter-object context feature construction module, a scene context-aware interaction module, a conditional mapping network, and a shape completion network; the cross-modal object representation module extracts the geometric, semantic, and spatial features of the object's residual point cloud, extracts the texture features of the object's RGB image, and fuses the geometric, semantic, spatial, and texture features into object features; the inter-object context feature construction module obtains the inter-object context features of different target objects; The scene context-aware interaction module fuses object features based on contextual features between objects to obtain a contextual condition vector; The conditional mapping network maps the context conditional vector into modulation parameters; the shape completion network completes the incomplete point cloud of the object based on the modulation parameters to obtain the initial complete point cloud. The initial complete point cloud and the object's residual point cloud are merged into the final complete point cloud. The total loss function of the 3D point cloud completion network includes the observation depth order loss constrained by the before and after occlusion relationship, and the depth order loss is constructed based on calibration parameters and two-dimensional instance masks; S4, use the trained 3D point cloud completion network to complete the 3D point cloud.
2. The point cloud completion method based on multi-object guidance and depth order constraints according to claim 1, characterized in that, The shape completion network includes an encoder and a decoder; the encoder includes a first set abstraction layer, a first point Transformer layer, a second set abstraction layer, a second point Transformer layer, and a third set abstraction layer connected in sequence; the decoder includes a first transposed convolution, a first multilayer perceptron, a second multilayer perceptron, a second transposed convolution, a third multilayer perceptron, and a fourth multilayer perceptron. The object defect cloud input to the encoder is processed layer by layer by the encoder to extract features and obtain global shape features; The global shape features are conditionally affine modulated to obtain the first modulated feature. The first modulated feature is then decoded by the first transposed convolution and the first multilayer perceptron to obtain the coarse point cloud. The coarse point cloud is concatenated with the global shape features to obtain the first concatenated feature. The first concatenated feature is then processed by the second multilayer perceptron to obtain the first coarse point cloud feature. The first coarse point cloud feature is then processed by the second transposed convolution and the third multilayer perceptron to output the upsampled feature. This upsampled feature is then concatenated with the global shape features to obtain the second concatenated feature. The second concatenated feature is then conditionally affine modulated and decoded by the fourth multilayer perceptron to obtain the initial complete point cloud.
3. The point cloud completion method based on multi-object guidance and depth order constraints according to claim 2, characterized in that, The conditional affine modulation is expressed as: ； in, Indicates the geometric features after modulation. Indicates the first One object, Indicates the first layer; Indicates the geometric features before modulation; Indicates the scaling parameter; This indicates element-wise multiplication; This represents the bias parameter.
4. The point cloud completion method based on multi-object guidance and depth order constraints according to claim 1, characterized in that, The contextual features between objects include one or more of the following: the center position of the defect cloud of the two objects, the center distance, the relative orientation, the logarithmic scale ratio, the occlusion order, the contact relationship, the support relationship, and the visual representation similarity.
5. The point cloud completion method based on multi-object guidance and depth order constraints according to claim 1, characterized in that, The cross-modal object representation module is specifically as follows: The object defect cloud is processed using a trained point cloud encoder, and the global geometric feature vector is extracted as geometric features. The object defect cloud is projected onto the RGB image feature plane of the same acquisition viewpoint using calibration parameters. The visual features at the corresponding positions are obtained by bilinear interpolation and pooling to obtain semantic features. The 3D axial bounding box parameters of the object defect cloud are used as spatial features. The object RGB image is processed using a trained visual encoder, and the global texture feature vector is extracted as texture features.
6. The point cloud completion method based on multi-object guidance and depth order constraints according to claim 1, characterized in that, The process of fusing object features based on inter-object contextual features to obtain a contextual condition vector is as follows: The interaction gating coefficient is calculated based on object characteristics and expressed as follows: ； in, Indicates the interaction gating coefficient; Representation Object For the target object Context contribution weight; Indicates an interactive gating coefficient encoder; Represents the target object Object characteristics; Representation Object Object characteristics; Represents the contextual characteristics between objects; Indicates feature concatenation operation; By aggregating the representations of other objects using interaction gating coefficients, we obtain context-aggregated features, represented as: ； in, Represents the target object Contextual aggregation representation; Represents an object feature mapping function; This indicates an extremely small positive number that prevents the denominator from being zero; By performing a residual concatenation between the context aggregation features and the object features, we obtain the context condition vector, which is represented as follows: ； in, Represents the context condition vector; Representation layer normalization; This represents a context-aware set interaction encoder.
7. The point cloud completion method based on multi-object guidance and depth order constraints according to claim 1, characterized in that, The total loss function of the 3D point cloud completion network includes contour consistency loss, visible structure consistency loss, and observation depth order loss; the total loss function is expressed as follows: ； ； in, Represents the total loss function; This represents the basic completion loss of the shape completion network; The weight representing the loss due to observation depth order; This indicates the loss of observation depth order; The weights representing the contour consistency loss; This indicates the loss of contour consistency; The weights representing the visible structural consistency loss; This represents the visible structural consistency loss.
8. The point cloud completion method based on multi-object guidance and depth order constraints according to claim 1, characterized in that, The observation depth order loss is specifically implemented as follows: Using the extrinsic and intrinsic parameters from the scene coordinate system to the camera coordinate system, candidate completion points are projected onto an image plane with the same acquisition viewpoint, and the camera depth is obtained. ； ； in, Indicates projected pixels; Represents perspective projection; Indicates camera intrinsic parameters; This represents the extrinsic parameters from the scene coordinate system to the camera coordinate system; Indicates candidate completion points homogeneous coordinates; This indicates the forward depth of the candidate completion point from the camera; Indicates taking Axial components; Other objects The actual lidar observation points are projected onto the image plane; when candidate completion points... Projected pixels Located in the object Within a two-dimensional instance mask, and within its neighborhood, there are objects. When determining the actual observation point, the forward depth of the observation point closest to the camera in the neighborhood is taken as the depth of the visible surface of that pixel. Candidate points that meet these conditions constitute a valid set of candidate completion points. Construct the depth-order violation quantity, represented as: ； in, This indicates a depth order violation. Indicates the depth of the visible surface; Indicates depth interval error; This means that when the value inside the parentheses is greater than zero, the value is itself; when the value inside the parentheses is less than or equal to zero, the value is zero. The observation depth order loss is constructed and expressed as: ； in, This indicates the loss of observation depth order; Represents the set of valid candidate completion points The cardinality of the set.
9. The point cloud completion method based on multi-object guidance and depth order constraints according to claim 7, characterized in that, The visible structure consistency loss is expressed as: ； in, Clouds representing the defects of an object The point in the middle; Indicates the prediction of the complete point cloud The point in the middle; The number of points representing the defects of an object; This indicates taking the minimum value; express distance.
10. A point cloud completion device based on multi-object guidance and depth order constraints, characterized in that, Including the following: The data acquisition module is used to acquire scene point clouds using a 3D sensor, acquire scene RGB using an image sensor, and acquire calibration parameters of the 3D sensor and the image sensor. The data processing module is used to input the scene point cloud into the trained point cloud segmentation model to obtain the object residual point cloud of each target object. The scene RGB image is input into the trained image segmentation model to obtain the two-dimensional instance mask and object RGB image of each target object; The network construction and training module is used to construct and train a 3D point cloud completion network to obtain a trained 3D point cloud completion network. The 3D point cloud completion network includes a cross-modal object representation module, an inter-object context feature construction module, a scene context-aware interaction module, a conditional mapping network, and a shape completion network. The cross-modal object representation module extracts the geometric, semantic, and spatial features of the object's residual point cloud, extracts the texture features of the object's RGB image, and fuses the geometric, semantic, spatial, and texture features into object features. The inter-object context feature construction module obtains the inter-object context features of different target objects. The scene context-aware interaction module fuses object features based on contextual features between objects to obtain a contextual condition vector; The conditional mapping network maps the context conditional vector into modulation parameters; the shape completion network completes the incomplete point cloud of the object based on the modulation parameters to obtain the initial complete point cloud. The initial complete point cloud and the object's residual point cloud are merged into the final complete point cloud. The total loss function of the 3D point cloud completion network includes the observation depth order loss constrained by the front and back occlusion relationship, and the depth order loss is constructed based on calibration parameters and two-dimensional instance masks; the 3D point cloud completion module is used to perform 3D point cloud completion using the trained 3D point cloud completion network.