A diversity shape completion method based on component analysis and decomposition

Abstract

The application relates to the field of point cloud generation and discloses a diversity shape completion method based on component analysis and decomposition, which comprises the following steps: representing an object as an abstract template structure, predicting and matching template structure parameters of a defective point cloud through a deep learning network, and deducing a complete structure representation of the object; performing component decomposition on the object, obtaining projection details of internal points of each component on a plane, inputting the projection details into a network, and generating solid details of each component; performing three-view completion, combining the details of symmetrical components, deducing complete component details of the object, comparing the three views of the defective and complete objects to identify a missing area, judging whether diversity generation is needed, masking the area needing diversity generation in the complete detail map, performing image-to-image diffusion generation, and obtaining diversified complete detail three views; reconstructing a three-dimensional area according to the three views, obtaining a corresponding signed distance field, and obtaining a watertight completion result through a marching cubes algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of point cloud generation, and particularly relates to a diversity shape completion method based on component analysis and decomposition. BACKGROUND

[0002] The background provided in this section is merely for the purpose of enhancing the understanding of the present disclosure, and does not necessarily constitute prior art.

[0003] Point cloud data is three-dimensional spatial data obtained by sensors such as laser scanning and depth cameras, which can accurately describe the geometric shape of an object or environment. Compared with traditional two-dimensional image data, point cloud not only provides the position of the object surface, but also has rich spatial information, and is widely used in automatic driving, robot perception, building modeling, AR / VR, digital twin and other fields. However, in real-world application scenarios, incomplete point clouds will inevitably be obtained, which will lead to ineffective use of point cloud data. Therefore, how to recover complete three-dimensional information from these incomplete point cloud data is a major challenge in computer vision and computer graphics at present.

[0004] Existing shape completion methods use incomplete signed distance fields or occupancy voxels as input to predict the complete signed distance field of the object. However, in actual scenarios, it is also unrealistic to obtain partial signed distance fields from partial point clouds. At the same time, it is extremely important to realize the diversity prediction of the missing area, and the existing methods often fail to preserve the input shape, resulting in poor robustness when applied to real scanned objects. SUMMARY

[0005] The present application discloses a diversity shape completion method based on component analysis and decomposition, which aims to outline the details inside the structure through three-view sketching after component decomposition, complete the missing details in the form of two-dimensional image generation, and recover the three-dimensional object from the complete three-view. The proposed technology reduces the granularity of generating completion through component structure analysis and decomposition, and realizes the clear division of existing and missing areas, which can realize the diversity completion of three-dimensional objects on the basis of preserving the input shape.

[0006] Technical scheme: A diversity shape completion method based on component analysis and decomposition, comprising the following steps:

[0007] Step 1: representing the object as an abstract template structure, the object being incomplete input data represented by three-dimensional point cloud, predicting and matching the template structure parameters of the incomplete point cloud through a deep learning network, and deriving the complete structure representation of the object according to the template structure parameters.

[0008] Step 2: Based on the complete structure representation of the object obtained in step 1, the object is decomposed into parts, and the projection details of the internal points of each part on three orthogonal planes are obtained. The projection details are input into the network to generate the solid details of each part.

[0009] Step 3: Three-view completion is performed, and the detailed information of the symmetric parts is combined to derive and complete the complete part details of the object.

[0010] Step 4: By comparing the incomplete and complete three views obtained in step 3, the missing areas are distinguished and identified, and it is determined whether diversity generation is needed according to the missing areas.

[0011] Step 5: In the complete detail map, the area that needs diversity generation is masked, image-to-image diffusion generation is performed, and diversified complete detail three views are obtained.

[0012] Step 6: According to the three-view reconstruction, the corresponding signed distance field is obtained, and the watertight completion result is obtained by the marching cubes algorithm.

[0013] Further, in step 1, the abstract template structure matched by the object is a general structure designed based on the category, which includes M original boxes that are associated with each other. Each box corresponds to a part in the object, and each box encloses a part of the object. The structural relationship, position and size of the boxes reflect the structural relationship, relative size and rough shape of the parts in the object.

[0014] Each original box is represented by 8 parameters, including 3 coordinate parameters of the starting point, 3 coordinate parameters of the ending point, and 2 dimension parameters representing the length and width of the original box. The association relationship between the boxes is defined by the connection constraint between the key points.

[0015] There are 26 key points, corresponding to the center points of the 6 faces, the 8 vertices and the 12 midpoints of the edges on the unit cube. In addition to the connection relationship between the parts, the definition of the template structure also includes the semantic information and symmetry of the parts, which are helpful for more accurately describing the overall structure and properties of the object.

[0016] Further, after matching to the corresponding abstract template structure, the network N s learns the parameters of each box that best fits the input incomplete point cloud. The complete object structure matched with the input point cloud is obtained. The deep network N s takes the three-channel coordinates of the input point cloud as the input features, and outputs n numerical values after the corresponding encoding and decoding process, which correspond to the parameters of the structure template. The general structure template is adjusted by the output parameters to make the template structure more fit the input point cloud, thereby obtaining the complete structure of the input point cloud.

[0017] The deep network N s A data augmentation strategy is used in the training scheme of the network N

[0018] Further, step 2 extracts the detailed information inside the input region structure by dividing the input point cloud according to the position of each component in the structure and obtaining the projection of each component point cloud on three orthogonal planes. A network is used to process these projections to supplement and perfect the closed contour line and solid three-view. The specific process can be divided into the following two sub-steps:

[0019] Step 2-1: According to the position and scale of each box obtained in step 1, the input point cloud is segmented by components to obtain the projection of each component point cloud on three orthogonal planes. The projection is used as the original multi-hole three-view to describe the detailed information inside the input region structure;

[0020] Step 2-2: The component projection map obtained in step 2-1 is used as the input to train the network N f , and the solid three-view details of each component are inferred from the multi-hole three-view details by the network N f This makes the detailed description more refined and accurate. The three-view obtained by projecting the original input component point cloud in step 2-1 is multi-hole, which cannot accurately outline the details of the input structure with a solid area enclosed by a contour, making it difficult for the subsequent steps to clearly define the input region and generate a watertight mesh result. The network N f is needed to predict the solid details of each component.

[0021] Further, the prediction process described in step 2-2 uses a data augmentation strategy, which allows the method to fill any multi-hole details. This strategy divides the image by a random straight line, and then randomly selects the area above or below the straight line as a mask. The mask is applied to both the multi-hole detail map obtained by projecting the complete point cloud and the solid three-view provided by the data set, denoted as and respectively, as the data pairs required for training the network N f During the training process, the network N f learns to recover from .

[0022] Further, step 3 aims to complete the details of the part. Although the solid three-view obtained in step 2 has provided the preliminary part profile, there are still details of missing areas. Through this step, the method can supplement and perfect these missing details to generate complete part information. The specific process can be divided into the following two sub-steps:

[0023] Step 3-1: Obtain the projection three-view of the point cloud in the symmetric component of each box as additional input of the network.

[0024] Step 3-2: Train network N c , which takes the original multi-hole three-view details of the part as input, predicts and outputs the complete solid point cloud details, and completes the supplement of the missing area.

[0025] The symmetry information between parts in step 3-1 is contained in the general template structure definition corresponding to the category. After obtaining the symmetric detail map, it is concatenated with the original detail map as the network input.

[0026] The step 3-2 needs to input the complete point cloud into the solid detail map obtained in the network N f of step 2, and take it as the label of the training network N c . Network N c learns to recover the complete details from the multi-hole details of the incomplete point cloud.

[0027] The network N c uses data augmentation strategy in the training process; the data augmentation strategy is the same as that in step 1. Thus, the robustness of the network is increased.

[0028] Further, step 4 identifies the missing area by comparing the detail maps, which provides the basis for preserving the input shape. The specific process can be divided into the following sub-steps:

[0029] Step 4-1: Compare the complete solid detail D c and the incomplete solid detail D s , identify the overlapping area as the original input area A o .

[0030] Step 4-2: For each part P, perform symmetric mirroring on its complete three-view to obtain the solid three-view of the mirrored part Check whether its complete three-view overlaps with the solid three-view of the mirrored part , respectively calculate the overlap rate of the three-view If one of the three-view overlap rates r does not exceed the threshold T1, the part is not a symmetric part, and the missing area needs to be generated with diversity.

[0031] The mirror image refers to the mirror image of the three views according to three orthogonal planes. For example, if the three views after the mirror image about the xy plane are needed, only the vertical mirror image of the xy view and the horizontal mirror image of the yz and xz views are needed.

[0032] Step 4-3: For each symmetric component P, obtain the symmetric relationship of the component from the complete structure of the object. If the component P has a symmetric component, merge the solid three-view of the mirror image part of the component and the solid three-view of the symmetric component as the to-be-checked region, and then define the overlapping region of the solid three-view of the component P and the to-be-checked region as the associated input region A r ; the overlapping region of the component detail map d c and d s is the original input region A o . Calculate the proportion of the region A o ∪A r to the complete part detail. If it exceeds the threshold T2, it means that the missing area can be directly completed by the symmetric information, and diversity generation is not needed; otherwise, the missing area needs to be generated by diversity.

[0033] Further, the threshold T1 in step 4-2 is in the range of 0.7-0.9, and the threshold T2 in step 4-3 is in the range of 0.5-0.7.

[0034] Further, step 5 is based on the diffusion model N g of graph-to-graph to realize diversity completion, and the diffusion model N g of graph-to-graph is injected with image information, and a noisy source graph is used as the initialization, and the source graph is the complete detail map after being masked by a random mask; in the denoising process, the diffusion model learns to recover the complete detail map without the mask from the noise. The randomness and data complexity of this process enable the model to capture more details in the generation process, thereby enhancing the diversity and accuracy of the output.

[0035] Step 6 recovers the three-dimensional face from the complete three views. According to the rays projected by the three orthogonal planes, a point in the three-dimensional space can be uniquely determined, and the distance between the projection point and the boundary is calculated, and the minimum value is selected as the spatial distance. If the point is uniquely determined by the points in the contour in the three projection views, it means that it is inside the closed three-dimensional figure, and the value is negative; if the point is uniquely determined by the points on the contour in the three projection views, it means that it is exactly on the surface of the closed three-dimensional figure, and the value is 0; otherwise, it is outside the closed three-dimensional figure, and the value is positive. Based on this, the corresponding signed distance field representation is obtained, and then the watertight face is recovered according to the marching cubes algorithm.

[0036] The signed distance field is a function representing distance, for any point in three-dimensional space, the signed distance field represents the shortest distance from the point to the object surface. If the point is located outside the object, the value is positive; if it is located inside the object, the value is negative; if it is located on the object surface, the value is zero. Through the signed field, the shape and position of the object in space can be determined, providing a basis for subsequent surface extraction and mesh reconstruction. For each three-dimensional query point, the distance between the projection of the query point on the three-view and the boundary is selected as the signed distance field of the query point.

[0037] Further, step 6 comprises the following steps,

[0038] Step 6-1: Obtain the signed distance field, and reconstruct the three-dimensional region according to the three-view. For each three-dimensional query point, the distance between the projection of the query point on the three-view and the boundary is selected as the signed distance field of the query point, and the shape and position of the object in space can be determined through the signed distance field.

[0039] Step 6-2: Extract the isosurface through the signed distance field obtained in step 6-1, and based on the isosurface, restore the watertight three-dimensional face through the marching cubes algorithm. The marching cubes algorithm is a classic three-dimensional surface reconstruction algorithm, which is used to restore a closed and complete three-dimensional surface without holes.

[0040] Beneficial effects:

[0041] The method provided by the application has the beneficial effects that:

[0042] The method fully utilizes the structural information for point cloud completion. The method can complete the incomplete point cloud to a complete mesh by analyzing the structure of the three-dimensional object. In this process, the existing structural information in the input point cloud is fully utilized to ensure that the completed mesh has continuity in geometry. This completion strategy based on structural information also helps to improve the completeness and interpretability of the completed point cloud data, and important details can still be effectively restored when there are large areas of missing regions.

[0043] The component decomposition completion and generation method proposed by the method only focuses on the diversified prediction of a single component, thereby effectively preserving the input shape. Through this method, the system can not only cope with the morphological changes of different components, but also reduce the risk of overfilling the missing area or deviating from the original shape, ensuring that the generated component is highly consistent with the original input.

[0044] The method has proven its superiority through a large number of experiments, and has achieved SOTA performance on both synthetic data sets and real scan data sets. Whether in terms of point cloud completion accuracy, diversity of generation effect, or in terms of the ability to handle complex class details recovery, the method has shown superior performance.

[0045] The present application is inspired by the structural information of an object, and introduces an innovative diversity completion method using local structure analysis. After structure analysis, each component can be separated and the relationship between components can be determined, and the missing parts can be more accurately completed and reconstructed according to the existing related parts (such as symmetrical corresponding parts). In addition, focusing on the diversity prediction of the missing area of a single part makes the completion more simple and accurate, and also brings more changes to the final completion result. The method uses a projection three-view to describe the details of the part, and through two-dimensional three-view completion and detail generation, it realizes the diversity completion of three-dimensional objects, which can obtain the reconstruction result on the basis of effectively preserving the input shape. BRIEF DESCRIPTION OF DRAWINGS

[0046] The above and / or other aspects of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0047] Figure 1 The above and / or other aspects of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0048] Figure 2 The above and / or other aspects of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0049] Figure 3 The above and / or other aspects of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0050] Figure 4 The above and / or other aspects of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0051] Figure 5 The above and / or other aspects of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0052] Figure 6 The above and / or other aspects of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0053] Figure 7 The above and / or other aspects of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0054] Figure 8 The above and / or other aspects of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0055] Figure 9 The above and / or other aspects of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0056] Figure 10 The above and / or other aspects of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0057] Figure 11 The above and / or other aspects of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which: DETAILED DESCRIPTION

[0058] AsFigure 1 and Figure 2 As shown, a method for completing diverse shapes based on component analysis and decomposition includes the following steps:

[0059] Step 1: Represent the object as an abstract template structure. The object is a fragmented input data of a 3D point cloud representation. The template structure parameters are predicted by a deep learning network and matched with the fragmented point cloud. Based on the template structure parameters, the complete structural representation of the object is derived.

[0060] Step 2: Based on the complete structural representation of the object obtained in Step 1, the object is decomposed into components, and the projection details of the internal points of each component on three orthogonal planes are obtained. The projection details are then input into the network to generate solid detail information for each component.

[0061] Step 3: Complete the three-view drawing. Combine the detailed information of symmetrical parts to deduce and complete the complete part details of the object.

[0062] Step 4: By comparing the incomplete and complete three-view drawings obtained in Step 3, distinguish and identify the missing areas, and determine whether diversity generation is needed based on the missing areas.

[0063] Step 5: Mask out the areas that need to generate diversity in the complete detail map, perform image-to-image diffusion generation, and obtain diverse complete detail three-view drawings.

[0064] Step 6: Reconstruct the 3D region based on the three views to obtain the corresponding symbolic distance field, and obtain the watertight completion result through the moving cube algorithm.

[0065] Figure 2 M1-M4 represent the results of four diversity generation methods.

[0066] like Figure 3 As shown, in step 1, the abstract template structure for object matching is a general structure based on category design, containing M interrelated primitive boxes, each box enclosing one component of the object. The structural relationships, positions, and sizes of these boxes reflect the structural relationships, relative sizes, and rough shapes between the object components.

[0067] Each original bounding box is represented by eight parameters, including three coordinate parameters for the starting point, three coordinate parameters for the ending point, and two dimension parameters representing the length and width of the original bounding box. The relationships between bounding boxes are defined by connection constraints between keypoints.

[0068] There are 26 key points in total, corresponding to the center points of the 6 faces, 8 vertices, and the midpoints of the 12 edges of the unit cube. In addition to the connection relationships between components, the definition of the template structure also includes the semantic information and symmetry of the components, which helps to more accurately describe the overall structure and properties of the object.

[0069] After matching the corresponding abstract template structure, through network N s To learn about the shortcomings of input cloud P p The optimal bounding box parameters p. This yields the complete object structure matching the input point cloud. Deep network N s Using the three-channel coordinates of the input point cloud as input features, the network outputs n values ​​after the corresponding encoding and decoding process, corresponding to the parameters of its structural template. By adjusting the general structural template through the output parameters, the template structure is made to better fit the input point cloud, thus obtaining the complete structure S of the input point cloud. c .

[0070] The deep network N s Their training scheme uses data augmentation strategies, such as Figure 11 As shown, the strategy involves randomly rotating the complete point cloud in the dataset according to the rotation matrix R, removing some points by segmenting them according to the z-axis, and then rotating the point cloud again using the rotation matrix R. -1 Flip it back to obtain the residual defect cloud of the input. P1 and P2 are the set z-axis threshold planes, i.e., z = a, the a1 region is the removal region, and the a2 region is the retention region.

[0071] This enhances the network's robustness to various defective cloud distributions.

[0072] N s The network is optimized using the mean squared error between the predicted structural parameters and the label parameters. N is trained. s The pseudocode representation of a network is as follows:

[0073]

[0074] like Figure 4 As shown, step 2 involves dividing the input point cloud according to the position of the bounding box of each component in the structure and obtaining the projection of each component's point cloud onto three orthogonal planes, thereby extracting detailed information within the input region's structure. A network is then used to process these projections, supplementing and refining the closed contour lines and solid three-view drawings. The specific process can be divided into the following two sub-steps:

[0075] Step 2-1: Based on the position and scale of each box obtained in Step 1, segment the input point cloud into components, obtain the projection of each component point cloud onto three orthogonal planes, and use the projection as the original multi-hole three-view detail D. p, the porous three-view description input area inside the structure details.

[0076] Step 2-2: Project the component projection map obtained in step 2-1 as input to train network N f , through network N f By the porous three-view details D p Predict the solid three-view details D s of each component by inference. Make the details description more refined and accurate. The three-view obtained in step 2-1 according to the projection of the point cloud of the original input component is porous, which cannot outline the details of the input structure with an accurate outline of the solid area, making it difficult for the subsequent step to clearly define the input area and generate a watertight mesh result. The network N f described in step 2-2 is needed to predict the solid details D s of each part.

[0077] The prediction process described in step 2-2 uses a data augmentation strategy, so that this method can fill in any porous details. This strategy divides the image by a random straight line, then randomly selects the area above or below the straight line as a mask, and the mask is applied to the porous detail map obtained by projecting the complete point cloud and the solid three-view provided by the dataset, respectively represented as and , as the data pairs required for training network N f During training, N f learns to recover from . The pseudo code for training the N f network is as follows:

[0078]

[0079] As shown in Figure 5 , step 3 aims to complete the completion of the component details. Although the solid three-view obtained in step 2 provides a preliminary part outline, there are still details of missing areas. Through this step, the method can supplement and perfect these missing details to generate complete details D c , that is, complete part information. The specific process can be divided into the following two sub-steps:

[0080] Step 3-1: Obtain the projection three-view of the point cloud within the symmetric component of each box as additional input to the network.

[0081] Step 3-2: Train network N c , which takes the original porous three-view details D p of the component as input, predicts and outputs the complete solid point cloud details, and completes the supplement of the missing area.

[0082] In step 3-1, the symmetry information between components is contained in the general template structure definition corresponding to the category. After obtaining the symmetrical detail map, it will be concatenated with the original detail map as the network input. Figure 5 Taking a chair as an example, after obtaining the perforated three-view drawings of the backrest, seat, and four legs based on the structural information, the three-view drawings of the symmetrical components are obtained based on the structural symmetry information and merged into a network N. c Input.

[0083] Step 3-2 requires inputting the complete point cloud into the network N described in step 2. f The solid three-view detail D obtained from s And use it as the training network N c The tag. Network N c Learning to recover complete details from the porous details of flawed clouds (D) c .

[0084] The network N c A data augmentation strategy is used during training; this strategy is the same as the one used in step 1. This increases the robustness of the network. Training N... c The process utilized pre-trained N s and N f The training pseudocode for the network is as follows:

[0085]

[0086] like Figure 6 As shown, step 4 clearly identifies the missing areas through comparison of detail images, providing a basis for preserving the input shape. The specific process can be divided into the following sub-steps:

[0087] Step 4-1: Compare the complete solid details D c and incomplete solid details D s The overlapping region is identified as the original input region A. o .

[0088] Step 4-2: For each component P, generate its complete three-view drawing. The solid three-view diagram of the mirrored portion is obtained by mirroring the image symmetrically along three orthogonal planes. Check its complete three views Is it the solid three-view drawing of its mirror image? Overlap, calculate the overlap rate of the three views respectively. If the overlap rate r of the three views does not exceed the threshold T1, then the part is not a symmetrical part, and its missing area needs to be generated using diversity generation.

[0089] The mirror image refers to the symmetrical mirror image of the three views according to three orthogonal planes. For example, if the three views after the mirror image about the xy plane are needed, only the vertical mirror image of the xy view and the horizontal mirror image of the yz and xz views are needed.

[0090] Step 4-3: For each symmetrical component P, obtain the symmetrical relationship of the component from the complete structure of the object. If the component P has a symmetrical component, merge the solid three-view of the mirror image part of the component and the solid three-view of the symmetrical component as the to-be-checked region, and then define the overlapping region of the solid three-view of the component P and the to-be-checked region as the associated input region A r ; the overlapping region of the component detail map d c and d s is the original input region A o . Calculate the proportion of the region A o ∪A r to the complete part detail. If it exceeds the threshold T2, it means that the missing area can be directly completed by the symmetrical information, and diversity generation is not needed; otherwise, the missing area needs to be generated by diversity.

[0091] The threshold T1 in step 4-2 is 0.8, and the threshold T2 in step 4-3 is 0.6.

[0092] As shown in Figure 7 , step 5 realizes diversity completion based on the diffusion model N g of graph-to-graph, injects image information into the diffusion model N g of graph-to-graph, and uses the source graph with noise as the initialization, the source graph being the complete detail map after being masked by a random mask; in the denoising process, the diffusion model learns to recover the complete detail map without the mask from the noise. The randomness and data complexity of this process enable the model to capture more details in the generation process, thereby enhancing the diversity and accuracy of the output. The pseudo code of the N g network is as follows:

[0093]

[0094] As shown in Figure 8As shown, step 6 recovers the three-dimensional patch from the complete three views. According to the rays of the three orthogonal plane projections, a point in three-dimensional space can be uniquely determined, and the distance of the projection point on the three views from the boundary is calculated, and the minimum value is selected as the spatial distance. If the point is uniquely determined by the points in the contour enclosed graph in the three projection views, it is inside the closed three-dimensional graph, and the value is negative; if the point is uniquely determined by the points on the contour in the three projection views, it is exactly on the surface of the closed three-dimensional graph, and the value is 0; otherwise, it is outside the closed three-dimensional graph, and the value is positive. Based on this, the corresponding signed distance field representation is obtained, and then the watertight patch is recovered according to the moving cube algorithm.

[0095] The pseudo code for obtaining the signed distance field is as follows:

[0096]

[0097] Figure 9 The comparison of the method proposed in this paper (Ours) with cGAN, SDFusion and ShapeFormer on the ShapNet dataset is shown in the figure, and the input Input, label GT and three kinds of diversity completion results of each method are shown.

[0098] Figure 10 The comparison of the method proposed in this paper (Ours) with the prior art (cGAN, SDFusion and ShapeFormer) on the real scan dataset Scannet, LiDAR-Net and consumer-grade device scanned data is shown. The first three rows show the diversity completion results of the swivel chair extracted from ScanNet. The next three rows show the completion results of the chair extracted from ScanNet, LiDAR-Net and consumer-grade device scanning.

[0099] The present application solves the problem that the current method cannot obtain high-quality output results in the field of three-dimensional completion under the premise of preserving the input shape. After analyzing and decomposing the structure of the parts, the method generates and completes the three views of the incomplete input parts. By completing the three views of each part of the incomplete three-dimensional object, the method realizes the conversion from complete three views to signed distance field, and then to watertight triangular patch, realizing the diversity completion of three-dimensional objects.

[0100] The application provides a method and a train of thought of shape completion based on component analysis and diversity decomposition. The above description is only the preferred embodiment of the application, and it should be pointed out that, for those skilled in the art, some improvements and refinements can be made without departing from the principle of the application, and these improvements and refinements should also be regarded as the protection scope of the application. The components not explicitly described in the embodiment can be realized by using the prior art.

Claims

1. A method for shape completion based on component analysis and diversity of decomposition, characterized in that, Comprising the following steps: Step 1: Representing the object as an abstract template structure, the object being a partial input data in a three-dimensional point cloud representation, by means of a deep learning network N s Predicting and matching the template structure parameters of the partial point cloud, deriving a complete structure representation of the object from the template structure parameters; Step 2: based on the complete structure representation of the object obtained in step 1, the object is decomposed into parts, the projection details of the internal points of each part on three orthogonal planes are obtained, and the projection details are input into the network to generate the solid details of each part; Step 3: three-view completion is performed, and the complete part details of the object are derived and completed in combination with the detail information of the symmetrical parts; Step 4: by comparing the incomplete and complete three views obtained in step 3, the missing area is distinguished and identified, and it is judged whether diversity generation is needed according to the missing area; Step 5: in the complete detail map, the area needing diversity generation is masked, image-to-image diffusion generation is performed, and diversified complete detail three views are obtained; Step 6: three-dimensional regions are reconstructed according to the three views, corresponding signed distance fields are obtained, and watertight completion results are obtained by moving cube algorithm.

2. The component analysis and decomposition based shape completion method of claim 1, wherein, The abstract template structure of step 1 is a general structure designed for three-dimensional objects, which describes the basic part structure relationship and the variation range of part parameters of the object. The structure is composed of M mutually related original boxes, and each original box corresponds to a part in the object; Each original box is represented by 8 parameters, including three coordinate parameters of the starting point, three coordinate parameters of the ending point, and two dimension parameters representing the length and width of the original box; The connection constraint relationship between the key points controls the connection relationship between the boxes, and the key points are 26, corresponding to the center points of the six faces, the eight vertices and the 12 midpoints of the edges of the unit cube; The association relationship includes the connection relationship between the parts and the semantics and symmetry information of the parts.

3. The component analysis and decomposition based shape completion method of claim 2, wherein, The deep network N described in step 1 s Taking the three-channel coordinates of the input point cloud as input features, the network outputs n numerical values after the corresponding encoding and decoding process, which correspond to the parameters of the structure template. By adjusting the general structure template through the output parameters, the template structure is more fitted to the input point cloud, thereby obtaining the complete structure of the input point cloud. The deep network N s A data augmentation strategy is used in the training scheme of the deep network N, which rotates the complete point cloud in the dataset randomly and removes part of the points according to the z-axis segmentation to obtain the incomplete point cloud as input.

4. The component analysis and decomposition based shape completion method of claim 3, wherein, Step 2 includes the following steps: Step 2-1: according to the position and scale of each box obtained in step 1, the input point cloud is segmented into parts, and the projection of each part point cloud on three orthogonal planes is obtained, which is used as the original multi-hole three view to describe the internal structure details of the input area; Step 2-2: Train network N with the part projection maps obtained in step 2-1 as input f D is obtained by a network N f from the porous three-view detail prediction inference s .

5. The component analysis and decomposition based shape completion method of claim 4, wherein, Step 2-2 prediction process uses a data augmentation strategy, which is to divide the image by a random straight line, and randomly select the area above or below the straight line as the mask. The mask is applied to the multi-hole detail map obtained by projecting the complete point cloud and the solid three-view of the data set white band, respectively represented as and As network N f The data pairs required for training, in the training process, network N f Learned by Recovery 6. The component analysis and decomposition based shape completion method of claim 5, wherein, Step 3 includes the following steps: Step 3-1: Obtain the projection tri-view of the point cloud within the symmetric component of each box as the additional input of network N c The symmetric information between components in step 3-1 is contained in the definition of the general template structure, and after obtaining the symmetric detail map, it is concatenated with the original detail map as the network input; Step 3-2: Train network N c , the network N c with the original porous three-view details of the component as input, predict and output the complete solid details D c , supplement the missing areas; Step 3-2 requires inputting the complete point cloud into the network N described in step 2. f The solid detail image obtained from the image is used as the training network N. c The tag, network N c Learn to recover complete details from the porous details of flawed clouds; The network N c A data augmentation strategy is used during training; the data augmentation strategy is the same as in step 1.

7. The component analysis and decomposition based shape completion method of claim 6, wherein, Step 4 includes the following steps: Step 4-1: Compare complete solid detail D c with incomplete solid detail D s Identify overlapping area as original input area A o ; Step 4-2: For each part P, generate its complete tri-view Symmetrically mirror the part P in three orthogonal planes to obtain the solid tri-views of the mirrored parts Check the complete tri-view of the part P with the solid tri-views of the mirrored parts Overlap, respectively, to calculate the overlap rate of the tri-views If one of the overlap rates r of the tri-views does not exceed a threshold T1, then the part is not a symmetric part and its missing area needs to be generated with diversity Step 4-3: For each symmetric part P, get the symmetric relationship of the part from the complete structure of the object, if the part P has a symmetric component, merge the solid three-view of the mirrored part and the solid three-view of its symmetric component as the to-be-checked region, define the overlapping area of the solid three-view of the part P and the to-be-checked region as the associated input region A r ; the part detail map d c and the overlapping area of d s as the original input region A o , calculate the proportion of the region A o ∪A r to the complete part detail, if it exceeds the threshold T2, the missing region is directly completed by the symmetric information, otherwise, the missing region needs to be generated with diversity.

8. The component analysis and decomposition based shape completion method of claim 7, wherein, The threshold T1 in step 4-2 is in the range of 0.7-0.9, and the threshold T2 in step 4-3 is in the range of 0.5-0.

7.

9. The component analysis and decomposition based shape completion method of claim 8, wherein, Step 5 based on the diffusion model of graph-to-graph N g Diversity completion is achieved for the diffusion model of graph-to-graph N g Image information is injected, and a noisy source graph is used as initialization, the source graph being a complete detail graph masked by a random mask; in the denoising process, the diffusion model learns to recover the complete detail graph without the mask from the noise, and the network realizes diversity generation through the randomness of the noisy denoising process and the complexity of the high-dimensional data.

10. The component analysis and decomposition based shape completion method of claim 9, wherein, Step 6 Comprising the following steps, Step 6-1: obtain the signed distance field, reconstruct the three-dimensional region according to the three views, for each three-dimensional query point, the projection of the query point on the three views is queried, and the minimum value is selected as the signed distance field of the query point. The shape and position of the object in space can be determined through the signed distance field; Step 6-2: extract the isosurface from the signed distance field obtained in step 6-1, and restore the watertight three-dimensional face based on the isosurface through the moving cube algorithm.

Images