Complex component point cloud completion method based on cross-modal alignment and related equipment

Abstract

The application provides a complex component point cloud completion method based on cross-modal alignment and related equipment, and relates to the technical field of three-dimensional computer vision. The method comprises the following steps: three-dimensional topological features and continuous visual feature fields are respectively extracted through a geometry branch and a vision branch, cross-modal space alignment is realized by using an active query mechanism, and global features are generated; meanwhile, feature vectors representing local surface attributes and having long-distance dependency relationships are extracted; then, sparse skeleton point clouds are generated based on the global features, multi-level geometric uncertainty analysis is performed by using a structure self-attention module, and the model is guided to retrieve high-frequency geometric details from the local features to modify and embed the skeleton in layers. The application maximizes point pair mutual information by using the active retrieval mechanism of the cross-modal, solves the problems of poor fine structure recovery and easy degradation of the model in the prior art, realizes accurate completion of fine structures such as thin walls and sharp corners, and can truly reflect individual differences such as machining allowances of workpieces.

Description

Technical Field

[0001] This invention relates to the technical field of point cloud completion, and in particular to a method and related equipment for point cloud completion of complex components based on cross-modal alignment. Background Technology

[0002] In high-end equipment manufacturing sectors such as rail transportation and aerospace, the precision manufacturing and quality control of complex components (such as bogies and engine blades) are crucial for ensuring product performance and safety. These components typically possess intricate geometric features such as deep holes, stiffeners, and complex free-form surfaces, demanding extremely high accuracy in 3D inspection. Currently, non-contact measurement methods, such as laser scanning or structured light vision, are primarily used to acquire their 3D point cloud data.

[0003] However, due to limitations imposed by the component's own geometric occlusion, high surface reflectivity, and sensor viewing angle constraints, point cloud data directly obtained using the aforementioned methods generally suffers from data gaps, holes, and local sparsity, making it difficult to form a complete and dense 3D model. This severely impacts the accuracy and reliability of subsequent dimensional measurements, morphological analysis, and quality assessment.

[0004] Therefore, how to achieve high-fidelity and high-precision point cloud completion for complex industrial components has become a key technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a method and related equipment for point cloud completion of complex components based on cross-modal alignment, so as to solve the above problems.

[0006] In a first aspect, embodiments of the present invention provide a method for completing point clouds of complex components based on cross-modal alignment, applied to a point cloud completion model, the method comprising: The incomplete 3D point cloud of the target component is input into the geometric feature extraction module to extract geometric feature vectors containing 3D topological information; The incomplete 3D point cloud is input into the visual feature extraction module to obtain the aggregated visual feature vector of each position in the incomplete 3D point cloud in order to generate a continuous visual feature field. Based on the geometric feature vector, a query is performed in the visual feature field to align and match the probability density image with the visual feature field space, thereby generating a global feature vector. The incomplete 3D point cloud data is input to the local feature extraction module to extract a first feature vector that characterizes local surface properties. The first feature is then input to the self-attention module, which is used to capture the long-distance dependencies of the first feature. The dependency relationship is fused with the first feature to generate a second feature vector; The global feature vector is input into a multilayer perceptron for dimensionality reduction, and the output of the multilayer perceptron is fused with the incomplete 3D point cloud to output a sparse skeleton point cloud. The output of the multilayer perceptron and the sparse skeleton point cloud are input into the first structure self-attention module for geometric uncertainty analysis to output a first target region of interest. The point cloud features in the first target region of interest are used as a query vector to perform a query in the second feature vector and output a first query result. Based on the first query result, a first local detail feature is determined. Based on the first local detail feature, it is embedded into the sparse skeleton point cloud for completion and outputs the intermediate result of point cloud completion. The output of the multilayer perceptron and the intermediate result of the point cloud completion are input into the second structural self-attention module for geometric uncertainty analysis to output a second target region of interest. The point cloud features in the second target region of interest are used as query vectors, and a query is performed in the second feature vectors to output a second query result. Based on the second query result, a second local detail feature is determined. Based on the second local detail feature, the point cloud completion intermediate result is embedded to complete the incomplete 3D point cloud, and the completion result of the incomplete 3D point cloud is output.

[0007] One possible approach is that the method further includes: The incomplete 3D point cloud is translated by coordinates and scaled diagonally to project the incomplete 3D point cloud onto the target area.

[0008] One possible approach is that, in the step of inputting the incomplete 3D point cloud into the visual feature extraction module to obtain the aggregated visual feature vector at each location in the incomplete 3D point cloud in order to generate a visual feature field, The RGB image of the incomplete 3D point cloud is determined based on the 3D coordinates of the incomplete 3D point cloud; The RGB image is input into the visual feature extraction module, and the Gaussian splashing algorithm is used to obtain the aggregated visual feature vector of each position in the incomplete 3D point cloud.

[0009] One possible approach is to input the RGB image into the visual feature extraction module and use the Gaussian splashing algorithm to obtain the aggregated visual feature vector at each location in the incomplete 3D point cloud. For any position The aggregated visual feature vector is calculated using the following formula: ; in, —in location The aggregated visual feature vector calculated at the location; —Falling into the neighboring area The first Three-dimensional projection points; —No. Coordinates of a three-dimensional point in a projection cluster Mapped to two dimensions ; —No. Each projection point pairs the query location Contribution weight; -constant; ; —No. Deterministic coordinates of a three-dimensional point projected onto a two-dimensional plane; — Coordinates of the projection point With query location The square of the Euclidean distance between them; —The bandwidth or standard deviation of the Gaussian kernel; —No. The depth value of a 3D point in the camera coordinate system; -constant.

[0010] One possible approach is to input the incomplete 3D point cloud data into a local feature extraction module to extract a first feature vector characterizing local surface properties, and then input the first feature vector into a self-attention module, which is used to capture the long-range dependencies of the first feature. Local feature extraction module: configured to acquire incomplete 3D point cloud, and extract the mean curvature and tangent space information of the incomplete 3D point cloud by performing edge convolution operation to approximate the Laplacian operator on the manifold. Local surface attributes include mean curvature and tangent space information.

[0011] The self-attention module is specifically used to: receive a first feature vector and capture long-distance dependencies based on a global message passing mechanism, the dependencies including: the hole structure, symmetric structure and long-distance links of the 3D point cloud.

[0012] One possible approach is to input the output of the multilayer perceptron and the sparse skeleton point cloud into a first structured self-attention module for geometric uncertainty analysis to output a first target region of interest. The point cloud features in the first target region of interest are used as a query vector to perform a query in the second feature vector, and a first query result is output. Based on the first query result, a first local detail feature is determined. Based on the first local detail feature, it is embedded into the sparse skeleton point cloud for completion, and an intermediate result of point cloud completion is output. and The steps of inputting the output of the multilayer perceptron and the intermediate result of the point cloud completion into the second structural self-attention module for geometric uncertainty analysis to output a second target region of interest; using the point cloud features in the second target region of interest as a query vector; querying within the second feature vector; outputting a second query result; determining a second local detail feature based on the second query result; embedding the second local detail feature into the intermediate result of the point cloud completion for completion; and outputting the completed result of the incomplete 3D point cloud are then discussed. The first self-attention module is configured to use the chamfer distance between the generated points of the sparse skeleton point cloud and the input points corresponding to the global feature vector as a measure of local reconstruction uncertainty, determine the high-entropy region based on the measurement result, and determine the first target attention region based on the high-entropy region. The second self-attention module is configured to use the chamfer distance between the intermediate result of point cloud completion and the input point corresponding to the global feature vector as a measure of local reconstruction uncertainty, determine the high-entropy region based on the measure result, and determine the second target attention region based on the high-entropy region.

[0013] One possible approach is to use the following loss function during the training of the point cloud completion model: ; —The total loss value during training; —Adjustment coefficient; —Chamfer distance operator; —The completed point cloud output generated by the model; —A true and complete point cloud.

[0014] Secondly, embodiments of this application provide a point cloud completion device for complex components based on cross-modal alignment, applied to a point cloud completion model, including: First extraction module: used to input the incomplete 3D point cloud into the geometric feature extraction module to extract geometric feature vectors containing 3D topological information; The second extraction module is used to input the incomplete 3D point cloud into the visual feature extraction module, and to obtain the aggregated visual feature vector of each position in the incomplete 3D point cloud to generate a continuous visual feature field. Alignment module: Performs a query in the visual feature field based on the geometric feature vector to align and match the probability density image with the visual feature field space, and generates a global feature vector; The third extraction module is used to input the incomplete 3D point cloud data into the local feature extraction module, extract a first feature vector to characterize local surface properties, and input the first feature into the self-attention module, which is used to capture the long-distance dependency of the first feature. Fusion module: used to fuse the dependency relationship with the first feature to generate a second feature vector; First output module: used to input the global feature vector into a multilayer perceptron for dimensionality reduction, and to fuse the output of the multilayer perceptron with the incomplete 3D point cloud to output a sparse skeleton point cloud; The second output module is used to input the output of the multilayer perceptron and the sparse skeleton point cloud into the first structure self-attention module for geometric uncertainty analysis, so as to output a first target region of interest, use the point cloud features in the first target region of interest as a query vector, perform a query in the second feature vector, output a first query result, determine a first local detail feature based on the first query result, embed the first local detail feature into the sparse skeleton point cloud for completion, and output the intermediate result of point cloud completion. The third output module is used to input the output of the multilayer perceptron and the intermediate result of the point cloud completion into the second structural self-attention module for geometric uncertainty analysis, so as to output a second target region of interest. The point cloud features in the second target region of interest are used as query vectors, and a query is performed in the second feature vectors to output a second query result. Based on the second query result, a second local detail feature is determined. Based on the second local detail feature, the point cloud completion is embedded into the intermediate result of the point cloud completion for completion, and the completion result of the incomplete 3D point cloud is output.

[0015] Thirdly, embodiments of this application provide an electronic device, including: At least one processor; and At least one memory communicatively connected to the processor, wherein: The memory stores program instructions that can be executed by the processor, which can invoke the program instructions to perform the method as described in the first aspect.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that cause the computer to perform the method described in the first aspect.

[0017] In the embodiments provided in this application, a sparse skeleton point cloud is generated using the aforementioned global feature vector. Through hierarchical upsampling and feature correction, the completed point cloud data is output. Using this differentiable dictionary lookup method, the point cloud coordinates are corrected using high-frequency geometric details, restoring fine structures such as thin walls and sharp corners, as well as invisible structures. This mechanism forces the model to actively retrieve semantic cues related to the currently missing geometric regions in the visual feature field. This design maximizes point-to-point mutual information, ensuring that the model strictly relies on the current visual observation data during inference, avoiding the model's degradation into a "shape template retriever" that only relies on training memory. This allows the model to accurately reflect the true individual differences (such as machining allowances) of the workpiece under test.

[0018] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0020] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a method for completing point clouds of complex components based on cross-modal alignment, provided in this application embodiment. Figure 2 A schematic diagram of the global feature vector generation steps in a point cloud completion method for complex components based on cross-modal alignment provided in this application embodiment; Figure 3 A schematic diagram of local feature extraction in a point cloud completion method for complex components based on cross-modal alignment provided in an embodiment of this application; Figure 4 A schematic diagram of point cloud completion steps for a complex component point cloud completion method based on cross-modal alignment provided in this application embodiment; Figure 5 A schematic diagram of the point cloud completion steps after the point cloud completion step in a point cloud completion method for complex components based on cross-modal alignment provided in this application embodiment; Figure 6(a) shows a detailed diagram of the visual feature extraction module and the geometric feature extraction module of a complex component point cloud completion method based on cross-modal alignment provided in an embodiment of this application; Figure 6(b) is a Gaussian splashing algorithm projection comparison diagram of a complex component point cloud completion method based on cross-modal alignment provided in an embodiment of this application; Figure 7 A point cloud completion method for complex components based on cross-modal alignment, provided for an embodiment of this application, is illustrated in the structural diagram of the point cloud completion device after the point cloud completion step; Figure 8 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] In high-end equipment manufacturing sectors such as rail transportation and aerospace, the precision manufacturing and quality control of complex components (such as bogies and engine blades) are crucial for ensuring product performance and safety. These components typically possess intricate geometric features such as deep holes, stiffeners, and complex free-form surfaces, demanding extremely high accuracy in 3D inspection. Currently, non-contact measurement methods, such as laser scanning or structured light vision, are primarily used to acquire their 3D point cloud data.

[0024] However, due to limitations imposed by the component's own geometric occlusion, high surface reflectivity, and sensor viewing angle constraints, point cloud data directly obtained using the aforementioned methods generally suffers from data gaps, holes, and local sparsity, making it difficult to form a complete and dense 3D model. This severely impacts the accuracy and reliability of subsequent dimensional measurements, morphological analysis, and quality assessment.

[0025] Therefore, how to achieve high-fidelity and high-precision point cloud completion for complex industrial components has become a key technical problem that urgently needs to be solved by those skilled in the art.

[0026] Based on this, embodiments of the present invention provide a method and related equipment for completing point clouds of complex components based on cross-modal alignment, in order to solve the above-mentioned problems.

[0027] Reference Figure 1 First, we will describe a method for completing point clouds of complex components based on cross-modal alignment, which is involved in the embodiments of this application.

[0028] In the embodiments provided in this application, the point cloud completion method for complex components with cross-modal alignment is applied to a point cloud completion model, and the method includes: Combination Figure 2 S1: Input the incomplete 3D point cloud of the target component into the geometric feature extraction module to extract geometric feature vectors containing 3D topological information.

[0029] As one feasible approach, the geometric feature extraction module can employ the Transformer module. Three-dimensional topological information can characterize the connectivity, adjacency, and local / global structural relationships between points in the incomplete three-dimensional point cloud within the three-dimensional space. See Figure 6(a).

[0030] Specifically, it describes how points form local surfaces (such as planes and corners) through their neighboring points, and further form global components (such as bogies and engine blades), as well as how these components are assembled (such as connecting and surrounding).

[0031] In this step, in order to facilitate subsequent processing of the discrete point cloud, in some examples, it is necessary to preprocess the incomplete 3D point cloud, that is, to perform coordinate translation and diagonal scaling on the incomplete 3D point cloud to project the incomplete 3D point cloud onto the target area.

[0032] S2: Input the incomplete 3D point cloud into the visual feature extraction module to obtain the aggregated visual feature vector of each position in the incomplete 3D point cloud, so as to generate a continuous visual feature field.

[0033] S3: Perform a query in the visual feature field based on the geometric feature vector to align and match the probability density image with the visual feature field space, thereby generating a global feature vector.

[0034] In this step, geometric feature vectors are used as query vectors to perform active queries in the visual feature field, thereby fusing the 3D topological information of the incomplete 3D point cloud with the corresponding continuous visual field. This global feature vector captures the long-distance dependencies of the entire component (such as the bogie side beam). At the same time, it captures the logical relationships between each sub-component.

[0035] S4: Input the incomplete 3D point cloud data into the local feature extraction module to extract the first feature vector used to characterize the local surface properties, and input the first feature into the self-attention module, which is used to capture the long-distance dependency of the first feature; Combination Figure 3Here, after acquiring the incomplete 3D point cloud, the local feature extraction module extracts the mean curvature and tangent space information of the incomplete 3D point cloud by performing an edge convolution operation to approximate the Laplacian operator on the manifold.

[0036] It should be noted that local surface properties include mean curvature and tangent space information.

[0037] Here, the local feature extraction module may specifically adopt the edge convolution module, which can be set by those skilled in the art and is not limited here.

[0038] Meanwhile, the self-attention module is specifically used to: receive the first feature vector and capture long-distance dependencies based on the global message passing mechanism.

[0039] The field dependencies include, but are not limited to: the void structure, symmetric structure, and long-distance links of 3D point clouds.

[0040] Specifically, the self-attention module can establish spatial feature mappings between symmetrical elements of components at long distances, and use the geometric priors of known regions to assist in feature inference of missing regions. At the same time, it uses global context information to distinguish between functional holes and occluded missing regions to maintain the connectivity and closure of components at the macro-topological level. Finally, it captures the spatial pose logic and connection constraints between components to ensure that the completed non-visual regions are strictly aligned geometrically and semantically with the global overall framework.

[0041] Using the above method, hybrid geometric features containing rich local details and global semantics can be generated.

[0042] S5: The dependency relationship is fused with the first feature to generate a second feature vector.

[0043] In this way, the second feature vector includes a hybrid geometric feature that combines rich local details with global semantics.

[0044] S6: Input the global feature vector into a multilayer perceptron for dimensionality reduction, and fuse the output of the multilayer perceptron with the incomplete 3D point cloud to output a sparse skeleton point cloud.

[0045] Combination Figure 4 S7: Input the output of the multilayer perceptron and the sparse skeleton point cloud into the first structure self-attention module for geometric uncertainty analysis to output a first target region of interest. Use the point cloud features in the first target region of interest as a query vector and perform a query in the second feature vector to output a first query result. Determine a first local detail feature based on the first query result. Embed the first local detail feature into the sparse skeleton point cloud for completion and output the intermediate result of point cloud completion.

[0046] Here, the first structure self-attention module is configured to use the chamfer distance between the generated points of the sparse skeleton point cloud and the input points corresponding to the global feature vector as a measure of local reconstruction uncertainty, determine the high-entropy region based on the measure result, and determine the first target attention region based on the high-entropy region.

[0047] The point cloud at the current level (i.e., the sparse skeleton point cloud) is used as the retrieval vector to perform retrieval in the second feature vector. Since the second feature vector incorporates the geometric properties of local details of complex structures and long dependencies, the detailed features of the first target interest region, i.e. the first local detail features, can be output here. The first local detail features are embedded into the sparse skeleton point cloud for completion, and the intermediate result of point cloud completion is output.

[0048] S8: Input the output of the multilayer perceptron and the intermediate result of the point cloud completion into the second structure self-attention module for geometric uncertainty analysis to output a second target region of interest. Use the point cloud features in the second target region of interest as a query vector, perform a query in the second feature vector, output a second query result, determine a second local detail feature based on the second query result, embed the second local detail feature into the intermediate result of the point cloud completion for completion, and output the completion result of the incomplete 3D point cloud.

[0049] Combination Figure 5 The second structure self-attention module uses the chamfer distance between the intermediate result of point cloud completion and the input point corresponding to the global feature vector as a measure of local reconstruction uncertainty, determines the high-entropy region based on the measure result, and determines the second target attention region based on the high-entropy region.

[0050] In this way, the model is guided to focus on the severely missing areas again. Then, the point cloud at the current level is used as the retrieval vector to retrieve the local details corresponding to the second target area of ​​interest in the second feature vector, that is, the second local detail features. By embedding the second local detail features into the intermediate result of point cloud completion, the completion result of the incomplete 3D point cloud is output.

[0051] In the embodiments provided in this application, a sparse skeleton point cloud is generated using the aforementioned global feature vector. Through hierarchical upsampling and feature correction, the completed point cloud data is output. Using this differentiable dictionary lookup method, the point cloud coordinates are corrected using high-frequency geometric details, restoring fine structures such as thin walls and sharp corners, as well as invisible structures. This mechanism forces the model to actively retrieve semantic cues related to the currently missing geometric regions in the visual feature field. This design maximizes point-to-point mutual information, ensuring that the model strictly relies on the current visual observation data during inference, avoiding the model's degradation into a "shape template retriever" that only relies on training memory. This allows the model to accurately reflect the true individual differences (such as machining allowances) of the workpiece under test.

[0052] Therefore, in the embodiments provided in this application, by utilizing the Gaussian splashing algorithm, the present invention can generate a continuous visual feature field, establishing effective visual correspondences even in areas with very sparse point clouds. This processing method significantly improves the completeness of non-visual areas and expands the support set range of effective features. Secondly, by querying the geometric feature vectors in the visual feature field, the present invention can generate global feature vectors, capturing the long-distance dependencies of the entire component and the logical associations between sub-components. Simultaneously, the combination of the local feature extraction module and the self-attention module further enhances the capture of local surface properties and long-distance dependencies. Finally, the use of hierarchical upsampling and feature correction methods can accurately recover fine structures such as thin walls and sharp corners, as well as invisible structures, ensuring that the completed point cloud data can truly reflect the actual shape of the workpiece under test.

[0053] Referring to Figure 6(b), the following will elaborate on the aforementioned S2: inputting the incomplete 3D point cloud into the visual feature extraction module to obtain the aggregated visual feature vector at each location in the incomplete 3D point cloud in order to generate a continuous visual feature field: In the embodiments provided in this application, the visual feature extraction module can be obtained by using the Gaussian splashing algorithm.

[0054] Specifically, firstly, the RGB image of the incomplete 3D point cloud is determined based on its 3D coordinates, and then the aggregated visual feature vector of each position in the incomplete 3D point cloud is obtained using the Gaussian splashing algorithm.

[0055] Specifically, for any position The aggregated visual feature vector is calculated using the following formula: ; in, —in location The aggregated visual feature vector calculated at the location; —Falling into the neighboring area The first Three-dimensional projection points —No. Coordinates of a three-dimensional point in a projection cluster Mapped to two dimensions

[0056] —No. Each projection point pairs the query location Contribution weight; -constant; ; —No. Deterministic coordinates of a three-dimensional point projected onto a two-dimensional plane; — Coordinates of the projection point With query location The square of the Euclidean distance between them; —Bandwidth or standard deviation of the Gaussian kernel —No. The depth value of a 3D point in the camera coordinate system; -constant.

[0057] This weight design It includes two specific synergistic effects: Space Gaussian kernel ( The function of ) is: Low-pass filtering: As a spatial low-pass filter, it can effectively smooth the quantization noise generated during the projection process.

[0058] Guaranteeing gradient continuity: different from The function has a value only at the coincident points. The "tailing" property of the Gaussian function ensures that even if there is a slight misalignment between the point and the grid center, its gradient remains constant. It is also always non-zero. This allows neural networks to efficiently update geometric coordinates through backpropagation, solving the "vanishing gradient" problem.

[0059] Soft depth buffer ( The function of ) is: Micro-occlusion handling: Introducing the inverse depth term In effect, a soft Z-Buffer was constructed.

[0060] Prospect Priority: Depth Value The smaller the value (i.e., the closer it is to the foreground point of the camera), the higher its corresponding weight. The larger the value, the better. This means that during feature aggregation, the model will prioritize retaining the feature information of the foreground geometry, thereby simulating the occlusion culling mechanism in traditional computer graphics while maintaining the differentiability of the entire process, and avoiding background noise interfering with foreground features.

[0061] Referring to Figure 6(b), the discrete point cloud can be transformed into a continuous visual field using the above method, which significantly expands the range of the support set of effective features. This enables the establishment of effective visual correspondences even in regions where the point cloud is very sparse (such as deep holes or thin-walled edges), thereby significantly improving the completeness of the incompleteness of non-visual regions.

[0062] Based on the foregoing embodiments, in this application, the loss function during the training process of the aforementioned point cloud completion model is as follows: ; —The total loss value during training; —Adjustment coefficient; —Chamfer distance operator; —The completed point cloud output generated by the model; —A true and complete point cloud.

[0063] The above method balances robustness to outliers with sensitivity to small errors. Network parameters are updated via gradient backpropagation; due to the differentiability of Gaussian soft splashing, the gradient can penetrate the entire network.

[0064] Reference Figure 7 Based on the foregoing embodiments, this application provides a point cloud completion device for complex components based on cross-modal alignment, applied to a point cloud completion model, including: First extraction module: used to input the incomplete 3D point cloud into the geometric feature extraction module to extract geometric feature vectors containing 3D topological information; The second extraction module is used to input the incomplete 3D point cloud into the visual feature extraction module, and to obtain the aggregated visual feature vector of each position in the incomplete 3D point cloud to generate a continuous visual feature field. Alignment module: Performs a query in the visual feature field based on the geometric feature vector to align and match the probability density image with the visual feature field space, and generates a global feature vector; The third extraction module is used to input the incomplete 3D point cloud data into the local feature extraction module, extract a first feature vector to characterize local surface properties, and input the first feature into the self-attention module, which is used to capture the long-distance dependency of the first feature. Fusion module: used to fuse the dependency relationship with the first feature to generate a second feature vector; First output module: used to input the global feature vector into a multilayer perceptron for dimensionality reduction, and to fuse the output of the multilayer perceptron with the incomplete 3D point cloud to output a sparse skeleton point cloud; The second output module is used to input the output of the multilayer perceptron and the sparse skeleton point cloud into the first structure self-attention module for geometric uncertainty analysis, so as to output a first target region of interest, use the point cloud features in the first target region of interest as a query vector, perform a query in the second feature vector, output a first query result, determine a first local detail feature based on the first query result, embed the first local detail feature into the sparse skeleton point cloud for completion, and output the intermediate result of point cloud completion. The third output module is used to input the output of the multilayer perceptron and the intermediate result of the point cloud completion into the second structural self-attention module for geometric uncertainty analysis, so as to output a second target region of interest. The point cloud features in the second target region of interest are used as query vectors, and a query is performed in the second feature vectors to output a second query result. Based on the second query result, a second local detail feature is determined. Based on the second local detail feature, the point cloud completion is embedded into the intermediate result of the point cloud completion for completion, and the completion result of the incomplete 3D point cloud is output.

[0065] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0066] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0067] Figure 8A block diagram is shown that is suitable for implementing embodiments of the present invention. Figure 8 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0068] like Figure 8 As shown, the electronic device is represented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: one or more processors 410, memory 430, and communication bus 440 connecting different system components (including memory 430 and processing unit 410).

[0069] Communication bus 440 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MAC) buses, Enhanced ISA buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses.

[0070] Electronic devices typically include a variety of computer-readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, and removable and non-removable media.

[0071] Memory 430 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. Memory 430 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0072] A program / utility having a set (at least one) of program modules can be stored in memory 430. Such program modules include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules typically perform the functions and / or methods described in the embodiments of this invention.

[0073] Processor 410 executes various functional applications and data processing by running programs stored in memory 430, such as implementing embodiments of the present invention. Figure 1 The method provided in the illustrated embodiment.

[0074] This invention provides a non-transitory computer-readable storage medium that stores computer instructions, which cause the computer to execute embodiments of this invention. Figure 1 The method provided in the illustrated embodiment.

[0075] The aforementioned computer-readable storage medium may be any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by or in connection with an instruction execution system, apparatus, or device.

[0076] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0077] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0078] Computer program code for performing the operations of embodiments of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0079] The foregoing has described specific embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0080] In the description of the embodiments of the present invention, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In the embodiments of the present invention, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in the embodiments of the present invention, as well as the features of the different embodiments or examples.

[0081] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of embodiments of the present invention, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0082] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.

[0083] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0084] It should be noted that the terminals involved in the embodiments of the present invention may include, but are not limited to, personal computers (PCs), personal digital assistants (PDAs), wireless handheld devices, tablet computers, mobile phones, MP3 players, MP4 players, etc.

[0085] In the several embodiments provided in this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0086] Furthermore, in the various embodiments of the present invention, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0087] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for completing point clouds of complex components based on cross-modal alignment, characterized in that, The method, applied to point cloud completion models, includes: The incomplete 3D point cloud of the target component is input into the geometric feature extraction module to extract geometric feature vectors containing 3D topological information; The incomplete 3D point cloud is input into the visual feature extraction module to obtain the aggregated visual feature vector of each position in the incomplete 3D point cloud, so as to generate a continuous visual feature field. Based on the geometric feature vector, a query is performed in the visual feature field to align and match the probability density image with the visual feature field space, thereby generating a global feature vector. The incomplete 3D point cloud data is input to the local feature extraction module to extract a first feature vector that characterizes local surface properties. The first feature is then input to the self-attention module, which is used to capture the long-distance dependencies of the first feature. The dependency relationship is fused with the first feature to generate a second feature vector; The global feature vector is input into a multilayer perceptron for dimensionality reduction, and the output of the multilayer perceptron is fused with the incomplete 3D point cloud to output a sparse skeleton point cloud. The output of the multilayer perceptron and the sparse skeleton point cloud are input into the first structure self-attention module for geometric uncertainty analysis to output a first target region of interest. The point cloud features in the first target region of interest are used as a query vector to perform a query in the second feature vector and output a first query result. Based on the first query result, a first local detail feature is determined. Based on the first local detail feature, it is embedded into the sparse skeleton point cloud for completion and outputs the intermediate result of point cloud completion. The output of the multilayer perceptron and the intermediate result of the point cloud completion are input into the second structural self-attention module for geometric uncertainty analysis to output a second target region of interest. The point cloud features in the second target region of interest are used as query vectors, and a query is performed in the second feature vectors to output a second query result. Based on the second query result, a second local detail feature is determined. Based on the second local detail feature, the point cloud completion intermediate result is embedded to complete the incomplete 3D point cloud, and the completion result of the incomplete 3D point cloud is output.

2. The method according to claim 1, characterized in that, The method further includes: The incomplete 3D point cloud is translated by coordinates and scaled diagonally to project the incomplete 3D point cloud onto the target area.

3. The method according to claim 2, characterized in that, In the step of inputting the incomplete 3D point cloud into the visual feature extraction module to obtain the aggregated visual feature vector at each location in the incomplete 3D point cloud in order to generate a visual feature field, The RGB image of the incomplete 3D point cloud is determined based on the 3D coordinates of the incomplete 3D point cloud; The RGB image is input into the visual feature extraction module, and the Gaussian splashing algorithm is used to obtain the aggregated visual feature vector of each position in the incomplete 3D point cloud.

4. The method according to claim 3, characterized in that, In the step of inputting the RGB image into the visual feature extraction module and using the Gaussian splashing algorithm to obtain the aggregated visual feature vector of each position in the incomplete 3D point cloud, For any position The aggregated visual feature vector is calculated using the following formula: ; in, —in location The aggregated visual feature vector calculated at the location; —Falling into the neighboring area The first Three-dimensional projection points; —No. Coordinates of a three-dimensional point in a projection cluster Mapped to two dimensions ; —No. Each projection point pairs the query location Contribution weight; -constant; ; —No. Deterministic coordinates of a three-dimensional point projected onto a two-dimensional plane; — Coordinates of the projection point With query location The square of the Euclidean distance between them; —The bandwidth or standard deviation of the Gaussian kernel; —No. The depth value of a 3D point in the camera coordinate system; -constant.

5. The method according to any one of claims 1 to 4, characterized in that, In the step of inputting the incomplete 3D point cloud data into the local feature extraction module to extract a first feature vector representing local surface properties, and inputting the first feature into the self-attention module, the self-attention module is used to capture the long-distance dependencies of the first feature: Local feature extraction module: configured to acquire incomplete 3D point cloud, and extract the mean curvature and tangent space information of the incomplete 3D point cloud by performing edge convolution operation to approximate the Laplacian operator on the manifold. Local surface attributes include mean curvature and tangent space information. The self-attention module is specifically used to: receive a first feature vector and capture long-distance dependencies based on a global message passing mechanism, the dependencies including: the hole structure, symmetric structure and long-distance links of the 3D point cloud.

6. The method according to claim 5, characterized in that, The output of the multilayer perceptron and the sparse skeleton point cloud are input into the first structure self-attention module for geometric uncertainty analysis to output a first target region of interest. The point cloud features in the first target region of interest are used as a query vector to perform a query in the second feature vector and output a first query result. Based on the first query result, a first local detail feature is determined. Based on the first local detail feature, it is embedded into the sparse skeleton point cloud for completion and an intermediate result of point cloud completion is output. and The steps of inputting the output of the multilayer perceptron and the intermediate result of the point cloud completion into the second structural self-attention module for geometric uncertainty analysis to output a second target region of interest; using the point cloud features in the second target region of interest as a query vector; querying within the second feature vector; outputting a second query result; determining a second local detail feature based on the second query result; embedding the second local detail feature into the intermediate result of the point cloud completion for completion; and outputting the completed result of the incomplete 3D point cloud are then discussed. The first self-attention module is configured to use the chamfer distance between the generated points of the sparse skeleton point cloud and the input points corresponding to the global feature vector as a measure of local reconstruction uncertainty, determine the high-entropy region based on the measurement result, and determine the first target attention region based on the high-entropy region. The second self-attention module is configured to use the chamfer distance between the intermediate result of point cloud completion and the input point corresponding to the global feature vector as a measure of local reconstruction uncertainty, determine the high-entropy region based on the measure result, and determine the second target attention region based on the high-entropy region.

7. The method according to claim 5, characterized in that, During the training of the point cloud completion model, the loss function is as follows: ; —The total loss value during training; —Adjustment coefficient; —Chamfer distance operator; —The completed point cloud output generated by the model; —A true and complete point cloud.

8. A point cloud completion device for complex components based on cross-modal alignment, characterized in that, Applied to point cloud completion models, including: First extraction module: used to input the incomplete 3D point cloud into the geometric feature extraction module to extract geometric feature vectors containing 3D topological information; The second extraction module is used to input the incomplete 3D point cloud into the visual feature extraction module, and to obtain the aggregated visual feature vector of each position in the incomplete 3D point cloud to generate a continuous visual feature field. Alignment module: Performs a query in the visual feature field based on the geometric feature vector to align and match the probability density image with the visual feature field space, and generates a global feature vector; The third extraction module is used to input the incomplete 3D point cloud data into the local feature extraction module, extract a first feature vector to characterize local surface properties, and input the first feature into the self-attention module, which is used to capture the long-distance dependency of the first feature. Fusion module: used to fuse the dependency relationship with the first feature to generate a second feature vector; First output module: used to input the global feature vector into a multilayer perceptron for dimensionality reduction, and to fuse the output of the multilayer perceptron with the incomplete 3D point cloud to output a sparse skeleton point cloud; The second output module is used to input the output of the multilayer perceptron and the sparse skeleton point cloud into the first structure self-attention module for geometric uncertainty analysis, so as to output a first target region of interest, use the point cloud features in the first target region of interest as a query vector, perform a query in the second feature vector, output a first query result, determine a first local detail feature based on the first query result, embed the first local detail feature into the sparse skeleton point cloud for completion, and output the intermediate result of point cloud completion. The third output module is used to input the output of the multilayer perceptron and the intermediate result of the point cloud completion into the second structural self-attention module for geometric uncertainty analysis, so as to output a second target region of interest. The point cloud features in the second target region of interest are used as query vectors, and a query is performed in the second feature vectors to output a second query result. Based on the second query result, a second local detail feature is determined. Based on the second local detail feature, the point cloud completion is embedded into the intermediate result of the point cloud completion for completion, and the completion result of the incomplete 3D point cloud is output.

9. An electronic device, characterized in that, include: At least one processor; as well as At least one memory communicatively connected to the processor, wherein: The memory stores program instructions that can be executed by the processor, and the processor can invoke the program instructions to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause the computer to perform the method as described in any one of claims 1 to 7.

Images