A method for completing an ancient building point cloud model based on multi-modal feedback and feature purification

Abstract

The application discloses a kind of based on multi-modal feedback and feature purification ancient building point cloud model completion method, including calculating the local density of each three-dimensional coordinate point in ancient building defect point cloud model, obtain adaptive neighbor quantity, utilize dynamic graph convolution network to process feature descriptor, obtain the component class to which three-dimensional coordinate point belongs;Retain the component class to which the three-dimensional coordinate point corresponding to the component class of classification confidence higher than dynamic confidence threshold belongs, obtain final belonging component class, obtain purified feature based on the class;Construct two-dimensional uniform grid point set, splice purified feature with grid point coordinates in two-dimensional uniform grid point set, obtain high-dimensional feature vector, utilize folding decoder to obtain the component point cloud set after completion, and then obtain the point cloud set after smoothing, utilize multi-modal large language model to process the set, complete point cloud completion.The application realizes high-fidelity fine repair, guarantees the completion stability under full scene.

Description

Technical Field

[0001] This invention relates to the fields of three-dimensional computer vision and digital heritage protection technology, specifically to a method for completing point cloud models of ancient buildings based on multimodal feedback and feature purification. Background Technology

[0002] In recent years, 3D point cloud scanning technology has been widely applied in cultural heritage protection, ancient building restoration, and virtual reconstruction. Using technologies such as laser scanning, photogrammetry, and structured light scanning, researchers can acquire high-precision 3D point cloud data of ancient buildings, providing a data foundation for their digital archiving and restoration. However, due to long-term weathering, earthquakes, and human-caused damage, many ancient buildings have suffered partial or large-scale component loss, resulting in existing point cloud data failing to fully reflect the building structure and hindering subsequent digital restoration and engineering repair. Therefore, effectively completing the missing point cloud data has become a key technical challenge in this field. Existing technical solutions can be broadly categorized into traditional geometry-driven methods and data-driven deep learning methods.

[0003] Early restoration work primarily relied on traditional geometry-driven methods. For example, document 1 (Li Haipeng. Research on Virtual Restoration Method of Cultural Relics Based on Point Cloud Processing [D]. Yunnan University, 2022. DOI:10.27456 / d.cnki.gyndu.2022.002536) mainly analyzed the boundaries of the damaged area and used mathematical surfaces such as Poisson surfaces and radial basis functions for fitting and interpolation. This method is effective for handling small-area holes with simple and smooth geometric shapes, but its core flaw lies in the lack of understanding of the global shape of the object. When faced with complex defects covering large areas and spanning multiple structural components, it cannot infer the complex form that conforms to the original structural logic, and the restoration results are often too smooth, losing the original edges, carvings, and other high-frequency details of the building or cultural relic.

[0004] With the growth of data volume, data-driven deep learning methods have gradually become mainstream. These methods train neural networks on large datasets, enabling them to learn rich shape priors. However, the limitations of these methods are also quite obvious: compressing the entire complex structure into a single global feature inevitably leads to the loss of local fine details, and the generated surfaces are often too blurry or have an average shape, failing to reproduce the complex geometric features of specific cultural relics and ancient buildings. In addition, the performance of data-driven deep learning methods is highly dependent on the distribution of the training data. When dealing with unique architectural structures that differ significantly from the training set, the generalization ability is insufficient, and the performance will drop significantly.

[0005] Therefore, recent research has begun to explore strategies that incorporate external information or prior knowledge for guidance. For example, document 2 (Gao S, Jiao C, Chen R, et al. Point Cloud Completion Guided by PriorKnowledge via Causal Inference[J]. arXiv preprint arXiv:2305.17770, 2023) explores how to use explicit structural symmetry, component relationships, and other prior knowledge to guide the completion network, improving the accuracy of the results. Document 3 (Lu Z, Jiao Q, Xu L. MMCNet: Multi-Modal Point Cloud Completion Network Based on Self-Projected Views[C] / / 2024 6th InternationalConference on Data-driven Optimization of Complex Systems (DOCS). IEEE, 2024:897-902) employs multimodal data, combining high-resolution two-dimensional images to provide richer clues for three-dimensional geometry generation. Nevertheless, these methods still face insurmountable obstacles: methods based on explicit prior knowledge have complex knowledge base constructions that struggle to cover the infinite variations of all buildings and artifacts, thus limiting their versatility; while multimodal methods heavily rely on additional high-quality, precisely registered data (such as images), which are often difficult to obtain at many archaeological sites or under scanning conditions.

[0006] In summary, when dealing with complex, detailed, and often isolated digital objects such as ancient buildings, existing technologies still suffer from core problems including inaccurate component segmentation, insufficient understanding of structural relationships, weak noise suppression, and the difficulty in balancing detail preservation and global smoothness. Therefore, there is an urgent need for a novel technical solution that can perform structural awareness and high-fidelity restoration of complex defect cloud models without requiring a large number of similar samples or additional modal data. Summary of the Invention

[0007] The technical problem to be solved by this invention is: in order to solve the problem of preserving high-frequency geometric details while suppressing weathering and erosion noise, and the lack of semantic constraints in pure geometric completion, a method for completing point cloud models of ancient buildings based on multimodal feedback and feature purification is provided, which significantly improves the robustness of damaged and deformed ancient building data and achieves high-fidelity and refined restoration.

[0008] To solve the above technical problems, the present invention adopts the following technical solution:

[0009] A method for completing point cloud models of ancient buildings based on multimodal feedback and feature purification includes the following steps:

[0010] S1. Obtain the actual point cloud model of the ancient building with defects, and calculate the local density of each three-dimensional coordinate point in the point cloud model with defects; obtain the adaptive number of neighbors based on the local density of each three-dimensional coordinate point, and dynamically adjust the neighborhood search range.

[0011] S2. Construct neighborhoods with different radii for each 3D coordinate point, calculate the covariance matrix of these neighborhoods using principal component analysis, and obtain explicit geometric features; construct a local geometric nearest neighbor graph, calculate the discrete Laplace-Beltrami operator matrix, and perform eigenvalue decomposition on this operator matrix to obtain the corresponding frontier features. 1 eigenvector, used as manifold harmonic spectrum features;

[0012] S3. The coordinates, local density, explicit geometric features, and manifold harmonic spectrum features of each 3D coordinate point are concatenated to obtain a feature descriptor. Based on the adaptive number of neighbors, a pre-trained dynamic graph convolutional network is used to extract features from the feature descriptor and aggregate neighborhood information to obtain the component category to which the 3D coordinate point belongs.

[0013] S4. Obtain the classification confidence of each three-dimensional coordinate point. When the classification confidence is higher than the dynamic confidence threshold, retain the component category to which the three-dimensional coordinate point belongs. Otherwise, mark it as unclassified to obtain the final component category.

[0014] S5. Construct a set of component point clouds based on the final component category, obtain the component feature vectors corresponding to the component point clouds in the set, input the vectors into the Transformer module, capture the long-distance dependencies between components, and obtain context-aware features.

[0015] S6. Input the context-aware features into the pre-trained gated attention cleansing model to obtain the original gated vector. Use the gated truncation threshold to correct the original gated vector. Use the corrected gated vector to perform weighted filtering of the context-aware features to obtain the cleaned features.

[0016] S7, Construction includes A two-dimensional uniform grid set of grid points will be used to replicate the purification features. The part is divided into parts and concatenated with the coordinates of each grid point in the two-dimensional uniform grid point set to obtain a high-dimensional feature vector. This high-dimensional feature vector is then input into a pre-trained folding decoder to obtain the complete part point cloud set.

[0017] S8. Combine the completed part point cloud set in the original coordinate system to obtain the final point cloud set. Improve the gradient field of each part's 3D coordinate point in the final point cloud set. Based on the smoothing factor, the improved gradient field, and the Poisson equation, obtain the smoothed target vertex. The smoothed target vertex constitutes the smoothed point cloud set.

[0018] S9. After generating a rendered image from the smoothed point cloud set, input it into a pre-trained multimodal large language model for evaluation. Guide the visual language model to evaluate the material and age of the ancient building through prompt words. Based on the evaluation results, obtain a set of structured parameter adjustment instructions for each component. Update the gating truncation threshold in step S6, the number of grid points in the two-dimensional uniform grid point set in step S7, and the smoothing factor in step S8 according to the instruction set until the evaluation is passed, and complete the point cloud model completion.

[0019] Furthermore, in step S1, obtaining the adaptive number of neighbors includes the following:

[0020] The formula for calculating local density is:

[0021] ;

[0022] in, This represents the i-th three-dimensional coordinate point. This represents the local density of the i-th 3D coordinate point. This indicates that the i-th three-dimensional coordinate point is located at a radius of... The neighborhood point set within, Indicates the attenuation coefficient. This represents the j-th neighboring 3D coordinate point of the i-th 3D coordinate point. express and The square of the Euclidean distance between them Represents an exponential function;

[0023] The formula for calculating the adaptive number of neighbors is:

[0024] ;

[0025] in, This represents the adaptive number of neighbors for the i-th 3D coordinate point. This represents the initial value indicating the number of neighbors. Indicates the balance coefficient;

[0026] In step S2, three neighborhoods with different radii are constructed for each 3D coordinate point. Then, for the i-th 3D coordinate point... Neighborhood The covariance matrix is ​​a 3×3 matrix. The motivation for using multiple scales is that a single-scale neighborhood cannot simultaneously capture fine structures (such as sculpted edges) and macroscopic shapes (such as the smoothness of a wall). The specific formula for obtaining explicit geometric features is:

[0027] ;

[0028] ;

[0029] ;

[0030] in, Indicates radius as Linearity within the neighborhood; Indicates radius as Flatness within the neighborhood; Indicates radius as The divergence within the neighborhood; This represents the maximum eigenvalue, which usually corresponds to the direction of maximum change in the local point set distribution, i.e., the extent to which the point cloud extends along that direction; This represents an intermediate eigenvalue, which typically corresponds to the second largest direction of change. This represents the minimum eigenvalue, which usually corresponds to the minimum direction of change, i.e., the thickness or degree of change of the point cloud perpendicular to the surface (normal direction); , Indicates the total number of scales;

[0031] Explicit geometric features include linearity, flatness, and divergence;

[0032] calculate and Gaussian weights between The specific formula is as follows:

[0033] ;

[0034] in, Indicates the thermal diffusivity;

[0035] Construct a normalized Laplace-Beltrami operator matrix of dimension N*N The specific formula is as follows:

[0036] ;

[0037] in, Represents the identity matrix. Degree matrix, Indicates Gaussian weights;

[0038] right Perform eigenvalue decomposition to obtain the corresponding eigenvalues. and eigenvectors Select the preceding mode corresponding to the low-frequency vibration mode. The eigenvectors corresponding to the smallest non-zero eigenvalues ​​are used to obtain the harmonic spectrum features of the manifold, as shown in the following formula:

[0039] ;

[0040] in, express Manifold harmonic spectrum characteristics, express The first of the manifold harmonic spectrum characteristics 1 eigenvector.

[0041] Furthermore, in step S3, the specific formula for obtaining the feature descriptor is as follows:

[0042] ;

[0043] in, The feature descriptor representing the i-th 3D coordinate point, This indicates a splicing operation. Indicates radius as linearity within the neighborhood, Indicates radius as Flatness within the neighborhood, Indicates radius as The divergence within the neighborhood, Indicates radius as linearity within the neighborhood, Indicates radius as Flatness within the neighborhood, Indicates radius as The divergence within the neighborhood, The first characteristic in the harmonic spectrum of a manifold 1 eigenvector;

[0044] Based on an adaptive number of neighbors, a k-NN graph is constructed in the enhanced feature space, and adjacency relationships are generated;

[0045] Dynamic graph convolutional networks consist of edge convolutional layers, max pooling layers, and a classification head;

[0046] The feature descriptors are input into a pre-trained dynamic graph convolutional network. After passing through edge convolutional layers, the edge convolution operation is used to aggregate neighborhood information and extract local structural features. The specific expression is as follows:

[0047] ;

[0048] in, This represents the local structural features of the i-th three-dimensional coordinate point. This represents the nearest neighbor of the adaptive neighborhood of the i-th 3D coordinate point. This represents a learnable function implemented by a multilayer perceptron. express Feature descriptors;

[0049] The local structural features are max-pooled to obtain the global structural features. These local and global features are then concatenated, processed by a classification head, and based on adjacency relationships, the component category of the 3D coordinate point is determined. The specific formula is as follows:

[0050] ;

[0051] in, This indicates the component category to which the i-th 3D coordinate point belongs. This represents the probability normalization function. This represents a multilayer perceptron. This represents global structural features.

[0052] Furthermore, in step S4, the formula for calculating the classification confidence is:

[0053] ;

[0054] in, This represents the classification confidence score of the i-th 3D coordinate point. This represents the highest probability that the predicted 3D coordinate point belongs to the component category. This represents the second-highest probability predicted for the i-th 3D coordinate point. This represents the i-th three-dimensional coordinate point;

[0055] The expression for the dynamic confidence threshold is:

[0056] ;

[0057] in, This represents the dynamic confidence threshold for the i-th 3D coordinate point. This represents the initial value of the dynamic confidence threshold. Indicates the regulating factor;

[0058] The formula for determining the final component category is:

[0059] ;

[0060] in, This indicates the final component category to which the i-th 3D coordinate point belongs.

[0061] Furthermore, in step S5, the 3D coordinate points of the same category within the final component category are constructed into a component point cloud set. An encoder network is used to extract the initial component feature vectors corresponding to the 3D coordinate points in this set, and the semantic vectors corresponding to the 3D coordinate points in this set are obtained. The initial component feature vectors and the semantic vectors are then added element-wise to obtain the component feature vectors. The specific formula is as follows:

[0062] ;

[0063] in, Indicates the first In the point cloud set of the component, the first The component feature vector of each component's three-dimensional coordinate points. Indicates the first In the point cloud set of the component, the first The initial component feature vector of each component's three-dimensional coordinate points. Indicates the first In the point cloud set of the component, the first The semantic vector of the three-dimensional coordinate points of each component Indicates the first Three-dimensional coordinates of each component , This indicates the total number of three-dimensional coordinate points of the component;

[0064] Inputting the component feature vector into the Transformer module yields the corresponding query matrix, key matrix, and value matrix. The specific formula is as follows:

[0065] ;

[0066] ;

[0067] ;

[0068] in, Represents the query matrix. Represents the key matrix, Represents a value matrix, The linear mapping matrix representing the query matrix. The linear mapping matrix representing the key matrix. The linear mapping matrix representing the value matrix. Indicates the first In the point cloud set of the component, the first The component feature vector of each component's three-dimensional coordinate points, superscript T Indicates transpose;

[0069] Attention weights are obtained using standard scaled dot product attention, thereby capturing long-distance dependencies between components. The specific formula is as follows:

[0070] ;

[0071] in, This represents the attention mechanism. This represents the probability normalization function. express transpose, Indicates the dimension of the key vector;

[0072] Long-range dependencies are encoded into the output features of each component to obtain context-aware features. .

[0073] Furthermore, in step S6, the purification characteristics obtained include the following:

[0074] Context-aware features are input into a pre-trained gated attention cleansing model. Structural and relational information is extracted using standard Transformer blocks to obtain latent structural features. These latent structural features are then processed sequentially through an MLP network and a Sigmoid activation function to generate the original gated vector. A gate truncation threshold is then used... The original gating vector is modified to obtain the modified gating vector, and the specific formula is as follows:

[0075] ;

[0076] in, Indicates the first The corrected gating vector in each dimension, Indicates the first The original gated vector of each dimension;

[0077] The context-aware features are weighted and filtered using the modified gating vector, and the purified features are obtained by element-wise multiplication. The specific formula is as follows:

[0078] ;

[0079] in, Indicates the first In the point cloud set of the component, the first The purification features of each component's three-dimensional coordinate points Indicates the first In the point cloud set of the component, the first Structural features of the three-dimensional coordinate points of each component It represents the Hadamardi (or Hadama) stack;

[0080] The specific formula for the separated suppressed noise features is:

[0081] ;

[0082] in, Indicates the first In the point cloud set of the component, the first Noise characteristics of three-dimensional coordinate points of each component.

[0083] Furthermore, in step S7, a process is performed within the two-dimensional unit square, including... A two-dimensional uniform grid point set of grid points ;

[0084] Copy the purification features The data is then combined with the coordinates of each grid point in a two-dimensional uniform grid to form a high-dimensional feature vector. The specific formula is as follows:

[0085] ;

[0086] in, Indicates the first The target coordinates of a uniform grid Indicates the first The target vertex of a uniform grid;

[0087] The high-dimensional feature vector is input into a pre-trained folding decoder composed of multilayer perceptrons. Through nonlinear transformation, the two-dimensional uniform grid points and their associated global features are mapped and folded into three-dimensional space, outputting three-dimensional coordinates to obtain the completed part point cloud set. The specific formula is as follows:

[0088] ;

[0089] in, This represents the complete point cloud set of components. This indicates a folded decoder.

[0090] Furthermore, in step S8, the completed component point cloud set is combined in the original coordinate system to obtain the final point cloud set. The specific formula is as follows:

[0091] ;

[0092] in, This represents the total number of point cloud sets for each component;

[0093] The gradient field of each component's 3D coordinate point in the final point cloud set is improved by utilizing local curvature, resulting in the improved gradient field. The specific formula is as follows:

[0094] ;

[0095] in, Represents the gradient operator. Represents the gradient field. This represents the improved gradient field. Indicates the first The weights of the three-dimensional coordinate points of each component Indicates the first The normal vector of the three-dimensional coordinate points of each component Indicates the first Local curvature of three-dimensional coordinate points of each component ;

[0096] Based on the improved gradient field and Poisson equation, the three-dimensional scalar field is obtained, with the specific formula as follows:

[0097] ;

[0098] in, Indicates Fourier transform, Represents a three-dimensional scalar field. Represents the frequency vector. Represents the regularization parameter. Represents the imaginary unit;

[0099] A triangular mesh is obtained by processing a three-dimensional scalar field using an adaptive threshold; where the adaptive threshold... The expression is:

[0100] ;

[0101] in, This represents the initial value of the adaptive threshold. Indicates the first The probability of building damage at three-dimensional coordinates of individual components. All represent equilibrium parameters;

[0102] The Laplacian smoothing method is used to process the triangular mesh to obtain the smoothed target vertices. The specific formula is as follows:

[0103] ;

[0104] in, Represents the smoothed first... A target vertex of a uniform grid. Represents the smoothing factor. Indicates the first The neighborhood point set of the three-dimensional coordinate points of each component Indicates the first The three-dimensional coordinates of the component and its first component The weights between neighboring points Indicates the first The target vertex of the neighborhood points.

[0105] Furthermore, in step S9, the smoothed point cloud set is subjected to multi-view projection rendering to obtain the corresponding rendered image. In the rendered image, unique component index labels are superimposed on the point clouds belonging to different component sets to generate a two-dimensional image set containing semantic mapping relationships between components. This two-dimensional image set is input into a pre-trained multimodal large language model. Prompt words guide the model to perform visual quality assessment on each independent component in the two-dimensional image set, automatically identifying and semantically inferring the age characteristics, component materials, and craftsmanship style of the ancient building. Based on this, the geometric shape and surface condition of each component are evaluated for quality, resulting in a structured parameter adjustment instruction set for each component. , , Indicates the first Each independent component This indicates the corresponding defect type; the structured parameter adjustment instruction set includes the target component index, defect type, and parameter adjustment range.

[0106] The structured parameter adjustment instruction set is parsed, and the results are processed as follows:

[0107] (1) When the instruction set indicates the first When the defect type of an individual component is excessively smooth texture, reduce the smoothing factor. And increase the balance parameter The specific formula is as follows:

[0108] ;

[0109] ;

[0110] in, All of these represent the preset adjustment step size coefficient. , Indicates the first Smoothing factor adjusted for each individual component Indicates the first Smoothing factor for each individual component Indicates the first Balance parameters adjusted for each individual component Indicates the first Balance parameters of individual components;

[0111] based on and The process of obtaining the smoothed target vertex is repeated until the score of the pre-trained multimodal large language model reaches the threshold.

[0112] (2) When the instruction set instructs the first When the defect type of an individual component is sparse point cloud, increase the number of grid points in step S7, specifically using the following formula:

[0113] ;

[0114] in, Indicates the first The number of grid points after adjustment for each individual component Indicates the first The number of grid points for each individual component This represents the preset grid increment constant;

[0115] based on Repeat steps S7-S8 until the score of the pre-trained multimodal large language model reaches the threshold.

[0116] (3) When the instruction set instructs the first When the defect type of an individual component is residual noise, increase the gate cutoff threshold in step S6, as shown in the following formula:

[0117] ;

[0118] in, Indicates the first The gating cutoff threshold is adjusted for each individual component. No. Gating cutoff threshold for each individual component Indicates the preset truncation increment;

[0119] based on Repeat steps S6-S8 until the score of the pre-trained multimodal large language model reaches the threshold.

[0120] The smoothed point cloud set obtained by re-execution is combined with the unadjusted point clouds of other components in the original coordinate system to obtain the final point cloud set after completion, thus completing the point cloud model.

[0121] Furthermore, pre-training of the dynamic graph convolutional network, the gated attention cleansing model, and the folded decoder includes:

[0122] The specific formula for calculating the overall probability of loss of three-dimensional coordinate points in ancient buildings is as follows:

[0123] ;

[0124] in, This represents the overall defect probability of the i-th three-dimensional coordinate point; Both represent weight parameters; Represents the curvature factor of the i-th three-dimensional coordinate point; This represents the gradient factor of the normal vector of the i-th three-dimensional coordinate point; Indicates weathering and erosion factor; Indicates the random damage factor;

[0125] Set defect threshold Remove the three-dimensional coordinate points whose overall defect probability is greater than the defect threshold to obtain the defect point cloud. The specific formula is as follows:

[0126] ;

[0127] in, Represents a complete set of point clouds;

[0128] Defective point cloud As the training set, the dynamic graph convolutional network, the gated attention sanitizing model, and the folded decoder are trained by backpropagation using the multi-task joint loss function, respectively, to obtain the pre-trained dynamic graph convolutional network, gated attention sanitizing model, and folded decoder; the specific formula of the multi-task joint loss function is as follows:

[0129] ;

[0130] in, This represents the joint loss function for multiple tasks. Represents the reconstruction loss function. Represents the balance coefficient. Represents the orthogonality loss function;

[0131] The curvature of a three-dimensional coordinate point is obtained using Gaussian curvature, and the specific expression is as follows:

[0132] ;

[0133] in, Represents the curvature of the i-th three-dimensional coordinate point. This represents the minimum principal radius of curvature in the neighborhood of the i-th 3D coordinate point. This represents the maximum principal radius of curvature in the neighborhood of the i-th three-dimensional coordinate point;

[0134] The curvature factor is obtained by normalizing the curvature.

[0135] Calculate the normal vector of each 3D coordinate point Then calculate the spatial gradient of the normal vector. ,Will Normalization is performed to obtain the gradient factor of the normal vector;

[0136] The weathering and exfoliation factor was obtained using the Poisson point process;

[0137] Each three-dimensional coordinate point is assigned a random number uniformly distributed in the interval [0, 1] to obtain the random damage factor. .

[0138] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0139] 1. By introducing manifold harmonic spectrum features and utilizing the isometric transformation invariance of the Laplace-Beltrami operator, this invention can extract fingerprint features that reflect the inherent topological structure of components, effectively overcoming the identification and completion bias caused by the non-rigid deformation of ancient building components, and significantly improving the robustness of damaged and deformed ancient building data.

[0140] 2. This invention designs a gated attention cleanup model and introduces an orthogonality loss function. By forcing noise features and structural features to remain orthogonal in the feature space, and combining a gating mechanism to adaptively filter information, it can accurately strip away random scanning noise (such as flying points) while preserving high-frequency texture details with structural regularity to the greatest extent, achieving a balance between noise suppression and high-frequency detail preservation. In addition, by combining gradient field optimization based on local curvature improvement, it further sharpens the edges and corners, achieving high-fidelity fine-grained repair.

[0141] 3. This invention utilizes the powerful visual understanding capabilities of a multimodal large language model to perform instance-level diagnosis on the completion results, such as excessively smooth texture and sparse point clouds. The diagnosis results are then converted into adaptive adjustment instructions for mathematical parameters (such as smoothing factors and thresholds), ensuring that the final completion set not only has low geometric errors but also conforms to the visual aesthetics and structural logic of ancient architecture, thus breaking through the limitations of optimizing a single geometric index.

[0142] 4. By calculating local density, adaptively adjusting the number of neighbors and dynamic confidence threshold, this invention can automatically expand the search range in sparse point cloud regions to capture sufficient information, and focus on local details in dense regions. This avoids feature extraction failure or misclassification caused by differences in sampling density, ensures completion stability in all scenarios, and improves the adaptability to point cloud data with uneven density. Attached Figure Description

[0143] Figure 1 This is a flowchart illustrating the overall implementation of the present invention.

[0144] Figure 2 This is a comparison diagram of the classification effects of manifold harmonic spectrum features and traditional geometric features in an embodiment of the present invention.

[0145] Figure 3 This is a schematic diagram illustrating the purification features obtained using a gated attention purification model in an embodiment of the present invention.

[0146] Figure 4 This is a schematic diagram illustrating the point cloud completion process of this invention.

[0147] Figure 5 This is a result image of point cloud repair for complex and deformed structures in an embodiment of the present invention.

[0148] Figure 6 This is a point cloud repair result diagram of a large defect structure in an embodiment of the present invention. Detailed Implementation

[0149] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0150] To achieve the above objectives, this invention provides a method for completing point cloud models of ancient buildings based on multimodal feedback and feature purification, such as... Figure 1 As shown, the specific steps are as follows:

[0151] S1. Obtain the actual missing point cloud model of the ancient building, and calculate the local density of each 3D coordinate point in the missing point cloud model; based on the local density of each 3D coordinate point, obtain an adaptive number of neighbors, and dynamically adjust the neighborhood search range to adapt to the uneven density of the ancient building's scanned point cloud, guiding the subsequent graph convolutional network to construct a more reasonable local graph structure in different density regions; specifically:

[0152] The formula for calculating local density is:

[0153] ;

[0154] in, This represents the i-th three-dimensional coordinate point. This represents the local density of the i-th 3D coordinate point. This indicates that the i-th three-dimensional coordinate point is located at a radius of... The neighborhood point set within, Indicates the attenuation coefficient. This represents the j-th neighboring 3D coordinate point of the i-th 3D coordinate point. express and The square of the Euclidean distance between them Represents an exponential function;

[0155] The formula for calculating the adaptive number of neighbors is:

[0156] ;

[0157] in, This represents the adaptive number of neighbors for the i-th 3D coordinate point. This represents the initial value indicating the number of neighbors. This represents the balance coefficient.

[0158] S2. Construct neighborhoods with different radii for each 3D coordinate point, calculate the covariance matrix of these neighborhoods using principal component analysis, and obtain explicit geometric features; construct a local geometric nearest neighbor graph, calculate the discrete Laplace-Beltrami operator matrix, and perform eigenvalue decomposition on this operator matrix to obtain the corresponding frontier features. 1 eigenvector is used as the harmonic spectrum feature of the manifold; specifically:

[0159] Construct three neighborhoods with different radii for each 3D coordinate point, then the i-th 3D coordinate point... Neighborhood The covariance matrix is ​​a 3×3 matrix. The motivation for using multiple scales is that a single-scale neighborhood cannot simultaneously capture fine structures (such as sculpted edges) and macroscopic shapes (such as the smoothness of a wall). The specific formula for obtaining explicit geometric features is:

[0160] ;

[0161] ;

[0162] ;

[0163] in, Indicates radius as Linearity within the neighborhood; Indicates radius as Flatness within the neighborhood; Indicates radius as The divergence within the neighborhood; This represents the maximum eigenvalue, which usually corresponds to the direction of maximum change in the local point set distribution, i.e., the extent to which the point cloud extends along that direction; This represents an intermediate eigenvalue, which typically corresponds to the second largest direction of change. This represents the minimum eigenvalue, which usually corresponds to the minimum direction of change, i.e., the thickness or degree of change of the point cloud perpendicular to the surface (normal direction); , Indicates the total number of scales;

[0164] Explicit geometric features include linearity, flatness, and divergence;

[0165] For example, a point located on the edge of a column will exhibit high linearity at a certain scale, while a point located on the surface of a wall will exhibit high flatness.

[0166] To overcome the interference of non-rigid deformation of ancient building components due to their age (such as beam and column bending and foundation settlement) on feature extraction, calculation and Gaussian weights between The specific formula is as follows:

[0167] ;

[0168] in, Indicates the thermal diffusivity;

[0169] Construct a normalized Laplace-Beltrami operator matrix of dimension N*N The specific formula is as follows:

[0170] ;

[0171] in, Represents the identity matrix. Degree matrix, Indicates Gaussian weights;

[0172] right Perform eigenvalue decomposition to obtain the corresponding eigenvalues. and eigenvectors Select the preceding mode corresponding to the low-frequency vibration mode. The eigenvectors corresponding to the smallest non-zero eigenvalues ​​are used to obtain the harmonic spectrum features of the manifold, as shown in the following formula:

[0173] ;

[0174] in, express Manifold harmonic spectrum characteristics, express The first harmonic spectral feature of the manifold 1 eigenvector.

[0175] S3. Concatenate the coordinates, local density, explicit geometric features, and manifold harmonic spectrum features of each 3D coordinate point to obtain a feature descriptor; based on the adaptive number of neighbors, use a pre-trained dynamic graph convolutional network to extract features from the feature descriptor and aggregate neighborhood information to obtain the component category to which the 3D coordinate point belongs; specifically:

[0176] The specific formula for obtaining the feature descriptor is:

[0177] ;

[0178] in, The feature descriptor representing the i-th 3D coordinate point, This indicates a splicing operation. Indicates radius as linearity within the neighborhood, Indicates radius as Flatness within the neighborhood, Indicates radius as The divergence within the neighborhood, Indicates radius as linearity within the neighborhood, Indicates radius as Flatness within the neighborhood, Indicates radius as The divergence within the neighborhood, The first characteristic of the harmonic spectrum of a manifold 1 eigenvector;

[0179] Based on an adaptive number of neighbors, a k-NN graph is constructed in the enhanced feature space, and adjacency relationships are generated;

[0180] Dynamic graph convolutional networks consist of edge convolutional layers, max pooling layers, and a classification head;

[0181] The feature descriptors are input into a pre-trained dynamic graph convolutional network. After passing through edge convolutional layers, the edge convolution operation is used to aggregate neighborhood information and extract local structural features. The specific expression is as follows:

[0182] ;

[0183] in, This represents the local structural features of the i-th three-dimensional coordinate point. This represents the nearest neighbor of the adaptive neighborhood of the i-th 3D coordinate point. This represents a learnable function implemented by a multilayer perceptron. express Feature descriptors;

[0184] The local structural features are max-pooled to obtain the global structural features. These local and global features are then concatenated, processed by a classification head, and based on adjacency relationships, the component category of the 3D coordinate point is determined. The specific formula is as follows:

[0185] ;

[0186] in, This indicates the component category to which the i-th 3D coordinate point belongs. This represents the probability normalization function. This represents a multilayer perceptron. This represents global structural features.

[0187] S4. Obtain the classification confidence score for each 3D coordinate point, reflecting the degree of certainty of the classification result by the dynamic graph convolutional network. When the classification confidence score is higher than the dynamic confidence score threshold, retain the component category to which the 3D coordinate point corresponding to that classification confidence score belongs; otherwise, mark it as unclassified, thus obtaining the final component category. Specifically:

[0188] The formula for calculating classification confidence is:

[0189] ;

[0190] in, This represents the classification confidence score of the i-th 3D coordinate point. This represents the highest probability that the predicted 3D coordinate point belongs to the component category. This represents the second-highest probability predicted for the i-th 3D coordinate point. This represents the i-th three-dimensional coordinate point;

[0191] The expression for the dynamic confidence threshold is:

[0192] ;

[0193] in, This represents the dynamic confidence threshold for the i-th 3D coordinate point. This represents the initial value of the dynamic confidence threshold. Indicates the regulating factor;

[0194] The formula for determining the final component category is:

[0195] ;

[0196] in, This indicates the final component category to which the i-th 3D coordinate point belongs.

[0197] S5. Construct a set of component point clouds based on the final component category, obtain the component feature vectors corresponding to the component point clouds in this set, and input this vector into the Transformer module to capture the long-distance dependencies between components and obtain context-aware features; specifically:

[0198] The final set of 3D coordinate points of the same category within the component category is constructed into a component point cloud set. An encoder network is used to extract the initial component feature vectors corresponding to the 3D coordinate points in this set. Semantic vectors corresponding to the 3D coordinate points in this set are then obtained. The initial component feature vectors and semantic vectors are added element-wise to obtain the component feature vectors. The specific formula is as follows:

[0199] ;

[0200] in, Indicates the first In the point cloud set of the component, the first The component feature vector of each component's three-dimensional coordinate points. Indicates the first In the point cloud set of the component, the first The initial component feature vector of each component's three-dimensional coordinate points. Indicates the first In the point cloud set of the component, the first The semantic vector of the three-dimensional coordinate points of each component Indicates the first Three-dimensional coordinates of each component , This indicates the total number of three-dimensional coordinate points of the component;

[0201] Inputting the component feature vector into the Transformer module yields the corresponding query matrix, key matrix, and value matrix. The specific formula is as follows:

[0202] ;

[0203] ;

[0204] ;

[0205] in, Represents the query matrix. Represents the key matrix, Represents a value matrix, The linear mapping matrix representing the query matrix. The linear mapping matrix representing the key matrix. The linear mapping matrix representing the value matrix. Indicates the first In the point cloud set of the component, the first The component feature vector of each component's three-dimensional coordinate points, superscript T Indicates transpose;

[0206] Attention weights are obtained using standard scaled dot product attention, thereby capturing long-distance dependencies between components. The specific formula is as follows:

[0207] ;

[0208] in, This represents the attention mechanism. This represents the probability normalization function. express transpose, Indicates the dimension of the key vector;

[0209] Long-range dependencies are encoded into the output features of each component to obtain context-aware features. Each context-aware feature contains global information about the relationships between components.

[0210] S6. Input the context-aware features into the pre-trained gated attention cleansing model to obtain the original gate vector. Correct this original gate vector using a gate truncation threshold. Then, use the corrected gate vector to perform weighted filtering of the context-aware features to obtain the cleaned features; specifically:

[0211] Context-aware features are input into a pre-trained gated attention cleansing model. Structural and relational information is extracted using standard Transformer blocks to obtain latent structural features. These latent structural features are then processed sequentially through an MLP network and a Sigmoid activation function to generate the original gated vector. An adaptively adjustable gate truncation threshold is then used. (Set to 0 in the initial iteration) The original gating vector is corrected to obtain the corrected gating vector. The specific formula is as follows:

[0212] ;

[0213] in, Indicates the first The corrected gating vector in each dimension, Indicates the first The original gated vector of each dimension;

[0214] The context-aware features are weighted and filtered using the modified gating vector, and the purified features are obtained by element-wise multiplication. The specific formula is as follows:

[0215] ;

[0216] in, Indicates the first In the point cloud set of the component, the first The purification features of each component's three-dimensional coordinate points Indicates the first In the point cloud set of the component, the first Structural features of the three-dimensional coordinate points of each component It represents the Hadamardi (or Hadama) stack;

[0217] The specific formula for the separated suppressed noise features is:

[0218] ;

[0219] in, Indicates the first In the point cloud set of the component, the first Noise characteristics of three-dimensional coordinate points of each component.

[0220] S7, Construction includes A two-dimensional uniform grid set of grid points will be used to replicate the purification features. The points are divided into portions and concatenated with the coordinates of each grid point in a two-dimensional uniform grid to obtain a high-dimensional feature vector. This high-dimensional feature vector is then input into a pre-trained folding decoder to obtain the completed part point cloud set; specifically:

[0221] Generate within a two-dimensional unit square, including A two-dimensional uniform grid point set of grid points ;

[0222] Copy the purification features The data is then combined with the coordinates of each grid point in a two-dimensional uniform grid to form a high-dimensional feature vector. The specific formula is as follows:

[0223] ;

[0224] in, Indicates the first The target coordinates of a uniform grid Indicates the first The target vertex of a uniform grid;

[0225] The high-dimensional feature vector is input into a pre-trained folding decoder composed of multilayer perceptrons. Through nonlinear transformation, the two-dimensional uniform grid points and their associated global features are mapped and folded into three-dimensional space, outputting three-dimensional coordinates to obtain the completed part point cloud set. The specific formula is as follows:

[0226] ;

[0227] in, This represents the complete point cloud set of components. This indicates a folded decoder.

[0228] It's worth noting that the folded decoder employs a shared weight structure based on a multilayer perceptron. Since the multilayer perceptron processes each point independently, it essentially learns a continuous mapping function from a two-dimensional planar domain to a three-dimensional curved manifold, rather than a simple correspondence of discrete points. Therefore, the folded decoder is resolution-independent during the inference phase. This means that although a fixed number of grid points may be used during pre-training, the density of the input two-dimensional uniform grid point set can be flexibly adjusted (i.e., the density of the input grid point set can be changed) based on feedback instructions during the actual inference completion phase. The value of is used to achieve higher resolution sampling of 3D surfaces in a physical way to solve the point cloud sparsity problem without retraining the model parameters.

[0229] S8. Combine the completed part point cloud set in the original coordinate system to obtain the final point cloud set. Improve the gradient field of each part's 3D coordinate point in the final point cloud set. Based on the smoothing factor, the improved gradient field, and the Poisson equation, obtain the smoothed target vertices. The smoothed target vertices constitute the smoothed point cloud set. Specifically:

[0230] The completed point cloud set of components is combined in the original coordinate system to obtain the final point cloud set. The specific formula is as follows:

[0231] ;

[0232] in, This represents the total number of point cloud sets for each component;

[0233] The gradient field of each component's 3D coordinate point in the final point cloud set is improved by utilizing local curvature, resulting in the improved gradient field. The specific formula is as follows:

[0234] ;

[0235] in, Represents the gradient operator. Represents the gradient field. This represents the improved gradient field. Indicates the first The weights of the three-dimensional coordinate points of each component Indicates the first The normal vector of the three-dimensional coordinate points of each component Indicates the first Local curvature of three-dimensional coordinate points of each component ;

[0236] Based on the improved gradient field and Poisson equation, the three-dimensional scalar field is obtained, with the specific formula as follows:

[0237] ;

[0238] in, Indicates Fourier transform, Represents a three-dimensional scalar field. Represents the frequency vector. Represents the regularization parameter. Represents the imaginary unit;

[0239] A triangular mesh is obtained by processing a three-dimensional scalar field using an adaptive threshold; where the adaptive threshold... The expression is:

[0240] ;

[0241] in, This represents the initial value of the adaptive threshold. Indicates the first The probability of building damage at three-dimensional coordinates of individual components. All represent equilibrium parameters;

[0242] The Laplacian smoothing method is used to process the triangular mesh to obtain the smoothed target vertices. The specific formula is as follows:

[0243] ;

[0244] in, Represents the smoothed first... A target vertex of a uniform grid. Represents the smoothing factor. Indicates the first The neighborhood point set of the three-dimensional coordinate points of each component Indicates the first The three-dimensional coordinates of the component and its first component The weights between neighboring points Indicates the first The target vertex of the neighborhood points.

[0245] S9. After generating a rendered image from the smoothed point cloud set, input it into a pre-trained multimodal large language model for evaluation. Use prompts to guide the visual language model in evaluating the material and age of the ancient building. Based on the evaluation results, obtain a set of structured parameter adjustment instructions for each component. Update the gating truncation threshold in step S6, the number of grid points in the two-dimensional uniform grid point set in step S7, and the smoothing factor in step S8 according to the instruction set until the evaluation passes, completing the point cloud model completion. Specifically:

[0246] The smoothed point cloud set is subjected to multi-view projection rendering to obtain the corresponding rendered image. In the rendered image, unique component index labels are superimposed on the point clouds belonging to different component sets to generate a two-dimensional image set containing semantic mapping relationships between components. This two-dimensional image set is input into a pre-trained multimodal large language model. Using prompt words to guide the model, visual quality assessment is performed on each independent component in the two-dimensional image set. Automatic identification and semantic inference are performed on the age characteristics, component materials, and craftsmanship style of the ancient building. Based on this, the geometric shape and surface condition of each component are evaluated for quality, resulting in a structured parameter adjustment instruction set for each component. , , Indicates the first Each independent component This indicates the corresponding defect type; the structured parameter adjustment instruction set includes the target component index, defect type, and parameter adjustment range.

[0247] The structured parameter adjustment instruction set is parsed, and the results are processed as follows:

[0248] (1) When the instruction set indicates the first When the defect type of an individual component is excessively smooth texture (i.e., the surface of the component is as smooth as plastic, losing the proper carving or weathering texture), reduce the smoothness factor. And increase the balance parameter The specific formula is as follows:

[0249] ;

[0250] ;

[0251] in, All of these represent the preset adjustment step size coefficient. , Indicates the first Smoothing factor adjusted for each individual component Indicates the first Smoothing factor for each individual component Indicates the first Balance parameters adjusted for each individual component Indicates the first Balance parameters of individual components;

[0252] based on and The process of obtaining the smoothed target vertex is repeated until the score of the pre-trained multimodal large language model reaches the threshold.

[0253] (2) When the instruction set instructs the first When the defect type of an individual component is sparse point cloud (i.e., there are visual holes or breaks on the component surface, and the point density is insufficient to support the structure), increase the number of grid points in step S7, specifically using the following formula:

[0254] ;

[0255] in, Indicates the first The number of grid points after adjustment for each individual component Indicates the first The number of grid points for each individual component This represents the preset grid increment constant;

[0256] based on Repeat steps S7-S8 until the score of the pre-trained multimodal large language model reaches the threshold.

[0257] (3) When the instruction set instructs the first When the defect type of an independent component is noise residue (i.e., discrete flying points exist around the component or the surface exhibits a non-structural burr-like appearance), the gate cutoff threshold in step S6 is increased, and the specific formula is as follows:

[0258] ;

[0259] in, Indicates the first The gating cutoff threshold is adjusted for each individual component. No. Gating cutoff threshold for each individual component Indicates the preset truncation increment;

[0260] based on Repeat steps S6-S8 until the score of the pre-trained multimodal large language model reaches the threshold.

[0261] The smoothed point cloud set obtained by re-execution is combined with the unadjusted point clouds of other components in the original coordinate system to obtain the final point cloud set after completion, thus completing the point cloud model;

[0262] In this process, the semantic understanding capability of the multimodal large language model provides crucial prior constraints for adjusting geometric parameters. Ancient architectural components from different historical periods and made of different materials possess distinctly different surface geometric characteristics. The multimodal large language model guides specific parameter optimization by identifying these characteristics. For example, when the multimodal large language model identifies a component as made of white marble or glazed tile (common in Qing Dynasty architecture, with a fine surface texture), if high-frequency noise is detected on the surface, it will be judged as residual noise, and an instruction will be generated to increase the gating threshold to enhance noise reduction. When the multimodal large language model identifies a component as made of weathered blue brick or wood carvings (common in Ming Dynasty or earlier, with a naturally rough surface texture), if the surface is detected as too smooth, it will be judged as excessively smoothed texture, and an instruction will be generated to reduce the smoothing factor to preserve the sense of historical vicissitude. This semantic-based feedback mechanism ensures that the completion results conform to the true characteristics of ancient architecture in both geometric accuracy and visual style.

[0263] Pre-training of dynamic graph convolutional networks, gated attention sanitization models, and folded decoders includes:

[0264] The specific formula for calculating the overall probability of loss of three-dimensional coordinate points in ancient buildings is as follows:

[0265] ;

[0266] in, This represents the overall defect probability of the i-th three-dimensional coordinate point; Both represent weight parameters; Represents the curvature factor of the i-th three-dimensional coordinate point; This represents the gradient factor of the normal vector of the i-th three-dimensional coordinate point; Indicates weathering and erosion factor; Indicates the random damage factor;

[0267] Set defect threshold Remove the three-dimensional coordinate points whose overall defect probability is greater than the defect threshold to obtain the defect point cloud. The specific formula is as follows:

[0268] ;

[0269] in, Represents a complete set of point clouds;

[0270] Defective point cloud As the training set, the dynamic graph convolutional network, the gated attention sanitizing model, and the folded decoder are trained by backpropagation using the multi-task joint loss function, respectively, to obtain the pre-trained dynamic graph convolutional network, gated attention sanitizing model, and folded decoder; the specific formula of the multi-task joint loss function is as follows:

[0271] ;

[0272] in, This represents the joint loss function for multiple tasks. The reconstruction loss function is represented by a weighted combination of chamfer distance and bulldozer distance. Represents the balance coefficient. Represents the orthogonality loss function;

[0273] Curvature reflects the degree of bending of the point cloud surface. High curvature regions typically correspond to higher structural stresses and are therefore more prone to defects. The curvature of three-dimensional coordinate points is obtained using Gaussian curvature, specifically expressed as:

[0274] ;

[0275] in, Represents the curvature of the i-th three-dimensional coordinate point. This represents the minimum principal radius of curvature in the neighborhood of the i-th 3D coordinate point. This represents the maximum principal radius of curvature in the neighborhood of the i-th three-dimensional coordinate point;

[0276] The curvature factor is obtained by normalizing the curvature.

[0277] In building structures, areas where the normal vector changes drastically (such as the joints of load-bearing structures) are often points of stress concentration. These areas can be identified by calculating the gradient of the normal vector field. The normal vector of each 3D coordinate point is calculated. Then calculate the spatial gradient of the normal vector. ,Will Normalization is performed to obtain the gradient factor of the normal vector;

[0278] The weathering and erosion factor is obtained using the Poisson point process; the weathering probability is related to the tilt angle of the point cloud surface normal vector (i.e., the degree of exposure to the environment);

[0279] Each three-dimensional coordinate point is assigned a random number uniformly distributed in the interval [0, 1] to obtain the random damage factor. .

[0280] Example:

[0281] The simulation platform was set up, and the experimental hardware and software environment parameters are as follows:

[0282] The operating system is Ubuntu 20.04 LTS, the CPU is a 13th Gen Intel(R) Core(TM) i7-13700H, and the GPU is an NVIDIA GeForce RTX 4060 Laptop GPU (8GB VRAM).

[0283] We use original CAD model files from the industry-recognized 3D model benchmark dataset ShapeNet. For each CAD model, we first uniformly sample its surface to generate a dense point cloud containing 16,384 points. This dense point cloud serves as the high-fidelity ground truth point cloud. Subsequently, from this dense point cloud, we downsample to 2,048 points using a farthest-point sampling algorithm, obtaining a standard-sized complete point cloud set that preserves the original shape and contour while being suitable for network processing. This set is denoted as […]. The size is 2048×3.

[0284] For each 3D coordinate point, construct three neighborhoods with different radii: 0.1, 0.2, and 0.4. The feature descriptor has a dimension of 1434×13. .

[0285] First, an end-to-end deep neural network is constructed and trained, consisting of a dynamic graph convolutional network (for classification), a Transformer module (for context extraction), a gated attention sanitizing model (for denoising), and a folded decoder (for generation).

[0286] Set defect threshold Remove points with a probability greater than the threshold to generate a realistic point cloud with defects. Corresponding As a supervisory label.

[0287] The deep neural network is trained by backpropagation using a multi-task joint loss function, resulting in a pre-trained deep neural network. The specific formula is as follows:

[0288] ;

[0289] in, This represents the joint loss function for multiple tasks. Represents the reconstruction loss function. Represents the balance coefficient. This represents the orthogonality loss function.

[0290] The eigenvectors corresponding to the first 16 smallest non-zero eigenvalues ​​of the low-frequency vibration modes are selected to obtain the manifold harmonic spectrum characteristics, as shown in the following formula:

[0291]

[0292] Figure 2 This is a schematic diagram comparing the classification effects of manifold harmonic spectrum features and traditional geometric features under non-rigid deformation in this invention. Figure 2 The left side shows two forms of the column component: the original straight state A and the bent and deformed state B, represented by a point. As the observation object for feature calculation, when using traditional geometric feature calculation, due to changes in local shape, the linearity decreases from 1.0 in the original straight state A to 0.9 in the bent and deformed state B, leading to feature mismatch and misclassification. However, when using the manifold harmonic spectrum feature calculation of this invention, due to its isometric transformation invariance, the first low-frequency component (0.37) and the second low-frequency component (-0.20) in both the original straight state A and the bent and deformed state B remain highly consistent, thus ensuring that the deformed component is still correctly classified as a column. Therefore, since the manifold harmonic spectrum feature depends on topological connectivity, even if the component is bent, the feature remains unchanged, ensuring classification accuracy.

[0293] Construct a two-dimensional uniform grid of 1024 points and set a smoothing factor. Balance parameters .

[0294] Figure 3 This is a schematic diagram illustrating the process of obtaining sanitized features using a gated attention sanitization model. Figure 3The left side of the diagram represents a mixed feature signal containing noise interference. After being processed by the gated attention cleansing model in the middle, it is decomposed into two sets of feature vectors: one set is the cleaned feature waveform representing the geometric structure, and the other set is the noise feature waveform representing random interference. It can be seen that the cleaned feature and the noise feature remain perpendicular in the vector space, thereby achieving complete decoupling between the effective signal and the noise signal.

[0295] Figure 4 This is a schematic diagram illustrating the point cloud model completion using the method proposed in this invention. Figure 4 The paper demonstrates the adaptive adjustment path of a multimodal large language model for different defects: semantic descriptions such as determining the dynasty to which the ancient building belongs, material information, and expected surface weathering degree are used to guide the multimodal large language model to give a visual quality evaluation consistent with the style of the ancient building in a specific cultural context. When sparse point cloud is detected, the number of grid points is increased; when excessively smooth texture is detected, the smoothing factor is decreased and the balance parameter is increased.

[0296] Figure 5 The results of point cloud restoration of complex and deformed structures using the method proposed in this invention are shown. The first column, showing the arch restoration, reveals a fractured gap at the critical stress points of the arch's crown, and a noticeable void in the left column. Compared to the actual values, the method successfully restores the continuous and smooth arc curve of the arch's crown. The transition between the two column segments is natural, without the common linear fitting or misalignment phenomena, demonstrating precise capture of the curvature structure. The second column, showing the column restoration, reveals a significant truncation in the middle of the column's body. The method not only fills this gap but also achieves extremely high alignment between the upper and lower column axes, uniform point cloud density, and perfectly restores the geometric integrity of the cylinder. As seen in the third column of the complex roof restoration results, the input point cloud shows large areas of flaky peeling on the roof tiles and incomplete eaves structure. Compared to the actual values, the result generated using the method proposed in this invention not only macroscopically completes the overall outline of the roof but also clearly preserves the layered texture of the roof tiles and the undulating details of the ridge at the microscopic level. There is no averaging phenomenon that erases texture in pursuit of smoothness, demonstrating the ability of the proposed method to preserve high-frequency details. In summary, whether facing geometric topological breaks or complex texture loss, the method proposed in this invention can generate high-fidelity point clouds that are highly consistent with the actual values, with significantly better visual effects than incomplete input point clouds.

[0297] Figure 6This image shows the actual effect of the point cloud repair method proposed in this invention on large defect structures. The method proposed in this invention innovatively introduces a multimodal large language model and constructs a closed-loop feedback mechanism of generation-evaluation-adjustment. This mechanism breaks through the limitations of traditional geometric indicators and can perform instance-level diagnosis of the generated ancient building point cloud from the perspective of visual perception, and automatically trigger differentiated parameter optimization strategies.

[0298] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for ancient building point cloud model completion based on multi-modal feedback and feature purification, characterized in that, The method comprises the following steps: S1, obtaining an actual defect point cloud model of an ancient building, calculating the local density of each three-dimensional coordinate point in the defect point cloud model, and obtaining an adaptive neighbor number based on the local density of each three-dimensional coordinate point to dynamically adjust the neighborhood search range; specifically, The calculation formula of the local density is: ； wherein, represents the i-th three-dimensional coordinate point, represents the local density of the i-th three-dimensional coordinate point, represents the neighborhood point set of the i-th three-dimensional coordinate point within a radius , represents the decay coefficient, represents the j-th neighborhood three-dimensional coordinate point of the i-th three-dimensional coordinate point, represents the square of the Euclidean distance between , represents the exponential function; The formula for calculating the adaptive neighbor number is: ； wherein, represents the adaptive number of neighbors of the i-th three-dimensional coordinate point, represents an initial value of the number of neighbors, represents a balancing coefficient; S2, construct a neighborhood of different radius for each three-dimensional coordinate point, calculate the covariance matrix of the neighborhood by using principal component analysis to obtain explicit geometric features; construct a local geometric neighborhood graph, calculate the discrete Laplace-Beltrami operator matrix, and perform eigenvalue decomposition on the operator matrix to obtain the corresponding first characteristic vectors as manifold harmonic spectrum features; S3, splicing the coordinate value, local density, explicit geometric feature and manifold harmonic spectrum feature of each three-dimensional coordinate point to obtain a feature descriptor; based on the adaptive neighbor number, using a pre-trained dynamic graph convolution network to extract features and aggregate neighborhood information of the feature descriptor to obtain a component category to which the three-dimensional coordinate point belongs; S4, obtaining the classification confidence of each three-dimensional coordinate point, when the classification confidence is higher than the dynamic confidence threshold, retaining the component category to which the three-dimensional coordinate point corresponding to the classification confidence belongs, otherwise marking it as unclassified, to obtain the final component category; S5, based on the final component category, constructing a component point cloud set, obtaining a component feature vector corresponding to the component point cloud in the set, inputting the vector into a Transformer module to capture long-distance dependency between components to obtain context-aware features; S6, inputting the context-aware features into a pre-trained gated attention purification model to obtain an original gating vector, modifying the original gating vector using a gating truncation threshold, and weighting and filtering the context-aware features using the modified gating vector to obtain purified features; In the formula, the context-aware feature is input into a pre-trained gated attention purification model, structural features and relationship information are extracted through a standard Transformer block extraction structure, and the potential structural features are sequentially processed through an MLP network and a Sigmoid activation function to generate an original gating vector; a gating truncation threshold is used The original gating vector is corrected to obtain a corrected gating vector, and the specific formula is as follows: ； wherein, represents the modified gating vector for the th dimension, represents the original gating vector for the th dimension; S7、constructing a two-dimensional uniform grid point set including copying the purification features splicing the purification features with the coordinates of each grid point in the two-dimensional uniform grid point set, obtaining a high-dimensional feature vector, and inputting the high-dimensional feature vector into the pre-trained folding decoder to obtain a completed component point cloud set; S8, combining the completed component point cloud set in the original coordinate system to obtain a final point cloud set, improving the gradient field of each component three-dimensional coordinate point in the final point cloud set, and obtaining a smoothed target vertex based on a smoothing factor, the improved gradient field and a Poisson equation; the smoothed target vertex constitutes a smoothed point cloud set; S9, inputting the smoothed point cloud set into a pre-trained multi-modal large language model after generating a rendering image to evaluate, guiding the visual language model to evaluate the material and year of the ancient building through a prompt word; based on the evaluation result, a structured parameter adjustment instruction set for each component is obtained, and the gating truncation threshold in step S6, the number of grid points of the two-dimensional uniform grid point set in step S7 and the smoothing factor in step S8 are updated according to the instruction set until the evaluation is passed, and the point cloud model completion is completed.

2. The ancient building point cloud model completion method based on multi-modal feedback and feature purification according to claim 1, characterized in that, In step S2, a neighborhood of three different radii is constructed for each three-dimensional coordinate point, and then the i-th three-dimensional coordinate point neighborhood The covariance matrix of the neighborhood is a 3x3 matrix, and the specific formula for obtaining the explicit geometric feature is: ； ； ； wherein, represents linearity within a neighborhood of radius represents planarity within a neighborhood of radius represents divergence within a neighborhood of radius each represents an eigenvalue; , represents the total number of scales;​​​ The explicit geometric feature includes linearity, planarity and divergence; Computing with Gaussian weights between , specifically given by ； wherein represents the thermal diffusivity; Constructing a normalized laplace-beltrami operator matrix with dimension N*N The specific formula is: ； wherein denotes the identity matrix, denotes the degree matrix, denotes the Gaussian weight; right Perform eigenvalue decomposition to obtain the corresponding eigenvalues. and eigenvectors Select the preceding mode corresponding to the low-frequency vibration mode. The eigenvectors corresponding to the smallest non-zero eigenvalues ​​are used to obtain the harmonic spectrum features of the manifold, as shown in the following formula: ； wherein represents manifold harmonic spectrum features of represents the i-th feature vector in the manifold harmonic spectrum features of ​ 3. The ancient building point cloud model completion method based on multi-modal feedback and feature purification according to claim 1, characterized in that, In step S3, the specific formula for obtaining the feature descriptor is: ； in, The feature descriptor representing the i-th 3D coordinate point, This indicates a splicing operation. Indicates radius as linearity within the neighborhood, This represents the local density of the i-th 3D coordinate point. Indicates radius as Flatness within the neighborhood, Indicates radius as The divergence within the neighborhood, Indicates radius as linearity within the neighborhood, Indicates radius as Flatness within the neighborhood, Indicates radius as The divergence within the neighborhood, The first characteristic in the harmonic spectrum of a manifold 1 eigenvector This represents the i-th three-dimensional coordinate point; Based on the adaptive neighbor number, a k-NN graph is constructed in the enhanced feature space, and an adjacency relationship is generated; The dynamic graph convolution network comprises an edge convolution layer, a maximum pooling layer and a classification head; The feature descriptor is input into the pre-trained dynamic graph convolution network, and the edge convolution layer is used to aggregate neighborhood information through edge convolution operation to extract local structure features, and the specific expression is: ； in, This represents the local structural features of the i-th three-dimensional coordinate point. This represents the j-th neighboring 3D coordinate point of the i-th 3D coordinate point. This represents the nearest neighbor of the adaptive neighborhood of the i-th 3D coordinate point. This represents a learnable function implemented by a multilayer perceptron. express Feature descriptors; The local structure features are subjected to a max-pooling operation through a max-pooling layer to obtain global structure features, and after the local structure features and the global structure features are spliced, the processing of a classification head and based on an adjacency relationship, a category to which the three-dimensional coordinate points belong is obtained, and a specific formula is as follows: ； wherein, represents a component category to which the i-th three-dimensional coordinate point belongs, represents a probability normalization function, represents a multi-layer perception, represents a global structure feature.

4. The ancient building point cloud model completion method based on multi-modal feedback and feature purification according to claim 1, characterized in that, In step S4, a calculation formula of the classification confidence is as follows: ； wherein, represents a classification confidence of the i-th three-dimensional coordinate point, represents a highest probability of a component class to which the predicted i-th three-dimensional coordinate point belongs, represents a second highest probability of the predicted i-th three-dimensional coordinate point, represents the i-th three-dimensional coordinate point; An expression of the dynamic confidence threshold is as follows: ； wherein, denotes a dynamic confidence threshold for the i-th three-dimensional coordinate point, denotes an initial value for the dynamic confidence threshold, denotes an adjustment factor, denotes a local density for the i-th three-dimensional coordinate point; A formula for obtaining the final category to which the parts belong is as follows: ； wherein, represents the final component class to which the i-th three-dimensional coordinate point belongs, represents the component class to which the i-th three-dimensional coordinate point belongs.

5. The ancient building point cloud model completion method based on multi-modal feedback and feature purification according to claim 1, characterized in that, In step S5, the three-dimensional coordinate points of the same category in the final category to which the parts belong are constructed into a part point cloud set, an initial part feature vector corresponding to the part three-dimensional coordinate points in the set is extracted by using an encoder network, a semantic vector corresponding to the part three-dimensional coordinate points in the set is obtained, and the initial part feature vector and the semantic vector are added element by element to obtain a part feature vector, and a specific formula is as follows: ； wherein, represents a component feature vector of a component three-dimensional coordinate point in a component point cloud set, represents a component feature vector of a component three-dimensional coordinate point in a component point cloud set, represents a component feature vector of a component three-dimensional coordinate point in a component point cloud set, represents a semantic vector of a component three-dimensional coordinate point, , represents a total number of component three-dimensional coordinate points; The part feature vector is input into a Transformer module to obtain a corresponding query matrix, key matrix and value matrix, and a specific formula is as follows: ； ； ； in, Represents the query matrix. Represents the key matrix. Represents a value matrix, The linear mapping matrix representing the query matrix. The linear mapping matrix representing the key matrix. The linear mapping matrix representing the value matrix. Indicates the first In the point cloud set of the component, the first The component feature vector of each component's three-dimensional coordinate points; A standard scaled dot-product attention is used to obtain attention weights, so as to capture long-distance dependency relationships between parts, and a specific formula is as follows: ； wherein, denotes the attention mechanism, denotes a probability normalization function, denotes the transpose of denotes the dimension of the key vector; Encoding long-range dependencies into the output features of each component, resulting in context-aware features .

6. The ancient building point cloud model completion method based on multi-modal feedback and feature purification according to claim 1, characterized in that, In step S6, the obtained purification features include the following contents: The context-aware features are weighted and screened by using the modified gating vector, and the purification features are obtained by element-wise multiplication, and a specific formula is as follows: ； wherein, represents a cleaned feature of a th component 3D coordinate point in a th component point cloud set, represents a structure feature of a th component 3D coordinate point in a th component point cloud set, represents a Hadamard product; A specific formula of the separated suppressed noise features is as follows: ； wherein, represents the noise characteristic of the i-th component three-dimensional coordinate point in the i-th component point cloud set. represents the noise characteristic of the i-th component three-dimensional coordinate point in the i-th component point cloud set.​ 7. The ancient building point cloud model completion method based on multi-modal feedback and feature purification according to claim 1, characterized in that, In step S7, a two-dimensional uniform grid point set including grid points is generated within a two-dimensional unit square ; Copy the purification feature The high-dimensional feature vector is formed by splicing the grid point coordinates of each two-dimensional uniform grid point set The specific formula is: ； in, Indicates the first In the point cloud set of the component, the first The purification features of each component's three-dimensional coordinate points Indicates the first The target coordinates of a uniform grid Indicates the first The target vertex of a uniform grid; The high-dimensional feature vector is input into a pre-trained folding decoder composed of multiple perception mechanisms, a two-dimensional uniform grid point and its accompanying global features are mapped and folded into a three-dimensional space through nonlinear transformation, and three-dimensional coordinates are output, so as to obtain a completed part point cloud set, and a specific formula is as follows: ； wherein, denotes the completed set of component point clouds, denotes the folding decoder.

8. The ancient building point cloud model completion method based on multi-modal feedback and feature purification according to claim 1, characterized in that, In step S8, the completed component point cloud sets are combined under the original coordinate system to obtain a final point cloud set The specific formula is: ； wherein, denotes the total number of component point cloud sets, denotes the completed component point cloud set; The gradient field of each part three-dimensional coordinate point in the final point cloud set is improved by using local curvature, and an improved gradient field is obtained, and a specific formula is as follows: ； wherein, denotes the gradient operator, denotes the gradient field, denotes the improved gradient field, denotes the weight of the third dimensional coordinate point of the component, denotes the normal vector of the third dimensional coordinate point of the component, denotes the local curvature of the third dimensional coordinate point of the component, ; Based on the improved gradient field and the Poisson equation, a three-dimensional scalar field is obtained, and a specific formula is as follows: ； wherein denotes the Fourier transform, denotes a three-dimensional scalar field, denotes a frequency vector, denotes a regularization parameter, denotes the imaginary unit; The three-dimensional scalar field is processed by using an adaptive threshold to obtain a triangular mesh; wherein, the expression of the adaptive threshold is: The expression is: ； wherein, represents an initial value of an adaptive threshold, represents a building damage probability of a third dimensional coordinate point of the component, all represent balance parameters; The triangular mesh is processed by using Laplace smoothing to obtain smoothed target vertices, and a specific formula is as follows: ； wherein, denotes the target vertex of the smoothed uniform grid, denotes a smoothing factor, denotes the neighborhood of the third component three-dimensional coordinate point, denotes the weight between the third component three-dimensional coordinate point and its third neighborhood point, denotes the target vertex of the third neighborhood point, denotes the target vertex of the third uniform grid.

9. The ancient building point cloud model completion method based on multi-modal feedback and feature purification according to claim 8, characterized in that, In step S9, the smoothed point cloud set is subjected to multi-view projection rendering to obtain a corresponding rendered image, and a unique component index label is superimposed on the point cloud belonging to different component sets in the rendered image to generate a two-dimensional image set containing component semantic mapping relationships; the two-dimensional image set is input into a pre-trained multi-modal large language model, a prompt word is used to guide the model to perform visual quality assessment on each independent component in the two-dimensional image set, automatically identify and semantically infer the age characteristics, component material and process style of the ancient building, and on this basis, perform quality evaluation on the geometric shape and surface state of each component to obtain a structured parameter adjustment instruction set for each component , , represents the first independent component, represents the corresponding defect type; the structured parameter adjustment instruction set includes a target component index, a defect type and a parameter adjustment amplitude; The structured parameter adjustment instruction set is parsed, and according to the parsed results, the following processing is performed respectively: (1) When the instruction set indicates that the defect type of the first independent component is texture over-smoothing, the smoothing factor is reduced and the balance parameter is increased , and the specific formula is: ； ； wherein, both represent preset adjustment step coefficients, , represents the smoothing factor of the first independent component after adjustment, represents the smoothing factor of the first independent component, represents the balance parameter of the first independent component after adjustment, represents the balance parameter of the first independent component. based on and re-executing the process of obtaining smoothed target vertices until the score of the pre-trained multi-modal large language model reaches a threshold stop; (2) When the instruction set indicates that the defect type of the first independent component is a point cloud sparsity, the number of grid points in step S7 is increased, and the specific formula is: ； wherein, represents the number of grid points adjusted by the th independent component, represents the number of grid points of the th independent component, represents a preset grid increment constant; Based on , the process of steps S7-S8 is re-executed until the score of the pre-trained multi-modal large language model reaches the threshold stop; (3) When the instruction set indicates that the defect type of the first independent component is noise residue, increase the gating clipping threshold in step S6, and the specific formula is: ； wherein, denotes the adjusted gating threshold of the denotes the gating threshold of the denotes a preset increment of the cut-off.​​ Based on , the process of steps S6-S8 is re-executed until the score of the pre-trained multi-modal large language model reaches the threshold stop; The smoothed point cloud set obtained by re-execution and other part point clouds that are not adjusted are recombined in the original coordinate system to obtain a completed final point cloud set, and the point cloud model completion is completed.

10. The ancient building point cloud model completion method based on multi-modal feedback and feature purification according to claim 1, characterized in that, The pre-training of the dynamic graph convolution network, the gated attention purification model and the folding decoder includes: The comprehensive damage probability of the three-dimensional coordinate points in the ancient building is calculated, and a specific formula is as follows: ； wherein, represents the i-th three-dimensional coordinate point; represents the i-th three-dimensional coordinate point; each represents a weight parameter; represents the i-th three-dimensional coordinate point; represents the i-th three-dimensional coordinate point; represents the weathering spalling factor; represents the random damage factor; Setting a defect threshold Removing the three-dimensional coordinate points with a comprehensive defect probability greater than the defect threshold to obtain a defect point cloud The specific formula is: ； wherein, denotes the complete set of point clouds; The point cloud of the defect is obtained As a training set, the dynamic graph convolution network, the gated attention purification model and the folding decoder are trained by using a multi-task joint loss function respectively, to obtain a pre-trained dynamic graph convolution network, a gated attention purification model and a folding decoder; the specific formula of the multi-task joint loss function is: ； wherein, denotes a multi-task joint loss function, denotes a reconstruction loss function, denotes a balancing coefficient, denotes an orthogonality loss function; The curvature of the three-dimensional coordinate points is obtained by using Gaussian curvature, and a specific expression is as follows: ； wherein, represents the curvature of the ith three-dimensional coordinate point, represents the minimum principal radius of curvature within the neighborhood of the ith three-dimensional coordinate point, represents the maximum principal radius of curvature within the neighborhood of the ith three-dimensional coordinate point; The curvature is normalized to obtain a curvature factor; calculating a normal vector for each three-dimensional coordinate point recomputing the spatial gradient of the normal vector normalizing the to obtain a normal vector gradient factor; The weathering spalling factor is obtained by using the Poisson point process; A random number uniformly distributed in the interval [0, 1] is assigned to each three-dimensional coordinate point to obtain a random damage factor, .

Images