Historic building point cloud model completion method based on multi-modal feedback and feature purification

Abstract

The invention discloses an ancient building point cloud model completion method based on multi-modal feedback and feature purification, and the method comprises the steps: calculating the local density of each three-dimensional coordinate point in an ancient building defect point cloud model, obtaining the number of adaptive neighbors, carrying out the processing of feature descriptors through employing a dynamic graph convolution network, and obtaining the local density of each three-dimensional coordinate point in the ancient building defect point cloud model. Obtaining a part category to which the three-dimensional coordinate point belongs; component categories to which the three-dimensional coordinate points corresponding to the classification confidence higher than the dynamic confidence threshold belong are reserved, a final component category to which the three-dimensional coordinate points belong is obtained, and purification features are obtained based on the category; the method comprises the steps of constructing a two-dimensional uniform grid point set, splicing purification features and grid point coordinates in the two-dimensional uniform grid point set to obtain a high-dimensional feature vector, obtaining a complemented part point cloud set by using a folding decoder, further obtaining a smoothed point cloud set, and processing the set by using a multi-modal large language model to complete point cloud complementation. According to the method, high-fidelity refined repair is realized, and the complementation stability in a whole scene is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional computer vision and digital heritage protection, and particularly relates to a method for completing a point cloud model of an ancient building based on multi-modal feedback and feature purification. BACKGROUND

[0002] In recent years, three-dimensional point cloud scanning technology has been widely used in the fields of cultural heritage protection, ancient building restoration, virtual reconstruction, etc. By using technologies such as laser scanning, photogrammetry, and structured light scanning, researchers can obtain high-precision three-dimensional point cloud data of ancient buildings, providing a data basis for their digital archiving and restoration. However, due to factors such as long-term weathering, earthquakes, and human damage to historical buildings, many ancient buildings have suffered partial or large-scale component loss, making it impossible for existing point cloud data to fully reflect the building structure and hindering subsequent digital restoration and engineering restoration. Therefore, effectively completing the incomplete point cloud has become a key technical challenge in this field. Existing technical solutions can be broadly divided into traditional geometry-driven methods and data-driven deep learning methods.

[0003] Early restoration work mainly relied on traditional geometry-driven methods. For example, in Related Document 1 (Li Haipeng. Virtual restoration method of cultural relics based on point cloud processing[D]. Yunnan University, 2022. DOI:10.27456 / d.cnki.gyndu.2022.002536), the boundary of the damaged area was analyzed, and mathematical surfaces such as Poisson surfaces and radial basis functions were used for fitting and interpolation. This method has certain effect in dealing with small-area holes with simple geometric shapes and smoothness, but its core defect lies in the lack of understanding of the overall shape of the object. When faced with complex damage that spans multiple structural components, it is unable to infer the complex morphology that conforms to the original structural logic, and the completion result is often too smooth, losing the original high-frequency details such as the edges and carvings of the building or cultural relics.

[0004] With the growth of data, data-driven deep learning methods have gradually become the mainstream. This type of method trains a neural network on a large dataset, allowing it to learn rich shape priors. However, the limitations of this type of method are also quite obvious: compressing the entire complex structure into a single global feature inevitably leads to the loss of local fine details, and the generated surface is often too vague or presents an averaged form, unable to restore the complex geometric features of specific cultural relics and ancient buildings. In addition, the performance of data-driven deep learning methods is highly dependent on the distribution of the training data, and when dealing with unique architectural structures that differ significantly from the training set, the generalization ability is insufficient, and the effect will decrease significantly.

[0005] Therefore, the latest research begins to explore the strategy of introducing external information or prior knowledge for guidance. For example, in related document 2 (Gao S, Jiao C, Chen R, et al. Point Cloud Completion Guided by Prior Knowledge via Causal Inference [J]. arXiv preprint arXiv:2305.17770, 2023), it explores how to use explicit structural symmetry, component relationship and other prior knowledge to guide the completion network, which improves the accuracy of the results. Related document 3 (Lu Z, Jiao Q, Xu L. MMCNet: Multi-Modal Point Cloud Completion Network Based on Self-Projected Views [C] / / 2024 6th International Conference on Data-driven Optimization of Complex Systems (DOCS). IEEE, 2024:897-902) adopts multi-modal data, and provides more rich clues for three-dimensional geometry generation by combining high-resolution two-dimensional images. However, these methods still have insurmountable obstacles: the method based on explicit prior knowledge has a complex knowledge base construction, and it is difficult to cover all the infinite changes of buildings and cultural relics, and its universality is limited; and the multi-modal method is severely dependent on high-quality and accurately registered additional data (such as images), which are often difficult to obtain in many archaeological sites or scanning conditions.

[0006] In summary, when processing digital objects such as ancient buildings, which have complex structures, rich details, various missing patterns and are often isolated examples, the existing technology still has core problems such as inaccurate component segmentation, insufficient understanding of structural relationships, weak noise suppression ability, and difficulty in balancing detail preservation and global smoothing. Therefore, there is an urgent need for a new technical solution that can perform structure perception and high-fidelity restoration on complex incomplete point cloud models without the need for a large number of similar samples or additional modal data. SUMMARY

[0007] The technical problem to be solved by the present application is to solve the problem of preserving high-frequency geometric details while suppressing weathering and peeling noise, and the problem of lack of semantic constraints in pure geometric completion. A method for completing ancient building point cloud models based on multi-modal feedback and feature purification is provided, which significantly improves the robustness of incomplete and deformed ancient building data and realizes high-fidelity fine repair.

[0008] To solve the above technical problems, the present application adopts the following technical solutions:

[0009] A kind of ancient building point cloud model completion method based on multi-modal feedback and feature purification, comprising the following steps:

[0010] S1, the actual defect point cloud model of ancient building is acquired, the local density of each three-dimensional coordinate point in the defect point cloud model is calculated;Based on the local density of each three-dimensional coordinate point, the adaptive neighbor number is obtained, and the neighborhood search range is dynamically adjusted;

[0011] S2, the neighborhood of different radius of each three-dimensional coordinate point is constructed, the covariance matrix of the neighborhood is calculated using principal component analysis, and the explicit geometric feature is obtained;Local geometric nearest neighbor graph is constructed, discrete Laplace-Beltrami operator matrix is calculated, and the corresponding first Feature vector is obtained by feature decomposition of the operator matrix, as manifold harmonic spectrum feature;

[0012] S3, the coordinate value of each three-dimensional coordinate point, local density, explicit geometric feature and manifold harmonic spectrum feature are spliced, and feature descriptor is obtained;Based on adaptive neighbor number, feature descriptor is extracted and neighborhood information is aggregated using pre-trained dynamic graph convolution network, and the component category to which the three-dimensional coordinate point belongs is obtained;

[0013] S4, the classification confidence of each three-dimensional coordinate point is obtained, when the classification confidence is higher than the dynamic confidence threshold, the component category to which the three-dimensional coordinate point corresponding to the classification confidence belongs is retained, otherwise it is marked as unclassified, and the final component category is obtained;

[0014] S5, based on the final component category, component point cloud set is constructed, the component feature vector corresponding to the component point cloud in the set is obtained, and the vector is input into the Transformer module, the long-distance dependence relationship between components is captured, and context-aware feature is obtained;

[0015] S6, the context-aware feature is input into the pre-trained gated attention purification model, and the original gating vector is obtained, the original gating vector is modified using the gating truncation threshold, the context-aware feature is weighted and selected using the modified gating vector, and the purified feature is obtained;

[0016] S7, a two-dimensional uniform grid point set including Grid points are constructed, the purified feature is copied Parts, and is spliced with the coordinate of each grid point in the two-dimensional uniform grid point set, to obtain a high-dimensional feature vector, the high-dimensional feature vector is input into the pre-trained folding decoder, and the completed component point cloud set is obtained;

[0017] S8, combine the completed component point cloud set in the original coordinate system to obtain a final point cloud set, and improve the gradient field of each component three-dimensional coordinate point in the final point cloud set, based on the smoothing factor, the improved gradient field and the Poisson equation, obtain the smoothed target vertex; the smoothed target vertex constitutes a smoothed point cloud set;

[0018] S9, the smoothed point cloud set is generated to render an image and input into a pre-trained multi-modal large language model for evaluation, and the visual language model is guided by a prompt word to evaluate the material and year of the ancient building; based on the evaluation result, a structured parameter adjustment instruction set for each component is obtained, and the gate truncation threshold in step S6, the grid point number of the two-dimensional uniform grid point set in step S7 and the smoothing factor in step S8 are updated according to the instruction set until the evaluation is passed, and the point cloud model completion is completed.

[0019] Further, in step S1, obtaining the adaptive neighbor number includes the following contents:

[0020] The calculation formula of the local density is:

[0021] ;

[0022] wherein, represents the i-th three-dimensional coordinate point, represents the local density of the i-th three-dimensional coordinate point, represents the neighbor point set of the i-th three-dimensional coordinate point within a radius represents the attenuation coefficient, represents the j-th neighbor three-dimensional coordinate point of the i-th three-dimensional coordinate point, represents the square of the Euclidean distance between and ; represents the exponential function;

[0023] The formula for calculating the adaptive neighbor number is:

[0024] ;

[0025] wherein, represents the adaptive neighbor number of the i-th three-dimensional coordinate point, represents the initial value of the neighbor number, represents the balance coefficient;

[0026] In step S2, three different radii of the neighborhood of each three-dimensional coordinate point are constructed, then the i-th three-dimensional coordinate point neighborhood ​The covariance matrix of the point cloud is a 3x3 matrix. The motivation of multi-scale is that a single scale neighborhood cannot capture both fine structures (e.g. sculpture edges) and macro shapes (e.g. wall flatness) simultaneously. The explicit geometric features are obtained by the following formulas:

[0027] ;

[0028] ;

[0029] ;

[0030] wherein, represents the linearity in a neighborhood with radius ; represents the planarity in a neighborhood with radius ; represents the divergence in a neighborhood with radius ; represents the maximum eigenvalue, which usually corresponds to the maximum change direction of the local point set distribution, i.e. the extension of the point cloud along this direction; represents the intermediate eigenvalue, which usually corresponds to the second largest change direction; represents the minimum eigenvalue, which usually corresponds to the minimum change direction, i.e. the thickness or change degree of the point cloud perpendicular to the surface (normal direction); , represents the total number of scales;

[0031] The explicit geometric features include linearity, planarity and divergence;

[0032] The Gaussian weight between and is calculated as , and the specific formula is:

[0033] ;

[0034] wherein, represents the thermal diffusivity;

[0035] A normalized Laplace-Beltrami operator matrix with a dimension of N*N is constructed , and the specific formula is:

[0036] ;

[0037] wherein, represents the unit matrix, represents the degree matrix, represents the Gaussian weight;

[0038] The eigenvalues of are obtained by eigenvalue decomposition and feature vectors ; select the feature vectors corresponding to the first minimum non-zero eigenvalues of the low-frequency vibration mode, obtain the manifold harmonic spectrum feature, and the specific formula is:

[0039] ;

[0040] wherein, denotes the manifold harmonic spectrum feature of the , and denotes the i-th feature vector in the manifold harmonic spectrum feature of the .

[0041] Further, in step S3, the specific formula for obtaining the feature descriptor is:

[0042] ;

[0043] wherein, denotes the feature descriptor of the i-th three-dimensional coordinate point, denotes a splicing operation, denotes linearity in a neighborhood with a radius of , and denotes planarity in a neighborhood with a radius of , and denotes divergence in a neighborhood with a radius of , and denotes linearity in a neighborhood with a radius of , and denotes planarity in a neighborhood with a radius of , and denotes divergence in a neighborhood with a radius of , and denotes the i-th feature vector in the manifold harmonic spectrum feature;

[0044] Based on the adaptive number of neighbors, a k-NN graph is constructed in the enhanced feature space, and a neighbor relationship is generated;

[0045] The dynamic graph convolution network includes an edge convolution layer, a maximum pooling layer and a classification head.

[0046] The feature descriptor is input into the pre-trained dynamic graph convolution network, and the edge convolution layer is passed through to aggregate neighborhood information using edge convolution operation to extract local structure features, and the specific expression is:

[0047] ;

[0048] wherein, denotes the local structure feature of the i-th three-dimensional coordinate point,​​ a nearest neighbor point representing an adaptive neighborhood of the i-th three-dimensional coordinate point, a learnable function implemented by a multi-layer perceptron, a feature descriptor of the i-th three-dimensional coordinate point,

[0049] The local structure features are subjected to a max-pooling operation through a max-pooling layer to obtain global structure features. After the local structure features and the global structure features are spliced, the processing of a classification head and based on an adjacency relationship, a part category to which the three-dimensional coordinate point belongs is obtained, and the specific formula is:

[0050]

[0051] wherein, a part category to which the i-th three-dimensional coordinate point belongs, a probability normalization function, a multi-layer perceptron, a global structure feature.

[0052] Further, in step S4, the calculation formula of the classification confidence is:

[0053]

[0054] wherein, a classification confidence of the i-th three-dimensional coordinate point, a highest probability of a predicted part category to which the i-th three-dimensional coordinate point belongs, a second highest probability of a predicted i-th three-dimensional coordinate point, the i-th three-dimensional coordinate point;

[0055] The expression of the dynamic confidence threshold is:

[0056]

[0057] wherein, a dynamic confidence threshold of the i-th three-dimensional coordinate point, an initial value of the dynamic confidence threshold, an adjustment factor;

[0058] The formula for obtaining the final part category to which the i-th three-dimensional coordinate point belongs is:

[0059]

[0060] wherein, a final part category to which the i-th three-dimensional coordinate point belongs.

[0061] ​​​​​Furthermore, in step S5, the 3D coordinate points of the same category within the final component category are constructed into a component point cloud set. An encoder network is used to extract the initial component feature vectors corresponding to the 3D coordinate points in this set, and the semantic vectors corresponding to the 3D coordinate points in this set are obtained. The initial component feature vectors and the semantic vectors are then added element-wise to obtain the component feature vectors. The specific formula is as follows:

[0062] ;

[0063] in, Indicates the first In the point cloud set of the component, the first The component feature vector of each component's three-dimensional coordinate points. Indicates the first In the point cloud set of the component, the first The initial component feature vector of each component's three-dimensional coordinate points. Indicates the first In the point cloud set of the component, the first The semantic vector of the three-dimensional coordinate points of each component Indicates the first Three-dimensional coordinates of each component , This indicates the total number of three-dimensional coordinate points of the component;

[0064] Inputting the component feature vector into the Transformer module yields the corresponding query matrix, key matrix, and value matrix. The specific formula is as follows:

[0065] ;

[0066] ;

[0067] ;

[0068] in, Represents the query matrix. Represents the key matrix. Represents a value matrix, The linear mapping matrix representing the query matrix. The linear mapping matrix representing the key matrix, The linear mapping matrix representing the value matrix. Indicates the first In the point cloud set of the component, the first The component feature vector of each component's three-dimensional coordinate points, superscript T Indicates transpose;

[0069] Attention weights are obtained using standard scaled dot product attention, thereby capturing long-distance dependencies between components. The specific formula is as follows:

[0070] ;

[0071] wherein, denotes the attention mechanism, denotes the probability normalization function, denotes the transpose of , denotes the dimension of the key vector;

[0072] Encode long-range dependencies into the output features of each component to obtain context-aware features .

[0073] Further, in step S6, obtaining the purified features includes the following contents:

[0074] Input the context-aware features into the pre-trained gated attention purification model, extract the structure and relationship information through the standard Transformer block extraction structure, and obtain the latent structure features. The latent structure features are processed by the MLP network and the Sigmoid activation function in turn to generate the original gating vector; utilize the gating truncation threshold correct the original gating vector to obtain the corrected gating vector, and the specific formula is:

[0075] ;

[0076] wherein, denotes the corrected gating vector of the th dimension, denotes the original gating vector of the th dimension;

[0077] Weight the context-aware features using the corrected gating vector, and obtain the purified features by element-wise multiplication, and the specific formula is:

[0078] ;

[0079] wherein, denotes the purified feature of the th component three-dimensional coordinate point in the th component point cloud set, denotes the structure feature of the th component three-dimensional coordinate point in the th component point cloud set, denotes the Hadamard product;

[0080] The specific formula of the suppressed noise feature separated is:

[0081] ;

[0082] wherein, represents the noise feature of the i-th component three-dimensional coordinate point in the i-th component point cloud set.

[0083] Further, in step S7, a two-dimensional uniform grid point set including K grid points is generated in a two-dimensional unit square.

[0084] Copy the purification feature parts, and splice with the grid point coordinates in each two-dimensional uniform grid point set to form a high-dimensional feature vector , the specific formula is:

[0085] ;

[0086] wherein, represents the target coordinates of the i-th uniform grid, represents the target vertex of the i-th uniform grid.

[0087] Input the high-dimensional feature vector into the pre-trained folding decoder composed of multiple layers of perception, map and fold the two-dimensional uniform grid points and their attached global features to three-dimensional space through nonlinear transformation, output three-dimensional coordinates, get the completed component point cloud set, the specific formula is:

[0088] ;

[0089] wherein, represents the completed component point cloud set, represents the folding decoder.

[0090] Further, in step S8, the completed component point cloud set is combined in the original coordinate system to obtain the final point cloud set , the specific formula is:

[0091] ;

[0092] wherein, represents the total number of component point cloud sets;

[0093] Using local curvature to improve the gradient field of each component three-dimensional coordinate point in the final point cloud set, get the improved gradient field, the specific formula is:

[0094] ;

[0095] wherein, ​​​​​​Represents the gradient operator, Represents the gradient field. This represents the improved gradient field. Indicates the first The weights of the three-dimensional coordinate points of each component Indicates the first The normal vector of each component's three-dimensional coordinate points Indicates the first Local curvature of three-dimensional coordinate points of each component ;

[0096] Based on the improved gradient field and Poisson equation, the three-dimensional scalar field is obtained, with the specific formula as follows:

[0097] ;

[0098] in, Indicates Fourier transform, Represents a three-dimensional scalar field. Represents the frequency vector. Represents the regularization parameter. Represents the imaginary unit;

[0099] A triangular mesh is obtained by processing a three-dimensional scalar field using an adaptive threshold; where the adaptive threshold... The expression is:

[0100] ;

[0101] in, This represents the initial value of the adaptive threshold. Indicates the first The probability of building damage at three-dimensional coordinates of individual components. All represent equilibrium parameters;

[0102] The Laplacian smoothing method is used to process the triangular mesh to obtain the smoothed target vertices. The specific formula is as follows:

[0103] ;

[0104] in, Represents the smoothed first... A target vertex of a uniform grid. Represents the smoothing factor. Indicates the first The neighborhood point set of the three-dimensional coordinate points of each component Indicates the first The three-dimensional coordinates of the component and its first component The weights between neighboring points Indicates the first The target vertex of the neighborhood points.

[0105] Further, in step S9, the smoothed point cloud set is subjected to multi-view projection rendering to obtain a corresponding rendered image. In the rendered image, the point clouds belonging to different component sets are superimposed with unique component index labels to generate a two-dimensional image set containing component semantic mapping relationships. The two-dimensional image set is input into a pre-trained multi-modal large language model. The model is guided by a prompt word to perform visual quality assessment on each independent component in the two-dimensional image set, automatically identify and semantically infer the age characteristics, component material, and process style of the ancient building, and on this basis, perform quality evaluation on the geometric morphology and surface state of each component to obtain a structured parameter adjustment instruction set for each component , , represents the th independent component, represents the corresponding defect type; the structured parameter adjustment instruction set includes a target component index, a defect type, and a parameter adjustment amplitude;

[0106] The structured parameter adjustment instruction set is parsed, and according to the parsed results, the following processing is performed respectively:

[0107] (1) When the instruction set indicates that the defect type of the th independent component is texture over-smoothing, the smoothing factor is reduced and the balance parameter is increased, and the specific formula is:

[0108] ;

[0109] ;

[0110] wherein, all represent preset adjustment step coefficients, , represents the adjusted smoothing factor of the th independent component, represents the smoothing factor of the th independent component, represents the adjusted balance parameter of the th independent component, represents the balance parameter of the th independent component;

[0111] Based on and , the process of obtaining the smoothed target vertex is re-executed until the score of the pre-trained multi-modal large language model reaches a threshold value to stop;

[0112] (2) When the instruction set indicates that the defect type of the When the defect type of the i-th independent component is a point cloud sparsity, the number of grid points in step S7 is increased, and the specific formula is:

[0113] ;

[0114] wherein, represents the adjusted number of grid points of the i-th independent component, represents the number of grid points of the i-th independent component, represents a preset grid increment constant; Based on , the process of steps S7-S8 is re-executed until the score of the pre-trained multi-modal large language model reaches the threshold value to stop;

[0115] (3) When the instruction set indicates that the defect type of the i-th independent component is noise residue, the gating threshold in step S6 is increased, and the specific formula is:

[0116] ;

[0117] ;

[0118] wherein, represents the adjusted gating threshold of the i-th independent component, represents the gating threshold of the i-th independent component, represents a preset truncation increment; Based on , the process of steps S6-S8 is re-executed until the score of the pre-trained multi-modal large language model reaches the threshold value to stop;

[0119] The re-executed smoothed point cloud set is recombined with the unadjusted point cloud of other components in the original coordinate system to obtain a completed final point cloud set, completing the point cloud model completion.

[0120] Further, the pre-training of the dynamic graph convolution network, the gating attention purification model and the folding decoder includes:

[0121] The comprehensive damage probability of the three-dimensional coordinate point in the ancient building is calculated, and the specific formula is:

[0122]

[0123] ;

[0124] wherein, represents the comprehensive damage probability of the i-th three-dimensional coordinate point; all represent weight parameters; ​​​​a curvature factor of the i-th three-dimensional coordinate point; a normal vector gradient factor of the i-th three-dimensional coordinate point; a weathering spalling factor; a random damage factor;

[0125] setting a defect threshold , removing the three-dimensional coordinate points with a comprehensive defect probability greater than the defect threshold to obtain a defect point cloud , and the specific formula is:

[0126] ;

[0127] wherein, the complete point cloud set is represented by

[0128] the defect point cloud is taken as a training set, and the dynamic graph convolution network, the gated attention purification model and the folding decoder are trained by back propagation using a multi-task joint loss function to obtain a pre-trained dynamic graph convolution network, gated attention purification model and folding decoder; the specific formula of the multi-task joint loss function is:

[0129] ;

[0130] wherein, the multi-task joint loss function is represented by the reconstruction loss function is represented by the balance coefficient is represented by the orthogonality loss function is represented by

[0131] the curvature of the three-dimensional coordinate point is obtained by using the Gaussian curvature, and the specific expression is:

[0132] ;

[0133] wherein, the curvature of the i-th three-dimensional coordinate point is represented by the minimum principal curvature radius in the neighborhood of the i-th three-dimensional coordinate point is represented by the maximum principal curvature radius in the neighborhood of the i-th three-dimensional coordinate point is represented by

[0134] the curvature is normalized to obtain a curvature factor;

[0135] the normal vector of each three-dimensional coordinate point is calculated , and the spatial gradient of the normal vector is calculated , and is normalized to obtain a normal vector gradient factor;

[0136] the weathering spalling factor is obtained by using the Poisson point process;

[0137] A random number uniformly distributed in the interval [0, 1] is assigned to each three-dimensional coordinate point to obtain a random damage factor, .

[0138] Compared with the prior art, the present application has the following technical effects:

[0139] 1. By introducing the manifold harmonic spectrum feature, using the isometry invariance of the Laplace-Beltrami operator, the fingerprint feature reflecting the intrinsic topological structure of the component can be extracted, effectively overcoming the recognition and completion deviation caused by the non-rigid deformation of the ancient building component, and significantly improving the robustness of the damaged and deformed ancient building data.

[0140] 2. The gating attention purification model is designed and the orthogonality loss function is introduced, the noise features and structure features are forced to maintain orthogonality in the feature space, and the information is adaptively filtered by combining the gating mechanism, which can accurately strip random scanning noise (such as flying points), while maximizing the retention of high-frequency texture details with structural regularity, achieving a balance between noise suppression and high-frequency detail retention; in addition, combined with the gradient field optimization based on the improved local curvature, the edges and corners are further sharpened, realizing high-fidelity fine repair.

[0141] 3. The present application uses the powerful visual understanding ability of the multi-modal large language model to perform instance-level diagnosis on the completion results such as texture over-smoothing and point cloud sparseness, and converts the diagnosis results into adaptive adjustment instructions of mathematical parameters (such as smoothing factor, threshold), ensuring that the final completion set not only has low geometric error, but also meets the visual aesthetics and structural logic of ancient buildings, breaking through the limitations of single geometric index optimization.

[0142] 4. By calculating the local density, adaptively adjusting the number of neighbors and dynamic confidence threshold, the search range can be automatically expanded in the sparse area of point cloud to capture sufficient information, and in the dense area, the local details are focused, avoiding feature extraction failure or misclassification caused by sampling density difference, ensuring the stability of completion in the whole scene and improving the adaptability to non-uniform density point cloud data. BRIEF DESCRIPTION OF DRAWINGS

[0143] Figure 1 is the overall implementation flowchart of the present application.

[0144] Figure 2 is a comparison chart of the classification effect of the manifold harmonic spectrum feature and the traditional geometric feature in the embodiment of the present application.

[0145] Figure 3 is a schematic diagram of obtaining purified features by using the gating attention purification model in the embodiment of the present application.

[0146] Figure 4 This is a schematic diagram illustrating the point cloud completion process of this invention.

[0147] Figure 5 This is a result image of point cloud repair for complex and deformed structures in an embodiment of the present invention.

[0148] Figure 6 This is a point cloud repair result diagram of a large defect structure in an embodiment of the present invention. Detailed Implementation

[0149] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0150] To achieve the above objectives, this invention provides a method for completing point cloud models of ancient buildings based on multimodal feedback and feature purification, such as... Figure 1 As shown, the specific steps are as follows:

[0151] S1. Obtain the actual missing point cloud model of the ancient building, and calculate the local density of each 3D coordinate point in the missing point cloud model; based on the local density of each 3D coordinate point, obtain an adaptive number of neighbors, and dynamically adjust the neighborhood search range to adapt to the uneven density of the ancient building's scanned point cloud, guiding the subsequent graph convolutional network to construct a more reasonable local graph structure in different density regions; specifically:

[0152] The formula for calculating local density is:

[0153] ;

[0154] in, This represents the i-th three-dimensional coordinate point. This represents the local density of the i-th 3D coordinate point. This indicates that the i-th three-dimensional coordinate point is located at a radius of... The neighborhood point set within, Indicates the attenuation coefficient. This represents the j-th neighboring 3D coordinate point of the i-th 3D coordinate point. express and The square of the Euclidean distance between them Represents an exponential function;

[0155] The formula for calculating the adaptive number of neighbors is:

[0156] ;

[0157] in, This represents the adaptive number of neighbors for the i-th 3D coordinate point. an initial value representing the number of neighbors, represents a balance coefficient.

[0158] S2, constructing a neighborhood of different radius for each three-dimensional coordinate point, calculating the covariance matrix of the neighborhood by principal component analysis to obtain explicit geometric features; constructing a local geometric neighborhood graph, calculating a discrete Laplace-Beltrami operator matrix, and performing eigenvalue decomposition on the operator matrix to obtain corresponding first characteristic vectors as manifold harmonic spectrum features; specifically:

[0159] Constructing three different radius neighborhoods for each three-dimensional coordinate point, the covariance matrix of the i-th three-dimensional coordinate point neighborhood is a 3x3 matrix, and the multi-scale motivation is that a single scale neighborhood cannot capture both fine structures (such as carved edges) and macroscopic shapes (such as wall flatness); the specific formula for obtaining explicit geometric features is:

[0160] ;

[0161] ;

[0162] ;

[0163] wherein, represents linearity in a neighborhood with a radius of ; represents planarity in a neighborhood with a radius of ; represents divergence in a neighborhood with a radius of ; represents the maximum eigenvalue, which usually corresponds to the maximum change direction of the local point set distribution, i.e., the extension of the point cloud along that direction; represents the intermediate eigenvalue, which usually corresponds to the second largest change direction; represents the minimum eigenvalue, which usually corresponds to the minimum change direction, i.e., the thickness or change degree of the point cloud perpendicular to the surface (normal direction); , represents the total number of scales;

[0164] Explicit geometric features include linearity, planarity, and divergence;

[0165] For example, a point located on the edge of a column will exhibit high linearity at a certain scale, while a point located on the surface of a wall will exhibit high planarity;

[0166] In order to overcome the interference of non-rigid deformation (such as beam bending and foundation settlement) of ancient building components due to long age on feature extraction, and Gaussian weights between The specific formula is as follows:

[0167] ;

[0168] in, Indicates the thermal diffusivity;

[0169] Construct a normalized Laplace-Beltrami operator matrix of dimension N*N The specific formula is as follows:

[0170] ;

[0171] in, Represents the identity matrix. Degree matrix, Indicates Gaussian weights;

[0172] right Perform eigenvalue decomposition to obtain the corresponding eigenvalues. and eigenvectors Select the preceding mode corresponding to the low-frequency vibration mode. The eigenvectors corresponding to the smallest non-zero eigenvalues ​​are used to obtain the harmonic spectrum features of the manifold, as shown in the following formula:

[0173] ;

[0174] in, express Manifold harmonic spectrum characteristics, express The first harmonic spectral feature of the manifold 1 eigenvector.

[0175] S3. Concatenate the coordinates, local density, explicit geometric features, and manifold harmonic spectrum features of each 3D coordinate point to obtain a feature descriptor; based on the adaptive number of neighbors, use a pre-trained dynamic graph convolutional network to extract features from the feature descriptor and aggregate neighborhood information to obtain the component category to which the 3D coordinate point belongs; specifically:

[0176] The specific formula for obtaining the feature descriptor is:

[0177] ;

[0178] in, The feature descriptor representing the i-th 3D coordinate point, This indicates a splicing operation. Indicates radius as linearity within the neighborhood, Indicates radius as Flatness within the neighborhood, Indicates radius as The divergence within the neighborhood of . Indicates radius as linearity within the neighborhood of . Indicates radius as Flatness within the neighborhood, Indicates radius as The divergence within the neighborhood of . The first characteristic of the harmonic spectrum of a manifold 1 eigenvector;

[0179] Based on an adaptive number of neighbors, a k-NN graph is constructed in the enhanced feature space, and adjacency relationships are generated;

[0180] Dynamic graph convolutional networks consist of edge convolutional layers, max pooling layers, and a classification head;

[0181] The feature descriptors are input into a pre-trained dynamic graph convolutional network. After passing through edge convolutional layers, the edge convolution operation is used to aggregate neighborhood information and extract local structural features. The specific expression is as follows:

[0182] ;

[0183] in, This represents the local structural features of the i-th three-dimensional coordinate point. This represents the nearest neighbor of the adaptive neighborhood of the i-th 3D coordinate point. This represents a learnable function implemented by a multilayer perceptron. express Feature descriptors;

[0184] The local structural features are max-pooled to obtain the global structural features. These local and global features are then concatenated, processed by a classification head, and based on adjacency relationships, the component category of the 3D coordinate point is determined. The specific formula is as follows:

[0185] ;

[0186] in, This indicates the component category to which the i-th 3D coordinate point belongs. This represents the probability normalization function. This represents a multilayer perceptron. This represents global structural features.

[0187] S4, obtain the classification confidence of each three-dimensional coordinate point, which reflects the determination degree of the dynamic graph convolution network on the classification result, when the classification confidence is higher than the dynamic confidence threshold, the component category to which the three-dimensional coordinate point corresponding to the classification confidence belongs is retained, otherwise it is marked as unclassified, and the final component category is obtained; specifically:

[0188] The calculation formula of the classification confidence is:

[0189] ;

[0190] Wherein, represents the classification confidence of the i-th three-dimensional coordinate point, represents the highest probability of the component category to which the i-th three-dimensional coordinate point belongs, represents the second highest probability of the i-th three-dimensional coordinate point, represents the i-th three-dimensional coordinate point;

[0191] The expression of the dynamic confidence threshold is:

[0192] ;

[0193] Wherein, represents the dynamic confidence threshold of the i-th three-dimensional coordinate point, represents the initial value of the dynamic confidence threshold, represents the adjustment factor;

[0194] The formula for obtaining the final component category is:

[0195] ;

[0196] Wherein, represents the final component category of the i-th three-dimensional coordinate point.

[0197] S5, based on the final component category, a component point cloud set is constructed, the component feature vector corresponding to the component point cloud in the set is obtained, the vector is input into the Transformer module, the long-distance dependence relationship between components is captured, and the context-aware feature is obtained; specifically:

[0198] The three-dimensional coordinate points of the same category in the final component category are constructed into a component point cloud set, the initial component feature vector corresponding to the component three-dimensional coordinate points in the set is extracted by using the encoder network, the semantic vector corresponding to the component three-dimensional coordinate points in the set is obtained, the initial component feature vector and the semantic vector are added element by element, and the component feature vector is obtained, and the specific formula is:

[0199] ;

[0200] wherein, represents the i-th component feature vector of the i-th component point cloud set, represents the i-th component feature vector of the i-th component point cloud set, represents the i-th component feature vector of the i-th component point cloud set, represents the i-th component feature vector of the i-th component point cloud set, represents the i-th component feature vector of the i-th component point cloud set, represents the i-th component feature vector of the i-th component point cloud set, represents the i-th component feature vector of the i-th component point cloud set, represents the i-th component feature vector of the i-th component point cloud set, represents the i-th component feature vector of the i-th component point cloud set, represents the i-th component feature vector of the i-th component point cloud set, represents the i-th component feature vector of the i-th component point cloud set, , represents the total number of component three-dimensional coordinate points;

[0201] The component feature vector is input into the Transformer module to obtain the corresponding query matrix, key matrix and value matrix, and the specific formula is:

[0202] ;

[0203] ;

[0204] ;

[0205] wherein, represents the query matrix, represents the key matrix, represents the value matrix, represents the linear mapping matrix of the query matrix, represents the linear mapping matrix of the key matrix, represents the linear mapping matrix of the value matrix, represents the i-th component feature vector of the i-th component point cloud set, represents the i-th component feature vector of the i-th component point cloud set, represents the i-th component feature vector of the i-th component point cloud set, T represents the transpose;

[0206] The standard scaled dot-product attention is used to obtain the attention weight, so as to capture the long-distance dependency relationship between components, and the specific formula is:

[0207] ;

[0208] wherein, represents the attention mechanism, represents the probability normalization function, represents the transpose of , represents the dimension of the key vector;

[0209] Long-range dependencies are encoded into the output features of each component to obtain context-aware features. Each context-aware feature contains global information about the relationships between components.

[0210] S6. Input the context-aware features into the pre-trained gated attention cleansing model to obtain the original gate vector. Correct this original gate vector using a gate truncation threshold. Then, use the corrected gate vector to perform weighted filtering of the context-aware features to obtain the cleaned features; specifically:

[0211] Context-aware features are input into a pre-trained gated attention cleansing model. Structural and relational information is extracted using standard Transformer blocks to obtain latent structural features. These latent structural features are then processed sequentially through an MLP network and a Sigmoid activation function to generate the original gated vector. An adaptively adjustable gate truncation threshold is then used. (Set to 0 in the initial iteration) The original gating vector is corrected to obtain the corrected gating vector. The specific formula is as follows:

[0212] ;

[0213] in, Indicates the first The corrected gating vector in each dimension, Indicates the first The original gated vector of each dimension;

[0214] The context-aware features are weighted and filtered using the modified gating vector, and the purified features are obtained by element-wise multiplication. The specific formula is as follows:

[0215] ;

[0216] in, Indicates the first In the point cloud set of the component, the first The purification features of each component's three-dimensional coordinate points Indicates the first In the point cloud set of the component, the first Structural features of the three-dimensional coordinate points of each component It represents the Hadamardi (or Hadama) stack;

[0217] The specific formula for the separated suppressed noise features is:

[0218] ;

[0219] in, Indicates the first In the point cloud set of the component, the first Noise characteristics of three-dimensional coordinate points of a component.

[0220] S7, constructing a two-dimensional uniform grid point set including copies of the purification feature and splicing with the coordinates of each grid point in the two-dimensional uniform grid point set to obtain a high-dimensional feature vector, inputting the high-dimensional feature vector into a pre-trained folding decoder to obtain a completed component point cloud set; specifically: In a two-dimensional unit square, a two-dimensional uniform grid point set including

[0221] copies of the purification feature is generated ;

[0222] copies of the purification feature are copied and spliced with the grid point coordinates in each two-dimensional uniform grid point set to form a high-dimensional feature vector , and the specific formula is:

[0223] ;

[0224] wherein, represents the target coordinates of the th uniform grid, represents the target vertex of the th uniform grid;

[0225] The high-dimensional feature vector is input into a pre-trained folding decoder composed of multiple layers of perception, and through a nonlinear transformation, the two-dimensional uniform grid point and its accompanying global feature are mapped and folded into a three-dimensional space, outputting three-dimensional coordinates to obtain a completed component point cloud set, and the specific formula is:

[0226] ;

[0227] wherein, represents the completed component point cloud set, represents the folding decoder.

[0228] It is worth noting that the folding decoder adopts a shared weight structure based on multiple layers of perception. Since the multiple layers of perception are independent point-by-point processing, it essentially learns a continuous mapping function from a two-dimensional plane domain to a three-dimensional curved surface manifold, rather than a simple correspondence of discrete points. Therefore, the folding decoder has resolution independence in the inference stage, which means that although a fixed number of grid points may be used in the pre-training stage, the density of the input two-dimensional uniform grid point set (i.e., the value of ) can be flexibly adjusted according to feedback instructions in the actual inference completion stage, thereby physically achieving higher resolution sampling of the three-dimensional curved surface to solve the point cloud sparsity problem without retraining the model parameters.​

[0229] S8. Combine the completed part point cloud set in the original coordinate system to obtain the final point cloud set. Improve the gradient field of each part's 3D coordinate point in the final point cloud set. Based on the smoothing factor, the improved gradient field, and the Poisson equation, obtain the smoothed target vertices. The smoothed target vertices constitute the smoothed point cloud set. Specifically:

[0230] The completed point cloud set of components is combined in the original coordinate system to obtain the final point cloud set. The specific formula is as follows:

[0231] ;

[0232] in, This represents the total number of point cloud sets for each component;

[0233] The gradient field of each component's 3D coordinate point in the final point cloud set is improved by utilizing local curvature, resulting in the improved gradient field. The specific formula is as follows:

[0234] ;

[0235] in, Represents the gradient operator, Represents the gradient field. This represents the improved gradient field. Indicates the first The weights of the three-dimensional coordinate points of each component Indicates the first The normal vector of each component's three-dimensional coordinate points Indicates the first Local curvature of three-dimensional coordinate points of each component ;

[0236] Based on the improved gradient field and Poisson equation, the three-dimensional scalar field is obtained, with the specific formula as follows:

[0237] ;

[0238] in, Indicates Fourier transform, Represents a three-dimensional scalar field. Represents the frequency vector. Represents the regularization parameter. Represents the imaginary unit;

[0239] A triangular mesh is obtained by processing a three-dimensional scalar field using an adaptive threshold; where the adaptive threshold... The expression is:

[0240] ;

[0241] in, This represents the initial value of the adaptive threshold. Indicates the first The probability of building damage at three-dimensional coordinates of individual components. All represent equilibrium parameters;

[0242] The Laplacian smoothing method is used to process the triangular mesh to obtain the smoothed target vertices. The specific formula is as follows:

[0243] ;

[0244] in, Represents the smoothed first... A target vertex of a uniform grid. Represents the smoothing factor. Indicates the first The neighborhood point set of the three-dimensional coordinate points of each component Indicates the first The three-dimensional coordinates of the component and its first component The weights between neighboring points Indicates the first The target vertex of the neighborhood points.

[0245] S9. After generating a rendered image from the smoothed point cloud set, input it into a pre-trained multimodal large language model for evaluation. Use prompts to guide the visual language model in evaluating the material and age of the ancient building. Based on the evaluation results, obtain a set of structured parameter adjustment instructions for each component. Update the gating truncation threshold in step S6, the number of grid points in the two-dimensional uniform grid point set in step S7, and the smoothing factor in step S8 according to the instruction set until the evaluation passes, completing the point cloud model completion. Specifically:

[0246] The smoothed point cloud set is subjected to multi-view projection rendering to obtain the corresponding rendered image. In the rendered image, unique component index labels are superimposed on the point clouds belonging to different component sets to generate a two-dimensional image set containing semantic mapping relationships between components. This two-dimensional image set is input into a pre-trained multimodal large language model. Using prompt words to guide the model, visual quality assessment is performed on each independent component in the two-dimensional image set. Automatic identification and semantic inference are performed on the age characteristics, component materials, and craftsmanship style of the ancient building. Based on this, the geometric shape and surface condition of each component are evaluated for quality, resulting in a structured parameter adjustment instruction set for each component. , , Indicates the first Each independent component This indicates the corresponding defect type; the structured parameter adjustment instruction set includes the target component index, defect type, and parameter adjustment range.

[0247] The structured parameter adjustment instruction set is parsed, and the results are processed as follows:

[0248] (1) When the instruction set indicates the first When the defect type of an individual component is excessively smooth texture (i.e., the surface of the component is as smooth as plastic, losing the proper carving or weathering texture), reduce the smoothness factor. And increase the balance parameter The specific formula is as follows:

[0249] ;

[0250] ;

[0251] in, All of these represent the preset adjustment step size coefficient. , Indicates the first Smoothing factor adjusted for each individual component Indicates the first Smoothing factor for each individual component Indicates the first Balance parameters adjusted for each individual component Indicates the first Balance parameters of individual components;

[0252] based on and The process of obtaining the smoothed target vertex is repeated until the score of the pre-trained multimodal large language model reaches the threshold.

[0253] (2) When the instruction set instructs the first When the defect type of an individual component is sparse point cloud (i.e., there are visual holes or breaks on the component surface, and the point density is insufficient to support the structure), increase the number of grid points in step S7, specifically using the following formula:

[0254] ;

[0255] in, Indicates the first The number of grid points after adjustment for each individual component Indicates the first The number of grid points for each individual component This represents the preset grid increment constant;

[0256] based on re-perform the process of steps S7-S8 until the score of the pre-trained multi-modal large language model reaches a threshold value to stop;

[0257] (3) When the instruction set indicates that the defect type of the first independent component is noise residue (i.e., there are discrete flying points around the component or the surface presents a non-structural burr feeling), increase the gating clipping threshold in step S6, and the specific formula is:

[0258] ;

[0259] wherein, represents the adjusted gating clipping threshold of the first independent component, the gating clipping threshold of the first independent component, represents the preset clipping increment;

[0260] based on , re-perform the process of steps S6-S8 until the score of the pre-trained multi-modal large language model reaches a threshold value to stop;

[0261] recombine the smoothed point cloud set obtained by re-performing with the point clouds of other components in the original coordinate system to obtain a completed final point cloud set, and complete the point cloud model completion;

[0262] In this process, the semantic understanding ability of the multi-modal large language model provides key prior constraints for the adjustment of geometric parameters. Ancient architectural components of different historical periods and materials have completely different surface geometric characteristics. The multi-modal large language model identifies these characteristics to guide specific parameter optimization. For example, when the multi-modal large language model identifies that the component material is white marble or glazed tile (common in Qing Dynasty buildings, with fine surface texture), if high-frequency noise is detected on the surface, it will be determined as noise residue and generate an instruction to increase the gating clipping threshold to strengthen denoising; when the multi-modal large language model identifies that the component material is weathered brick or wooden carving (common in the Ming Dynasty or earlier, with natural rough texture on the surface), if the surface is too smooth, it will be determined as excessive smoothing of the texture, and an instruction will be generated to reduce the smoothing factor to retain the historical vicissitudes. This semantic-based feedback mechanism ensures that the completion result meets the real characteristics of ancient buildings in terms of geometric accuracy and visual style.

[0263] The pre-training of the dynamic graph convolution network, the gated attention purification model, and the folding decoder includes:

[0264] Calculate the comprehensive damage probability of the three-dimensional coordinate points in the ancient building, and the specific formula is:

[0265] ;

[0266] wherein, denotes the comprehensive defect probability of the i-th three-dimensional coordinate point; both denote weight parameters; denotes the curvature factor of the i-th three-dimensional coordinate point; denotes the normal vector gradient factor of the i-th three-dimensional coordinate point; denotes the weathering spalling factor; denotes the random damage factor;

[0267] a defect threshold is set , the three-dimensional coordinate points with a comprehensive defect probability greater than the defect threshold are removed to obtain a defect point cloud , and the specific formula is:

[0268] ;

[0269] wherein, denotes the complete point cloud set;

[0270] the defect point cloud is taken as a training set, and the dynamic graph convolution network, the gated attention purification model and the folding decoder are trained by back propagation using a multi-task joint loss function to obtain a pre-trained dynamic graph convolution network, gated attention purification model and folding decoder; the specific formula of the multi-task joint loss function is:

[0271] ;

[0272] wherein, denotes the multi-task joint loss function, denotes the reconstruction loss function (calculated by weighted combination of chamfer distance and bulldozer distance), denotes the balance coefficient, denotes the orthogonality loss function;

[0273] Curvature reflects the bending degree of the point cloud surface, and generally, a high curvature area corresponds to a higher structural stress, and thus is more prone to defects. The curvature of the three-dimensional coordinate point is obtained by using the Gaussian curvature, and the specific expression is:

[0274] ;

[0275] wherein, denotes the curvature of the i-th three-dimensional coordinate point, denotes the minimum principal curvature radius within the neighborhood of the i-th three-dimensional coordinate point, denotes the maximum principal curvature radius within the neighborhood of the i-th three-dimensional coordinate point;

[0276] The curvature is normalized to obtain the curvature factor;

[0277] In building structures, areas where the normal vector changes drastically (such as the joints of load-bearing structures) are often points of stress concentration. These areas can be identified by calculating the gradient of the normal vector field. The normal vector of each 3D coordinate point is calculated. Then calculate the spatial gradient of the normal vector. ,Will Normalization is performed to obtain the gradient factor of the normal vector;

[0278] The weathering and erosion factor is obtained using the Poisson point process; the weathering probability is related to the tilt angle of the point cloud surface normal vector (i.e., the degree of exposure to the environment);

[0279] Each three-dimensional coordinate point is assigned a random number uniformly distributed in the interval [0, 1] to obtain the random damage factor. .

[0280] Example:

[0281] The simulation platform was set up, and the experimental hardware and software environment parameters are as follows:

[0282] The operating system is Ubuntu 20.04 LTS, the CPU is a 13th Gen Intel(R) Core(TM) i7-13700H, and the GPU is an NVIDIA GeForce RTX 4060 Laptop GPU (8GB VRAM).

[0283] We use original CAD model files from the industry-recognized 3D model benchmark dataset ShapeNet. For each CAD model, we first uniformly sample its surface to generate a dense point cloud containing 16,384 points. This dense point cloud serves as the high-fidelity ground truth point cloud. Subsequently, from this dense point cloud, we downsample to 2,048 points using a farthest-point sampling algorithm, obtaining a standard-sized complete point cloud set that preserves the original shape and contour while being suitable for network processing. This set is denoted as […]. The size is 2048×3.

[0284] For each 3D coordinate point, construct three neighborhoods with different radii: 0.1, 0.2, and 0.4. The feature descriptor has a dimension of 1434×13. .

[0285] First, an end-to-end deep neural network is constructed and trained, consisting of a dynamic graph convolutional network (for classification), a Transformer module (for context extraction), a gated attention sanitizing model (for denoising), and a folded decoder (for generation).

[0286] Set defect threshold , remove points with a removal probability greater than the threshold, generate realistic point cloud with missing points , corresponding to As a supervised label.

[0287] The deep neural network is trained by back propagation using the multi-task joint loss function, and a pre-trained deep neural network is obtained, and the specific formula is:

[0288]

[0289] Among them, The multi-task joint loss function is represented by L, The reconstruction loss function is represented by L, The balance coefficient is represented by λ, The orthogonality loss function is represented by L.

[0290] The first 16 smallest non-zero eigenvalues corresponding to the low-frequency vibration mode are selected to obtain the manifold harmonic spectrum feature, and the specific formula is:

[0291]

[0292] Figure 2 The classification effect comparison diagram of the manifold harmonic spectrum feature and the traditional geometric feature in the non-rigid deformation in the present application, Figure 2 The left side of the figure shows two shapes of the column member: the original straight state A and the state B of the bending deformation, and the point As the observation object for feature calculation; when using traditional geometric features for calculation, due to the change of local shape, the linearity decreases from 1.0 in the original straight state A to 0.9 in the state B of the bending deformation, resulting in feature mismatch and false classification; when using the manifold harmonic spectrum feature of the present application for calculation, due to its invariance under isometric transformation, the first low-frequency component (0.37) and the second low-frequency component (-0.20) in the original straight state A and the state B of the bending deformation are highly consistent, thereby ensuring that the deformed member is still correctly classified as a column. Therefore, due to the dependence of the manifold harmonic spectrum feature on topological connection, even if the member is bent, the feature remains unchanged, ensuring the classification accuracy.

[0293] A two-dimensional uniform grid point set including 1024 points is constructed, and the smoothing factor The balance parameter .

[0294] Figure 3 The schematic diagram for obtaining the purified feature by using the gated attention purification model, Figure 3 ​The left side in the figure represents a mixed feature signal containing noise interference, which is decomposed into two groups of feature vectors after being processed by the middle gated attention purification model: one group is a purified feature waveform representing the geometric structure, and the other group is a noise feature waveform representing random interference; it can be seen that the purified features and the noise features remain perpendicular in the vector space, thereby achieving complete decoupling of the effective signal and the noise signal.

[0295] Figure 4 A schematic diagram for completing the point cloud model by using the method proposed in the present application is shown in the figure, Figure 4 The adaptive adjustment path of the multi-modal large language model for different defects is shown in the figure: judging the dynasty to which the ancient building belongs, the material information and the expected surface weathering degree and other semantic descriptions are used to guide the multi-modal large language model to give a visual quality evaluation consistent with the ancient building style in a specific cultural context. When detecting sparse point clouds, increase the number of grid points; when detecting excessively smooth texture, reduce the smoothing factor and increase the balance parameter.

[0296] Figure 5 The result figure of point cloud repair for complex structures and deformation structures by using the method proposed in the present application is shown in the figure. As can be seen from the arch repair result in the first column, the input point cloud has a broken missing at the key stress position of the arch top, and the left column has a obvious hollow. Compared with the true value, the result generated by using the method proposed in the present application successfully restores the continuous and smooth circular curve of the arch top, and the connection between the two columnar parts is smooth and natural, without the common straight line fitting or misalignment phenomenon, which reflects the accurate capture of the curvature structure. As can be seen from the columnar repair result in the second column, there is a large amount of truncation missing in the middle of the column body of the input point cloud. The result generated by using the method proposed in the present application not only fills the blank area in the middle, but also has a high axial alignment degree between the upper and lower column bodies, and the point cloud density is uniform, perfectly restoring the geometric integrity of the cylindrical body. As can be seen from the complex roof repair result in the third column, there is a large area of flaky peeling on the roof tiles in the input point cloud, and the eave structure is incomplete. Compared with the true value, the result generated by using the method proposed in the present application not only completes the overall outline of the roof in a macroscopic way, but also clearly retains the layered texture of the eave and the ups and downs of the ridge in a microscopic way, without the averaging phenomenon of sacrificing texture for smoothness, which proves the maintaining ability of the method proposed in the present application for high-frequency details. In summary, whether it is facing geometric topological fracture or complex texture missing, the method proposed in the present application can generate high-fidelity point clouds highly consistent with the true value, and the visual effect is significantly better than that of the incomplete input point cloud.

[0297] Figure 6The actual effect diagram of the method proposed in the application for point cloud repair of large defect structures, the method proposed in the application innovatively introduces a multimodal large language model, and a closed-loop feedback mechanism of generation-evaluation-adjustment is constructed, the mechanism breaks through the limitation of traditional geometric indexes, can diagnose the generated ancient building point cloud from the angle of visual perception, and automatically triggers a differentiated parameter optimization strategy.

[0298] The above only describes the preferred embodiments of the present application, and it should be noted that those skilled in the art can make several improvements and modifications without departing from the technical principles of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A method for ancient building point cloud model completion based on multi-modal feedback and feature purification, characterized in that, The method comprises the following steps: S1, obtaining an actual defect point cloud model of an ancient building, and calculating the local density of each three-dimensional coordinate point in the defect point cloud model; Based on the local density of each three-dimensional coordinate point, an adaptive neighbor number is obtained, and the neighborhood search range is dynamically adjusted; S2, constructing a neighborhood of different radii for each three-dimensional coordinate point, calculating the covariance matrix of the neighborhood by principal component analysis, and obtaining explicit geometric features; Construct local geometric neighbor graph, compute discrete Laplace-Beltrami operator matrix, and perform eigen-decomposition on the operator matrix to obtain corresponding first eigenvectors as manifold harmonic spectrum features; S3, splicing the coordinate value, local density, explicit geometric feature and manifold harmonic spectrum feature of each three-dimensional coordinate point to obtain a feature descriptor; based on the adaptive neighbor number, the feature descriptor is subjected to feature extraction and aggregation of neighborhood information by using a pre-trained dynamic graph convolution network to obtain a component category to which the three-dimensional coordinate point belongs; S4, obtaining the classification confidence of each three-dimensional coordinate point, when the classification confidence is higher than the dynamic confidence threshold, retaining the component category to which the three-dimensional coordinate point corresponding to the classification confidence belongs, otherwise marking it as unclassified, and obtaining the final component category; S5, based on the final component category, a component point cloud set is constructed, a component feature vector corresponding to the component point cloud in the set is obtained, and the vector is input into a Transformer module to capture long-distance dependency between components and obtain context-aware features; S6, inputting the context-aware features into a pre-trained gated attention purification model to obtain an original gating vector, modifying the original gating vector by using a gating truncation threshold, and weighting and screening the context-aware features by using the modified gating vector to obtain purified features; S7、constructing a two-dimensional uniform grid point set including copying the purification features parts, and splicing with the coordinates of each grid point in the two-dimensional uniform grid point set to obtain a high-dimensional feature vector, inputting the high-dimensional feature vector into a pre-trained folding decoder to obtain a completed part point cloud set; S8, combining the completed component point cloud set in the original coordinate system to obtain a final point cloud set, improving the gradient field of each component three-dimensional coordinate point in the final point cloud set, and obtaining a smoothed target vertex based on a smoothing factor, the improved gradient field and a Poisson equation; The smoothed target vertex constitutes a smoothed point cloud set; S9, inputting the rendered image of the smoothed point cloud set into a pre-trained multi-modal large language model for evaluation, guiding the visual language model to evaluate the material and year of the ancient building through a prompt word; based on the evaluation result, a structured parameter adjustment instruction set for each component is obtained, and the gating truncation threshold in step S6, the number of grid points of the two-dimensional uniform grid point set in step S7 and the smoothing factor in step S8 are updated according to the instruction set until the evaluation passes, and the point cloud model completion is completed.

2. The ancient building point cloud model completion method based on multi-modal feedback and feature purification according to claim 1, characterized in that, In step S1, the adaptive neighbor number comprises the following contents: The calculation formula of the local density is: ； in, This represents the i-th three-dimensional coordinate point. This represents the local density of the i-th 3D coordinate point. This indicates that the i-th three-dimensional coordinate point is located at a radius of... The neighborhood point set within, Indicates the attenuation coefficient. This represents the j-th neighboring 3D coordinate point of the i-th 3D coordinate point. express and The square of the Euclidean distance between them Represents an exponential function; The formula for calculating the adaptive neighbor number is: ； wherein, represents the adaptive number of neighbors of the i-th three-dimensional coordinate point, represents an initial value of the number of neighbors, represents a balance coefficient; In step S2, a neighborhood of three different radii is constructed for each three-dimensional coordinate point, and then the i-th three-dimensional coordinate point neighborhood The covariance matrix of the neighborhood is a 3x3 matrix, and the specific formula for obtaining the explicit geometric feature is: ； ； ； wherein, represents linearity within a neighborhood of radius ; represents flatness within a neighborhood of radius ; represents divergence within a neighborhood of radius ; each represents an eigenvalue; , represents the total number of scales; The explicit geometric features include linearity, planarity and divergence; Computing The Gaussian weight between and is given by ； wherein represents the thermal diffusivity; Constructing a normalized laplace-beltrami operator matrix with dimension N*N The specific formula is: ； wherein denotes the identity matrix, denotes the degree matrix, denotes the Gaussian weight; right Perform eigenvalue decomposition to obtain the corresponding eigenvalues. and eigenvectors Select the preceding mode corresponding to the low-frequency vibration mode. The eigenvectors corresponding to the smallest non-zero eigenvalues ​​are used to obtain the harmonic spectrum features of the manifold, as shown in the following formula: ； wherein represents a manifold harmonic spectrum feature, represents a first eigenvector in the manifold harmonic spectrum features of .

3. The ancient building point cloud model completion method based on multi-modal feedback and feature purification according to claim 1, characterized in that, In step S3, the specific formula for obtaining the feature descriptor is: ； in, The feature descriptor representing the i-th 3D coordinate point, This indicates a splicing operation. Indicates radius as linearity within the neighborhood of . This represents the local density of the i-th 3D coordinate point. Indicates radius as Flatness within the neighborhood, Indicates radius as The divergence within the neighborhood of . Indicates radius as linearity within the neighborhood of . Indicates radius as Flatness within the neighborhood, Indicates radius as The divergence within the neighborhood of . The first characteristic in the harmonic spectrum of a manifold 1 eigenvector This represents the i-th three-dimensional coordinate point; Based on the adaptive neighbor number, a k-NN graph is constructed in an enhanced feature space, and an adjacency relationship is generated; The dynamic graph convolution network comprises an edge convolution layer, a maximum pooling layer and a classification head; The feature descriptor is input into the pre-trained dynamic graph convolution network, and the edge convolution layer is used to aggregate neighborhood information, extract local structure features, and the specific expression is: ； wherein, denotes a local structure feature of the i-th three-dimensional coordinate point, denotes the j-th neighbor three-dimensional coordinate point of the i-th three-dimensional coordinate point, denotes the nearest neighbor point of the adaptive neighborhood of the i-th three-dimensional coordinate point, denotes a learnable function implemented by a multi-layer perceptron, denotes a feature descriptor of The local structure features are subjected to a max-pooling operation through a max-pooling layer to obtain global structure features, and after the local structure features and the global structure features are spliced, the part category to which the three-dimensional coordinate point belongs is obtained through the processing of a classification head and based on an adjacency relationship, and the specific formula is: ； wherein, represents a component category to which the i-th three-dimensional coordinate point belongs, represents a probability normalization function, represents a multi-layer perception, represents a global structure feature.

4. The ancient building point cloud model completion method based on multi-modal feedback and feature purification according to claim 1, characterized in that, In step S4, the calculation formula of the classification confidence is: ； wherein, represents a classification confidence of the i-th three-dimensional coordinate point, represents a highest probability of a component class to which the predicted i-th three-dimensional coordinate point belongs, represents a second highest probability of the predicted i-th three-dimensional coordinate point, represents the i-th three-dimensional coordinate point; The expression of the dynamic confidence threshold is: ； wherein, denotes a dynamic confidence threshold for the i-th three-dimensional coordinate point, denotes an initial value of the dynamic confidence threshold, denotes an adjustment factor, denotes a local density of the i-th three-dimensional coordinate point; The formula for obtaining the final part category to which the three-dimensional coordinate point belongs is: ； wherein, represents the final component class to which the i-th three-dimensional coordinate point belongs, represents the component class to which the i-th three-dimensional coordinate point belongs.

5. The ancient building point cloud model completion method based on multi-modal feedback and feature purification according to claim 1, characterized in that, In step S5, the three-dimensional coordinate points of the same category in the final part category are constructed into a part point cloud set, an initial part feature vector corresponding to the part three-dimensional coordinate points in the set is extracted by using an encoder network, a semantic vector corresponding to the part three-dimensional coordinate points in the set is obtained, and the initial part feature vector and the semantic vector are added element by element to obtain a part feature vector, and the specific formula is: ； wherein, represents a component feature vector of a component three-dimensional coordinate point in a component point cloud set of the i-th component, represents a component feature vector of a component three-dimensional coordinate point in a component point cloud set of the i-th component, represents an initial component feature vector of a component three-dimensional coordinate point in a component point cloud set of the i-th component, represents a semantic vector of a component three-dimensional coordinate point in a component point cloud set of the i-th component, represents a component three-dimensional coordinate point of the i-th component, , represents a total number of component three-dimensional coordinate points;​​​​​​ The part feature vector is input into a Transformer module to obtain a corresponding query matrix, key matrix and value matrix, and the specific formula is: ； ； ； in, Represents the query matrix. Represents the key matrix. Represents a value matrix, The linear mapping matrix representing the query matrix. The linear mapping matrix representing the key matrix. The linear mapping matrix representing the value matrix. Indicates the first In the point cloud set of the component, the first The component feature vector of each component's three-dimensional coordinate points, superscript T Indicates transpose; A standard scaled dot-product attention is used to obtain attention weights, so as to capture long-distance dependency relationships between parts, and the specific formula is: ； wherein, denotes the attention mechanism, denotes a probability normalization function, denotes the transpose of denotes the dimension of the key vector; Encoding long-range dependencies into the output features of each component, resulting in context-aware features .

6. The ancient building point cloud model completion method based on multi-modal feedback and feature purification according to claim 1, characterized in that, In step S6, the obtained purification features include the following contents: The context-aware feature is input into a pre-trained gated attention purification model, structure and relationship information are extracted through a standard Transformer block extraction structure, and potential structure features are obtained. The potential structure features are sequentially processed by an MLP network and a Sigmoid activation function to generate an original gating vector. A gating truncation threshold is used The original gating vector is corrected to obtain a corrected gating vector, and the specific formula is: ； wherein, represents the modified gating vector for the thdimension, represents the original gating vector for the thdimension; The context-aware features are weighted and screened by using the modified gate vector, and the purification features are obtained by element-wise multiplication, and the specific formula is: ； wherein, represents a purification feature of a k-th component three-dimensional coordinate point in a k-th component point cloud set, represents a purification feature of a k-th component three-dimensional coordinate point in a k-th component point cloud set, represents a structure feature of a k-th component three-dimensional coordinate point in a k-th component point cloud set, represents a structure feature of a k-th component three-dimensional coordinate point in a k-th component point cloud set, represents a structure feature of a k-th component three-dimensional coordinate point in a k-th component point cloud set, represents a structure feature of a k-th component three-dimensional coordinate point in a k-th component point cloud set, represents a Hadamard product; The specific formula of the separated suppressed noise features is: ； wherein, represents the noise characteristic of the i-th component three-dimensional coordinate point in the i-th component point cloud set. represents the noise characteristic of the i-th component three-dimensional coordinate point in the i-th component point cloud set.​ 7. The ancient building point cloud model completion method based on multi-modal feedback and feature purification according to claim 1, characterized in that, In step S7, a two-dimensional uniform grid point set including grid points is generated within a two-dimensional unit square ; Copy the purification feature The high-dimensional feature vector is formed by splicing the grid point coordinates of each two-dimensional uniform grid point set, and the number of grid points is 2n The specific formula is: ； in, Indicates the first In the point cloud set of the component, the first The purification features of each component's three-dimensional coordinate points Indicates the first The target coordinates of a uniform grid Indicates the first The target vertex of a uniform grid; The high-dimensional feature vector is input into a pre-trained folding decoder composed of multiple perception mechanisms, a two-dimensional uniform grid point and its accompanying global features are mapped and folded into a three-dimensional space through nonlinear transformation, and three-dimensional coordinates are output, so as to obtain a completed part point cloud set, and the specific formula is: ； wherein, denotes the completed set of component point clouds, denotes a folding decoder.

8. The ancient building point cloud model completion method based on multi-modal feedback and feature purification according to claim 1, characterized in that, In step S8, the completed component point cloud sets are combined under the original coordinate system to obtain a final point cloud set The specific formula is: ； wherein, denotes the total number of component point cloud sets, denotes the completed component point cloud set; The gradient field of each part three-dimensional coordinate point in the final point cloud set is improved by using local curvature, and an improved gradient field is obtained, and the specific formula is: ； wherein, denotes the gradient operator, denotes the gradient field, denotes the improved gradient field, denotes the weight of the third dimensional coordinate point of the component, denotes the normal vector of the third dimensional coordinate point of the component, denotes the local curvature of the third dimensional coordinate point of the component, ; Based on the improved gradient field and the Poisson equation, a three-dimensional scalar field is obtained, and the specific formula is: ； wherein denotes the Fourier transform, denotes a three-dimensional scalar field, denotes a frequency vector, denotes a regularization parameter, denotes the imaginary unit; The three-dimensional scalar field is processed by using an adaptive threshold to obtain a triangular mesh; wherein the adaptive threshold is expressed as: ； wherein, represents an initial value of the adaptive threshold, represents the building damage probability of the third dimensional coordinate point of the component, all represent balance parameters; The triangular mesh is processed by using Laplace smoothing to obtain smoothed target vertices, and the specific formula is: ； wherein, represents a target vertex of the smoothed uniform mesh, represents a smoothing factor, represents a neighborhood point set of the third component three-dimensional coordinate point, represents a weight between the third component three-dimensional coordinate point and its first neighborhood point, represents a target vertex of the first neighborhood point, represents a target vertex of the uniform mesh.

9. The ancient building point cloud model completion method based on multi-modal feedback and feature purification according to claim 8, characterized in that, In step S9, the smoothed point cloud set is subjected to multi-view projection rendering to obtain a corresponding rendered image, and a unique component index label is superimposed on the point cloud belonging to different component sets in the rendered image to generate a two-dimensional image set containing component semantic mapping relationships; the two-dimensional image set is input into a pre-trained multi-modal large language model, a prompt word is used to guide the model to perform visual quality assessment on each independent component in the two-dimensional image set, automatically identify and semantically infer the age characteristics, component material and process style of the ancient building, and on this basis, perform quality evaluation on the geometric shape and surface state of each component to obtain a structured parameter adjustment instruction set for each component , , represents the first independent component, represents the corresponding defect type; the structured parameter adjustment instruction set includes a target component index, a defect type and a parameter adjustment amplitude; The structured parameter adjustment instruction set is parsed, and according to the parsed results, the following processing is performed respectively: (1) When the instruction set indicates that the defect type of the first independent component is texture over-smoothing, the smoothing factor is reduced and the balance parameter is increased , and the specific formula is: ； ； wherein, both represent preset adjustment step coefficients, , represents the adjusted smoothing factor of the th independent component, represents the smoothing factor of the th independent component, represents the adjusted balance parameter of the th independent component, represents the balance parameter of the th independent component; based on and re-executing the process of obtaining smoothed target vertices until the score of the pre-trained multi-modal large language model reaches a threshold stop; (2) When the instruction set indicates that the defect type of the first independent component is a point cloud sparsity, the number of grid points in step S7 is increased, and the specific formula is: ； wherein, represents the number of grid points adjusted by the th independent component, represents the number of grid points of the th independent component, represents a preset grid increment constant; Based on , the process of steps S7-S8 is re-executed until the score of the pre-trained multi-modal large language model reaches the threshold stop; (3) When the instruction set indicates that the defect type of the first independent component is noise residue, increase the gating clipping threshold in step S6, and the specific formula is: ； wherein, denotes the adjusted gating threshold of the th individual component, the gating threshold of the th individual component, denotes a preset increment of the cutoff. Based on , the process of steps S6-S8 is re-executed until the score of the pre-trained multi-modal large language model reaches the threshold stop; The smoothed point cloud set obtained by re-execution and the other part point clouds that are not adjusted are recombined in the original coordinate system to obtain a completed final point cloud set, and the point cloud model completion is completed.

10. The ancient building point cloud model completion method based on multi-modal feedback and feature purification according to claim 1, characterized in that, The pre-training of the dynamic graph convolution network, the gated attention purification model and the folding decoder includes: The comprehensive damage probability of the three-dimensional coordinate point in the ancient building is calculated, and the specific formula is: ； wherein, represents the i-th three-dimensional coordinate point; represents the i-th three-dimensional coordinate point; each represents a weight parameter; represents the i-th three-dimensional coordinate point; represents the i-th three-dimensional coordinate point; represents the i-th three-dimensional coordinate point; represents the i-th three-dimensional coordinate point; Setting a defect threshold Removing the three-dimensional coordinate points with a comprehensive defect probability greater than the defect threshold to obtain a defect point cloud The specific formula is: ； wherein, denotes the complete set of point clouds; The point cloud of the defect is obtained As a training set, the dynamic graph convolution network, the gated attention purification model and the folding decoder are trained by using a multi-task joint loss function respectively, to obtain a pre-trained dynamic graph convolution network, a pre-trained gated attention purification model and a pre-trained folding decoder; a specific formula of the multi-task joint loss function is: ； wherein, denotes a multi-task joint loss function, denotes a reconstruction loss function, denotes a balancing coefficient, denotes an orthogonality loss function; The curvature of the three-dimensional coordinate point is obtained by using Gaussian curvature, and the specific expression is: ； wherein, represents the curvature of the ith three-dimensional coordinate point, represents the minimum principal radius of curvature within the neighborhood of the ith three-dimensional coordinate point, represents the maximum principal radius of curvature within the neighborhood of the ith three-dimensional coordinate point; The curvature is normalized to obtain a curvature factor; calculating a normal vector for each three-dimensional coordinate point recomputing the spatial gradient of the normal vector normalizing the to obtain a normal vector gradient factor The weathering spalling factor is obtained by using the Poisson point process; A random number uniformly distributed in the interval [0, 1] is assigned to each three-dimensional coordinate point to obtain a random damage factor.

Images