Three-dimensional reconstruction method, system, device and medium for restoring damaged cultural relics

Abstract

The application relates to a three-dimensional reconstruction method, system, device and medium for restoring damaged cultural relics. The method comprises the following steps: obtaining text data, image data and initial point cloud data of the damaged cultural relics, and preprocessing the point cloud data to obtain enhanced point cloud data; performing reverse modeling on a registration point cloud model to generate a three-dimensional reverse model of the damaged cultural relics; inputting the text data, the image data and the three-dimensional reverse model of the damaged cultural relics into a multi-source fusion restoration reconstruction model to generate a three-dimensional restoration model of the damaged cultural relics. The method can realize high-precision point cloud collection and registration through multi-source data fusion; in combination with the generalization capability of a large model in form generation and semantic reasoning, the method can integrate image data and literature description, complete intelligent completion and reasonable reconstruction of missing structures, break through the traditional repair mode, and promote the transformation of cultural relic repair from artificial experience driving to intelligentization.

Description

Technical Field

[0001] This application belongs to the field of three-dimensional reconstruction, and in particular relates to a three-dimensional reconstruction method, system, equipment and medium for the restoration of damaged cultural relics. Background Technology

[0002] Cultural relics are important witnesses to the development of human civilization, carrying multiple values ​​of history, culture, science, and art. They are key carriers for studying the evolution of social life, ideologies, religious beliefs, and economic activities. As carriers of cultural information from specific historical periods, cultural relics not only have archaeological research value but also serve as an important material foundation for shaping national identity, inheriting cultural heritage, and protecting historical memory. Their protection and restoration are an important part of the national cultural strategy. With technological advancements and the development of the times, digital preservation of cultural relics has become a mainstream trend, especially the digital restoration technology for damaged and incomplete relics, which urgently needs to overcome existing technological bottlenecks.

[0003] Current digital restoration technologies mainly rely on indirect references from existing complete cultural relic samples, resulting in problems such as insufficient restoration accuracy, poor structural rationality, and low semantic consistency. Furthermore, existing point cloud stitching technologies are prone to getting trapped in local optima and produce poor stitching and registration results. Summary of the Invention

[0004] Therefore, it is necessary to provide a three-dimensional reconstruction method, system, equipment, and medium that can improve the point cloud stitching effect and accurately restore the appearance of cultural relics to address the above-mentioned technical problems.

[0005] Firstly, this application provides a three-dimensional reconstruction method for the restoration of damaged cultural relics, including:

[0006] The textual data, image data, and initial point cloud data of the damaged cultural relics were acquired, and the point cloud data was preprocessed to obtain enhanced point cloud data.

[0007] The enhanced point cloud data is stitched together to obtain a registered point cloud model;

[0008] The registered point cloud model is reverse-engineered to generate a 3D reverse model of the damaged cultural relic;

[0009] Textual data, image data, and 3D reverse model of damaged cultural relics are input into the multi-source fusion restoration and reconstruction model to generate a 3D restoration model of the damaged cultural relics.

[0010] In one embodiment, the multi-source fusion reconstruction model includes a text encoder, a visual encoder, a mesh encoder, and a cross-modal fusion converter. Textual data, image data, and a 3D inverse model of the damaged artifact are input into the multi-source fusion reconstruction model to generate a 3D reconstruction model of the damaged artifact, including:

[0011] The text data is input into the text encoder to obtain the semantic features of the text data;

[0012] The image data is input into the visual encoder to obtain the visual features of the cultural relic image;

[0013] The 3D reverse model of the damaged cultural relic is input into the mesh encoder to obtain the semantic features of the model;

[0014] The semantic features of textual data, the visual features of cultural relic images, and the semantic features of the model are input into a cross-modal fusion converter to generate a three-dimensional restoration model of the damaged cultural relic.

[0015] In one embodiment, the text encoder is a guided language-image pre-trained text encoder, the visual encoder is a guided language-image pre-trained visual encoder, and the grid encoder is a grid BERT encoder.

[0016] In one embodiment, for a single training sample of the multi-source fusion restoration and reconstruction model, the expression for the loss function of the multi-source fusion restoration and reconstruction model is:

[0017]

[0018]

[0019]

[0020] In the formula, The loss function of the multi-source fusion restoration and reconstruction model is... and These are the mesh reconstruction loss term and the semantic consistency loss term, respectively. and These are the coefficients of the mesh reconstruction loss term and the coefficients of the semantic consistency loss term, respectively. To train a 3D reconstruction model of the damaged cultural relics in the training samples, This serves as the standard reference model corresponding to the 3D reconstruction model of the damaged cultural relics in the training samples. The semantic consistency score of the language-image pre-trained model for the 3D reconstruction model of damaged cultural relics is compared. Image features for the three-dimensional reconstruction model of damaged cultural relics, Preset text features for the three-dimensional reconstruction model of damaged cultural relics.

[0021] In one embodiment, the registered point cloud model is reverse-engineered to generate a 3D reverse model of the damaged cultural relic, including:

[0022] The registered point cloud model is input into the point cloud semantic segmentation network to perform point cloud semantic segmentation, resulting in a semantically segmented point cloud model.

[0023] Based on the semantic segmentation point cloud model, the Poisson surface reconstruction algorithm and the sphere rotation algorithm are combined to solve the Poisson equation and obtain the triangular mesh model;

[0024] The triangular mesh model is smoothed using the Taubin smoothing algorithm to obtain a smoothed mesh model.

[0025] B-spline surface fitting is performed on the smooth mesh model to generate a three-dimensional inverse model of the damaged cultural relic.

[0026] In one embodiment, the enhanced point cloud data is stitched together to obtain a registered point cloud model, including:

[0027] The enhanced point cloud data is coarsely registered using the fast point feature histogram algorithm and / or the random sampling consensus algorithm to obtain the initial rigid transformation matrix.

[0028] Based on the enhanced point cloud data and the initial rigid transformation matrix, combined with the point cloud stitching multi-scale feature fusion network, iterative optimization registration is performed to obtain the iteratively registered point cloud model.

[0029] The iterative nearest point algorithm is used to perform fine registration on the iterative registration point cloud model to obtain the registered point cloud model.

[0030] In one embodiment, the point cloud stitching multi-scale feature fusion network includes a source point cloud feature extraction component, a reference point cloud feature extraction component, a source point cloud mask prediction component, a reference point cloud mask prediction component, and a rigid linear transformation regression component. The source point cloud feature extraction component includes a source point cloud multi-scale feature extraction unit, a source point cloud mask fusion unit, and a source point cloud multi-scale fusion feature extraction unit. The reference point cloud feature extraction component includes a reference point cloud multi-scale feature extraction unit, a reference point cloud mask fusion unit, and a reference point cloud multi-scale fusion feature extraction unit. The enhanced point cloud data includes an enhanced source point cloud model and an enhanced reference point cloud model. Based on the enhanced point cloud data and the initial rigid transformation matrix, combined with the point cloud stitching multi-scale feature fusion network, iterative optimization registration is performed to obtain an iteratively registered point cloud model, including:

[0031] Step S1: Obtain the initial source point cloud mask and the initial reference point cloud mask, and initialize the enhanced source point cloud rigid transformation iterative optimization parameters based on the initial rigid transformation matrix, initialize the source point cloud mask based on the initial source point cloud mask, and initialize the reference point cloud mask based on the initial reference point cloud mask.

[0032] Step S2: Based on the iterative optimization parameters of the enhanced source point cloud rigid transformation, perform rigid transformation on the enhanced source point cloud model to obtain the iterative registration source point cloud model for this iteration round;

[0033] Step S3: Input the iteratively registered source point cloud model into the source point cloud multi-scale feature extraction unit to obtain the first-scale source point cloud features and the second-scale source point cloud features of this iteration. Then, input the second-scale source point cloud features and the source point cloud mask into the source point cloud mask fusion unit to obtain the source point cloud mask fusion feature map of this iteration.

[0034] Step S4: Input the enhanced reference point cloud model into the reference point cloud multi-scale feature extraction unit to obtain the first-scale reference point cloud features and the second-scale reference point cloud features. Then, input the second-scale reference point cloud features and the reference point cloud mask into the reference point cloud mask fusion unit to obtain the reference point cloud mask fusion feature map for this iteration.

[0035] Step S5: The source point cloud features at the first scale, the source point cloud mask fusion feature map, and the reference point cloud mask fusion feature map are concatenated and input into the source point cloud multi-scale fusion feature extraction unit to obtain the source point cloud fusion features for this iteration. The reference point cloud features at the first scale, the source point cloud mask fusion feature map, and the reference point cloud mask fusion feature map are concatenated and input into the reference point cloud multi-scale fusion feature extraction unit to obtain the reference point cloud fusion features for this iteration.

[0036] Step S6: Multiply the source point cloud fusion feature and the source point cloud mask element by element and input them into the source point cloud mask prediction component to obtain the source point cloud mask prediction feature and the source point cloud prediction mask for this iteration; and multiply the reference point cloud fusion feature and the reference point cloud mask element by element and input them into the reference point cloud mask prediction component to obtain the reference point cloud mask prediction feature and the reference point cloud prediction mask for this iteration.

[0037] Step S7: Multiply the source point cloud mask prediction feature and the source point cloud mask element by element to obtain the source point cloud mask prediction feature map for this iteration round. Multiply the reference point cloud mask prediction feature and the reference point cloud mask element by element to obtain the reference point cloud mask prediction feature map for this iteration round. Then, concatenate the source point cloud mask prediction feature map, the reference point cloud mask prediction feature map, the source point cloud fusion feature, and the reference point cloud fusion feature, and input them into the rigid linear transformation regression component to obtain the source point cloud rigid transformation iterative update parameters for this iteration round.

[0038] Step S8: Determine the end of the iteration according to the preset iteration end rule. If the determination result is the end of the iteration, construct the iterative registration point cloud model based on the enhanced source point cloud model, the enhanced reference point cloud model, and the source point cloud rigid transformation iterative update parameters of this iteration round. Otherwise, update the source point cloud mask based on the source point cloud prediction mask, update the reference point cloud mask based on the reference point cloud prediction mask, update the enhanced source point cloud rigid transformation iterative optimization parameters based on the source point cloud rigid transformation iterative update parameters, and return to step S2.

[0039] Secondly, this application also provides a three-dimensional reconstruction system for the restoration of damaged cultural relics, including:

[0040] The damaged cultural relic data acquisition module is used to acquire textual data, image data, and initial point cloud data of damaged cultural relics, and to preprocess the point cloud data to obtain enhanced point cloud data.

[0041] The damaged cultural relic point cloud stitching module is used to stitch together the enhanced point cloud data to obtain a registered point cloud model;

[0042] The 3D reverse model building module is used to reverse model the registered point cloud model and generate a 3D reverse model of the damaged cultural relics.

[0043] The multi-source fusion intelligent restoration module is used to input textual data, image data, and 3D reverse models of damaged cultural relics into the multi-source fusion restoration and reconstruction model to generate a 3D restoration model of the damaged cultural relics.

[0044] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method as described in any of the first aspects of this application.

[0045] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the first aspects of this application.

[0046] The aforementioned three-dimensional reconstruction methods, systems, equipment, and media for the restoration of damaged cultural relics achieve cross-modal fusion of geometric information from the three-dimensional inverse model of the damaged cultural relics with multi-source heterogeneous data such as the historical background, stylistic features, and image textures of the relics. This fusion is then input into a pre-trained model with cross-modal alignment capabilities. Based on the model's ability to combine visual-semantic joint embedding space and contextual learning, semantic understanding and stylistic completion of missing parts can be achieved. Furthermore, the semantic constraints of textual materials enhance the cultural accuracy and restoration rationality of the three-dimensional reconstruction, thereby enabling intelligent three-dimensional reconstruction that combines data-driven and knowledge-guided approaches. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1A flowchart illustrating a three-dimensional reconstruction method for restoring damaged cultural relics, provided as an embodiment of this application;

[0049] Figure 2 This application provides a schematic diagram of the structure of a point cloud stitching multi-scale feature fusion network according to one embodiment of the present application;

[0050] Figure 3 This is a schematic diagram of the structure of a three-dimensional reconstruction system for the restoration of damaged cultural relics, provided as an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] In one exemplary embodiment of this application, such as Figure 1 As shown, a three-dimensional reconstruction method for restoring damaged cultural relics is provided. This embodiment illustrates the application of this method to a three-dimensional reconstruction terminal. It is understood that this method can also be applied to a three-dimensional reconstruction server, and further to a three-dimensional reconstruction system including both a three-dimensional reconstruction terminal and a three-dimensional reconstruction server, and is implemented through the interaction between the three-dimensional reconstruction terminal and the three-dimensional reconstruction server. In this embodiment, the method includes the following steps:

[0053] Step S101: Obtain textual data, image data, and initial point cloud data of the damaged cultural relic, and preprocess the point cloud data to obtain enhanced point cloud data.

[0054] Specifically, the 3D reconstruction terminal can acquire historical textual data, historical image data, and initial point cloud data of damaged cultural relics. The 3D reconstruction terminal can preprocess the initial point cloud data of the damaged cultural relics to obtain enhanced point cloud data of the damaged cultural relics.

[0055] Optionally, the 3D reconstruction terminal can use a scanning system based on the phase-shifting structured light 3D reconstruction principle to control a projector to project a modulated sinusoidal fringe grating onto the surface of the artifact, and control a camera to simultaneously acquire images of the fringe distortion caused by the geometric shape of the object's surface from the side. After calculating the phase map using a phase decoding algorithm, the 3D reconstruction terminal can reconstruct the precise position of each pixel in 3D space using the principle of triangulation, thus obtaining the initial point cloud data of the damaged artifact.

[0056] Furthermore, the 3D reconstruction terminal can project the scanned point cloud data onto a two-dimensional plane and then classify the point cloud according to its density on each surface. The 3D reconstruction terminal can resample based on a uniform point cloud volume density, retrieve the minimum elevation value, and obtain the point density. The 3D reconstruction terminal can extract the facade wall point cloud by selecting an appropriate point density threshold, thereby reducing the noise generated by the point cloud.

[0057] Optionally, the preprocessing of the initial point cloud data of the damaged cultural relics by the 3D reconstruction terminal may include, but is not limited to, point cloud denoising, point cloud downsampling, and point cloud filtering.

[0058] For example, a 3D reconstruction terminal may, but is not limited to, use Statistical Outlier Removal (SOR) and / or Radius Outlier Removal (ROR) algorithms to denoise the initial point cloud data of damaged cultural relics.

[0059] Optionally, the statistical outlier removal algorithm is a denoising algorithm in point cloud preprocessing. It can improve the purity of point clouds by statistically analyzing the neighborhood distance of each point and removing outliers whose average neighborhood distance is significantly greater than the global mean.

[0060] Optionally, the radius outlier removal algorithm is a denoising algorithm in point cloud preprocessing. It can remove isolated points within the radius range that do not have enough neighbor points to reduce point cloud noise by setting a radius and a minimum number of neighbors.

[0061] For example, a 3D reconstruction terminal may, but is not limited to, use voxel mesh filtering to downsample the initial point cloud data of the damaged cultural relic.

[0062] Specifically, the 3D reconstruction terminal can construct a voxel mesh of equal size in 3D space, and replace all points in the voxel with the centroid of all points in the voxel, thereby achieving the purpose of sparse point cloud.

[0063] For example, a 3D reconstruction terminal can, but is not limited to, perform point cloud filtering on the initial point cloud data of damaged cultural relics based on curvature or density detection of abnormal points, thereby improving the recognition stability of the enhanced point cloud data.

[0064] Step S102: Perform point cloud stitching on the enhanced point cloud data to obtain a registered point cloud model.

[0065] Specifically, the 3D reconstruction terminal can stitch together the enhanced point cloud data of damaged cultural relics to obtain a registered point cloud model of the damaged cultural relics, thereby ensuring that the structure of the cultural relics remains intact.

[0066] Optionally, the 3D reconstruction terminal can perform coarse and fine registration on the enhanced point cloud data of damaged cultural relics, stitching together point cloud data from multiple scanning perspectives into a complete model under a unified coordinate system.

[0067] For example, the 3D reconstruction terminal can extract features and perform semantic analysis on the enhanced point cloud data of damaged cultural relics based on Fast Point Feature Histogram (FPFH), calculate feature vectors describing the geometric relationships of local neighborhoods, and establish corresponding point pairs through feature matching. The 3D reconstruction terminal can also remove mismatches based on the Random Sample Consensus (RANSAC) algorithm, estimate the preliminary rigid transformation matrix, and obtain the initial rigid transformation matrix.

[0068] Optionally, Fast Point Feature Histogram (FPS) is a point cloud feature extraction algorithm that generates fixed-dimensional feature vectors by statistically analyzing geometric relationships such as the angle between the normal vectors of a point and its neighboring points, and the distance between them. This vector is used for feature matching in coarse registration of point clouds.

[0069] Optionally, the random sampling consensus algorithm is a robust algorithm in point cloud registration. It can estimate the rigid transformation matrix by randomly sampling a small number of matching point pairs, count the number of interior points, and iteratively select the optimal matrix to eliminate mismatched points in feature matching.

[0070] For example, the 3D reconstruction terminal can perform iterative optimization registration based on the initial rigid transformation matrix provided by RANSAC, according to the enhanced point cloud data, combined with the point cloud stitching multi-scale feature fusion network, to obtain an iteratively registered point cloud model, thereby further iteratively optimizing the pose.

[0071] For example, the 3D reconstruction terminal can further refine the iterative registration point cloud model using the Iterative Closest Point (ICP) algorithm to obtain the registration point cloud model of the damaged cultural relic.

[0072] Optionally, the iterative nearest neighbor algorithm is a point cloud fine registration algorithm that can find the nearest neighbor pairs between the source point cloud and the target point cloud iteratively, minimize the sum of squared projected distances to optimize the rotation and translation matrices, and achieve high-precision alignment.

[0073] As an illustration, since the surface of damaged cultural relics is often curved, the 3D reconstruction terminal can use the normal direction information to minimize the projection distance of the plane where the corresponding normal vector is located.

[0074] Step S103: Reverse model the registered point cloud model to generate a 3D reverse model of the damaged cultural relic.

[0075] In illustrative terms, a 3D reconstruction terminal can extract key structural features and semantic information of damaged cultural relics based on a registered point cloud model. Combined with skeleton and texture information, it can intelligently recognize boundaries, contours, and symmetry. The 3D reconstruction terminal can employ various strategies such as implicit surface modeling, multi-resolution reconstruction, and parametric modeling to construct high-quality 3D geometric models, ensuring the accuracy of reverse engineering and high-fidelity restoration of the original appearance of the cultural relics.

[0076] Step S104: Input the text data, image data and the 3D reverse model of the damaged cultural relic into the multi-source fusion restoration and reconstruction model to generate the 3D restoration model of the damaged cultural relic.

[0077] For example, a 3D reconstruction terminal can input text data into the text encoder of a multi-source fusion reconstruction model to obtain the semantic features of the text data. The 3D reconstruction terminal can input image data into the visual encoder of the multi-source fusion reconstruction model to obtain the visual features of the artifact image. The 3D reconstruction terminal can input a 3D inverse model of a damaged artifact into the mesh encoder of the multi-source fusion reconstruction model to obtain the semantic features of the model. The 3D reconstruction terminal can then input the semantic features of the text data, the visual features of the artifact image, and the semantic features of the model into a cross-modal fusion converter to generate a 3D reconstruction model of the damaged artifact.

[0078] In the aforementioned 3D reconstruction method for the restoration of damaged cultural relics, point cloud stitching of enhanced point cloud data enables high-precision alignment and integration of point cloud data from different acquisition perspectives and / or different point cloud fragments, resulting in a complete and continuous registered point cloud model. This allows for the complete restoration of the overall form of the cultural relic, providing structurally complete point cloud input for the multi-source fusion restoration and reconstruction model. This ensures the continuity and consistency of geometric form during multi-source fusion, achieving high-precision point cloud acquisition and registration. By inputting textual data, image data, and the 3D reverse model of the damaged cultural relic into the multi-source fusion restoration and reconstruction model, image data and documentary descriptions can be integrated. This ensures that the geometric form of the restored 3D model of the damaged cultural relic is highly consistent with its historical and cultural connotations, truly restoring the historical appearance of the cultural relic. It also enables intelligent completion and reasonable reconstruction of missing structures, promoting the transformation of cultural relic restoration from manual experience-driven to intelligent. Furthermore, by performing reverse modeling on the registered point cloud model, it can be transformed into a 3D reverse model of the damaged cultural relic with continuous surfaces and editable structures, ensuring the integrity of the 3D restoration of the damaged cultural relic and enhancing its practical value.

[0079] In an optional embodiment of this application, the multi-source fusion restoration and reconstruction model may include a text encoder, a visual encoder, a mesh encoder, and a cross-modal fusion converter. Inputting textual data, image data, and a 3D inverse model of the damaged cultural relic into the multi-source fusion restoration and reconstruction model to generate a 3D restoration model of the damaged cultural relic may include:

[0080] Specifically, the 3D reconstruction terminal can input text data into a text encoder to obtain the semantic features of the text data.

[0081] Specifically, the 3D reconstruction terminal can input image data into a visual encoder to obtain the visual features of the cultural relic image.

[0082] Specifically, the 3D reconstruction terminal can input the 3D reverse model of the damaged cultural relic into the mesh encoder to obtain the semantic features of the model.

[0083] Specifically, the 3D reconstruction terminal can input the semantic features of textual data, the visual features of cultural relic images, and the semantic features of the model into a cross-modal fusion converter to generate a 3D restoration model of the damaged cultural relic.

[0084] Optionally, the 3D reconstruction terminal can uniformly input the semantic features of textual materials, the visual features of artifact images, and the semantic features of the model into a cross-modal transformer fusion converter to generate a 3D reconstruction model of the damaged artifact. The cross-modal transformer can achieve semantic collaboration and unified alignment between the image / text and point cloud through cross-attention, mixing attention mappings of different modal sequences in a multi-head attention structure. This results in a unified latent representation used to guide the 3D generation of the 3D reconstruction model of the damaged artifact.

[0085] In an optional embodiment of this application, the text encoder can be a guided language-image pre-trained text encoder, the visual encoder can be a guided language-image pre-trained visual encoder, and the mesh encoder can be a Mesh-BERT encoder.

[0086] Optionally, the guided language-image pre-trained text encoder may be, but is not limited to, the TextEncoder in BLIP-2, and the guided language-image pre-trained vision encoder may be, but is not limited to, the Vision Encoder in BLIP-2.

[0087] In an optional embodiment of this application, for a single training sample of the multi-source fusion restoration and reconstruction model, the expression for the loss function of the multi-source fusion restoration and reconstruction model can be:

[0088]

[0089]

[0090]

[0091] In the formula, The loss function of the multi-source fusion restoration and reconstruction model is... and These are the mesh reconstruction loss term and the semantic consistency loss term, respectively. and These are the coefficients of the mesh reconstruction loss term and the coefficients of the semantic consistency loss term, respectively. To train a 3D reconstruction model of the damaged cultural relics in the training samples, This serves as the standard reference model corresponding to the 3D reconstruction model of the damaged cultural relics in the training samples. The semantic consistency score of the language-image pre-trained model for the 3D reconstruction model of damaged cultural relics is compared. Image features for the three-dimensional reconstruction model of damaged cultural relics, Preset text features for the three-dimensional reconstruction model of damaged cultural relics.

[0092] Furthermore, the loss function of the multi-source fusion restoration and reconstruction model can also include an expert scoring loss term. In this case, the expression for the loss function of the multi-source fusion restoration and reconstruction model can be:

[0093]

[0094] In the formula, For expert scoring of loss items, The coefficient for the expert scoring loss term.

[0095] Indicatively, the expert rating loss item can be used to characterize the cultural compatibility between the 3D reconstruction model of the damaged cultural relic and the damaged cultural relic.

[0096] In one optional embodiment of this application, reverse modeling of the registered point cloud model to generate a 3D reverse model of the damaged cultural relic may include:

[0097] Specifically, the 3D reconstruction terminal can input the registered point cloud model into the point cloud semantic segmentation network to perform point cloud semantic segmentation and obtain a semantically segmented point cloud model.

[0098] Optionally, the point cloud semantic segmentation network can be, but is not limited to, PointNet++. The 3D reconstruction terminal can combine PointNet++ for point cloud semantic segmentation. In the abstract set layer, keypoints are sampled from the point cloud, a local neighborhood is constructed for each sampled point, and local features are extracted within each neighborhood. Multiple layers are stacked to extract features at different scales. In the feature propagation layer, high-level features are interpolated back to the original point cloud space to achieve pixel-level prediction. Finally, a Softmax layer is used to regress and predict the semantic category probability distribution of each point.

[0099] Specifically, the 3D reconstruction terminal can solve the Poisson equation based on the semantic segmentation point cloud model, combined with the Poisson surface reconstruction algorithm and the sphere rotation algorithm, to obtain a triangular mesh model.

[0100] Specifically, the 3D reconstruction terminal can smooth the triangular mesh model based on the Taubin smoothing algorithm to obtain a smooth mesh model.

[0101] Optionally, the Taubin smoothing algorithm, proposed by Gabriel Taubin, is a smoothing algorithm for handling surface noise in 3D models. The Taubin smoothing algorithm iteratively adjusts the vertex positions of the model, removing high-frequency noise while preserving the model's geometric features and overall shape as much as possible. The core idea of ​​the Taubin smoothing algorithm is to divide the vertex position update into two steps: forward smoothing and reverse smoothing, to prevent the model from over-shrinking or collapsing during the smoothing process. The Taubin smoothing algorithm first moves the vertices towards the average position of their neighborhood to achieve smoothing (forward smoothing), and then moves the vertices in the opposite direction by a certain proportion to counteract the volume shrinkage caused by over-smoothing (reverse smoothing). The Taubin smoothing algorithm is widely used in fields such as 3D reconstruction, helping to improve the visual quality of the model and the accuracy of subsequent processing.

[0102] Specifically, the 3D reconstruction terminal can perform B-spline surface fitting on a smooth mesh model to generate a 3D inverse model of the damaged cultural relic.

[0103] In an optional embodiment of this application, performing point cloud stitching on the enhanced point cloud data to obtain a registered point cloud model may include:

[0104] Specifically, the 3D reconstruction terminal can perform coarse registration of the enhanced point cloud data using the fast point feature histogram algorithm and / or random sampling consistency algorithm to obtain the initial rigid transformation matrix.

[0105] Specifically, the 3D reconstruction terminal can perform iterative optimization registration based on enhanced point cloud data and an initial rigid transformation matrix, combined with a point cloud stitching multi-scale feature fusion network, to obtain an iteratively registered point cloud model.

[0106] Specifically, the 3D reconstruction terminal can perform fine registration of the iterative registration point cloud model based on the iterative nearest point algorithm to obtain the registered point cloud model.

[0107] In an optional embodiment of this application, please refer to Figure 2The point cloud stitching multi-scale feature fusion network can include a source point cloud feature extraction component, a reference point cloud feature extraction component, a source point cloud mask prediction component, a reference point cloud mask prediction component, and a rigid linear transformation regression component. The source point cloud feature extraction component can include a source point cloud multi-scale feature extraction unit, a source point cloud mask fusion unit, and a source point cloud multi-scale fusion feature extraction unit. The reference point cloud feature extraction component includes a reference point cloud multi-scale feature extraction unit, a reference point cloud mask fusion unit, and a reference point cloud multi-scale fusion feature extraction unit. The enhanced point cloud data can include an enhanced source point cloud model and an enhanced reference point cloud model. Based on the enhanced point cloud data and the initial rigid transformation matrix, combined with the point cloud stitching multi-scale feature fusion network, iterative optimization registration is performed to obtain an iteratively registered point cloud model, which can include:

[0108] Step S1: Obtain the initial source point cloud mask and the initial reference point cloud mask, and initialize the enhanced source point cloud rigid transformation iterative optimization parameters based on the initial rigid transformation matrix, initialize the source point cloud mask based on the initial source point cloud mask, and initialize the reference point cloud mask based on the initial reference point cloud mask.

[0109] Step S2: Based on the iterative optimization parameters of the enhanced source point cloud rigid transformation, perform rigid transformation on the enhanced source point cloud model to obtain the iterative registration source point cloud model for this iteration round.

[0110] Step S3: Input the iteratively registered source point cloud model into the source point cloud multi-scale feature extraction unit to obtain the first-scale source point cloud features and the second-scale source point cloud features of this iteration. Then, input the second-scale source point cloud features and the source point cloud mask into the source point cloud mask fusion unit to obtain the source point cloud mask fusion feature map of this iteration.

[0111] Step S4: Input the enhanced reference point cloud model into the reference point cloud multi-scale feature extraction unit to obtain the first-scale reference point cloud features and the second-scale reference point cloud features. Then, input the second-scale reference point cloud features and the reference point cloud mask into the reference point cloud mask fusion unit to obtain the reference point cloud mask fusion feature map for this iteration.

[0112] Step S5: The source point cloud features at the first scale, the source point cloud mask fusion feature map, and the reference point cloud mask fusion feature map are concatenated and input into the source point cloud multi-scale fusion feature extraction unit to obtain the source point cloud fusion features for this iteration. The reference point cloud features at the first scale, the source point cloud mask fusion feature map, and the reference point cloud mask fusion feature map are concatenated and input into the reference point cloud multi-scale fusion feature extraction unit to obtain the reference point cloud fusion features for this iteration.

[0113] Step S6: The source point cloud fusion feature and the source point cloud mask are multiplied element-wise and then input into the source point cloud mask prediction component to obtain the source point cloud mask prediction feature and the source point cloud prediction mask for this iteration. Similarly, the reference point cloud fusion feature and the reference point cloud mask are multiplied element-wise and then input into the reference point cloud mask prediction component to obtain the reference point cloud mask prediction feature and the reference point cloud prediction mask for this iteration.

[0114] Step S7: Multiply the source point cloud mask prediction feature and the source point cloud mask element-wise to obtain the source point cloud mask prediction feature map for this iteration. Multiply the reference point cloud mask prediction feature and the reference point cloud mask element-wise to obtain the reference point cloud mask prediction feature map for this iteration. Then, concatenate the source point cloud mask prediction feature map, the reference point cloud mask prediction feature map, the source point cloud fusion feature, and the reference point cloud fusion feature, and input them into the rigid linear transformation regression component to obtain the source point cloud rigid transformation iterative update parameters for this iteration.

[0115] Step S8: Determine the end of the iteration according to the preset iteration end rule. If the determination result is the end of the iteration, construct the iterative registration point cloud model based on the enhanced source point cloud model, the enhanced reference point cloud model, and the source point cloud rigid transformation iterative update parameters of this iteration round. Otherwise, update the source point cloud mask based on the source point cloud prediction mask, update the reference point cloud mask based on the reference point cloud prediction mask, update the enhanced source point cloud rigid transformation iterative optimization parameters based on the source point cloud rigid transformation iterative update parameters, and return to step S2.

[0116] Indicative, Figure 2 The term "convolutional layer F2" indicates that the number of output channels of this convolutional layer is 2. Figure 2 The term "convolutional layer F7" indicates that the number of output channels of this convolutional layer is 7. Figure 2 The "convolutional layer F64" indicates that the number of output channels of this convolutional layer is 64. Figure 2 The "convolutional layer F128" indicates that the number of output channels of this convolutional layer is 128. Figure 2 The "convolutional layer F256" indicates that the number of output channels of this convolutional layer is 256. Figure 2 The "convolutional layer F512" indicates that the number of output channels of this convolutional layer is 512. Figure 2 The "convolutional layer F1024" indicates that the number of output channels of this convolutional layer is 1024.

[0117] For example, the preset iteration termination rules may include, but are not limited to, registration error convergence to below a preset registration error threshold, reaching a preset maximum number of iterations, and the change in rigid transformation parameters being less than a preset threshold for the change in rigid transformation parameters.

[0118] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0119] Based on the same inventive concept, this application also provides a three-dimensional reconstruction system for restoring damaged cultural relics, which implements the three-dimensional reconstruction method for restoring damaged cultural relics as described above. The solution provided by this system is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the three-dimensional reconstruction system for restoring damaged cultural relics provided below can be found in the limitations of the three-dimensional reconstruction method for restoring damaged cultural relics described above, and will not be repeated here.

[0120] In one exemplary embodiment, such as Figure 3 As shown, a three-dimensional reconstruction system 300 for the restoration of damaged cultural relics is provided, including:

[0121] The damaged cultural relic data acquisition module 301 can be used to acquire textual data, image data and initial point cloud data of damaged cultural relics, and preprocess the point cloud data to obtain enhanced point cloud data.

[0122] The damaged cultural relic point cloud stitching module 302 can be used to stitch together enhanced point cloud data to obtain a registered point cloud model.

[0123] The 3D reverse model building module 303 can be used to reverse model a registered point cloud model to generate a 3D reverse model of a damaged cultural relic.

[0124] The multi-source fusion intelligent restoration module 304 can be used to input textual data, image data, and 3D reverse model of damaged cultural relics into the multi-source fusion restoration and reconstruction model to generate a 3D restoration model of the damaged cultural relics.

[0125] In an optional embodiment of this application, the multi-source fusion intelligent restoration module 304 can also be used for:

[0126] The text data is input into the text encoder to obtain the semantic features of the text data.

[0127] The image data is input into the visual encoder to obtain the visual features of the cultural relic image.

[0128] The 3D reverse model of the damaged cultural relic is input into the mesh encoder to obtain the semantic features of the model.

[0129] The semantic features of textual data, the visual features of cultural relic images, and the semantic features of the model are input into a cross-modal fusion converter to generate a three-dimensional restoration model of the damaged cultural relic.

[0130] In an optional embodiment of this application, the three-dimensional reverse model construction module 303 can also be used for:

[0131] The registered point cloud model is input into the point cloud semantic segmentation network to perform point cloud semantic segmentation, resulting in a semantically segmented point cloud model.

[0132] Based on the semantic segmentation point cloud model, the Poisson surface reconstruction algorithm and the sphere rotation algorithm are combined to solve the Poisson equation and obtain the triangular mesh model.

[0133] The triangular mesh model is smoothed using the Taubin smoothing algorithm to obtain a smoothed mesh model.

[0134] B-spline surface fitting is performed on the smooth mesh model to generate a three-dimensional inverse model of the damaged cultural relic.

[0135] In an optional embodiment of this application, the damaged cultural relic point cloud stitching module 302 can also be used for:

[0136] The enhanced point cloud data is coarsely registered using the fast point feature histogram algorithm and / or the random sampling consensus algorithm to obtain the initial rigid transformation matrix.

[0137] Based on enhanced point cloud data and an initial rigid transformation matrix, combined with a point cloud stitching multi-scale feature fusion network, iterative optimization registration is performed to obtain an iteratively registered point cloud model.

[0138] The iterative nearest point algorithm is used to perform fine registration on the iterative registration point cloud model to obtain the registered point cloud model.

[0139] In an optional embodiment of this application, the damaged cultural relic point cloud stitching module 302 can also be used for:

[0140] Step S1: Obtain the initial source point cloud mask and the initial reference point cloud mask, and initialize the enhanced source point cloud rigid transformation iterative optimization parameters based on the initial rigid transformation matrix, initialize the source point cloud mask based on the initial source point cloud mask, and initialize the reference point cloud mask based on the initial reference point cloud mask.

[0141] Step S2: Based on the iterative optimization parameters of the enhanced source point cloud rigid transformation, perform rigid transformation on the enhanced source point cloud model to obtain the iterative registration source point cloud model for this iteration round.

[0142] Step S3: Input the iteratively registered source point cloud model into the source point cloud multi-scale feature extraction unit to obtain the first-scale source point cloud features and the second-scale source point cloud features of this iteration. Then, input the second-scale source point cloud features and the source point cloud mask into the source point cloud mask fusion unit to obtain the source point cloud mask fusion feature map of this iteration.

[0143] Step S4: Input the enhanced reference point cloud model into the reference point cloud multi-scale feature extraction unit to obtain the first-scale reference point cloud features and the second-scale reference point cloud features. Then, input the second-scale reference point cloud features and the reference point cloud mask into the reference point cloud mask fusion unit to obtain the reference point cloud mask fusion feature map for this iteration.

[0144] Step S5: The source point cloud features at the first scale, the source point cloud mask fusion feature map, and the reference point cloud mask fusion feature map are concatenated and input into the source point cloud multi-scale fusion feature extraction unit to obtain the source point cloud fusion features for this iteration. The reference point cloud features at the first scale, the source point cloud mask fusion feature map, and the reference point cloud mask fusion feature map are concatenated and input into the reference point cloud multi-scale fusion feature extraction unit to obtain the reference point cloud fusion features for this iteration.

[0145] Step S6: Multiply the source point cloud fusion feature and the source point cloud mask element-wise and input them into the source point cloud mask prediction component to obtain the source point cloud mask prediction feature and the source point cloud prediction mask for this iteration; and multiply the reference point cloud fusion feature and the reference point cloud mask element-wise and input them into the reference point cloud mask prediction component to obtain the reference point cloud mask prediction feature and the reference point cloud prediction mask for this iteration.

[0146] Step S7: Multiply the source point cloud mask prediction feature and the source point cloud mask element-wise to obtain the source point cloud mask prediction feature map for this iteration. Multiply the reference point cloud mask prediction feature and the reference point cloud mask element-wise to obtain the reference point cloud mask prediction feature map for this iteration. Then, concatenate the source point cloud mask prediction feature map, the reference point cloud mask prediction feature map, the source point cloud fusion feature, and the reference point cloud fusion feature, and input them into the rigid linear transformation regression component to obtain the source point cloud rigid transformation iterative update parameters for this iteration.

[0147] Step S8: An iteration termination determination is performed according to a preset iteration termination rule. If the determination result indicates iteration termination, a registration point cloud model is constructed based on the enhanced source point cloud model, the enhanced reference point cloud model, and the source point cloud rigid transformation iterative update parameters of this iteration. Otherwise, the source point cloud mask is updated based on the source point cloud prediction mask, the reference point cloud mask is updated based on the reference point cloud prediction mask, and the enhanced source point cloud rigid transformation iterative optimization parameters are updated based on the source point cloud rigid transformation iterative update parameters. The process then returns to step S2. In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the three-dimensional reconstruction method for restoring damaged cultural relics as described above.

[0148] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0149] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0150] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A three-dimensional reconstruction method for restoring damaged cultural relics, characterized in that, The method includes: The textual data, image data, and initial point cloud data of the damaged cultural relic are acquired, and the point cloud data is preprocessed to obtain enhanced point cloud data. The enhanced point cloud data is stitched together to obtain a registered point cloud model; The registered point cloud model is reverse-engineered to generate a three-dimensional reverse model of the damaged cultural relic; The text data, the image data, and the 3D reverse model of the damaged cultural relic are input into the multi-source fusion restoration and reconstruction model to generate a 3D restoration model of the damaged cultural relic. The multi-source fusion reconstruction model includes a text encoder, a visual encoder, a mesh encoder, and a cross-modal fusion converter. The text encoder is a guided language-image pre-trained text encoder, the visual encoder is a guided language-image pre-trained visual encoder, and the mesh encoder is a mesh BERT encoder. The step of inputting the text data, the image data, and the 3D inverse model of the damaged cultural relic into the multi-source fusion reconstruction model to generate a 3D reconstruction model of the damaged cultural relic includes: The text data is input into the text encoder to obtain the semantic features of the text data; The image data is input into the visual encoder to obtain the visual features of the cultural relic image; The three-dimensional reverse model of the damaged cultural relic is input into the mesh encoder to obtain the semantic features of the model; The semantic features of the textual data, the visual features of the cultural relic image, and the semantic features of the model are input into the cross-modal fusion converter to generate the three-dimensional restoration model of the damaged cultural relic.

2. The method according to claim 1, characterized in that, For a single training sample of the multi-source fusion restoration and reconstruction model, the expression for the loss function of the multi-source fusion restoration and reconstruction model is: ； ； ； In the formula, The loss function of the multi-source fusion restoration and reconstruction model is... and These are the mesh reconstruction loss term and the semantic consistency loss term, respectively. and These are the coefficients of the mesh reconstruction loss term and the coefficients of the semantic consistency loss term, respectively. The three-dimensional reconstruction model of the damaged cultural relic in the training samples. This is the standard reference model corresponding to the three-dimensional reconstruction model of the damaged cultural relic in the training samples. The semantic consistency score of the language-image pre-trained model for the 3D reconstruction model of the damaged cultural relic is used to compare the semantic consistency of the model. The image features of the three-dimensional reconstruction model of the damaged cultural relic are... These are the preset text features for the three-dimensional reconstruction model of the damaged cultural relic.

3. The method according to claim 1, characterized in that, The process of reverse modeling the registered point cloud model to generate a 3D reverse model of the damaged cultural relic includes: The registered point cloud model is input into the point cloud semantic segmentation network to perform point cloud semantic segmentation, thereby obtaining a semantic segmentation point cloud model. Based on the semantic segmentation point cloud model, the Poisson surface reconstruction algorithm and the sphere rotation algorithm are combined to solve the Poisson equation and obtain a triangular mesh model. The triangular mesh model is smoothed using the Taubin smoothing algorithm to obtain a smoothed mesh model. The smooth mesh model is fitted with a B-spline surface to generate a three-dimensional inverse model of the damaged cultural relic.

4. The method according to any one of claims 1 to 3, characterized in that, The step of stitching together the enhanced point cloud data to obtain a registered point cloud model includes: The enhanced point cloud data is coarsely registered using a fast point feature histogram algorithm and / or a random sampling consensus algorithm to obtain an initial rigid transformation matrix. Based on the enhanced point cloud data and the initial rigid transformation matrix, combined with the point cloud splicing multi-scale feature fusion network, iterative optimization registration is performed to obtain the iteratively registered point cloud model. The iterative nearest point algorithm is used to perform fine registration on the iterative registration point cloud model to obtain the registered point cloud model.

5. The method according to claim 4, characterized in that, The point cloud stitching multi-scale feature fusion network includes a source point cloud feature extraction component, a reference point cloud feature extraction component, a source point cloud mask prediction component, a reference point cloud mask prediction component, and a rigid linear transformation regression component. The source point cloud feature extraction component includes a source point cloud multi-scale feature extraction unit, a source point cloud mask fusion unit, and a source point cloud multi-scale fusion feature extraction unit. The reference point cloud feature extraction component includes a reference point cloud multi-scale feature extraction unit, a reference point cloud mask fusion unit, and a reference point cloud multi-scale fusion feature extraction unit. The enhanced point cloud data includes an enhanced source point cloud model and an enhanced reference point cloud model. Based on the enhanced point cloud data and the initial rigid transformation matrix, combined with the point cloud stitching multi-scale feature fusion network, iterative optimization registration is performed to obtain an iteratively registered point cloud model, including: Step S1: Obtain the initial source point cloud mask and the initial reference point cloud mask, and initialize the enhanced source point cloud rigid transformation iterative optimization parameters based on the initial rigid transformation matrix, initialize the source point cloud mask based on the initial source point cloud mask, and initialize the reference point cloud mask based on the initial reference point cloud mask. Step S2: Based on the iterative optimization parameters of the enhanced source point cloud rigid transformation, perform a rigid transformation on the enhanced source point cloud model to obtain the iterative registration source point cloud model for this iteration round; Step S3: Input the iteratively registered source point cloud model into the source point cloud multi-scale feature extraction unit to obtain the first-scale source point cloud features and the second-scale source point cloud features of this iteration round. Then, input the second-scale source point cloud features and the source point cloud mask into the source point cloud mask fusion unit to obtain the source point cloud mask fusion feature map of this iteration round. Step S4: Input the enhanced reference point cloud model into the reference point cloud multi-scale feature extraction unit to obtain the first-scale reference point cloud features and the second-scale reference point cloud features. Then, input the second-scale reference point cloud features and the reference point cloud mask into the reference point cloud mask fusion unit to obtain the reference point cloud mask fusion feature map for this iteration. Step S5: The source point cloud features at the first scale, the source point cloud mask fusion feature map, and the reference point cloud mask fusion feature map are concatenated and input into the source point cloud multi-scale fusion feature extraction unit to obtain the source point cloud fusion features for this iteration. The reference point cloud features at the first scale, the source point cloud mask fusion feature map, and the reference point cloud mask fusion feature map are concatenated and input into the reference point cloud multi-scale fusion feature extraction unit to obtain the reference point cloud fusion features for this iteration. Step S6: Multiply the source point cloud fusion feature and the source point cloud mask element-wise and input them into the source point cloud mask prediction component to obtain the source point cloud mask prediction feature and the source point cloud prediction mask for this iteration; and multiply the reference point cloud fusion feature and the reference point cloud mask element-wise and input them into the reference point cloud mask prediction component to obtain the reference point cloud mask prediction feature and the reference point cloud prediction mask for this iteration. Step S7: Multiply the source point cloud mask prediction feature and the source point cloud mask element-wise to obtain the source point cloud mask prediction feature map for this iteration round; multiply the reference point cloud mask prediction feature and the reference point cloud mask element-wise to obtain the reference point cloud mask prediction feature map for this iteration round; and concatenate the source point cloud mask prediction feature map, the reference point cloud mask prediction feature map, the source point cloud fusion feature, and the reference point cloud fusion feature, and input them into the rigid linear transformation regression component to obtain the source point cloud rigid transformation iterative update parameters for this iteration round. Step S8: Determine the end of the iteration according to the preset iteration end rule. If the determination result is that the iteration has ended, construct the iterative registration point cloud model based on the enhanced source point cloud model, the enhanced reference point cloud model, and the source point cloud rigid transformation iterative update parameters of this iteration round. Otherwise, update the source point cloud mask based on the source point cloud prediction mask, update the reference point cloud mask based on the reference point cloud prediction mask, update the enhanced source point cloud rigid transformation iterative optimization parameters based on the source point cloud rigid transformation iterative update parameters, and return to step S2.

6. A three-dimensional reconstruction system for the restoration of damaged cultural relics, characterized in that, The system includes various functional modules required to implement the method of any one of claims 1 to 5.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images