Cultural relic fragment splicing, repairing and complementing system and method

Abstract

The invention provides a cultural relic fragment splicing, repairing and complementing system and method, and the system comprises a multi-modal large model understanding and vectorization processing module which is used for carrying out the understanding of cultural relic fragments through a multi-modal large model, and generating a multi-modal semantic vector through cross-modal vectorization processing; the knowledge graph construction and reasoning module is used for constructing a knowledge graph of the cultural relic fragments and carrying out reasoning and verification on splicing of the cultural relic fragments; and the fragment matching and splicing module is used for performing reasoning by utilizing the multi-modal semantic vector and the knowledge graph enhanced historical information to complete splicing and repairing of fragments. According to the technical scheme, by fusing the multi-modal data and generating the unified semantic vector, the form, texture and historical background of the fragments can be considered at the same time, and the splicing and repairing accuracy and integrity are remarkably improved.

Description

Background Art

[0002] Traditional cultural relic restoration usually relies on manual judgment and processing, requiring professional restorers to have deep knowledge and experience in cultural relics. The splicing and completion of cultural relic fragments is a complex and time-consuming task. Especially in the case of severe fragmentation, manual restoration has problems such as low efficiency and high error rate. At the same time, background information such as the age and purpose of cultural relics usually relies on literature research, which further increases the difficulty of restoration.

[0003] In recent years, with the development of computer vision, 3D modeling and natural language processing technology, more and more cultural relics restoration work is carried out with the help of intelligent means. For example, image processing technology is used to identify the shape and texture of cultural relics fragments, and 3D scanning technology is used to reconstruct the shape of cultural relics. However, existing technologies often only target the characteristics of cultural relics in a single mode, lack the comprehensive utilization of multimodal information, and cannot fully consider the morphology, texture, historical information and other aspects of cultural relics, resulting in the lack of accuracy and completeness of the restoration results. Summary of the invention

[0004] The present application provides a system and method for splicing, repairing and completing cultural relic fragments, which are used to achieve deep association between cultural relic characteristics, historical information and knowledge base content, and improve the accuracy of splicing and repair.

[0005] In the first aspect, a system for splicing, repairing and completing cultural relics fragments is provided, comprising:

[0006] The multimodal large model understanding and vectorization processing module is used to understand the cultural relic fragments using the multimodal large model and generate multimodal semantic vectors through cross-modal vectorization processing;

[0007] The knowledge graph construction and reasoning module is used to construct the knowledge graph of cultural relic fragments and to perform reasoning and verification on the splicing of cultural relic fragments;

[0008] The fragment matching and splicing module is used to complete the splicing and repair of fragments by utilizing the multimodal semantic vector and the historical information reasoning enhanced by the knowledge graph.

[0009] In the above technical scheme, a multimodal large model understanding and vectorization processing module is set up to understand the cultural relic fragments using the multimodal large model and generate a multimodal semantic vector through cross-modal vectorization processing; a knowledge graph construction and reasoning module is used to construct a knowledge graph of cultural relic fragments and perform reasoning and verification of the splicing of cultural relic fragments; a fragment matching and splicing module is used to complete the splicing and restoration of fragments using the multimodal semantic vector and the historical information reasoning enhanced by the knowledge graph; by fusing multimodal data (including images, 3D models and texts), a unified semantic vector is generated, which can simultaneously consider the morphology, texture and historical background of the fragments, significantly improving the accuracy and completeness of splicing and restoration; based on the reasoning ability of the multimodal large model and the knowledge graph, the splicing and restoration of cultural relic fragments can be automatically completed, reducing the dependence on manual judgment, greatly improving the restoration efficiency, and reducing the splicing error rate; through the knowledge graph constructed by GraphRAG, the cultural relic fragments are associated with their historical information, documentary materials and archaeological background, enhancing the reasoning ability in the splicing process. This enables the system to not only piece together fragments based on physical features, but also to infer information such as the age and excavation location of the fragments from historical documents, supplementing the contextual knowledge in the restoration process and ensuring the historical authenticity of the restoration.

[0010] In a specific embodiment, it also includes:

[0011] The three-dimensional model reconstruction and completion module is used to generate a complete three-dimensional model of the cultural relic by inferring the shape of the missing parts of the cultural relic after completing the splicing of the cultural relic fragments.

[0012] In a specific embodiment, it also includes:

[0013] The feedback and correction module is used to interact with users and adjust the repair strategy based on user feedback.

[0014] In a specific implementation scheme, the multimodal large model understanding and vectorization processing module includes:

[0015] A data acquisition and preprocessing unit, used to acquire multimodal data of the cultural relic fragments and perform preprocessing;

[0016] A feature extraction and vectorization unit, used for performing feature extraction and vectorization on the preprocessed multimodal data using a multimodal large model;

[0017] The multimodal feature fusion unit is used to fuse the vectorized multimodal data to generate the multimodal semantic vector.

[0018] In a specific implementation scheme, the knowledge graph construction and reasoning module includes:

[0019] A knowledge graph construction unit, used to construct a knowledge graph based on the characteristics and historical background of the cultural relic fragments;

[0020] A unit for combining cultural relics with knowledge graphs, used to compare the multimodal semantic vectors of the cultural relic fragments with the nodes in the knowledge graphs, and to find the relevance of the cultural relic fragments with cultural relics, documents, and historical events;

[0021] The knowledge reasoning and expansion unit is used to obtain relevant information of the cultural relic fragments from an external document library through reasoning during the process of matching and splicing the cultural relic fragments, and expand the knowledge graph; and to infer unmatched fragments and enhance the accuracy of splicing by analyzing the relationship between nodes and edges in the knowledge graph.

[0022] In a specific implementation scheme, the fragment matching and splicing module includes:

[0023] A geometry and texture matching unit, used to match the geometry, edge features and texture information of the fragments based on the similarity of the multimodal semantic vectors; and to determine whether there is a splicing relationship between the cultural relic fragments by measuring the similarity of the feature vectors in the multimodal semantic vectors;

[0024] The knowledge graph enhanced matching unit is used to supplement the verification of the cultural relic fragments through the historical relevance in the knowledge graph.

[0025] In a specific implementation scheme, the three-dimensional model reconstruction and completion module uses NeRF technology or 3DGS technology to infer the shape of the missing parts of the cultural relics and generate a complete three-dimensional model of the cultural relics.

[0026] In a specific embodiment, it also includes:

[0027] The restoration result verification and display module is used to compare and verify the generated three-dimensional model of the cultural relic with the existing cultural relic database and display it.

[0028] In a specific implementation scheme, the repair result verification and display module includes:

[0029] A verification unit for the splicing result is used to compare and verify the generated three-dimensional model of the cultural relic with an existing cultural relic database;

[0030] The restoration effect display unit is used to display the finally generated three-dimensional model of the cultural relic through virtual reality or augmented reality technology.

[0031] In a second aspect, a method for splicing, repairing and completing cultural relic fragments is provided, comprising the following steps:

[0032] Using the understanding and vectorization processing module of the multimodal large model, the multimodal large model is used to understand the cultural relic fragments, and a multimodal semantic vector is generated through cross-modal vectorization processing;

[0033] Use the knowledge graph construction and reasoning module to build a knowledge graph of cultural relic fragments, and perform reasoning and verification on the splicing of cultural relic fragments;

[0034] Using the fragment matching and splicing module, the multimodal semantic vector and the historical information reasoning enhanced by the knowledge graph are used to complete the splicing and repair of the fragments;

[0035] By using the 3D model reconstruction and completion module, after completing the splicing of the cultural relic fragments, a complete 3D model of the cultural relic is generated by inferring the shape of the missing parts of the cultural relic.

[0036] In the above technical scheme, a multimodal large model understanding and vectorization processing module is set up to understand the cultural relic fragments using the multimodal large model and generate a multimodal semantic vector through cross-modal vectorization processing; a knowledge graph construction and reasoning module is used to construct a knowledge graph of cultural relic fragments and perform reasoning and verification of the splicing of cultural relic fragments; a fragment matching and splicing module is used to complete the splicing and restoration of fragments using the multimodal semantic vector and the historical information reasoning enhanced by the knowledge graph; by fusing multimodal data (including images, 3D models and texts), a unified semantic vector is generated, which can simultaneously consider the morphology, texture and historical background of the fragments, significantly improving the accuracy and completeness of splicing and restoration; based on the reasoning ability of the multimodal large model and the knowledge graph, the splicing and restoration of cultural relic fragments can be automatically completed, reducing the dependence on manual judgment, greatly improving the restoration efficiency, and reducing the splicing error rate; through the knowledge graph constructed by GraphRAG, the cultural relic fragments are associated with their historical information, documentary materials and archaeological background, enhancing the reasoning ability in the splicing process. This enables the system to not only piece together fragments based on physical features, but also to infer information such as the age and excavation location of the fragments from historical documents, supplementing the contextual knowledge in the restoration process and ensuring the historical authenticity of the restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A structural block diagram of a system for splicing, repairing and completing cultural relics fragments provided in an embodiment of the present application;

[0038] Figure 2 A flowchart of the method for splicing, repairing and completing cultural relic fragments provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The present application is further described in detail below through the accompanying drawings and embodiments. Through these descriptions, the characteristics and advantages of the present application will become clearer and more specific.

[0040] The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise noted.

[0041] In addition, the technical features involved in different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0042] To facilitate understanding of the cultural relic fragment splicing, repair and completion system and method provided in the embodiment of the present application, its application scenario is first explained. The cultural relic fragment splicing, repair and completion system and method provided in the embodiment of the present application are used to achieve the deep association of cultural relic features, historical information and knowledge base content, and improve the accuracy of splicing and repair. Traditional cultural relic restoration usually relies on manual judgment and processing, requiring professional repair personnel to have deep cultural relic knowledge and experience. Splicing and completing cultural relic fragments is a complex and time-consuming task. Especially in the case of severe fragmentation, manual repair has problems such as low efficiency and high error rate. At the same time, background information such as the age and use of cultural relics usually depends on literature research, which further increases the difficulty of repair. In recent years, with the development of computer vision, 3D modeling and natural language processing technology, more and more cultural relic restoration work is carried out with the help of intelligent means. For example, image processing technology is used to identify the shape and texture of cultural relic fragments, and three-dimensional scanning technology is used to reconstruct the morphology of cultural relics. However, the existing technology often only targets the cultural relic features of a single modality, lacks the comprehensive utilization of multimodal information, and cannot fully consider the morphology, texture, historical information and other aspects of cultural relics, resulting in a lack of accuracy and completeness in the restoration results. To this end, the present application embodiment provides a cultural relic fragment splicing, repairing and completing system and method to achieve a deep association between cultural relic features, historical information and knowledge base content, and improve the accuracy of splicing and repairing. The following is a detailed description of the embodiment in conjunction with specific drawings.

[0043] Terminology explanation:

[0044] 1. GraphRAG (Graph Retrieval-Augmented Generation): GraphRAG is a technology that combines knowledge graph and graph machine learning. It can improve the retrieval, generation and reasoning capabilities based on spatiotemporal graph data. It is particularly suitable for prediction tasks in complex, multi-source data environments.

[0045] 2. Multimodal Large Models, MLM.

[0046] 3. Knowledge Graph, KG.

[0047] refer to Figure 1 and Figure 2 , Figure 1 A structural block diagram of a system for splicing, repairing and completing cultural relics fragments provided in an embodiment of the present application; Figure 2 A flowchart of the method for splicing, repairing and completing cultural relic fragments provided in an embodiment of the present application.

[0048] exist Figure 1 In the embodiment of the present application, a cultural relic fragment splicing, repairing and completing system is provided, including:

[0049] The multimodal large model understanding and vectorization processing module is used to understand the cultural relic fragments using the multimodal large model and generate multimodal semantic vectors through cross-modal vectorization processing;

[0050] The knowledge graph construction and reasoning module is used to construct the knowledge graph of cultural relic fragments and to perform reasoning and verification on the splicing of cultural relic fragments;

[0051] The fragment matching and splicing module is used to complete the splicing and repair of fragments by utilizing the multimodal semantic vector and the historical information reasoning enhanced by the knowledge graph.

[0052] In the above technical scheme, a multimodal large model understanding and vectorization processing module is set up to understand the cultural relic fragments using the multimodal large model and generate a multimodal semantic vector through cross-modal vectorization processing; a knowledge graph construction and reasoning module is used to construct a knowledge graph of cultural relic fragments and perform reasoning and verification of the splicing of cultural relic fragments; a fragment matching and splicing module is used to complete the splicing and restoration of fragments using the multimodal semantic vector and the historical information reasoning enhanced by the knowledge graph; by fusing multimodal data (including images, 3D models and texts), a unified semantic vector is generated, which can simultaneously consider the morphology, texture and historical background of the fragments, significantly improving the accuracy and completeness of splicing and restoration; based on the reasoning ability of the multimodal large model and the knowledge graph, the splicing and restoration of cultural relic fragments can be automatically completed, reducing the dependence on manual judgment, greatly improving the restoration efficiency, and reducing the splicing error rate; through the knowledge graph constructed by GraphRAG, the cultural relic fragments are associated with their historical information, documentary materials and archaeological background, enhancing the reasoning ability in the splicing process. This enables the system to not only piece together fragments based on physical features, but also to infer information such as the age and excavation location of the fragments from historical documents, supplementing the contextual knowledge in the restoration process and ensuring the historical authenticity of the restoration.

[0053] In a specific embodiment, it also includes:

[0054] The three-dimensional model reconstruction and completion module is used to generate a complete three-dimensional model of the cultural relic by inferring the shape of the missing parts of the cultural relic after completing the splicing of the cultural relic fragments.

[0055] In a specific embodiment, it also includes:

[0056] The feedback and correction module is used to interact with users and adjust the repair strategy based on user feedback.

[0057] Specifically, the cultural relic fragment splicing, restoration and completion system is a cultural relic restoration and completion system based on a multimodal large model combined with knowledge graph technology. Through the large model's understanding and vectorization of multimodal inputs (such as images, 3D models, text, etc.), the system can intelligently splice incomplete cultural relic fragments, and combine with the literature knowledge base for reasoning to automatically complete the completion and age inference of cultural relics. At the same time, GraphRAG technology is used to construct a knowledge graph of cultural relics, to achieve a deep association between cultural relic features, historical information and knowledge base content, in order to improve the accuracy of splicing and restoration. Including:

[0058] 1. Understanding and vectorization processing module of multimodal large models

[0059] Receive and process images, 3D models, and text information from cultural relic fragments, vectorize and fuse these data to generate multimodal semantic vectors of cultural relic fragments.

[0060] Through multimodal large models (such as images, 3D models and text), we can deeply understand the cultural relic fragments, use cross-modal vectorization processing, integrate visual, geometric, semantic and other features, and generate a unified multimodal semantic vector. This fusion can more accurately restore the shape and historical background information of the fragments, and improve the accuracy of splicing and completion.

[0061] 2.GraphRAG knowledge graph construction and reasoning module

[0062] The knowledge graph of cultural relics is constructed through GraphRAG technology, and nodes such as fragment characteristics, historical background, excavation site, and documentary information are associated to conduct reasoning and verification of the splicing of cultural relic fragments.

[0063] Through GraphRAG technology, the system constructs a knowledge graph of cultural relics, associates the characteristics of cultural relics fragments, historical background, excavation sites and other information, and realizes intelligent reasoning and completion. The knowledge graph supports the acquisition of new information from external document libraries, and dynamically expands and adjusts the graph to improve the accuracy of splicing and restoration results. Compared with the simple geometric matching in the prior art, the knowledge graph constructed by GraphRAG in the present invention associates cultural relics fragments with their historical information, documentary materials and archaeological background, and enhances the reasoning ability in the splicing process. This enables the system to not only splice based on physical features, but also infer the age, excavation site and other information of the fragments from historical documents, supplement the contextual knowledge in the restoration process, and ensure the historical authenticity of the restoration. Relying on the knowledge graph and external document library of GraphRAG, the knowledge graph can be dynamically expanded by continuously learning the characteristics and archaeological information of new cultural relics. This capability enables the system to have the characteristics of adaptive learning and expansion, and can continuously improve the accuracy of restoration results over time, with the update of cultural relics database or the increase of archaeological discoveries.

[0064] 3. Fragment matching and splicing module

[0065] Combined with the vector features generated by the multimodal large model, fragment matching is performed using geometric morphology, texture features, etc., and the matching accuracy is enhanced through the knowledge graph.

[0066] By using the automatic matching algorithm of multimodal features and geometric morphology, combined with the historical information reasoning enhanced by the knowledge graph, the system can automatically complete the intelligent splicing and repair of fragments. In particular, for severely broken cultural relics, through knowledge reasoning and historical association verification, the system can accurately predict the splicing relationship between fragments. Traditional splicing of cultural relic fragments relies on manual experience, which is time-consuming, labor-intensive and error-prone. Based on the reasoning ability of the multimodal large model and the knowledge graph, the present invention can automatically complete the splicing and repair of cultural relic fragments, reduce the reliance on manual judgment, greatly improve the repair efficiency, and reduce the splicing error rate.

[0067] 4. 3D model reconstruction and completion module

[0068] After the fragments are pieced together, the shape of the missing parts of the cultural relic is inferred through NeRF or 3DGS technology to generate a complete three-dimensional model.

[0069] By integrating NeRF (neural radiation field) or 3DGS (3D geometry) technology, after the fragments are assembled, the known morphology and texture information of the cultural relics are analyzed to infer the morphology of the missing parts of the cultural relics and generate a high-precision 3D model. This function can provide complete morphological completion for the cultural relics.

[0070] 5. Feedback and correction module

[0071] The system allows users to make interactive corrections during the repair process and adjusts the repair strategy based on user feedback.

[0072] A user interaction function is provided to allow users to make manual adjustments based on their professional experience during the restoration process. The system will make real-time corrections and optimizations based on user feedback to ensure that the final splicing and completion results meet the professional requirements of cultural relics research. Existing systems are often closed automated tools that are difficult to flexibly adjust based on expert feedback. The present invention provides a user interaction function that allows restorers to correct splicing and completion results based on their experience and knowledge, and optimize the system's processing logic in real time through feedback to ensure that the final restoration results meet both technical standards and the professional judgment of experts.

[0073] A dynamically updated cultural relic database is built in to compare the restoration results with the existing cultural relic database to further verify the accuracy of the splicing and provide a reference for future restoration work. This database is constantly expanding and can be upgraded with new cultural relic discoveries and research results.

[0074] In a specific implementation scheme, the multimodal large model understanding and vectorization processing module includes:

[0075] A data acquisition and preprocessing unit, used to acquire multimodal data of the cultural relic fragments and perform preprocessing;

[0076] A feature extraction and vectorization unit, used for performing feature extraction and vectorization on the preprocessed multimodal data using a multimodal large model;

[0077] The multimodal feature fusion unit is used to fuse the vectorized multimodal data to generate the multimodal semantic vector.

[0078] Specifically, the multimodal large model understanding and vectorization processing module includes:

[0079] 1. Data acquisition and preprocessing

[0080] The system accepts multimodal data uploaded by users, including multi-angle high-definition images of cultural relic fragments, three-dimensional models generated by 3D scanning, archaeological reports, historical documents and other text information.

[0081] De-noise and enhance contrast of images, and simplify and smooth the mesh of 3D models to ensure the quality and consistency of input data.

[0082] 2. Feature extraction and vectorization of different modalities

[0083] Image feature extraction: The visual features of the cultural relic fragments, such as shape, edge, color, and texture, are extracted through a convolutional neural network (CNN) and converted into an image feature vector.

[0084] 3D model processing: Extract three-dimensional geometric features such as surfaces, edge contours, etc. through PointNet or Transformer-based 3D model processing networks to generate 3D feature vectors.

[0085] Text information processing: Use natural language processing models (such as BERT and GPT) to extract information such as the age and excavation site of cultural relics from literature and convert it into text feature vectors.

[0086] 3. Multimodal feature fusion

[0087] The above image feature vectors, 3D feature vectors, and text feature vectors are fused through cross-modal LLM to generate a unified multimodal semantic vector for subsequent splicing and completion tasks.

[0088] When processing data of different modalities (images, 3D models, text), the system needs to adaptively adjust the weight of each modality due to their different feature dimensions, information density, and reliability.

[0089] The system first normalizes the features of each modality to ensure that they are fused at similar scales.

[0090] Using contrastive learning, a multimodal alignment model is pre-trained to learn similarity metrics across modalities.

[0091] Through training, the model automatically adjusts the weight of each modality, giving greater weight to modalities with higher information content and better matching. For example, when the texture information of an image is relatively complete, the system will prioritize visual features, while when the document provides more historical background information, the weight of the text modality will increase.

[0092] Finally, the system dynamically assigns weights to each modality by calculating the similarity of the feature vectors of each modality and combining data integrity and quality assessment (such as the clarity of the fragments, the accuracy of the 3D model, etc.).

[0093] In a specific implementation scheme, the knowledge graph construction and reasoning module includes:

[0094] A knowledge graph construction unit, used to construct a knowledge graph based on the characteristics and historical background of the cultural relic fragments;

[0095] A unit for combining cultural relics with knowledge graphs, used to compare the multimodal semantic vectors of the cultural relic fragments with the nodes in the knowledge graphs, and to find the relevance of the cultural relic fragments with cultural relics, documents, and historical events;

[0096] The knowledge reasoning and expansion unit is used to obtain relevant information of the cultural relic fragments from an external document library through reasoning during the process of matching and splicing the cultural relic fragments, and expand the knowledge graph; and to infer unmatched fragments and enhance the accuracy of splicing by analyzing the relationship between nodes and edges in the knowledge graph.

[0097] Specifically, the knowledge graph construction and reasoning module includes:

[0098] 1. Knowledge graph construction

[0099] The system builds a knowledge graph based on the characteristics and historical background of cultural relic fragments. The nodes in the graph represent information such as cultural relic fragments, historical events, excavation sites, and documentary records, while the edges represent the associations or inferred relationships between them.

[0100] 2. Combination of cultural relics and knowledge graphs

[0101] The multimodal semantic vectors of cultural relic fragments are compared with the nodes in the knowledge graph to find the relevance of the fragments with cultural relics, documents, historical events, etc. For example, if the geometric features of two fragments match and they were unearthed in the same area, the system can infer that the two fragments may come from the same cultural relic through the relationship in the graph.

[0102] 3. Knowledge Reasoning and Extension

[0103] During the fragment matching and splicing process, the system uses GraphRAG to perform reasoning, automatically obtain relevant information from external literature libraries, and expand the knowledge graph. By analyzing the relationship between nodes and edges in the graph, the system can infer unmatched fragments and enhance the accuracy of splicing.

[0104] In a specific implementation scheme, the fragment matching and splicing module includes:

[0105] A geometry and texture matching unit, used to match the geometry, edge features and texture information of the fragments based on the similarity of the multimodal semantic vectors; and to determine whether there is a splicing relationship between the cultural relic fragments by measuring the similarity of the feature vectors in the multimodal semantic vectors;

[0106] The knowledge graph enhanced matching unit is used to supplement the verification of the cultural relic fragments through the historical relevance in the knowledge graph.

[0107] Specifically, the fragment matching and splicing module includes:

[0108] 1. Geometry and texture matching

[0109] Based on the similarity of multimodal semantic vectors, the system first matches the geometric shape, edge features and texture information of the fragments. Through the similarity measurement of feature vectors (such as cosine similarity), it determines whether there is a possible splicing relationship between the fragments.

[0110] 2. Knowledge Graph Enhanced Matching

[0111] The system not only relies on geometric features, but also supplements verification through historical relevance in the knowledge graph. For example, if the historical background of two fragments has a direct or indirect correlation in the graph, the splicing scheme will be given priority.

[0112] In a specific implementation scheme, the three-dimensional model reconstruction and completion module uses NeRF technology or 3DGS technology to infer the shape of the missing parts of the cultural relics and generate a complete three-dimensional model of the cultural relics.

[0113] Specifically, the three-dimensional model reconstruction and completion module includes:

[0114] 1. NeRF / 3DGS technology completion

[0115] When the fragments are assembled, the system uses NeRF (Neural Radiance Fields) or 3DGS (3D Geometric Shape) technology to infer the shape of the missing parts. By analyzing the overall shape of the artifact and the geometric features of the assembled parts, a complete three-dimensional model of the artifact is generated.

[0116] 2. Interactive feedback and correction

[0117] Users can interactively modify the repair results automatically generated by the system. The system will adjust the repair strategy in real time based on user feedback to optimize the model generation effect.

[0118] In a specific embodiment, it also includes:

[0119] The restoration result verification and display module is used to compare and verify the generated three-dimensional model of the cultural relic with the existing cultural relic database and display it.

[0120] In a specific implementation scheme, the repair result verification and display module includes:

[0121] A verification unit for the splicing result is used to compare and verify the generated three-dimensional model of the cultural relic with an existing cultural relic database;

[0122] The restoration effect display unit is used to display the finally generated three-dimensional model of the cultural relic through virtual reality or augmented reality technology.

[0123] Specifically, the repair result verification and display module includes:

[0124] 1. Verification of splicing results

[0125] Through historical background reasoning, geometric feature matching and completion of incomplete parts based on the knowledge graph, the system compares the generated cultural relic model with the existing cultural relic database to verify the accuracy of the restoration.

[0126] 2.Repair effect display

[0127] The final three-dimensional model of the cultural relic can be displayed through virtual reality (VR) or augmented reality (AR) technology. Users can view the restoration effect of the cultural relic in 360 degrees and understand its historical background and restoration process through the cultural relic information display interface.

[0128] In a specific embodiment, the cultural relic fragment splicing, repairing and completing system includes:

[0129] 1. Case Background

[0130] An archaeological team discovered a batch of broken ceramic artifacts in an ancient site. There are a large number of these fragments, some of which are severely worn or missing, making them extremely difficult to piece together manually. The surface of the ceramic fragments has rich textures and patterns, but due to their age, these features are blurred and cannot be effectively pieced together manually. At the same time, the archaeological report also recorded the site's location, historical background, and possible cultural era, but no complete artifact form was found.

[0131] 2. System Application

[0132] 1. Data acquisition and preprocessing

[0133] Fragment data collection: Archaeologists used high-definition cameras to take multi-angle images of the ceramic fragments from different angles, and used 3D scanning equipment to generate a three-dimensional model of the fragments.

[0134] Document and historical data entry: At the same time, the archaeological team uploaded the archaeological report and related historical documents of the site, and the system used these text data as part of the background knowledge base.

[0135] Data preprocessing: The system removes noise and enhances contrast of uploaded images, and simplifies and smoothes the mesh of 3D models to ensure data clarity and consistency.

[0136] 2. Multimodal feature extraction and fusion

[0137] Visual feature extraction: The system processes the image of each ceramic fragment through a convolutional neural network (CNN) to extract its visual features such as shape, edges, color and texture.

[0138] 3D geometric feature extraction: Use PointNet to process the 3D model of each fragment and extract geometric shape features, including surface contours and fragment edge features.

[0139] Text information extraction: Based on natural language processing models (such as BERT), the system extracts key information related to the cultural relics, such as the age, cultural background, and excavation site, from archaeological reports.

[0140] Cross-modal feature fusion: The system fuses the above-extracted image, 3D model and text features to generate a unified multimodal semantic vector for subsequent splicing and restoration.

[0141] 3. Intelligent splicing of fragments

[0142] Geometry and texture matching: The system first matches the geometric shape and texture features of the fragments based on the similarity of the multimodal semantic vectors. Using the edge similarity of the fragments and the surface contours of the incomplete parts, the system infers the possible splicing positions.

[0143] Knowledge graph enhanced reasoning: Through the knowledge graph built by GraphRAG technology, the system combines the historical background and archaeological information of the fragments for reasoning verification. For example, if two fragments were unearthed at the same location and of similar age, the system infers that they may belong to the same cultural relic and gives priority to splicing them.

[0144] The splicing results are preliminarily completed: based on geometric matching and knowledge graph reasoning, the system successfully and automatically splices multiple fragments into a partially restored ceramic object model.

[0145] 4. 3D model reconstruction and completion

[0146] 3D model completion: For missing parts after splicing, the system uses NeRF technology to infer the geometric shape of the missing parts, generates the missing areas based on the existing surface structure, and completes the overall shape of the ceramic.

[0147] Model optimization: The system combines the texture information of the surface of the cultural relic to optimize the texture of the completed part, so that the completed model is seamlessly connected with the original part.

[0148] 5. User interaction and feedback

[0149] Expert feedback: Archaeological experts can make minor adjustments to the splicing and completion results generated by the system. For example, if there is uncertainty in the splicing position of a fragment, the expert will make minor adjustments based on experience, and the system will readjust the model based on the expert feedback.

[0150] Real-time correction: The system updates the stitching strategy and completion results in real time based on the experts’ interactive operations, ensuring that the final results meet the professional standards in the field of archaeology.

[0151] 6. Verification and display of repair results

[0152] Verification: The system compares the assembled 3D model of the ceramic with existing ceramics of the same type in the cultural relics database to verify the accuracy of the restoration. For example, by comparing the similarity of texture patterns, the historical authenticity of the restoration results can be confirmed.

[0153] Display: The final restored ceramic model is displayed through VR / AR technology. Archaeologists can view the artifact in 360 degrees and learn about the restoration process, unearthed information and historical background of the cultural relics in detail through the interface.

[0154] 3. Case Results

[0155] After the intelligent splicing and completion of this system, the fragments of this batch of ceramic artifacts were successfully repaired, and the originally incomplete ceramics were reconstructed into a complete three-dimensional model. The restoration results show that the ceramics are a precious artifact from a certain historical period, and the shape, texture and historical background information of the cultural relics have been highly restored. The restoration process is significantly more efficient and accurate than traditional manual restoration methods, and the incomplete parts of the cultural relics have been reasonably completed, providing an important basis for further historical research and exhibition displays.

[0156] exist Figure 2 In the embodiment of the present application, a method for splicing, repairing and completing cultural relic fragments is provided, comprising the following steps:

[0157] Using the understanding and vectorization processing module of the multimodal large model, the multimodal large model is used to understand the cultural relic fragments, and a multimodal semantic vector is generated through cross-modal vectorization processing;

[0158] Use the knowledge graph construction and reasoning module to build a knowledge graph of cultural relic fragments, and perform reasoning and verification on the splicing of cultural relic fragments;

[0159] Using the fragment matching and splicing module, the multimodal semantic vector and the historical information reasoning enhanced by the knowledge graph are used to complete the splicing and repair of the fragments;

[0160] By using the 3D model reconstruction and completion module, after completing the splicing of the cultural relic fragments, a complete 3D model of the cultural relic is generated by inferring the shape of the missing parts of the cultural relic.

[0161] In the above technical scheme, a multimodal large model understanding and vectorization processing module is set up to understand the cultural relic fragments using the multimodal large model and generate a multimodal semantic vector through cross-modal vectorization processing; a knowledge graph construction and reasoning module is used to construct a knowledge graph of cultural relic fragments and perform reasoning and verification of the splicing of cultural relic fragments; a fragment matching and splicing module is used to complete the splicing and restoration of fragments using the multimodal semantic vector and the historical information reasoning enhanced by the knowledge graph; by fusing multimodal data (including images, 3D models and texts), a unified semantic vector is generated, which can simultaneously consider the morphology, texture and historical background of the fragments, significantly improving the accuracy and completeness of splicing and restoration; based on the reasoning ability of the multimodal large model and the knowledge graph, the splicing and restoration of cultural relic fragments can be automatically completed, reducing the dependence on manual judgment, greatly improving the restoration efficiency, and reducing the splicing error rate; through the knowledge graph constructed by GraphRAG, the cultural relic fragments are associated with their historical information, documentary materials and archaeological background, enhancing the reasoning ability in the splicing process. This enables the system to not only piece together fragments based on physical features, but also to infer information such as the age and excavation location of the fragments from historical documents, supplementing the contextual knowledge in the restoration process and ensuring the historical authenticity of the restoration.

[0162] Those skilled in the art will appreciate that the present application may be implemented as a system, method or computer program product.

[0163] Therefore, the present disclosure may be specifically implemented in the following forms, namely: it may be completely hardware, it may be completely software (including firmware, resident software, microcode, etc.), or it may be a combination of hardware and software, generally referred to herein as a "circuit", "module" or "system". In addition, in some embodiments, the present application may also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium may contain computer-readable program code.

[0164] Any combination of one or more computer-readable media can be used. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0165] Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art can change, modify, replace and modify the above embodiments within the scope of the present application. On this basis, a variety of replacements and improvements can be made to the present application, all of which fall within the scope of protection of the present application.

Claims

1. A cultural relic fragment splicing, repair and completion system, characterized in that: include: The multimodal large model understanding and vectorization processing module is used to understand the cultural relic fragments using the multimodal large model and generate multimodal semantic vectors through cross-modal vectorization processing; The knowledge graph construction and reasoning module is used to construct the knowledge graph of cultural relic fragments and to perform reasoning and verification on the splicing of cultural relic fragments; The fragment matching and splicing module is used to complete the splicing and repair of fragments by utilizing the multimodal semantic vector and the historical information reasoning enhanced by the knowledge graph.

2. The cultural relic fragment splicing, repairing and completing system according to claim 1 is characterized in that: Also includes: The three-dimensional model reconstruction and completion module is used to generate a complete three-dimensional model of the cultural relic by inferring the shape of the missing parts of the cultural relic after completing the splicing of the cultural relic fragments.

3. The cultural relic fragment splicing, repairing and completing system according to claim 2 is characterized in that: Also includes: The feedback and correction module is used to interact with users and adjust the repair strategy based on user feedback.

4. The cultural relic fragment splicing, repairing and completing system according to claim 3 is characterized in that: The multimodal large model understanding and vectorization processing module includes: A data acquisition and preprocessing unit, used to acquire multimodal data of the cultural relic fragments and perform preprocessing; A feature extraction and vectorization unit, used for performing feature extraction and vectorization on the preprocessed multimodal data using a multimodal large model; The multimodal feature fusion unit is used to fuse the vectorized multimodal data to generate the multimodal semantic vector.

5. The cultural relic fragment splicing, repairing and completing system according to claim 4 is characterized in that: The knowledge graph construction and reasoning module includes: A knowledge graph construction unit, used to construct a knowledge graph based on the characteristics and historical background of the cultural relic fragments; A unit for combining cultural relics with knowledge graphs, used to compare the multimodal semantic vectors of the cultural relic fragments with the nodes in the knowledge graphs, and to find the relevance of the cultural relic fragments with cultural relics, documents, and historical events; The knowledge reasoning and expansion unit is used to obtain relevant information of the cultural relic fragments from an external document library through reasoning during the process of matching and splicing the cultural relic fragments, and expand the knowledge graph; and to infer unmatched fragments and enhance the accuracy of splicing by analyzing the relationship between nodes and edges in the knowledge graph.

6. The cultural relic fragment splicing, repairing and completing system according to claim 5 is characterized in that: The fragment matching and splicing module includes: A geometry and texture matching unit, used to match the geometry, edge features and texture information of the fragments based on the similarity of the multimodal semantic vectors; and to determine whether there is a splicing relationship between the cultural relic fragments by measuring the similarity of the feature vectors in the multimodal semantic vectors; The knowledge graph enhanced matching unit is used to supplement the verification of the cultural relic fragments through the historical relevance in the knowledge graph.

7. The cultural relic fragment splicing, repairing and completing system according to claim 6 is characterized in that: The three-dimensional model reconstruction and completion module uses NeRF technology or 3DGS technology to infer the shape of the missing parts of the cultural relics and generate a complete three-dimensional model of the cultural relics.

8. The cultural relic fragment splicing, repairing and completing system according to claim 7 is characterized in that: Also includes: The restoration result verification and display module is used to compare and verify the generated three-dimensional model of the cultural relic with the existing cultural relic database and display it.

9. The cultural relic fragment splicing, repairing and completing system according to claim 8, characterized in that: The repair result verification and display module includes: A verification unit for the splicing result is used to compare and verify the generated three-dimensional model of the cultural relic with an existing cultural relic database; The restoration effect display unit is used to display the finally generated three-dimensional model of the cultural relic through virtual reality or augmented reality technology.

10. A method for splicing, repairing and completing cultural relic fragments, characterized in that: The following steps are involved: Using the understanding and vectorization processing module of the multimodal large model, the multimodal large model is used to understand the cultural relic fragments, and a multimodal semantic vector is generated through cross-modal vectorization processing; Use the knowledge graph construction and reasoning module to build a knowledge graph of cultural relic fragments, and perform reasoning and verification on the splicing of cultural relic fragments; Using the fragment matching and splicing module, the multimodal semantic vector and the historical information reasoning enhanced by the knowledge graph are used to complete the splicing and repair of the fragments; By using the 3D model reconstruction and completion module, after completing the splicing of the cultural relic fragments, a complete 3D model of the cultural relic is generated by inferring the shape of the missing parts of the cultural relic.

Images