Intelligent management and non-destructive analysis method and system for cultural relics multi-modal data

Abstract

The application provides a kind of cultural relic multi-modal data intelligent management and non-destructive analysis method and system, through tree index and two-level registration strategy, solve the problem that the coordinate of cultural relic external three-dimensional scanning data and internal CT data is not unified, management is dispersed, so that researchers can intuitively analyze the "surface and internal relationship" of cultural relics in the same window;Through the standardized reasoning interface, the system can flexibly mount the high-precision deep learning model trained for specific materials, overcoming the problem of insufficient accuracy of traditional general algorithms in complex cultural relic disease extraction, significantly improving the analysis efficiency;Combined with fusion rendering and virtual cutting technology, non-destructive exploration and quantitative evaluation of cultural relic internal corrosion depth and crack trend are realized without touching the cultural relic entity, providing visual basis for human-computer collaborative labeling and repair scheme.

Description

Technical Field

[0001] This invention belongs to the field of computer-aided cultural relic protection and digital analysis technology, and in particular relates to a method and system for intelligent management and non-destructive analysis of multimodal data of cultural relics. Background Technology

[0002] With the widespread application of computed tomography (CT) and 3D laser scanning technologies in archaeological excavation and cultural relic preservation, the digitization of cultural relics has shifted from simply "recording their appearance" to "exploring their internal structure" and "multimodal comprehensive analysis." Researchers not only need to master the high-precision surface texture of cultural relics, but also need to use X-rays to see through their internal manufacturing processes (such as mold seams and tenons) and the distribution of defects (such as powdery rust inside bronzes and invisible cracks inside ceramics).

[0003] However, existing technologies for processing and analyzing cultural relics data mainly suffer from the following problems:

[0004] 1. Fragmented Management of Heterogeneous Data from Multiple Sources, Lacking a Unified Viewpoint: Current workflows for digitizing cultural relics are typically fragmented. X-ray / CT data are usually processed using medical imaging software (such as Mimics and ImageJ), while 3D laser scan data (such as OBJ / STL format) are processed using reverse engineering software (such as Geomagic). This results in the separation of data on the "inner core" and "outer shell" of the cultural relic, making it difficult for researchers to intuitively establish the spatial correspondence between the two within the same software environment, and preventing them from simultaneously observing the relationship between the surface patterns and internal defects of the cultural relic.

[0005] 2. General-purpose segmentation algorithms struggle to handle complex artifact damage: Existing software often integrates general-purpose image segmentation algorithms (such as global thresholding and watershed algorithms), which are primarily designed for medical tissues or industrial standard parts. However, unearthed artifacts (especially bronzes and ceramics) have undergone long periods of burial, resulting in extremely complex density distributions (for example, the mineralized corrosion layer of bronzes has a density close to the body, while the boundary between the glaze and the body of ceramics is blurred). Traditional general-purpose algorithms alone are insufficient to extract effective damage features, while manually annotating CT slices layer by layer is time-consuming and labor-intensive, severely hindering research efficiency.

[0006] 3. Lack of intelligent analysis and extension capabilities for specific materials: Although deep learning technology has developed rapidly in the field of image segmentation, existing cultural relic software is often closed and lacks standardized model interfaces. Dedicated segmentation models trained by researchers for specific cultural relics (such as the Sanxingdui bronzes) are difficult to directly embed into existing visualization software for interactive verification and fine-tuning, making it difficult to quickly transform research results into practical analysis tools.

[0007] 4. Lack of Virtual Restoration and Interactive Analysis Tools: Before formulating cultural relic protection plans (such as rust removal and splicing), preliminary assessments are often required. Existing software focuses mainly on "viewing" and lacks "movement" capabilities. It is difficult to perform interactive operations such as virtual cutting, virtual peeling of rust layers, or simulated splicing in three-dimensional space, thus failing to provide quantitative reference data for physical restoration.

[0008] Therefore, there is an urgent need to develop a system and method that can uniformly manage multi-source heterogeneous data of cultural relics, integrate targeted intelligent segmentation algorithms, and support multimodal fusion and interactive virtual restoration. Summary of the Invention

[0009] To address the challenges of unified management of multimodal data, low accuracy in segmenting specific defects, and lack of interactive virtual restoration methods in existing technologies, this invention provides an intelligent management and non-destructive analysis method and system for multimodal data of cultural relics. It integrates X-ray computed tomography (CT / X-ray) image data with high-precision three-dimensional surface scan data, and utilizes artificial intelligence and computer graphics technology to intelligently identify defects in cultural relics, fuse and display multimodal data, and perform virtual restoration.

[0010] A method for intelligent management and non-destructive analysis of multimodal data of cultural relics includes the following steps: S1: Obtain the CT data of the internal tomographic scan of the cultural relic to be analyzed and the surface data of the external three-dimensional surface scan; construct a hierarchical tree index structure based on the metadata of the CT data and surface data. In the tree index structure, each cultural relic corresponds to a parent node, and the CT data and surface data of the same cultural relic are associated with the same parent node as child nodes. S2: Use a two-level spatial registration strategy to perform coordinate transformation on CT data and surface data so that they are aligned in a unified spatial coordinate system; S3: Based on the material properties of the cultural relic, a pre-trained deep neural network model is called to perform serialized reasoning on the registered CT data, extract the features of the cultural relic itself and the diseased area, and obtain a two-dimensional segmentation mask sequence. S4: Use the surface drawing algorithm to reconstruct the two-dimensional segmentation mask sequence into a three-dimensional disease mesh model, and attach the three-dimensional disease mesh model to the corresponding parent node of the cultural relic in the tree index structure; S5: Based on a dual-channel rendering pipeline, it mixes and displays surface data, CT data, and 3D disease mesh models in the same rendering window; responding to virtual cutting commands, it uses clipping planes to calculate and map cut surface textures in real time, showing the correspondence between the internal structure and external form of cultural relics; S6: In response to the user's interactive correction operation on the segmentation results, the two-dimensional segmentation mask sequence is updated in real time, and the reconstruction process in step S4 is triggered to generate an updated three-dimensional disease mesh model and virtual repair scheme.

[0011] Furthermore, in step S2, a two-level spatial registration strategy is used to perform coordinate transformation on the CT data and surface data to align them in a unified spatial coordinate system. Specifically, this involves: Coarse registration: In response to at least three pairs of corresponding feature points picked by the user on the 3D reconstruction of the CT data and the surface data model, the initial rigid body transformation matrix from CT data to surface data is calculated. Then, based on the initial rigid body transformation matrix, the CT data is transformed to the coordinate system of the surface data to complete the coarse registration. Fine registration: Based on coarse registration, the nearest point iteration algorithm is used to calculate and minimize the spatial distance between CT data and surface data, and output the fine registration rigid body transformation matrix. Then, in response to the user's rotation, scaling or translation operation, the superimposed display effect of CT data and surface data is obtained based on the fine registration rigid body transformation matrix. The spatial position of CT data is manually compensated according to the superimposed display effect until the overlap accuracy between the outline of CT data and the geometric shape of surface data meets the preset requirements, and the fine registration is completed.

[0012] Furthermore, in step S3, the method for performing serialization inference on the registered CT data to extract features of the artifact itself and the diseased area, and obtaining the two-dimensional segmentation mask sequence is as follows: Model loading: Through standardized inference interface units, the corresponding pre-trained deep neural network model is automatically loaded from the model library according to the material category of the cultural relic; Serialization inference: Extract the three-dimensional CT data into a two-dimensional slice sequence along a preset axis, and convert the two-dimensional slice sequence into a tensor format and input it into the pre-trained deep neural network model; Mask generation: The pre-trained deep neural network model outputs a probability map corresponding to the classification prediction results of the represented pixels, where the classification is either the artifact itself or a specific diseased area; then, the probability is adjusted according to a preset threshold. Figure 2 Value-enhanced, generating a two-dimensional segmentation mask sequence marked with the area of ​​the cultural relic itself and the area of ​​damage.

[0013] Furthermore, in step S5, the method for mixing and displaying surface data, CT data, and the 3D disease mesh model in the same rendering window based on a dual-channel rendering pipeline is as follows: The surface data is rendered using solid rendering mode, and the rendering effect is set to semi-transparent. The CT data is rendered using volume rendering mode, and the rendering effect is set to volume rendering material. Depth testing is used to determine whether the occlusion relationship of surface data, CT data, and 3D disease mesh model in 3D space is displayed correctly. If the result is negative, the model is re-rendered.

[0014] Further, in step S5, the virtual cutting command includes the three-dimensional position coordinates (x, y, z) of the cutting plane and the normal vector parameter (n). x , n y , n z ).

[0015] Furthermore, in step S5, the method for responding to the virtual cutting command by using the cutting plane to calculate and map the cut surface texture in real time, and displaying the correspondence between the internal structure and external form of the artifact, is as follows: First, a clipping plane is defined in the 3D rendering space. The spatial geometric intersection of the clipping plane with the external surface data model and the internal 3D disease mesh model is calculated in real time to determine the polygon section profile. Then, along the spatial coordinates corresponding to the clipping plane, the corresponding 2D tomographic pixel values ​​are extracted from the CT data, and the 2D tomographic pixel values ​​are mapped as a section texture into the polygon section profile in real time. Finally, by removing the rendering data on one side of the clipping plane, while retaining the external surface morphology profile of the unclipped part, the internal CT density information and the depth distribution of the disease are displayed based on the section texture.

[0016] Furthermore, in step S1, the metadata of the CT data and surface data includes the name of the artifact, the time of excavation, the location of excavation, the material type, and the scanning equipment parameters.

[0017] Furthermore, the pre-trained deep neural network model is a U-Net, V-Net, or Mask R-CNN network structure.

[0018] Furthermore, in step S6, the interactive correction operations include brush smearing and eraser erasing.

[0019] A system for intelligent management and non-destructive analysis of multimodal data of cultural relics includes a multi-source data management module, a spatial registration module, an intelligent computing engine module, and a visualization interaction module; The multi-source data management module is used to acquire CT data of internal tomographic scans and surface data of external three-dimensional surface scans of the cultural relic to be analyzed; a hierarchical tree index structure is constructed based on the metadata of CT data and surface data. In the tree index structure, each cultural relic corresponds to a parent node, and the CT data and surface data of the same cultural relic are associated as child nodes under the same parent node. The spatial registration module uses a two-level spatial registration strategy to perform coordinate transformation on CT data and surface data, so that the two are aligned in a unified spatial coordinate system. The intelligent computing engine module is used to call a pre-trained deep neural network model based on the material properties of the cultural relic, perform serialized reasoning on the registered CT data, extract the features of the cultural relic body and the diseased area, and obtain a two-dimensional segmentation mask sequence. The intelligent computing engine module is also used to reconstruct the two-dimensional segmentation mask sequence into a three-dimensional disease mesh model using a surface drawing algorithm, and to attach the three-dimensional disease mesh model to the corresponding parent node of the cultural relic in the tree index structure. The visualization interaction module is used to display surface data, CT data, and three-dimensional disease mesh models in the same rendering window based on a dual-channel rendering pipeline; in response to virtual cutting commands, it uses the clipping plane to calculate and map the cut surface texture in real time, showing the correspondence between the internal structure and external form of the cultural relic; The visualization interaction module is also used to respond to the user's interactive correction operation on the segmentation results, update the two-dimensional segmentation mask sequence in real time, and trigger the reconstruction process in the intelligent computing engine module to generate an updated three-dimensional disease mesh model and virtual repair scheme.

[0020] Beneficial effects: This invention provides an intelligent management and non-destructive analysis method for multimodal data of cultural relics. Through a tree-structured index and a two-level registration strategy, it solves the problems of inconsistent coordinates and fragmented management between external 3D scan data and internal CT data of cultural relics, allowing researchers to intuitively analyze the "internal and external relationships" of relics under a single view. Through a standardized inference interface, the system can flexibly mount high-precision deep learning models trained for specific materials, overcoming the insufficient accuracy of traditional general-purpose algorithms in extracting complex cultural relic defects, and significantly improving analysis efficiency. Combined with fusion rendering and virtual cutting technology, it enables non-destructive exploration and quantitative assessment of the depth of internal corrosion and the direction of cracks in cultural relics without physical contact, providing a visual basis for human-machine collaborative annotation and restoration plan formulation. Therefore, this invention achieves unified spatiotemporal management of multi-source heterogeneous data, and based on a hybrid-driven intelligent segmentation mechanism for cultural relic defects, it provides an intuitive means for virtual restoration and interactive analysis. Attached Figure Description

[0021] Figure 1 This is a main flowchart of an intelligent management and non-destructive analysis method for multi-source heterogeneous data of cultural relics provided in an embodiment of the present invention; Figure 2 This is a diagram illustrating the overall functional module architecture of the intelligent analysis system provided in this embodiment of the invention. Figure 3 This is a schematic diagram of the user interface of the multi-source heterogeneous data management module in an embodiment of the present invention, showing the custom tree-structured classification and metadata management functions; Figure 4 This is a schematic diagram of the working interface of the intelligent computing engine module in an embodiment of the present invention, illustrating the intelligent segmentation and manual annotation fine-tuning process of CT images; Figure 5This is a schematic diagram illustrating the intelligent segmentation and three-dimensional reconstruction effect of bronze corrosion in Embodiment 1 of the present invention; Figure 6 This is a schematic diagram of high-precision registration and interactive virtual cutting analysis of multimodal data for the cloisonné enamel incense burner in Embodiment 2 of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0023] Example 1 like Figure 1 As shown, a method for intelligent management and non-destructive analysis of multimodal data of cultural relics includes the following steps: S1: Construct a multi-source heterogeneous data index tree Acquire CT data of the internal tomographic scan of the cultural relic to be analyzed and surface data of the external three-dimensional surface scan; construct a hierarchical tree index structure based on the metadata of the CT data and surface data. In the tree index structure, each cultural relic corresponds to a parent node, and the CT data and surface data of the same cultural relic are associated as child nodes under the same parent node, realizing the logical association of heterogeneous data. The metadata of the CT data and surface data includes attribute information such as the name of the artifact, excavation time, excavation location, material type, and scanning equipment parameters. The system builds a database index based on this metadata, allowing users to quickly locate specific artifact data nodes through keyword searches.

[0024] S2: Multimodal data spatial registration To analyze the data in a unified spatial coordinate system, the CT data and surface data imported in step S1 are subjected to two-level spatial registration. A two-level spatial registration strategy is used to transform the coordinates of the CT data and surface data, aligning them in a unified spatial coordinate system. Specifically: Coarse registration: In response to at least three pairs of corresponding feature points (Landmarks) picked by the user on the 3D reconstruction of the CT data and the surface data model, the initial rigid transformation matrix from CT data to surface data is calculated. Then, based on the initial rigid transformation matrix, the CT data is transformed to the coordinate system of the surface data to complete the coarse registration. Fine registration: Based on coarse registration, the nearest point iteration algorithm (ICP) is used to calculate and minimize the spatial distance between CT data and surface data, and output the fine registration rigid body transformation matrix. Then, in response to the user's rotation, scaling or translation operation, the superimposed display effect of CT data and surface data is obtained based on the fine registration rigid body transformation matrix. The spatial position of CT data is manually compensated according to the superimposed display effect until the overlap accuracy between the outline of CT data and the geometric shape of surface data meets the preset requirements, and the fine registration is completed.

[0025] S3: Intelligent Segmentation Based on Hybrid Drive Based on the material properties of the cultural relic, a pre-trained deep neural network model is invoked to perform sequential inference on the registered CT data, extracting features of the cultural relic itself and the diseased areas to obtain a two-dimensional segmentation mask sequence, as follows: Material definition and model loading: Read the material properties of cultural relics (such as bronze and ceramics), and automatically load the corresponding pre-trained deep neural network model from the model library according to the material category of the cultural relics through a standardized inference interface unit; Slice Sequencing Inference: Extract three-dimensional CT data into two-dimensional slice sequences along a preset axis (such as the Z-axis), and convert the two-dimensional slice sequences into tensor format and input them into the pre-trained deep neural network model; Mask generation: The pre-trained deep neural network model outputs a probability map corresponding to the predicted classification of the represented pixels (e.g., the cultural relic itself, a specific diseased area), where the classification is either the cultural relic itself or a specific diseased area; then, the probability is adjusted according to a preset threshold. Figure 2 Value-enhanced, generating a two-dimensional segmentation mask sequence that marks the area of ​​the artifact itself and the areas of damage (such as corrosion and cracks).

[0026] The inference interface unit adopts a plug-in architecture design, transmits CT slice data to the background inference engine (such as a Python inference script) through a cross-process communication mechanism, and receives the segmentation results returned by the inference engine; the pre-trained deep neural network model includes, but is not limited to, U-Net, V-Net or Mask R-CNN network structures.

[0027] S4: 3D Reconstruction and Node Update The two-dimensional segmented mask sequence is reconstructed into a three-dimensional disease mesh model using a surface rendering algorithm (such as the Marching Cubes algorithm). The three-dimensional disease mesh model is then used as a new child node and automatically attached to the corresponding parent node of the cultural relic in the tree index structure. At the same time, preset rendering colors and transparency attributes can be automatically assigned according to the type of disease (such as the body and corrosion). S5: Multimodal Fusion Display and Virtual Segmentation Based on a dual-channel rendering pipeline (such as the VTK pipeline), surface data, CT data, and 3D disease mesh models are mixed and displayed in the same rendering window; in response to virtual cutting commands, the cutting plane is used to calculate and map the cut surface texture in real time to show the correspondence between the internal structure and external form of the cultural relic; Furthermore, the method for mixing and displaying surface data, CT data, and 3D disease mesh models in the same rendering window based on a dual-channel rendering pipeline is as follows: The surface data is rendered using solid rendering mode, and the rendering effect is set to semi-transparent. The CT data is rendered using volume rendering mode, and the rendering effect is set to volume rendering material. Depth testing is used to determine whether the occlusion relationship of surface data, CT data, and 3D disease mesh model in 3D space is displayed correctly. If the result is negative, the model is re-rendered.

[0028] Furthermore, the virtual cutting command includes the three-dimensional position coordinates (x, y, z) of the cutting plane and the normal vector parameter (n). x , n y , n z ),like Figure 4 As shown, users can modify the above parameters in real time by dragging the visual handle with the mouse, and the system refreshes the rendering buffer in real time according to the modified parameters; in response to the virtual cutting command, the method of using the clipping plane to calculate and map the cut surface texture in real time to show the correspondence between the internal structure and external form of the cultural relic is as follows: First, a clipping plane is defined in the 3D rendering space. The spatial geometric intersection of the clipping plane with the external surface data model and the internal 3D disease mesh model is calculated in real time to determine the polygon section profile. Then, along the spatial coordinates corresponding to the clipping plane, the corresponding 2D tomographic pixel values ​​are extracted from the CT data, and the 2D tomographic pixel values ​​are mapped as a section texture into the polygon section profile in real time. Finally, by removing the rendering data on one side of the clipping plane, while retaining the external surface morphology profile of the unclipped part, the internal CT density information and the depth distribution of the disease are displayed based on the section texture.

[0029] S6: Human-Computer Collaborative Annotation and Virtual Repair In response to the user's interactive correction operations such as brush strokes and eraser operations on the segmentation results, the two-dimensional segmentation mask sequence is updated in real time, and the reconstruction process in step S4 is triggered to generate an updated three-dimensional disease mesh model and virtual repair scheme.

[0030] Example 2 Based on Embodiment 1, the present invention also provides an intelligent management and non-destructive analysis system for multimodal data of cultural relics, such as... Figure 2As shown, it includes a multi-source data management module, a spatial registration module, an intelligent computing engine module, and a visualization interaction module; The multi-source data management module is used to acquire CT data from the internal tomographic scans of the artifact to be analyzed and surface data from the external three-dimensional surface scans; based on the metadata of the CT data and surface data, a hierarchical tree-like index structure is constructed, such as... Figure 3 As shown, in the tree-like index structure, each cultural relic corresponds to a parent node, and the CT data and surface data of the same cultural relic are associated as child nodes under the same parent node. The spatial registration module uses a two-level spatial registration strategy to perform coordinate transformation on CT data and surface data, so that the two are aligned in a unified spatial coordinate system. The intelligent computing engine module is used to call a pre-trained deep neural network model based on the material properties of the cultural relic, perform serialized reasoning on the registered CT data, extract the features of the cultural relic body and the diseased area, and obtain a two-dimensional segmentation mask sequence. The intelligent computing engine module is also used to reconstruct the two-dimensional segmentation mask sequence into a three-dimensional disease mesh model using a surface drawing algorithm, and to attach the three-dimensional disease mesh model to the corresponding parent node of the cultural relic in the tree index structure. The visualization interaction module is used to display surface data, CT data, and three-dimensional disease mesh models in the same rendering window based on a dual-channel rendering pipeline; in response to virtual cutting commands, it uses the clipping plane to calculate and map the cut surface texture in real time, showing the correspondence between the internal structure and external form of the cultural relic; The visualization interaction module is also used to respond to the user's interactive correction operation on the segmentation results, update the two-dimensional segmentation mask sequence in real time, and trigger the reconstruction process in the intelligent computing engine module to generate an updated three-dimensional disease mesh model and virtual repair scheme.

[0031] Example 3 Based on intelligent analysis and virtual restoration of bronze artifacts, the following is the complete process of non-destructive exploration, corrosion disease segmentation, and virtual restoration of an unearthed bronze ding (a type of ancient Chinese cooking vessel) using this invention: • Step S1: Building a multi-source database for bronze artifacts Users log in to the system through the "Multi-Source Data Management Module" and create a new cultural relic case node named "Bronze_Tripod_001".

[0032] Import the high-precision X-ray computed tomography (CT) data of the bronze tripod, in DICOM sequence format, with a resolution of 512×512×800; at the same time, import the external three-dimensional surface model (Surface) obtained using a handheld laser scanner, in OBJ file format with texture mapping.

[0033] The system automatically parses the metadata, mounts the CT volume data and surface mesh data to the same parent node, and displays their excavation information and preservation status in the attribute panel.

[0034] • Step S2: Internal and external data spatial registration In the "Spatial Registration Module", the system first renders the iso-surface reconstruction model of the CT data and the imported external OBJ model.

[0035] Coarse registration (S2-1): The user picks corresponding feature point pairs on the CT model and OBJ model at three iconic anatomical locations of the bronze tripod (such as the tip of the left ear, the sole of the right foot, and the center of the animal mask pattern on the abdomen). The system calculates the rigid body transformation matrix and roughly aligns the CT data to the coordinate system of the OBJ model.

[0036] Fine registration (S2-2): Based on coarse registration, the system automatically calls the nearest point iteration algorithm (ICP) to find the nearest point pair between the CT isosurface and the surface of the external OBJ model and performs iterative optimization to automatically complete high-precision spatial alignment. Subsequently, the system switches to four-view mode. For local areas with severe corrosion or defects, the user can manually compensate by fine-tuning the rotation and translation vectors of the CT data with the mouse, observing the overlaid contour lines until the edges of the CT data completely fit the shell of the OBJ model, and locking the transformation matrix.

[0037] • Step S3: Intelligent segmentation of rust defects Users can access the "Intelligent Computing Engine Module" and select "Bronze" in the material options.

[0038] The system automatically loads the pre-built "Bronze-Rust-SegNet" in the background through the inference interface unit.

[0039] When the user clicks the "Start Segmentation" command, the system serializes the CT data and inputs it into the model. The model uses a convolutional neural network to extract the feature differences between the high-density metal matrix and the medium-to-low-density mineralized corrosion layer in the CT image, and outputs a binary mask for each slice, where a pixel value of 1 represents corrosion and 0 represents the background or the matrix.

[0040] • Step S4: 3D Reconstruction and Node Update The system receives the segmented mask sequence and uses surface rendering algorithms (such as Marching Cubes) to reconstruct the "rust layer 3D mesh" and the "bronze body 3D mesh" in real time.

[0041] These two newly generated mesh models are automatically added as new child nodes to the "Bronze_Tripod_001" case tree, with the rust layer set to green (RGB: 0, 1, 0) and the body set to bronze (RGB: 0.8, 0.5, 0.2) by default.

[0042] • Step S5: Integrate Perspective Display In the "Visual Interaction and Virtual Repair Module", the system enables dual-channel rendering.

[0043] User-adjusted transparency properties (Opacity): Set the external OBJ model to 30% transparency, the internal "rust layer 3D mesh" to 50% transparency, and the "bronze body 3D mesh" to 100% opaque.

[0044] Through this integrated display, such as Figure 5 As shown, researchers can visually see the depth of internal corrosion growth and whether it has eroded the inscription area of ​​the bronze artifact through the outer shell.

[0045] • Step S6: Virtual Rust Removal and Repair Assessment Users utilize the "virtual repair tool" provided by the system. In the 3D view, users select to hide the "rust layer 3D mesh" node, retaining only the "bronze body 3D mesh" and the external OBJ model.

[0046] For areas with uneven edges, users can use the "3D Sculpting Brush" to smooth them, simulating the surface appearance after physical rust removal.

[0047] The system calculates the ratio of rust volume to body volume based on the virtual repaired model, generating a quantitative disease assessment report to provide data support for subsequent physical protection schemes (such as whether to use mechanical rust removal or chemical sealing).

[0048] Example 4 The following example, a Qing Dynasty cloisonné enamel double-eared incense burner, demonstrates how this invention utilizes spatial registration and virtual cutting technology to explore the connection process and hidden cracks of the internal metal body without damaging the exquisite glaze of the artifact. The specific process of multimodal fusion display and interactive virtual slicing based on the cloisonné enamel incense burner is as follows: • Step S1: Importing multimodal data and mounting metadata Users create a new case named "Enamel_Incense_Burner_002" in the "Multi-Source Data Management Module".

[0049] Import the industrial CT scan data of the incense burner (reflecting the density and structure of the internal metal body) and the high-precision photogrammetric 3D model (reflecting the external enamel glaze color and decoration, in OBJ format).

[0050] The system parses the metadata, records its scanning resolution as 0.05mm, and attaches the two sets of data as child nodes to the case tree.

[0051] • Step S2: Multimodal data spatial registration Coarse Registration (S2-1): Due to the different orientations during CT scanning and photogrammetry, the initial positions of the two sets of data have significant deviations. In the "Spatial Registration Module," the user selects three pairs of corresponding feature points on both the CT 3D reconstruction model and the external OBJ model: the tips of the ears, the center of the mouth, and the edge of the foot. The system calculates the rigid body transformation matrix, quickly translating and rotating the CT data to the vicinity of the OBJ model.

[0052] Fine Registration (S2-2): Based on coarse registration, the system automatically extracts the highlight edge isosurface features from the CT data and calls the Iterative Closest Point (ICP) algorithm to automatically calculate and minimize the spatial distance between the CT isosurface and the surface of the external OBJ model, outputting a precise rigid body transformation matrix to complete the automatic fine registration. Subsequently, the system switches to the "fused perspective view" to render the superimposed effect of the two in real time and provides a 3D joystick (Gizmo) interactive tool. For local complex defect areas, users can observe the CT highlight edge and OBJ model outline at the rim of the incense burner and perform minor manual translation and rotation compensation on the automatic registration results until the two fit together perfectly.

[0053] • Steps S3 & S4: Data Preprocessing and Rendering Attribute Settings This embodiment focuses primarily on structural fusion; therefore, in the "Intelligent Computing Engine Module," the user selects "Metal Extraction Preset." The system automatically adjusts the window width and window level of the CT data, filtering out low-density background noise (such as foam filler) and retaining only the high-density signals of the metal matrix and enamel glaze layer.

[0054] The system sets the processed CT data to volume rendering mode and uses "hot metal" pseudo-color mapping; it sets the external OBJ model to surface rendering mode to preserve its true color and texture.

[0055] • Step S5: Interactive Virtual Cutting and Process Analysis In the "Visual Interaction Module", the user activates the "Virtual Slicer".

[0056] The system generates a semi-transparent clipping plane in the viewport. The user can drag this plane with the mouse to move it from the left ear of the incense burner towards the right ear.

[0057] Dynamic fusion rendering: The system utilizes the hybrid clipping technology of the VTK graphics pipeline to calculate and update the display status of multimodal data in real time based on the positive and negative half-regions of space defined by the normal vector of the clipping plane. For external OBJ models, the system automatically removes mesh data located on the side to be removed from the clipping plane (such as the area behind the cutting direction), while retaining the solid mesh on the other side to reveal the internal cavity of the artifact. For internal CT data, the system calculates the spatial intersection of the clipping plane and the three-dimensional volume data in real time, and maps the corresponding two-dimensional tomographic density image onto the geometric section formed by this intersection in real time, such as... Figure 6 As shown.

[0058] Non-destructive testing results: By observing the cut surface, the researchers clearly saw the connection method between the incense burner's two handles and the body of the burner. CT slices showed that there was an obvious riveting structure at the connection point rather than a one-piece casting. Furthermore, an internal casting sand hole (blind void) that did not extend to the surface was found at the bottom of the burner body.

[0059] • Step S6: Labeling Measurements and Outputting Results Users can use the "Measurement Tool" to directly click on the cut surface to measure the length of the rivet (approximately 12.5 mm) and the diameter of the internal sand hole.

[0060] When the user clicks "Save View", the system saves the current cutting angle, fusion status and measurement data as a snapshot, and automatically generates a digital inspection report containing an "internal process structure analysis diagram".

[0061] In summary, this invention aims to address the problems of fragmented management of internal X-ray data and external 3D scan data, low efficiency in defect segmentation, and difficulties in multimodal display in existing cultural relic digitization work. This invention achieves unified management of cultural relic CT volume data, 3D surface meshes, and metadata through a tree-like index structure; utilizes a two-level spatial registration strategy to achieve high-precision alignment of internal and external structures; and achieves targeted intelligent segmentation and 3D reconstruction updates for defects in cultural relics of different materials such as bronze and ceramics through a pre-built deep learning inference interface and traditional image processing algorithms. Based on a dual-channel rendering pipeline, it realizes multimodal data fusion perspective, virtual cutting, and human-computer collaborative annotation and restoration within the same coordinate system. This invention provides archaeological researchers with a one-stop solution from data archiving, non-destructive exploration, virtual restoration assessment to output results, significantly improving the efficiency and accuracy of cultural relic research and conservation. It can uniformly manage multi-source heterogeneous data of cultural relics, integrate targeted intelligent segmentation algorithms, and support multimodal fusion and interactive virtual restoration.

[0062] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A method for intelligent management and non-destructive analysis of multimodal data of cultural relics, characterized in that, Includes the following steps: S1: Obtain the CT data of the internal tomographic scan of the cultural relic to be analyzed and the surface data of the external three-dimensional surface scan; construct a hierarchical tree index structure based on the metadata of the CT data and surface data. In the tree index structure, each cultural relic corresponds to a parent node, and the CT data and surface data of the same cultural relic are associated with the same parent node as child nodes. S2: Use a two-level spatial registration strategy to perform coordinate transformation on CT data and surface data so that they are aligned in a unified spatial coordinate system; S3: Based on the material properties of the cultural relic, a pre-trained deep neural network model is called to perform serialized reasoning on the registered CT data, extract the features of the cultural relic itself and the diseased area, and obtain a two-dimensional segmentation mask sequence. S4: Use the surface drawing algorithm to reconstruct the two-dimensional segmentation mask sequence into a three-dimensional disease mesh model, and attach the three-dimensional disease mesh model to the corresponding parent node of the cultural relic in the tree index structure; S5: Based on a dual-channel rendering pipeline, it mixes and displays surface data, CT data, and 3D disease mesh models in the same rendering window; responding to virtual cutting commands, it uses clipping planes to calculate and map cut surface textures in real time, showing the correspondence between the internal structure and external form of cultural relics; S6: In response to the user's interactive correction operation on the segmentation results, the two-dimensional segmentation mask sequence is updated in real time, and the reconstruction process in step S4 is triggered to generate an updated three-dimensional disease mesh model and virtual repair scheme.

2. The intelligent management and non-destructive analysis method for multimodal data of cultural relics as described in claim 1, characterized in that, In step S2, a two-level spatial registration strategy is used to perform coordinate transformation on the CT data and surface data to align them in a unified spatial coordinate system. Specifically: Coarse registration: In response to at least three pairs of corresponding feature points picked by the user on the 3D reconstruction of the CT data and the surface data model, the initial rigid body transformation matrix from CT data to surface data is calculated. Then, based on the initial rigid body transformation matrix, the CT data is transformed to the coordinate system of the surface data to complete the coarse registration. Fine registration: Based on coarse registration, the nearest point iteration algorithm is used to calculate and minimize the spatial distance between CT data and surface data, and output the fine registration rigid body transformation matrix; Then, in response to the user's rotation, scaling or translation operations, the superimposed display effect of CT data and surface data is obtained based on the rigid body transformation matrix of the fine registration, and the spatial position of CT data is manually compensated according to the superimposed display effect until the overlap accuracy of the outline of CT data and the geometric shape of surface data meets the preset requirements, thus completing the fine registration.

3. The intelligent management and non-destructive analysis method for multimodal data of cultural relics as described in claim 1, characterized in that, In step S3, the registered CT data is subjected to sequential inference to extract features of the artifact itself and the diseased area, resulting in a two-dimensional segmentation mask sequence. The specific method is as follows: Model loading: Through standardized inference interface units, the corresponding pre-trained deep neural network model is automatically loaded from the model library according to the material category of the cultural relic; Serialization inference: Extract the three-dimensional CT data into a two-dimensional slice sequence along a preset axis, and convert the two-dimensional slice sequence into a tensor format and input it into the pre-trained deep neural network model; Mask generation: The pre-trained deep neural network model outputs a probability map corresponding to the classification prediction result of the pixel points, wherein the classification is the cultural relic body or a specific disease area; then the probability map is binarized according to a preset threshold to generate a two-dimensional segmentation mask sequence marked with the cultural relic body area and the disease area.

4. The intelligent management and non-destructive analysis method for multimodal data of cultural relics as described in claim 1, characterized in that, In step S5, the method for mixing and displaying surface data, CT data, and the 3D disease mesh model in the same rendering window based on a dual-channel rendering pipeline is as follows: The surface data is rendered using solid rendering mode, and the rendering effect is set to semi-transparent. The CT data is rendered using volume rendering mode, and the rendering effect is set to volume rendering material. Depth testing is used to determine whether the occlusion relationship of surface data, CT data, and 3D disease mesh model in 3D space is displayed correctly. If the result is negative, the model is re-rendered.

5. The intelligent management and non-destructive analysis method for multimodal data of cultural relics as described in claim 1, characterized in that, In step S5, the virtual cutting command includes the three-dimensional position coordinates (x, y, z) of the cutting plane and the normal vector parameter (n). x , n y ,n z ).

6. The intelligent management and non-destructive analysis method for multimodal data of cultural relics as described in claim 1, characterized in that, In step S5, the method for responding to the virtual cutting command by using the cutting plane to calculate and map the cut surface texture in real time, and displaying the correspondence between the internal structure and external form of the cultural relic, is as follows: First, a clipping plane is defined in the 3D rendering space. The spatial geometric intersection of the clipping plane with the external surface data model and the internal 3D disease mesh model is calculated in real time to determine the polygon cutting profile. Then, along the spatial coordinates corresponding to the clipping plane, the corresponding two-dimensional tomographic pixel values ​​are extracted from the CT data, and the two-dimensional tomographic pixel values ​​are mapped onto the polygonal section contour in real time as a section texture. Finally, by removing the rendering data on one side of the clipping plane, while retaining the external surface morphology contour of the un-clipped part, the internal CT density information and the depth distribution of the lesions are displayed based on the section texture.

7. The intelligent management and non-destructive analysis method for multimodal data of cultural relics as described in claim 1, characterized in that, In step S1, the metadata of the CT data and surface data includes the name of the cultural relic, the time of excavation, the location of excavation, the material category, and the scanning equipment parameters.

8. The intelligent management and non-destructive analysis method for multimodal data of cultural relics as described in claim 1, characterized in that, The pre-trained deep neural network model is a U-Net, V-Net, or Mask R-CNN network structure.

9. The intelligent management and non-destructive analysis method for multimodal data of cultural relics as described in claim 1, characterized in that, In step S6, the interactive correction operations include brush painting and eraser removal.

10. A system for intelligent management and non-destructive analysis of multimodal data of cultural relics, characterized in that, It includes a multi-source data management module, a spatial registration module, an intelligent computing engine module, and a visualization interaction module; The multi-source data management module is used to acquire CT data of the internal tomographic scan of the cultural relic to be analyzed and surface data of the external three-dimensional surface scan; a hierarchical tree index structure is constructed based on the metadata of the CT data and surface data. In the tree index structure, each cultural relic corresponds to a parent node, and the CT data and surface data of the same cultural relic are associated as child nodes under the same parent node. The spatial registration module uses a two-level spatial registration strategy to perform coordinate transformation on CT data and surface data, so that the two are aligned in a unified spatial coordinate system. The intelligent computing engine module is used to call a pre-trained deep neural network model based on the material properties of the cultural relic, perform serialized reasoning on the registered CT data, extract the features of the cultural relic body and the diseased area, and obtain a two-dimensional segmentation mask sequence. The intelligent computing engine module is also used to reconstruct the two-dimensional segmentation mask sequence into a three-dimensional disease mesh model using a surface drawing algorithm, and to attach the three-dimensional disease mesh model to the corresponding parent node of the cultural relic in the tree index structure. The visualization interaction module is used to display surface data, CT data, and three-dimensional disease mesh models in the same rendering window based on a dual-channel rendering pipeline; in response to virtual cutting commands, it uses the clipping plane to calculate and map the cut surface texture in real time, showing the correspondence between the internal structure and external form of the cultural relic; The visualization interaction module is also used to respond to the user's interactive correction operation on the segmentation results, update the two-dimensional segmentation mask sequence in real time, and trigger the reconstruction process in the intelligent computing engine module to generate an updated three-dimensional disease mesh model and virtual repair scheme.

Images