An immersive cultural relic fragment splicing and repairing system

Abstract

This invention discloses an immersive system for piecing together and restoring fragmented cultural relics, comprising: an inspection module for piecing together fragments and visualizing the matching results, receiving user modifications to the matching relationships, separating incorrectly matched fragments, and selecting a target fragment group; an exploration module for filtering candidate matching fragments from the unmatched fragments in the target fragment group; and a confirmation module for sensing the matching between candidate matching fragments and the target fragment group through feedback, and, upon confirming a matching candidate fragment, achieving precise matching between the confirmed candidate fragment and the target fragment group and visualizing the precise matching result. This system enables cultural relic restoration experts to complete the task of correcting incorrectly pieced together fragments in a virtual reality space.

Description

Technical Field

[0001] This invention belongs to the field of cultural relic assembly technology, specifically relating to an immersive cultural relic fragment assembly and restoration system. Background Technology

[0002] Existing methods for piecing together fragments of cultural relics include automatic piecing algorithms, immersive manual piecing methods, and human-machine combined piecing methods.

[0003] The purpose of automatic splicing algorithms is to calculate the possible relative positions between fragments based on their geometric features and ultimately assemble them into a whole. Automatic algorithms offer advantages such as high efficiency and avoidance of secondary damage to the fragments. Papaioannou et al. systematically divided the automatic splicing algorithm process into two parts: pairwise matching and multi-part optimization. The pairwise matching process searches for all possible pairwise matching pairs based on the geometric features of the fragments and assigns a score to each pair to indicate its matching quality. Based on the results of pairwise matching, multi-part optimization is responsible for selecting the correct matching pairs and splicing the matched fragment pairs into a complete object. However, due to the low-quality characteristics of real fragments caused by corrosion, loss, etc., and the existence of local optima in the matching optimization, the matching results of automatic algorithms are not always reliable.

[0004] Based on the development of immersive technology in 3D visualization and interaction methods, immersive manual assembly methods allow users to more efficiently attempt to restore cultural relics by manipulating 3D scanned virtual fragments. These methods mainly include two types of technologies: interactive visual feedback prompts and high-precision interaction. Interactive visual feedback technology primarily uses augmented reality visualization techniques, displaying visual symbols (such as arrows, charts, etc.) outside the fragments to enhance the user's perception of the specific spatial relationships between the current fragments (such as distance, fragment orientation, angle, etc.). High-precision interaction technology aims to improve the efficiency of manual fragment alignment and reduce fatigue. These technologies mainly include designing a mapping relationship between user hand movement and fragment movement (e.g., allowing fragments to rotate only along a certain axis) and using automatic fragment alignment algorithms. However, the above immersive manual assembly methods mainly focus on the manual alignment of pairs of fragments and do not consider the problem of simultaneously aligning multiple fragments in the assembly and restoration process. As the number of fragments to be aligned increases, observing and accurately specifying multiple alignment positions becomes a challenge.

[0005] Human-machine collaborative splicing methods combine the efficiency of automated splicing algorithms with the accuracy of human splicing. One strategy involves having experts assess the correctness of each result after the pairwise matching algorithm produces pairwise matches and before the complete artifact is obtained through multi-part optimization, thus eliminating incorrect results in real time. Another strategy involves having experts specify key matching features before the pairwise matching algorithm, thereby improving the splicing success rate of the matching algorithm. In addition, the 3DPuzzling Engine presents the fragment alignment results of pairwise matching in an immersive space and uses visual feedback of matching lines to enhance the user's perception of specific matching positions. Users can use this to evaluate the correctness of fragment matching and manually improve the alignment results. However, the above human-machine collaborative methods do not address the issue of correcting the final result after multi-part optimization. To correct the final splicing result of the automated algorithm, experts need to identify erroneous fragment matches and find the correct result from numerous splicing alignment possibilities. Summary of the Invention

[0006] In view of the above, the purpose of this invention is to provide an immersive artifact fragment assembly and restoration system, so as to enable artifact restoration experts to complete the task of correcting erroneous splicing in artifacts in a virtual reality space.

[0007] To achieve the above-mentioned objectives, an embodiment provides an immersive artifact fragment assembly and restoration system, comprising:

[0008] The inspection module is used to piece together fragments of cultural relics and visualize the matching results. It can receive user modifications to the matching relationships of fragments, separate incorrectly matched fragments, and select the target fragment group.

[0009] The exploration module is used to filter out corresponding candidate matching fragments from the unmatched fragments in the target fragment group;

[0010] The confirmation module is used to realize the matching perception between candidate matching fragments and target fragment groups through feedback. When confirming the matching candidate fragments, it realizes the accurate matching between the confirmed candidate fragments and target fragment groups and visualizes the accurate matching results.

[0011] Preferably, in the inspection module, the process of piecing together the fragments of the cultural relic includes: identifying the triangular facets of each fragment, dividing each fragment into matching facets and intact facets, calculating the matching value between the matching facets of two fragments, filtering and determining the best match between two fragments based on the size of the matching value, and matching and aligning the two fragments corresponding to the best match to form a preliminary matching result.

[0012] Preferably, in the inspection module, the three-dimensional mesh embedded visualization of the matching results is used, including: using a matching relationship diagram embedded in the three-dimensional fragments to represent the matching relationship between the matching surfaces, wherein the nodes in the matching relationship diagram are placed at the geometric center of the fragments to represent a fragment, and the edges represent a pair of matching surfaces, thereby integrating the three-dimensional shape of the fragments with the visualization of the matching relationship diagram to realize the visualization of the matching results;

[0013] In visualization, the matching relationship in each pair of matching faces is represented by a matching relationship line connecting the centers of the two corresponding fragments. The matching value of each match is encoded as the color of the matching face and the relationship line. The darker the color, the larger the matching value. The unmatched relationship of the unmatched matching faces is represented by the unmatched relationship line connecting the center of the matching face and the center of the fragment to which it belongs.

[0014] When visualizing the matching results, edge filtering functions are provided, including: normalizing all matching values ​​to the range of 0 to 1; and filtering out edges of matching faces with matching values ​​below the threshold by setting a threshold on the interactive panel.

[0015] Preferably, the inspection module provides a navigation interaction function, which uses handle gripping and movement gestures to realize the rotation of the entire spliced ​​object. The matching relationship map and the spliced ​​object will move synchronously and proportionally with the movement of the handle. The user's movement in the real space and head conversion are proportionally mapped to the viewpoint movement in the virtual space.

[0016] It provides a matching modification function and uses touch gestures to support users to specify incorrect matches. Users control the corresponding virtual hand in the virtual space using the gamepad. When the user's virtual hand touches the matching relationship line, the touched matching relationship is deleted. The matching relationship line is replaced by two broken lines. The broken lines represent the unmatched matching relationships of the unmatched matching surfaces after the matching relationship is deleted. The remaining mutually matched fragments form a fragment group.

[0017] It provides a target selection function, which uses touch gestures to support users in selecting target fragment groups. When the user's virtual hand is inside a three-dimensional fragment and the touch button is used, the fragment group corresponding to this fragment is selected as the target fragment group.

[0018] Preferably, in the exploration module, initial candidate matching fragments are generated by arranging and combining fragments in the target fragment group with unmatched fragments, and the initial candidate matching fragments are aligned and spliced ​​with the target fragment group. During the alignment and splicing process, unreliable initial candidate matching fragments are filtered out according to the following two rules:

[0019] Rule 1: When aligning matching faces in each splice combination, the alignment results within each splice combination must not be disrupted; Rule 2: Fragments must not be connected after alignment;

[0020] The remaining initial candidate matching fragments are used as candidate matching fragments, and the average matching value of each pair of matching faces in each pair of splicing combinations is calculated as the matching value of the candidate matching fragment.

[0021] Preferably, the exploration module provides a candidate matching layout function, which arranges candidate matching fragments in a regular manner based on the similarity of candidate skeleton shapes, including:

[0022] First, the distance between the skeleton nodes of each pair of candidate matching fragments is calculated as similarity. Then, based on the similarity of each pair of candidate matching fragments, the point distribution of the candidate matching fragments on the two-dimensional plane is obtained. Next, the two-dimensional plane is placed vertically in front of the user in the immersive space, and the position of each point on the two-dimensional plane is used as the position of the corresponding candidate matching fragment in the space. After the layout of the candidate positions is completed, two interaction methods, transformation and selection, are provided to facilitate the search for candidate matching fragments.

[0023] Preferably, the exploration module provides a candidate matching clustering function, employing clustering methods to provide users with multi-level search filtering, including:

[0024] Each candidate matching fragment is treated as a point on a two-dimensional plane, and the candidate matching fragments are clustered. The average node value of the nodes in the skeleton of each candidate matching fragment is calculated to generate an average skeleton, which is used as a visual summary of the clustering results to replace the original candidate matching fragments. The average skeleton is placed at the center of the clustering results. The matching value of the average skeleton is the mean of the matching values ​​of each candidate matching fragment, and the color of the average skeleton is encoded in the same way as each candidate matching fragment. When a user selects an average skeleton, all candidate matching fragments within this clustering result will be displayed in their original positions. Each displayed candidate matching fragment includes its three-dimensional shape and the visualization results of the skeleton relationship.

[0025] Preferably, the confirmation module provides a manual alignment function, which assists in manual alignment through feedback, including:

[0026] When the virtual hand is inside the fragment shape and grips the controller, the fragment's movement is synchronized with the controller's movement, indicating a manual alignment state. In manual alignment, real-time visual feedback is provided to show the alignment of the mating surfaces. Specifically, it detects mating surfaces that are getting closer in real-time. If the distance between a pair of mating surfaces is less than a distance threshold and the angle between their normal vectors is greater than an angle threshold, a dashed line is used to connect the mating surfaces to show that they are being aligned. When the user releases the controller, the mating surfaces shown with the dashed line are considered to have been aligned by the user. When the user grips the fragment again to align other mating surfaces, if previously confirmed mating surfaces are dragged away or their facing direction changes, resistance feedback is provided between these mating surfaces. In this case, the degree of movement change of the fragment is the controller's change degree multiplied by a decreasing factor, requiring the user to move or rotate a larger amount to separate the previously aligned mating surfaces.

[0027] Preferably, the confirmation module provides a precise automatic matching function, including determining the spatial position of the fragment group where the matching surface is located based on the point cloud of the matching surface to accurately and quickly align the matching surface. When the user completes the confirmation of all alignment relationships, the query, exploration and confirmation of this error matching are completed.

[0028] Compared with existing inventions, the beneficial effects of the present invention are as follows:

[0029] (1) Recommended Artifact Assembly for Restoration: Compared to simply having restoration experts search for the correct match from a large number of assembly combinations and try to align fragments through frequent trial and error, this invention adopts a human-machine combined approach. It utilizes an assembly alignment algorithm to traverse all possible assembly combinations and presents them according to certain rules, thereby providing experts with recommendations for correct matches and quickly selecting the correct matching scheme. The automatically generated assembly results allow experts to eliminate incorrect results before careful alignment, and the rough alignment scheme presented in the results also provides experts with preliminary alignment guidance.

[0030] (2) Visualizing Relationships to Aid Piecing Together: This invention uses graph data to construct the matching relationships between fragments and displays them synchronously with the fragment model using embedded 3D network visualization. The 3D network visualization intuitively presents the relationships between matching surfaces of fragments (e.g., which matching surfaces align fragments with each other), and effectively indicates key information such as which matching surfaces are not matched and the degree of alignment between matched surfaces. This provides important clues for experts to understand the quality of artifact piecing together and quickly locate problematic matches from numerous fragments.

[0031] (3) Improving the efficiency of searching for matching solutions through multi-level selection: This invention improves the efficiency of expert searching for matching solutions by using a two-level screening mechanism based on multiple levels of detail. In the first level of screening, the cluster visualization of different matching types can help experts eliminate a large number of invalid matches, while the second level of screening allows experts to accurately select appropriate results by showing the details of the matching results.

[0032] (4) Reduce the difficulty of alignment interaction through real-time feedback: Based on the traditional immersive direct interaction, this invention enhances the user's perception of the operation effect by visualizing the correspondence of the current matching surface in real time, and maps the relationship to the movement of the operated fragments, thus limiting the user's incorrect alignment operation and improving the user's efficiency in aligning fragments. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of the structure of an immersive artifact fragment assembly and restoration system provided in the embodiment;

[0035] Figure 2 This is a schematic diagram of the splicing and restoration process of the Gargoyle 3D sculpture provided in the embodiment. In this process, (a) the automatic algorithm first splices the fragments and obtains a preliminary result. The system then visualizes the result, including the 3D shape of the fragments and the embedded node link diagram representing the skeleton of the link relationship; (b1, b2) the user uses touch gestures to delete links with incorrect matching relationships; (c) based on the user's modifications, possible matching candidates are displayed through clustering and average skeleton visualization; (d) when the user selects a cluster, the specific shape of the candidate in this category and its relationship skeleton are displayed; (e) the user manually confirms and aligns the matching surfaces; (f) after the manual alignment operation is completed, the system automatically further refines the alignment of the confirmed matching surfaces.

[0036] Figure 3 The visualization summary for the candidates is generated and laid out. (a) The clustering algorithm generates clustering results based on the position of each candidate on the plane; (b) The average skeleton result is used as the visualization summary for each cluster, while preserving the shape of the target group; (c) The visualization summary of each cluster is placed at the center of each cluster for display, while the visualization result at the center is magnified for easy observation and selection.

[0037] Figure 4There are two types of timely feedback. When the user manually aligns the pieces, (a) visual feedback uses a dashed line link to show the matching surfaces in the aligned state; (b) force feedback reduces the range of movement of the pieces when the user attempts to disrupt the aligned matching surfaces, making the user feel that a force is being applied to the pieces in the opposite direction along the dashed line link to guide the user's movement of the pieces. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0039] Because immersive spaces provide intuitive 3D perception and interactive operation for digitized virtual artifact fragments, this embodiment aims to allow artifact restoration experts to correct erroneous joins within a virtual reality space. To effectively repair these joins, experts first observe the overall appearance of the artifact, the details between each fragment, and analyze the correctness of the joins, identifying incorrect fragment matches. Based on this, experts then explore other matching methods for the incorrectly matched fragments and find those that correctly align the fragments. Subsequently, experts manually align all correctly matched fragments by adjusting their relative spatial positions. Finally, the system completes the join of all fragments based on the expert's alignment, forming a new artifact. Furthermore, experts need to repeat the above steps until no new join errors remain in the artifact.

[0040] However, existing immersive environments for artifact restoration present two main challenges. First, finding the optimal restoration solution is extremely time-consuming and labor-intensive. Restoration experts need to repeatedly try different splicing methods between fragments to find the correct match. This process requires observing and comparing the shapes and alignments of various fragment pairs, and judging the correctness of the match based on features such as complex surface textures. Second, correcting the alignment of multiple fragments in an immersive environment is very difficult. Although immersive interactive technology allows users to naturally change the spatial position of fragments and indicate the aligned parts directly through hand gestures, restoration experts need to observe multiple factors while interacting to determine the degree and correctness of alignment. These factors include the overall appearance as well as information on local misaligned surfaces and the degree of stitching between aligned surfaces. Therefore, manual alignment correction becomes increasingly difficult as the number of aligned fragments and the complexity of their shapes increase.

[0041] To enable cultural relic restoration experts to correct errors in relic assembly within an immersive environment, the embodiment provides an immersive cultural relic fragment assembly and restoration system that outlines a restoration process and clarifies the tasks to be addressed in each step. Specifically, the proposed restoration process consists of three modules as follows:

[0042] Inspection Module: Experts need to identify stitching errors in the objects generated by the automatic stitching algorithm. Error matching is determined based on multiple features, including overall appearance, discontinuous local surface texture, and the matching value calculated by the matching algorithm. After an error match is identified, experts need to separate the corresponding fragments, leaving them in an unmatched state, so that the correct matching fragments can be found and aligned in subsequent steps.

[0043] Exploration Module: Experts need to explore possible correspondences and alignments between unmatched fragments and identify potentially correct matches. These alignment candidates are obtained by traversing all the faces to be matched on the unmatched fragments and aligning them. However, since the number of candidates increases exponentially with the number of faces to be matched, the goal of this stage is to enable experts to efficiently eliminate impossible matches from the numerous results and quickly identify a subset of potentially correct candidates.

[0044] Confirmation Module: After selecting a set of candidates, experts need to carefully observe them, select one with a correct match, and then accurately align the fragments. Once the alignment is complete, the next round of the repair process can begin to check and repair the remaining incorrect matches. However, considering the complexity of aligning multiple fragments simultaneously, this stage aims to provide experts with interactive alignment assistance to reduce operational difficulty and improve alignment accuracy.

[0045] The following section provides a detailed description of the function of each module.

[0046] During the inspection phase, the inspection module needs to complete the following: (I1) Visualize the 3D shape and matching relationships of the fragments: The shape of the fragments, the texture of their surfaces, and the correspondence between fragments (e.g., which faces the fragments are aligned with and the degree of alignment) are crucial for experts to judge the correctness of the object assembly and identify incorrect matches. Therefore, the visualization of matching information needs to be integrated into the 3D fragments to facilitate the identification of incorrect matches; (I2) Provide intuitive interaction for modifying fragment matching relationships: Modifying fragment matching relationships aims to allow experts to identify and remove incorrect matching relationships between fragments. Therefore, the system should provide experts with intuitive and convenient interaction for modifying matching relationships.

[0047] Specifically, the inspection module comprises two parts: automatic fragment stitching and visualization of matching results, as well as error matching identification. For automatic fragment stitching, the fragment data in this embodiment is a 3D point cloud in Wavefront.obj format, containing basic geometric information of the fragment surfaces. This embodiment employs the following computer graphics methods to sequentially generate the fragment stitching results: The screen poison reconstruction method is used to identify the triangular facets of each fragment; matching facet recognition and edge recognition algorithms are used to divide the triangular facets of each fragment into matching faces and intact faces; a pairwise matching algorithm is used to calculate the matching value between the matching faces of two fragments; a multi-part reconstruction algorithm is used to filter and determine the best match between two fragments based on the matching value; and OpenGR is used to match, align, and stitch the two fragments corresponding to the best match to form a preliminary matching result.

[0048] To visualize the matching results and identify incorrect matches, a 3D network embedded visualization is first used to display the fragments and their matching relationships. Then, a gesture interaction is designed to provide intuitive relationship modification.

[0049] In this embodiment, a 3D mesh embedding is used to visualize the matching results. A matching relationship graph embedded in the 3D fragments represents the matching relationships between matching faces, where nodes are placed at the geometric center of the fragments to represent a fragment, and edges represent a pair of matching faces. Therefore, this invention integrates the 3D shape of the fragments with the visualization of the matching relationship graph, enhancing the intuitiveness and efficiency of users observing the matching results by using the fragment skeleton metaphorical visualization method of the matching relationship graph.

[0050] When visualizing, such as Figure 2 As shown in Figure a, the matching relationship in each pair of matching faces is represented by a matching relationship line connecting the centers of the two corresponding fragments. The matching value of each match is encoded by the color of the matching face and the relationship line. The darker the color, the larger the matching value. In particular, the unmatched relationship of an unmatched face is represented by an unmatched relationship line connecting the center of the matching face and the center of the fragment to which it belongs.

[0051] To reduce visual clutter caused by complex matching relationships, this invention provides an edge filtering function, which includes: normalizing all matching values ​​to the range of 0 to 1; and filtering out edges of matching faces with matching values ​​below the threshold by receiving a threshold set on the interactive panel.

[0052] This embodiment is based on the HTC Vive Pro device and provides three interaction methods to give users an intuitive way to observe and modify the matching results.

[0053] For navigation interaction, this embodiment uses handle gripping and movement gestures to achieve the overall rotation of the assembled object. The relationship diagram and the object move synchronously and proportionally with the movement of the handle. Furthermore, this example maps the user's movement and head rotation in real space proportionally to the viewpoint movement in virtual space.

[0054] For match modification, this embodiment uses touch gestures to support user-specified incorrect matches. The user controls the corresponding virtual hand in the virtual space using a gamepad. When the user's virtual hand touches a matching line representing a matching relationship, the corresponding matching relationship is deleted. This matching line is replaced by two broken lines, such as... Figure 1 As shown in b1 and b2, these broken lines represent the unmatched relationships of the unmatched surfaces after the matching relationships are deleted. At this point, the remaining mutually matching fragments form fragment groups.

[0055] For selecting target fragment groups, this embodiment uses touch gestures to support user selection. When the user's virtual hand is inside a 3D fragment and touches the button, the corresponding fragment group is selected, and the system enters the exploration module of the workflow.

[0056] The exploration module needs to complete the following tasks during the exploration phase: (E1) Presenting traversal results based on matching features: To enable experts to quickly identify potentially correct matches, these candidate matches should be organized in a certain order. Since experts judge the correctness of candidates based on matching features, the system should arrange and present them according to the similarity of matching features between candidates. (E2) Revealing the matching patterns of candidates: In order to eliminate incorrect candidates from a large number of traversal results, experts need to understand the general matching types, such as which positions the fragments can be matched to and what shape the object will roughly take after each position is matched. To this end, the system should cluster the candidates based on matching features and provide a visual summary of each cluster to help experts quickly filter the correct matching types.

[0057] Specifically, based on the incorrect matching in the previous stage, the exploration module includes three parts: candidate matching generation, candidate matching layout, and candidate matching clustering visualization, aiming to help users find the correct matching method.

[0058] For candidate matching generation, based on the fragment group G = {G1, G2, ..., G...} obtained in the previous round... n Candidate matching fragments are generated by arranging fragments in the target fragment group with unmatched faces in other groups: Among them G i G represents the target fragment group. ′ The set of the remaining groups; and n G′ For G iThe number of unmatched faces in G′, and

[0059] After obtaining the initial candidate matching fragments, OpenGR is used to align and stitch the initial candidate matching fragments with the target fragment group. During the alignment and stitching process, unreliable initial candidate matching fragments are filtered out according to the following two rules: Rule 1: When aligning the matching faces in each stitching combination, the alignment results within each stitching combination must not be destroyed; Rule 2: Fragments cannot be connected after alignment; The remaining initial candidate matching fragments are used as candidate matching fragments, and the average matching value of each pair of matching faces in each stitching combination is calculated as the matching value of the candidate matching fragment.

[0060] For candidate matching layout, to accelerate the expert's search for correct candidates, candidates are systematically arranged based on the similarity of their skeleton shapes, and a dimensionality reduction algorithm is used to ensure that candidates at adjacent positions are morphologically similar. Specifically, this includes:

[0061] First, the Chamfer distance between the skeleton nodes of each pair of candidate matching fragments is calculated as similarity. Then, based on the similarity of each pair of candidate matching fragments, the t-SNE algorithm is used to obtain the point distribution of the candidate matching fragments on a two-dimensional plane. Next, in the immersive space, the two-dimensional plane is placed vertically in front of the user, and the position of each point on the two-dimensional plane is used as the position of the corresponding candidate matching fragment in the space. After the candidate positions are laid out, two interactive methods, transformation and selection, are provided to facilitate the search for candidate matching fragments. For transforming candidate matching fragments, this embodiment uses an air-grabbing gesture. When the user grasps the handle in the air and moves or rotates it, all candidates will move and rotate synchronously in the laid-out plane. For selecting candidate matching fragments, this embodiment magnifies the candidate near the center of the plane; when the user presses the handle, the magnified candidate is considered selected and enters the confirmation module.

[0062] For candidate matching clustering, this invention utilizes clustering methods to provide users with multi-level search filtering, thereby avoiding the difficulty of directly finding the correct candidate from too many candidates. For example... Figure 3 As shown, it specifically includes:

[0063] Each candidate matching fragment is treated as a point on a two-dimensional plane, and the DBSCAN algorithm is used to cluster them. The average node value in the skeleton of each candidate matching fragment is calculated to generate an average skeleton, which is then used as a visual summary of the clustering results, replacing the original candidate matching fragments. The average skeleton is placed at the center of its respective clustering result. The matching value of the average skeleton is the average of the matching values ​​of all candidate matching fragments, and its color is encoded in the same way as each candidate matching fragment. When a user selects an average skeleton, all candidate matching fragments within that clustering result will be displayed in their original positions. Figure 2 In sections c and d, each displayed candidate matching fragment includes its 3D shape and a visualization of its skeletal relationship. Once the user selects a candidate matching fragment, the system proceeds to the next confirmation module.

[0064] The confirmation module needs to complete the following tasks during the confirmation phase: (C1) Improve the perception of matching surface relationships using real-time feedback technology: The difficulty in aligning multiple fragments stems primarily from the fact that the correspondence between matching surfaces changes in real time as the spatial positions of the fragments change, thus posing a significant challenge to both real-time observation and the precision of manual operation. For example, experts need to check in real time which matching surfaces are currently facing each other, and in which direction and angle the fragments should be moved to simultaneously align multiple fragment surfaces. Providing real-time feedback on the matching surface relationships to experts can guide the alignment operation and improve the efficiency of the interaction. (C2) Generate accurate alignment results based on the alignment interaction: Considering the imprecision of manual operation, the system should promptly perform further precise alignment of manually aligned fragments to improve the overall accuracy of the splicing.

[0065] Specifically, the confirmation module includes two parts: manual alignment and automatic stitching. For manual alignment, real-time feedback is used to assist in manual alignment, allowing users to intuitively complete the manual alignment of fragments through grasping gestures. When the virtual hand is inside the fragment shape and grips the controller, the movement of the fragment will be synchronized with the movement of the controller, which is the manual alignment state. In the manual alignment state, real-time visual feedback is provided to show the matching surface relationship being aligned, that is, real-time detection of matching surfaces that are close to each other. If the distance between a pair of matching surfaces is less than a distance threshold and the angle between the normal vectors of the surfaces is greater than an angle threshold, then the matching surfaces are connected by a dashed line to show the matching surfaces being aligned, such as... Figure 4 As shown in Figure a; when the user releases the controller, the mating surfaces displayed by the dashed lines are considered to be in alignment as confirmed by the user. When the user grasps the piece again to align other mating surfaces, if the previously confirmed mating surfaces are dragged away or their facing direction changes, resistance feedback is provided between these mating surfaces. At this point, the degree of movement change of the piece is the controller's change degree multiplied by a decreasing factor, requiring the user to move or rotate a larger amplitude to separate the previously aligned mating surfaces. This reduced change degree mapping reduces the impact of misoperation (e.g., matching a new surface but disrupting an already aligned match), while simultaneously constructing the feeling of the piece being dragged by the dashed lines, such as... Figure 4 As shown in b.

[0066] For precise automatic stitching: Once the alignment relationship is confirmed, the matched surfaces that have been confirmed to be aligned are automatically and precisely aligned to compensate for the inaccuracies of manual operation. Specifically, 1) This embodiment uses global registration technology to determine the spatial position of the fragment group where the matched surface is located based on the point cloud of the matched surface, so as to accurately and quickly align the matched surfaces. After the user completes the confirmation of all alignment relationships, the system re-enters the detection module to allow the user to review the remaining mismatches.

[0067] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An immersive system for piecing together and restoring fragmented cultural relics, characterized in that, include: The inspection module is used to piece together fragments of cultural relics and visualize the matching results. It can receive user modifications to the matching relationships of fragments, separate incorrectly matched fragments, and select the target fragment group. Specifically, it includes: The process of piecing together fragments of cultural relics includes: identifying the triangular facets of each fragment, dividing each fragment into matching facets and intact facets, calculating the matching value between matching facets of two fragments, filtering and determining the best match between two fragments based on the size of the matching value, and matching and aligning the two fragments corresponding to the best match to form a preliminary matching result. The matching results are visualized using an embedded 3D mesh, including: a matching relationship graph embedded in 3D fragments to represent the matching relationships between matching faces. Nodes in the matching relationship graph are placed at the geometric center of the fragment to represent a fragment, while edges represent a pair of matching faces. This integrates the 3D shape of the fragments with the visualization of the matching relationship graph, achieving visualization of the matching results. During visualization, the matching relationship in each pair of matching faces is represented by a straight line connecting the centers of the two corresponding fragments. The matching value of each match is encoded as the color of the matching face and the line; the darker the color, the larger the matching value. The unmatched relationship of an unmatched face is represented by a straight line connecting the center of the matching face to the center of its corresponding fragment. When visualizing the matching results, edge filtering is provided, including: normalizing all matching values ​​to the range of 0 to 1; and filtering out edges of matching faces with matching values ​​below a threshold set on the interactive panel. The inspection module provides navigation interaction functions, which use handle grip and movement gestures to realize the rotation of the spliced ​​object as a whole. The matching relationship map and the spliced ​​object will move synchronously and proportionally with the movement of the handle. The user's movement and head conversion in the real space are proportionally mapped to the viewpoint movement in the virtual space. It provides a matching modification function and uses touch gestures to support users to specify incorrect matches. Users control the corresponding virtual hand in the virtual space using the gamepad. When the user's virtual hand touches the matching relationship line, the touched matching relationship is deleted. The matching relationship line is replaced by two broken lines. The broken lines represent the unmatched matching relationships of the unmatched matching surfaces after the matching relationship is deleted. The remaining mutually matched fragments form a fragment group. It provides a target selection function, which uses touch gestures to support users in selecting target fragment groups. When the user's virtual hand is inside a 3D fragment and the touch button is used, the fragment group corresponding to this fragment is selected as the target fragment group; the exploration module is used to filter corresponding candidate matching fragments among the fragments that do not match the target fragment group; The confirmation module is used to realize the matching perception between candidate matching fragments and target fragment groups through feedback. When confirming the matching candidate fragments, it realizes the accurate matching between the confirmed candidate fragments and target fragment groups and visualizes the accurate matching results.

2. The immersive cultural relic fragment assembly and restoration system according to claim 1, characterized in that, In the exploration module, initial candidate matching fragments are generated by arranging and combining fragments in the target fragment group with unmatched fragments. These initial candidate matching fragments are then aligned and spliced ​​with the target fragment group. During the alignment and splicing process, unreliable initial candidate matching fragments are filtered out using the following two rules: Rule 1: When aligning matching faces in each splice combination, the alignment results within each splice combination must not be disrupted; Rule 2: Fragments must not be connected after alignment; The remaining initial candidate matching fragments are used as candidate matching fragments, and the average matching value of each pair of matching faces in each pair of splicing combinations is calculated as the matching value of the candidate matching fragment.

3. The immersive cultural relic fragment assembly and restoration system according to claim 1, characterized in that, The exploration module provides a candidate matching layout function, which arranges candidate matching fragments in a regular manner based on the similarity of candidate skeleton shapes, including: First, the distance between the skeleton nodes of each pair of candidate matching fragments is calculated as similarity. Then, based on the similarity of each pair of candidate matching fragments, the point distribution of the candidate matching fragments on the two-dimensional plane is obtained. Next, the two-dimensional plane is placed vertically in front of the user in the immersive space, and the position of each point on the two-dimensional plane is used as the position of the corresponding candidate matching fragment in the space. After the layout of the candidate positions is completed, two interaction methods, transformation and selection, are provided to facilitate the search for candidate matching fragments.

4. The immersive cultural relic fragment assembly and restoration system according to claim 1, characterized in that, The exploration module provides a candidate matching clustering function, which uses clustering methods to provide users with multi-level search filtering, including: Each candidate matching fragment is treated as a point on a two-dimensional plane, and the candidate matching fragments are clustered. The average node value of the nodes in the skeleton of each candidate matching fragment is calculated to generate an average skeleton, which is used as a visual summary of the clustering results to replace the original candidate matching fragments. The average skeleton is placed at the center of the clustering results. The matching value of the average skeleton is the mean of the matching values ​​of each candidate matching fragment, and the color of the average skeleton is encoded in the same way as each candidate matching fragment. When a user selects an average skeleton, all candidate matching fragments within this clustering result will be displayed in their original positions. Each displayed candidate matching fragment includes its three-dimensional shape and the visualization results of the skeleton relationship.

5. The immersive cultural relic fragment assembly and restoration system according to claim 1, characterized in that, The confirmation module provides a manual alignment function, which assists in manual alignment through feedback, including: When the virtual hand is inside the fragment shape and grips the controller, the fragment's movement is synchronized with the controller's movement, indicating a manual alignment state. In manual alignment, real-time visual feedback is provided to show the alignment of the mating surfaces. Specifically, it detects mating surfaces that are getting closer in real-time. If the distance between a pair of mating surfaces is less than a distance threshold and the angle between their normal vectors is greater than an angle threshold, a dashed line is used to connect the mating surfaces to show that they are being aligned. When the user releases the controller, the mating surfaces shown with the dashed line are considered to have been aligned by the user. When the user grips the fragment again to align other mating surfaces, if previously confirmed mating surfaces are dragged away or their facing direction changes, resistance feedback is provided between these mating surfaces. In this case, the degree of movement change of the fragment is the controller's change degree multiplied by a decreasing factor, requiring the user to move or rotate a larger amount to separate the previously aligned mating surfaces.

6. The immersive cultural relic fragment assembly and restoration system according to claim 1, characterized in that, The confirmation module provides a precise automatic matching function, which includes determining the spatial position of the fragment group where the matching surface is located based on the point cloud of the matching surface to accurately and quickly align the matching surface. Once the user has completed the confirmation of all alignment relationships, the query, exploration and confirmation of this error matching are completed.

Images