A three-dimensional space knowledge question and answer system and method for cultural relics with visual field perception

Abstract

The application relates to a kind of three-dimensional space knowledge question and answer systems and methods of cultural relic visual field perception, comprising: the semantic segmentation of component level, subject matter level and feature level is carried out to cultural relic three-dimensional model respectively, and spatial region descriptor is constructed for each semantic segmentation area of various semantic segmentation results;Descriptor records the attribute information of time and space;Based on descriptor, the spatial topology relationship and time topology relationship between any two semantic segmentation areas are calculated, and the spatial topology relationship graph and time topology relationship graph are stored in the knowledge graph storage module in the data format of knowledge graph;Receive the question and answer instruction input by user;Temporal and spatial intent analysis is carried out to question and answer instruction, the candidate area of question and answer instruction is dynamically filtered according to the current visual field of user, the answer is generated based on the content stored in knowledge graph storage module and is returned to user through user interaction module;Cultural relic space interaction intelligent question and answer are realized by combining three-dimensional semantic segmentation, space-time topology modeling and visual field perception constraint filtering.

Description

Technical Field

[0001] This invention relates to the fields of digital preservation of cultural heritage, computer vision and natural language processing, and in particular to a three-dimensional spatial knowledge question-and-answer system and method for cultural relics based on field perception. Background Technology

[0002] With the popularization of 3D laser scanning and photogrammetry technologies, important cultural sites such as the Dunhuang Mogao Grottoes, Kizil Grottoes, and Longmen Grottoes have completed high-precision 3D reconstructions, forming massive amounts of 3D point cloud data and mesh models. Simultaneously, cultural heritage institutions are continuously building knowledge graphs to systematically store structured knowledge such as the historical background, image themes, and craftsmanship characteristics of cultural relics. However, the 3D models and knowledge graphs have long been disconnected, and existing technologies suffer from the following systemic defects: Defect 1: The 3D interactive system lacks deep semantic understanding capabilities. Existing 3D cultural relic display systems (such as Sketchfab and Matterport) only support simple attribute labeling (clicking to display "This is a Buddha statue"), lacking deep semantic understanding capabilities and unable to answer knowledge-based questions such as "What does this gesture mean?", "In which period was this pattern most popular?", and "Who is the Bodhisattva to the left of the main statue?". There is an insurmountable modal gap between users' 3D browsing and knowledge acquisition.

[0003] Defect 2: Knowledge graph question answering systems lack spatial anchoring capabilities. Existing cultural heritage knowledge graph question-answering systems (such as those based on DBpedia and Wikidata) are entirely text-based, and answers cannot be correlated with specific spatial locations on 3D models. When a user asks for "the pattern on the east wall," the system cannot use 3D spatial information to constrain the scope of the answer, easily returning all relevant content for the entire artifact with low accuracy, preventing users from obtaining knowledge in a "what you see is what you get" manner.

[0004] Defect 3: The existing system cannot handle spatial relationship problems. In cultural heritage research, spatial relationships are one of the core research dimensions (such as the spatial layout of sutra illustrations, the positional relationships of donors, and the spatial combination patterns of statue groups). Existing retrieval systems based on two-dimensional images can only handle the identification problem of "what is this?", but cannot answer spatial relationship questions such as "what is directly above this pattern?", "what is the adjacent subject on the left?", and "the spatial location of this area in the entire cave".

[0005] Defect 4: The question-and-answer results lack visual perception constraints and have low relevance. Existing question-answering systems do not consider the user's current perspective when returning answers. When the user is focused on a specific area, the system may still return irrelevant content outside the user's field of vision, resulting in low answer relevance. Experimental data from this invention shows that without field-of-view constraints, the answer relevance is only 61%; after introducing field-of-view perception constraint filtering, the answer relevance increases to 92%, an improvement of 37.2 percentage points.

[0006] Defect 5: Spatial relationships only describe the spatial dimension and lack the ability to model the temporal dimension. Existing 3D cultural relic display systems rely solely on spatial topological relationships (adjacent, above, containment, etc.) to describe spatial location in the current state. However, cultural heritage exhibits distinct temporal layering characteristics: the same wall may have undergone multiple renovations (three layers overlapping in the early Tang, high Tang, and late Tang periods), and the statues in the same niche may have been added to or moved in different historical periods. Spatial relationships alone cannot express temporal information such as "this mural was painted before that mural" or "the lower layer of murals was covered by the upper layer," thus limiting the system's ability to support core issues in cultural heritage research. Summary of the Invention

[0007] This invention addresses the technical problems existing in the prior art by providing a vision-aware three-dimensional spatial knowledge question-answering method for cultural relics. It combines three-dimensional semantic segmentation, spatiotemporal topology modeling, vision-aware constraint filtering, and knowledge graph question answering to achieve intelligent interactive question answering for cultural relics in space. It is suitable for interactive knowledge services of "what you see is what you ask" in cultural heritage scenarios such as grottoes and museum artifacts.

[0008] According to a first aspect of the present invention, a vision-aware three-dimensional spatial knowledge question-answering system for cultural relics is provided, comprising: a three-dimensional semantic segmentation and spatial index construction module, a spatiotemporal topological relationship graph construction module, a knowledge graph storage module, a vision-aware constrained question-answering engine, and a user interaction module; The 3D semantic segmentation and spatial indexing construction module is used to perform component-level, subject-level, and feature-level semantic segmentation on the 3D model of cultural relics, and to construct spatial region descriptors for each semantic segmentation region of various semantic segmentation results; the descriptors record time and space attribute information. The spatiotemporal topology graph construction module is used to calculate the spatial and temporal topology relationships between any two semantic segmentation regions based on the descriptor, construct spatial and temporal topology graphs respectively, and store the spatial and temporal topology graphs in the knowledge graph storage module in the data format of a knowledge graph. The user interaction module is used to receive question and answer instructions input by the user; The vision-aware constraint question-answering engine performs spatiotemporal intent parsing on the question-answering command, dynamically filters the candidate region of the question-answering command based on the user's current vision, generates an answer based on the content stored in the knowledge graph storage module, and returns it to the user through the user interaction module.

[0009] According to a second aspect of the present invention, a visual perception-based three-dimensional spatial knowledge question-and-answer method for cultural relics is provided, comprising: Step 1: Perform semantic segmentation on the 3D model of the cultural relic at the component level, subject level, and feature level, and construct spatial region descriptors for each semantic segmentation region of each semantic segmentation result; the descriptors record temporal and spatial attribute information. Step 2 is used to calculate the spatial and temporal topological relationships between any two semantic segmentation regions based on the descriptor, construct spatial and temporal topological relationship graphs respectively, and store the spatial and temporal topological relationship graphs in the knowledge graph storage module in the data format of a knowledge graph. Step 3: Receive the user's question and answer instructions; Step 4: Perform spatiotemporal intent parsing on the question-and-answer instruction, dynamically filter the candidate region of the question-and-answer instruction based on the user's current field of view, generate an answer based on the content stored in the knowledge graph storage module, and return it to the user through the user interaction module.

[0010] This invention provides a vision-aware 3D spatial knowledge question-answering system and method for cultural relics, with the following beneficial effects: Firstly, it achieves a "what you see is what you ask" cultural heritage interaction paradigm: when a user clicks on any location on the 3D model, the system can identify the corresponding cultural relic element and answer related questions, improving interaction efficiency by over 40% (compared to the traditional "visual recognition → system switching → manual retrieval" process). Secondly, vision-aware constraint filtering significantly improves answer relevance: vision-aware constraint filtering increases answer relevance from 61% to 92% (an increase of 37.2 percentage points), reduces the average number of returned candidate regions from 23.4 to 6.1, and increases user satisfaction scores from 2.8 to 4.6 (based on scores from 10 cultural heritage experts, on a 1-5 scale). Thirdly, three-layer semantic segmentation supports multi-scale knowledge services: the component-level / theme-level / feature-level three-layer segmentation framework meets different levels of query needs from coarse-grained to fine-grained, with segmentation accuracies mIoU reaching 0.91, 0.87, and 0.83 respectively, covering the full-scale knowledge needs of cultural heritage research. The spatiotemporal topology graph enables joint spatial and temporal reasoning: Based on five spatial topological relationships, three new temporal topological relationships (earlier / contemporaneous / overlapping) are added, constructing a spatiotemporal topology graph with eight relationship types. This graph is integrated with the CIDOC-CRM knowledge graph in the form of RDF triples, providing a structured expression of the spatiotemporal joint dimension for cultural heritage knowledge for the first time. The accuracy rate for spatial relationship resolution reaches 87.5%, and the accuracy rate for temporal relationship resolution reaches 84.2%. The system can answer temporal-dimensional questions such as "Is this mural earlier or later than that one?" and "What is covered at the bottom?", filling the gap in the temporal topology expression of existing cultural heritage knowledge graphs. Efficient spatial indexing ensures a real-time interactive experience: The octree spatial index achieves a 3D spatial query response time of less than 80ms (for a 1.2 million-facet model), meeting real-time interaction requirements, with a spatial mapping accuracy exceeding 92%. Attached Figure Description

[0011] Figure 1 A general technical architecture diagram of an embodiment of a vision-aware 3D spatial knowledge question-answering system for cultural relics provided by the present invention; Figure 2 This is a schematic diagram of a three-layer semantic segmentation framework provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the eleven-field structure of the spatial region descriptor provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of five types of spatial topological relationships provided in the embodiments of the present invention; Figure 5 This is a schematic diagram illustrating three types of temporal topological relationships provided in embodiments of the present invention; Figure 6 A complete interactive flowchart of what you see is what you ask, provided for an embodiment of the present invention; Figure 7This is a schematic diagram of a field-of-view perception constraint filtering algorithm provided in an embodiment of the present invention; Figure 8 This is a comparison chart of the field-of-view perception constraint filtering effect provided in an embodiment of the present invention; Figure 9 A flowchart of 3D point cloud semantic segmentation and region descriptor generation provided in this embodiment of the invention; Figure 10 A flowchart illustrating the construction of a spatial topology graph and the mapping of a knowledge graph, provided in an embodiment of the present invention. Detailed Implementation

[0012] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0013] Figure 1 A general technical architecture diagram of an embodiment of a vision-aware 3D spatial knowledge question-answering system for cultural relics provided by the present invention is shown below. Figure 1 As shown, the knowledge question answering system includes: a 3D semantic segmentation and spatial indexing construction module, a spatiotemporal topological relationship graph construction module, a knowledge graph storage module, a vision-aware constraint question answering engine, and a user interaction module.

[0014] The 3D semantic segmentation and spatial indexing construction module is used to perform component-level, subject-level, and feature-level semantic segmentation on the 3D model of cultural relics, and to construct spatial region descriptors for each semantic segmentation region of various semantic segmentation results; the descriptors record temporal and spatial attribute information.

[0015] The spatiotemporal topology graph construction module is used to calculate the spatial and temporal topology relationships between any two semantically segmented regions based on descriptors, construct spatial and temporal topology graphs respectively, and store the spatial and temporal topology graphs in the knowledge graph storage module in the data format of a knowledge graph.

[0016] The user interaction module is used to receive user input for question and answer instructions.

[0017] The vision-aware constraint question-answering engine performs spatiotemporal intent parsing of question-answering commands, dynamically filters candidate regions of question-answering commands based on the user's current field of vision, generates answers based on the content stored in the knowledge graph storage module, and returns them to the user through the user interaction module.

[0018] This invention provides a vision-aware 3D spatial knowledge question-and-answer system for cultural relics. It combines 3D semantic segmentation, spatiotemporal topology modeling, vision-aware constraint filtering, and knowledge graph question-and-answer to achieve interactive intelligent question-and-answer for cultural relics in space. It is suitable for interactive knowledge services of "what you see is what you ask" in cultural heritage scenarios such as grottoes and museum artifacts.

[0019] Example 1

[0020] Embodiment 1 provided by this invention is an embodiment of a vision-aware three-dimensional spatial knowledge question-and-answer system for cultural relics provided by this invention, combined with... Figure 1 It is known that the implementation of this question-answering system includes: a 3D semantic segmentation and spatial indexing construction module, a spatiotemporal topological relationship graph construction module, a knowledge graph storage module, a vision-aware constraint question-answering engine, and a user interaction module.

[0021] The 3D semantic segmentation and spatial indexing construction module is used to perform component-level, subject-level, and feature-level semantic segmentation on the 3D model of cultural relics, and to construct spatial region descriptors for each semantic segmentation region of various semantic segmentation results; the descriptors record temporal and spatial attribute information.

[0022] To address the unique characteristics of 3D models of cultural relics (complex curved surfaces, dense images, and large scale span), this invention designs a three-level semantic segmentation framework, such as... Figure 2 The diagram shown is a schematic of a three-layer semantic segmentation framework provided in an embodiment of the present invention. Figure 2 It can be seen that, in one possible embodiment, the process of semantic segmentation by the 3D semantic segmentation and spatial index construction module includes: UV unpacking the 3D mesh of the cultural relic 3D model into a parametric 2D texture map, performing semantic segmentation (Mask R-CNN) operation on the 2D texture map to obtain the segmentation result, and mapping the segmentation result back to the 3D coordinate system of the cultural relic 3D model through back projection to achieve joint 3D and 2D segmentation.

[0023] Component-level semantic segmentation of 3D models of cultural relics involves dividing different types of cultural relics into various component regions based on their physical structure.

[0024] Table 1 below shows an example table of cultural relic component-level segmentation provided in the embodiments of the present invention: Table 1. Examples of cultural relic component-level segmentation

[0025] Thematic-level semantic segmentation of the 3D model of cultural relics involves semantically segmenting each component region of different types into image units with independent semantics.

[0026] Table 2 below shows an example table of cultural relic type and subject-level segmentation provided by the embodiments of the present invention: Table 2. Examples of cultural relic type and subject-based segmentation:

[0027] Feature-level semantic segmentation of the 3D model of the cultural relic involves fine-grained feature annotation of the designated key regions.

[0028] Table 3 below shows the fine-grained feature annotation table for key regions provided in the embodiments of the present invention: Table 3. Fine-grained feature annotation table for key regions:

[0029] Specifically, different semantic segmentation algorithms can be used for segmentation at each layer. Table 4 below shows the performance data of the segmentation algorithms provided in the embodiments of the present invention: Table 4 Performance data of segmentation algorithms

[0030] An eleven-field spatial region descriptor is constructed for each semantic segmentation region, such as... Figure 3 The diagram shown is a schematic diagram of the eleven-field structure of the spatial region descriptor provided in an embodiment of the present invention. Figure 3 It is known that, in one possible embodiment, the information recorded by the descriptor includes: region ID, semantic label, 3D bounding box, centroid coordinates, surface area, number of point clouds, segmentation confidence, knowledge graph node ID, normal vector, period label, and stack number.

[0031] Specifically, the structure of a descriptor can be: RegionDescriptor{ region_id: A unique identifier (format: {cavity number}_{level}_{serial number}) Example: K17_comp_001 (Component No. 001 of Kizil Cave 17) semantic_label: Semantic label (corresponding to CIDOC-CRM entity type) bounding_box: 3D bounding box [x_min, x_max, y_min, y_max, z_min, z_max] (unit: meters) centroid: Centroid coordinates (x, y, z) surface_area: Surface area (square meters) point_count: The number of point clouds contained. confidence: Segment confidence level (0-1, < 0.70 triggers manual review) kg_node_id: Corresponding knowledge graph node ID (bidirectional mapping key) normal_vector: Normal vector (used to determine orientation: East / West / South / North / Top / Bottom) temporal_period: Creation period marker (CIDOC-CRM E4_Period reference) [New] Example: "Early Tang Dynasty (618-712)" / "High Tang Dynasty (712-766)" temporal_layer: Stack index (0 = bottommost / earliest, larger values ​​indicate more recent) [New] Example: 0 (bottom layer mural), 1 (overlay mural), 2 (outermost layer) } The specific content of the descriptor for the head of the main Buddha statue in Kizil Cave 17 is as follows: RegionDescriptor{ region_id: "K17_comp_001" semantic_label: "Buddha's head" bounding_box: [x_min=-0.35, x_max=0.38, y_min=1.25, y_max=1.68, z_min=-0.18, z_max=0.20] centroid: [0.02, 1.47, 0.01] surface_area: 0.42 point_count: 128450 confidence: 0.94 kg_node_id: "E22_Main Buddha Statue of Cave 17, Kizil Caves" normal_vector: [0.01, 0.02, 1.00] / / Facing forward temporal_period: "Kucha Kingdom period (circa 5th-6th century AD)" temporal_layer: 0 / / Original layer, no overlay } In the specific implementation, an octree spatial index is used to store all region descriptors: Tree depth: Adaptive (minimum leaf node volume = 0.01m) 3 ).

[0032] Query interface: queryByPoint(x,y,z) → Returns the nearest region and its descriptor.

[0033] Query response time: <80ms (1.2 million polygon model, actual test).

[0034] Spatial mapping accuracy: over 92% (verified manually by 10 cultural heritage experts).

[0035] In one possible implementation, the descriptor includes a segmentation confidence field, which automatically triggers a manual review process when the confidence level is less than 0.70, ensuring that the spatial mapping accuracy reaches more than 92%.

[0036] The spatiotemporal topology graph construction module is used to calculate the spatial and temporal topology relationships between any two semantically segmented regions based on descriptors, construct spatial and temporal topology graphs respectively, and store the spatial and temporal topology graphs in the knowledge graph storage module in the data format of a knowledge graph.

[0037] like Figure 4 and Figure 5 The figures shown are schematic diagrams of five spatial topological relationship types and three temporal topological relationship types provided in the embodiments of the present invention. Figure 4 and Figure 5 It can be seen that, in one possible embodiment, the spatiotemporal topology graph construction module calculates the spatial topology relationship between any two semantic segmentation regions based on the descriptor as follows: the type of spatial topology relationship is determined based on the difference or positional relationship between the bounding boxes, centroid coordinate axis plane and normal vector of the two semantic segmentation regions. The types of spatial topology relationships include: adjacent, above, containing, symmetric and opposite.

[0038] Table 5 below shows the relationship type information table of spatial topology provided in the embodiments of the present invention: Table 5. Information on Spatial Topological Relationship Types

[0039] As shown in Table 5, the spatial topology graph uses an adaptive threshold parameter: the adjacent relationship distance threshold. =0.05m, threshold for height difference between the upper and lower levels =0.1m, symmetry deviation threshold =0.08m, the threshold of the angle between the normal vectors of the positive relationship is 150°, and the threshold parameter can be adaptively adjusted according to the scale of the cultural relic.

[0040] The process of calculating the temporal topological relationship between any two semantic segmentation regions based on descriptors is as follows: the type of temporal topological relationship is determined based on the temporal_period and temporal_layer fields of the semantic segmentation region; the types of temporal topological relationships include: earlier than, concurrent, and superimposed.

[0041] Specifically, the automatic calculation method for this time relationship includes: Step 0, Input data, including: a set of semantic segmentation regions Regions={R1,R2,...,Rn}, each semantic segmentation region containing the temporal_period and temporal_layer fields; and revision records of the knowledge graph's chronology information (CIDOC-CRMP4_with time span attribute) (if any).

[0042] Step 1: Determine the stacking relationship based on the temporal layer: Traverse all semantic segmentation region combinations (Ri, Rj) with spatial overlap exceeding 50%: If Ri's temporal_layer is less than Rj's temporal_layer: determine that Rj is superimposed on Ri and Ri is earlier than Rj.

[0043] Step 2, determine the relationship between earlier and concurrent times based on the time interval temporal_period: Traverse all spatially adjacent semantic segmentation region combinations (Ri, Rj): The temporal_period of the two semantic segmentation regions Ri and Rj is parsed respectively, and the time intervals period_i and period_j between the start year and the end year are obtained.

[0044] If the end year of Ri is less than the start year of Rj, then Ri is considered to be earlier than Rj.

[0045] If the end year of Rj is less than the start year of Ri, then Rj is considered to be earlier than Ri.

[0046] If the overlapping time range of Ri and Rj exceeds half of the maximum value of the time interval length of Ri and Rj, then Ri and Rj are considered to be contemporaneous.

[0047] Specifically, the pseudocode for steps 1 and 2 is as follows: Step 1: Determine the stacking relationship based on the temporal_layer FOR each pair (Ri, Rj) where spatial_overlap(Ri, Rj) > 0.50: IF Ri.temporal_layer < Rj.temporal_layer: add_relation(Rj, "overlaps", Ri) / / Rj overlaps Ri. add_relation(Ri, "before", Rj) / / Ri is earlier than Rj (implicit) Step 2: Determine the relationship between earlier and concurrent based on temporal_period. FOR each pair (Ri, Rj) where spatial_adjacent(Ri, Rj): period_i = parse_period(Ri.temporal_period) / / Parses as [start_year, end_year] period_j = parse_period(Rj.temporal_period) IF period_i.end < period_j.start: add_relation(Ri, "before", Rj) ELIF period_j.end < period_i.start: add_relation(Rj, "before", Ri) ELIF overlap(period_i, period_j) > 0.5 × max(duration_i, duration_j): add_relation(Ri, "contemporary", Rj) Step 3, verify time consistency Detect time relationship loops (A earlier than B, B earlier than C, C earlier than A → contradiction warning).

[0048] The detection of superposition and age discrepancies (the age of the upper spatial region should not be earlier than that of the lower layer).

[0049] Step 4, Output: Set of time topology triples + list of contradiction warnings.

[0050] Table 6 below shows the relationship type information table of the time topology provided in the embodiments of the present invention: Table 6. Information on Relationship Types in Temporal Topological Relationships

[0051] Table 7 below shows the test data table of the accuracy of time relationship analysis calculated according to the time relationship automatic calculation algorithm provided in the embodiments of the present invention: Table 7 Test data on the accuracy of time relationship analysis

[0052] Both spatial and temporal topological relationships are stored in RDF triples, seamlessly integrated with the CIDOC-CRM knowledge graph. #Namespace declaration @prefix spatial:<http: / / kg.cultural-heritage.cn / spatial#> @prefix temporal:<http: / / kg.cultural-heritage.cn / temporal#> @prefix region:<http: / / kg.cultural-heritage.cn / region#> #Spatial topological triplet region:K17_comp_001 spatial:above region:K17_comp_002. region:K17_comp_001 spatial:adjacent region:K17_feat_001. region:K17_comp_003 spatial:symmetric region:K17_comp_004. region:K17_comp_001 spatial:contains region:K17_feat_001. #Time-based topological triplet [New] region:K17_layer_0 temporal:before region:K17_layer_1. region:K17_layer_1 temporal:overlaps region:K17_layer_0. region:K17_subj_001 temporal:contemporary region:K17_subj_002. #Mapping relationship with knowledge graph nodes region:K17_comp_001 spatial:mapsTo kg:E22_Kizil Cave 17 Main Buddha Statue.

[0053] region:K17_feat_001 spatial:mapsTo kg:E55_Abhaya Mudra.

[0054] The example of spatiotemporal-semantic joint query provided by the embodiment of the present invention includes: / / Query 1: What is above the main Buddha statue in the 17th Kizil Grotto? (Spatial query) MATCH (anchor:Region {id: 'K17_comp_001'}) -[:spatial_above]->(neighbor:Region) -[:mapsTo]->(entity:Entity) RETURN entity.norm_name, entity.et_type, entity.description ORDER BY neighbor.centroid_z DESC / / Query 2: Is this mural earlier or later than the one next to it? (Temporal query) [Newly added] MATCH (anchor:Region {id: 'K17_subj_001'}) -[:temporal_before|temporal_contemporary]->(neighbor:Region) -[:mapsTo]->(entity:Entity) RETURN entity.norm_name, type(r) AS temporal_relation, neighbor.temporal_period AS period / / Query 3: What is the theme of the mural at the bottom of the east wall (covered)? (Superimposed overlap query) [Newly added] MATCH (top:Region {id: 'K17_east_layer_1'}) -[:temporal_overlaps]->(bottom:Region) -[:mapsTo]->(entity:Entity) WHERE bottom.temporal_layer = 0 RETURN entity.norm_name, entity.description, bottom.temporal_period The user interaction module is used to receive user input for question and answer instructions.

[0055] In one possible embodiment, the question-and-answer instructions received by the user interaction module include: click operations, box selection operations, and natural language questioning.

[0056] The user interaction module receives the user's click operation on the 3D model, obtains the 3D coordinates through ray projection, and queries the corresponding semantic region through the octree spatial index.

[0057] The generated natural language answers with spatial and temporal source information are highlighted in the corresponding areas on the 3D model displayed in the user interaction module.

[0058] The vision-aware constraint question-answering engine performs spatiotemporal intent parsing of question-answering commands, dynamically filters candidate regions of question-answering commands based on the user's current field of vision, generates answers based on the content stored in the knowledge graph storage module, and returns them to the user through the user interaction module.

[0059] like Figure 7 The diagram shown is a schematic of the field-of-view perception constraint filtering algorithm provided in an embodiment of the present invention. Figure 7 It can be seen that, in one possible embodiment, the method by which the view-aware constrained question-answering engine performs spatiotemporal intent parsing of question-answering commands includes: Construct a time lexicon mapping table; the time lexicon mapping table records the types of time topological relationships, time constraints, and the correspondences of various natural language expressions; the time constraints are the time ranges determined by the time intent parsed from the natural language expression of user input.

[0060] Specifically, Table 8 below shows the time-related vocabulary mapping table provided in the embodiments of the present invention: Table 8 Time-based vocabulary mapping table

[0061] Specifically, Table 9 below shows the spatial vocabulary mapping table provided in the embodiments of the present invention: Table 9 Spatial Vocabulary Map

[0062] Construct a spatial vocabulary mapping table; the spatial vocabulary mapping table records the correspondence between the types of spatial topological relationships, directional constraints and various natural language expressions; the directional constraints are the spatial range determined by the spatial intent parsed from the natural language expression of user input.

[0063] A large language model is used to parse users' natural language questions, identify spatial and temporal relation words, and map them into structured constraints.

[0064] Based on the temporal and spatial lexical mapping tables, the temporal and directional constraints corresponding to the user's input question-and-answer instructions are determined.

[0065] In practice, an example of spatiotemporal intent parsing is as follows: Question: "Is the mural next to this one earlier or later?" Analysis results: Anchor point entity: The region corresponding to the current clicked location (region_id=R042) Spatial relationship: Next to → Adjacent (any direction) Time Intent: Determine whether it is early or late → Query Before / Contemporary Relationship Query objective: Comparison of time attributes between adjacent regions Generate constraints: Step 1: Spatial Constraints → Obtain the set of Adjacent neighbors for R042 Step 2: Field of View Perception Filtering → Preserve Neighbors Within the Field of View Step 3: Query the temporary relationship of the retained neighbors → determine the order of return times. In one possible embodiment, the process by which the view-aware constrained question-answering engine dynamically filters candidate regions for question-answering instructions based on the user's current view includes: enter: User camera position cam_pos=(cx,cy,cz) Camera orientation vector cam_dir=(dx,dy,dz) (unit vector) Field of View (FOV, unit: degrees) Candidate region set Regions={R1,R2,...,Rn} Obtain the user's current camera position, orientation, and field of view.

[0066] Calculate the angle dir_i between the centroid of each candidate region Ri and the camera orientation.

[0067] Specifically, dir_i = normalize(centroid_i - cam_pos); where centroid_i represents the centroid of the candidate region Ri, cam_pos is the user's camera position, and normalize represents normalization processing.

[0068] Calculate the angle_i between the direction vector dir_i and the camera orientation vector.

[0069] When the included angle_i ≤ FOV / 2, the candidate region Ri is determined to be within the field of view and is retained; when the included angle_i > FOV / 2, the candidate region Ri is determined to be outside the field of view and is filtered; FOV is the field of view angle.

[0070] The remaining candidate regions Ri are sorted in ascending order according to the size of the included angle_i (the closer to the center of the field of view, the higher the priority), resulting in an ordered list of candidate regions.

[0071] like Figure 8 The image shown is a comparison chart of the field-of-view perception constraint filtering effect provided in the embodiments of the present invention. Table 10 below shows a comparison data table of the field-of-view constraint filtering effect indicators provided in the embodiments of the present invention: Table 10 Comparison of Field-of-Vision Constraint Filtering Effectiveness Indicators

[0072] In one possible embodiment, the vision-aware constraint question-answering engine further includes: performing multi-scale semantic understanding of the user-input question-answering instructions; performing joint constraint retrieval of the temporal topology graph, spatial topology graph, and knowledge graph based on spatial constraints, temporal constraints, and vision filtering results; and calling a large language model to generate natural language answers with spatial and temporal source information.

[0073] Multi-scale semantic understanding automatically selects the segmentation level based on the granularity of the user's question. Coarse-grained questions return component-level knowledge, while fine-grained questions return feature-level knowledge. For time-dimensional questions, it combines temporal_layer to filter the content of specific layers, supporting drill-down queries from caves to patterns and browsing time slices. Granularity judgment rules: Questions containing words such as "entire cave", "whole", or "all" → cave-level (component-level segmentation results); Questions containing specific component names ("East Wall", "Coffered Ceiling") → Component Level; Questions containing subject matter titles ("Sutra Transformation" or "Jataka Tales") → Subject-level; Problems containing detailed features ("gestures", "objects held", "patterns") → feature level; The question contains time-level words ("bottom layer", "original", "reconstruction") → combined with temporal_layer filtering [new].

[0074] Example of drill-down at different levels: Coarse-grained: "What are the stories behind Fengxian Temple?" →Return to cave-level knowledge (overall history of Fengxian Temple, composition of statues, construction background) Fine-grained question: "What instrument is that flying apsara holding?" →Return to feature-level knowledge (Pipa, symbolizing musical offerings, characteristics of the High Tang style). Time dimension: "What was originally painted on the bottom layer of this wall?" [New] → Filter the area where temporal_layer=0 to return the underlying mural theme information. When the answer is generated, 3D spatial and temporal source information is automatically added, and the relevant areas are highlighted on the 3D model: An example of the answer output structure is as follows: { "answer_text": "Natural Language Answer (LLM Generation)", "evidence_chain": "Original evidence sentence (from the source_sentence field of the knowledge graph)", "spatial_anchor": { "region_id": "R043", "centroid": [12.3, 8.7, 2.1], "distance_to_anchor": 0.82, "spatial_relation": "left_adjacent", "highlight_bbox": [x_min,x_max,y_min,y_max,z_min,z_max] }, "temporal_anchor":{ / / Add time-based tracing "temporal_period": "The High Tang Dynasty (712-766)", "temporal_layer": 1, "temporal_relation_to_anchor": "contemporary", "time_note": "Drawn during the same period as the anchor point area" }, "confidence": 0.91 } ``` Example answer (including time information): > "The area to the left of the location you clicked (north side of the east wall, 2.3m from the ground) is an image of a Bodhisattva offering (CIDOC-CRM:E36 visual artwork, depicting Avalokitesvara Bodhisattva on page 138), located approximately 0.82m to the left of the north side of the east wall, belonging to the retinue of the Bhaisajyaguru Sutra transformation. This Bodhisattva holds a vase and wears red celestial robes, in the style of the High Tang Dynasty. This area belongs to the High Tang (712-766) painting layer (temporal_layer=1), and was painted at the same time as the mural at your clicked location. The bottom layer (temporal_layer=0) is a beaded decorative band from the Early Tang Dynasty, which has been covered. [Spatial Origin: region_id=R043, centroid coordinates (12.3,8.7,2.1), distance from anchor point 0.82m, spatial relationship = left_adjacent, confidence level = 0.91] [Temporal Origin: Period = High Tang (712-766), layer = 1 / 2, time relationship with anchor point = contemporary]" Table 11 below shows the system architecture module information table of a vision-aware three-dimensional spatial knowledge question-and-answer system for cultural relics provided in an embodiment of the present invention.

[0075] Table 11 System Architecture Module Information Table

[0076] Example 2

[0077] Embodiment 2 of this invention is an embodiment of a visual perception-based three-dimensional spatial knowledge question-answering method for cultural relics provided by this invention. This embodiment of the knowledge question-answering method includes: Step 1: Perform semantic segmentation on the 3D model of the cultural relic at the component level, subject level, and feature level, respectively, and construct spatial region descriptors for each semantic segmentation region of each semantic segmentation result; the descriptors record time and space attribute information.

[0078] Step 2 is used to calculate the spatial and temporal topological relationships between any two semantic segmentation regions based on descriptors, construct spatial and temporal topological relationship graphs respectively, and store the spatial and temporal topological relationship graphs in the knowledge graph storage module in the data format of a knowledge graph.

[0079] Step 3: Receive the user's input of question and answer instructions.

[0080] Step 4: Perform spatiotemporal intent parsing on the question-and-answer command, dynamically filter the candidate region of the question-and-answer command based on the user's current field of view, generate the answer based on the content stored in the knowledge graph storage module, and return it to the user through the user interaction module.

[0081] It is understood that the visual perception-based three-dimensional spatial knowledge question-and-answer method for cultural relics provided by the present invention corresponds to the visual perception-based three-dimensional spatial knowledge question-and-answer system for cultural relics provided in the foregoing embodiments. The relevant technical features of the visual perception-based three-dimensional spatial knowledge question-and-answer method for cultural relics can be referred to the relevant technical features of the visual perception-based three-dimensional spatial knowledge question-and-answer system for cultural relics, and will not be repeated here.

[0082] An example of the complete interactive process (including time dimension enhancement) of the "what you see is what you ask" system and method for three-dimensional spatial knowledge of cultural relics provided by this invention is as follows: Step 1: The user rotates / zooms in the 3D model browser to locate the area of ​​interest.

[0083] Step 2: Click on a point on the surface of the 3D model → trigger Raycasting.

[0084] Step 3: Find the intersection of the ray and the 3D mesh → Obtain the 3D coordinates (x, y, z) of the hit point.

[0085] Step 4: Octree spatial index query → return the semantic region (<80ms).

[0086] Step 5: Pop up an information card: Region Name + Era Label + Knowledge Graph Summary + 3D Highlighting.

[0087] Step 6: The user inputs a natural language question (automatically associated with the currently clicked area as the anchor point).

[0088] Step 7: Spatiotemporal intent analysis → Extract spatial constraints + temporal constraints.

[0089] Step 8: Visual perception constraint filtering → narrowing down the candidate region range.

[0090] Step 9: Spatiotemporal topology graph query → obtain the relevant regions that satisfy both spatial and temporal constraints.

[0091] Step 10: Knowledge Graph Subgraph Retrieval → Obtain relevant entities and relationships.

[0092] Step 11: LLM generates natural language answers + spatial source information + temporal source information.

[0093] Step 12: Other areas related to the answer are highlighted synchronously on the 3D model (different colors are used to mark different periods).

[0094] Enhanced interactive features based on the time dimension include: automatic labeling of the period and layer number of each area in the information card; support for a "time slice" highlighting mode: users can choose to view only content from a specific period (e.g., "show only the early Tang Dynasty layer"); and support for a "timeline" browsing mode: highlighting each area sequentially according to time to show the evolution of the mural's creation. Figure 9 The diagram shown is a flowchart of the three-dimensional point cloud semantic segmentation and region descriptor generation provided in an embodiment of the present invention. Figure 10 The diagram illustrates the spatial topology graph construction and knowledge graph mapping process provided in this embodiment of the invention.

[0095] This invention provides a field-sensing three-dimensional spatial knowledge question-and-answer system and method for cultural relics, which can achieve: 1. Perform three-level semantic segmentation (component-level / theme-level / feature-level) on the 3D model of the cultural relic and construct a refined spatial region descriptor with a segmentation accuracy mIoU≥0.83.

[0096] 2. Establish a bidirectional and precise mapping between three-dimensional spatial coordinates and knowledge graph nodes, with a mapping accuracy of ≥92% and a query response time of <100ms.

[0097] 3. Construct a spatiotemporal topology graph containing 8 types of relationships (5 spatial relationships + 3 temporal relationships), supporting joint queries of spatial and temporal relationships. The accuracy rate of spatial relationship parsing reaches 87.5%, and the accuracy rate of temporal relationship parsing reaches 84.2%.

[0098] 4. Based on the field of view perception constraint filtering of the user's current camera view, the relevance of the answer is improved from 61% to 92% (an increase of 37.2 percentage points).

[0099] 5. Enables spatially anchored answer generation that allows users to "see what you ask," supporting multi-scale, hierarchical drill-down queries from caves to patterns.

[0100] 6. Supports time-based Q&A, answering questions about time relationships such as "Is this mural earlier or later than the one next to it?" and "What is the underlying mural?"

[0101] Example 3

[0102] Embodiment 3 of this invention is a specific application embodiment of the visual perception-based three-dimensional spatial knowledge question-and-answer system and method for cultural relics provided by this invention. This specific application embodiment takes the main Buddha statue in Cave 263 of the North Grottoes as an example to perform spatial perception question-and-answer, including: (1) 3D model acquisition Three-dimensional point cloud data of the main Buddha statue in Cave 263 of the North Grottoes were obtained using laser scanning. A three-dimensional mesh model was generated by Poisson reconstruction, containing approximately 1.2 million faces with a point cloud density of approximately 1 point / mm. 2 .

[0103] (2) Three-dimensional semantic segmentation The PointNet++ algorithm is used for component-level segmentation, MaskR-CNN (UV unwrapping) for subject-level segmentation, and a fine-tuned version of Swin3D for feature-level segmentation, resulting in a three-layer segmentation outcome. - Component level: Buddha head (K17_comp_001), Buddha body (K17_comp_002), lotus pedestal (K17_comp_003), halo (K17_comp_004) - Subject-specific: Explanation of the Dharma (K17_subj_001), Maitreya with crossed legs (K17_subj_002) - Characteristic Level: Abhaya Mudra (K17_feat_001), Varada Mudra (K17_feat_002) The overall segmentation accuracy mIoU = 0.87.

[0104] (3) Calculation of spatial topological relationships Automatically calculate spatial topological relationships between regions: - Above the Buddha's head (K17_comp_001) is a halo (K17_comp_004). - Below the Buddha's head (K17_comp_001) is the Buddha's torso (K17_comp_002). - The backlight (K17_comp_004) is symmetrical to the center line of the Buddha's head. - The Abhaya Mudra (K17_feat_001) is included in the head of the Buddha statue (K17_comp_001). (4) Calculation of temporal topological relationships [New] Based on chronological information and overlay analysis in the knowledge graph, the temporal topological relationships are automatically calculated: - The illustration of the Maitreya Buddha (K17_subj_001, temporal_layer=0, circa 5th century AD) predates the overlay of the Maitreya Buddha with crossed legs (K17_subj_002_overlay, temporal_layer=1, circa 6th-7th century AD). -The Maitreya Buddha with crossed legs overlay (K17_subj_002_overlay) is superimposed on the bottom layer of the preaching diagram. The donors on the east wall (K17_subj_003) and the donors on the west wall (K17_subj_004) are contemporaneous (both from the 5th-6th centuries AD). - Actual accuracy rate of time relationship analysis: 84.2% (5) User interaction and query When a user clicks on the Buddha statue's head (3D coordinates [0.01, 1.49, 0.02]), the octree query response time is 68ms. The location is K17_comp_001, and an information card pops up displaying "Buddha statue head, in the Abhaya Mudra (gesture of fearlessness), Kizil style, 5th-6th century AD, original layer (temporal_layer=0)".

[0105] (6) Spatial Relationship Questions and Answers A user asked, "What does this gesture mean?" - Spatial intent parsing: Anchor point = K17_comp_001, feature-level query, target = handprint meaning - Knowledge Graph Search: K17_feat_001 (Applying the Abhaya Mudra) → P138 Describes "Applying the Abhaya Mudra" → E55 Type "Symbolizing the Elimination of Fear in All Beings" -LLM generated the answer: "The Buddha's right hand is raised, palm facing outward, in the Abhaya Mudra (gesture of fearlessness)." The mudra (gesture) symbolizes the elimination of fear and danger for all beings and is one of the most common hand gestures in the Kizil Caves, prevalent from the 5th to 7th centuries AD. - The right hand area of ​​the Buddha statue is highlighted in the 3D model (K17_feat_001) (7) Time Management Q&A [New] A user asked, "What was originally painted underneath this mural?" - Spatiotemporal Intent Analysis: Anchor Point = K17_subj_002_overlay (currently visible Maitreya overlay layer), Temporal Relationship = Overlaps (check lower layers), Target = Underlying Theme - Spatiotemporal topology graph query: K17_subj_002_overlay-[:temporal_overlaps]→ K17_subj_001 (the bottom layer of the graph) - Knowledge Graph Search: K17_subj_001 → P138 Depicting the "Preaching Scene" → Describing "A scene of Shakyamuni Buddha preaching, drawn around the 5th century AD" -LLM generated the answer: "The currently visible Maitreya Buddha statue with crossed legs (repainted around the 6th-7th century AD) has a preaching scene (originally painted around the 5th century AD) at its base, depicting the Buddha preaching. This base mural is no longer directly observable due to later repainting, but its existence can be confirmed through infrared transmission photography and archaeological excavation records." - The 3D model is superimposed on the location of the underlying mural in a semi-transparent manner. (8) Verification of the effect of visual perception constraint User camera position (2.5, 1.5, 1.0), FOV=60°, camera facing directly forward: - No field-of-view constraints: Returns 23 candidate regions, including the backlit area behind the user, with a relevance of 61%. - With field-of-view constraints (regions with an angle > 30° are filtered): Returns 6 candidate regions, improving relevance to 92%. Example 4

[0106] Embodiment 4 of this invention is a specific application embodiment of the visual perception-based three-dimensional spatial knowledge question-and-answer system and method for cultural relics provided by this invention. This specific application embodiment takes the Vairocana Buddha of Fengxian Temple in Longmen Grottoes as an example for spatial perception question-and-answer, including: (1) 3D model acquisition A three-dimensional model of the Vairocana Buddha at Fengxian Temple in Longmen Grottoes was reconstructed using photogrammetry (128 high-resolution photos), containing approximately 2 million faces.

[0107] (2) Three-dimensional semantic segmentation The Swin3D algorithm was used for segmentation, and the following components were identified: Vairocana Buddha (comp_fx_001), disciple Kasyapa (comp_fx_002), disciple Ananda (comp_fx_003), Manjushri Bodhisattva (comp_fx_004), Samantabhadra Bodhisattva (comp_fx_005), Heavenly King (comp_fx_006), and Vajrapani (comp_fx_007).

[0108] (3) Calculation of spatial topological relationships Above the Vairocana Buddha (comp_fx_001) are Bodhisattvas listening to the Dharma and flying apsaras. -To the left of the Vairocana Buddha (comp_fx_001) are the disciples Kasyapa (comp_fx_002) and Ananda (comp_fx_003). - To the right of the Vairocana Buddha (comp_fx_001) are Manjushri Bodhisattva (comp_fx_004) and Samantabhadra Bodhisattva (comp_fx_005). - The Vairocana Buddha (comp_fx_001) faces the direction of the cave entrance. (4) Calculation of temporal topological relationships [New] -The main body of the Vairocana Buddha (comp_fx_001, completed in 675 during the Xianqing era) predates the Heavenly King (comp_fx_006, which, according to stylistic analysis, was added during the Wu Zhou period). -The Vairocana Buddha (comp_fx_001) and its disciples Kasyapa / Ananda (comp_fx_002 / 003) were created during the same period (both were built during the reign of Emperor Gaozong). - The base (comp_fx_007_base, the original base from the Tang Dynasty) was repaired and layered during the Song Dynasty. (5) Spatial Relationship Questions and Answers A user asked, "What's above the Buddha statue?" - Spatial intent resolution: Anchor point = comp_fx_001, Spatial relationship = Above - Field of View (FOV) Filtering: Filters out irrelevant areas in the horizontal direction from the user's upward viewing angle (FOV) of 45°. -Knowledge graph retrieval: Returns images of the Bodhisattva Listening to the Dharma and the Flying Apsaras, with a confidence level of 0.91. - Highlight the area above the Buddha's head in the 3D model. (6) Time Management Q&A [New] A user asked, "Were these statues made at the same time?" - Spatiotemporal intent analysis: Anchor point = comp_fx_001 (Vairocana Buddha), temporal relationship = Contemporary determination, query range = all statues within the field of view. - Field-of-view filtering: Retain comp_fx_001 to comp_fx_007 within the field of view. - Spatiotemporal topology graph query: -comp_fx_001-[:contemporary]→comp_fx_002,comp_fx_003 (contemporary) -comp_fx_001-[:before]→comp_fx_006 (earlier than the King) -LLM generated the answer: "The Vairocana Buddha and its two disciples, Kasyapa and Ananda, were built around the same time (during the Xianqing-Shangyuan period of Emperor Gaozong of Tang, 672-675 AD), while the statues of the Heavenly Kings on the left and right were added during the Wu Zhou period (approximately 690-705 AD), about 15-30 years later than the main Buddha. The base of the guardian statues shows traces of Song Dynasty repairs." - The 3D model uses colors to distinguish different periods: blue = Emperor Gaozong's reign (675 AD), green = Wu Zhou period, orange = Song Dynasty repairs. (7) Multi-scale semantic understanding verification - Coarse-grained question "What is the story of Fengxian Temple?" → Return to cave-level knowledge (Wu Zetian donated funds for its construction, and it was excavated during the Xianqing era of Emperor Gaozong of Tang, etc.) - Fine-grained question "What instrument is this flying apsara holding?" → Return to feature-level knowledge (Pipa, symbolizing musical offerings, Tang Dynasty style). - Regarding the time dimension question "Which sculpture was the earliest on this wall?" → Return the region where temporal_layer=0 and temporal_period is the earliest. (8) Verification of comprehensive technical effects Table 12 below shows a scoring table provided by an embodiment of the present invention, which invites 10 cultural heritage experts to score the system's responses (1-5 points): Table 12 Scoring Sheet

[0109] As shown in Table 12, the spatial knowledge question-and-answer system and method for three-dimensional cultural relics with spatial field perception provided by the present invention has an accuracy rate of 87.5% for relationship analysis and an accuracy rate of 84.2% for temporal relationship analysis.

[0110] This invention provides a vision-aware 3D spatial knowledge question-and-answer system and method for cultural relics, with the following beneficial effects: Firstly, it achieves a "what you see is what you ask" cultural heritage interaction paradigm: when a user clicks on any location on the 3D model, the system can identify the corresponding cultural relic element and answer related questions, improving interaction efficiency by over 40% (compared to the traditional "visual recognition → system switching → manual retrieval" process). Secondly, vision-aware constraint filtering significantly improves answer relevance: vision-aware constraint filtering increases answer relevance from 61% to 92% (an increase of 37.2 percentage points), reduces the average number of returned candidate regions from 23.4 to 6.1, and increases user satisfaction scores from 2.8 to 4.6 (based on scores from 10 cultural heritage experts, on a 1-5 scale). Thirdly, three-layer semantic segmentation supports multi-scale knowledge services: the component-level / theme-level / feature-level three-layer segmentation framework meets different levels of query needs from coarse-grained to fine-grained, with segmentation accuracies mIoU reaching 0.91, 0.87, and 0.83 respectively, covering the full-scale knowledge needs of cultural heritage research. The spatiotemporal topology graph enables joint spatial and temporal reasoning: Based on five spatial topological relationships, three new temporal topological relationships (earlier / contemporaneous / overlapping) are added, constructing a spatiotemporal topology graph with eight relationship types. This graph is integrated with the CIDOC-CRM knowledge graph in the form of RDF triples, providing a structured expression of the spatiotemporal joint dimension for cultural heritage knowledge for the first time. The accuracy rate for spatial relationship resolution reaches 87.5%, and the accuracy rate for temporal relationship resolution reaches 84.2%. The system can answer temporal-dimensional questions such as "Is this mural earlier or later than that one?" and "What is covered at the bottom?", filling the gap in the temporal topology expression of existing cultural heritage knowledge graphs. Efficient spatial indexing ensures a real-time interactive experience: The octree spatial index achieves a 3D spatial query response time of less than 80ms (for a 1.2 million-facet model), meeting real-time interaction requirements, with a spatial mapping accuracy exceeding 92%.

[0111] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0112] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0113] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0114] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0116] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0117] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A three-dimensional spatial knowledge question-and-answer system for cultural relics based on visual perception, characterized in that, The knowledge question answering system includes: a 3D semantic segmentation and spatial indexing construction module, a spatiotemporal topological relationship graph construction module, a knowledge graph storage module, a vision-aware constraint question answering engine, and a user interaction module; The 3D semantic segmentation and spatial indexing construction module is used to perform component-level, subject-level, and feature-level semantic segmentation on the 3D model of cultural relics, and to construct spatial region descriptors for each semantic segmentation region of various semantic segmentation results; the descriptors record time and space attribute information. The spatiotemporal topology graph construction module is used to calculate the spatial and temporal topology relationships between any two semantic segmentation regions based on the descriptor, construct spatial and temporal topology graphs respectively, and store the spatial and temporal topology graphs in the knowledge graph storage module in the data format of a knowledge graph. The user interaction module is used to receive question and answer instructions input by the user; The vision-aware constraint question-answering engine performs spatiotemporal intent parsing on the question-answering command, dynamically filters the candidate region of the question-answering command based on the user's current vision, generates an answer based on the content stored in the knowledge graph storage module, and returns it to the user through the user interaction module.

2. The knowledge question-answering system according to claim 1, characterized in that, The semantic segmentation process of the three-dimensional semantic segmentation and spatial indexing construction module includes: UV unpacking the three-dimensional mesh of the cultural relic three-dimensional model into a parametric two-dimensional texture map, performing semantic segmentation operation on the two-dimensional texture map to obtain the segmentation result, and mapping the segmentation result back to the three-dimensional coordinate system of the cultural relic three-dimensional model through back projection; Component-level semantic segmentation of 3D models of cultural relics involves dividing different types of cultural relics into various component regions based on their physical structure. Thematic-level semantic segmentation of the three-dimensional model of cultural relics involves semantically segmenting each component region of different types into image units with independent semantics. Feature-level semantic segmentation of the 3D model of the cultural relic involves fine-grained feature annotation of the designated key regions.

3. The knowledge question-answering system according to claim 1, characterized in that, The information recorded in the descriptor includes: region ID, semantic label, 3D bounding box, centroid coordinates, surface area, number of point clouds, segmentation confidence, knowledge graph node ID, normal vector, period label, and stack number.

4. The knowledge question-answering system according to claim 1, characterized in that, The spatiotemporal topology graph construction module calculates the spatial topology relationship between any two semantic segmentation regions based on the descriptor as follows: the type of spatial topology relationship is determined based on the difference or positional relationship between the bounding boxes, centroid coordinate planes, and normal vectors of the two semantic segmentation regions. The types of spatial topology relationships include: adjacent, above, containing, symmetric, and opposite. The process of calculating the temporal topological relationship between any two semantic segmentation regions based on the descriptor is as follows: the type of the temporal topological relationship is determined based on the time information recorded in the temporal_period and temporal_layer fields of the semantic segmentation region; the types of the temporal topological relationship include: earlier than, concurrent, and superimposed.

5. The knowledge question-answering system according to claim 4, characterized in that, The process by which the spatiotemporal topology graph construction module determines the type of the temporal topology relationship includes: For any i-th and j-th semantic segmentation regions Ri and Rj; Traverse all semantic segmentation region combinations (Ri, Rj) with spatial overlap exceeding 50%; If Ri's temporal_layer is less than Rj's temporal_layer: determine that Rj is superimposed on Ri and Ri is earlier than Rj; Traverse all semantic segmentation region combinations (Ri, Rj) that are spatially adjacent; The temporal_period of the two semantic segmentation regions Ri and Rj is parsed separately and transformed to obtain the time interval from the start year to the end year; If the end year of Ri is less than the start year of Rj, then Ri is determined to be earlier than Rj. If the end year of Rj is less than the start year of Ri, then Rj is determined to be earlier than Ri. If the overlapping time range of Ri and Rj exceeds half of the maximum value of the time interval length of Ri and Rj, then Ri and Rj are considered to be contemporaneous.

6. The knowledge question-answering system according to claim 4, characterized in that, The method by which the vision-aware constrained question-answering engine performs spatiotemporal intent parsing of the question-answering instructions includes: A time lexicon mapping table is constructed; the time lexicon mapping table records the types of time topological relationships, time constraints, and correspondences of various natural language expressions; the time constraints are time ranges determined based on the time intent parsed from the natural language expression input by the user; A spatial vocabulary mapping table is constructed; the spatial vocabulary mapping table records the correspondence between the types and directional constraints of the spatial topological relationships and various natural language expressions; the directional constraints are constraints on the spatial range determined by the spatial intent parsed from the natural language expression input by the user; Based on the time-based vocabulary mapping table and the space-based vocabulary mapping table, the time constraints and directional constraints corresponding to the user's input question-and-answer instructions are determined.

7. The knowledge question-answering system according to claim 1, characterized in that, The process by which the field-of-view-aware, constrained question-answering engine dynamically filters candidate regions for the question-answering instructions based on the user's current field of view includes: Obtain the user's current camera position, orientation, and field of view; Calculate the angle dir_i between the centroid of each candidate region Ri and the camera orientation; Calculate the angle_i between the direction vector dir_i and the camera orientation vector; When the included angle_i ≤ FOV / 2, the candidate region Ri is retained; when the included angle_i > FOV / 2, the candidate region Ri is filtered; FOV is the field of view angle. The remaining candidate regions Ri are sorted in ascending order according to the size of the included angle_i to obtain an ordered list of candidate regions.

8. The knowledge question-answering system according to claim 1, characterized in that, The vision-aware constraint question-answering engine also includes: performing multi-scale semantic understanding of the question-answering instructions input by the user; performing joint constraint retrieval of the temporal topology graph, spatial topology graph, and knowledge graph based on spatial constraints, temporal constraints, and vision filtering results; and calling a large language model to generate natural language answers with spatial and temporal source information.

9. The knowledge question-answering system according to claim 1, characterized in that, The question-and-answer instructions received by the user interaction module include: click operations, box selection operations, and natural language questioning; The user interaction module receives the user's click operation on the 3D model, obtains the 3D coordinates through ray projection, and queries the corresponding semantic region through the octree spatial index; The generated natural language answers with spatial and temporal source information are highlighted in the corresponding areas on the 3D model displayed in the user interaction module.

10. A visual perception-based three-dimensional spatial knowledge question-and-answer method for cultural relics, characterized in that, include: Step 1: Perform semantic segmentation on the 3D model of the cultural relic at the component level, subject level, and feature level, and construct spatial region descriptors for each semantic segmentation region of each semantic segmentation result; the descriptors record temporal and spatial attribute information. Step 2: Calculate the spatial and temporal topological relationships between any two semantic segmentation regions based on the descriptor, construct spatial and temporal topological relationship graphs respectively, and store the spatial and temporal topological relationship graphs in the knowledge graph storage module in the data format of a knowledge graph. Step 3: Receive the user's question and answer instructions; Step 4: Perform spatiotemporal intent parsing on the question-and-answer instruction, dynamically filter the candidate region of the question-and-answer instruction based on the user's current field of view, generate an answer based on the content stored in the knowledge graph storage module, and return it to the user through the user interaction module.

Images