Methods and devices for accessing historical building construction records
By generating refined query requests through multimodal perception and combining retrieval from a dense semantic vector memory and a domain knowledge graph, the problem of insufficient semantic understanding and cross-modal association in historical building construction archives queries is solved, achieving efficient and reliable intelligent question answering and dynamic knowledge injection.
Patent Information
- Application Number
- CN202611584677.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-14
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies for querying historical building construction records suffer from insufficient semantic understanding, cross-modal association, and knowledge tracing, resulting in low query efficiency and a lack of credible evidence for verification results.
By generating detailed query requests through multimodal perception, retrieving data using a dense semantic vector memory and domain knowledge graph, and generating natural language responses by combining concept routing mechanisms and pointer networks, we can achieve accurate understanding of historical building construction archives and traceable intelligent question answering.
It improves the recognition of professional terminology boundaries, expands the scope of knowledge coverage, enhances query efficiency and the credibility of generated content, reduces the risk of generating illusions, and achieves accurate understanding and dynamic knowledge injection of historical building construction archives.
Smart Images

Figure CN123115357A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of cultural heritage digitization and artificial intelligence, specifically to a method and apparatus for querying historical building construction archives. Background Technology
[0002] Currently, the search and verification of Yangshi Lei drawings mainly rely on researchers manually comparing multimodal archives such as reports, work logs, and hot sample annotations, supplemented by keyword retrieval systems and general text matching models for literature review.
[0003] However, in the process of implementing the relevant technologies, at least the following problems were found: Since the Yangshi Lei diagram files cover a variety of document formats and the terminology used is domain-specific, the above methods are obviously insufficient in terms of semantic understanding, cross-modal association and knowledge tracing, making it difficult to support structured question answering for complex academic queries, resulting in low query efficiency and a lack of credible evidence for the verification results. Summary of the Invention
[0004] In view of the above problems, the present invention provides a method and apparatus for querying historical building construction records.
[0005] According to a first aspect of the present invention, a method for querying historical building construction archives is provided, comprising: responding to receiving a document query request for historical building construction archives; performing multimodal perception on the document query request to perform concept segmentation on the document query request and generate a refined query request; based on the refined query request, performing searches in a dense semantic vector memory and a domain knowledge graph respectively to determine historical memory data and historical graph data, wherein the dense semantic vector memory and the domain knowledge graph are constructed based on formatted text sequences obtained after identifying and semantically aligning historical building construction archives; fusing the refined query request, historical memory data, and historical graph data in a target model to generate an enhanced feature representation; inputting concept metadata associated with the historical memory data and historical graph data into a concept routing mechanism to invoke routing units of a number of routing activations corresponding to the concept metadata, performing concept-level reasoning on the enhanced feature representation to obtain a reasoned feature representation; and using a pointer network, with the reasoned feature representation as a constraint, controlling the target model to extract target memory data from the historical memory data to generate a natural language response to the document query request.
[0006] According to an embodiment of the present invention, a multimodal perception is performed on a document query request to perform concept segmentation and generate a refined query request, including: performing word segmentation on the document query request to obtain an initial word sequence; generating a text semantic boundary signal based on the semantic coherence between adjacent terms in the initial word sequence; matching the initial word sequence with a construction terminology dictionary to obtain a matching result; generating a terminology boundary signal based on the matching result and the semantic abruptness between adjacent terms, wherein the terminology boundary signal is used to constrain complete terms from being split during segmentation; determining the coordinates of the region corresponding to the image pointing information in the image of the historical building construction archive according to the image pointing information carried in the document query request, and determining a visual boundary signal based on the region coordinates, wherein the visual boundary signal is used to associate and align the pointing region corresponding to the image pointing information with the corresponding term in the document query request; performing weighted fusion of the text semantic boundary signal, the terminology boundary signal, and the visual boundary signal to obtain a fusion boundary probability; and performing concept merging on the initial word sequence based on the fusion boundary probability to generate a refined query request, wherein the refined query request includes at least one concept unit with complete semantics.
[0007] According to an embodiment of the present invention, based on a fine-grained query request, retrieval is performed in a dense semantic vector memory and a domain knowledge graph to determine historical memory data and historical graph data, including: mapping the concept units in the fine-grained query request to query vectors, wherein the query vectors are used to represent the semantic information of the concept units; performing multi-granularity semantic retrieval in the dense semantic vector memory based on the query vectors to recall historical memory data that semantically matches the query vectors; performing entity recognition on the concept units in the fine-grained query request to link the identified entities to the corresponding entity nodes in the domain knowledge graph, and using the linked entity nodes as the starting point, performing relationship expansion along a preset relationship path to recall historical graph data related to the fine-grained query request.
[0008] According to an embodiment of the present invention, the method further includes: dividing the image of the historical building construction archive into a layout area to obtain a text area and a symbol area; performing typesetting correction on the text content identified from the text area to obtain corrected content, wherein the typesetting correction includes text direction conversion and size description standardization; performing terminology standardization processing on the special symbols identified from the symbol area to obtain standard symbols, wherein the terminology standardization processing includes parsing the special symbols into corresponding historical unit values and metric unit values; and aligning the standard symbols with the corrected content through spatial location association and semantic association to obtain a formatted text sequence.
[0009] According to an embodiment of the present invention, a dense semantic vector memory is constructed as follows: semantic boundary detection is performed on a formatted text sequence to obtain detection results; the formatted text sequence is then segmented into semantic units based on the detection results to obtain a set of semantic units; each semantic unit in the set of semantic units is vector-encoded using a preset domain embedding model to obtain a domain semantic vector; multi-granularity semantic vectors are generated based on standard symbols in the formatted text sequence; the domain semantic vectors and multi-granularity semantic vectors are stored in a vector database, and a multi-level index structure is constructed to obtain a dense semantic vector memory.
[0010] According to an embodiment of the present invention, a domain knowledge graph is constructed as follows: multiple core entities are extracted from historical building construction specifications and formatted text sequences, wherein the historical building construction specifications record historical building construction terminology and form specifications, and the core entities include at least building entities and component entities; a relationship network is constructed based on the association relationships between the multiple core entities, wherein the association relationships include at least entity inclusion relationships and size association relationships; the multiple core entities and the relationship network are stored in a graph database to form a domain knowledge graph.
[0011] According to an embodiment of the present invention, fine-grained query requests, historical memory data, and historical graph data are fused in the target model to generate an enhanced feature representation, including: embedding the fine-grained query requests, historical memory data, and historical graph data respectively, and concatenating them in the target model to obtain a feature fusion representation; performing cross-attention calculation on the feature fusion representation in the cross-attention layer at the bottom of the target model to obtain an enhanced feature representation.
[0012] According to an embodiment of the present invention, conceptual metadata associated with historical memory data and historical map data is input into a conceptual routing mechanism to invoke routing units corresponding to the number of routing activations corresponding to the conceptual metadata, and to perform conceptual-level reasoning on the enhanced feature representation to obtain a reasoning feature representation. This includes: performing a fusion projection transformation on the enhanced feature representation and conceptual metadata through a conceptual bridge to obtain a conceptual feature representation that the conceptual routing mechanism can process; determining the number of routing activations based on the data attributes indicated by the conceptual metadata, wherein the number of routing activations controls the number of routing units participating in conceptual-level reasoning; selecting multiple target routing units corresponding to the number of routing activations from multiple candidate routing units; inputting the conceptual feature representation into each target routing unit for parallel computation, and weighting and fusing the computation results output by each target routing unit to obtain the reasoning feature representation.
[0013] According to an embodiment of the present invention, a pointer network is used to control a target model to extract target memory data from historical memory data, using inference feature representation as a constraint, to generate a natural language response to a document query request. This includes: inputting the inference feature representation into the output layer of the target model as a constraint; during the process of the target model generating a natural language response based on the constraint, triggering the pointer network when it is determined that the next output word belongs to a preset category, wherein the preset categories include numerical categories and terminology categories; using the pointer network, based on the inference feature representation and the currently generated partial response content, locating and copying the corresponding original text data from the historical memory data as the target memory data; and controlling the target model to output the target memory data as the next output word to continue generating a natural language response until the natural language response generation is complete.
[0014] A second aspect of the present invention provides a historical building construction archive query device, comprising: a request segmentation module, configured to, in response to receiving a document query request for historical building construction archives, perform multimodal perception on the document query request to perform conceptual segmentation on the document query request and generate a refined query request; a history determination module, configured to, based on the refined query request, perform retrieval in a dense semantic vector memory and a domain knowledge graph respectively to determine historical memory data and historical graph data, wherein the dense semantic vector memory and the domain knowledge graph are constructed based on formatted text sequences obtained by recognizing and semantically aligning historical building construction archives; a data fusion module, configured to fuse the refined query request, historical memory data, and historical graph data in a target model to generate an enhanced feature representation; a concept reasoning module, configured to input concept metadata associated with historical memory data and historical graph data into a concept routing mechanism to call routing units of the number of routing activations corresponding to the concept metadata, perform concept-level reasoning on the enhanced feature representation, and obtain a reasoning feature representation; and a response generation module, configured to, using a pointer network and constrained by the reasoning feature representation, control the target model to extract target memory data from the historical memory data to generate a natural language response to the document query request.
[0015] A third aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.
[0016] A fourth aspect of the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.
[0017] A fifth aspect of the present invention also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.
[0018] According to embodiments of the present invention, multimodal boundary awareness is used to conceptually segment document query requests to obtain refined query requests, thereby improving the recognition effect of domain terminology boundaries. A dual-channel retrieval system using a dense semantic vector memory and a domain knowledge graph is employed to determine historical memory data and historical graph data, expanding the knowledge coverage. By fusing refined query requests, historical memory data, and historical graph data within the target model, the model's ability to jointly represent heterogeneous semantics is enhanced. Conceptual metadata is input into a concept routing mechanism, and a corresponding number of routing units are invoked to perform concept-level reasoning on the enhanced feature representation, improving the reasoning depth under limited computational resources. A pointer network, constrained by the reasoning feature representation, extracts target memory data from historical memory data to generate a natural language response, reducing the risk of generating illusions. Therefore, this approach at least partially improves the problems of inaccurate professional terminology representation, difficulty in dynamic knowledge injection, lack of cross-modal semantic alignment, and low credibility of generated content, thereby achieving accurate understanding, dynamic knowledge injection, and traceable intelligent question answering of historical building construction archives. Attached Figure Description
[0019] The above-mentioned contents, as well as other objects, features and advantages of the present invention, will become clearer from the following description of embodiments of the present invention with reference to the accompanying drawings.
[0020] Figure 1 The illustration shows an application scenario of the historical building construction archive query method, apparatus, equipment, medium, and program product according to embodiments of the present invention.
[0021] Figure 2 A flowchart of a method for querying historical building construction records according to an embodiment of the present invention is shown.
[0022] Figure 3 A data flow diagram of a method for querying historical building construction records according to an embodiment of the present invention is shown.
[0023] Figure 4 A schematic diagram showing images of historical building construction archives according to an embodiment of the present invention is provided.
[0024] Figure 5 A structural block diagram of a historical building construction record query device according to an embodiment of the present invention is shown.
[0025] Figure 6 A block diagram of an electronic device suitable for implementing a method for querying historical building construction records according to an embodiment of the present invention is shown. Detailed Implementation
[0026] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.
[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.
[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.
[0029] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).
[0030] In the technical solution of this invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.
[0031] Embodiments of the present invention provide a method for querying historical building construction archives, comprising: responding to receiving a document query request for historical building construction archives; performing multimodal perception on the document query request to perform concept segmentation on the document query request and generate a refined query request; based on the refined query request, performing searches in a dense semantic vector memory and a domain knowledge graph respectively to determine historical memory data and historical graph data, wherein the dense semantic vector memory and the domain knowledge graph are constructed based on formatted text sequences obtained after identifying and semantically aligning historical building construction archives; fusing the refined query request, historical memory data, and historical graph data in a target model to generate an enhanced feature representation; inputting concept metadata associated with the historical memory data and historical graph data into a concept routing mechanism to call routing units of a number of routing activations corresponding to the concept metadata, performing concept-level reasoning on the enhanced feature representation to obtain a reasoned feature representation; and using a pointer network, with the reasoned feature representation as a constraint, controlling the target model to extract target memory data from the historical memory data to generate a natural language response to the document query request.
[0032] Figure 1 The illustration shows an application scenario of the historical building construction archive query method, apparatus, equipment, medium, and program product according to embodiments of the present invention.
[0033] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.
[0034] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).
[0035] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.
[0036] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.
[0037] It should be noted that the historical building construction record query method provided in this embodiment of the invention can generally be executed by server 105. Correspondingly, the historical building construction record query device provided in this embodiment of the invention can generally be located in server 105. The historical building construction record query method provided in this embodiment of the invention can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the historical building construction record query device provided in this embodiment of the invention can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.
[0038] It should be understood that Figure 1 The number of first terminal devices, second terminal devices, third terminal devices, networks, and servers shown in the diagram is merely illustrative. Depending on implementation needs, any number of first terminal devices, second terminal devices, third terminal devices, networks, and servers can be included.
[0039] The following will be based on Figure 1 The described scene, through Figures 2-4 The method for querying historical building construction records according to embodiments of the present invention will be described in detail.
[0040] Figure 2 A flowchart of a method for querying historical building construction records according to an embodiment of the present invention is shown.
[0041] like Figure 2 As shown, this embodiment includes operations S210 to S250.
[0042] In operation S210, in response to receiving a document query request for the construction archives of historical buildings, the document query request is subjected to multimodal perception to perform conceptual segmentation of the document query request and generate a refined query request.
[0043] In operation S220, based on the fine-grained query request, searches are performed in the dense semantic vector memory and the domain knowledge graph to determine the historical memory data and historical graph data. The dense semantic vector memory and the domain knowledge graph are constructed based on the formatted text sequences obtained after identifying and semantically aligning the historical building construction archives.
[0044] In operation S230, the fine query request, historical memory data, and historical graph data are fused in the target model to generate an enhanced feature representation.
[0045] In operation S240, the concept metadata associated with historical memory data and historical map data is input into the concept routing mechanism to call the routing units corresponding to the number of routing activations of the concept metadata, perform concept-level reasoning on the enhanced feature representation, and obtain the reasoned feature representation.
[0046] In operating the S250, a pointer network is used, with inference feature representation as a constraint, to control the target model to extract target memory data from historical memory data in order to generate a natural language response to a document query request.
[0047] According to an embodiment of the present invention, a document query request for historical building construction archives input by a user is received. The document query request may be a natural language question such as "From which documents does the size data of the golden pillar in the main hall of Dingling Ling'en Hall come?", or it may also include the information of the golden pillar area circled by the user on the hot-tempered image associated with the question; or, the query request may be a complex query that requires multi-hop reasoning, such as "Find all halls that use swastika-shaped caisson ceilings and their designers, and explain in which year the form first appeared in the report."
[0048] The document query request is segmented to obtain an initial word sequence. Based on the semantic coherence between adjacent terms in this initial word sequence, a text semantic boundary signal is generated. Simultaneously, this initial word sequence is matched with a dictionary of architectural terms to obtain matching results. Based on these matching results and the degree of semantic abruptness between adjacent terms, a term boundary signal is generated. This term boundary signal is used to constrain complete terms from being split during segmentation. For example, it ensures that complex architectural terms like "Mingjian Jinzhu" are not incorrectly split into two independent concepts, "Mingjian" and "Jinzhu," or that "Qilan Qianhou Lang Jujia Wuju" is not incorrectly segmented into fragmented terminology pieces.
[0049] Upon detecting that the document query request carries image pointing information, the coordinates of the corresponding region on the image of the historical building construction archive are obtained, and a visual boundary signal is determined based on these coordinates. This visual boundary signal is used to align the pointing region corresponding to the image pointing information with the corresponding term in the document query request. Subsequently, the text semantic boundary signal, the term boundary signal, and the visual boundary signal are weighted and fused to obtain a fusion boundary probability. Based on this fusion boundary probability, the initial word sequence is conceptually merged to generate the refined query request, which includes at least one conceptual unit with complete semantics.
[0050] After obtaining a detailed query request, the conceptual units in the detailed query request are mapped to query vectors, whereby the query vectors represent the semantic information of the conceptual units. Based on the query vectors, multi-granularity semantic retrieval is performed in the dense semantic vector memory to recall historical memory data that semantically matches the query vector. For example, when a user's query contains ancient and modern expressions such as "one foot two inches" and "a column diameter of over a foot," the multi-granularity retrieval mechanism can achieve cross-granularity semantic matching and accurate recall; or when a user queries with the metric unit "80cm," the semantic vector at the metric unit granularity can match the record corresponding to "twenty-five inches" in the original archival text.
[0051] Simultaneously, entity recognition is performed on the conceptual units in the refined query request. The identified entities are then linked to corresponding entity nodes in the domain knowledge graph. Starting from the linked entity nodes, relationships are expanded along a pre-defined path to retrieve historical graph data related to the refined query request. For example, "Dingling" is linked to the corresponding architectural entity node in the knowledge graph, and its component nodes such as "Ling'en Hall" and "Golden Pillar," along with their associated dimensional information, are retrieved along a pre-defined path. The dense semantic vector memory and domain knowledge graph are pre-constructed based on formatted text sequences obtained after recognizing and semantically aligning historical building construction archives.
[0052] The detailed query request, the historical memory data, and the historical graph data are embedded separately and then concatenated in the target model to obtain a feature fusion representation. Then, in the cross-attention layer at the bottom of the target model, cross-attention calculation is performed on the feature fusion representation to obtain an enhanced feature representation, enabling full interaction between textual concepts and image regions, and between retrieval memory and graph relationships, within the same feature space.
[0053] Through a concept bridger, fusion projection transformation is performed on the enhanced feature representation and the concept metadata to obtain a concept feature representation processable by the concept routing mechanism. The number of routing activations is determined based on the data attributes indicated by the concept metadata. The number of routing activations is used to control the number of routing units participating in concept-level reasoning. For example, more routing units are activated when the data attributes are core engineering attributes such as "diameter" and "height", and fewer routing units are activated when the data attributes are non-engineering attributes such as "time". A plurality of target routing units corresponding to the number of routing activations are selected from a plurality of candidate routing units. The concept feature representation is input into each of the target routing units for parallel computation, and the computation results output by each of the target routing units are weighted and fused to obtain the reasoning feature representation.
[0054] The reasoning feature representation is input to the output layer of the target model as a constraint condition. In the process that the target model generates the natural language response based on the constraint condition, in response to determining that the next output word belongs to a preset category, a pointer network is triggered, where the preset category includes a numerical category and a term category. Using the pointer network, based on the reasoning feature representation and the currently generated partial response content, corresponding original text data is located and copied from the historical memory data as the target memory data.
[0055] For example, when a specific value of the diameter of a gold pillar needs to be output, the pointer network locates and copies the original text "one chi and two cun" from the historical memory data, instead of generating a possible approximate expression from the model vocabulary; or when the proprietary term "Wanzi Branch" needs to be output, the pointer network directly copies the term from the original official report to ensure the accuracy of the professional term.
[0056] The target model is controlled to output the target memory data as the next output word, so as to continue generating the natural language response until the generation of the natural language response is completed. In this embodiment, the finally generated response content is "The diameter of the gold pillar in the central bay of the Ling'en Hall of the Dingling Tomb is one chi and two cun (about 38.4 cm), which is from the official report submitted by Lei Tingchang on the 9th day of the 3rd month of the 13th year of the Guangxu reign", accompanied by a structured evidence traceability package, which includes information such as archive number SL-04-01-003-0187, document type "official report", evidence type "main text", and original position coordinates, for researchers to verify item by item.
[0057] By employing multimodal boundary awareness to segment document query requests conceptually, refined query requests are obtained, improving the recognition of domain terminology boundaries. A dual-channel retrieval system using a dense semantic vector memory and a domain knowledge graph identifies historical memory data and historical graph data, expanding knowledge coverage. By fusing refined query requests, historical memory data, and historical graph data within the target model, the model's ability to jointly represent heterogeneous semantics is enhanced. By inputting concept metadata into a concept routing mechanism and calling corresponding routing units to perform concept-level reasoning on enhanced feature representations, the reasoning depth is improved under limited computational resources. A pointer network, constrained by reasoning feature representations, extracts target memory data from historical memory data to generate natural language responses, effectively reducing the risk of generating illusions. This approach at least partially improves the problems of inaccurate professional terminology representation, difficulty in dynamic knowledge injection, lack of cross-modal semantic alignment, and low credibility of generated content, thereby achieving accurate understanding, dynamic knowledge injection, and traceable intelligent question answering of historical building construction archives.
[0058] According to an embodiment of the present invention, a multimodal perception is performed on a document query request to perform concept segmentation and generate a refined query request, including: performing word segmentation on the document query request to obtain an initial word sequence; generating a text semantic boundary signal based on the semantic coherence between adjacent terms in the initial word sequence; matching the initial word sequence with a construction terminology dictionary to obtain a matching result; generating a terminology boundary signal based on the matching result and the semantic abruptness between adjacent terms, wherein the terminology boundary signal is used to constrain complete terms from being split during segmentation; determining the coordinates of the region corresponding to the image pointing information in the image of the historical building construction archive according to the image pointing information carried in the document query request, and determining a visual boundary signal based on the region coordinates, wherein the visual boundary signal is used to associate and align the pointing region corresponding to the image pointing information with the corresponding term in the document query request; performing weighted fusion of the text semantic boundary signal, the terminology boundary signal, and the visual boundary signal to obtain a fusion boundary probability; and performing concept merging on the initial word sequence based on the fusion boundary probability to generate a refined query request, wherein the refined query request includes at least one concept unit with complete semantics.
[0059] For a document query request input by a user, the document query request is first processed by word segmentation and part-of-speech tagging, which decomposes it into an initial word sequence at the term level. At the same time, a pre-trained language model is used to extract the hidden state vector of each term, providing a semantic representation basis for subsequent boundary detection.
[0060] For the generation of text semantic boundary signals, semantic similarity is calculated based on the hidden states of adjacent word terms in the initial word sequence. During the online query phase, the similarity is calculated based on the hidden states of adjacent word terms in the user's query text. When the semantic vector directions of adjacent word terms tend to be consistent, it indicates a higher probability that they belong to the same semantic unit, and the boundary probability decreases accordingly; conversely, when the semantic vectors shift significantly, the boundary probability increases. This text semantic boundary signal is denoted as... , represents the semantic boundary signal of the text at the nth position, and its calculation method is inherited from the original boundary detection capability of the concept segmentation module.
[0061] To generate the terminology boundary signal, the initial word sequence is first matched with a pre-constructed dictionary of construction terms. This dictionary covers all construction terms recorded in documents such as "Regulations on Construction Practices of the Ministry of Works of the Qing Dynasty" and "Construction Calculation Examples", as well as frequently used official terms in official documents and construction logs, such as "Xiantai", "Petition", "Review", and "Approval". It is stored using a prefix tree data structure to support efficient matching. Based on this, the terminology boundary signal is generated using formula (1).
[0062] (1)
[0063] in, This represents the term boundary signal at the nth position. Indicates the balancing weight coefficient. This indicates an indicator function that takes the value 1 when its internal condition is true, and 0 otherwise. Dictionary of construction terminology This represents the activation function. This represents the learnable scaling weight parameters. The vector abrupt change at position n represents the degree of change, which is specifically calculated using formula (2). Indicates from the first A sequence of consecutive terms from the nth term to the nth term.
[0064] (2)
[0065] in, This function retrieves the maximum value; it takes the difference if the difference within the parentheses is greater than 0, otherwise it takes 0. This represents the cosine similarity function, used to calculate the cosine similarity between two vectors. Representing positions respectively The semantic vector of the term.
[0066] This formula measures the degree of semantic abruptness at position n by comparing the semantic similarity difference between the preceding pair of adjacent terms and the following pair of adjacent terms. When a continuous span hits the terminology dictionary, the term boundary signal within that span is forcibly suppressed to ensure that the complete term is not incorrectly split in subsequent concept segmentation; at the same time, the boundary signal at the span boundary is correspondingly enhanced, thereby constraining the complete term to participate in subsequent processing as an indivisible whole.
[0067] For the generation of visual boundary signals, the first step is to detect whether the document query request carries image pointing information. This visual boundary signal is only triggered in multimodal query scenarios, i.e., when the user submits a text query and points to a specific image area of a historical building construction archive (such as the location of a component in a model image) on the interface. When the user points to a specific image area, the coordinates of the area corresponding to the pointed location are obtained, and the boundary probability is extracted as the visual boundary signal from the corresponding area visual features pre-stored in the offline construction phase. If the document query request is in plain text mode and does not carry image pointing information, the visual boundary signal component is zero, and its weight in the fusion is zero. The corresponding value is set to 0. This signal ensures cross-modal alignment between the area the user points to on the image and the corresponding term in the document query request text.
[0068] After obtaining the above triple boundary signals, the fusion boundary probability is calculated by weighted fusion, as shown in formula (3).
[0069] (3)
[0070] in, This represents the fusion boundary probability at the nth position. This represents the visual boundary signal at the nth position. , and These represent the weighting coefficients of the three signals.
[0071] When performing concept merging on the initial word sequence based on the fusion boundary probability, Exceeding the preset merging boundary threshold At this point, position n is determined as the concept boundary. The boundary position divides the initial word sequence into multiple consecutive concept intervals, and the terms within the same interval are merged into a concept unit with complete semantics. The final generated refined query request contains at least one of the above concept units. Each concept unit encapsulates the corresponding term sequence, semantic vector, and related metadata information for use in the subsequent dual-channel retrieval and inference stages.
[0072] By integrating three boundary signals—textual semantic coherence, mandatory protection from a terminology dictionary, and image-guided region localization—this approach solves the technical problem of domain-specific terms being incorrectly segmented in general segmentation algorithms. It ensures that composite terms enter the subsequent retrieval and reasoning stages as complete semantic units, thereby improving the accuracy of query intent understanding.
[0073] Figure 3 A data flow diagram of a method for querying historical building construction records according to an embodiment of the present invention is shown.
[0074] like Figure 3 As shown, upon receiving document query request 301, a multimodal boundary signal 302 is generated based on the document query request 301, specifically including text semantic boundary signal, terminology boundary signal, and visual boundary signal. The text semantic boundary signal, terminology boundary signal, and visual boundary signal are then weighted and fused to generate a refined query request 303.
[0075] Subsequently, based on the fine-grained query request 303, a dual-channel retrieval is performed: first, multi-granular semantic retrieval is performed in the dense semantic vector memory 304 to recall historical memory data 306; second, entity linking and relationship expansion are performed in the domain knowledge graph 305 to recall historical graph data 307.
[0076] Next, the detailed query request 303, historical memory data 306, and historical graph data 307 are input into the target model 308 for multimodal fusion and concept-level reasoning. The target model 308 generates a natural language response 309 for the document query request 301. The entire process demonstrates a complete question-answering chain from query understanding, knowledge retrieval, multimodal fusion to controllable generation and source-tracing output.
[0077] According to an embodiment of the present invention, based on a fine-grained query request, retrieval is performed in a dense semantic vector memory and a domain knowledge graph to determine historical memory data and historical graph data, including: mapping the concept units in the fine-grained query request to query vectors, wherein the query vectors are used to represent the semantic information of the concept units; performing multi-granularity semantic retrieval in the dense semantic vector memory based on the query vectors to recall historical memory data that semantically matches the query vectors; performing entity recognition on the concept units in the fine-grained query request to link the identified entities to the corresponding entity nodes in the domain knowledge graph, and using the linked entity nodes as the starting point, performing relationship expansion along a preset relationship path to recall historical graph data related to the fine-grained query request.
[0078] After generating the refined query request, each conceptual unit in the refined query request is mapped to a query vector. Specifically, for each conceptual unit, a pre-trained domain-specific embedding model is used to encode it into a dense vector representation. This query vector is used to characterize the semantic information of the conceptual unit for similarity calculation in subsequent retrieval stages.
[0079] In the dense semantic vector memory retrieval channel, multi-granularity semantic retrieval is performed based on the above query vector. The dense semantic vector memory stores semantic vectors of four granularities: original symbol granularity, Qing Dynasty unit granularity, metric unit granularity, and component context granularity, which are injected into different partitions of the memory respectively. During retrieval, the similarity between the query vector and the vectors in each partition is calculated, and the final comprehensive similarity score is calculated by the following weighted fusion formula, as shown in formula (4).
[0080] (4)
[0081] in, This represents the overall similarity score between semantic unit u and the fine-grained query request q. These represent the weighting coefficients for the original symbol granularity, Qing Dynasty unit granularity, metric unit granularity, and component context granularity, respectively, with the weights set to... This multi-granularity retrieval mechanism enables users to effectively retrieve semantically matching historical memory data regardless of the granularity of their query, such as directly entering Suzhou code symbols, Qing Dynasty construction rulers, metric dimensions, or component context descriptions. These represent the semantic vectors of semantic unit u at the original symbol granularity, Qing Dynasty unit granularity, metric unit granularity, and component context granularity, respectively. This represents the query vector for the fine-grained query request q.
[0082] During retrieval, nearest neighbor search is performed within a pre-built three-level index architecture: the first level divides all vectors into 128 clusters through clustering to achieve vector filtering; the second level organizes vectors within each cluster using a graph structure; and the third level performs vector compression. Through the progressive filtering and ranking of the three-level index, millisecond-level vector retrieval response is achieved, ultimately recalling and querying multiple historical memory data with the highest vector semantic similarity. Each historical memory data is associated with a corresponding original text fragment, archive source, location coordinates, and confidence information.
[0083] In the domain knowledge graph retrieval channel, entity recognition is first performed on each conceptual unit in the detailed query request. This identifies entity names from the text content of the conceptual unit that may refer to building entities, component entities, craftsman entities, or document entities. For each identified entity, an entity linking operation is performed, linking it to the corresponding entity node in the domain knowledge graph. This domain knowledge graph is pre-constructed based on standard documents such as the "Qing Dynasty Ministry of Works Engineering Practice Rules and Regulations" and Yangshi Lei drawings, and is stored in a graph database. It contains 12 core entity categories, including building entities, component entities, craftsman entities, document entities, dimension entities, and parameter entities, as well as core relationships.
[0084] When linking entities, the entity name in the conceptual unit is matched and disambiguated with the name attribute of the entity node in the knowledge graph to determine the most likely corresponding node. Then, starting from the linked entity node, the relationship is expanded by one to two hops along the preset relationship path to form a subgraph structure related to the current query request. The entity nodes and their relationships in the subgraph are then retrieved as historical graph data.
[0085] After the dual-channel retrieval is completed, the candidate memory units returned by the dense semantic memory and the subgraph paths returned by the knowledge graph are jointly re-ranked. The re-ranking is based on a comprehensive score of the confidence of each candidate result, the weight of the source type, the path length penalty, and the relation matching degree. Then, noisy data with a confidence of less than 0.6 is filtered out, and finally, the most relevant historical memory data and their associated metadata, as well as multiple historical graph data and their path descriptions, are output for use in the subsequent fusion and inference stages.
[0086] By simultaneously mapping refined query concepts to semantic vectors for dense memory retrieval and linking them to entity nodes for knowledge graph subgraph expansion, a dual-channel complementary recall of "fuzzy semantic association" and "structured relational reasoning" is achieved. This solves the technical problem that a single retrieval method cannot simultaneously cover imprecise descriptions and multi-hop relational queries, thus improving the comprehensiveness of knowledge recall.
[0087] According to an embodiment of the present invention, the query of historical building construction archives further includes: dividing the image of the historical building construction archive into page areas to obtain text areas and symbol areas respectively; performing typesetting correction on the text content identified from the text areas to obtain corrected content, wherein the typesetting correction includes text direction conversion and size description standardization; performing terminology standardization processing on special symbols identified from the symbol areas to obtain standard symbols, wherein the terminology standardization processing includes parsing the special symbols into corresponding historical unit values and metric unit values; and aligning the standard symbols with the corrected content through spatial location association and semantic association to obtain a formatted text sequence.
[0088] Layout analysis algorithm is applied to the input file image for region division. Specifically, the layout analysis algorithm divides the input image into five types of regions: a text region, a table region, a seal region, a tangyang image region, and a symbol region, and outputs the bounding box and category confidence of each type of region. The text region covers the handwritten text regions of official memorials and on-site work diaries, the table region covers structured data regions such as engineering budget sheets and material lists, and the symbol region covers Suzhou numeral symbol regions.
[0089] For the recognition of the text region, a handwritten recognition algorithm based on an encoder-decoder structure is used to extract text from the official memorial image and the on-site work diary image. Before recognition, in view of the characteristics of scribbled handwriting in the on-site work diary, a contrast enhancement algorithm is adopted to perform handwriting image enhancement on the image, so as to improve the clarity of low-illumination regions. In view of the characteristics of vertically arranged traditional Chinese characters, the encoder adds a receptive field in the vertical direction, and the decoder supports an output order from right to left.
[0090] After the recognition output, post-processing is performed on cursive characters and variant characters by adopting language model-based fuzzy recognition error correction. For the tangyang image region, a special image recognition algorithm is used to perform labeling extraction of material recognition (such as "nanmu", "pinewood", "stone material", etc.) and structure recognition (such as "mortise and tenon structure", "dougong form", etc.).
[0091] For the symbol region, a convolutional recurrent neural network is used to perform Suzhou numeral symbol sequence recognition, and the network includes three parts: a convolutional layer for extracting visual features, a recurrent layer for modeling sequence relationships, and a transcription layer for outputting symbol sequences. The Suzhou numeral symbol set is defined as , where "one, two, three" are the horizontal writing forms of 1, 2, 3, which are used to avoid confusion with vertical strokes.
[0092] For the seal region, a seal classification and recognition algorithm is used for content extraction, and the recognition result is used to determine the issuing authority or signatory identity of the document (such as "the Ministry of Works", "the Yangshifang", etc.). For the table region, a table parsing algorithm based on connected domain analysis and heuristic rules is used to extract the table frame, recognize the text and Suzhou numeral symbols in the cells, and reorganize them into structured data according to the row and column logic, so as to realize accurate capture of information such as engineering budget sheets and material lists. The recognition results of the above regions respectively output text sequences, region coordinates and semantic labels, wherein the region coordinates and semantic labels of tangyang image recognition are transmitted as visual boundary signals to the subsequent term-aware boundary fusion step for use.
[0093] After the recognition is completed, typesetting correction and semantic association alignment are performed on the recognition result. Text direction correction is performed for the vertical writing format of official memorials. The reading order of vertically arranged text is from top to bottom and from right to left, and the correction algorithm rearranges the recognized text sequence into a standard horizontal arrangement format conforming to modern reading habits, while retaining the original bounding box position information to facilitate subsequent traceability.
[0094] Carry out term standardization on the material descriptions in the recognition results of architectural mock-ups and the table contents in the table recognition results. For example, standardize "3 cun fir" into "component diameter three cun material Chinese fir", standardize "pine rectangular timber" into "component name square timber material pine", and uniformly map the recognized descriptions to the standardized expressions in the standard term library.
[0095] Perform term standardization processing on Suzhou numeral symbols, and parse the recognized Suzhou numeral symbol sequence into the corresponding numerical value. The conversion reference adopts the Qing Dynasty construction chi, as specifically shown in Formula (5).
[0096] ; (5)
[0097] wherein, represents the metric unit value, with the unit of centimeter, represents the value after Suzhou numeral parsing (the unit is cun), that is, 1 chi = 10 cun ≈ 32.0 centimeters. Accordingly, the Suzhou numeral "〢〥" is parsed into both the Qing Dynasty unit value "25 cun" and the metric unit value "80.0 cm", forming a standardized multi-granularity dimension expression.
[0098] Finally, perform spatial position association and semantic association alignment on the standardized Suzhou numeral symbols, the corrected text content and the table cells. Specifically, based on the bounding box coordinates of each region output in the layout analysis stage, the relative positional relationship between the symbol region and the text region is determined. Insert the symbol sequence into the correct position in the corresponding text sequence according to the positional relationship; determine the contextual meaning of the symbol based on the semantics of adjacent texts, for example, when "diameter... cun" appears around the symbol, it is determined to have a dimension meaning, and when "labor... person-time" appears, it is determined to have a labor quantity meaning. The Suzhou numeral symbols recognized in the table area are semantically associated with the corresponding table headers and row labels according to the row and column positions of the cells where they are located, so as to form structured key-value pair data.
[0099] Through the above typesetting correction, term standardization, and spatial and semantic association alignment, a complete computable formatted text sequence is finally output. Each piece of information in this sequence is marked with its source type (petition document, work log, architectural mock-up or archive), evidence carrier type (main text, table, seal or atlas relationship) and original position coordinates, for subsequent semantic modeling and knowledge graph construction.
[0100] By performing layout region division, vertical text direction correction, dimension expression standardization, and ancient-modern unit dual-granularity parsing of special symbols (Suzhou numerals) on the images of historical building construction archives, and performing spatial and semantic association alignment between the symbols and the text content, the technical problem that handwritten traditional Chinese characters, variant symbols and unconventional typesetting in heterogeneous archives are difficult to be directly read by downstream models is solved, and the input quality of subsequent semantic modeling and the traceability of dimension conversion are guaranteed.
[0101] Figure 4 A schematic diagram showing images of historical building construction archives according to an embodiment of the present invention is provided.
[0102] like Figure 4 As shown, the images in the historical building construction archives are divided into multiple functional areas with different information carriers after the layout is divided. These areas include at least: text areas, table areas, hot stamping image areas, and seal areas.
[0103] The text area further includes a symbol area, which specifically refers to the image sub-region where the Suzhou code symbols are located, and is used for subsequent symbol recognition and terminology standardization processing.
[0104] The table area corresponds to structured data carriers such as project budget sheets and material lists in the archives. This area uses a table parsing algorithm to extract the table framework and identify the cell content.
[0105] The hot-tempered image area corresponds to the three-dimensional model image in the archive. The material and structural annotations of this area are extracted using a specialized image recognition algorithm.
[0106] The seal area corresponds to the official seal affixed at the end of formal documents such as petitions. This area uses seal classification and recognition algorithms to extract the seal content, which is used to determine the issuing institution or signatory of the document. The recognition results for each of the above areas output text sequences, area coordinates, and semantic labels, providing basic data support for subsequent typesetting correction, terminology standardization, and multimodal semantic alignment.
[0107] According to an embodiment of the present invention, a dense semantic vector memory is constructed as follows: semantic boundary detection is performed on a formatted text sequence to obtain detection results; the formatted text sequence is then segmented into semantic units based on the detection results to obtain a set of semantic units; each semantic unit in the set of semantic units is vector-encoded using a preset domain embedding model to obtain a domain semantic vector; multi-granularity semantic vectors are generated based on standard symbols in the formatted text sequence; the domain semantic vectors and multi-granularity semantic vectors are stored in a vector database, and a multi-level index structure is constructed to obtain a dense semantic vector memory.
[0108] Semantic unit boundary-aware segmentation is performed on a formatted text sequence. A boundary-aware segmentation algorithm is used to determine the boundary probability of the k-th term in the input text sequence. Calculated using the following formula (6).
[0109] (6)
[0110] in, Indicates the position of the k-th term. Boundary probability at the given location, This represents the k-th word in the input text sequence. and These represent the forward and backward hidden states of a bidirectional long short-term memory network, respectively. Show a learnable weight matrix, represents the bias term, and sigmoid represents the activation function.
[0111] The model predicts whether each term's location is a semantic unit boundary. The set of semantic units is defined as: .in, Represents a set of semantic units. This represents the continuous span from the i-th term to the j-th term. Indicates the threshold for semantic boundary determination. This means that for all terms k in the interval (i,j), the boundary probability must be lower than the threshold.
[0112] That is, a semantic unit contains all terms from i to j, and the boundary probabilities of all terms within this interval are lower than a preset threshold for semantic boundaries. This segmentation algorithm can accurately identify the boundaries of complex construction terms such as "seven purlins, front and rear corridors, and five-tiered roof structure," dividing them into two independent semantic units: "seven purlins, front and rear corridors" and "five-tiered roof structure." At the same time, it maintains the indivisibility of "five-tiered roof structure" as a complete technical expression. It can also completely segment official document terms such as "approval from the Censorate" and "request for cancellation" in official documents.
[0113] After obtaining the set of semantic units, each semantic unit is vector-encoded using a pre-defined domain embedding model. This pre-defined domain embedding model is then fine-tuned to construct a dense vector model specifically for Yangshi Lei. The vector representation of a single semantic unit u is calculated using the following formula (7).
[0114] (7)
[0115] in, The vector representation of the semantic unit u. This represents a domain-specific embedding model, where α represents the pooling weight coefficient, used to balance the contributions of the [CLS] vector and the weighted average vector. Preferably, α = 0.3. Indicates the hidden state of the [CLS] tag. This represents the hidden state of the t-th term. Indicates the total number of terms in a semantic unit. This represents the overall weight of the t-th term. Overall Weight The calculation method is shown in formula (8).
[0116] (8)
[0117] in, Indicate the part of speech of the t-th term. Let λ represent the natural exponential function, and let λ represent the position decay coefficient, preferably λ=0.15. This represents the distance between the t-th term and the center position of the semantic unit, where center represents the center position of the semantic unit. That is, terms that belong to the noun or quantifier category receive additional weight gain, while positional decay is applied based on their distance from the center position of the semantic unit.
[0118] While generating domain semantic vectors, multi-granularity semantic vectors are generated based on standard symbols in the formatted text sequence, specifically Suzhou code symbols. Suzhou code symbols are standardized into semantic vectors of four granularities: the original symbol granularity is mapped to the original Suzhou code symbol sequence (e.g., "〢〥"), the Qing Dynasty unit granularity is mapped to a numerical expression in Qing Dynasty construction feet (e.g., "25 inches"), the metric unit granularity is mapped to the converted metric value (e.g., "80.0cm"), and the component context granularity is mapped to a complete expression incorporating the component (e.g., "the diameter of the gold pillar is 25 inches"). These four granularity semantic vectors are injected into different partitions of a dense semantic vector memory, allowing users to perform searches using expressions of arbitrary granularity.
[0119] To achieve effective alignment of ancient and modern semantic representations, a multi-granularity semantic alignment loss function was designed during model training. The total loss function is shown in formula (9).
[0120] (9)
[0121] in, This represents the total loss function for multi-granularity semantic alignment. This represents the loss of multiple similarity boundary triples. This represents the loss from knowledge distillation. Indicates anchor point loss. This represents the semantic alignment loss, used to constrain the distance between ancient and modern representations in the vector space to not exceed a preset threshold. , , These represent the weight coefficients of the knowledge distillation loss, anchor point loss, and semantic alignment loss, respectively. Semantic alignment loss The distance between the ancient and modern representations in the vector space is forced to not exceed a preset threshold, as shown in formula (10).
[0122] (10)
[0123] in, A semantic vector representing ancient expressions. A semantic vector representing a modern expression; This represents the square of the L2 norm, used to calculate the square of the Euclidean distance between two vectors. This indicates the preset alignment distance threshold.
[0124] This loss ensures that the distance between ancient expressions such as "the diameter of the column exceeds one foot" and modern expressions such as "one foot and two inches" in the vector space does not exceed 0.3, thus achieving accurate recall of fuzzy expressions. The multi-similarity boundary triplet loss is defined as shown in formula (11).
[0125] (11)
[0126] in, Describe the set of triples The size of the triplet, i.e., the number of triples; Let represent a triple, where 'a' represents the anchor sample, 'p' represents the positive sample, and 'n' represents the negative sample. Represents the hard boundary threshold, preferably =0.45; , , These represent the semantic vectors of the anchor sample, positive sample, and negative sample, respectively. Indicates the soft boundary threshold, preferably =0.25, Represents the target similarity threshold, preferably =0.35. This loss function achieves refined training of the model's discriminative ability through the dual constraints of hard and soft boundaries.
[0127] After vector encoding, domain semantic vectors and multi-granularity semantic vectors are stored in a vector database, and a multi-level index structure is constructed to achieve millisecond-level retrieval of hundreds of millions of vectors. The first level is a semantic clustering layer, which uses a clustering algorithm to divide all vectors into 128 clusters, enabling fast pre-filtering of vectors. The second level is a graph indexing layer, where vectors within each cluster are organized using a graph structure to ensure the accuracy and efficiency of intra-cluster retrieval. The third level is a vector compression layer, which performs vector compression to further reduce storage space and retrieval computation. During retrieval, the query vector undergoes three levels of processing: clustering filtering, graph index retrieval, and vector fine ranking. Finally, the candidate memory unit with the highest similarity is returned, and a four-granularity weighted fusion calculation is used to calculate the final comprehensive similarity.
[0128] By performing semantic boundary detection and unit segmentation on formatted text, using a domain-specific embedding model for vector encoding, and generating multi-granularity semantic vectors such as original symbols, historical units, and metric units from Suzhou code symbols, combined with a multi-level index structure, the technical problems of sparse semantic terminology, large differences in the form of ancient and modern size descriptions, and low vector retrieval efficiency in ancient building construction archives have been solved, achieving unified recall and millisecond-level retrieval response for different granular expressions.
[0129] According to an embodiment of the present invention, a domain knowledge graph is constructed as follows: multiple core entities are extracted from historical building construction specifications and formatted text sequences, wherein the historical building construction specifications record historical building construction terminology and form specifications, and the core entities include at least building entities and component entities; a relationship network is constructed based on the association relationships between the multiple core entities, wherein the association relationships include at least entity inclusion relationships and size association relationships; the multiple core entities and the relationship network are stored in a graph database to form a domain knowledge graph.
[0130] Multiple core entities were extracted from historical building construction standards and formatted text sequences. The historical building construction standards specifically refer to documents such as the *Qing Dynasty Ministry of Works Engineering Practice Rules and Regulations* and *Construction Calculation Examples*, which record historical building construction terminology and structural specifications. The formatted text sequences are derived from the output results of identifying and semantically aligning historical building construction archives. During entity extraction, core entities included at least architectural entities (such as Dingling Mausoleum and Ling'en Hall) and component entities (such as main columns and bracket sets), as well as craftsman entities, document entities, dimensional entities, and parameter entities, covering a total of 12 core entity types.
[0131] Entity extraction can be achieved by combining rule-based pattern matching with a sequence labeling-based named entity recognition model. Rule matching uses a terminology dictionary to accurately extract standard terms from normative documents, while the sequence labeling model performs open-domain entity recognition on free text in a formatted text sequence.
[0132] After obtaining the core entities, a relationship network is constructed based on the associations between multiple core entities. The associations include at least entity containment relationships (such as a building containing components) and size relationships (such as a component having dimensions), as well as designed_by (design relationship), authorized (authorized relationship), mentions (mention relationship), has_parameter (parameter relationship), etc., covering a total of 18 core relationship types.
[0133] Relation extraction is performed as follows: For highly structured normative documents, a template-based rule matching method is used to extract explicit relationships between entities from the text using predefined relation extraction templates; for free text in formatted text sequences, a remotely supervised relation extraction model is used to align existing entity pairs in the knowledge base with co-occurrence patterns in the text, and a relation classifier is trained after automatically labeling the training data. Each relation record in the relation network includes a head entity, relation type, tail entity, corresponding confidence score, and source document identifier.
[0134] Recorded with typical confiscation entities For example, its entity record format is shown in formula (12).
[0135] (12)
[0136] Here, `type` represents the document type field, `date` represents the date field, `author` represents the author field, `archive_id` represents the archive number field, and `weather` represents the weather field. The values of each of these fields correspond to the specific content of the corresponding attributes in the report or the work log.
[0137] Recorded in a typical work log For example, its entity record format is shown in formula (13).
[0138] (13)
[0139] After entity extraction and relation network construction are completed, multiple core entities and relation networks are stored in a graph database to form a domain knowledge graph. Entities are stored as nodes, each containing attribute information such as entity name, entity type, source archive number, original text fragment, and location coordinates; relations are stored as directed edges with type labels, each containing attribute information such as relation type, confidence level, and source document identifier.
[0140] Once the knowledge graph is constructed, in the subsequent online query phase, when a user submits a query request, the entity name in the query concept is linked to the corresponding entity node in the graph. Starting from this node, a one- to two-hop subgraph recall is performed along a preset relationship path, converting the subgraph structure into a graph concept vector for use in the downstream multimodal fusion and reasoning phase.
[0141] By extracting multiple core entities, such as building entities and component entities, as well as their inclusion relationships and dimensional relationships from historical building construction standard documents and archival texts, and storing them in a graph database to form a structured knowledge network, the technical problem that implicit knowledge in archival texts (such as component attribution and dimensional relationships) is difficult to be used for structured querying and reasoning is solved, and traversable relational path support is provided for multi-hop reasoning queries.
[0142] According to an embodiment of the present invention, fine-grained query requests, historical memory data, and historical graph data are fused in the target model to generate an enhanced feature representation, including: embedding the fine-grained query requests, historical memory data, and historical graph data respectively, and concatenating them in the target model to obtain a feature fusion representation; performing cross-attention calculation on the feature fusion representation in the cross-attention layer at the bottom of the target model to obtain an enhanced feature representation.
[0143] The target model employs a large language model, and its feedforward neural network has been modified into a hybrid expert structure to support subsequent dynamic routing reasoning. In the fusion phase, the fine-grained query requests, historical memory data, and historical graph data are first embedded separately. Specifically, each conceptual unit in the fine-grained query request corresponds to an internal concept. Visual concepts are obtained by mean pooling the hidden states of each term within a concept unit; Visual features of image regions pre-stored during the layout analysis and specialized identification stages; external concepts Embedding vectors corresponding to candidate memory units retrieved from a dense semantic vector memory; graph concept. The embedding vectors are derived from the entity nodes and relational paths retrieved by the knowledge graph subgraph retrieval. After the embedding processing of the above four types of concepts is completed, they are concatenated in the target model to obtain the feature fusion representation, as shown in formula (14).
[0144] (14)
[0145] in, This represents the feature fusion representation; Concat(⋅) represents the concatenation operation, used to concatenate the concept vectors along the feature dimension. This concatenation operation occurs at the bottom layer of the target model, that is, in the first core computational unit after the input embedding layer. Its design purpose is to enable the injected external concepts to fully influence the expert weight allocation and feature extraction process in subsequent hybrid expert inference.
[0146] Subsequently, in the cross-attention layer at the bottom of the target model, cross-attention calculation is performed on the above-mentioned feature fusion representation. Specifically, the query, key, and value matrices are calculated based on the fused feature representation, as shown in Equations (15) to (18). In this case, each modality uses an independent projection matrix to map the original vector to the shared attention space. Furthermore, a mask matrix based on modality weights is introduced to control the interaction strength between concepts of each modality, and the cross-attention calculation result is shown in Equation (19).
[0147]
[0148] in, This represents the merged query matrix. This represents the fused key matrix. This represents the fused value matrix; , , These represent the query, key, and value projection matrices of the text concepts, respectively. , , These represent the query, key, and value projection matrices of visual concepts, respectively. , , These represent the query, key, and value projection matrices of the external concept, respectively. , , These represent the query, key, and value projection matrices of a graph concept, respectively. This represents the dimension of each attention head. express Matrix transpose operation, This represents the result of the cross-attention calculation.
[0149] The mask matrix is used to selectively control the propagation range of cross-modal attention weights. For example, in the early stages of training or in a specific task, it can limit the direct interaction between visual concepts and graph concepts to avoid introducing too much noise.
[0150] Through the aforementioned underlying cross-attention calculation, the four types of concepts complete deep interaction and fusion at the bottom layer of the target model, generating an enhanced feature representation. This enhanced feature representation includes not only the semantic information of the user query, but also the retrieved historical archive memory, the structured relationships of the knowledge graph, and the visual features of the image region. It is then passed up to the subsequent hybrid expert layer as an enhanced input representation.
[0151] This underlying injection mechanism enables externally retrieved knowledge to participate in feature transformation from the lowest level of the model, thereby profoundly influencing the evolution of hidden states at each subsequent step and the scoring results of experts in the hybrid expert layer by the gating network, ultimately determining which professional knowledge experts are activated to participate in conceptual reasoning.
[0152] According to an embodiment of the present invention, conceptual metadata associated with historical memory data and historical map data is input into a conceptual routing mechanism to invoke routing units corresponding to the number of routing activations corresponding to the conceptual metadata, and to perform conceptual-level reasoning on the enhanced feature representation to obtain a reasoning feature representation. This includes: performing a fusion projection transformation on the enhanced feature representation and conceptual metadata through a conceptual bridge to obtain a conceptual feature representation that the conceptual routing mechanism can process; determining the number of routing activations based on the data attributes indicated by the conceptual metadata, wherein the number of routing activations controls the number of routing units participating in conceptual-level reasoning; selecting multiple target routing units corresponding to the number of routing activations from multiple candidate routing units; inputting the conceptual feature representation into each target routing unit for parallel computation, and weighting and fusing the computation results output by each target routing unit to obtain the reasoning feature representation.
[0153] After completing the fusion of the four-modal concepts at the bottom layer, the enhanced feature representation and associated concept metadata are first input into the concept bridge. The enhanced feature representation received by the concept bridge includes multimodal enhanced features after cross-attention fusion, while the concept metadata includes metadata fields associated with historical memory data and historical graph data, specifically including seven fields: component name, attribute type, size value, dynasty, confidence level, document type, and evidence type. Among them, the document type is used to identify whether the data comes from a petition, work log, sample, or archive, and the evidence type is used to identify whether the data comes from the main text, table, seal, or graph relationship path. The concept bridge fuses and maps the enhanced feature representation and concept metadata through the following projection transformation to obtain the concept feature representation that the concept routing mechanism can process, as shown in formula (19).
[0154] (19)
[0155] in, This represents the j-th concept feature representation output by the concept bridge. Presentation layer normalization operation, The projection matrix representing the enhanced features. This represents the enhanced feature representation corresponding to the j-th candidate knowledge unit. The projection matrix representing metadata. This indicates a metadata embedding function. This represents the concept metadata corresponding to the j-th candidate knowledge unit. This formula maps multimodal features to a unified dimensional representation that can be processed by the concept routing mechanism by fusing and projecting the enhanced features and metadata embeddings. Projection matrix The training optimization is performed using the following loss function, as shown in formula (20).
[0156] (20)
[0157] in, Indicates projection loss, Let (u, v) represent an aligned dataset, where u represents a semantic unit and v represents the corresponding input text. This represents the vector after the semantic unit u is encoded by the domain embedding model. This represents the hidden state output by the first layer of a Large Language Model (LLM) for input text v. The loss function constrains the alignment between the projected conceptual feature representation and the underlying hidden state of the target model.
[0158] After obtaining the concept feature representation, the number of route activations is determined based on the data attributes indicated by the concept metadata. This number of route activations is used to control the number of route units participating in the current concept-level inference, and its calculation method is shown in formula (21).
[0159] ;(twenty one)
[0160] in, This represents the number of routes activated corresponding to concept c, where floor(⋅) represents the floor function, and the base number of activations. =8, minimum activation number =4, maximum number of activations =16, The scaling factor represents the metadata of the concept cc. The scaling factor is dynamically determined by the attribute type and component type in the metadata. The specific calculation method is shown in formula (22).
[0161] ;(twenty two)
[0162] in, Indicates the attribute type in the concept metadata. This indicates the component type in the conceptual metadata. When the attribute type belongs to core engineering attributes such as diameter, height, and taper, the scaling factor is increased to 1.8, which activates more experts to participate in deep reasoning; when the attribute type is a non-engineering attribute such as time description, the scaling factor is reduced to 0.6, which activates only a few experts to save computing resources.
[0163] After determining the number of active routes, multiple target routes corresponding to the number of active routes are selected from a pre-set pool of candidate routes. The selection process is based on gating scores, which take into account both concept feature vectors and metadata embedding. The probability distribution of each candidate expert is calculated in the following way, as shown in formula (23).
[0164] ;(twenty three)
[0165] in, This represents the gating score vector on the candidate routing unit. Let represent the learnable gated projection matrix, d represent the feature dimension, and c represent the concept feature vector. Represents metadata embedding vectors, This represents the concatenated vector of concept features and metadata embeddings. Subsequently, the top vectors with the highest gating scores are selected. Each routing unit is used as the target routing unit, as shown in formula (24).
[0166] ;(twenty four)
[0167] in, This represents the selected gated score vector. (⋅,⋅) represents the operation of selecting the top k maximum values and their corresponding indices. This formula is used to select the top k highest-scoring routing units from all candidate routing unit gating scores. Each routing unit serves as the target routing unit.
[0168] The conceptual feature representations are input into each target routing unit for parallel forward computation. Each routing unit is essentially a feedforward network expert, and it has undergone differentiated and specialized training for different categories of conceptual knowledge. The computation results output by the routing unit are weighted and fused according to the corresponding gating scores, and the specific calculation method is shown in formula (25).
[0169] (25)
[0170] in, This represents the final output after concept c is routed by dynamic concept experts, and e represents the index number of the candidate routing unit. Indicates the number of routing units actually activated. This represents the gating score corresponding to the e-th routing unit. This represents the computational output of the e-th feedforward network expert on the concept feature c. Unselected experts do not participate in the calculation, and their output contribution is zero. The formula obtains the final inference feature representation by weighting and aggregating the outputs of each target routing unit according to its gating score.
[0171] Through the aforementioned gating score selection mechanism, dynamic routing effectively controls computational resource consumption while ensuring inference depth. Finally, the weighted aggregated output is passed up to the output layer of the target model as the inference feature representation for use in subsequent natural language generation and source tracing stages.
[0172] According to an embodiment of the present invention, a pointer network is used to control a target model to extract target memory data from historical memory data, using inference feature representation as a constraint, to generate a natural language response to a document query request. This includes: inputting the inference feature representation into the output layer of the target model as a constraint; during the process of the target model generating a natural language response based on the constraint, triggering the pointer network when it is determined that the next output word belongs to a preset category, wherein the preset categories include numerical categories and terminology categories; using the pointer network, based on the inference feature representation and the currently generated partial response content, locating and copying the corresponding original text data from the historical memory data as the target memory data; and controlling the target model to output the target memory data as the next output word to continue generating a natural language response until the natural language response generation is complete.
[0173] After multi-layered inference involving bottom-level four-modal fusion and dynamic concept expert routing, the inference feature representation output from the top layer is passed to the output layer of the target model (i.e., the language model head) as a constraint in the generation process. The decoder of the target model uses a beam search algorithm to progressively generate natural language answers that conform to human reading habits. In each step of the decoder's autoregressive generation, the probability distribution of the next output word is calculated based on the inference feature representation and the currently generated partial response content.
[0174] During the generation process described above, the output probability distribution of the target model is monitored in real time to determine whether the predicted next output word belongs to a preset category. Preset categories include numerical categories and terminology categories, specifically covering key information types that need to be consistent with the original archives, such as size values, component names, and construction terms. When the predicted category of the next output word is detected to belong to one of these preset categories, a pointer network mechanism is triggered.
[0175] After the pointer network is triggered, cross-attention weights are calculated based on the hidden state of the current decoding step and the hidden states of each term in the historical memory data. This attention mechanism locates the term in the historical memory data that is most relevant to the current generation context. Specifically, for each term in the historical memory data, its relevance score to the current decoding hidden state is calculated. After normalization, an attention distribution is obtained, and the term position with the highest attention weight is selected. The original text data corresponding to that position is copied as the target memory data. This copying mechanism forces the model to directly extract the original values and proper nouns from the historical memory data recalled during the retrieval phase, rather than generating them from the model's own vocabulary. This ensures zero-illusion output of key data; for example, "the diameter of the gold pillar is one foot and two inches" in the text is completely copied, without the model generating an approximate but incorrect expression.
[0176] After the pointer network completes the copying operation, the target model uses the copied target memory data as the output word of the current step, replacing the content that might have been generated from the vocabulary, and continues the decoding process. During the generation process, metadata generated at each stage is collected synchronously to generate a structured evidence tracing package. Evidence tracing package The format is shown in formula (26).
[0177] (26)
[0178] Here, `text` represents the original text cited in the answer, `source` indicates the type of source document for the evidence, used to indicate whether the data comes from a report, work log, sample, or archive, `archive_id` represents the archive number, `component` represents the name of the building component involved, `attribute` represents the type of attribute involved, such as diameter, height, etc., and `bbox` represents the coordinates of the original location bounding box. , These represent the x and y coordinates of the top-left corner of the bounding box, respectively; w represents the width of the bounding box; h represents the height of the bounding box; image_region represents the image region link; confidence represents the confidence level; and γ represents the specific value of the confidence level. This evidence tracing package records the source information of each piece of data in the answer through structured fields, supporting item-by-item verification. The meanings of the remaining symbols are the same as described above.
[0179] For evidence derived from tables, additional row and column coordinates are added; for evidence derived from seals, additional seal identification attributes are added; for evidence derived from graph relationships, additional relationship path descriptions are added. The final generated answer is output along with the evidence tracing package, enabling researchers to verify the source of each data point in the answer, meeting the needs of academic literature verification.
[0180] By triggering a pointer network when the next output word belongs to a preset category such as a numerical value or a proper noun during the decoding process of the target model generating natural language responses, the corresponding original source data is forcibly located and copied from historical memory data as output. This solves the technical problems that large language models are prone to illusions when generating professional data and that key dimensions and terminology cannot be guaranteed to be consistent with the original files. It achieves zero-illusion output of key data and native traceability capability.
[0181] Based on the above-mentioned method for querying historical building construction records, this invention also provides a device for querying historical building construction records. The following will be combined with... Figure 5 The device is described in detail.
[0182] Figure 5 A structural block diagram of a historical building construction record query device according to an embodiment of the present invention is shown.
[0183] like Figure 5 As shown, the historical building construction record query device 500 of this embodiment includes a request segmentation module 510, a history determination module 520, a data fusion module 530, a concept reasoning module 540, and a response generation module 550.
[0184] The request segmentation module 510 is used to respond to a document query request for the construction archives of historical buildings, perform multimodal perception on the document query request, perform conceptual segmentation on the document query request, and generate a refined query request.
[0185] The history determination module 520 is used to retrieve historical memory data and historical graph data based on a fine-grained query request in the dense semantic vector memory and the domain knowledge graph, respectively. The dense semantic vector memory and the domain knowledge graph are constructed based on formatted text sequences obtained by recognizing and semantically aligning historical building construction archives.
[0186] The data fusion module 530 is used to fuse detailed query requests, historical memory data, and historical graph data in the target model to generate enhanced feature representations.
[0187] The concept reasoning module 540 is used to input the concept metadata associated with historical memory data and historical graph data into the concept routing mechanism, so as to call the routing units corresponding to the number of routing activations of the concept metadata, perform concept-level reasoning on the enhanced feature representation, and obtain the reasoned feature representation.
[0188] The response generation module 550 is used to control the target model to extract target memory data from historical memory data by using pointer networks and inference feature representations as constraints, in order to generate a natural language response to a document query request.
[0189] According to an embodiment of the present invention, the request segmentation module 510 includes a word segmentation processing submodule, a semantic boundary generation submodule, a dictionary matching submodule, a terminology boundary generation submodule, a visual boundary generation submodule, a signal fusion submodule, and a request generation submodule.
[0190] The word segmentation processing submodule is used to segment document query requests into words to obtain an initial word sequence.
[0191] The semantic boundary generation submodule is used to generate text semantic boundary signals based on the degree of semantic coherence between adjacent words in the initial word sequence.
[0192] The dictionary matching submodule is used to match the initial word sequence with the constructed terminology dictionary to obtain the matching results.
[0193] The term boundary generation submodule is used to generate term boundary signals based on the degree of semantic aberration between the matching results and adjacent terms. The term boundary signals are used to constrain the complete terms from being split during segmentation.
[0194] The visual boundary generation submodule is used to determine the coordinates of the region corresponding to the image pointer information in the image of the historical building construction archive based on the image pointer information carried in the document query request, and to determine the visual boundary signal based on the region coordinates. The visual boundary signal is used to associate and align the pointer region corresponding to the image pointer information with the corresponding term in the document query request.
[0195] The signal fusion submodule is used to perform weighted fusion of text semantic boundary signals, term boundary signals and visual boundary signals to obtain the fusion boundary probability.
[0196] The request generation submodule is used to merge concepts in the initial word sequence based on the fusion boundary probability to generate a fine-grained query request, which includes at least one concept unit with complete semantics.
[0197] According to an embodiment of the present invention, the history determination module 520 includes a unit mapping submodule, a semantic retrieval submodule, and a relationship expansion submodule.
[0198] The Unit Mapping submodule is used to map concept units in a fine-grained query request into query vectors, where the query vectors are used to represent the semantic information of the concept units.
[0199] The semantic retrieval submodule is used to perform multi-granular semantic retrieval based on the query vector in a dense semantic vector memory to recall historical memory data that semantically matches the query vector.
[0200] The relation extension submodule is used to perform entity recognition on the conceptual units in the fine query request, link the identified entities to the corresponding entity nodes in the domain knowledge graph, and extend the relation along the preset relation path starting from the linked entity nodes to recall historical graph data related to the fine query request.
[0201] According to an embodiment of the present invention, the historical building construction archive query device 500 further includes a region division module, a typesetting correction module, a standard processing module, and an association alignment module.
[0202] The area division module is used to divide the images of historical building construction archives into text areas and symbol areas.
[0203] The layout correction module is used to correct the layout of the text content identified from the text region to obtain the corrected content. The layout correction includes text direction conversion and size representation standardization.
[0204] The standard processing module is used to perform terminology standardization processing on the special symbols identified from the symbol area to obtain standard symbols. The terminology standardization processing includes parsing the special symbols into their corresponding historical unit values and metric unit values.
[0205] The alignment module is used to align standard symbols with the correction content through spatial and semantic association, resulting in a formatted text sequence.
[0206] According to an embodiment of the present invention, the historical building construction record query device 500 further includes a boundary detection module, a vector encoding module, a vector generation module, and a vector storage module.
[0207] The boundary detection module is used to perform semantic boundary detection on the formatted text sequence, obtain the detection results, and then segment the formatted text sequence into semantic units based on the detection results to obtain a set of semantic units.
[0208] The vector encoding module is used to perform vector encoding on each semantic unit in the semantic unit set using a preset domain embedding model, so as to obtain a domain semantic vector.
[0209] The vector generation module is used to generate multi-granularity semantic vectors based on standard symbols in a formatted text sequence.
[0210] The vector storage module stores domain semantic vectors and multi-granularity semantic vectors into the vector database and constructs a multi-level index structure to obtain a dense semantic vector memory.
[0211] According to an embodiment of the present invention, the historical building construction record query device 500 further includes an entity extraction module, a network construction module, and a map formation module.
[0212] The entity extraction module is used to extract multiple core entities from historical building construction specifications and formatted text sequences. The historical building construction specifications record historical building construction terminology and form specifications, and the core entities include at least building entities and component entities.
[0213] The network building module is used to construct a relationship network based on the relationships between multiple core entities, wherein the relationships include at least entity containment relationships and size relationships.
[0214] The graph formation module is used to store multiple core entities and relationship networks in a graph database to form a domain knowledge graph.
[0215] According to an embodiment of the present invention, the data fusion module 530 includes an embedding processing submodule and a cross-computation submodule.
[0216] The embedding processing submodule is used to embed fine query requests, historical memory data and historical graph data respectively, and then concatenate them in the target model to obtain a feature fusion representation.
[0217] The cross-computation submodule is used to perform cross-attention computation on the feature fusion representation in the cross-attention layer at the bottom of the target model to obtain the enhanced feature representation.
[0218] According to an embodiment of the present invention, the concept reasoning module 540 includes a projection transformation submodule, a quantity determination submodule, a route selection submodule, and a feature calculation submodule.
[0219] The projection transformation submodule is used to perform a fusion projection transformation on the enhanced feature representation and concept metadata through the concept bridge to obtain a concept feature representation that can be processed by the concept routing mechanism.
[0220] The quantity determination submodule is used to determine the number of route activations based on the data attributes indicated by the concept metadata. The number of route activations is used to control the number of route units participating in concept-level reasoning.
[0221] The route selection submodule is used to select multiple target route units from multiple candidate route units, corresponding to the number of active routes.
[0222] The feature calculation submodule is used to input the concept feature representation into each target routing unit for parallel calculation, and to weight and fuse the calculation results output by each target routing unit to obtain the inference feature representation.
[0223] According to an embodiment of the present invention, the response generation module 550 includes a feature input submodule, a network triggering submodule, a data copying submodule, and an output determination submodule.
[0224] The feature input submodule is used to input the inference feature representation into the output layer of the target model as a constraint.
[0225] The network triggering submodule is used to trigger the pointer network when the next output word belongs to a preset category during the process of the target model generating a natural language response based on constraints. The preset categories include numerical categories and term categories.
[0226] The data copying submodule is used to locate and copy the corresponding original text data from historical memory data, based on the pointer network, inference feature representation, and currently generated partial response content, as the target memory data.
[0227] The output determination submodule is used to control the target model to output the target memory data as the next output word to continue generating natural language responses until the natural language responses are generated.
[0228] According to embodiments of the present invention, any plurality of modules among the request segmentation module 510, history determination module 520, data fusion module 530, concept reasoning module 540, and response generation module 550 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the request segmentation module 510, history determination module 520, data fusion module 530, concept reasoning module 540, and response generation module 550 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the request segmentation module 510, history determination module 520, data fusion module 530, concept reasoning module 540, and response generation module 550 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.
[0229] Figure 6 A block diagram of an electronic device suitable for implementing a method for querying historical building construction records according to an embodiment of the present invention is shown.
[0230] like Figure 6As shown, an electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory ROM 602 or a program loaded from a storage portion 608 into a random access memory RAM 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.
[0231] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 602 and / or RAM 603. It should be noted that programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in one or more memories.
[0232] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.
[0233] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.
[0234] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.
[0235] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the historical building construction archive query method provided in the embodiments of the present invention.
[0236] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.
[0237] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.
[0238] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.
[0239] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, languages such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0240] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0241] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.
[0242] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.
Claims
1. A method for querying historical building construction records, characterized in that, The method includes: In response to receiving a document query request for the construction archives of historical buildings, the document query request is subjected to multimodal perception to perform concept segmentation and generate a refined query request; Based on the refined query request, searches are performed in the dense semantic vector memory and the domain knowledge graph to determine historical memory data and historical graph data. The dense semantic vector memory and the domain knowledge graph are constructed based on formatted text sequences obtained after identifying and semantically aligning the historical building construction archives. The fine-grained query request, the historical memory data, and the historical graph data are fused in the target model to generate an enhanced feature representation; The concept metadata associated with the historical memory data and the historical map data is input into the concept routing mechanism to call the routing units corresponding to the number of routing activations of the concept metadata, and to perform concept-level reasoning on the enhanced feature representation to obtain the reasoning feature representation; Using a pointer network and constrained by the inference feature representation, the target model is controlled to extract target memory data from the historical memory data to generate a natural language response to the document query request.
2. The method according to claim 1, characterized in that, The step of performing multimodal perception on the document query request to perform concept segmentation and generate refined query requests includes: The document query request is segmented to obtain an initial word sequence; Based on the semantic coherence between adjacent terms in the initial word sequence, a text semantic boundary signal is generated; The initial word sequence is matched with the constructed terminology dictionary to obtain the matching results; Based on the matching results and the degree of semantic mutation between adjacent terms, a term boundary signal is generated, wherein the term boundary signal is used to constrain the complete term from being split during segmentation; Based on the image pointing information carried in the document query request, the coordinates of the region corresponding to the image pointing information in the image of the historical building construction archive are determined, and a visual boundary signal is determined based on the region coordinates. The visual boundary signal is used to associate and align the pointing region corresponding to the image pointing information with the corresponding term in the document query request. The text semantic boundary signal, the term boundary signal, and the visual boundary signal are weighted and fused to obtain the fusion boundary probability; Based on the fusion boundary probability, the initial word sequence is merging concepts to generate the refined query request, which includes at least one concept unit with complete semantics.
3. The method according to claim 1, characterized in that, Based on the refined query request, searches are performed in the dense semantic vector memory and the domain knowledge graph to determine historical memory data and historical graph data, including: The concept units in the fine-grained query request are mapped to query vectors, wherein the query vectors are used to characterize the semantic information of the concept units; Based on the query vector, multi-granularity semantic retrieval is performed in the dense semantic vector memory to recall historical memory data that semantically matches the query vector; Entity identification is performed on the conceptual units in the refined query request, and the identified entities are linked to the corresponding entity nodes in the domain knowledge graph. Starting from the linked entity nodes, the relationship is expanded along a preset relationship path to recall historical graph data related to the refined query request.
4. The method according to claim 1, characterized in that, The method further includes: The images in the historical building construction archives are divided into page areas to obtain text areas and symbol areas respectively; The text content identified from the text region is corrected in terms of layout to obtain corrected content, wherein the layout correction includes text direction conversion and size representation standardization; Special symbols identified from the symbol region are subjected to terminology standardization processing to obtain standard symbols, wherein the terminology standardization processing includes parsing the special symbols into corresponding historical unit values and metric unit values; The standard symbols are aligned with the correction content through spatial and semantic association to obtain the formatted text sequence.
5. The method according to claim 4, characterized in that, The dense semantic vector memory is constructed in the following way: Semantic boundary detection is performed on the formatted text sequence to obtain detection results. Based on the detection results, the formatted text sequence is segmented into semantic units to obtain a set of semantic units. Using a preset domain embedding model, each semantic unit in the semantic unit set is vector-encoded to obtain a domain semantic vector; Based on the standard symbols in the formatted text sequence, generate a multi-granularity semantic vector; The domain semantic vector and the multi-granularity semantic vector are stored in a vector database, and a multi-level index structure is constructed to obtain the dense semantic vector memory.
6. The method according to claim 4, characterized in that, The domain knowledge graph is constructed in the following way: Multiple core entities are extracted from the historical building construction specifications and the formatted text sequence. The historical building construction specifications record historical building construction terminology and form specifications. The core entities include at least building entities and component entities. A relationship network is constructed based on the associations between multiple core entities, wherein the associations include at least entity inclusion relationships and size relationships; The core entities and the relationship network are stored in a graph database to form the domain knowledge graph.
7. The method according to claim 1, characterized in that, The step of fusing the fine-grained query request, the historical memory data, and the historical graph data in the target model to generate an enhanced feature representation includes: The fine query request, the historical memory data, and the historical graph data are respectively embedded and then concatenated in the target model to obtain a feature fusion representation. In the cross-attention layer at the bottom of the target model, cross-attention calculation is performed on the feature fusion representation to obtain the enhanced feature representation.
8. The method according to claim 1, characterized in that, The step of inputting concept metadata associated with the historical memory data and the historical atlas data into the concept routing mechanism to call routing units corresponding to the number of routing activations of the concept metadata, performing concept-level reasoning on the enhanced feature representation, and obtaining the reasoning feature representation includes: By using a concept bridge, the enhanced feature representation and the concept metadata are fused and projected to obtain a concept feature representation that the concept routing mechanism can process; Based on the data attributes indicated by the concept metadata, the number of route activations is determined, wherein the number of route activations is used to control the number of route units participating in concept-level reasoning; Select multiple target routing units corresponding to the number of activated routes from multiple candidate routing units; The conceptual feature representation is input into each of the target routing units for parallel computation, and the computation results output by each target routing unit are weighted and fused to obtain the inference feature representation.
9. The method according to claim 1, characterized in that, The method of using a pointer network, with the inference feature representation as a constraint, to control the target model to extract target memory data from the historical memory data in order to generate a natural language response to the document query request includes: The inference feature representation is input into the output layer of the target model as a constraint condition. During the process of the target model generating the natural language response based on the constraints, if it is determined that the next output word belongs to a preset category, the pointer network is triggered, wherein the preset category includes numerical category and term category; Using the pointer network, based on the inference feature representation and the currently generated partial response content, the corresponding original text data is located and copied from the historical memory data as the target memory data; The target model is controlled to output the target memory data as the next output word to continue generating the natural language response until the natural language response generation is complete.
10. A device for querying historical building construction records, characterized in that, The device includes: The request segmentation module is used to respond to a document query request for the construction archives of historical buildings, perform multimodal perception on the document query request, perform conceptual segmentation on the document query request, and generate a refined query request. The history determination module is used to perform searches in the dense semantic vector memory and the domain knowledge graph based on the fine query request to determine the historical memory data and the historical graph data, wherein the dense semantic vector memory and the domain knowledge graph are constructed based on the formatted text sequence obtained by recognizing and semantically aligning the historical building construction archives; The data fusion module is used to fuse the fine query request, the historical memory data, and the historical graph data in the target model to generate an enhanced feature representation; The concept reasoning module is used to input the concept metadata associated with the historical memory data and the historical map data into the concept routing mechanism, so as to call the routing units corresponding to the number of routing activations of the concept metadata, perform concept-level reasoning on the enhanced feature representation, and obtain the reasoning feature representation; The response generation module is used to control the target model to extract target memory data from the historical memory data using a pointer network and the inference feature representation as constraints, so as to generate a natural language response to the document query request.
