Multi-modal question and answer implementation method and device for heterogeneous historical and cultural heritage data

Abstract

The invention discloses a multi-modal question and answer implementation method and device for heterogeneous historical and cultural heritage data, and relates to the field of historical and cultural heritage data processing.The method comprises the steps that a multi-modal knowledge base is formed, and conversion from unstructured text data to standardized Markdown is achieved; hierarchical annotation of image semantics is achieved, focusing enhancement of professional terms is achieved, and therefore annotation of image data is achieved. A special large language model suitable for the field of historical and cultural heritage is constructed; the structured text data and the converted unstructured text data are mapped to a multi-dimensional continuous vector space, and construction of a multi-modal knowledge base index is achieved; a retrieval enhancement generation technology and a special large language model are adopted, based on a multi-modal knowledge base and a multi-modal knowledge base index, knowledge is obtained through a double-stage retrieval mechanism according to user questions, and intelligent question answering is achieved. According to the method, the professionality, the credibility and the practicability in a complex historical research scene can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of historical and cultural heritage data processing, specifically to a multimodal question-answering method and apparatus for heterogeneous historical and cultural heritage data. Background Technology

[0002] In the field of historical and cultural heritage, the current mainstream technology system mainly relies on the traditional information retrieval architecture. Its core implementation can be divided into two technical levels: First, there is the library automation management system based on MARC 21, which adopts an indexing mechanism that combines structured query based on Boolean algebra with a controlled thesaurus. Its retrieval model is based on an exact string matching algorithm, resulting in significant defects in semantic indexing capabilities. Second, there is the electronic document database system based on the Dublin Core metadata specification. Although it has achieved metadata extension through extended markup language, its underlying layer still adopts a relational algebra query optimization strategy, which cannot achieve cross-modal semantic vector space mapping.

[0003] It is evident that the aforementioned technical system suffers from two key flaws: firstly, the heterogeneity of feature distribution in multimodal data renders traditional distance measurement methods ineffective; secondly, the static retrieval architecture based on inverted indexes lacks a differentiable associative reasoning mechanism, severely restricting the dynamic ontology construction capabilities of knowledge graphs. These technical bottlenecks make it difficult to achieve deep semantic association and dynamic knowledge discovery of historical and cultural heritage data, particularly exhibiting significant limitations when handling cross-media and cross-temporal heterogeneous data associations.

[0004] Among emerging technologies, using large-scale pre-trained language models is the closest solution, but it suffers from two key technical constraints: first, model training requires massive computational resources, leading to excessively high costs for domain-adaptive optimization; second, there is a lack of specialized model architectures for the field of historical and cultural heritage. Although parameter fine-tuning can improve the model's semantic understanding of historical texts, its single-modal processing mechanism has inherent flaws, failing to support cross-modal feature alignment and multi-source data collaborative verification. These limitations directly impact the three core indicators in historical research scenarios: factual accuracy, information completeness, and multimodal relevance. Summary of the Invention

[0005] This application provides a multimodal question-answering method and apparatus for heterogeneous historical and cultural heritage data, which can effectively improve the professionalism, credibility and practicality in complex historical research scenarios.

[0006] Firstly, embodiments of this application provide a multimodal question-answering method for heterogeneous historical and cultural heritage data, the method comprising: We collect structured text data, unstructured text data, and image data related to historical and cultural heritage to form a multimodal knowledge base. Based on the constructed multimodal document parsing engine, we realize the conversion of unstructured text data into standardized Markdown through a hierarchical parsing architecture. A multimodal joint annotation framework based on deep learning is used to achieve hierarchical annotation of image semantics, and hierarchical attention fusion is used to enhance the focus of technical terms, thereby realizing the annotation of image data; By modifying the basic model using a low-rank adaptive method and combining it with multi-source professional data, a specialized large language model suitable for the field of historical and cultural heritage is constructed. By using a pre-trained cross-language text embedding model, structured text data and transformed unstructured text data are mapped to a multi-dimensional continuous vector space, thereby enabling the construction of a multi-modal knowledge base index. It employs retrieval-enhanced generation technology and a dedicated large language model, based on a multimodal knowledge base and a multimodal knowledge base index, to acquire knowledge through a two-stage retrieval mechanism according to the user's question, thereby achieving intelligent question answering.

[0007] In conjunction with the first aspect, in one implementation, the collection of structured text data, unstructured text data, and image data related to historical and cultural heritage to form a multimodal knowledge base specifically includes: We collect official archives, documents, archaeological excavation reports, and authoritative academic publications related to historical and cultural heritage to obtain structured and unstructured text data. Collect high-resolution digital image data related to historical and cultural heritage; The collected data is standardized, and a multimodal knowledge base is obtained based on the standardized data.

[0008] In conjunction with the first aspect, in one implementation method, The multimodal document parsing engine employs a triple processing pipeline. First, it performs preliminary structured segmentation of the text using a rule-based regular expression matching layer. Then, it uses a conditional random field model to identify semantic blocks. Finally, it reconstructs the text's logical structure using an adaptive layout analysis algorithm. In the process of converting unstructured text data to standardized Markdown, an abstract syntax tree converter maps the original text to a markup language that conforms to the Markdown specification, while preserving the XPath positioning information of the original text.

[0009] In conjunction with the first aspect, in one implementation method, The multimodal document parsing engine also has a built-in difference verification module; The difference verification module is used to ensure that the output results retain table nesting and mathematical formulas.

[0010] In conjunction with the first aspect, in one implementation, the hierarchical annotation of image semantics is achieved using a deep learning-based multimodal joint annotation framework, and the focus enhancement of technical terms is achieved using hierarchical attention fusion, thereby realizing the annotation of image data, specifically including: Visual features are extracted using an object detection architecture, and a multi-scale feature fusion network is constructed using feature suppression algorithms and attention mechanisms. A cross-modal contrastive learning model is used to map visual features to a semantic space, generating preliminary semantic annotation results, i.e., annotated text; Low-confidence labeled texts are manually verified using a confidence screening algorithm and standardized using a professional terminology labeling system. By performing semantic role analysis and relationship extraction on the labeled text using a pre-trained language model, an image-text alignment vector space that conforms to the standards for historical and cultural heritage data is constructed, thereby enabling the labeling of image data.

[0011] In conjunction with the first aspect, in one implementation, the modification of the basic model using a low-rank adaptive method and the integration of multi-source professional data to construct a specialized large language model suitable for the field of historical and cultural heritage specifically includes: By modifying the basic large language model using a low-rank adaptive method and combining it with multi-source professional data, the basic large language model is optimized in a targeted manner for the identification of cultural relic features, dating, and generation of protection strategies. Thus, a special large language model suitable for the field of historical and cultural heritage is constructed based on the parameter fine-tuning technical route.

[0012] In conjunction with the first aspect, in one implementation, the multi-source professional data includes historical document corpora, cultural relic archive datasets, and conservation and restoration records.

[0013] In conjunction with the first aspect, in one implementation method, The process involves mapping structured text data and transformed unstructured text data to a multidimensional continuous vector space, where semantic similarity is measured by cosine distance. The index architecture of the multimodal knowledge base index adopts a hierarchical design. First, the candidate set is initially screened through an approximate nearest neighbor search algorithm, and then the results are finely ranked by a re-ranking module based on an attention mechanism.

[0014] In conjunction with the first aspect, in one implementation method, The knowledge acquisition based on user questions through a two-stage retrieval mechanism specifically involves: performing an approximate nearest neighbor search using a high-dimensional semantic embedding space, followed by optimizing the relevance of the retrieval results through an attention-based reordering module; In the result generation stage of intelligent question answering, the filtered multimodal context is input into the domain-adapted generative model, and the beam search algorithm and kernel sampling technique are used.

[0015] Secondly, embodiments of this application provide a multimodal question-answering implementation device for heterogeneous historical and cultural heritage data, the multimodal question-answering implementation device for heterogeneous historical and cultural heritage data comprising: The knowledge base construction module is used to collect structured text data, unstructured text data, and image data related to historical and cultural heritage to form a multimodal knowledge base. Based on the constructed multimodal document parsing engine, it realizes the conversion of unstructured text data into standardized Markdown through a hierarchical parsing architecture. The image data annotation module is used to achieve hierarchical annotation of image semantics using a deep learning-based multimodal joint annotation framework, and to enhance the focus of technical terms using hierarchical attention fusion, thereby achieving the annotation of image data. A dedicated model building module is used to modify the basic model through a low-rank adaptive method and combine it with multi-source professional data to build a dedicated large language model suitable for the field of historical and cultural heritage. The knowledge base index building module is used to map structured text data and transformed unstructured text data to a multi-dimensional continuous vector space through a pre-trained cross-language text embedding model, thereby realizing the construction of a multi-modal knowledge base index. The intelligent question answering module employs retrieval enhancement generation technology and a dedicated large language model. Based on a multimodal knowledge base and its index, it acquires knowledge through a two-stage retrieval mechanism according to the user's question, thereby achieving intelligent question answering.

[0016] The beneficial effects of the technical solutions provided in this application include: (1) By innovatively integrating cross-modal representation learning and dynamic knowledge routing mechanism, the relevant technical bottlenecks have been effectively overcome. The parameter-efficient fine-tuning technology is used to adapt the large language model to the domain, which significantly reduces the consumption of computing resources while maintaining the model's semantic understanding ability of historical texts. At the same time, by integrating the structured domain knowledge base, a professional knowledge enhancement system containing professional terms and historical event associations is constructed, which makes up for the knowledge limitations of general pre-trained large language models in the field of cultural heritage. (2) While maintaining the authenticity of historical data, this application constructs a multimodal unified representation space for cultural heritage, develops a multi-source data verification algorithm based on attention mechanism, and designs an interpretable knowledge retrieval and generation pipeline. This technical approach not only optimizes the model training cost, but also significantly improves the professionalism, credibility and practicality of the system in complex historical research scenarios through the deep integration of knowledge base. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the multimodal question-answering implementation method for heterogeneous historical and cultural heritage data in this application; Figure 2 This is a flowchart of the data acquisition process. Figure 3 A flowchart of document preprocessing; Figure 4 Flowchart for visual data annotation; Figure 5 Flowchart for building a dedicated large language model; Figure 6 Flowchart for building a multimodal knowledge base index; Figure 7 Flowchart for intelligent question answering implementation; Figure 8 This is a schematic diagram of the functional modules of the multimodal question-answering device for heterogeneous historical and cultural heritage data in this application. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0020] In one aspect, embodiments of this application provide a multimodal question-answering method for heterogeneous historical and cultural heritage data.

[0021] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating the multimodal question-answering implementation method for heterogeneous historical and cultural heritage data in this application. Figure 1 As shown, the multimodal question-answering implementation methods for heterogeneous historical and cultural heritage data include: S1: Collect structured text data, unstructured text data, and image data related to historical and cultural heritage to form a multimodal knowledge base. Based on the constructed multimodal document parsing engine, the unstructured text data is converted into standardized Markdown (a lightweight markup language) through a hierarchical parsing architecture. S2: A multimodal joint annotation framework based on deep learning is used to achieve hierarchical annotation of image semantics, and hierarchical attention fusion is used to enhance the focus of professional terms, thereby realizing the annotation of image data; S3: By modifying the basic model using a low-rank adaptive method and combining it with multi-source professional data, a specialized large language model suitable for the field of historical and cultural heritage is constructed. S4: By using a pre-trained cross-language text embedding model, structured text data and transformed unstructured text data are mapped to a multi-dimensional continuous vector space to build a multi-modal knowledge base index. S5: Employs retrieval enhancement generation technology and a dedicated large language model, based on a multimodal knowledge base and a multimodal knowledge base index, and acquires knowledge through a two-stage retrieval mechanism according to the user's question to achieve intelligent question answering.

[0022] Furthermore, in one embodiment, a multimodal knowledge base is formed by collecting structured text data, unstructured text data, and image data related to historical and cultural heritage, specifically including: S101: Collect official archives, documents, archaeological excavation reports, and authoritative academic publications related to historical and cultural heritage to obtain structured and unstructured text data; S102: Acquire high-resolution digital image data related to historical and cultural heritage; S103: Standardize the collected data and obtain a multimodal knowledge base based on the standardized data.

[0023] For details, see Figure 2 As shown, a comprehensive and authoritative knowledge base system, namely a multimodal knowledge base, is constructed based on multi-source heterogeneous data in the field of historical and cultural heritage. During the data acquisition phase, structured and unstructured text data, including official archives, archaeological excavation reports, and authoritative academic publications, were integrated, along with high-resolution digital image data. All raw data underwent standardized preprocessing to ensure compliance with international standards and specifications. Experimental results show that by integrating authoritative published data sources verified by domain experts, the accuracy of historical and cultural heritage knowledge representation was improved by over 80% during intelligent question answering.

[0024] Furthermore, in one embodiment, the multimodal document parsing engine employs a three-stage processing pipeline. First, it performs preliminary structured segmentation of the text using a rule-based regular expression matching layer. Then, it uses a conditional random field model for semantic block recognition. Finally, it reconstructs the text's logical structure using an adaptive layout analysis algorithm. During the conversion of unstructured text data to standardized Markdown, an abstract syntax tree converter maps the original text to a markup language conforming to Markdown specifications, while retaining the original text's XPath positioning information.

[0025] For details, see Figure 3 As shown, the document preprocessing stage employs a multimodal document parsing engine, which uses a hierarchical parsing architecture to convert unstructured text data into standardized Markdown. The multimodal document parsing engine uses a three-stage processing pipeline: first, a rule-based regular expression matching layer performs preliminary structured segmentation of the text; then, an improved Conditional Random Field (CRF) model is used for semantic block recognition; and finally, an adaptive layout analysis algorithm reconstructs the text's logical structure. During the conversion process, an Abstract Syntax Tree (AST) transformer maps the original document elements to a markup language conforming to Markdown specifications, while preserving the precise XPath positioning information of the source document.

[0026] Furthermore, the multimodal document parsing engine also has a built-in discrepancy verification module; this module ensures that the output retains nested tables and mathematical formulas. In other words, the built-in discrepancy verification module of the multimodal document parsing engine ensures that the output retains complex structures such as nested tables and mathematical formulas (LaTeX compatible).

[0027] All processing steps achieve non-destructive editing through versioned intermediate representations, supporting end-to-end traceability of the conversion process via content-addressed hashing (based on SHA-256). Experimental data shows that, by employing a three-tiered processing pipeline architecture with a multimodal document parsing engine, this step achieves a 98.7% accuracy rate in the standardized conversion of unstructured historical documents, an improvement of over 10 percentage points compared to traditional methods, and an improvement of over 30 percentage points compared to traditional printed versions of ancient books. Sampling evaluations by experts in the field of cultural heritage have shown that the conversion results perform excellently in maintaining the key indicator of the semantic integrity of the original documents.

[0028] Furthermore, in one embodiment, a multimodal joint annotation framework based on deep learning is used to achieve hierarchical annotation of image semantics, and hierarchical attention fusion is used to enhance the focus of technical terms, thereby achieving the annotation of image data, specifically including: S201: Visual feature extraction is performed through an object detection architecture, and a multi-scale feature fusion network is constructed through feature suppression algorithms and attention mechanisms; S202: A cross-modal contrastive learning model is used to map visual features to the semantic space, generating preliminary semantic annotation results, i.e., annotated text; S203: Low-confidence labeled texts are manually verified using a confidence screening algorithm and standardized using a professional terminology labeling system; S204: By using a pre-trained language model to perform semantic role analysis and relationship extraction on the labeled text, a text-image alignment vector space that conforms to the standards for historical and cultural heritage data is constructed, thereby achieving the labeling of image data.

[0029] For details, see Figure 4 As shown, in the visual data annotation stage, a multimodal joint annotation framework based on deep learning is adopted to achieve hierarchical annotation of image semantics, and hierarchical attention fusion is used to enhance the focus of professional terms. First, an improved object detection architecture is deployed to extract visual features. A multi-scale feature fusion network is constructed through feature suppression algorithms and attention mechanisms. Then, a cross-modal contrastive learning model is used to map visual features to the semantic space to generate preliminary semantic annotation results. In the quality control stage, an active learning mechanism supervised by domain experts is adopted. Low-confidence samples are manually verified through a confidence screening algorithm, and a professional terminology annotation system is used for standardization. Finally, a pre-trained language model is used to perform semantic role analysis and relation extraction on the annotated text to construct an image-text alignment vector space that conforms to the standards of the cultural heritage field.

[0030] The entire annotation process employs containerization technology to isolate computational resources and is equipped with a real-time performance monitoring system to ensure that the annotation results meet predetermined quality standards. Experimental verification shows that this multimodal joint annotation framework achieves a semantic annotation accuracy of 72.4% in cultural heritage image annotation tasks, an improvement of 18.3% compared to traditional single-modal methods. The final generated annotation data achieves leading levels in two key indicators: cultural heritage terminology coverage and semantic consistency, at 95.2% and 93.8%, respectively.

[0031] Furthermore, in one embodiment, a specialized large language model suitable for the field of historical and cultural heritage is constructed by modifying the basic model using a low-rank adaptive method and combining it with multi-source professional data, specifically including: A low-rank adaptive method is used to modify a basic large-scale language model (i.e., a general-purpose artificial intelligence large-scale model that has not been trained and configured for a specific field). Combined with multi-source professional data, the basic large-scale language model is optimized in a targeted manner for cultural relic feature recognition, dating, and conservation strategy generation. Based on parameter fine-tuning techniques, a specialized large-scale language model suitable for the field of historical and cultural heritage is constructed. The multi-source professional data includes historical document corpora, cultural relic archive datasets, and conservation and restoration records.

[0032] For details, see Figure 5 As shown, in the domain-specific model construction stage, a technical approach based on efficient parameter fine-tuning is adopted to achieve vertical domain adaptation of the large language model. The basic model is professionally modified through a low-rank adaptive method, combining multi-source professional data such as historical document corpora, cultural relic archive datasets, and conservation and restoration records, to perform targeted optimization for core tasks such as cultural relic feature recognition, dating, and conservation strategy generation. This technical solution significantly improves the model's semantic understanding depth and professional knowledge reasoning ability in the cultural heritage field. The generated text content strictly adheres to academic research norms and archaeological practice standards, meeting professional application requirements in key indicators such as terminology accuracy, logical rigor, and domain adaptability.

[0033] The entire fine-tuning process employs techniques such as gradient accumulation and mixed-precision training, effectively controlling computational resource consumption while ensuring improved model performance. Experimental data shows that the domain-specific large language model constructed using the parameter-efficient fine-tuning scheme performs exceptionally well in cultural heritage-related tasks: improving the accuracy of artifact feature recognition by 31.5%. Blind testing by domain experts revealed that the model output significantly improves upon general-purpose large-scale artificial intelligence models in three core metrics: accuracy of professional terminology (95.2%), logical rigor (93.1%), and domain adaptability (90.8%). Furthermore, the training process requires only 18% of the original computational resources, achieving an optimal balance between performance and efficiency.

[0034] Furthermore, in one embodiment, structured text data and transformed unstructured text data are mapped to a multidimensional continuous vector space, wherein semantic similarity is measured by cosine distance; the index architecture of the multimodal knowledge base index adopts a hierarchical design, firstly screening the candidate set through an approximate nearest neighbor search algorithm, and then re-ranking the results based on an attention mechanism re-ranking module.

[0035] For details, see Figure 6 As shown, in the multimodal knowledge base indexing construction phase, a high-dimensional semantic embedding technique based on deep neural networks is adopted to achieve a unified vectorized representation of heterogeneous data. Through a pre-trained cross-language text embedding model, preprocessed structured and unstructured data are mapped to a 4096-dimensional continuous vector space, where semantic similarity is measured by cosine distance. The index architecture adopts a hierarchical design, firstly using an approximate nearest neighbor search algorithm for initial screening of the candidate set, and then using an attention-based re-ranking module for fine-tuning the results. This scheme significantly improves retrieval efficiency in large-scale knowledge base scenarios, with significant improvements in core metrics such as recall and precision. The vector index uses quantization compression technology to optimize storage efficiency, while supporting an incremental update mechanism to ensure the timeliness of the multimodal knowledge base. Furthermore, the multimodal knowledge base can be horizontally scaled through a distributed computing framework, supporting real-time retrieval needs for billions of vectors.

[0036] Furthermore, in one embodiment, knowledge acquisition is performed through a two-stage retrieval mechanism based on the user's question. Specifically, this involves: performing an approximate nearest neighbor search using a high-dimensional semantic embedding space, followed by optimizing the relevance of the retrieval results through a re-ranking module based on an attention mechanism; and in the intelligent question answering result generation stage, inputting the filtered multimodal context into a domain-adaptive generative model, and employing a beam search algorithm and kernel sampling technology.

[0037] For details, see Figure 7 As shown, in the intelligent question-answering implementation stage, a multimodal interaction system is constructed using a retrieval-enhanced generation technology framework. The processing path is automatically selected based on the user's question type (classification questions such as grotto temples and stone carvings, ancient tombs, ancient murals, and ancient architecture identification). Based on a locally deployed dedicated large language model and a fine-tuned version of the multimodal understanding model, knowledge acquisition is achieved through a two-stage retrieval mechanism: first, an approximate nearest neighbor search is performed using a high-dimensional semantic embedding space; then, the relevance of the retrieval results is optimized through a re-ranking module based on an attention mechanism. In the generation stage, the filtered multimodal context (including textual, visual, and other multi-source information) is input into a domain-adapted generation model. A bundle search algorithm and kernel sampling technique are used to ensure the accuracy and diversity of the output results.

[0038] To achieve full-modal interaction capabilities, intelligent question-answering systems can also integrate an automatic speech recognition module to convert speech input into text representation and achieve speech feedback through speech synthesis technology.

[0039] The entire processing flow adopts a microservice architecture for modular deployment. Components exchange data via high-speed message queues, ensuring the intelligent question-answering system meets real-time interaction requirements in terms of key performance indicators such as throughput and response latency. The knowledge update mechanism employs incremental indexing technology, supporting dynamic loading of new data and index reconstruction to guarantee the timeliness of knowledge within the intelligent question-answering system.

[0040] Experimental results show that this enhanced retrieval generation method achieves significant results in the intelligent question-answering task of cultural heritage: on the cultural heritage test set, the multimodal retrieval accuracy reaches over 90%, and the professionalism score of the generated answers reaches over 90%, which is more than 30% higher than that of general artificial intelligence models on the market. Furthermore, the knowledge supports dynamic updates, enabling real-time updates for newly emerging professional concepts, effectively ensuring the timeliness of knowledge services.

[0041] Compared to traditional single-modal retrieval architectures and general language models based on keyword matching, this application proposes a multimodal retrieval enhancement and generation framework. In the field of historical and cultural heritage, it achieves multi-dimensional performance breakthroughs through heterogeneous data fusion and domain-adaptive optimization. At the knowledge representation level, the constructed cross-modal unified semantic space (d=4096) uses contrastive learning loss to optimize feature alignment, and combined with a domain-specific embedding index, it improves Hamming distance retrieval efficiency by 3.2 times (based on 10,000 test queries). At the model optimization level, the efficient parameter fine-tuning strategy reduces the domain adaptation cost by more than 90%, and the F1 score for professional terminology understanding reaches 0.91 (on a test set containing 5,000 professional terms). At the performance level, it achieves end-to-end query response latency of <350ms (for a knowledge base of tens of millions of terms) and improves the accuracy of historical facts to 95.3% (based on 10,000 test queries). Its technological breakthroughs are reflected in core innovations such as multi-granularity attention fusion mechanism, dynamic negative sampling retrieval optimization algorithm and knowledge-enhanced bundle search decoding strategy. After A / B testing comparison with traditional retrieval systems, all performance improvements passed the significance test.

[0042] Secondly, embodiments of this application also provide a multimodal question-answering device for heterogeneous historical and cultural heritage data.

[0043] In one embodiment, reference is made to Figure 8 , Figure 8 This is a schematic diagram of the functional modules of the multimodal question-answering device for heterogeneous historical and cultural heritage data, as described in this application. Figure 8 As shown, the multimodal question-answering device for heterogeneous historical and cultural heritage data includes: a knowledge base construction module, an image data annotation module, a dedicated model construction module, a knowledge base index construction module, and an intelligent question-answering implementation module.

[0044] The knowledge base construction module collects structured, unstructured, and image data related to historical and cultural heritage to form a multimodal knowledge base. Based on the constructed multimodal document parsing engine, it converts unstructured text data into standardized Markdown through a hierarchical parsing architecture. The image data annotation module uses a deep learning-based multimodal joint annotation framework to achieve hierarchical annotation of image semantics and employs hierarchical attention fusion to enhance the focus of professional terms, thereby achieving image data annotation. The dedicated model construction module modifies the basic model using a low-rank adaptive method and combines it with multi-source professional data to construct a dedicated large language model suitable for the field of historical and cultural heritage. The knowledge base index construction module uses a pre-trained cross-language text embedding model to map structured text data and transformed unstructured text data to a multi-dimensional continuous vector space, realizing the construction of a multimodal knowledge base index. The intelligent question answering implementation module uses retrieval enhancement generation technology and a dedicated large language model. Based on the multimodal knowledge base and the multimodal knowledge base index, it acquires knowledge through a two-stage retrieval mechanism according to the user's question to achieve intelligent question answering.

[0045] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0046] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0047] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0048] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0049] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0050] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A multimodal question-answering method for heterogeneous historical and cultural heritage data, characterized in that, The multimodal question-answering implementation method for heterogeneous historical and cultural heritage data includes: We collect structured text data, unstructured text data, and image data related to historical and cultural heritage to form a multimodal knowledge base. Based on the constructed multimodal document parsing engine, we realize the conversion of unstructured text data into standardized Markdown through a hierarchical parsing architecture. A multimodal joint annotation framework based on deep learning is used to achieve hierarchical annotation of image semantics, and hierarchical attention fusion is used to enhance the focus of technical terms, thereby realizing the annotation of image data; By modifying the basic model using a low-rank adaptive method and combining it with multi-source professional data, a specialized large language model suitable for the field of historical and cultural heritage is constructed. By using a pre-trained cross-language text embedding model, structured text data and transformed unstructured text data are mapped to a multi-dimensional continuous vector space, thereby enabling the construction of a multi-modal knowledge base index. It employs retrieval-enhanced generation technology and a dedicated large language model, based on a multimodal knowledge base and a multimodal knowledge base index, to acquire knowledge through a two-stage retrieval mechanism according to the user's question, thereby achieving intelligent question answering.

2. The multimodal question-answering method for heterogeneous historical and cultural heritage data as described in claim 1, characterized in that, The collection of structured text data, unstructured text data, and image data related to historical and cultural heritage forms a multimodal knowledge base, specifically including: We collect official archives, documents, archaeological excavation reports, and authoritative academic publications related to historical and cultural heritage to obtain structured and unstructured text data. Collect high-resolution digital image data related to historical and cultural heritage; The collected data is standardized, and a multimodal knowledge base is obtained based on the standardized data.

3. The multimodal question-answering method for heterogeneous historical and cultural heritage data as described in claim 1, characterized in that: The multimodal document parsing engine employs a triple processing pipeline. First, it performs preliminary structured segmentation of the text using a rule-based regular expression matching layer. Then, it uses a conditional random field model to identify semantic blocks. Finally, it reconstructs the text's logical structure using an adaptive layout analysis algorithm. In the process of converting unstructured text data to standardized Markdown, an abstract syntax tree converter maps the original text to a markup language that conforms to the Markdown specification, while preserving the XPath positioning information of the original text.

4. The multimodal question-answering method for heterogeneous historical and cultural heritage data as described in claim 3, characterized in that: The multimodal document parsing engine also has a built-in difference verification module; The difference verification module is used to ensure that the output results retain table nesting and mathematical formulas.

5. The multimodal question-answering method for heterogeneous historical and cultural heritage data as described in claim 1, characterized in that, The method employs a deep learning-based multimodal joint annotation framework to achieve hierarchical annotation of image semantics, and uses hierarchical attention fusion to enhance the focus of technical terms, thereby achieving image data annotation. Specifically, this includes: Visual features are extracted using an object detection architecture, and a multi-scale feature fusion network is constructed using feature suppression algorithms and attention mechanisms. A cross-modal contrastive learning model is used to map visual features to a semantic space, generating preliminary semantic annotation results, i.e., annotated text; Low-confidence labeled texts are manually verified using a confidence screening algorithm and standardized using a professional terminology labeling system. By performing semantic role analysis and relationship extraction on the labeled text using a pre-trained language model, an image-text alignment vector space that conforms to the standards for historical and cultural heritage data is constructed, thereby enabling the labeling of image data.

6. The multimodal question-answering method for heterogeneous historical and cultural heritage data as described in claim 1, characterized in that, The process involves modifying the basic model using a low-rank adaptive method and combining it with multi-source professional data to construct a specialized large language model suitable for the field of historical and cultural heritage. Specifically, this includes: By modifying the basic large language model using a low-rank adaptive method and combining it with multi-source professional data, the basic large language model is optimized in a targeted manner for the identification of cultural relic features, dating, and generation of protection strategies. Thus, a special large language model suitable for the field of historical and cultural heritage is constructed based on the parameter fine-tuning technical route.

7. The multimodal question-answering method for heterogeneous historical and cultural heritage data as described in claim 6, characterized in that: The multi-source professional data includes historical document corpora, cultural relic archive datasets, and conservation and restoration records.

8. The multimodal question-answering method for heterogeneous historical and cultural heritage data as described in claim 1, characterized in that: The process involves mapping structured text data and transformed unstructured text data to a multidimensional continuous vector space, where semantic similarity is measured by cosine distance. The index architecture of the multimodal knowledge base index adopts a hierarchical design. First, the candidate set is initially screened through an approximate nearest neighbor search algorithm, and then the results are finely ranked by a re-ranking module based on an attention mechanism.

9. The multimodal question-answering method for heterogeneous historical and cultural heritage data as described in claim 1, characterized in that: The knowledge acquisition based on user questions through a two-stage retrieval mechanism specifically involves: performing an approximate nearest neighbor search using a high-dimensional semantic embedding space, followed by optimizing the relevance of the retrieval results through an attention-based reordering module; In the result generation stage of intelligent question answering, the filtered multimodal context is input into the domain-adapted generative model, and the beam search algorithm and kernel sampling technique are used.

10. A multimodal question-answering device for heterogeneous historical and cultural heritage data, characterized in that, The multimodal question-answering device for heterogeneous historical and cultural heritage data includes: The knowledge base construction module is used to collect structured text data, unstructured text data, and image data related to historical and cultural heritage to form a multimodal knowledge base. Based on the constructed multimodal document parsing engine, it realizes the conversion of unstructured text data into standardized Markdown through a hierarchical parsing architecture. The image data annotation module is used to achieve hierarchical annotation of image semantics using a deep learning-based multimodal joint annotation framework, and to enhance the focus of technical terms using hierarchical attention fusion, thereby achieving the annotation of image data. A dedicated model building module is used to modify the basic model through a low-rank adaptive method and combine it with multi-source professional data to build a dedicated large language model suitable for the field of historical and cultural heritage. The knowledge base index building module is used to map structured text data and transformed unstructured text data to a multi-dimensional continuous vector space through a pre-trained cross-language text embedding model, thereby realizing the construction of a multi-modal knowledge base index. The intelligent question answering module employs retrieval enhancement generation technology and a dedicated large language model. Based on a multimodal knowledge base and its index, it acquires knowledge through a two-stage retrieval mechanism according to the user's question, thereby achieving intelligent question answering.

Images