Cultural heritage digitization method based on multi-modal fusion
By integrating cultural heritage information through multimodal fusion, the problems of low efficiency and narrow audience in traditional methods have been solved, achieving efficient and systematic digitization of cultural heritage, generating vivid multimedia content, and expanding audience reach and dissemination efficiency.
Patent Information
- Application Number
- CN202511511992.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-22
- Publication Date
- 2026-02-13
AI Technical Summary
Existing technologies are insufficient for efficiently integrating and disseminating cultural heritage information. Traditional methods are inefficient, have limited information dimensions, and narrow audience reach, making it impossible to deeply understand and systematically showcase the complex relationships within cultural heritage.
By employing a multimodal fusion approach, through ancient text processing, multimodal information integration, knowledge graph construction, AIGC narrative visualization, and multi-terminal interactive presentation, we achieve semantic alignment and deep association between text entities and multimedia data, generate vivid multimedia content, and construct a systematic knowledge network.
It has improved the efficiency of cultural heritage information integration and dissemination appeal, expanded the audience coverage, achieved accurate correlation and mutual verification of cross-modal information, generated high-quality and historically accurate multimedia content, and broken the limitations of time and space.
Smart Images
Figure CN121524907A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of cultural heritage digitization and artificial intelligence technology, specifically a method for digitizing cultural heritage based on multimodal fusion. Background Technology
[0002] In recent years, the national government has vigorously promoted the digitalization strategy for culture, successively issuing a series of important policy documents such as the "Opinions on Promoting the Implementation of the National Digitalization Strategy for Culture," which clearly require the full utilization of digital technologies to revitalize the vitality of outstanding traditional Chinese culture. Cultural heritage, as an irreplaceable treasure of human civilization, is of paramount importance for its protection and transmission. However, with the passage of time and changes in the natural environment, a large number of precious cultural heritages, especially intangible cultural heritage represented by ancient books and documents, are facing problems such as broken transmission chains, limited dissemination, and disconnection from modern society.
[0003] Traditional methods of cultural heritage protection and transmission mainly rely on physical exhibitions, guided tours, academic research, and print publications. These methods have inherent limitations: First, they are inefficient, as manual recording and organization are time-consuming and labor-intensive, making it difficult to handle the vast amount of cultural heritage resources. Second, information dissemination is not intuitive; static exhibits and professional academic language cannot vividly and comprehensively showcase the historical context, processes, and cultural narratives behind cultural heritage, making it difficult for the public to understand. Third, the audience reach is narrow, as the influence is limited by geographical location and opening hours, making it difficult to transcend the limitations of physical space.
[0004] To address the aforementioned issues, digital technologies have been widely applied in the cultural field. Existing technological solutions mainly focus on the following aspects:
[0005] (1) Single-modal digital archiving: This is currently the most common approach. For example, ancient books, paintings, and cultural relics are digitized through high-precision scanning or photography to generate electronic images or PDF files; or, ancient documents are converted into editable electronic text using optical character recognition (OCR) technology. However, such technical solutions usually stop at "digital archiving," and the result is merely a digital copy of the physical object, with a single information dimension, failing to conduct in-depth understanding and structured processing of the content.
[0006] (2) Isolated database construction: Some institutions have established databases of cultural relics or ancient books, but these databases are mostly based on traditional relational databases, mainly recording basic attribute information (metadata) such as the name, age, and material of the cultural relics. The internal data correlation is weak, like isolated information islands, which cannot effectively reveal the complex spatiotemporal, social and cultural connections between cultural heritage elements, and it is difficult to form a systematic knowledge network.
[0007] (3) Preliminary multimedia presentations: Some museums or websites provide simple pictures, text descriptions, or short videos for digital cultural relics. However, these multimedia contents are usually pre-made and fixed, lacking machine-understandable semantic connections with the core textual knowledge (such as ancient records), and the content generation process has not been automated or intelligent.
[0008] No solutions have yet been proposed for the relevant technical issues. Summary of the Invention
[0009] To address the problems in related technologies, this invention proposes a multimodal fusion-based method for digitizing cultural heritage, overcoming the aforementioned technical issues in existing technologies. The purpose of this invention is to achieve efficient integration of multimodal information about cultural heritage. Compared to traditional single-modal digitization, it offers richer information dimensions, laying the foundation for subsequent knowledge mining and dissemination. The constructed knowledge graph can uncover deep connections within cultural heritage, making the cultural heritage knowledge system more systematic and logical, thus helping researchers and the public to gain a deeper understanding of cultural heritage. AIGC technology transforms abstract narratives into vivid multimedia content, enhancing the visual impact and appeal of cultural heritage, making it easier for the public to accept and appreciate. Multi-terminal interactive presentation expands the audience reach of cultural heritage, enabling it to reach more people on different platforms and in different scenarios, effectively promoting the inheritance and dissemination of cultural heritage.
[0010] To achieve the above objectives, the present invention provides the following technical solution: a method for digitizing cultural heritage based on multimodal fusion, comprising the following steps:
[0011] S1, Ancient Books and Documents Processing: Digitally identify ancient books and documents and extract key entities from them;
[0012] S2, Multimodal information integration: Acquire multimedia data related to the content of the ancient documents, and align and annotate the multimedia data with the key entities;
[0013] S3, Knowledge Graph Construction: Using the key entities as nodes and the relationships between entities as edges, construct a knowledge graph and mine the deep relationships between entities based on graph algorithms;
[0014] S4, AIGC Narrative Visualization: Extract narrative clues from the knowledge graph and multimodal data, and generate multimedia content based on the narrative clues using a multimodal AIGC model;
[0015] S5, multi-terminal interactive presentation: adapts and presents the generated multimedia content and knowledge graph data to multiple terminals.
[0016] Preferably, the ancient book document processing specifically includes:
[0017] (1) Construct a sample database of ancient books and documents;
[0018] (2) Use a customized OCR model to recognize ancient book images or texts to extract the text;
[0019] (3) The key entities are identified from the extracted text using a finely tuned large language model (LLM).
[0020] Preferably, in the multimodal information integration, the data alignment and annotation are achieved by mapping the text descriptions of the key entities and the multimedia data to the same semantic vector space through a multimodal embedding model, and by calculating the similarity between the semantic vectors.
[0021] Preferably, in the construction of the knowledge graph, the mining of deep associations is achieved by running at least one of the following algorithms on the knowledge graph: path discovery algorithm, community discovery algorithm, and centrality analysis algorithm.
[0022] Preferably, the AIGC narrative visualization specifically includes:
[0023] (1) Use LLM and graph algorithms to extract narrative clues from the knowledge graph and multimodal data;
[0024] (2) Input the narrative clues as prompt words into the multimodal AIGC model to generate image or video content; wherein, the style and historical accuracy of the generated content are controlled by prompt word engineering.
[0025] Preferably, the process of generating image content includes: based on the narrative clues, performing sketch construction, image editing, anomaly detection and region redrawing, and style transfer sequentially by repeatedly calling the image generation model and the visual evaluation model.
[0026] Preferably, the method employs a hybrid storage architecture for data storage, the hybrid storage architecture comprising:
[0027] Relational databases are used to store structured metadata;
[0028] A graph database is used to store the knowledge graph;
[0029] Object storage is used to store multimedia data;
[0030] A vector database used to store semantic vectors for data alignment.
[0031] Preferably, in the multi-terminal interactive presentation, the multi-terminal includes at least two of the following: website platform, mini-program, mobile application (APP), and digital media art installation.
[0032] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described above.
[0033] Compared with the prior art, the beneficial effects of the present invention are:
[0034] (1) This invention is a method for digitizing cultural heritage based on multimodal fusion. By setting up a multimodal information integration step, the multimodal embedding model is used to align text entities with multimedia data such as images and videos in the semantic vector space. This overcomes the drawback of the fragmentation of multimodal data in traditional methods, improves the efficiency and scale of data integration, and realizes accurate association and mutual verification of cross-modal information. This lays a solid data foundation for building a three-dimensional digital resource system for cultural heritage, far exceeding the effect of single-modal digitization or simple superposition.
[0035] (2) This invention is a digital method for cultural heritage based on multimodal fusion. By setting up knowledge graph construction steps, it organizes scattered entities and relationships into a structured knowledge network and uses graph algorithms such as path discovery and community discovery for automated mining. This enables the visualization and quantitative analysis of deep connections that are difficult to detect and rely on manual methods in traditional research. This elevates scattered cultural heritage knowledge into a systematic knowledge system with internal logic, which greatly assists academic research and the public's in-depth understanding.
[0036] (3) This invention is a digital method for cultural heritage based on multimodal fusion. By setting up an AIGC narrative visualization step, the narrative clues mined from the knowledge graph are combined with the multimodal AIGC model. This method can generate images, videos and other content in batches and automatically, which are both historically accurate and artistically expressive. This fundamentally solves the problems of abstract cultural heritage narratives and rigid dissemination forms. It transforms dry text descriptions into multimedia content with visual impact and emotional resonance, significantly improving the attractiveness and dissemination efficiency of cultural heritage.
[0037] (4) This invention is a digital method for cultural heritage based on multimodal fusion. When generating image content, it constructs a refined workflow that includes sketch construction, image editing, anomaly detection and region redrawing, and style transfer by calling the image generation and visual evaluation model multiple times. This can effectively control the randomness of AIGC output and ensure that the generated content is of high quality and strong consistency in detail, style and historical facts, making the final result more authentic and credible and meeting the high requirements of accuracy for cultural heritage dissemination.
[0038] (5) This invention is a digital method for cultural heritage based on multimodal fusion. By setting up a multi-terminal interactive presentation and hybrid storage architecture, the hybrid storage architecture provides the optimal storage and retrieval scheme for different types of data, ensuring system performance. On this basis, the same core knowledge system can be quickly adapted and deployed on multiple terminals such as websites, mini-programs, APPs, and digital art installations, breaking the limitations of time and space in traditional methods and greatly expanding the audience coverage and application scenarios of cultural heritage. Attached Figure Description
[0039] Figure 1 This is a schematic diagram of the method of the present invention. Detailed Implementation
[0040] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
[0041] Example
[0042] Please see Figure 1 This invention proposes a technical solution for a multimodal fusion-based method for digitizing cultural heritage: a multimodal fusion-based method for digitizing cultural heritage, comprising the following steps:
[0043] S1, Ancient Books and Documents Processing: Digitally identify ancient books and documents and extract key entities from them;
[0044] S2, Multimodal information integration: Acquire multimedia data related to the content of the ancient documents, and align and annotate the multimedia data with the key entities;
[0045] S3, Knowledge Graph Construction: Using the key entities as nodes and the relationships between entities as edges, construct a knowledge graph and mine the deep relationships between entities based on graph algorithms;
[0046] S4, AIGC Narrative Visualization: Extract narrative clues from the knowledge graph and multimodal data, and generate multimedia content based on the narrative clues using a multimodal AIGC model;
[0047] S5, multi-terminal interactive presentation: adapts and presents the generated multimedia content and knowledge graph data to multiple terminals.
[0048] In this embodiment, in S1, the labeled data for specific types of ancient books (such as medical books, local chronicles, and classic texts) is fine-tuned, and an image preprocessing module (such as deep learning-based layout analysis, noise reduction, and character segmentation) is integrated to achieve accurate understanding of the ancient book format and high-precision extraction of text. The text is first automatically punctuated and segmented, and then the named entity recognition is performed using an LLM fine-tuned on professional ancient book corpora (such as the abstract of the Siku Quanshu and biographies of historical figures). The fine-tuning enables the LLM to more accurately identify and understand complex entities with characteristics of the era.
[0049] In S2, multimodal pre-trained models such as CLIP and BLIP are used. These models have been trained on a massive number of text-image pairs and can map the text description of a landscape painting and the image of the landscape painting itself to very close positions in a high-dimensional vector space. By calculating the cosine similarity between the text vector of the key entity (such as "A Panorama of Rivers and Mountains") and all image vectors in the image library, cross-modal semantic retrieval and accurate association can be achieved, overcoming the limitations of traditional keyword-based matching.
[0050] In S3, knowledge graphs are not only a form of data storage, but also an engine for analysis. Path discovery algorithms (such as Dijkstra) can be used to explore hidden relationship paths between two historical figures; community discovery algorithms (such as Louvain) can automatically cluster different cultural schools or event groups from complex relationships; and centrality analysis (such as eigenvector centrality) can identify the most critical and influential figures or works in the entire cultural network.
[0051] In S4, the semantic understanding capabilities of LLM and the structural analysis capabilities of graph algorithms are combined. LLM is responsible for summarizing narrative segments from text descriptions associated with the graph, while graph algorithms provide structured clues such as "all relevant events surrounding a core entity." The combination of the two ensures that the narrative clues are both vivid and comprehensive. By performing low-rank adaptation on the basic text-generated graph model (such as Stable Diffusion), it can learn the artistic style (such as ink painting and blue-green landscape) and artifact characteristics of a specific historical period (such as the Song Dynasty), thus outputting images with a consistent style and historical accuracy. The "sketching construction → editing → anomaly detection and redrawing → style transfer" process is an iterative optimization loop. For example, the first round generates a composition sketch, the second round refines the clothing of the characters based on feedback from the evaluation model, the third round redraws the areas with generation errors, and finally unifies the overall artistic style. This workflow, through multiple "generation-evaluation-correction" loops, effectively solves the common problems of random content and detailed errors in AIGC technology, ensuring the quality and consistency of the final result.
[0052] Furthermore, the processing of ancient texts specifically includes:
[0053] (1) Construct a sample database of ancient books and documents;
[0054] (2) Use a customized OCR model to recognize ancient book images or texts to extract the text;
[0055] (3) The key entities are identified from the extracted text using a finely tuned large language model (LLM).
[0056] Furthermore, in the multimodal information integration, the data alignment and annotation are achieved by mapping the text descriptions of the key entities and the multimedia data to the same semantic vector space through a multimodal embedding model, and by calculating the similarity between the semantic vectors.
[0057] Furthermore, in the construction of the knowledge graph, the mining of deep associations is achieved by running at least one of the following algorithms on the knowledge graph: path discovery algorithm, community discovery algorithm, and centrality analysis algorithm.
[0058] Furthermore, the AIGC narrative visualization specifically includes:
[0059] (1) Use LLM and graph algorithms to extract narrative clues from the knowledge graph and multimodal data;
[0060] (2) Input the narrative clues as prompt words into the multimodal AIGC model to generate image or video content; wherein, the style and historical accuracy of the generated content are controlled by prompt word engineering.
[0061] Furthermore, the process of generating image content includes: based on the narrative clues, by repeatedly calling the image generation model and the visual evaluation model, performing sketch construction, image editing, anomaly detection and region redrawing, and style transfer in sequence.
[0062] Furthermore, the method employs a hybrid storage architecture for data storage, the hybrid storage architecture comprising:
[0063] Relational databases are used to store structured metadata; specifically, they efficiently manage structured data such as document entries and user information.
[0064] A graph database is used to store the knowledge graph; specifically, it natively supports the storage and complex querying of knowledge graph relationships.
[0065] Object storage is used to store multimedia data; specifically, it can cost-effectively store massive amounts of images, videos, and audio files.
[0066] Vector databases are used to store semantic vectors for data alignment. Specifically, they are optimized for high-dimensional vector similarity search and are the infrastructure for achieving fast data alignment in S2.
[0067] Furthermore, in the multi-terminal interactive presentation, the multi-terminal includes at least two of the following: website platform, mini-program, mobile application (APP), and digital media art installation.
[0068] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described above.
[0069] This method relies on conventional computer hardware systems and requires no custom hardware. The hardware architecture includes:
[0070] Data acquisition module: Relying on a desktop computer (CPU: 12th Gen Intel(R) Core(TM) i5-12400F or above, memory ≥32GB), it acquires multimodal data such as ancient books, documents, images, and videos from public resources such as museums and digital libraries through web crawlers, API interfaces, or direct reading of local storage.
[0071] Data processing module: Relying on the application server (CPU≥4 cores vCPU, memory≥8GB), it runs OCR model, LLM model, data alignment algorithm, graph algorithm and AIGC model to realize data processing, knowledge graph construction and content generation;
[0072] Storage module: It adopts a hybrid storage architecture (MySQL relational database to store structured data, Neo4j graph database to store knowledge graphs, and OSS object storage to store multimedia data), supporting data retention and multi-terminal access.
[0073] This invention, through a multimodal information integration step, uses a multimodal embedding model to align text entities with multimedia data such as images and videos in the semantic vector space. This overcomes the drawbacks of fragmented multimodal data in traditional methods, improves the efficiency and scale of data integration, and achieves precise association and mutual verification of cross-modal information. It lays a solid data foundation for building a three-dimensional digital resource system for cultural heritage, far exceeding the effect of single-modal digitization or simple superposition.
[0074] By constructing knowledge graphs, scattered entities and relationships are organized into a structured knowledge network. Graph algorithms such as path discovery and community discovery are used for automated mining. This allows for the visualization and quantitative analysis of deep connections that are traditionally difficult to detect and rely on manual methods. As a result, scattered cultural heritage knowledge is elevated into a systematic knowledge system with internal logic, greatly contributing to academic research and the public's in-depth understanding.
[0075] By combining the narrative clues mined from the knowledge graph with the multimodal AIGC model through the AIGC narrative visualization process, images, videos, and other content that are both historically accurate and artistically expressive can be generated in batches and automatically. This fundamentally solves the problems of abstract cultural heritage narratives and rigid dissemination forms, transforming dry text descriptions into multimedia content with visual impact and emotional resonance, and significantly improving the attractiveness and dissemination efficiency of cultural heritage.
[0076] When generating image content, a refined workflow is constructed by repeatedly calling the image generation and visual evaluation model, which includes sketch construction, image editing, anomaly detection and region redrawing, and style transfer. This effectively controls the randomness of AIGC output, ensuring high quality and strong consistency of the generated content in terms of details, style, and historical facts, making the final result more authentic and credible, and meeting the high requirements for accuracy in the dissemination of cultural heritage.
[0077] Through multi-terminal interactive presentation and hybrid storage architecture, the hybrid storage architecture provides the optimal storage and retrieval solution for different types of data, ensuring system performance. On this basis, the same core knowledge system can be quickly adapted and deployed on multiple terminals such as websites, mini-programs, apps, and digital art installations, breaking the limitations of time and space in traditional methods and greatly expanding the audience coverage and application scenarios of cultural heritage.
[0078] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A method for digitizing cultural heritage based on multimodal fusion, characterized in that, Includes the following steps: S1, Ancient Books and Documents Processing: Digitally identify ancient books and documents and extract key entities from them; S2, Multimodal information integration: Acquire multimedia data related to the content of the ancient documents, and align and annotate the multimedia data with the key entities; S3, Knowledge Graph Construction: Using the key entities as nodes and the relationships between entities as edges, construct a knowledge graph and mine the deep relationships between entities based on graph algorithms; S4, AIGC Narrative Visualization: Extract narrative clues from the knowledge graph and multimodal data, and generate multimedia content based on the narrative clues using a multimodal AIGC model; S5, multi-terminal interactive presentation: adapts and presents the generated multimedia content and knowledge graph data to multiple terminals.
2. The method for digitizing cultural heritage based on multimodal fusion according to claim 1, characterized in that, The processing of ancient books and documents specifically includes: (1) Construct a sample database of ancient books and documents; (2) Use a customized OCR model to recognize ancient book images or texts to extract the text; (3) The key entities are identified from the extracted text using a finely tuned large language model (LLM).
3. The method for digitizing cultural heritage based on multimodal fusion according to claim 1, characterized in that, In the multimodal information integration, the data alignment and annotation are achieved by mapping the text descriptions of the key entities and the multimedia data to the same semantic vector space through a multimodal embedding model, and by calculating the similarity between the semantic vectors.
4. The method for digitizing cultural heritage based on multimodal fusion according to claim 1, characterized in that, In the construction of the knowledge graph, the mining of deep associations is achieved by running at least one of the following algorithms on the knowledge graph: path discovery algorithm, community discovery algorithm, and centrality analysis algorithm.
5. The method for digitizing cultural heritage based on multimodal fusion according to claim 1, characterized in that, The AIGC narrative visualization specifically includes: (1) Use LLM and graph algorithms to extract narrative clues from the knowledge graph and multimodal data; (2) Input the narrative clues as prompt words into the multimodal AIGC model to generate image or video content; wherein, the style and historical accuracy of the generated content are controlled by prompt word engineering.
6. The method for digitizing cultural heritage based on multimodal fusion according to claim 5, characterized in that, The process of generating image content includes: based on the narrative clues, by repeatedly calling the image generation model and the visual evaluation model, sketching, image editing, anomaly detection and region redrawing, and style transfer are performed in sequence.
7. The method for digitizing cultural heritage based on multimodal fusion according to claim 1, characterized in that, The method employs a hybrid storage architecture for data storage, the hybrid storage architecture comprising: Relational databases are used to store structured metadata; A graph database is used to store the knowledge graph; Object storage is used to store multimedia data; A vector database used to store semantic vectors for data alignment.
8. The method for digitizing cultural heritage based on multimodal fusion according to claim 1, characterized in that, In the multi-terminal interactive presentation, the multi-terminal includes at least two of the following: website platform, mini-program, mobile application (APP), and digital media art installation.