Multi-modal real-time semantic segmentation intelligent analysis method and device based on cultural relic model

Abstract

The invention discloses a multi-modal real-time semantic segmentation intelligent analysis method and device based on a cultural relic model, and relates to the technical field of cultural relic digital analysis. Performing hierarchical labeling on the semantic information of the multi-modal data on the basis of three-dimensional space feature coding and cross-modal projection; based on a parameter fine tuning mode of spatial perception, realizing adaptation of a large language model to multi-modal data, and according to high-dimensional semantic embedding of a deep neural network, performing unified vector space representation on the multi-modal data to realize construction of a multi-modal knowledge base index; and based on a retrieval enhancement generation technology and in combination with the constructed multi-modal knowledge base index, realizing and applying the cultural relic three-dimensional model intelligent question-answering system. According to the invention, a more efficient, accurate and convenient intelligent solution can be provided for historical and cultural heritage research.

Description

Technical Field

[0001] This application relates to the field of digital analysis technology of cultural relics, specifically to a multimodal real-time semantic segmentation intelligent analysis method and device based on cultural relic models. Background Technology

[0002] In the current field of digital cultural heritage, the development of intelligent interactive systems is still in its early stages. Their functions are mainly concentrated on static identification and text retrieval of cultural relics, with relatively low accuracy. For example, a company's latest open-source large model, based on a test set of 1000 images of different grottoes in China, achieved an accuracy rate of only about 40%. Furthermore, while this technology can achieve basic information retrieval of cultural relics, it has significant functional limitations in practical applications. The most prominent problem is the system's lack of understanding of three-dimensional space, making it unable to support users in observing details of cultural relics from multiple angles and engaging in interactive questioning. Simultaneously, image recognition and knowledge-based question-answering modules typically operate independently, making it difficult for users to obtain real-time knowledge feedback related to their perspective during 3D exploration. In addition, most existing question-answering systems rely on plain text input and cannot combine the perspective transformation, scaling, and sectioning operations of the 3D model to achieve context-aware intelligent responses.

[0003] The root of these technical bottlenecks lies in the fact that the existing system architecture fails to achieve deep multimodal integration of 3D vision, semantic understanding, knowledge reasoning, and interactive control. This fragmented technical architecture prevents users from obtaining an intelligent question-and-answer experience that is deeply connected with spatial perspectives and structural features when exploring 3D models of cultural relics. Due to the non-standard nature and diversity of cultural relic data, the generalization ability of existing AI models is limited, resulting in narrow applications and low maturity in the vertical field of cultural heritage. Summary of the Invention

[0004] This application provides a multimodal real-time semantic segmentation intelligent analysis method and device based on cultural relic models, which can provide a more efficient, accurate and convenient intelligent solution for the study of historical and cultural heritage.

[0005] In a first aspect, embodiments of this application provide a multimodal real-time semantic segmentation intelligent analysis method based on a cultural relic model, the multimodal real-time semantic segmentation intelligent analysis method based on a cultural relic model includes: Collect multimodal data related to three-dimensional cultural heritage and perform standardized preprocessing; hierarchical annotation of semantic information of multimodal data based on three-dimensional spatial feature encoding and cross-modal projection. Based on spatially aware parameter fine-tuning, a large language model is adapted to multimodal data. Based on the high-dimensional semantic embedding of deep neural networks, multimodal data is represented by a unified vector space, and a multimodal knowledge base index is constructed. Based on retrieval-enhanced generation technology and combined with the constructed multimodal knowledge base index, an intelligent question-and-answer system for 3D models of cultural relics was realized and applied.

[0006] In conjunction with the first aspect, in one implementation method, The multimodal data includes structured data, unstructured data, digital image data, and 3D models of cultural heritage. The structured data includes archaeological reports; The unstructured data includes official archives, academic monographs, and research papers.

[0007] In conjunction with the first aspect, in one implementation, the hierarchical annotation of multimodal data semantic information based on three-dimensional spatial feature encoding and cross-modal projection specifically includes: Based on the 3D object detection architecture, multi-view feature extraction is performed on 3D visual data in multimodal data. By integrating semantic segmentation network and the synergistic effect of spatial attention mechanism and feature suppression algorithm, a multi-scale feature pyramid with depth perception is constructed. A visual-semantic dual-stream spatial projection network is established. The 3D feature space is processed in the visual stream, and a high-dimensional concept space is constructed in the semantic stream. Cross-modal alignment is achieved through a spatial contrast loss function to generate preliminary annotation results with depth information. The preliminary annotation results are actively verified.

[0008] In conjunction with the first aspect, in one implementation method, the spatially aware parameter fine-tuning method enables the large language model to adapt to multimodal data, specifically including: By integrating multimodal data, a three-dimensional semantic space with spatial topological relationships is constructed, enabling large language models to dynamically capture the correlation of geometric features of cultural relics. Based on three-dimensional spatial multi-channel, the attention of two-dimensional images is extended to three-dimensional voxel space, enabling large language models to analyze local features of cultural relics components and achieve cross-scale semantic understanding.

[0009] In conjunction with the first aspect, in one implementation method, For the fusion of multimodal data, specifically, it involves fusing UAV oblique photogrammetry texture, close-range photogrammetry details, and orthophotos; For three-dimensional space with multiple channels, including multi-angle depth maps and normal maps.

[0010] In conjunction with the first aspect, in one implementation, the step of constructing a multimodal knowledge base index by performing unified vector space representation of multimodal data based on the high-dimensional semantic embedding of a deep neural network specifically includes: By using a pre-trained cross-language text embedding model, structured and unstructured data are uniformly mapped to a 4096-dimensional continuous vector space, where semantic similarity is measured using cosine distance. A hierarchical design approach is adopted to construct the multimodal knowledge base index architecture.

[0011] In conjunction with the first aspect, in one implementation method, the construction of a multimodal knowledge base index architecture using a hierarchical design approach specifically includes: The candidate set is initially screened using an approximate nearest neighbor search algorithm; The search results are sorted using an attention-based reordering module.

[0012] In conjunction with the first aspect, in one implementation, the intelligent question-answering system for 3D models of cultural relics, based on retrieval enhancement generation technology and combined with a constructed multimodal knowledge base index, is implemented and applied, specifically including: A multimodal interaction system is constructed based on retrieval-enhanced generation technology, and the locally deployed large language model and multimodal understanding model are used as the core components of the intelligent question-answering system for 3D models of cultural relics. Knowledge acquisition is achieved through a two-stage retrieval mechanism, an approximate nearest neighbor search is performed in a high-dimensional semantic embedding space, and a re-ranking module based on an attention mechanism optimizes the relevance ranking of the retrieval results. In the content generation stage, the filtered multimodal contextual information is input into the domain-adapted generative model. A strategy combining the beam search algorithm and kernel sampling technology is adopted to obtain the generated result, thereby realizing the intelligent question-and-answer system for 3D models of cultural relics and its application.

[0013] Secondly, embodiments of this application provide a multimodal real-time semantic segmentation intelligent analysis device based on a cultural relic model, characterized in that the multimodal real-time semantic segmentation intelligent analysis device based on the cultural relic model includes: The acquisition module is used to acquire multimodal data related to three-dimensional cultural heritage and perform standardized preprocessing, and hierarchically annotate the semantic information of multimodal data based on three-dimensional spatial feature encoding and cross-modal projection. The module is used to implement a spatially-aware parameter fine-tuning method to adapt large language models to multimodal data. Based on the high-dimensional semantic embedding of deep neural networks, it performs unified vector space representation of multimodal data and realizes the construction of multimodal knowledge base index. The application module is used to realize and apply an intelligent question-and-answer system for 3D models of cultural relics based on retrieval-enhanced generation technology and combined with the constructed multimodal knowledge base index.

[0014] In conjunction with the second aspect, in one implementation method, The multimodal data includes structured data, unstructured data, digital image data, and 3D models of cultural heritage. The structured data includes archaeological reports; The unstructured data includes official archives, academic monographs, and research papers.

[0015] The beneficial effects of the technical solutions provided in this application include: This application breaks through the technical bottlenecks of traditional digital cultural heritage systems in terms of 3D spatial understanding and multimodal interaction. First, by deeply integrating 3D computer vision with a large language model, a cultural relic question-and-answer system with spatial perception capabilities is constructed, solving the key problem that existing technologies cannot combine 3D perspectives for intelligent response. Second, an innovative cross-modal dynamic alignment technology is developed, realizing real-time association between 3D model features, semantic segmentation results, and knowledge graphs, enabling the question-and-answer system to automatically generate context-related multimodal feedback based on changes in the user's perspective. This application effectively promotes a paradigm shift in digital cultural heritage from static retrieval to 3D intelligent interaction, realizing a closed-loop interactive experience of "spatial positioning-semantic retrieval-knowledge enhancement," providing a more efficient, accurate, and convenient intelligent solution for historical and cultural heritage research, and offering the first implementable 3D spatial cognition solution for scenarios such as museum guidance, cultural relic restoration, and education. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the multimodal real-time semantic segmentation intelligent analysis method based on cultural relic models proposed in this application. Figure 2 This is a schematic diagram of the functional modules of the multimodal real-time semantic segmentation intelligent analysis device based on cultural relic models in this application; Figure 3 This is a schematic diagram of the hardware structure of the multimodal real-time semantic segmentation intelligent analysis device based on cultural relic models, as described in this application. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0019] In the first aspect, the embodiments of this application provide a multimodal real-time semantic segmentation intelligent analysis method based on cultural relic models, which achieves a leapfrog development from "static retrieval" to "three-dimensional intelligent interaction" through multimodal fusion and domain knowledge injection.

[0020] This paper innovatively proposes a multimodal question-and-answer system for cultural relics based on a three-dimensional perspective, namely, an intelligent question-and-answer system for three-dimensional models of cultural relics. This system deeply integrates three-dimensional computer vision, large language models, and spatial knowledge graph technologies to construct an intelligent interactive paradigm of "what you see is what you ask." Specifically, the system uses three-dimensional object detection technology to analyze the structure of cultural relics from the user's perspective in real time, enabling the question-and-answer system to have spatial understanding capabilities. Cross-modal alignment technology achieves dynamic association between vision, language, and knowledge, and a three-dimensional rendering and edge computing architecture ensures rapid and efficient question-and-answer response time. When a user asks questions such as "What are the characteristics of the decorative craftsmanship from this angle?" while observing a three-dimensional model, the system can intelligently combine the current perspective features, model structure, and knowledge base information to generate accurate multimodal answers. This propels digital cultural heritage from the traditional static retrieval mode to a new stage of three-dimensional intelligent interaction, and has broad application prospects in museum guidance, cultural relic restoration research, and immersive education.

[0021] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating the multimodal real-time semantic segmentation intelligent analysis method based on cultural relic models proposed in this application. Figure 1 As shown, the multimodal real-time semantic segmentation intelligent analysis method based on cultural relic models includes: S1: Collect multimodal data related to 3D cultural heritage and perform standardized preprocessing; hierarchical annotation of semantic information of multimodal data based on 3D spatial feature encoding and cross-modal projection. S2: Based on spatial awareness parameter fine-tuning, the large language model is adapted to multimodal data. According to the high-dimensional semantic embedding of deep neural networks, multimodal data is represented by a unified vector space, and the multimodal knowledge base index is constructed. S3: Based on retrieval enhancement generation technology and combined with the constructed multimodal knowledge base index, an intelligent question-and-answer system for 3D models of cultural relics is realized and applied.

[0022] In this application, multimodal data includes structured data, unstructured data, digital image data, and three-dimensional models of cultural heritage; structured data includes archaeological reports; and unstructured data includes official archives, academic monographs, and research papers.

[0023] Specifically, this application proposes a complete solution to the problem of integrating multi-source heterogeneous data and constructing a knowledge base in the field of historical and cultural heritage. In the data acquisition phase, comprehensive acquisition of multimodal data is achieved, including: 1) structured data (archaeological reports, etc.); 2) unstructured data (official archives, academic monographs, research papers, etc.); 3) high-precision digital image data; and 4) high-precision 3D models of cultural heritage. All raw data undergoes standardized preprocessing according to standards.

[0024] Furthermore, in one embodiment, hierarchical annotation of multimodal data semantic information based on three-dimensional spatial feature encoding and cross-modal projection specifically includes: S101: Based on the 3D object detection architecture, multi-view feature extraction is performed on 3D visual data in multimodal data. By integrating semantic segmentation network and the synergistic effect of spatial attention mechanism and feature suppression algorithm, a multi-scale feature pyramid with depth perception is constructed. In terms of structure, the multi-scale feature pyramid is a multi-module collaborative system: First, based on an improved 3D object detection architecture as the backbone network, multi-level feature extraction is performed to generate a series of feature maps with decreasing resolution but increasing semantic information; Second, through a "top-down" upsampling path and lateral connections, the strong semantic features of the high-level layers are fused with the fine geometric features of the low-level layers, so that each scale has both rich semantics and details; Finally, a spatial attention mechanism and a feature suppression algorithm are introduced as collaborative enhancement modules. The former focuses on key regions, while the latter suppresses noise and inconsistent responses, and the output of the semantic segmentation network is fused to inject clear category semantic guidance, thereby forming a deep-perception multi-scale feature representation system. S102: Establish a visual-semantic dual-stream spatial projection network, process the 3D feature space in the visual stream, construct a high-dimensional concept space in the semantic stream, achieve cross-modal alignment through the spatial contrast loss function, generate preliminary annotation results with depth information, and actively verify the preliminary annotation results; Among them, for the visual stream of the visual-semantic two-stream spatial projection network, the visual stream is a computational pipeline specifically for processing the original data in the three-dimensional space. Its core task is to extract and construct an accurate spatial feature representation that can be understood by machines from the three-dimensional model of cultural relics. Specifically, the content processed by the visual stream includes multiple levels: First, there is the basic three-dimensional geometric data, such as point clouds (XYZ coordinates), voxels, or mesh models, which form the skeleton of spatial understanding; Second, there are rich visual attributes, including RGB color information, surface normal vectors, and multi-angle depth maps corresponding to each spatial point. These data provide details such as texture, material, and geometric orientation; Finally, the above data is deeply encoded through a multi-scale feature pyramid to generate feature maps at different levels of abstraction from microscopic details to macroscopic structures, and the spatial attention mechanism is used to weight and focus on these features, so as to highlight the key regions most relevant to the current analysis task and suppress irrelevant background interference. The output of the visual stream is a three-dimensional feature space rich in geometric, texture, multi-scale semantic, and spatial weight information, laying a solid foundation for subsequent cross-modal alignment with the semantic stream; Among them, for the semantic stream of the visual-semantic two-stream spatial projection network, the core task of the semantic stream is to construct a high-dimensional abstract concept space parallel and corresponding to the visual space. Its processing content does not directly depend on the geometric or pixel information of the three-dimensional model, but focuses on the semantic knowledge extracted and organized from sources such as texts and knowledge bases. Specifically, the semantic stream first processes various text corpora, including unstructured data such as archaeological reports, academic papers, and official archives, and maps these text information into high-dimensional vectors through a pre-trained cross-language text embedding model (such as a Transformer-based model). Each vector represents a mathematical expression of a specific concept, attribute, or relationship; Second, the semantic stream will construct or utilize a domain knowledge graph, which contains entities related to cultural relics (such as "bronze tripod", "Tang Dynasty"), attributes (such as "the decoration is taotie pattern"), and complex relationships between entities (such as "unearthed in", "belongs to"), thus forming a structured semantic network; Finally, through mechanisms such as the spatial contrast loss function, the semantic stream learns to accurately align these abstract concept vectors with the corresponding three-dimensional space features in the visual stream. For example, it associates the concept of "the decoration on the tripod foot" in the text description with the three-dimensional area of the tripod foot detected in the visual stream, and finally generates a preliminary semantic annotation result with depth information and bound to the spatial position; Among them, for the preliminary annotation result, it is specifically used to label each recognizable part or area in the three-dimensional model with a three-dimensional bounding box (or cube), and assign a semantic label and its confidence to each box. For example: {bbox: {min_point: [x1, y1, z1], max_point: [x2, y2, z2]}, label: "Tripod ear", confidence: 0.95}.

[0025] Specifically, in the implementation of 3D visual data annotation, this application proposes a deep learning-based multimodal stereo annotation framework, which achieves hierarchical annotation of semantic information through 3D spatial feature encoding and cross-modal projection. First, an improved 3D object detection architecture is used for multi-view feature extraction. By fusing a semantic segmentation network and through the synergistic effect of spatial attention mechanisms and feature suppression algorithms, a multi-scale feature pyramid with depth perception is constructed. Subsequently, a visual-semantic dual-stream spatial projection network is established. The 3D feature space (including XYZ coordinates and RGB channels) is processed in the visual stream, while a high-dimensional concept space is constructed in the semantic stream. Cross-modal alignment is achieved through a spatial contrast loss function, generating preliminary annotation results with depth information. These results are then manually verified through an expert collaborative annotation platform. This application achieves dimensional transformation from physical space to semantic space through a complete processing flow of "3D feature encoding → spatial semantic projection → stereo quality closed loop," and its architecture fully reflects the collaborative mapping relationship between spatial coordinate axes and semantic dimensions.

[0026] Furthermore, in one embodiment, a spatially aware parameter fine-tuning method is used to adapt a large language model to multimodal data, specifically including: S201: By integrating multimodal data, a three-dimensional semantic space with spatial topological relationships is constructed, enabling large language models to dynamically capture the correlation of geometric features of cultural relics. Among them, the three-dimensional semantic space is an abstract information space that deeply integrates and uniformly represents multimodal data (such as three-dimensional geometry, texture, and textual knowledge). Its structure is not a simple three-dimensional model, but a complex architecture composed of a spatial topological network and embedded semantic vectors. The spatial topological network accurately records the three-dimensional spatial relationships (such as adjacency, inclusion, and orientation) between various parts of the cultural relic (such as the handles and legs of the tripod), while the embedded semantic vectors map the semantic features (such as "decorative style" and "historical period") from multiple sources of information such as text and images to specific locations or regions in this space. In terms of function, the core value of this space lies in serving as a "translator" and "association engine" between the Large Language Model (LLM) and three-dimensional visual data. It binds the geometric features of the cultural relic (such as a specific curved surface structure) and its corresponding semantic concepts (such as "the curved part of the Kui dragon pattern") under the same coordinate frame, so that the Large Language Model, which is originally good at processing text sequences, can "understand" and dynamically capture the complex relationships between various geometric features in three-dimensional space, and achieve true cross-modal semantic understanding and reasoning. S202: Based on three-dimensional spatial multi-channel, the attention of two-dimensional images is extended to three-dimensional voxel space to realize the local feature analysis and cross-scale semantic understanding of cultural relics components by a large language model.

[0027] In this application, the fusion of multimodal data specifically involves fusing UAV oblique photogrammetry texture, close-range photogrammetry details, and orthophotos; and fusing three-dimensional spatial multi-channel data, including multi-angle depth maps and normal maps.

[0028] Specifically, in the construction of the 3D cultural heritage model, this application adopts a parameter-efficient fine-tuning technique based on spatial perception (such as geospatial information) to achieve professional adaptation of the large language model to 3D cultural heritage data. By integrating multimodal data (such as UAV oblique photography texture, close-range photogrammetry details, orthophotos, etc.), a 3D semantic space with spatial topological relationships is constructed, enabling the large language model to dynamically capture the correlation of geometric features of cultural relics. In terms of technical implementation, the performance of the large language model in core tasks such as 3D cultural relic feature recognition is particularly optimized. By introducing multi-channel based 3D space (multi-angle depth maps, normal maps, etc.), the attention of traditional 2D images is extended to 3D voxel space, significantly improving the large language model's ability to analyze local features of complex components (such as areas of paint peeling) and its depth of cross-scale semantic understanding. Through an AI + human collaborative solution, a precise mapping relationship between the 3D semantic space and textual concepts is established, realizing multi-dimensional knowledge fusion.

[0029] Furthermore, in one embodiment, multimodal data is represented in a unified vector space based on the high-dimensional semantic embedding of a deep neural network to construct a multimodal knowledge base index, specifically including: S211: Through a pre-trained cross-lingual text embedding model, structured and unstructured data are uniformly mapped to a 4096-dimensional continuous vector space, where semantic similarity is measured by cosine distance. Among them, the pre-trained cross-language text embedding model is the core technical component for building a multimodal knowledge base index. It is essentially a large-scale language model based on the Transformer architecture (such as chatGPT), which is specially trained on massive multilingual text data and has the ability to convert text fragments of different languages ​​(such as Chinese archaeological reports or English papers) into high-dimensional numerical vectors. The function of this model is to uniformly map heterogeneous structured and unstructured text data to a 4096-dimensional continuous vector space. In this space, the semantic similarity of texts is measured by calculating the cosine distance between vectors. The closer the vector directions are, the higher the semantic similarity. This process effectively solves the problem of unified semantic representation of multi-source cultural relics data and lays a solid foundation for efficient semantic retrieval based on approximate nearest neighbor search. It enables accurate and fast location of multimodal information related to user questions from a knowledge base of billions of data points. S212: A hierarchical design approach is used to construct the multimodal knowledge base index architecture.

[0030] In this application, the construction of a multimodal knowledge base index architecture using a hierarchical design approach specifically includes: a: Use the approximate nearest neighbor search algorithm to perform preliminary screening of the candidate set; Specifically, regarding the candidate set, when a user asks a question (e.g., "What are the characteristics of the patterns at this angle?"), the question is first converted into a high-dimensional query vector using a pre-trained text embedding model. Then, the algorithm searches a vast vector index for vectors with the closest cosine distance to this query vector. The original data entries corresponding to these vectors constitute the "candidate set." The candidate set is multimodal; it is not a single list of text but a collection containing various types of information, primarily covering: Text excerpts: such as paragraphs in archaeological reports describing similar patterns, or related discussions in research papers; 3D model fragments or features: 3D model components or their high-dimensional feature representations that are similar to the current user's viewing perspective or decorative features; Related images or drawings: such as high-resolution close-up images or line drawings of the pattern; The candidate set is characterized by pursuing high recall, that is, to avoid missing any potentially relevant information as much as possible. However, it may also contain some results that are less relevant or redundant. Therefore, it needs to be passed to the subsequent attention-based re-ranking module for fine-tuning and sorting to ultimately improve the accuracy and relevance of the retrieval results. This initial screening step greatly optimizes the retrieval efficiency in massive data (billions of vectors). b: The search results are sorted using an attention-based reordering module.

[0031] Specifically, in the multimodal knowledge base indexing construction stage, this application employs high-dimensional semantic embedding technology based on deep neural networks to achieve a unified vector space representation of heterogeneous data. Through a pre-trained cross-language text embedding model, pre-processed structured and unstructured data are uniformly mapped to a 4096-dimensional continuous vector space, where semantic similarity is measured using cosine distance. The index architecture adopts a hierarchical design, firstly using an approximate nearest neighbor search algorithm to complete the initial screening of the candidate set, and then using an attention-based re-ranking module to achieve refined ranking of the retrieval results. This effectively improves retrieval performance in large-scale knowledge base environments, achieving excellent levels in key evaluation indicators such as recall and precision. To optimize storage efficiency, vector quantization compression technology can also be used to optimize the index, while supporting incremental update mechanisms to ensure the timeliness of knowledge base content. At the architectural level, a distributed computing framework enables horizontal scalability, stably supporting real-time retrieval needs for billions of vectors. This application fully presents the end-to-end technical implementation from data vectorization and index construction to retrieval optimization, providing a systematic solution for efficient retrieval of multimodal knowledge bases.

[0032] Furthermore, in one embodiment, based on retrieval-enhanced generation technology and combined with a constructed multimodal knowledge base index, an intelligent question-answering system for 3D models of cultural relics is implemented and applied, specifically including: A multimodal interaction system is constructed based on retrieval-enhanced generation technology, and the locally deployed large language model and multimodal understanding model are used as the core components of the intelligent question-answering system for 3D models of cultural relics. Knowledge acquisition is achieved through a two-stage retrieval mechanism, an approximate nearest neighbor search is performed in a high-dimensional semantic embedding space, and a re-ranking module based on an attention mechanism optimizes the relevance ranking of the retrieval results. In the content generation stage, the filtered multimodal contextual information is input into the domain-adapted generative model. A strategy combining the beam search algorithm and kernel sampling technology is adopted to obtain the generated result, thereby realizing the intelligent question-and-answer system for 3D models of cultural relics and its application.

[0033] Among them, the multimodal understanding model is one of the core components of the intelligent question-answering system for 3D cultural relic models. Its structure is usually based on the Transformer architecture and consists of a visual encoder, a text encoder, and a cross-modal fusion module. The visual encoder (such as a 3D convolutional network based on ViT) is responsible for parsing visual data such as multi-view images and depth maps of the 3D model and extracting spatial geometric and texture features. The text encoder processes the text query input by the user or the semantic information in the knowledge base. The most critical part is the cross-modal fusion module (usually using a cross-attention mechanism), which can establish a fine alignment relationship between visual features and text concepts. For example, it can associate the user's description of "tripod ear ornamentation" with the visual features of the corresponding parts in the 3D model. The core function of this model is to achieve a unified understanding of visual and linguistic semantics, transforming unstructured 3D visual information into a semantic representation that can be processed by a large language model, thereby supporting the system to achieve accurate question-answering capabilities of "what you see is what you ask". It is the technical foundation for the entire multimodal interaction system to achieve cross-modal intelligent reasoning. Specifically, the process of "filtering" multimodal contextual information is a refined decision-making process based on multi-dimensional relevance scores and attention weights. This filtering mechanism is not a single condition, but a comprehensive result determined by successive retrieval stages. First, the filtering criteria directly rely on the output of the previous stage's "attention-based reordering module." This module performs deep semantic analysis on the initially retrieved candidate set, calculating a comprehensive relevance score for each candidate piece of information relative to the user's current query (including the text question and the 3D perspective context). Therefore, the core criteria for "filtering" here include: Semantic relevance: Cosine similarity score between candidate information (such as text paragraphs, 3D component features) and user query intent in vector space; Spatial context relevance: The degree to which candidate information matches the user's current 3D model perspective and the focused spatial area (such as a specific cultural relic component); Attention weights: During the re-ranking process, the model calculates weight values ​​through a cross-attention mechanism to identify which information segments are more critical to generating the final answer; Domain expertise confidence level: The confidence level assessment of candidate information belonging to authoritative knowledge in the field of cultural relics by a domain-adapted model; Ultimately, the intelligent question-answering system for 3D models of cultural relics will set a comprehensive threshold or select the Top-K highest-scoring multimodal information fragments to form the most relevant and high-quality multimodal context after "filtering," and then input it as the prompt context into the generation model; this rigorous filtering mechanism is the key to ensuring that the generated answer is both accurate and context-relevant.

[0034] Specifically, in the implementation of the intelligent question-answering system for 3D models of cultural relics, this application constructs a multimodal interaction system based on the Retrieval Augmentation Generation (RAG) technology framework. The intelligent question-answering system for 3D models of cultural relics uses a locally deployed, large-scale pre-trained large language model and a fine-tuned version of the multimodal understanding model as core components. Knowledge acquisition is achieved through a two-stage retrieval mechanism: the first stage performs an approximate nearest neighbor search in a high-dimensional semantic embedding space; the second stage optimizes the relevance ranking of the retrieval results using an attention-based re-ranking module.

[0035] In the content generation stage, the filtered multimodal contextual information (including text, current scene of the 3D model and other multi-source data) is input into the domain-adapted generation model. A strategy combining the beam search algorithm and kernel sampling technology is adopted to ensure the accuracy of the generated results while maintaining reasonable output diversity.

[0036] To support multimodal interaction, the intelligent question-and-answer system for 3D cultural relic models integrates an automatic speech recognition module to convert speech to text and employs speech synthesis technology to generate natural speech feedback. At the system architecture level, a microservice architecture is used to modularly deploy each functional component, and a high-speed message queue enables efficient data exchange between components, ensuring that the system meets real-time interaction requirements in terms of key performance indicators such as throughput and response latency.

[0037] This application addresses the 3D intelligent interaction needs in the field of historical and cultural heritage by constructing a multimodal knowledge base that integrates multimodal data acquisition, standardized modeling, and dynamic knowledge association. In the data acquisition phase, it innovatively integrates spatial data such as 3D laser scanning point clouds and high-precision texture maps, forming a multi-source heterogeneous data system with traditional documents, archives, and archaeological reports, providing data support for the accurate reconstruction of 3D scenes. At the data modeling level, it constructs a 3D ontology model containing spatial coordinates, material properties, and structural features. Through deep learning-based semantic segmentation technology, it performs refined data classification, forming a multi-dimensional data architecture with traditional artifact attributes and spatiotemporal information. This breaks through the limitations of traditional two-dimensional knowledge bases, laying the data foundation for "what you see is what you ask" 3D intelligent interaction, and dynamically triggering context-aware feedback of related knowledge.

[0038] This application addresses key issues in the digital preservation and intelligent application of historical and cultural heritage, proposing a complete solution. Through five technological breakthroughs—multimodal data integration, 3D semantic annotation, professional domain knowledge transfer, cross-modal retrieval optimization, and intelligent interactive experience—it achieves end-to-end innovation in the entire process of historical and cultural heritage preservation, from data acquisition to intelligent application. It innovatively integrates multi-source heterogeneous data, including archaeological reports, documentary archives, 3D point clouds, and high-precision textures, resolving compatibility issues related to data from different sources and with varying precision. In cross-modal retrieval, the system supports billion-level vector retrieval (with response times in the seconds) and innovative modes such as "searching for cultural relics by image," maintaining a precision rate of over 80% in test data. The interactive experience achieves second-level 3D model-knowledge linkage response, with a question-and-answer system accuracy rate exceeding 90%. This application provides a new approach for research in cultural relic preservation and efficient educational research.

[0039] Secondly, this application also provides a multimodal real-time semantic segmentation intelligent analysis device based on a cultural relic model.

[0040] In one embodiment, reference is made to Figure 2 , Figure 2 This is a schematic diagram of the functional modules of the multimodal real-time semantic segmentation intelligent analysis device based on cultural relic models, as described in this application. Figure 2 As shown, the multimodal real-time semantic segmentation intelligent analysis device based on cultural relic models includes: a data acquisition module, a construction module, and an application module.

[0041] The acquisition module is used to collect multimodal data related to 3D cultural heritage and perform standardized preprocessing. It performs hierarchical annotation of the semantic information of multimodal data based on 3D spatial feature encoding and cross-modal projection. The construction module is used to adapt the large language model to multimodal data based on spatial perception parameter fine-tuning. It performs unified vector space representation of multimodal data based on high-dimensional semantic embedding of deep neural networks to realize the construction of multimodal knowledge base index. The application module is used to realize and apply an intelligent question-answering system for 3D cultural relic models based on retrieval enhancement generation technology and combined with the constructed multimodal knowledge base index.

[0042] In this application, the multimodal data includes structured data, unstructured data, digital image data, and three-dimensional models of cultural heritage; the structured data includes archaeological reports; and the unstructured data includes official archives, academic monographs, and research papers.

[0043] Thirdly, this application provides a multimodal real-time semantic segmentation intelligent analysis device based on a cultural relic model. The multimodal real-time semantic segmentation intelligent analysis device based on a cultural relic model can be a personal computer (PC), a laptop, a server, or other device with data processing capabilities.

[0044] Reference Figure 3 , Figure 3 This is a schematic diagram of the hardware structure of the multimodal real-time semantic segmentation intelligent analysis device based on a cultural relic model involved in the embodiments of this application. In the embodiments of this application, the multimodal real-time semantic segmentation intelligent analysis device based on a cultural relic model may include a processor, a memory, a communication interface, and a communication bus.

[0045] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0046] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces. These interfaces enable interconnection of internal components within the multimodal real-time semantic segmentation intelligent analysis device based on artifact models, and also enable interconnection between the device and other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0047] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0048] The processor can be a general-purpose processor, which can call the multimodal real-time semantic segmentation intelligent analysis program based on the cultural relic model stored in the memory and execute the multimodal real-time semantic segmentation intelligent analysis method based on the cultural relic model provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the multimodal real-time semantic segmentation intelligent analysis program based on the cultural relic model is called can refer to the various embodiments of the multimodal real-time semantic segmentation intelligent analysis method based on the cultural relic model in this application, and will not be repeated here.

[0049] Those skilled in the art will understand that Figure 3The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0050] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0051] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0052] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0053] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0054] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0055] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A multimodal real-time semantic segmentation intelligent analysis method based on cultural relic models, characterized in that, The multimodal real-time semantic segmentation intelligent analysis method based on cultural relic models includes: Collect multimodal data related to three-dimensional cultural heritage and perform standardized preprocessing; hierarchical annotation of semantic information of multimodal data based on three-dimensional spatial feature encoding and cross-modal projection. Based on spatially aware parameter fine-tuning, a large language model is adapted to multimodal data. Based on the high-dimensional semantic embedding of deep neural networks, multimodal data is represented by a unified vector space, and a multimodal knowledge base index is constructed. Based on retrieval-enhanced generation technology and combined with the constructed multimodal knowledge base index, an intelligent question-and-answer system for 3D models of cultural relics was realized and applied.

2. The multimodal real-time semantic segmentation intelligent analysis method based on cultural relic models as described in claim 1, characterized in that: The multimodal data includes structured data, unstructured data, digital image data, and 3D models of cultural heritage. The structured data includes archaeological reports; The unstructured data includes official archives, academic monographs, and research papers.

3. The multimodal real-time semantic segmentation intelligent analysis method based on cultural relic models as described in claim 2, characterized in that, The hierarchical annotation of multimodal data semantic information based on three-dimensional spatial feature encoding and cross-modal projection specifically includes: Based on the 3D object detection architecture, multi-view feature extraction is performed on 3D visual data in multimodal data. By integrating semantic segmentation network and the synergistic effect of spatial attention mechanism and feature suppression algorithm, a multi-scale feature pyramid with depth perception is constructed. A visual-semantic dual-stream spatial projection network is established. The 3D feature space is processed in the visual stream, and a high-dimensional concept space is constructed in the semantic stream. Cross-modal alignment is achieved through a spatial contrast loss function to generate preliminary annotation results with depth information. The preliminary annotation results are actively verified.

4. The multimodal real-time semantic segmentation intelligent analysis method based on cultural relic models as described in claim 2, characterized in that, The spatially-aware parameter fine-tuning method enables the large language model to adapt to multimodal data, specifically including: By integrating multimodal data, a three-dimensional semantic space with spatial topological relationships is constructed, enabling large language models to dynamically capture the correlation of geometric features of cultural relics. Based on three-dimensional spatial multi-channel, the attention of two-dimensional images is extended to three-dimensional voxel space, enabling large language models to analyze local features of cultural relics components and achieve cross-scale semantic understanding.

5. The multimodal real-time semantic segmentation intelligent analysis method based on cultural relic models as described in claim 4, characterized in that: For the fusion of multimodal data, specifically, it involves fusing UAV oblique photogrammetry texture, close-range photogrammetry details, and orthophotos; For three-dimensional space with multiple channels, including multi-angle depth maps and normal maps.

6. The multimodal real-time semantic segmentation intelligent analysis method based on cultural relic models as described in claim 2, characterized in that, The method of constructing a multimodal knowledge base index by representing multimodal data in a unified vector space based on the high-dimensional semantic embedding of deep neural networks specifically includes: By using a pre-trained cross-language text embedding model, structured and unstructured data are uniformly mapped to a 4096-dimensional continuous vector space, where semantic similarity is measured using cosine distance. A hierarchical design approach is adopted to construct the multimodal knowledge base index architecture.

7. The multimodal real-time semantic segmentation intelligent analysis method based on cultural relic models as described in claim 6, characterized in that, The construction of a multimodal knowledge base index architecture using a hierarchical design approach specifically includes: The candidate set is initially screened using an approximate nearest neighbor search algorithm; The search results are sorted using an attention-based reordering module.

8. The multimodal real-time semantic segmentation intelligent analysis method based on cultural relic models as described in claim 2, characterized in that, The aforementioned intelligent question-answering system for 3D models of cultural relics, based on retrieval-enhanced generation technology and combined with a constructed multimodal knowledge base index, specifically includes: A multimodal interaction system is constructed based on retrieval-enhanced generation technology, and the locally deployed large language model and multimodal understanding model are used as the core components of the intelligent question-answering system for 3D models of cultural relics. Knowledge acquisition is achieved through a two-stage retrieval mechanism, an approximate nearest neighbor search is performed in a high-dimensional semantic embedding space, and a re-ranking module based on an attention mechanism optimizes the relevance ranking of the retrieval results. In the content generation stage, the filtered multimodal contextual information is input into the domain-adapted generative model. A strategy combining the beam search algorithm and kernel sampling technology is adopted to obtain the generated result, thereby realizing the intelligent question-and-answer system for 3D models of cultural relics and its application.

9. A multimodal real-time semantic segmentation intelligent analysis device based on cultural relic models, characterized in that, The multimodal real-time semantic segmentation intelligent analysis device based on cultural relic models includes: The acquisition module is used to acquire multimodal data related to three-dimensional cultural heritage and perform standardized preprocessing, and hierarchically annotate the semantic information of multimodal data based on three-dimensional spatial feature encoding and cross-modal projection. The module is used to implement a spatially-aware parameter fine-tuning method to adapt large language models to multimodal data. Based on the high-dimensional semantic embedding of deep neural networks, it performs unified vector space representation of multimodal data and realizes the construction of multimodal knowledge base index. The application module is used to realize and apply an intelligent question-and-answer system for 3D models of cultural relics based on retrieval-enhanced generation technology and combined with the constructed multimodal knowledge base index.

10. The intelligent analysis device for multimodal real-time semantic segmentation based on cultural relic models as described in claim 9, characterized in that: The multimodal data includes structured data, unstructured data, digital image data, and 3D models of cultural heritage. The structured data includes archaeological reports; The unstructured data includes official archives, academic monographs, and research papers.

Images