Image format geological map enhanced understanding method based on multi-modal large model
By constructing a geoscience knowledge graph and integrating a multimodal large model, the problems of lack of professional knowledge and insufficient multi-component correlation reasoning in the intelligent understanding of geological maps are solved, and the intelligent understanding of geological maps with high accuracy and reliability is achieved.
Patent Information
- Application Number
- CN202511595134.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-03
- Publication Date
- 2026-02-24
AI Technical Summary
Existing multimodal large models suffer from a lack of professional knowledge, difficulties in visual-semantic alignment, and insufficient multi-component correlation reasoning when understanding highly specialized and structurally complex geological maps, resulting in insufficient accuracy and reliability of intelligent understanding of geological maps.
By constructing a geoscience knowledge graph and combining retrieval-enhanced generation (RAG) technology with a multimodal large model, this method integrates the structured semantic organization capabilities of the knowledge graph with the cross-modal understanding capabilities of the multimodal large model to achieve a high-precision understanding of the professional elements and geological processes of geological maps. The method includes geological entity and relation extraction, metadata extraction and retrieval enhancement, and cross-modal semantic alignment and deep reasoning of the multimodal large model.
It significantly improves the accuracy and reliability of understanding professional elements and geological processes in geological maps, solves the problems of lack of systematic organization of domain knowledge and weak semantic connections, and realizes the interpretability and efficiency of intelligent understanding and knowledge services of geological maps.
Smart Images
Figure CN121564751A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent geological map understanding technology, and specifically discloses a method for enhancing the understanding of image-format geological maps based on a multimodal large model. Background Technology
[0002] In geological research and production practices, geological maps and written reports typically serve as complementary core materials, jointly carrying and conveying professional geological information. Geological maps systematically record the spatial distribution and interrelationships of elements such as rock units, stratigraphic boundaries, structural morphology, and mineral distribution in a visual form, serving as a key carrier for expressing geological results. Written reports, on the other hand, interpret, analyze, and demonstrate the content of the geological maps, providing textual descriptions of genetic mechanisms, chronological basis, and exploration recommendations. Together, they form a complete system of geological knowledge.
[0003] Geological maps have irreplaceable value in practical applications, serving a wide range of aspects such as mineral resource exploration, geological disaster monitoring and early warning, site selection for major projects, environmental protection and land spatial planning, playing a key role in supporting decision-making and guiding practice. However, the understanding and use of geological maps has long relied on professionals to interpret legends, symbols, colors and spatial relationships. Behind them lies a large number of professional terms, regional geological structural backgrounds and metallogenic theories, which limits public use and the development of related intelligent applications.
[0004] In recent years, with the rapid advancement of pre-trained large language model technology, Multimodal Large Models (MMLMs) have demonstrated remarkable capabilities in tasks such as image generation, natural image understanding, and scene reasoning, showcasing powerful potential for cross-modal semantic perception and contextual reasoning. However, despite MMLMs' excellent performance in general vision-language tasks, they still face significant challenges in understanding highly specialized and structurally complex geological maps. These challenges mainly stem from the characteristics of geological maps themselves: First, geological maps typically have extremely high spatial resolution and information density, containing a vast amount of detailed elements, such as legends, stratigraphic boundaries, faults, and attitude symbols, requiring comprehensive detection, segmentation, and recognition; second, geological maps are multi-component interconnected images, often organically composed of multiple parts such as main maps, cross-sectional views, legends, and comprehensive columnar sections, necessitating the establishment of semantic relationships and spatial mappings between components; third, the correct interpretation of geological maps heavily relies on domain background knowledge, such as specific stratigraphic names, lithological assemblages, and tectonic evolution history, which far exceeds the scope of general model training corpora. Therefore, relying solely on end-to-end MMLM is insufficient for reliable geological map understanding. It is still necessary to combine computer vision technologies (such as object detection, semantic segmentation, and optical character recognition OCR) with professional knowledge injection methods to achieve truly intelligent geological map understanding.
[0005] Technical solutions for intelligent understanding of geological maps require the integration of cutting-edge technologies from multiple fields. The core challenge lies in achieving high-precision, interpretable professional image semantic parsing and question answering. This process mainly relies on two key aspects: first, the construction of a high-quality domain knowledge base; and second, deep cross-modal information retrieval and semantic matching. Knowledge graphs, with their powerful structured knowledge organization capabilities, have become a core means of constructing knowledge bases in the geosciences. They integrate geological concepts through <head entity-relationship-tail entity> triples, providing interpretable and reasonable semantic support, laying the foundation for the systematic management of complex geological knowledge. Traditional methods often rely on rule-based or statistical models (such as BERT-BiLSTM-CRF) to extract entities and relationships from unstructured geological texts, gradually constructing knowledge graphs. With the development of large language models, their powerful semantic understanding and knowledge implicit capabilities provide new avenues for the automatic construction and expansion of knowledge graphs.
[0006] In information retrieval and semantic matching, Retrieval Enhanced Generation (RAG) technology is gradually becoming a key solution. It deeply integrates external structured knowledge retrieval with generative models, significantly improving question-answering accuracy. Specifically, in geological map understanding, this method first uses a visual model to decompose the geological map into components and extract metadata, such as legend recognition, feature detection, and OCR text extraction. Then, using the extracted metadata as a query, it retrieves relevant entities and relational subgraphs in the knowledge graph, forming background knowledge-enhanced context. Finally, a multimodal large model integrates image visual information with the retrieved knowledge for joint reasoning, thereby significantly improving the understanding and credibility of professional elements, spatial structures, and geological processes in geological maps.
[0007] Currently, intelligent understanding technology for geological maps still faces a series of key bottlenecks. In knowledge graph construction, traditional methods heavily rely on manually predefined extraction rules and ontology patterns. This is not only cumbersome and poorly scalable, but also severely limited by the completeness and consistency of expert experience, making it difficult to adapt to multi-source, heterogeneous, and continuously growing geological text data, resulting in low knowledge acquisition efficiency and limited coverage. In the application of Retrieval Enhancement Generation (RAG) technology, existing methods mostly rely on a single text modality for retrieval and enhancement, lacking the ability to collaboratively model and align multi-modal information such as images and spatial structures. This makes it difficult to effectively support deep understanding of strongly multi-source semantically related objects like geological maps.
[0008] While multimodal large modeling (MMLM) has demonstrated great potential in general image understanding tasks, it still has significant shortcomings when dealing with highly specialized and structurally complex images such as geological maps. First, the model lacks domain prior knowledge, making it difficult to accurately perceive and semantically associate the numerous specialized symbols, legends, stratigraphic boundaries, and structural features in the map. Second, due to the scarcity of geological map-text pairs in the training data, the model is prone to generating "illusions," i.e., generating descriptions or inferences that do not conform to geological facts, seriously affecting the credibility and usability of the results. Third, existing MMLMs often process the entire geological map in an end-to-end manner, failing to fully consider the semantic associations and decoupling analysis needs between its multi-component structures (such as legends, profiles, and columnar sections), resulting in the loss of detailed information and overall inference bias.
[0009] In view of this, the present invention provides a method for enhancing the understanding of image-format geological maps based on multimodal large models, in order to solve the above problems. Summary of the Invention
[0010] This invention proposes an enhanced understanding method for image-based geological maps based on a multimodal large model. By integrating the structured semantic organization capabilities of knowledge graphs, the dynamic knowledge retrieval mechanism of Retrieval Enhanced Generation (RAG), and the cross-modal understanding capabilities of the multimodal large model (MMLM), this method effectively addresses the problems of lack of professional knowledge, difficulties in visual-semantic alignment, and insufficient multi-component associative reasoning in geological map understanding using MMLM. This method can significantly improve the accuracy and reliability of understanding professional elements, metallogenic regularities, and geological processes in geological maps, providing an interpretable and efficient solution for intelligent understanding and knowledge services of geological maps.
[0011] To achieve the above objectives, the present invention provides the following basic solution: A method for enhancing the understanding of image-format geological maps based on multimodal large models includes the following steps: S01: Geological Knowledge Graph Construction: Geological entities and relationships are extracted from geological corpora, professional terms and their semantic associations are identified using a large language model, vector indexes are constructed and community summaries are generated to form a geological knowledge graph that supports complex semantic reasoning, and the graph is persistently stored in the Neo4j graph database. After constructing the geoscience knowledge graph, a knowledge graph index and community are built based on the geoscience knowledge graph; S02: Geological map metadata extraction: A pre-trained visual model is used to perform component parsing and structural decomposition on image-format geological maps to accurately extract key metadata from image-format geological maps; S03: Metadata retrieval enhancement: Based on key metadata extracted from image-format geological maps, entity and relationship subgraph retrieval is performed in the constructed geological knowledge graph. The image content and geological background knowledge are integrated through the retrieval enhancement generation mechanism to form an enhanced semantic context. S04: Multimodal Large Model Understanding: The retrieved enhanced metadata and image-format geological maps are input into the multimodal large model to drive cross-modal semantic alignment and deep reasoning.
[0012] Furthermore, in step S02, the key metadata in the image-format geological map includes the geological map name, legend symbols, and structural annotations.
[0013] Furthermore, in step S01, the method for constructing the geoscience knowledge graph is as follows: First, a large language model is used to identify geological entities from unstructured text and extract semantic relationships between entities, and a knowledge network with deep associations is constructed; this knowledge network is a triplet knowledge network of head entity-relationship-tail entity. Then, based on the GraphRAG framework, the geological corpus is divided into semantic units using an intelligent block partitioning algorithm to perform text block partitioning and obtain text blocks after text block partitioning. Entity relations in the text blocks are extracted, and relation triples are built based on the triple knowledge network of head entity-relation-tail entity. Finally, semantically similar entities are merged through an entity alignment mechanism, and the geological knowledge graph data is persistently stored in the Neo4j graph database.
[0014] Furthermore, the steps for building a knowledge graph index and community are as follows: First, a vectorized index of entity nodes is constructed, semantic communities are divided according to the topological structure of the geological knowledge graph, and a high-level semantic community summary is generated using a large language model. Then, under the retrieval enhancement generation mechanism, the geological knowledge map is combined with the vector index to construct a knowledge enhancement mechanism that supports multi-granularity retrieval; The large language model uses structured prompts to guide its execution, specifically as follows: Entity recognition and standardization: Entities are identified and normalized based on predefined types, their names, types, and descriptions are standardized, and synonym expressions are unified; Relationship recognition and classification: Semantic associations are established between entities, and they are classified into predefined relationship types based on context, generating relationship descriptions and strength scores; Quality control and deduplication: Consistency, uniqueness, and completeness checks are performed on entities and relationships through rule constraints, duplicate entities are merged, and ambiguous referential phenomena are eliminated.
[0015] Furthermore, the entity alignment mechanism is as follows: A01: First, merge: Numerical encoding of entity semantic information is performed. The bge-m3 embedding model is deployed locally in Ollam to encode entity names, attributes, and context, generating 1024-dimensional semantic vectors. Large-scale semantic similarity retrieval is achieved by leveraging the FAISS vector index library. In this vector space, semantically similar entities exhibit high cosine similarity, calculated as follows: In the formula, and Entities and The embedding vector, where · represents the vector dot product. This represents the magnitude of the vector. The closer this value is to 1, the more semantically similar the two entities are. A threshold of 0.95 is set, and entity pairs higher than this threshold will be merged. A02: Further Optimization: The Leiden algorithm is introduced to conduct community detection, in order to identify semantically closely related sub-communities in the geological knowledge graph. The Leiden algorithm is based on modularity optimization, and the modularity calculation formula is as follows: In the formula, A ij k represents the connection weight between node i and node j in the adjacency matrix. i k j Let m be the degree of node i and node j, m be the total weight of the edges in the graph, and c be the degree of node i and node j. i and c j For the community to which the node belongs, It is the Kronecker function (when c) i =c j (If the value is 1, otherwise it is 0). This method can effectively decompose large-scale geological knowledge maps and maintain high connectivity within the community.
[0016] Furthermore, the specific steps for generating high-level semantic community summaries using a large language model are as follows: First, priorities are assigned to entities within the community, calculated as follows: In the formula, deg(e) represents the node degree, PR(e) represents the PageRank value, Cen(e) represents the centrality index, and α, β, and γ are all weight parameters, satisfying α+β+γ=1; Then, an iterative information aggregation strategy is adopted to input representative entities, relationships, and descriptive text within the community into a large language model to generate a semantic summary. When the community size exceeds the model context limit, the system adopts a dynamic truncation mechanism. In the formula, This indicates that the final community summary has been obtained. Represents core entity information. For context to be added, Indicates the maximum context length of the LLM. The function is used to truncate on demand.
[0017] Furthermore, the pre-trained visual model is a pre-trained YOLOv10 model for geological map detection and legend unit detection. In the component parsing stage, the multi-layered and heterogeneous information structure within the geological map is treated as a hierarchical tree structure with the entire map as the root node, and decomposed from top to bottom into the legend area, main map area, scale bar, and map name. Subsequently, preset prompt words are used to extract key metadata for each component using multimodal large model optical character recognition. The key metadata includes color coding, symbol name, geological unit description, spatial location, and map sheet number, and the key metadata retains its contextual semantics.
[0018] Furthermore, in step S03, in order to achieve semantic enhancement of metadata, two retrieval mechanisms are introduced based on the GraphRAG framework. The first retrieval mechanism is a local retrieval for precise entity queries, and the second retrieval mechanism is a global retrieval for comprehensive concept reasoning. Local retrieval takes the user query q and the extracted geological map metadata, including entity names and attribute keywords, as input. It performs precise retrieval on specific knowledge submaps for specific questions. The specific filtering process can be represented as follows:
[0019] in These are candidate entity / relationship nodes. It is the vector embedding of the user query. It is metadata embedding. It is the vector representation of the candidate node. and The weight parameter is used to balance the similarity between the query and the metadata. The top-k nodes and triples retrieved are sorted and filtered and then integrated into a structured context, and the output is semantically enhanced metadata. Global retrieval addresses macro-level issues, performing retrieval and reasoning across communities. The process can be divided into two stages: First, the Map phase: selecting relevant community sets based on input metadata. Generate intermediate answers for each community. ; Secondly, the Reduce phase: [This involves] all [elements / items]. After filtering by the input scoring model, the data is integrated into the final enhanced metadata according to the weights. The scoring model can be expressed as: in It's an intermediate answer. Embedded vector, This is the usefulness score for the answer. These are weight parameters that ultimately sort the highest-scoring intermediate answers by weight and integrate them into the context, generating enhanced metadata with a global perspective.
[0020] Furthermore, in step S04, the multimodal large model is GraphAgent, and the specific steps are as follows: First, based on the user's query intent, the metadata is filtered and standardized, and the core geological elements related to the query semantics are extracted as search conditions. Then, these core metadata are used as query inputs, and semantic retrieval is performed in the constructed geological knowledge graph to obtain entities and relationships closely related to the elements in the graph, as well as subgraph structures.
[0021] The principle and effect of this solution are as follows: 1. Compared with existing technologies, this method deeply integrates the structured knowledge representation and organization capabilities of knowledge graphs, the knowledge retrieval mechanism of RAG, and the cross-modal semantic understanding and reasoning capabilities of multimodal large models. By constructing geological reports as knowledge graphs, and based on metadata extracted from geological maps, entity relationship subgraphs are retrieved in the knowledge graphs to enhance the retrieval of metadata. Finally, through the multimodal fusion of geological maps and enhanced metadata retrieval, the multimodal large model's ability to understand professional elements and geological processes in geological maps is improved.
[0022] 2. Compared with existing technologies, this method effectively solves the limitations of traditional manual interpretation relying on expert experience and existing single-modal processing methods in terms of professional semantic understanding. Specifically, by constructing a geological knowledge graph, it systematically integrates and structures professional background knowledge in the geosciences field, solving the problems of lack of systematic organization and weak semantic connections in domain knowledge; by utilizing Retrieval Enhancement Generation (RAG) technology to achieve cross-modal knowledge retrieval and context enhancement, it breaks through the bottleneck of traditional retrieval methods being limited to a single modality and lacking dynamic knowledge fusion; by leveraging the powerful perception and reasoning capabilities of the multimodal large model, it achieves deep fusion of image and text information, solving the problems of separation of geological map and text elements perception and insufficient deep semantic reasoning. This solution significantly improves the accuracy, reliability, and interpretability of the multimodal large model in identifying professional elements, analyzing spatial relationships, and inferring mineralization regularities in geological maps. Attached Figure Description
[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1 The illustration shows a flowchart of a method for enhancing the understanding of image-format geological maps based on a multimodal large model, as proposed in an embodiment of this application. Figure 2 The flowchart of the knowledge graph construction based on GraphRAG in the image format geological map enhancement understanding method based on multimodal large model proposed in this application is shown. Figure 3 This illustration shows a geological ontology model diagram in an image-format geological map enhancement understanding method based on a multimodal large model proposed in an embodiment of this application. Figure 4 This illustration shows a geological map metadata extraction diagram in an image-format geological map enhancement understanding method based on a multimodal large model proposed in an embodiment of this application. Figure 5 The illustration shows an example of geological map detection and legend unit detection in a method for enhancing understanding of image-format geological maps based on a multimodal large model proposed in this application. Figure 6 This illustration shows a schematic diagram of metadata knowledge graph retrieval enhancement in a method for enhancing the understanding of image-format geological maps based on a multimodal large model, as proposed in an embodiment of this application. Figure 7 The flowchart illustrating the metadata retrieval-enhanced geological map understanding process in an image-format geological map enhancement understanding method based on a multimodal large model proposed in this application is shown. Figure 8 The illustration shows an example of the question-answering effect of a method for enhancing understanding of image-format geological maps based on a multimodal large model, as proposed in this application. Detailed Implementation
[0025] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.
[0026] Implementation, for example Figures 1-8 As shown: A method for enhancing the understanding of image-format geological maps based on multimodal large models includes the following steps:
[0027] S01: Geoscience Knowledge Graph Construction: Geological entities and relationships are extracted from geological corpora, professional terms and their semantic associations are identified using a large language model, vector indexes are constructed and community summaries are generated to form a geological knowledge graph that supports complex semantic reasoning, and the graph is persistently stored in the Neo4j graph database; after the geoscience knowledge graph is constructed, knowledge graph indexes and communities are built based on the geoscience knowledge graph.
[0028] Specifically: In step S01, the method for constructing the geoscience knowledge graph is as follows: First, a large language model is used to identify geological entities from unstructured text and extract semantic relationships between entities, and a knowledge network with deep associations is constructed; this knowledge network is a triplet knowledge network of head entity-relationship-tail entity.
[0029] Then, based on the GraphRAG framework, the geological corpus is further divided into semantic units using an intelligent segmentation algorithm to segment the text into blocks. This intelligent segmentation algorithm differs from fixed-length English segmentation methods; it incorporates HanLP word segmentation, sentence boundary detection, and paragraph structure analysis into the word segmentation and sentence breaking stages, taking into account the linguistic characteristics and domain requirements of the Chinese geological corpus, to ensure consistency between segmentation boundaries and professional semantic boundaries. Simultaneously, a sliding window mechanism is used to retain cross-block entity and relational information, and an adaptive segmentation strategy is employed for extremely long paragraphs, thereby improving retrieval efficiency while maintaining semantic integrity. Figure 3 As shown, entity relations are finally extracted from the block results, and relation triples are constructed based on the triple knowledge network of head entity-relation-tail entity.
[0030] Finally, semantically similar entities are merged through an entity alignment mechanism, and the geological knowledge graph data is persistently stored in the Neo4j graph database.
[0031] See appendix for details Figure 2 and Figure 3 Within the GraphRAG framework, the geological corpus is first divided into multiple text blocks, and entity relations are extracted based on a head entity-relation-tail entity triple knowledge network to construct corresponding knowledge triples. Subsequently, a knowledge graph is built based on these triples, and an index and community structure are further generated. The specific steps are as follows: First, a vectorized index of entity nodes is constructed, semantic communities are divided according to the topological structure of the geological knowledge graph, and a high-level semantic community summary is generated using a large language model. Then, under the retrieval enhancement generation mechanism, the geological knowledge map is combined with the vector index to construct a knowledge enhancement mechanism that supports multi-granularity retrieval; The large language model uses structured prompts to guide its execution, specifically as follows: Entity recognition and standardization: Entities are identified and normalized based on predefined types, their names, types, and descriptions are standardized, and synonym expressions are unified; Relationship recognition and classification: Semantic associations are established between entities, and they are classified into predefined relationship types based on context, generating relationship descriptions and strength scores; Quality control and deduplication: Consistency, uniqueness, and completeness checks are performed on entities and relationships through rule constraints, duplicate entities are merged, and ambiguous referential phenomena are eliminated.
[0032] Due to the complex structure of geological corpora and the ambiguity of technical terms, semantically redundant entity nodes are prone to appear during the map construction process. For example, different expressions such as "a medium-sized copper-cobalt deposit," "a copper-cobalt mining area," and "a mining occurrence" all refer to the same geological object. Such redundant nodes not only increase the size of the map but also affect the accuracy of retrieval and inference; therefore, entity alignment is necessary to address this issue.
[0033] Therefore, an entity alignment mechanism was designed to merge semantically similar entities.
[0034] Specifically as follows: The entity alignment mechanism is as follows: A01: First, merge: Numerical encoding of entity semantic information is performed. The bge-m3 embedding model is deployed locally in Ollam to encode entity names, attributes, and context, generating 1024-dimensional semantic vectors. Large-scale semantic similarity retrieval is achieved by leveraging the FAISS vector index library. In this vector space, semantically similar entities exhibit high cosine similarity, calculated as follows: In the formula, and Entities and The embedding vector, where · represents the vector dot product. This represents the magnitude of the vector. The closer this value is to 1, the more semantically similar the two entities are. A threshold of 0.95 is set; entity pairs exceeding this threshold will be merged.
[0035] A02: Further Optimization: The Leiden algorithm is introduced to conduct community detection, in order to identify semantically closely related sub-communities in the geological knowledge graph. The Leiden algorithm is based on modularity optimization, and the modularity calculation formula is as follows: In the formula, A ij = represents the connection weight between node i and node j in the adjacency matrix, k is the number of nodes. i k j Let m be the degree of node i and node j, m be the total weight of the edges in the graph, and c be the degree of node i and node j. i and c j For the community to which the node belongs, It is the Kronecker function (1 when ci=cj, 0 otherwise), which can effectively decompose large-scale geological knowledge maps and maintain high connectivity within communities.
[0036] The specific steps for generating high-level semantic community summaries from a large language model are as follows: First, priorities are assigned to entities within the community, calculated as follows:
[0037] In the formula, deg(e) represents the node degree, PR(e) represents the PageRank value, Cen(e) represents the centrality index, and α, β, and γ are all weight parameters that satisfy α+β+γ=1.
[0038] Then, an iterative information aggregation strategy is adopted to input representative entities, relationships, and descriptive text within the community into a large language model to generate a semantic summary. When the community size exceeds the model context limit, the system adopts a dynamic truncation mechanism. In the formula, This indicates that the final community summary has been obtained. Represents core entity information. For context to be added, Indicates the maximum context length of the LLM. The function is used to truncate on demand.
[0039] Through the above steps, the present invention can realize the incremental construction and dynamic updating of knowledge graphs. Newly introduced geological corpora can be quickly integrated into the existing structure, ensuring that the knowledge graph is continuously enhanced in terms of coverage, semantic consistency and application effectiveness.
[0040] S02: Geological map metadata extraction: A pre-trained visual model is used to parse and decompose the components of the image-format geological map, and to accurately extract key metadata from the image-format geological map.
[0041] Specifically, addressing the multi-layered and heterogeneous information structure within geological maps during the component parsing stage, this invention employs a pre-trained YOLOv10 model for geological map detection and legend unit detection. The entire geological map is viewed as a hierarchical tree structure rooted at the map itself, and decomposed from top to bottom into key components such as the legend area, main map area, scale bar, and map title. Subsequently, pre-defined prompts are used to extract metadata for each component using multimodal large-scale optical character recognition, including key metadata such as color coding, symbol names, geological unit descriptions, spatial locations, and map sheet numbers, while preserving their contextual semantics to ensure the integrity of geological terminology and professional structures. The processing flow is as follows: Figure 4 As shown, the results of geological map detection and legend unit detection are as follows: Figure 5 As shown.
[0042] S03: Metadata Retrieval Enhancement: Based on key metadata extracted from image-format geological maps, entity and relationship subgraph retrieval is performed in the constructed geological knowledge graph. The image content and geological background knowledge are integrated through the retrieval enhancement generation mechanism to form an enhanced semantic context.
[0043] Specifically, in step S03, in order to achieve semantic enhancement of metadata, two retrieval mechanisms are introduced based on the GraphRAG framework. The first retrieval mechanism is a local retrieval for precise entity queries, and the second retrieval mechanism is a global retrieval for comprehensive concept reasoning.
[0044] Local retrieval takes the user query q and the extracted geological map metadata, including entity names and attribute keywords, as input. It performs precise retrieval on specific knowledge submaps for specific questions. The specific filtering process can be represented as follows: in These are candidate entity / relationship nodes. It is the vector embedding of the user query. It is metadata embedding. It is the vector representation of the candidate node. and The weighting parameter is used to balance the similarity between the query and the metadata. The top-k nodes and triples retrieved are sorted and filtered before being integrated into a structured context, and the output is semantically enhanced metadata. Global retrieval, on the other hand, addresses macro-level issues and performs retrieval and reasoning across communities. The process can be divided into two stages: First, the Map phase: selecting relevant community sets based on input metadata. Generate intermediate answers for each community. ; Secondly, the Reduce phase: [This involves] all [elements / items]. After filtering by the input scoring model, the data is integrated by weight into the final enhanced metadata.
[0045] The scoring model can be expressed as: in It's an intermediate answer. Embedded vector, This is the usefulness score for the answer. These are weight parameters that ultimately sort the highest-scoring intermediate answers by weight and integrate them into the context, generating enhanced metadata with a global perspective.
[0046] By combining local and global retrieval, this method can not only meet the high-precision response to specific entity attribute problems, but also support macro-reasoning tasks such as mineralization processes and evolution trends in geological maps, thereby comprehensively improving the performance and interpretability of multimodal large models in geological map understanding.
[0047] S04: Multimodal Large Model Understanding: The retrieved enhanced metadata and image-format geological maps are input into the multimodal large model to drive cross-modal semantic alignment and deep reasoning.
[0048] Specifically, after obtaining geological map metadata, this invention first filters and standardizes the metadata based on the user's query intent, extracting core geological elements relevant to the query semantics as search conditions. Subsequently, these core metadata elements are used as query input to perform semantic retrieval within the constructed geological knowledge graph, obtaining entities, relationships, and subgraph structures closely related to elements in the map, thus injecting rich domain background knowledge into the geological map understanding task. Unlike traditional text-based retrieval methods, this invention introduces GraphAgent as a retrieval scheduling module. GraphAgent can invoke different retrieval strategies according to query requirements and fuse and reorder the retrieval results, thereby balancing the accuracy of fact-finding queries with the systematic nature of reasoning tasks. Its advantages are: on the one hand, avoiding knowledge gaps or redundancy caused by a single retrieval strategy; on the other hand, effectively supplementing the implicit professional context in the geological map through cross-community, multi-hop semantic subgraph integration, such as... Figure 7 As shown. The search results are organized in the form of a structured subgraph, containing information such as entity attributes, association paths, and community summaries, forming a semantic context that enhances the search. To formally describe this process, the multimodal context fusion formula is defined as follows: in, This represents a set of metadata extracted from geological maps. For user queries, This represents the retrieval results obtained by fusing LocalSearch and GlobalSearch after scheduling by GraphAgent. The Fusion function realizes the semantic alignment and context concatenation of image metadata, user query and knowledge graph retrieval results to obtain the final multimodal context C. This context C is fed in as input and outputs MMLM to infer the answer. The overall logic of GraphAgent can be abstracted into pseudocode, as shown in Table 1 below.
[0049]
[0050] Table 1
[0051] By introducing GraphAgent, this invention achieves unified scheduling and result fusion for different retrieval strategies, maintaining accurate response to factual elements of geological maps while significantly enhancing reasoning and interpretation capabilities in complex contexts.
[0052] For example, this invention uses exploration data and geological maps of a gold deposit as an example to systematically verify the effectiveness of the proposed method. This case employs GraphRAG technology to construct a knowledge graph from the geological report, and uses a pre-trained YOLOv10 model to automatically segment the geological map and identify metadata. Finally, it calls the GPT-4o multimodal API to achieve intelligent question answering on the geological map.
[0053] Taking a report related to a "gold deposit" as an example, the system automatically extracts entities such as "gold deposit," "Indosinian granodiorite body," and "fault structure," as well as relationships such as "mineralization control," "originating from," and "formed from," to form a local geological knowledge subgraph, which is then imported into Neo4j for visual browsing and structured querying. Examples of the resulting geological ternary sets are shown in Table 2, and the entity information stored in Neo4j is shown in Table 3.
[0054] Specifically as follows:
[0055]
[0056] Table 2
[0057]
[0058] Table 3
[0059] like Figure 8 As shown, the question-answering performance exhibits a significant difference as knowledge is injected: when only the large model is used (without enhancement), the answers are relatively general, failing to focus on key elements in the geological map and accurately capturing the core of the question; after introducing metadata extracted from the map, the model identifies key information such as "South China Plate" and "fault and fold structure," significantly improving the relevance of the answers; finally, by retrieving and enhancing semantic subgraphs related to metadata from the knowledge graph, the model not only integrates deep background knowledge but also demonstrates a complete logical reasoning chain of "tectonic evolution—magmatic activity—metallogenic background—exploration direction," achieving optimal levels of structure, accuracy, and geological rationality in the answers.
[0060] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.
Claims
1. A method for enhancing the understanding of image-format geological maps based on a multimodal large model, characterized in that, Includes the following steps: S01: Geological Knowledge Graph Construction: Geological entities and relationships are extracted from geological corpora, professional terms and their semantic associations are identified using a large language model, vector indexes are constructed and community summaries are generated to form a geological knowledge graph that supports complex semantic reasoning, and the graph is persistently stored in the Neo4j graph database. After constructing the geoscience knowledge graph, a knowledge graph index and community are built based on the geoscience knowledge graph; S02: Geological map metadata extraction: A pre-trained visual model is used to perform component parsing and structural decomposition on image-format geological maps to accurately extract key metadata from image-format geological maps; S03: Metadata retrieval enhancement: Based on key metadata extracted from image-format geological maps, entity and relationship subgraph retrieval is performed in the constructed geological knowledge graph. The image content and geological background knowledge are integrated through the retrieval enhancement generation mechanism to form an enhanced semantic context. S04: Multimodal Large Model Understanding: The retrieved enhanced metadata and image-format geological maps are input into the multimodal large model to drive cross-modal semantic alignment and deep reasoning.
2. The method for enhancing the understanding of image-format geological maps based on a multimodal large model according to claim 1, characterized in that, In step S02, the key metadata in the image-format geological map includes the geological map name, legend symbols, and structural annotations.
3. The method for enhancing the understanding of image-format geological maps based on a multimodal large model according to claim 2, characterized in that, In step S01, the method for constructing the geoscience knowledge graph is as follows: First, a large language model is used to identify geological entities from unstructured text and extract semantic relationships between entities, and a knowledge network with deep associations is constructed; this knowledge network is a triplet knowledge network of head entity-relationship-tail entity. Then, based on the GraphRAG framework, the geological corpus is divided into semantic units using an intelligent block partitioning algorithm to perform text block partitioning and obtain text blocks after text block partitioning. Entity relations in the text blocks are extracted, and relation triples are built based on the triple knowledge network of head entity-relation-tail entity. Finally, semantically similar entities are merged through an entity alignment mechanism, and the geological knowledge graph data is persistently stored in the Neo4j graph database.
4. The method for enhancing the understanding of image-format geological maps based on a multimodal large model according to claim 3, characterized in that, The steps to build a knowledge graph index and community are as follows: First, a vectorized index of entity nodes is constructed, semantic communities are divided according to the topological structure of the geological knowledge graph, and a high-level semantic community summary is generated using a large language model. Then, under the retrieval enhancement generation mechanism, the geological knowledge map is combined with the vector index to construct a knowledge enhancement mechanism that supports multi-granularity retrieval; The large language model uses structured prompts to guide its execution, specifically as follows: Entity recognition and standardization: Recognize and normalize entities based on predefined entity types, standardize names, types and descriptions, and unify synonym expressions; Relationship recognition and classification: Establish semantic relationships between entities, classify them into predefined relationship types based on context, and generate relationship descriptions and strength scores. Quality control and deduplication: Consistency, uniqueness and integrity checks are performed on entities and relationships through rule constraints, duplicate entities are merged and ambiguous referencing is eliminated.
5. The method for enhancing the understanding of image-format geological maps based on a multimodal large model according to claim 4, characterized in that, The entity alignment mechanism is as follows: A01: First, merge: Numerical encoding of entity semantic information is performed. The bge-m3 embedding model is deployed locally in Ollam to encode entity names, attributes, and context, generating 1024-dimensional semantic vectors. Large-scale semantic similarity retrieval is achieved by leveraging the FAISS vector index library. In this vector space, semantically similar entities exhibit high cosine similarity, calculated as follows: In the formula, and Entities and The embedded vector, where · represents the vector dot product. This represents the magnitude of the vector. The closer this value is to 1, the more semantically similar the two entities are. A threshold of 0.95 is set, and entity pairs higher than this threshold will be merged. A02: Further Optimization: The Leiden algorithm is introduced to conduct community detection, in order to identify semantically closely related sub-communities in the geological knowledge graph. The Leiden algorithm is based on modularity optimization, and the modularity calculation formula is as follows: In the formula, A ij k represents the connection weight between node i and node j in the adjacency matrix. i k j Let m be the degree of node i and node j, m be the total weight of the edges in the graph, and c be the degree of node i and node j. i and c j For the community to which the node belongs, It is the Kronecker function (when c) i =c j (If the value is 1, otherwise it is 0). This method can effectively decompose large-scale geological knowledge maps and maintain high connectivity within the community.
6. The method for enhancing the understanding of image-format geological maps based on a multimodal large model according to claim 5, characterized in that, The specific steps for generating high-level semantic community summaries from a large language model are as follows: First, priorities are assigned to entities within the community, calculated as follows: In the formula, deg(e) represents the node degree, PR(e) represents the PageRank value, Cen(e) represents the centrality index, and α, β, and γ are all weight parameters, satisfying α+β+γ=1; Then, an iterative information aggregation strategy is adopted to input representative entities, relationships, and descriptive text within the community into a large language model to generate a semantic summary. When the community size exceeds the model context limit, the system adopts a dynamic truncation mechanism. In the formula, This indicates that the final community summary has been obtained. Represents core entity information. For context to be added, Indicates the maximum context length of the LLM. The function is used to truncate on demand.
7. A method for enhancing the understanding of image-format geological maps based on a multimodal large model, as described in claim 2 or 6, characterized in that... The pre-trained visual model is a pre-trained YOLOv10 model for geological map detection and legend unit detection. In the component parsing stage, the multi-level and heterogeneous information structure within the geological map is treated as a hierarchical tree structure with the entire map as the root node, and decomposed from top to bottom into the legend area, main map area, scale bar and map name. Then, the preset prompt words are used to extract key metadata for each component using multimodal large model optical character recognition.
8. The method for enhancing the understanding of image-format geological maps based on a multimodal large model according to claim 7, characterized in that, In step S03, in order to achieve semantic enhancement of metadata, two retrieval mechanisms are introduced based on the GraphRAG framework. The first retrieval mechanism is a local retrieval for precise entity queries, and the second retrieval mechanism is a global retrieval for comprehensive concept reasoning. Local retrieval takes the user query q and the extracted geological map metadata, including entity names and attribute keywords, as input. It performs precise retrieval on specific knowledge submaps for specific questions. The specific filtering process can be represented as follows: in These are candidate entity / relationship nodes. It is the vector embedding of the user query. It is metadata embedding. It is the vector representation of the candidate node. and The weight parameter is used to balance the similarity between the query and the metadata. The top-k nodes and triples retrieved are sorted and filtered and then integrated into a structured context, and the output is semantically enhanced metadata. Global retrieval addresses macro-level issues, performing retrieval and reasoning across communities. The process can be divided into two stages: First, the Map phase: selecting relevant community sets based on input metadata. Generate intermediate answers for each community. ; Secondly, the Reduce phase: [This involves] all [elements / items]. After filtering by the input scoring model, the data is integrated into the final enhanced metadata according to the weights. The scoring model can be expressed as: in It's an intermediate answer. Embedded vector, This is the usefulness score for the answer. These are weight parameters that ultimately sort the highest-scoring intermediate answers by weight and integrate them into the context, generating enhanced metadata with a global perspective.
9. The method for enhancing the understanding of image-format geological maps based on a multimodal large model according to claim 8, characterized in that, In step S04, GraphAgent is introduced as a retrieval scheduling module. The specific steps are as follows: First, based on the user's query intent, GraphAgent automatically filters and standardizes the metadata, extracts the core geological elements that are related to the query semantics as retrieval conditions, and then uses these core metadata as query inputs to perform semantic retrieval in the constructed geological knowledge graph to obtain entities, relationships and subgraph structures that are closely related to the elements in the graph.