A multi-source heterogeneous geological data integration management method and system
Patent Information
- Application Number
- CN202611200221.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-10
- Publication Date
- 2026-10-30
AI Technical Summary
[0006]鉴于此，本发明的目的在于提供一种多源异构地质数据集成治理方法及系统，已解决或至少部分解决现有技术中所存在的上述问题
本发明提出一种多源异构地质数据集成治理方法及系统，通过对多源异构地质数据进行异常检测，提升数据质量，避免因异常值影响治理精度，通过构建异构地质数据模型实现不同来源多类型结构化地质数据之间的有效关联，通过构建异构图数据结构实现对地质数据集成治理效果的评估，避免影响地质业务的分析精度。本发明有效解决地质数据多源异构、地质数据异常等问题，显著提升地质数据集成效率与数据利用率。
Smart Images
Figure CN123011857A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data integration and data governance technology, and in particular to a method and system for integrating and governing multi-source heterogeneous geological data. Background Technology
[0002] Geological survey projects constructed at different times have dynamically generated a large amount of structured, semi-structured, and unstructured geological data, which vary significantly in terms of data structure, data storage methods and formats, and data environment. Faced with multi-source heterogeneous data with high update frequency and complex and ever-changing structure, traditional data integration methods suffer from problems such as poor scalability, low real-time performance, and high cost.
[0003] Existing geological data governance practices suffer from several shortcomings: First, data source access adaptability is weak; traditional governance tools only support access to a single type of structured data and cannot simultaneously support multiple types of structured geological data. Second, there is a lack of anomaly detection for multi-source heterogeneous geological data before integration, leading to outliers during the integration process. Third, heterogeneous data fusion is difficult; similar geological data suffer from multiple field naming and semantic ambiguity issues, and the lack of dynamic association between different types of structured geological data makes data fusion conflicts highly likely. Fourth, data governance effectiveness is poor; existing governance methods only perform simple checks on data integrity and lack the ability to calculate the integrated governance effect of geological data based on user task instructions, affecting the accuracy of subsequent geological business analysis.
[0004] Furthermore, existing general data governance technologies face the following proprietary challenges when applied to geological data: First, the strong spatiotemporal correlation of multimodal geological data: the spatial correlation (coordinate system, scale, stratigraphic correlation) of borehole data, geophysical data, and geochemical data is the core of geological data fusion, and general solutions lack the ability to understand geological spatial semantics; Second, the confusion of multi-source naming of geological terms: the same lithology has a large number of synonyms under different geological survey specifications (such as the CGS standard), and general semantic alignment methods are not applicable to the geological terminology system; Third, the trend sensitivity of geological anomaly detection: geological data (such as borehole depth and stratigraphic thickness) have significant trend changes, and the direct threshold method has a high misjudgment rate.
[0005] This invention addresses the unique challenges of the aforementioned geological data by designing a dedicated heterogeneous geological data model and semantic association mechanism, rather than a simple migration and application of a general data governance framework. Summary of the Invention
[0006] Therefore, the purpose of this invention is to provide a method and system for the integrated management of multi-source heterogeneous geological data, which has solved or at least partially solved the above-mentioned problems existing in the prior art.
[0007] To achieve the above objectives, the first aspect of the present invention provides a method for integrating and managing multi-source heterogeneous geological data, the method comprising the following steps: S101. Based on geological survey projects, obtain multi-source heterogeneous geological data and perform anomaly detection on the multi-source heterogeneous geological data; S102. The multi-source heterogeneous geological data that has undergone anomaly detection is classified into multiple types of structured geological data according to the data storage structure. S103. Perform preprocessing operations on various types of structured geological data, including data cleaning, format standardization, and redundancy removal. S104. Construct a heterogeneous geological data model and use the heterogeneous geological data model to dynamically associate multiple types of structured geological data to form an integrated governance geological dataset. S105. Construct a heterogeneous map data structure based on the integrated governance geological dataset, and calculate the integrated governance effect of the integrated governance geological dataset based on the heterogeneous map data structure.
[0008] Furthermore, the multi-type structured geological data includes structured geological data, semi-structured geological data, and unstructured geological data. The structured geological data is geological data with a fixed data table structure and standardized field definitions. The semi-structured geological data is geological data with a variable custom field structure, no fixed data table paradigm, containing self-descriptive tags, labels, or hierarchical structures, and whose field information can be parsed and extracted. The unstructured geological data is geological data without a fixed data structure, without a standardized field system, and stored in binary stream or text format.
[0009] Furthermore, the anomaly detection of multi-source heterogeneous geological data specifically includes the following steps: S11. Obtain historical multi-source heterogeneous geological data sequences based on geological survey projects; S12. Construct a multi-source heterogeneous geological data sequence based on historical multi-source heterogeneous geological data sequences and multi-source heterogeneous geological data; S13. Perform finite difference on the multi-source heterogeneous geological data sequence to obtain a differential geological data sequence, including historical differential geological data sequences and current differential geological data sequences; S14. Calculate the mean and standard deviation of the historical difference geological data series; S15. Based on the mean and standard deviation of historical differential geological data sequences, the three sigma criterion is introduced to determine whether the current differential geological data sequence is an outlier.
[0010] Furthermore, the construction of the heterogeneous geological data model specifically includes the following steps: S21. Collect metadata from each data source in the geological survey project, parse the metadata, and obtain the primary key of each data source; S22. Based on the parsed metadata, analyze the different semantic expressions of the same geological data concepts in geological survey projects, identify semantic conflicts, and establish a unified semantic description; S23. Based on a unified semantic description, a heterogeneous geological data model is constructed. The heterogeneous geological data model is used to define entities and their relationships within a geological survey project, as well as to define the attributes, primary keys, foreign keys, and relationships between entities.
[0011] Furthermore, the method of dynamically associating multiple types of structured geological data using a heterogeneous geological data model specifically includes the following steps: S31. Use some entities within the geological survey project as the first associated primary key, and establish the correspondence between the first associated primary key and the primary keys of each data source in the geological survey project. S32. Based on the heterogeneous geological data model, obtain entities from multiple types of structured geological data, and use some entities as the second association primary key; S33. Establish a mapping relationship between the second associated primary key and the first associated primary key, generate a mapping relationship table, and integrate multiple types of structured geological data based on the mapping relationship table to form an integrated governance geological dataset.
[0012] Furthermore, the heterogeneous graph data structure includes a node set and an edge set. The node set is the entity set of the integrated governance geological dataset, and the edge set is the set of semantic edges. The semantic edges are constructed based on a mapping table and are used to find entities in the integrated governance geological dataset that have semantic equivalence relationships in the heterogeneous graph data structure and connect them.
[0013] Furthermore, the calculation of the integrated governance effect of the integrated governance geological dataset based on the heterogeneous map data structure specifically includes the following steps: S41. Obtain the user's task instructions, convert the task instructions into task vectors, and introduce the task vectors into the heterogeneous graph data structure; S42. Obtain the semantic information of the task vector, and interact the semantic information with the semantic edges to calculate the importance weight of the semantic edges to the task vector. S43. Based on importance weights, obtain the entity set of the integrated governance geological dataset corresponding to the task vector through semantic edges; S44. Based on the entity set of the integrated governance geological dataset, detect whether the corresponding geological data in the integrated governance geological dataset has been successfully obtained. If the data is successfully obtained, it means that the integrated governance has been successful. Otherwise, mark the entity set of the integrated governance geological dataset and dynamically associate the marked entity set again.
[0014] A second aspect of the present invention provides a multi-source heterogeneous geological data integration and management system, the system comprising: Anomaly detection module: used to acquire multi-source heterogeneous geological data and perform anomaly detection on the multi-source heterogeneous geological data; The structure partitioning module is used to partition multi-source heterogeneous geological data that has undergone anomaly detection into multiple types of structured geological data according to the data storage structure. Data processing module: used for preprocessing various types of structured geological data, including data cleaning, format standardization and redundancy removal; Data association module: used to build heterogeneous geological data models, and dynamically associate multiple types of structured geological data using heterogeneous geological data models to form an integrated governance geological dataset; Integrated governance assessment module: This module is used to construct a heterogeneous map data structure based on the integrated governance geological dataset, and to calculate the integrated governance effect of the integrated governance geological dataset based on the heterogeneous map data structure.
[0015] Compared with the prior art, the beneficial effects of the present invention are: This invention proposes a method and system for integrating and managing multi-source heterogeneous geological data. By detecting anomalies in the multi-source heterogeneous geological data, it improves data quality and avoids the impact of outliers on the accuracy of data management. It achieves effective correlation between structured geological data of different types from different sources by constructing a heterogeneous geological data model, and evaluates the effectiveness of geological data integration and management by constructing a heterogeneous map data structure, thus avoiding any impact on the analytical accuracy of geological operations. This invention effectively solves the problems of multi-source heterogeneity and geological data anomalies, significantly improving the efficiency of geological data integration and data utilization. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only preferred embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 This is a schematic diagram of a multi-source heterogeneous geological data integration and management method provided in an embodiment of the present invention; Figure 2 A schematic diagram of the overall structure of a multi-source heterogeneous geological data integration and management system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the anomaly detection process for multi-source heterogeneous geological data provided in an embodiment of the present invention; Figure 4 A schematic diagram illustrating the process of constructing a heterogeneous geological data model provided in an embodiment of the present invention; Figure 5This is a schematic diagram illustrating the process of dynamically associating multiple types of structured geological data using a heterogeneous geological data model, as provided in an embodiment of the present invention. Figure 6 This is a schematic diagram illustrating the process of calculating the integrated governance effect of the integrated governance geological dataset based on the heterogeneous graph data structure, as provided in an embodiment of the present invention. Detailed Implementation
[0018] The principles and features of the present invention are described below with reference to the accompanying drawings. The listed embodiments are only used to explain the present invention and are not intended to limit the scope of the present invention.
[0019] Reference Figure 1 This embodiment provides a method for integrating and managing multi-source heterogeneous geological data, the method comprising the following steps: S101. Based on geological survey projects, obtain multi-source heterogeneous geological data and perform anomaly detection on the multi-source heterogeneous geological data; S102. The multi-source heterogeneous geological data that has undergone anomaly detection is classified into multiple types of structured geological data according to the data storage structure. S103. Perform preprocessing operations on various types of structured geological data, including data cleaning, format standardization, and redundancy removal. S104. Construct a heterogeneous geological data model and use the heterogeneous geological data model to dynamically associate multiple types of structured geological data to form an integrated governance geological dataset. S105. Construct a heterogeneous map data structure based on the integrated governance geological dataset, and calculate the integrated governance effect of the integrated governance geological dataset based on the heterogeneous map data structure.
[0020] In this embodiment, anomaly detection is first performed on the acquired multi-source heterogeneous geological data (such as geophysical exploration, geochemical exploration, and other multi-source heterogeneous geological data types) to remove invalid geological anomalies in advance, preventing them from participating in subsequent processes and affecting the accuracy of geological data integration and governance. The multi-source heterogeneous geological data is then divided into multiple types of structured geological data and preprocessed to improve the consistency and efficiency of multi-source heterogeneous geological data integration, unify the storage format and data benchmark of the multiple types of structured geological data, and reduce subsequent structural conflicts and correlation matching errors.
[0021] Simultaneously, by utilizing heterogeneous geological data models to dynamically correlate multiple types of structured geological data, effective association between structured geological data from different sources can be achieved, forming an integrated governance geological dataset. This unifies the field specifications, semantic definitions, and structural standards of multiple types of structured geological data. Finally, the integrated governance effect is calculated by constructing heterogeneous map structure data, and the integrated governance geological dataset is judged based on the integrated governance effect to determine whether it meets the integrated governance requirements.
[0022] In step S101, the acquisition of multi-source heterogeneous geological data based on geological survey projects specifically includes: for structured geological data (such as borehole logging data and core analysis data) stored in relational databases, batch extraction or incremental synchronization is performed through the JDBC / ODBC standard interface; for unstructured geological data such as geological survey reports and drilling columnar sections, access to the unstructured geological data is achieved through the FTP protocol; for data sources without access permissions, a file upload interface is used to access the unstructured geological data, and the client information of the accessed unstructured geological data is parsed and converted into a text model; finally, the file upload interface is called to pass in the text model to obtain the unstructured geological data from data sources without access permissions; for semi-structured geological data in GeoJSON and XML formats, access is achieved through a RESTful API or file parsing channel.
[0023] Reference Figure 3 In step S101, the anomaly detection of multi-source heterogeneous geological data specifically includes the following steps: S11. Obtain historical multi-source heterogeneous geological data sequences based on geological survey projects; S12, Based on historical multi-source heterogeneous geological data sequences Compared with current multi-source heterogeneous geological data Constructing multi-source heterogeneous geological data sequences ; S13. Perform finite difference on the multi-source heterogeneous geological data sequence to obtain a differential geological data sequence, including historical differential geological data sequences and current differential geological data sequences; The finite difference method for multi-source heterogeneous geological data sequences is represented as follows:
[0024] in, For the first A differential geological data sequence, For the first Multi-source heterogeneous geological data, For the first Multi-source heterogeneous geological data.
[0025] S14. Calculate the mean and standard deviation of the historical difference geological data series. The mean is expressed as follows:
[0026] in, The mean of the historical differential geological data series. This represents the total number of multi-source heterogeneous geological data.
[0027] The standard deviation is expressed as follows:
[0028] in, This represents the standard deviation of the historical differential geological data series.
[0029] S15. Based on the mean and standard deviation of historical differential geological data sequences, the three sigma criterion is introduced to determine whether the current differential geological data sequence is an outlier. The outlier judgment condition is expressed as follows:
[0030] For multi-source heterogeneous geological data that are identified as outliers, the following differentiated handling strategies are implemented: For occasional noise points, linear interpolation is used to replace and repair them with the weighted average of adjacent historical data; for persistent outlier segments (such as k consecutive data points that meet the outlier judgment criteria), the entire segment is marked as invalid and removed, and recorded as an anomaly log for subsequent source tracing.
[0031] In this embodiment, historical multi-source heterogeneous geological data sequences are first obtained based on geological survey projects. The trends of these historical data sequences are then analyzed to avoid affecting the accuracy of anomaly detection. Finally, using these historical sequences as a starting point, the three-sigma criterion is introduced, combining the mean and standard deviation to determine if outliers exist. If outliers are found, the corresponding data are removed to prevent them from impacting subsequent processes.
[0032] In step S102, the multi-type structured geological data includes structured geological data, semi-structured geological data, and unstructured geological data. The structured geological data is geological data with a fixed data table structure and standardized field definitions. The semi-structured geological data is geological data with a variable custom field structure, no fixed data table paradigm, containing self-descriptive tags, labels, or hierarchical structures, and whose field information can be parsed and extracted. The unstructured geological data is geological data without a fixed data structure, without a standardized field system, and stored in binary stream or text format.
[0033] In this embodiment, by dividing multi-source heterogeneous geological data into multiple types of structured geological data, the problems of poor adaptability and low processing accuracy in unified processing of multi-source heterogeneous geological data are effectively solved. Furthermore, it enables corresponding preprocessing operations for different types of structured geological data, avoiding data loss and formatting issues caused by using a uniform preprocessing operation. Simultaneously, it provides a well-organized data foundation for subsequent dynamic correlation, reducing fusion conflicts between multiple types of structured geological data.
[0034] In step S103, the preprocessing operation for the multi-type structured geological data specifically includes the following: Data cleaning is performed on structured geological data. The data cleaning mainly includes the following operations: Operation 1: Handling empty values: Mark empty values in required fields or fill them with default values; Operation 2: Duplicate value handling, based on primary key deduplication, retaining the latest version record; Step 3: Standardize the format; ensure that fields such as date, code, and unit have the same format. Step 4: Data cleaning, standardization, and transformation based on regular expressions.
[0035] The preprocessing operation for unstructured geological data is shown below: (1) Use parsing tools to parse the document content; (2) The original file is stored in the HDFS storage architecture, and the file metadata and parsed text content are stored in the ES search engine; (3) Establish the mapping relationship between documents and entities; (4) A full-text search index is built based on the ES search engine to support high-speed keyword retrieval.
[0036] Preprocessing is performed on semi-structured geological data, including cleaning up invalid and garbled characters, parsing hierarchical structures, removing redundant fields, unifying the parsing into a standard format, and initially completing missing core geological business fields to extract effective geological business information. Furthermore, spatial coordinate system unification processing and old / new map sheet number conversion are performed on the semi-structured geological data. Specifically, spatial coordinate system unification processing is implemented for semi-structured geological data of latitude and longitude from different sources, based on a third-party conversion interface, to achieve unified processing of coordinate data, including the following: Configure coordinate transformation interfaces for geological survey projects and incorporate them into automatic scheduling; Query the total number of latitude and longitude records for a specific year in semi-structured geological data, obtain the vernier records of the latitude and longitude table, and set up batch nodes; The latitude and longitude coordinates are read from the local database by batch nodes based on the cursor records; The latitude and longitude conversion interface is called repeatedly based on latitude and longitude coordinates; Obtain the JSON data returned by the latitude and longitude conversion interface, parse it, and extract the key latitude and longitude coordinate fields from the semi-structured geological data; Preprocessing operations such as data cleaning and transformation are performed on key fields of latitude and longitude coordinates to standardize semi-structured geological data of latitude and longitude from different sources.
[0037] The conversion of old and new map sheet numbers specifically involves converting old and new map sheet numbers from semi-structured geological data from different sources into a unified standard, classifying the scale of old map sheet numbers into preset categories, and converting old map sheet numbers into new map sheet numbers according to formulas for converting old map sheet numbers to new map sheet numbers at different scales.
[0038] Reference Figure 4 In step S104, the construction of the heterogeneous geological data model specifically includes the following steps: S21. Collect metadata from each data source in the geological survey project and parse the metadata, including parsing data structure, field definition, data type, etc., to obtain the primary key of each data source; S22. Based on the parsed metadata, analyze the different semantic expressions of the same geological data concepts in geological survey projects, identify semantic conflicts, and establish a unified semantic description, specifically including the following: First, preprocessing operations are performed on the field names and descriptions in the parsed metadata of each data source, including case unification and removal of special characters. Second, based on a pre-trained geological domain word vector model, the semantic similarity between similar fields from different data sources is calculated; when the semantic similarity between two fields exceeds a preset threshold and the field names are inconsistent, they are identified as semantically conflicting field pairs. Finally, national geological standards (GB / T9649 Geological and Mineral Terminology, DZ / T0079 Solid Mineral Exploration Specification, etc.) are used as authoritative semantic references to establish a unified standard semantic description for semantically conflicting field pairs, forming a semantic mapping dictionary. This semantic mapping dictionary is used to record the bidirectional mapping relationship from the original field names to standard semantic identifiers.
[0039] S23. Based on a unified semantic description, construct a heterogeneous geological data model. The heterogeneous geological data model is used to define the core business entities within the geological survey project and the relationships between entities, as well as the attributes, primary keys, foreign keys, and relationships between entities.
[0040] In this embodiment, a heterogeneous geological data model defines entities, entity attributes, and relationships between entities, facilitating the unified classification and semantic unification of entities in various types of structured geological data. Furthermore, before collecting metadata from various data sources in a geological survey project, data analysis and data source investigation can be conducted to gain a deeper understanding of geological operations and data conditions, and to clarify the sources and structures of various geological data in the project, thus aiding in the subsequent collection of metadata from each data source. The heterogeneous geological data model also features a hybrid storage structure, specifically a hybrid storage structure of Oracle / HDFS / ES / Hive / HBase. The Oracle storage structure stores structured geological data, specifically structured relational business data; the HDFS storage structure stores unstructured geological data, specifically unstructured original document files; the ES (Elasticsearch) storage structure stores the parsed content of unstructured documents and supports high-speed document content retrieval; the Hive storage structure stores large-scale geological processing and analysis data; and the HBase storage structure stores semi-structured geological data. The hybrid storage structure is used to optimally store different types of geological data and avoid data loss.
[0041] Reference Figure 5 In step S104, the heterogeneous geological data model is used to dynamically correlate multiple types of structured geological data, and geological age / stratigraphic time series constraints are added to the dynamic correlation to avoid erroneous correlation of geological data across different eras. Specifically, this includes the following steps: S31. Take the entity that is commonly used in the geological survey project and can uniquely identify the geological survey business object as the first associated primary key. The unique identifier of the geological survey business object is defined by the attributes of the entity, and establish the correspondence between the first associated primary key and the primary keys of each data source in the geological survey project. S32. Based on the heterogeneous geological data model, obtain entities from various types of structured geological data, and use commonly used entities as the second association primary key; S33. Establish a mapping relationship between the second associated primary key and the first associated primary key, generate a mapping relationship table, and integrate multiple types of structured geological data based on the mapping relationship table to form an integrated governance geological dataset.
[0042] In this embodiment, firstly, by establishing a correspondence between the first associated primary key and the primary keys of each data source, the problem of inconsistent primary keys between different data sources can be solved, avoiding erroneous associations caused by semantically consistent primary keys but inconsistent descriptions. Finally, mapping the second associated primary key to the first associated primary key allows entities in various types of structured geological data that are semantically consistent with the first associated primary key but inconsistent in description to successfully establish a mapping association with the first associated primary key, avoiding any missing associated entities, thereby forming a complete integrated governance geological dataset.
[0043] In step S105, the heterogeneous graph data structure includes a node set and an edge set. The node set is the entity set of the integrated governance geological dataset, and the edge set is a set of semantic edges. The semantic edges are constructed based on a mapping table and are used to find entities in the heterogeneous graph data structure that have semantic equivalence relationships in the integrated governance geological dataset and connect them. The heterogeneous graph data structure is represented as follows:
[0044] in, It is a heterogeneous graph data structure. For a set of nodes, It is a set of semantic edges.
[0045] In this embodiment, by constructing a heterogeneous graph data structure and establishing semantic verification pairs for the geological data in the integrated governance geological dataset at the semantic level through semantic edges, the geological data in the integrated governance geological dataset can be transformed into a heterogeneous graph data structure with entity-semantic association. Entities with the same semantics are connected through semantic edges, accurately depicting the potential associations of geological data. The structural topological relationship of geological data can be presented intuitively, providing highly correlated data support for geological-related businesses and improving the in-depth mining value of geological data and the accuracy of geological business analysis.
[0046] Reference Figure 6 In step S105, the calculation of the integrated governance effect of the integrated governance geological dataset based on the heterogeneous map data structure specifically includes the following steps: S41. Obtain the user's task instructions, convert the task instructions into task vectors, and introduce the task vectors into the heterogeneous graph data structure, specifically including the following: The task instructions are parsed, modifiers are removed, and a BERT-like pre-trained language model is used to semantically encode the task instructions based on a geological survey terminology database. The output is a multi-dimensional task vector, which is mapped to a vector space with the same dimensions as the heterogeneous graph data structure through a linear projection layer.
[0047] S42. Obtain the semantic information of the task vector, and interact the semantic information with the semantic edges to calculate the importance weight of the semantic edges to the task vector, as shown below:
[0048] in, The importance weights of semantic edges to the task vector. For inclusion Task vectors containing semantic information of the class. For inclusion Task vectors containing semantic information of the class. The amount of semantic information contained in a semantic edge. This is a matrix transpose operation. It is a global semantic vector, obtained by linear transformation of the task vector, representing the global semantic intent of the current task instruction.
[0049] S43. Based on importance weights, obtain the entity set of the integrated governance geological dataset corresponding to the task vector through semantic edges; S44. Based on the entity set of the integrated governance geological dataset, check whether the corresponding geological data in the integrated governance geological dataset has been successfully obtained. If the data is successfully obtained, it means that the integrated governance has been successful. Otherwise, mark the entity set of the integrated governance geological dataset, re-associate the marked entity set dynamically, establish a mapping association between the marked entity set and the first association primary key, update the mapping relationship table, and form a new integrated governance geological dataset based on the updated mapping relationship table. Finally, check whether the corresponding geological data has been successfully obtained based on the new integrated governance geological dataset. If it is still unsuccessful, record the marked entity set as an abnormal entity and report the abnormal entity alarm to the user.
[0050] In this embodiment, by obtaining the user's task instructions, such as sorting lithological materials, hydrogeological evaluation, and lithological identification and classification, the task instructions are converted into task vectors. Geological data related to the task instructions are obtained from the integrated governance geological dataset through semantic edges in the heterogeneous graph data structure, making it convenient for the user to obtain the corresponding sets of different structured geological data through the task instructions.
[0051] In the semantic edges, the same semantic meaning may implicitly contain multi-dimensional entity associations. Multiple semantic feature subspaces can be constructed by introducing a multi-head self-attention mechanism. These subspaces represent entity associations at different dimensions. Finally, the semantic feature subspaces are concatenated along the feature dimensions and mapped to the same space as the final semantic edge representation, ensuring that the semantic information of the semantic edge is fully associated with entities. Based on the importance weights of the semantic edges to the task vector, the multi-head self-attention mechanism filters semantic features through importance weights, thereby constructing semantic feature subspaces based on the selected semantic features, as shown below:
[0052] in, For the first A semantic feature subspace constructed from self-attention heads. For the first Semantic features selected by self-attention head filtering.
[0053] Reference Figure 2 Another embodiment of the present invention provides a multi-source heterogeneous geological data integration and management system, the system being applied to any of the multi-source heterogeneous geological data integration and management methods described in any one of the claims, the system comprising: Anomaly detection module: used to acquire multi-source heterogeneous geological data and perform anomaly detection on the multi-source heterogeneous geological data; Structure partitioning module: used to partition multi-source heterogeneous geological data that has undergone anomaly detection into multiple types of structured geological data according to the data storage structure. The multiple types of structured geological data include structured geological data, semi-structured geological data, and unstructured geological data. Data processing module: used for preprocessing various types of structured geological data, including data cleaning, format standardization, and redundancy removal; Data association module: used to build heterogeneous geological data models, and dynamically associate multiple types of structured geological data using heterogeneous geological data models to form an integrated governance geological dataset; Integrated governance evaluation module: This module is used to construct a heterogeneous graph data structure based on the integrated governance geological dataset. The heterogeneous graph data structure includes a node set and an edge set. The node set is the entity set of the integrated governance geological dataset, and the edge set is the set of semantic edges. The semantic edges are constructed based on a mapping table. The module calculates the integrated governance effect of the integrated governance geological dataset based on the heterogeneous graph data structure.
[0054] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for integrating and managing multi-source heterogeneous geological data, characterized in that, The method includes the following steps: S101. Based on geological survey projects, obtain multi-source heterogeneous geological data and perform anomaly detection on the multi-source heterogeneous geological data; S102. The multi-source heterogeneous geological data that has undergone anomaly detection is classified into multiple types of structured geological data according to the data storage structure. S103. Perform preprocessing operations on various types of structured geological data, including data cleaning, format standardization, and redundancy removal. S104. Construct a heterogeneous geological data model and use the heterogeneous geological data model to dynamically associate multiple types of structured geological data to form an integrated governance geological dataset. S105. Construct a heterogeneous map data structure based on the integrated governance geological dataset, and calculate the integrated governance effect of the integrated governance geological dataset based on the heterogeneous map data structure.
2. The method for integrating and managing multi-source heterogeneous geological data according to claim 1, characterized in that, The multi-type structured geological data includes structured geological data, semi-structured geological data, and unstructured geological data. The structured geological data is geological data with a fixed data table structure and standardized field definitions. The semi-structured geological data is geological data with a variable custom field structure, no fixed data table paradigm, containing self-describing tags, labels, or hierarchical structures, and whose field information can be parsed and extracted. The unstructured geological data is geological data without a fixed data structure, without a standardized field system, and stored in binary stream or text format.
3. The method for integrating and managing multi-source heterogeneous geological data according to claim 1, characterized in that, The anomaly detection of multi-source heterogeneous geological data specifically includes the following steps: S11. Obtain historical multi-source heterogeneous geological data sequences based on geological survey projects; S12. Construct a multi-source heterogeneous geological data sequence based on historical multi-source heterogeneous geological data sequences and multi-source heterogeneous geological data; S13. Perform finite difference on the multi-source heterogeneous geological data sequence to obtain a differential geological data sequence, including historical differential geological data sequences and current differential geological data sequences; S14. Calculate the mean and standard deviation of the historical difference geological data series; S15. Based on the mean and standard deviation of historical differential geological data sequences, the three sigma criterion is introduced to determine whether the current differential geological data sequence is an outlier.
4. The method for integrating and managing multi-source heterogeneous geological data according to claim 1, characterized in that, The construction of the heterogeneous geological data model specifically includes the following steps: S21. Collect metadata from each data source in the geological survey project, parse the metadata, and obtain the primary key of each data source; S22. Based on the parsed metadata, analyze the different semantic expressions of the same geological data concepts in geological survey projects, identify semantic conflicts, and establish a unified semantic description; S23. Based on a unified semantic description, a heterogeneous geological data model is constructed. The heterogeneous geological data model is used to define entities and their relationships within a geological survey project, as well as to define the attributes, primary keys, foreign keys, and relationships between entities.
5. The method for integrating and managing multi-source heterogeneous geological data according to claim 4, characterized in that, The method of dynamically associating multiple types of structured geological data using a heterogeneous geological data model specifically includes the following steps: S31. Use some entities within the geological survey project as the first associated primary key, and establish the correspondence between the first associated primary key and the primary keys of each data source in the geological survey project. S32. Based on the heterogeneous geological data model, obtain entities from multiple types of structured geological data, and use some entities as the second association primary key; S33. Establish a mapping relationship between the second associated primary key and the first associated primary key, generate a mapping relationship table, and integrate multiple types of structured geological data based on the mapping relationship table to form an integrated governance geological dataset.
6. The method for integrating and managing multi-source heterogeneous geological data according to claim 1, characterized in that, The heterogeneous graph data structure includes a node set and an edge set. The node set is the entity set of the integrated governance geological dataset, and the edge set is the set of semantic edges. The semantic edges are constructed based on a mapping table and are used to find entities in the heterogeneous graph data structure that have semantic equivalence relationships in the integrated governance geological dataset and connect them.
7. The method for integrating and managing multi-source heterogeneous geological data according to claim 6, characterized in that, The calculation of the integrated governance effect of the integrated governance geological dataset based on the heterogeneous map data structure specifically includes the following steps: S41. Obtain the user's task instructions, convert the task instructions into task vectors, and introduce the task vectors into the heterogeneous graph data structure; S42. Obtain the semantic information of the task vector, and interact the semantic information with the semantic edges to calculate the importance weight of the semantic edges to the task vector. S43. Based on importance weights, obtain the entity set of the integrated governance geological dataset corresponding to the task vector through semantic edges; S44. Based on the entity set of the integrated governance geological dataset, detect whether the corresponding geological data in the integrated governance geological dataset has been successfully obtained. If the data is successfully obtained, it means that the integrated governance has been successful. Otherwise, mark the entity set of the integrated governance geological dataset and dynamically associate the marked entity set again.
8. A multi-source heterogeneous geological data integration and management system, wherein the system is applied to the multi-source heterogeneous geological data integration and management method as described in any one of claims 1-7, characterized in that, The system includes: Anomaly detection module: used to acquire multi-source heterogeneous geological data and perform anomaly detection on the multi-source heterogeneous geological data; The structure partitioning module is used to partition multi-source heterogeneous geological data that has undergone anomaly detection into multiple types of structured geological data according to the data storage structure. Data processing module: used for preprocessing various types of structured geological data, including data cleaning, format standardization and redundancy removal; Data association module: used to build heterogeneous geological data models, and dynamically associate multiple types of structured geological data using heterogeneous geological data models to form an integrated governance geological dataset; Integrated governance assessment module: This module is used to construct a heterogeneous map data structure based on the integrated governance geological dataset, and to calculate the integrated governance effect of the integrated governance geological dataset based on the heterogeneous map data structure.