Intelligent adaptation system and method for heterogeneous data lake in a simulation engine

Abstract

This invention discloses an intelligent adaptation system and method for heterogeneous data lakes in a simulation engine. The system includes: a data source layer, a data access layer, a data processing layer, a data storage layer, a data management layer, and a data service layer. The data source layer is used to enable access to various types of heterogeneous data sources. The data access layer is responsible for the automatic identification and adaptation of heterogeneous data sources. The data processing layer is responsible for format conversion, quality checks, and semantic mapping of heterogeneous data. The data storage layer adopts a hierarchical storage architecture, distributing data across different storage media according to access frequency and importance. The data management layer provides data governance and management functions to ensure the orderly operation of the data lake. The data service layer provides a unified data access interface and services for upper-layer applications. This invention achieves rapid access, intelligent format conversion, automatic semantic mapping, and quality assurance of heterogeneous data sources in the simulation engine system.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an intelligent adaptation system and method for heterogeneous data lakes in a simulation engine. Background Technology

[0002] Simulation engine systems are crucial technological supports for modern scientific research and engineering training, with their core function being the modeling, analysis, and performance evaluation of complex systems. As the complexity of simulation engines continues to increase, these systems need to process massive amounts of heterogeneous data from various data sources, including sensor data, monitoring data, historical simulation data, configuration parameter data, and other data types. These data are characterized by diverse formats, complex structures, and inconsistent semantics, posing significant challenges to the data management and processing of simulation engine systems.

[0003] Traditional simulation engine data management solutions primarily employ relational databases or simple file storage systems, which struggle to effectively handle the unified management of heterogeneous data. While existing data lake technologies have addressed heterogeneous data storage to some extent, they still exhibit numerous limitations in areas such as data access, format conversion, and semantic mapping.

[0004] Currently, mainstream heterogeneous data processing technologies mainly include ETL-based data integration solutions, which integrate data through three steps: extraction, transformation, and loading. However, these solutions suffer from low processing efficiency and poor real-time performance. Data warehouse-based solutions store data using predefined schema structures, but lack flexibility and struggle to adapt to dynamically changing data sources. NoSQL-based distributed storage processes large-scale data through distributed databases, but lacks a unified data model and semantic management capabilities. Stream processing-based real-time data processing uses stream computing frameworks to handle real-time data, but it has shortcomings in batch processing and historical data management.

[0005] Existing technologies generally suffer from problems such as low data access efficiency, complex format conversion, and difficulty in semantic mapping when facing the complex and ever-changing data environment of simulation engine systems, and cannot meet the requirements of simulation engine systems for real-time, accuracy, and scalability of data processing. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide an intelligent adaptation system and method for heterogeneous data lakes in simulation engines, so as to realize rapid access, intelligent format conversion, automatic semantic mapping and quality assurance of heterogeneous data sources in simulation engine systems.

[0007] The objective of this invention is achieved through the following technical solution: an intelligent adaptation system for heterogeneous data lakes in a simulation engine, comprising: a data source layer, a data access layer, a data processing layer, a data storage layer, a data management layer, and a data service layer; The data source layer is used to enable access to various types of heterogeneous data sources; The data access layer is responsible for the automatic identification and adaptation of heterogeneous data sources; The data processing layer is responsible for format conversion, quality checking, and semantic mapping of heterogeneous data. The data storage layer adopts a hierarchical storage architecture, distributing data across different storage media according to the frequency and importance of data access. The data management layer provides data governance and management functions to ensure the orderly operation of the data lake; The data service layer provides a unified data access interface and services for upper-layer applications.

[0008] A method for intelligent adaptation of heterogeneous data lakes in a simulation engine includes the following steps: S1. Enables access to various types of heterogeneous data sources through the data source layer; S2. Automatic identification and adaptation of heterogeneous data sources through the data access layer; S3. Through the data processing layer, perform format conversion, quality checks, and semantic mapping of heterogeneous data; S4. In the data storage layer, a hierarchical storage architecture is adopted, and the data is distributed on different storage media according to the access frequency and importance of the data; S5. Through the data management layer, provide data governance and management functions to ensure the orderly operation of the data lake; S6. Through the data service layer, a unified data access interface and services are provided for upper-layer applications. The beneficial effects of this invention are: Attached Figure Description

[0009] Figure 1 This is a schematic diagram of the system principle of the present invention; Figure 2 This is a schematic diagram illustrating the specific processing steps of the present invention. Detailed Implementation

[0010] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the following description.

[0011] This invention achieves intelligent access, transformation, mapping, and management of heterogeneous data by constructing a multi-layered data processing system. The architecture adopts a six-layer design pattern, including a data source layer, a data access layer, a data processing layer, a data storage layer, a data management layer, and a data service layer. Each layer works collaboratively to achieve full lifecycle management of heterogeneous data. Specifically: like Figure 1 As shown, an intelligent adaptation system for heterogeneous data lakes in a simulation engine includes: The data source layer supports access to various types of heterogeneous data sources, including sensor data, monitoring data, historical simulation data, configuration parameter data, and real-time streaming data. Each data source has different data formats, transmission protocols, and access methods, posing challenges to subsequent unified processing. Sensor data is typically transmitted in binary format, characterized by high frequency and small data volume. Monitoring data is mostly in JSON or XML format, containing system operating status and performance indicators. Historical simulation data is stored in relational databases or file systems, with large data volume and complex structure. Configuration parameter data is usually stored in key-value pair format, changing infrequently but crucial to system operation.

[0012] The data access layer is the first key technology layer of this invention, responsible for the automatic identification and adaptation of heterogeneous data sources. The data source adapter adopts a plug-in design, supporting dynamic loading and configuration of multiple data sources. The data access layer includes: The data source adapter adopts a plug-in design, supporting dynamic loading and configuration of multiple data sources. Each data source adapter includes an adapter identifier, data source type, configuration parameters, and a connection pool. For each type of data source, a dedicated adapter identifier and configuration parameters are set. After the data source is connected, the appropriate adapter and corresponding configuration parameters are selected according to the data source type. The connection pool is used to retain the data source and adapter channel for a certain period of time for data sources that have selected adapters, to ensure efficient reuse of database connections. By maintaining the lifecycle and status information of connections, the performance overhead caused by frequent creation and destruction of connections is avoided. The format recognizer uses machine learning algorithms to automatically identify data formats: For multiple data sets with known formats, it extracts data features, including character frequency, distribution of special symbols, and row and column structure; then, it uses the known data formats as labels to train a classifier built with machine learning algorithms; finally, it extracts data features from the data to be recognized and feeds them into the trained classifier for format recognition; for new formats that cannot be recognized, it supports an online learning mechanism to continuously improve its recognition capabilities through user feedback and sample data. The data collector is used to achieve efficient data acquisition and transmission. The acquisition strategy supports three modes: full acquisition, incremental acquisition, and real-time acquisition. Full acquisition is suitable for the one-time import of historical data, which improves efficiency through batch reading and block transmission. Incremental acquisition is suitable for periodic data updates, which only acquires newly added or modified data through timestamp comparison and change detection. Real-time acquisition is suitable for the continuous acquisition of streaming data, which achieves low-latency data transmission through an event-driven mechanism. The buffer manager uses a multi-level buffering mechanism to balance data acquisition speed and system processing capacity. The first-level buffer is located inside the data collector and is used to temporarily store the raw data obtained from the data source. The second-level buffer is located at the entry point of the data processing layer and is used to balance the difference between acquisition speed and processing speed. The third-level buffer is located at the front end of the storage layer and is used for batch write optimization. The buffer size and refresh strategy are dynamically adjusted according to data characteristics and system load to avoid data backlog and memory overflow. An error handler provides a comprehensive exception handling mechanism to ensure the reliability of data acquisition: the retry mechanism implements an exponential backoff retry strategy for temporary errors to avoid frequent retries putting pressure on the data source; the degradation strategy automatically switches to a backup data source when the data source is unavailable, and if there is no backup data source, it uses historical data from the current data source; the alarm notification mechanism monitors the data acquisition status in real time and promptly notifies maintenance personnel when anomalies occur, including connection timeouts, authentication errors, and authorization errors.

[0013] The data processing layer is the core technology layer of this invention, responsible for format conversion, quality inspection, and semantic mapping of heterogeneous data. The format converter employs a conversion strategy combining rule-based and machine learning approaches to achieve unified conversion of various data formats. The conversion rule base pre-defines conversion templates between common formats, including field mapping, data type conversion, and encoding conversion. For complex nested data structures, the converter uses a recursive parsing algorithm to decompose the data structure layer by layer and perform corresponding conversions. The machine learning module automatically learns and optimizes conversion rules by analyzing historical conversion records, improving conversion accuracy and efficiency.

[0014] The conversion process employs a pipelined processing model, breaking down complex conversion tasks into multiple simple processing steps. Each step corresponds to a conversion operation, such as data cleaning, format standardization, field renaming, and data type conversion. The pipeline supports parallel processing, allowing multiple conversion steps to be executed simultaneously, significantly improving processing efficiency. Conversion results are temporarily stored through an intermediate caching mechanism to avoid redundant calculations and data loss.

[0015] The quality inspector provides multi-dimensional data quality assessment and repair functions. Integrity checks verify the completeness of data records, including mandatory field checks and record integrity verification. Accuracy checks ensure data accuracy through data range validation, format verification, and business rule checks. Consistency checks verify data consistency across different systems, identifying data conflicts through cross-validation and correlation analysis. Timeliness checks ensure that data timestamps are correct and meet business requirements.

[0016] The data repair mechanism offers both automatic and manual intervention methods for addressing identified quality issues. Automatic repair is suitable for common data quality problems such as missing value imputation, outlier handling, and deduplication. Missing value imputation employs various strategies, including mean imputation, median imputation, forward imputation, backward imputation, and machine learning-based predictive imputation. Outlier detection combines statistical methods with machine learning algorithms, using Z-score, IQR, and Isolation Forest to identify anomalous data points. Deduplication is efficiently achieved through hash comparison and similarity calculation.

[0017] The semantic mapping engine is a key innovative technology in the data processing layer, enabling semantic unification and knowledge extraction from heterogeneous data. The ontology construction module builds a unified data ontology based on domain knowledge, defining semantic elements such as entities, attributes, relationships, and constraints. The ontology is described using the OWL (Web Ontology Language) standard, supporting reasoning and query functions. The entity recognition algorithm identifies entity objects in the data using natural language processing techniques, including named entity recognition, relationship extraction, and event detection.

[0018] The automatic mapping algorithm employs a similarity-based mapping strategy, evaluating the mapping relationships between data fields using multi-dimensional metrics such as string similarity, semantic similarity, and structural similarity. String similarity uses algorithms like edit distance, Jaccard coefficient, and cosine similarity to calculate the similarity of field names. Semantic similarity is based on word vector models to calculate the semantic similarity of fields, supporting various word vector algorithms such as Word2Vec, GloVe, and BERT. Structural similarity analyzes the hierarchical structure and relationships of the data, finding the optimal mapping scheme through graph matching algorithms. Different data sources may describe the same thing differently (e.g., some call it "User ID," some call it "user_id," and some call it "Customer Number"). The mapping algorithm determines which unified semantic concept in the ontology these different fields should correspond to by calculating string similarity, semantic similarity, and structural similarity.

[0019] Mapping results are assessed for quality using confidence scores; high-confidence mappings are applied automatically, while low-confidence results are submitted for manual review. Mapping rules support incremental learning, continuously optimizing the accuracy of the mapping algorithm through user feedback and historical mapping records. The mapping process generates detailed mapping logs, recording the basis and process of mapping decisions, supporting the traceability and auditing of mapping results.

[0020] The confidence level scoring method is as follows: 1. Obtain the previously calculated string similarity (such as Jaccard or edit distance normalization), semantic similarity (such as Word2Vec cosine similarity), and structural similarity (such as whether the data types are consistent, 1 if consistent, 0 otherwise).

[0021] 2. The three similarities are weighted and summed according to preset weights (0.5, 0.3, 0.2) to obtain the final confidence score (range 0~1).

[0022] 3. Based on the confidence threshold: ≥0.8 is automatically mapped, 0.5~0.8 is marked for manual review, and <0.5 is not mapped.

[0023] 4. Record the score and decision result of each mapping for subsequent adjustment of weights or thresholds.

[0024] In the embodiments of this application, each time the mapping results are manually reviewed, the mapping pairs confirmed or corrected by the user are recorded (e.g., correctly mapping "user ID" to "user_identifier"). These feedback pairs are added to the historical mapping rule base as positive samples; mapping pairs rejected by the user are treated as negative samples. The weights in the similarity calculation are dynamically adjusted; for example, if historical data shows that semantic similarity is more reliable, its weight is increased. In subsequent automatic mapping, verified mapping pairs in the rule base are matched first; if a perfect match is found, it is directly adopted.

[0025] The logic for dynamically adjusting similarity weights is as follows: (1) After each manual review, compare the automatic scoring results with the user's final decision: if a certain dimension has a high score but the mapping is incorrect, reduce the weight of that dimension; if a certain dimension has a low score but the mapping should actually be successful, increase its weight.

[0026] (2) Update weights: New weights = Original weights + α × (User judgment results - Contribution of this dimension in automatic confidence) Where α is the learning rate (0.05), and the user's judgment result is 1 (correct) or 0 (incorrect).

[0027] (3) After updating the weights of each dimension, normalize them again to ensure that the sum is 1.

[0028] (4) Set upper and lower limits for weight changes (e.g., a single weight should not be lower than 0.1 or higher than 0.8) to prevent a certain dimension from being excessively suppressed or dominated.

[0029] (5) Periodically fine-tune the weights based on the most recent N feedback samples to avoid drastic fluctuations caused by a single noise feedback.

[0030] For example, the initial state: String similarity = 0.85, weight = 0.4 Semantic similarity = 0.70, weight = 0.4 Structural similarity = 0.60, weight = 0.2 Dimensional contribution calculation: String dimension contribution = 0.85 × 0.4 = 0.34 Semantic dimension contribution = 0.70 × 0.4 = 0.28 Structural dimension contribution = 0.60 × 0.2 = 0.12 The overall confidence level = 0.34 + 0.28 + 0.12 = 0.74, requiring manual review; If the user approves and confirms that the mapping should be performed, then the user's judgment is 1. Weight adjustment (taking the string dimension as an example, let α=0.1): New weight = 0.4 + 0.1 × (1 - 0.34) = 0.4 + 0.066 = 0.466 The data storage layer employs a tiered storage architecture, distributing data across different storage media based on access frequency and importance. Raw data storage preserves the original format and content of all incoming data, ensuring data integrity and traceability. Raw data utilizes object storage, supporting efficient storage and retrieval of massive amounts of data. Storage strategies automatically adjust based on data lifecycle: hot data is stored on high-performance SSDs, lukewarm data on hard disk drives (HDDs), and cold data in tape libraries or cloud storage.

[0031] Structured data storage organizes and stores processed, standardized data according to a unified model. The storage model uses a columnar storage format, supporting efficient analytical queries and data compression. Partitioning strategies partition data based on dimensions such as time, region, and business type, improving query performance and management efficiency. Indexing mechanisms include various types such as primary key indexes, secondary indexes, and full-text indexes, supporting rapid responses to complex queries.

[0032] Metadata storage manages descriptive information for all data, including data schema, lineage, quality metrics, and access permissions. Metadata is stored using a graph database, supporting efficient querying and analysis of complex relationships. A version control mechanism records the metadata change history, supporting metadata rollback and auditing. A metadata synchronization mechanism ensures consistency between metadata and actual data, maintaining data accuracy through regular verification and incremental synchronization.

[0033] The data management layer includes: The ontology manager is used to maintain the integrity and consistency of domain ontology, and supports ontology version management, conflict detection, and merging operations; it automatically adjusts and optimizes the ontology structure based on the analysis of data change trends and user needs. The lineage manager tracks the source, transformation process, and usage of data, and builds a complete data lineage map; lineage information includes data source identifier, processing steps, transformation rules, and output results.

[0034] The data service layer provides a unified data access interface and services for upper-layer applications. The query service supports multiple query languages, including SQL, NoSQL, and graph queries, providing a unified query entry point. The query optimizer improves query performance through cost estimation and execution plan optimization. The distributed query engine supports joint queries across storage systems, enabling transparent data access.

[0035] The analytics service provides advanced analytical functions such as data mining, machine learning, and statistical analysis. The analytics algorithm library includes commonly used algorithms such as clustering, classification, regression, association rules, and time series analysis. The analytics task scheduler supports both batch and stream processing modes, selecting appropriate computing resources based on task characteristics. A result caching mechanism avoids redundant calculations, improving analytical efficiency.

[0036] The visualization service offers rich data visualization features, supporting various display formats such as charts, dashboards, and reports. The visualization component library includes multiple chart types such as bar charts, line charts, pie charts, scatter plots, heatmaps, and network diagrams. Interactive visualizations support drill-down, filtering, and linked operations. The visualization configuration engine supports saving, sharing, and reusing visualization solutions.

[0037] In the embodiments of this application, at the data access layer, in addition to machine learning-based format recognizers, a rule engine-based format recognition scheme can be adopted. Automatic data format recognition is achieved through predefined format recognition rules and pattern matching algorithms. The advantage of this scheme is that the rules are clear and highly interpretable, but it requires manual addition of rules when processing new formats, resulting in relatively poor flexibility.

[0038] At the data processing layer, in addition to using automatic mapping algorithms based on similarity calculation, semantic mapping engines can also employ knowledge graph-based mapping schemes. By constructing a domain knowledge graph, semantic mapping is achieved using graph neural networks and graph matching algorithms. This approach can better handle complex semantic relationships, but it requires significant investment in constructing domain knowledge.

[0039] In terms of data storage, besides a tiered storage architecture, a hybrid storage solution based on data lakes and data warehouses can be adopted. Structured data is stored in the data warehouse, and unstructured data is stored in the data lake, with a unified data access interface provided through a virtualization layer. This solution can fully leverage the advantages of different storage systems, but its system complexity is relatively high.

[0040] Regarding the data management layer, in addition to using graph databases to store kinship relationships, kinship management can also employ blockchain-based kinship tracing solutions. The immutability of blockchain ensures the credibility and integrity of kinship information. This solution offers advantages in data security, but presents challenges in terms of performance and cost.

[0041] like Figure 2 As shown, an intelligent adaptation method for heterogeneous data lakes in a simulation engine includes the following steps: S1. Enables access to various types of heterogeneous data sources through the data source layer and receives heterogeneous data; S2. The data access layer performs format recognition and parsing on heterogeneous data, automatically identifies data types, and completes preliminary adaptation. S3. In the data processing layer, if the format recognition is successful, the conversion rules are applied to perform the format conversion; if it fails, the recognition error is recorded and the manual annotation process begins. S4. If the format conversion is successful, perform a data quality check; if it fails, record the conversion error and try an alternative conversion strategy. S5. Continue data quality checks at the data processing layer: if the checks pass, proceed to semantic mapping processing; if they fail, initiate the data quality repair process. S6. After data quality repair, if the repair is successful, continue semantic mapping processing; if it fails, record the quality anomaly and terminate the process. S7. During the semantic mapping process, if the mapping is successful, standardized data is generated and metadata is updated; if it fails, the mapping exception is recorded and manual intervention is initiated. S8. For data that has successfully completed semantic mapping, generate standardized data and update metadata to complete the data processing loop; S9. Through the data management layer, perform unified governance and monitoring of anomalies, quality status, and metadata changes during the processing to ensure the orderly operation of the data lake; S10. Finally, through the data service layer, a unified data access interface and services are provided to upper-layer applications, supporting efficient and reliable data consumption.

[0042] In summary, this invention, by constructing a heterogeneous data lake intelligent adaptation architecture for simulation engines, has the following characteristics: First, through the collaborative work of the format recognizer and data source adapter, intelligent and automatic access to heterogeneous data sources is achieved. The system can automatically identify and adapt to more than 200 data formats, reducing the access time for new data sources from two weeks in traditional solutions to less than four hours, and lowering access costs by more than 80%. The plug-in design gives the system excellent scalability, eliminating the need to develop separate adapters for each data source and significantly reducing manual configuration workload.

[0043] Secondly, the multi-dimensional quality assessment mechanism of the quality checker ensures the integrity, accuracy, consistency, and timeliness of the data. The automatic repair mechanism can handle over 90% of common data quality issues, reducing the proportion of dirty data from 20% to below 5%, significantly improving data quality. The semantic mapping engine, through ontology construction and automatic mapping algorithms, achieves semantic unification of heterogeneous data, increasing mapping accuracy from 55% to over 95%, resolving issues of data misunderstanding and usage errors.

[0044] Third, a scalable data storage system was built through the design of a layered storage architecture. The layered design of raw data storage, structured data storage, and metadata storage enables the system to support efficient storage and management of petabyte-scale data. Partitioning strategies and indexing mechanisms improve query performance by more than 10 times and reduce storage costs by more than 60%, effectively solving the performance bottleneck of large-scale data storage and retrieval.

[0045] Fourth, through the collaborative work of the ontology manager, lineage manager, and version manager, comprehensive data governance and management functions are provided. The lineage tracking function reduces the impact analysis time for data changes from hours to minutes, significantly improving operational efficiency. The version control mechanism ensures data traceability and consistency, supporting data rollback and auditing functions.

[0046] Fifth, a unified data service layer design enables diverse data access and analysis capabilities. The query service supports multiple query languages, including SQL, NoSQL, and graph queries; the analysis service provides a rich library of algorithms such as clustering, classification, and regression; and the visualization service supports diverse display formats. This unified service interface reduces the development cost of upper-layer applications by more than 50%, significantly improves system integration efficiency, and provides robust data support for the simulation engine.

Claims

1. An intelligent adaptation system for heterogeneous data lakes in simulation engines, characterized in that: include: Data source layer, data access layer, data processing layer, data storage layer, data management layer, and data service layer; The data source layer is used to enable access to various types of heterogeneous data sources; The data access layer is responsible for the automatic identification and adaptation of heterogeneous data sources; The data processing layer is responsible for format conversion, quality checking, and semantic mapping of heterogeneous data. The data storage layer adopts a hierarchical storage architecture, distributing data across different storage media according to the frequency and importance of data access. The data management layer provides data governance and management functions to ensure the orderly operation of the data lake; The data service layer provides a unified data access interface and services for upper-layer applications.

2. The intelligent adaptation system of heterogeneous data lakes in a simulation engine according to claim 1, characterized in that: The heterogeneous data sources are classified into sensor data sources, monitoring data sources, historical simulation data sources, configuration parameter data sources, and real-time streaming data sources according to their different types.

3. The intelligent adaptation system of heterogeneous data lakes in a simulation engine according to claim 1, characterized in that: The data access layer includes: The data source adapter adopts a plug-in design, supporting dynamic loading and configuration of multiple data sources. Each data source adapter includes an adapter identifier, data source type, configuration parameters, and a connection pool. For each type of data source, a dedicated adapter identifier and configuration parameters are set. After the data source is connected, the appropriate adapter and corresponding configuration parameters are selected according to the data source type. The connection pool is used to retain the data source and adapter channel for a certain period of time for data sources that have selected adapters, to ensure efficient reuse of database connections. By maintaining the lifecycle and status information of connections, the performance overhead caused by frequent creation and destruction of connections is avoided. The format recognizer uses machine learning algorithms to automatically identify data formats: For multiple data sets with known formats, it extracts data features, including character frequency, distribution of special symbols, and row and column structure; then, it uses the known data formats as labels to train a classifier built with machine learning algorithms; finally, it extracts data features from the data to be recognized and feeds them into the trained classifier for format recognition; for new formats that cannot be recognized, it supports an online learning mechanism to continuously improve its recognition capabilities through user feedback and sample data. The data collector is used to achieve efficient data acquisition and transmission. The acquisition strategy supports three modes: full acquisition, incremental acquisition, and real-time acquisition. Full acquisition is suitable for the one-time import of historical data, which improves efficiency through batch reading and block transmission. Incremental acquisition is suitable for periodic data updates, which only acquires newly added or modified data through timestamp comparison and change detection. Real-time acquisition is suitable for the continuous acquisition of streaming data, which achieves low-latency data transmission through an event-driven mechanism. The buffer manager uses a multi-level buffering mechanism to balance data acquisition speed and system processing capacity. The first-level buffer is located inside the data collector and is used to temporarily store the raw data obtained from the data source. The second-level buffer is located at the entry point of the data processing layer and is used to balance the difference between acquisition speed and processing speed. The third-level buffer is located at the front end of the storage layer and is used for batch write optimization. The buffer size and refresh strategy are dynamically adjusted according to data characteristics and system load to avoid data backlog and memory overflow. An error handler provides a comprehensive exception handling mechanism to ensure the reliability of data acquisition: the retry mechanism implements an exponential backoff retry strategy for temporary errors to avoid frequent retries putting pressure on the data source; the degradation strategy automatically switches to a backup data source when the data source is unavailable, and if there is no backup data source, it uses historical data from the current data source; the alarm notification mechanism monitors the data acquisition status in real time and promptly notifies maintenance personnel when anomalies occur, including connection timeouts, authentication errors, and authorization errors.

4. The intelligent adaptation system of heterogeneous data lakes in a simulation engine of claim 1, wherein: The data processing layer includes: The format converter employs a rule-based conversion strategy to achieve unified conversion of various data formats: It pre-defines conversion templates between multiple formats in its conversion rule base, including field mapping, data type conversion, and encoding conversion; the converter uses a recursive parsing algorithm to decompose the data structure layer by layer and perform corresponding conversions; the conversion process adopts a pipelined processing mode, breaking down complex conversion tasks into multiple simple processing steps, each corresponding to a conversion operation; the pipeline supports parallel processing, allowing multiple conversion steps to be executed simultaneously; conversion results are temporarily stored through an intermediate caching mechanism to avoid redundant calculations and data loss. The quality checker is used to implement multi-dimensional data quality assessment and repair functions. The data quality assessment includes: integrity checks, verifying the completeness of data records, including mandatory field checks and record integrity verification; accuracy checks, ensuring data accuracy through data range verification, format validation, and business rule checks; consistency checks, verifying data consistency across different systems, identifying data conflicts through cross-validation and correlation analysis; and timeliness checks, ensuring that data timestamps are correct and meet business requirements. The repair functions include: providing both automatic and manual intervention for identified quality issues. Automatic repair includes missing value filling, outlier handling, and duplicate data removal; manual intervention involves manual intervention to repair identified quality issues. The semantic mapping engine is used to achieve semantic unification and knowledge extraction from heterogeneous data. It includes: an ontology construction module, which builds a unified data ontology template based on domain knowledge. The template defines semantic elements, including entities, attributes, relationships, and constraints. The data ontology is described using the OWL standard and supports reasoning and querying functions. An entity recognition algorithm module identifies entity objects in the data using natural language processing techniques, including named entity recognition, relationship extraction, and attribute detection. An automatic mapping algorithm module maps the identified content to the content in the data ontology, employing a similarity-based mapping strategy. It evaluates the mapping relationship between data fields using multi-dimensional metrics, including string similarity, semantic similarity, and structural similarity.

5. The intelligent adaptation system of heterogeneous data lakes in a simulation engine according to claim 4, characterized in that: The semantic mapping engine also includes a quality assessment module: Obtain the previously calculated string similarity, semantic similarity, and structural similarity, and sum the three similarities according to preset weights to obtain the final confidence score; If the value is ≥0.8, it will be automatically mapped; if it is 0.5~0.8, it will be marked for manual review; if it is <0.5, it will not be mapped. Record the score and decision result for each mapping.

6. The intelligent adaptation system of heterogeneous data lakes in a simulation engine of claim 5, wherein: The semantic mapping engine also includes an incremental learning module, which continuously optimizes the accuracy of the mapping algorithm based on user feedback and historical mapping records. Each time the mapping results are manually reviewed, the mapping pairs that are confirmed or corrected by the user are recorded and added to the historical mapping rule base as positive samples; the mapping pairs that are rejected by the user are used as negative samples; the mapping pairs refer to the correspondence between entities in the original data and semantic concepts defined in the data ontology. Dynamically adjust the weights in similarity calculation: After each manual review, compare the automatic scoring results with the user's final decision: If a dimension scores high but the mapping is incorrect, decrease the weight of that dimension; if a dimension scores low but the mapping should have been successful, increase its weight and update the weights. Among them, mapping error is the difference between the confidence level of the system's identification and the result of human judgment; In automatic confidence, "the contribution of a dimension" refers to the specific contribution value of a particular dimension in the calculation of the total confidence. Dimension contribution = similarity score of that dimension × weight of that dimension. New weight = Original weight + α × (User judgment result - Contribution of this dimension in automatic confidence), where α is the learning rate, and the user judgment result is 1 or 0; After updating the weights for each dimension, normalize them again to ensure the sum is 1; and set upper and lower limits for weight changes to prevent a certain dimension from being excessively suppressed or dominated. Regularly fine-tune the weights based on the most recent N feedback samples to avoid drastic fluctuations caused by a single noisy feedback. In subsequent automatic mapping, the mapping pairs that have been verified in the rule base are matched first, and if a complete match is found, the mapping is used directly.

7. The intelligent adaptation system of heterogeneous data lakes in a simulation engine of claim 1, wherein: The data storage layer includes: The original data storage module retains the original format and content of all incoming data, ensuring data integrity and traceability; The structured data storage module organizes and stores the processed standardized data according to a unified model: the storage model adopts a columnar storage format, supports efficient analysis and querying and data compression, and adopts a partitioning strategy to partition the data according to the dimensions of time, region and business type, thereby improving query performance and management efficiency. The metadata storage management module is used to manage the descriptive information of all data, including data schema, lineage, quality indicators, and access permissions. The metadata is stored in a graph database, which supports efficient querying and analysis of complex relationships. The version control mechanism records the change history of metadata, supports the rollback and auditing of metadata, ensures the consistency between metadata and actual data through the metadata synchronization mechanism, and maintains the accuracy of data through regular verification and incremental synchronization. Metadata is used to describe data, including the source of the data, access method, data field information, data relationships, and data quality. Data field information includes entities, attributes, relationships between entities / attributes, data associations refer to the relationships between multiple data, and data quality refers to the quality assessment results. According to claim 1, the intelligent adaptation system for heterogeneous data lakes in a simulation engine is characterized in that: the data management layer includes: The ontology manager is used to maintain the integrity and consistency of domain ontology, and supports ontology version management, conflict detection, and merging operations; it automatically adjusts and optimizes the ontology structure based on the analysis of data change trends and user needs. The lineage manager tracks the source, transformation process, and usage of data, and builds a complete data lineage map; lineage information includes data source identifier, processing steps, transformation rules, and output results.

8. The intelligent adaptation system of heterogeneous data lakes in a simulation engine of claim 1, wherein: The data service layer includes: The query service unit supports multiple query languages ​​such as SQL, NoSQL, and graph queries, and provides a unified query entry point, including: a query optimizer that improves query performance through cost estimation and execution plan optimization; and a distributed query engine that supports joint queries across storage systems to achieve transparent data access. The analysis service unit provides advanced analysis functions including data mining, machine learning, and statistical analysis, including: an analysis algorithm library containing clustering, classification, regression, association rule, and time series analysis algorithms; an analysis task scheduler that supports both batch processing and stream processing computing modes, selecting appropriate computing resources based on task characteristics; and a result caching mechanism to avoid redundant calculations and improve analysis efficiency. The visualization service unit provides data visualization functions and supports various display formats such as charts, dashboards, and reports, including: a visualization component library containing various chart types such as bar charts, line charts, pie charts, scatter plots, heatmaps, and network diagrams; an interactive visualization unit that supports drill-down, filtering, and linked interactive operations; and a visualization configuration engine.

9. Supports saving, sharing, and reusing visual solutions.

10. An intelligent adaptation method of a heterogeneous data lake in a simulation engine based on the system of any one of claims 1-9, characterized in that: Includes the following steps: S1. Enables access to various types of heterogeneous data sources through the data source layer and receives heterogeneous data; S2. The data access layer performs format recognition and parsing on heterogeneous data, automatically identifies data types, and completes preliminary adaptation. S3. In the data processing layer, if the format recognition is successful, the conversion rules are applied to perform the format conversion; if it fails, the recognition error is recorded and the manual annotation process begins. S4. If the format conversion is successful, perform a data quality check; if it fails, record the conversion error and try an alternative conversion strategy. S5. Continue data quality checks at the data processing layer: if the checks pass, proceed to semantic mapping processing; if they fail, initiate the data quality repair process. S6. If the data quality repair is successful, continue with semantic mapping processing; If it fails, record the quality anomaly and terminate the process; S7. During the semantic mapping process, if the mapping is successful, standardized data is generated and metadata is updated; if it fails, the mapping exception is recorded and manual intervention is initiated. S8. For data that has successfully completed semantic mapping, generate standardized data and update metadata to complete the data processing loop; S9. Through the data management layer, perform unified governance and monitoring of anomalies, quality status, and metadata changes during the processing to ensure the orderly operation of the data lake; S10. Finally, through the data service layer, a unified data access interface and services are provided to upper-layer applications.

Images