Data modeling method based on cross-library federation technology

Through the cross-repository federal technology data modeling method, the problems of inefficiency, insufficient security and privacy protection in heterogeneous data source processing are solved, efficient and secure data processing and privacy protection are achieved, and data sharing is provided for distributed environments. with analysis support.

CN118939712APending Publication Date: 2024-11-12浪潮智慧城市科技有限公司 +1
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202410978352.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing data integration technologies are inefficient, insecurity, and difficult to protect data privacy when dealing with heterogeneous data sources. They often require cumbersome data migration and conversion processes, resulting in data inconsistency problems.

Method used

Using a data modeling method based on cross-base federal technology, data sources are discovered through network scanning, secure connections are established, metadata is extracted and mapped, data cleaning and normalization are carried out, and cross-border using intelligent metadata management system and federated data query engine Library query optimization and merging, train machine learning algorithms to generate optimal data models, realize efficient and secure data processing, and use access control and encryption measures to protect data privacy.

Benefits of technology

It realizes efficient and secure data modeling of heterogeneous data sources, improves data processing efficiency, protects data privacy, and provides technical support for data sharing and analysis in a distributed environment, solving the inefficiency of traditional methods, Issues of insufficient security and difficulty in privacy protection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939712A_ABST
    Figure CN118939712A_ABST
Patent Text Reader

Abstract

The invention discloses a data modeling method based on a cross-library federation technology, and relates to the technical field of data processing and analysis, and the method comprises the steps: discovering an available data source on a network through network scanning, and automatically extracting metadata information from the available data source; when a user submits a cross-library query request, establishing secure connection with the available data source, and performing identity authentication and authority authentication at the same time; identifying the connected available data sources, extracting metadata information, mapping the extracted information into a data model according to a preset rule, and normalizing the data model according to a predefined data standard; the method comprises the following steps: intelligently analyzing and optimizing a query statement according to a user request, and decomposing the query statement into a plurality of subtasks for a single data source according to an optimization result; and merging and integrating the query results returned by the plurality of subtasks to form a final query result and returning the final query result to the user. According to the method, the problem that the efficiency is low when heterogeneous data sources are processed by a traditional data modeling method is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data processing and analysis, and in particular to a data modeling method based on cross-database federation technology. Background Art

[0002] With the rapid development of technologies such as cloud computing and the Internet of Things, enterprises and organizations at all levels are facing the challenge of data being scattered across multiple heterogeneous systems. The data storage forms of these systems include but are not limited to relational databases, NoSQL databases, data lakes, etc., and each system generally adopts different data models and storage structures.

[0003] Existing data integration technologies mainly include ETL (Extract, Transform, Load), data federation and data virtualization. ETL requires that data be extracted from the source system first, then loaded into the target system after transformation. This process is complex to implement and has high maintenance costs. Although data federation technology can achieve cross-source queries to a certain extent, it usually has high consistency requirements for the access protocol and data model of the data source. Data virtualization creates a unified abstract layer to access the underlying heterogeneous data, but its performance and security are often limited by the differences and complexity of the underlying system.

[0004] In general, existing data integration technologies often require cumbersome data migration and conversion processes, which are not only time-consuming and labor-intensive, but may also introduce data inconsistency issues. Therefore, there is an urgent need for a data modeling solution that can cross these data silos and achieve efficient, secure, and flexible data modeling. Summary of the invention

[0005] In view of the needs and shortcomings of current technological development, the present invention provides a data modeling method based on cross-database federation technology, aiming to solve the problems of low efficiency, insufficient security and difficulty in privacy protection in traditional data modeling methods when processing heterogeneous data sources. Through cross-database federation technology, efficient and secure data modeling of heterogeneous data sources is achieved, which improves data processing efficiency while protecting data privacy, and provides strong technical support for data sharing and analysis in distributed environments.

[0006] The data modeling method based on cross-database federation technology of the present invention solves the above technical problems by adopting the following technical solutions:

[0007] A data modeling method based on cross-database federation technology, the implementation of which includes the following steps:

[0008] S1. Through network scanning, available data sources on the network are discovered, and metadata information is automatically extracted from the available data sources and stored in the central source database;

[0009] S2. When a user submits a cross-database query request, a secure connection is established with the available data source, and identity authentication and permission authentication are performed during the connection establishment process;

[0010] S3. Identify the available connected data sources and extract metadata information from them. Map the extracted metadata information to a unified data model according to preset rules, and perform syntax verification during the mapping process. Then, according to pre-defined data standards, normalize the data model through data cleaning, conversion and mapping operations.

[0011] S4. Intelligently parse and optimize the query statement according to the cross-database query request submitted by the user; decompose the query task corresponding to the query statement into multiple subtasks for a single data source according to the optimization result of the query statement; merge and integrate the query results returned by each data source to form the final query result and return it to the user;

[0012] S5. Based on the historical execution process and results of the cross-database query request in steps S1-S4, the machine learning algorithm is trained to generate the optimal data model to dynamically adapt to changes in the data source and achieve rapid merging and integration of query results.

[0013] Optionally, steps S1-S3 are performed by an intelligent metadata management system;

[0014] The intelligent metadata management system involved supports the following operations:

[0015] Based on the contextual information of metadata, the association between fields and the semantic information of data dependencies are inferred, and query optimization is performed based on the inferred semantic information.

[0016] Further optionally, the intelligent metadata management system involved supports the following operations:

[0017] Through the replication and synchronization mechanism, the data source is divided into multiple shards and stored on different nodes. When the metadata on a node changes, the change is automatically synchronized to the remaining nodes.

[0018] Monitor the status of data sources and changes in metadata in real time. When an abnormal situation is found or the preset threshold is reached, the alarm mechanism is automatically triggered to notify the administrator for processing.

[0019] Further optionally, step S2 is performed, and after a secure connection is successfully established between the user who submits the cross-database query request and the available data source, the connection is regularly monitored and maintained, including monitoring the network status, checking the health status of the data source, and optimizing the connection performance.

[0020] Further optionally, step S4 is performed by a federated data query engine;

[0021] Based on the cross-database query request submitted by the user, the federated data query engine performs intelligent parsing and optimization on the query statement, including:

[0022] Parse the query statement and generate an abstract syntax tree;

[0023] Infer semantic information based on the contextual information of metadata to optimize the query statement. The optimization process includes selecting the optimal query path, connection method and aggregation strategy.

[0024] The optimized query statement is decomposed into multiple subtasks targeting a single data source, and multiple subtasks are executed independently and in parallel through the concurrency control mechanism.

[0025] Further optionally, the federated data query engine involved has intelligent routing and load balancing functions. During the query execution process, the intelligent routing function automatically selects the optimal query execution path according to the data distribution and data source system characteristics, and the load balancing function dynamically adjusts the distribution strategy of the query request according to the load conditions of the node.

[0026] Further optionally, the federated data query engine involved supports the use of standard SQL language and extended language to write cross-database query statements to achieve joint query and analysis of multiple data sources; during the entire query process, the federated data query engine adopts access control, data encryption and desensitization security measures to protect data security and privacy, ensuring that sensitive data is not leaked during transmission and processing.

[0027] Further optionally, the federated data query engine involved introduces a distributed cache mechanism. For the same query request, the federated data query engine first checks whether there are already results in the cache. If so, the cached results are directly returned.

[0028] Compared with the prior art, the data modeling method based on the cross-database federation technology of the present invention has the following beneficial effects:

[0029] 1. The present invention realizes efficient and secure data modeling for heterogeneous data sources through cross-database federation technology, which not only improves the efficiency of data processing, but also strictly protects data privacy, provides strong technical support for data sharing and analysis in distributed environments, and solves the problems of low efficiency, insufficient security and difficulty in privacy protection in traditional data modeling methods when processing heterogeneous data sources;

[0030] 2. The present invention can realize virtual integration of distributed heterogeneous data sources and build a unified data model without migrating the original data. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Attached Figure 1It is a flow chart for implementing the method described in Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0032] In order to make the technical solution, the technical problem solved and the technical effect of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.

[0033] Embodiment 1:

[0034] Combined with Figure 1 This embodiment proposes a data modeling method based on cross-database federation technology, and the implementation of the method includes the following steps:

[0035] S1. Through network scanning, available data sources on the network are discovered, and metadata information is automatically extracted from the available data sources and stored in the central source database;

[0036] S2. When a user submits a cross-database query request, a secure connection is established with the available data source, and identity authentication and permission authentication are performed during the connection establishment process. After the authentication is passed, a secure connection is successfully established between the user and the available data source. After the connection is established, the connection is regularly monitored and maintained, including monitoring of network conditions, health checks on data sources, and optimization of connection performance.

[0037] S3. Identify the available connected data sources and extract metadata information from them. Map the extracted metadata information to a unified data model according to preset rules, and perform syntax verification during the mapping process. Then, according to pre-defined data standards, normalize the data model through data cleaning, conversion and mapping operations.

[0038] S4. Intelligently parse and optimize the query statements according to the cross-database query requests submitted by the user, including: parsing the query statements and generating an abstract syntax tree; inferring semantic information based on the context information of the metadata to optimize the query statements, and the optimization process includes selecting the optimal query path, connection method and aggregation strategy; decomposing the optimized query statements into multiple subtasks for a single data source, and executing multiple subtasks independently and in parallel through a concurrent control mechanism; merging and integrating the query results returned by each data source to form the final query result and return it to the user;

[0039] S5. Based on the historical execution process and results of the cross-database query request in steps S1-S4, the machine learning algorithm is trained to generate the optimal data model to dynamically adapt to changes in the data source and achieve rapid merging and integration of query results.

[0040] Based on the data modeling method of this embodiment, steps S1-S3 are performed by the intelligent metadata management system, and step S4 is performed by the federated data query engine.

[0041] The intelligent metadata management system involved in this embodiment also supports the following operations:

[0042] Based on the contextual information of metadata, the association between fields and the semantic information of data dependencies are inferred, and query optimization is performed based on the inferred semantic information.

[0043] Through the replication and synchronization mechanism, the data source is divided into multiple shards and stored on different nodes. When the metadata on a node changes, the change is automatically synchronized to the remaining nodes.

[0044] Monitor the status of data sources and changes in metadata in real time. When an abnormal situation is found or the preset threshold is reached, the alarm mechanism is automatically triggered to notify the administrator for processing.

[0045] The federated data query engine involved in this embodiment has intelligent routing and load balancing functions. During the query execution process, the intelligent routing function automatically selects the optimal query execution path according to the data distribution and data source system characteristics, and the load balancing function dynamically adjusts the distribution strategy of the query request according to the load conditions of the node.

[0046] The federated data query engine involved in this embodiment supports the use of standard SQL language and extended language to write cross-database query statements to realize joint query and analysis of multiple data sources; during the entire query process, the federated data query engine adopts access control, data encryption and desensitization security measures to protect data security and privacy, and ensure that sensitive data is not leaked during transmission and processing.

[0047] The federated data query engine involved in this embodiment introduces a distributed cache mechanism. For the same query request, the federated data query engine first checks whether there are results in the cache. If so, the cached results are directly returned.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data modeling method based on cross-database federation technology, characterized in that: The implementation of this method includes the following steps: S1. Through network scanning, available data sources on the network are discovered, and metadata information is automatically extracted from the available data sources and stored in the central source database; S2. When a user submits a cross-database query request, a secure connection is established with the available data source, and identity authentication and permission authentication are performed during the connection establishment process; S3, identifying the available connected data sources, extracting metadata information from them, mapping the extracted metadata information to a unified data model according to preset rules, and performing syntax verification during the mapping process; Subsequently, the data model is normalized through data cleaning, transformation, and mapping operations according to pre-defined data standards; S4. Intelligently parse and optimize the query statement according to the cross-database query request submitted by the user; According to the optimization result of the query statement, the query task corresponding to the query statement is decomposed into multiple subtasks for a single data source; The query results returned by each data source are merged and integrated to form the final query results and returned to the user.

2. According to the data modeling method based on cross-database federation technology according to claim 1, it is characterized in that: The implementation of the method further comprises the following steps: S5. Based on the historical execution process and results of the cross-database query request in steps S1-S4, the machine learning algorithm is trained to generate the optimal data model to dynamically adapt to changes in the data source and achieve rapid merging and integration of query results.

3. According to the data modeling method based on cross-database federation technology of claim 1, it is characterized in that: Execute steps S1-S3 by intelligent metadata management system; The intelligent metadata management system supports the following operations: Based on the contextual information of metadata, the association between fields and the semantic information of data dependencies are inferred, and query optimization is performed based on the inferred semantic information.

4. According to the data modeling method based on cross-database federation technology of claim 3, it is characterized in that: The intelligent metadata management system supports the following operations: Through the replication and synchronization mechanism, the data source is divided into multiple shards and stored on different nodes. When the metadata on a node changes, the change is automatically synchronized to the remaining nodes. Monitor the status of data sources and changes in metadata in real time. When an abnormal situation is found or the preset threshold is reached, the alarm mechanism is automatically triggered to notify the administrator for processing.

5. According to the data modeling method based on cross-database federation technology according to claim 4, it is characterized in that: Execute step S2. After a secure connection is successfully established between the user who submits the cross-database query request and the available data source, the connection is regularly monitored and maintained, including monitoring the network status, checking the health status of the data source, and optimizing the connection performance.

6. The data modeling method based on cross-database federation technology according to claim 1 is characterized in that: Execute step S4 through the federated data query engine; Based on the cross-database query request submitted by the user, the federated data query engine performs intelligent parsing and optimization on the query statement, including: Parse the query statement and generate an abstract syntax tree; Infer semantic information based on the contextual information of metadata to optimize the query statement. The optimization process includes selecting the optimal query path, connection method and aggregation strategy. The optimized query statement is decomposed into multiple subtasks targeting a single data source, and multiple subtasks are executed independently and in parallel through the concurrency control mechanism.

7. A data modeling method based on cross-database federation technology according to claim 6, characterized in that: The federated data query engine has intelligent routing and load balancing functions. During the query execution process, the intelligent routing function automatically selects the optimal query execution path based on data distribution and data source system characteristics, and the load balancing function dynamically adjusts the query request distribution strategy based on the node load conditions.

8. The data modeling method based on cross-database federation technology according to claim 6 is characterized in that: The federated data query engine supports the use of standard SQL language and extended language to write cross-database query statements, realizing joint query and analysis of multiple data sources; During the entire query process, the federated data query engine uses access control, data encryption and desensitization security measures to protect data security and privacy, ensuring that sensitive data is not leaked during transmission and processing.

9. The data modeling method based on cross-database federation technology according to claim 6 is characterized in that: The federated data query engine introduces a distributed cache mechanism. For the same query request, the federated data query engine first checks whether there are results in the cache. If so, it directly returns the cached results.

Citation Information

Cited By

  • AI-based data cross-library and file stream integration checking system

    CN121262288A

  • AI-based cross-database and file stream integration and query system

    CN121262288B

  • Data routing method and device based on metadata index, equipment and medium

    CN121681614A

  • Data routing methods, devices, equipment, and media based on metadata indexes

    CN121681614B