Data invocation method and device based on data virtualization, storage medium and server
By constructing a data retrieval model and lineage graph using data virtualization technology, the problem of cross-database data association and retrieval analysis is solved, and SQL syntax is automatically adapted to any database type, reducing the requirements for developers.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- DONGGUAN MENGDA INDAL INVESTMENT
- Filing Date
- 2023-12-25
- Publication Date
- 2026-07-21
AI Technical Summary
Existing technologies cannot achieve cross-database data association and analysis, and require highly skilled data developers.
By using data virtualization technology, data dictionary information from different databases is received, a data calling model is constructed, a parent data lineage graph is generated, which is then split into child data lineage graphs, unique identification information is assigned, and a data query script is generated to realize cross-database data association and calling.
It enables cross-database data association and analysis, reducing the requirements for developers and improving data retrieval efficiency.
Smart Images
Figure CN117609251B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of big data technology, and more specifically, to a data retrieval method, apparatus, storage medium, and server based on data virtualization. Background Technology
[0002] In existing BI platforms, establishing connections with multiple different databases requires writing corresponding SQL syntax based on different database types, then running the data in the corresponding database, and returning the results to the BI platform. On the one hand, this cannot achieve cross-database data association and analysis, and on the other hand, it places high demands on data analysts and other data developers.
[0003] There is currently no effective solution to the problem that related technologies cannot achieve cross-database data association and analysis, and that require a high level of expertise from data developers. Summary of the Invention
[0004] The main purpose of this application is to provide a data retrieval method, device, storage medium and server based on data virtualization to solve the problems of not being able to realize cross-database data association retrieval and analysis, and the high requirements for data developers.
[0005] To achieve the above objectives, according to one aspect of this application, a data retrieval method based on data virtualization is provided.
[0006] The data retrieval method based on data virtualization according to this application includes: receiving data dictionary information from different databases; constructing a data retrieval model based on the data dictionary information, and generating a parent data lineage graph based on the data retrieval model; generating at least one sub-data lineage graph corresponding to each database based on the parent data lineage graph; assigning corresponding unique identification information to each sub-data lineage graph, and recording the flow and generation relationship between each sub-data lineage graph; generating a data query script based on the sub-data lineage graphs and database types; querying and obtaining the corresponding source dataset through the data query script; first associating the unique identification information with the source dataset, then replacing the sub-data lineage graphs with the source dataset according to the association relationship, and finally generating a target data lineage graph based on the flow and generation relationship between each sub-data lineage graph; and obtaining the target dataset based on the target data lineage graph and the source dataset.
[0007] Furthermore, receiving data dictionary information from different databases includes: the data virtualization manager interfacing with multiple different types of databases to obtain data dictionary information from multiple different databases, wherein the data dictionary information includes at least the database name, table name, field name, field-related information, conditions, and domain.
[0008] Furthermore, constructing a data retrieval model based on data dictionary information includes: using icons or lines to represent the retrieval relationships of one or more databases, data tables, fields, field-related information, conditions, and domains in the data dictionary information to obtain a data retrieval model; first, creating a virtual library, then generating virtual tables and virtual fields for the intermediate datasets in the data retrieval model corresponding to the virtual library, and finally generating a parent data lineage diagram based on the icons and lines of the data retrieval model, the set conditions, and association rules.
[0009] Furthermore, generating at least one sub-data lineage graph corresponding to each database based on the parent data lineage graph includes: for a parent data lineage graph generated from a data table of a single database, using the parent data lineage graph as a sub-data lineage graph of that database; for a parent data lineage graph generated from data tables of multiple databases, splitting the parent data lineage graph into at least one sub-data lineage graph corresponding to a virtual database or a physical database.
[0010] Furthermore, generating data query scripts based on subdata lineage diagrams and database types includes: determining the corresponding script content in the script library according to the subdata lineage diagrams and database types to obtain the corresponding data query scripts.
[0011] To achieve the above objectives, according to another aspect of this application, a data retrieval device based on data virtualization is provided.
[0012] The data retrieval device based on data virtualization according to this application includes: a receiving module for receiving data dictionary information from different databases; a construction and generation module for constructing a data retrieval model based on the data dictionary information and generating a parent data lineage diagram based on the data retrieval model; a splitting module for generating a target data lineage diagram corresponding to each database based on the parent data lineage diagram; an allocation module for assigning corresponding unique identification information to each sub-data lineage diagram and recording the process and generation relationship between each sub-data lineage diagram; a script generation module for generating a data query script based on the sub-data lineage diagrams and database types; a query module for querying and obtaining the corresponding source dataset through the data query script; an association module for first associating the unique identification information with the source dataset, then replacing the sub-data lineage diagrams with the source dataset according to the association relationship, and finally generating the target data lineage diagram based on the process and generation relationship between each sub-data lineage diagram; and an acquisition module for obtaining the target dataset based on the parent data lineage diagram and the source dataset.
[0013] Furthermore, the receiving module includes: a data virtualization manager that interfaces with multiple different types of databases to obtain data dictionary information from multiple different databases, wherein the data dictionary information includes at least the database name, data table name, field name, field-related information, conditions, and domain.
[0014] Furthermore, the construction and generation module includes: using icons or lines to represent the calling relationships of one or more databases, data tables, fields, field-related information, conditions, and domains in the data dictionary information to obtain a data calling model; first, a virtual library is created, then virtual tables and virtual fields are generated for the intermediate datasets in the data calling model corresponding to the virtual library, and finally, a parent data lineage diagram is generated based on the icons and lines of the data calling model, the set conditions, and the association rules.
[0015] To achieve the above objectives, according to another aspect of this application, a computer-readable storage medium is provided.
[0016] According to the computer-readable storage medium of this application, the computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the data retrieval method based on data virtualization at runtime.
[0017] To achieve the above objectives, according to another aspect of this application, a server is provided.
[0018] The server according to this application includes: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the data retrieval method based on data virtualization.
[0019] In this embodiment, a data retrieval method based on data virtualization is adopted. This involves receiving data dictionary information from different databases; constructing a data retrieval model based on the data dictionary information; generating a parent data lineage graph based on the data retrieval model; generating at least one sub-data lineage graph corresponding to each database based on the parent data lineage graph; assigning corresponding unique identification information to each sub-data lineage graph and recording the flow and generation relationships between each sub-data lineage graph; generating a data query script based on the sub-data lineage graphs and database types; retrieving the corresponding source dataset through the data query script; first associating the unique identification information with the source dataset, then replacing the sub-data lineage graphs with the source dataset according to the association relationship; finally generating a target data lineage graph based on the flow and generation relationships between each sub-data lineage graph; and obtaining the target dataset based on the target data lineage graph and the source dataset. This achieves the goal of automatically adapting to the SQL syntax corresponding to any database type, thereby realizing cross-database data association retrieval analysis and reducing the technical requirements for developers. This solves the technical problem of not being able to achieve cross-database data association retrieval analysis and having high requirements for data developers. Attached Figure Description
[0020] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application. In the drawings:
[0021] Figure 1 This is a flowchart illustrating a data retrieval method based on data virtualization according to an embodiment of this application;
[0022] Figure 2 This is a schematic diagram of the structure of a data retrieval device based on data virtualization according to an embodiment of this application. Detailed Implementation
[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0025] In this application, the terms "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," and "longitudinal" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing the invention and its embodiments, and are not intended to limit the indicated device, element, or component to having a specific orientation, or to be constructed and operated in a specific orientation.
[0026] Furthermore, some of the aforementioned terms, besides indicating direction or positional relationships, may also have other meanings. For example, the term "above" may, in certain circumstances, indicate a dependency or connection. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances.
[0027] Furthermore, the terms "installation," "setup," "equipped with," "connection," "linking," and "socketing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral structure; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium, or an internal connection between two devices, components, or parts. Those skilled in the art can understand the specific meaning of these terms in this invention according to the specific circumstances.
[0028] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0029] According to embodiments of the present invention, a data retrieval method based on data virtualization is provided, such as... Figure 1 As shown, the method includes the following steps S101 to S105:
[0030] Step S101: Receive data dictionary information from different databases;
[0031] Specifically, receiving data dictionary information from different databases includes: the data virtualization manager interfacing with multiple databases of different types to obtain data dictionary information from these databases. This data dictionary information includes at least the database name, table name, field name, field-related information, conditions, and domains. This provides data support for subsequent modeling.
[0032] In this embodiment, field-related information includes field name comments, data type, data range, and other information.
[0033] It is important to understand that data dictionary information is stored in each database system in the form of data tables or data files. After the data virtualization management system connects to the database, it can obtain this information in real time. The data volume statistics virtualization system can also collect and synchronize numerical information in real time.
[0034] Obtaining the above information is to help developers better understand the data structures, data information, and characteristics from different databases in data virtualization systems, which will facilitate subsequent data modeling applications.
[0035] Step S102: Construct a data retrieval model based on the data dictionary information, and generate a parent data lineage diagram based on the data retrieval model;
[0036] Specifically, constructing a data retrieval model based on data dictionary information includes: using nodes, icons, or lines to represent the retrieval relationships of databases, data tables, fields, field-related information, conditions, and domains in the data dictionary information, thereby obtaining the data retrieval model.
[0037] It is important to understand that data modeling tools can be used to create a relationship diagram of databases, tables, field names, conditions, domains, etc., based on data requirements, thereby achieving data modeling.
[0038] It is also important to understand that the data virtualization system stores information such as the database to which the data table belongs, the data table to which the data field belongs, the relationships, conditions and rules between data tables, and the fields specified for the data fields.
[0039] Specific storage examples are as follows:
[0040] Main table: Source database 1, source data fields (1, 2...n), and data fields (one or more) satisfying the following conditions;
[0041] Related Table 1: Related source database 2, source data fields (1,2...n), and data fields (one or more) satisfying the conditions;
[0042] Association Rule 1: Based on the creation of the data model, multiple tables can be associated, or one or more tables can be associated in one or more nested layers;
[0043] Generate Table 1 with data fields (1, 2... n) and corresponding generation rules (1, 2... n). Based on the creation of the data model, multiple tables can be generated.
[0044] A data lineage diagram illustrates the origin and destination of each data field or each piece of data, and is a tool used for data generation and source tracing.
[0045] Based on the data model storage information generated by the pre-data retrieval model, identify the upstream data tables, data fields, and the relationship between each data point and the downstream data tables, data fields, and each data point.
[0046] For example, if a certain data field or combination of data fields in Table 1 satisfies condition 1, then this data field or combination of data fields in Table 1 and the data field or combination of data fields in Table 2 that satisfies condition 2 will generate one or more data fields in the downstream dataset based on a certain rule, thus forming a parent data lineage graph.
[0047] Step S103: Generate a sub-data lineage diagram corresponding to each database based on the parent data lineage diagram;
[0048] The parent data lineage graph generated above is a whole lineage graph that integrates data tables corresponding to different or multiple databases. Since cross-database data calls may come from different databases, there are cross-database or cross-database types. Therefore, it is necessary to split the whole lineage graph (parent data lineage graph) into corresponding lineage graph (sub-data lineage graph) parts corresponding to each database.
[0049] For example, in the parent data lineage graph, the lineage graph related to database a is split into lineage graph a1, lineage graph a2, etc., based on database a; and in the parent data lineage graph, the lineage graph related to database b is split into lineage graph b1, lineage graph b2, etc., based on database b.
[0050] Specifically, generating sub-data lineage diagrams corresponding to each database based on the parent data lineage diagram includes:
[0051] For a parent data lineage graph generated from data tables of a single database, the parent data lineage graph is used as the child data lineage graph of that database;
[0052] For a parent data lineage graph generated from data tables of multiple databases, the parent data lineage graph is split into at least one sub-data lineage graph corresponding to a virtual library or an entity library.
[0053] Since the parent data lineage diagram illustrates the source and destination of each data field or each piece of data, that is, it can identify the relationship between upstream data tables, data fields, and each piece of data and downstream data tables, data fields, and each piece of data; therefore, by using a pre-set database name or other unique identification information that can represent the database, it can first identify which database's data table the components of the parent data lineage diagram belong to, or which database's data tables are combined to generate them.
[0054] Therefore, if it is identified that the components of a parent data lineage graph are generated from a single database table, there is no need to split it; the parent data lineage graph can be directly used as the sub-data lineage graph for subsequent generation scripts.
[0055] If it is identified that a parent data lineage graph is generated by combining data tables from multiple databases, then the parent data lineage graph is split into at least one sub-data lineage graph corresponding to the virtual database or the physical database.
[0056] It's important to understand that the data virtualization management system can generate a virtual library for each overall lineage graph, making it easier for developers to manage based on different data calls and queries. Virtual tables are generated in the virtual library corresponding to the overall lineage graph and are intermediate datasets generated during the data modeling process. Because these intermediate datasets are not managed by entity tables, virtual tables need to be created to facilitate the management of relationships between data tables or data fields when the data lineage graph is split.
[0057] For example, data tables from database 1 (table 1) and database 2 (table 2) can generate intermediate dataset virtual tables (table a) with fields a1, a2, a3 and fields b1, b2, b3, ... or more by establishing rules and conditions. These virtual tables may then generate new dataset M by establishing rules and conditions, or virtual tables and entity tables may generate new dataset N by establishing rules and conditions. Because these virtual tables are intermediate data, without the management of these virtual tables, after the parent lineage graph is split and the datasets are obtained from the source databases, the lack of management of these intermediate datasets will cause the generation of dataset M and dataset N to become disconnected, making it impossible to reconstruct the original parent lineage graph and thus unable to generate the desired dataset result.
[0058] This allows for the processing and splitting of different lineage diagram components based on whether they are generated from a single or multiple database tables. This ensures that the corresponding lineage diagram components in the database are maintained, while also guaranteeing the automatic creation of a virtual database based on virtual tables and fields even without a source database. This provides a guarantee for adapting to the SQL syntax corresponding to any database type.
[0059] Step S104: Assign corresponding unique identification information to each sub-data lineage map, and record the process and generation relationship between each sub-data lineage map;
[0060] While generating sub-data lineage maps, the system will automatically assign a unique identification information to each sub-data lineage map and record the process and generation relationship between each sub-data lineage map; so as to facilitate the subsequent generation of new target data lineage maps.
[0061] Step S105: Generate a data query script based on the sub-data lineage diagram and database type;
[0062] The sub-data lineage diagram contains the relationships between database names, tables, and fields, ensuring that there are corresponding paths for querying each other. Therefore, appropriate data query scripts can be determined based on this relationship. Furthermore, the data virtualization management system stores relevant information during database integration, allowing for the estimation of the database type. Thus, based on the sub-data lineage diagram and database type, the corresponding script content is determined from the script library to obtain the appropriate data query script. Alternatively, the appropriate script syntax can be selected from the script library based on the data lineage diagram and database type. This entire process is handled automatically by the data virtualization management system without manual intervention. This enables automatic adaptation to the SQL syntax corresponding to any database type, achieving cross-database data association and analysis, while reducing the requirements for developers.
[0063] Step S106: Obtain the corresponding source dataset by querying the data query script.
[0064] Based on the generated data query scripts, the corresponding databases can be connected to generate the corresponding source datasets. Furthermore, the query scripts generated based on the lineage diagrams of each subdatabase can be connected to the corresponding databases to achieve cross-database and cross-database type queries and calls. This eliminates the need for developers to write corresponding data query scripts to adapt to different database types, reducing the requirements for developers.
[0065] Step S107: First, associate the unique identification information with the source dataset, then replace the sub-data lineage map with the source dataset according to the association relationship, and finally generate the target data lineage map based on the process and generation relationship between each sub-data lineage map.
[0066] Step S108: Based on the target data lineage map and the source dataset, obtain the target dataset.
[0067] By associating assigned unique identification information with source datasets to generate relationships, the unique identification information can be used to link corresponding source datasets. The source datasets are then used to replace the sub-data lineage diagrams. Finally, the target data lineage diagram is generated based on the flow and generation relationships between each sub-data lineage diagram. This achieves the updating of the overall data lineage diagram, ensuring that subsequent data retrieval can be achieved using the overall target data lineage diagram and source datasets, resulting in high efficiency.
[0068] As can be seen from the above description, the present invention achieves the following technical effects:
[0069] In this embodiment, a data retrieval method based on data virtualization is adopted. This involves receiving data dictionary information from different databases; constructing a data retrieval model based on the data dictionary information; generating a parent data lineage graph based on the data retrieval model; generating at least one sub-data lineage graph corresponding to each database based on the parent data lineage graph; assigning corresponding unique identification information to each sub-data lineage graph and recording the flow and generation relationships between each sub-data lineage graph; generating a data query script based on the sub-data lineage graphs and database types; retrieving the corresponding source dataset through the data query script; first associating the unique identification information with the source dataset; then replacing the sub-data lineage graphs with the source dataset according to the association relationship; finally generating a target data lineage graph based on the flow and generation relationships between each sub-data lineage graph; and finally obtaining the target dataset based on the target data lineage graph and the source dataset. This achieves the goal of automatically adapting to the SQL syntax corresponding to any database type, thereby realizing cross-database data association retrieval analysis and reducing the technical requirements for developers. This solves the technical problem of not being able to implement cross-database data association retrieval analysis and having high requirements for data developers.
[0070] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0071] According to embodiments of the present invention, an apparatus for implementing the above-described data retrieval method based on data virtualization is also provided, such as... Figure 2 As shown, the device includes:
[0072] The receiving module 10 is used to receive data dictionary information from different databases;
[0073] Specifically, receiving data dictionary information from different databases includes: the data virtualization manager interfacing with multiple databases of different types to obtain data dictionary information from these databases. This data dictionary information includes at least the database name, table name, field name, field-related information, conditions, and domains. This provides data support for subsequent modeling.
[0074] In this embodiment, field-related information includes field name comments, data type, data range, and other information.
[0075] It is important to understand that data dictionary information is stored in each database system in the form of data tables or data files. After the data virtualization management system connects to the database, it can obtain this information in real time. The data volume statistics virtualization system can also collect and synchronize numerical information in real time.
[0076] Obtaining the above information is to help developers better understand the data structures, data information, and characteristics from different databases in data virtualization systems, which will facilitate subsequent data modeling applications.
[0077] The generation module 20 is used to build a data retrieval model based on the data dictionary information and generate a parent data lineage diagram based on the data retrieval model.
[0078] Specifically, constructing a data retrieval model based on data dictionary information includes: using nodes, icons, or lines to represent the retrieval relationships of databases, data tables, fields, field-related information, conditions, and domains in the data dictionary information, thereby obtaining the data retrieval model.
[0079] It is important to understand that data modeling tools can be used to create a relationship diagram of databases, tables, field names, conditions, domains, etc., based on data requirements, thereby achieving data modeling.
[0080] It is also important to understand that the data virtualization system stores information such as the database to which the data table belongs, the data table to which the data field belongs, the relationships, conditions and rules between data tables, and the fields specified for the data fields.
[0081] Specific storage examples are as follows:
[0082] Main table: Source database 1, source data fields (1, 2...n), and data fields (one or more) satisfying the following conditions;
[0083] Related Table 1: Related source database 2, source data fields (1,2...n), and data fields (one or more) satisfying the conditions;
[0084] Association Rule 1: Based on the creation of the data model, multiple tables can be associated, or one or more tables can be associated in one or more nested layers;
[0085] Generate Table 1 with data fields (1, 2... n) and corresponding generation rules (1, 2... n). Based on the creation of the data model, multiple tables can be generated.
[0086] A data lineage diagram illustrates the origin and destination of each data field or each piece of data, and is a tool used for data generation and source tracing.
[0087] Based on the data model storage information generated by the pre-data retrieval model, identify the upstream data tables, data fields, and the relationship between each data point and the downstream data tables, data fields, and each data point.
[0088] For example, if a certain data field or combination of data fields in Table 1 satisfies condition 1, then this data field or combination of data fields in Table 1 and the data field or combination of data fields in Table 2 that satisfies condition 2 will generate one or more data fields in the downstream dataset based on a certain rule, thus forming a parent data lineage graph.
[0089] The splitting module 30 is used to generate a target data lineage map corresponding to each database based on the parent data lineage map;
[0090] The parent data lineage graph generated above is a whole lineage graph that integrates data tables corresponding to different or multiple databases. Since cross-database data calls may come from different databases, there are cross-database or cross-database types. Therefore, it is necessary to split the whole lineage graph (parent data lineage graph) into corresponding lineage graph (sub-data lineage graph) parts corresponding to each database.
[0091] For example, in the parent data lineage graph, the lineage graph related to database a is split into lineage graph a1, lineage graph a2, etc., based on database a; and in the parent data lineage graph, the lineage graph related to database b is split into lineage graph b1, lineage graph b2, etc., based on database b.
[0092] Specifically, generating sub-data lineage diagrams corresponding to each database based on the parent data lineage diagram includes:
[0093] For a parent data lineage graph generated from data tables of a single database, the parent data lineage graph is used as the child data lineage graph of that database;
[0094] For a parent data lineage graph generated from data tables of multiple databases, the parent data lineage graph is split into at least one sub-data lineage graph corresponding to a virtual library or an entity library.
[0095] Since the parent data lineage diagram illustrates the source and destination of each data field or each piece of data, that is, it can identify the relationship between upstream data tables, data fields, and each piece of data and downstream data tables, data fields, and each piece of data; therefore, by using a pre-set database name or other unique identification information that can represent the database, it can first identify which database's data table the components of the parent data lineage diagram belong to, or which database's data tables are combined to generate them.
[0096] Therefore, if it is identified that the components of a parent data lineage graph are generated from a single database table, there is no need to split it; the parent data lineage graph can be directly used as the sub-data lineage graph for subsequent generation scripts.
[0097] If it is identified that a parent data lineage graph is generated by combining data tables from multiple databases, then a virtual library is generated based on the parent data lineage graph. Virtual data table names and virtual field names are then established through the virtual library. Finally, the parent data lineage graph is split into at least one sub-data lineage graph corresponding to the virtual library as the sub-data lineage graph.
[0098] It's important to understand that the data virtualization management system can generate a virtual library for each overall lineage graph, making it easier for developers to manage based on different data calls and queries. Virtual tables are generated in the virtual library corresponding to the overall lineage graph and are intermediate datasets generated during the data modeling process. Because these intermediate datasets are not managed by entity tables, virtual tables need to be created to facilitate the management of relationships between data tables or data fields when the data lineage graph is split.
[0099] For example, data tables from database 1 (table 1) and database 2 (table 2) can generate intermediate dataset virtual tables (table a) with fields a1, a2, a3 and fields b1, b2, b3, ... or more by establishing rules and conditions. These virtual tables may then generate new dataset M by establishing rules and conditions, or virtual tables and entity tables may generate new dataset N by establishing rules and conditions. Because these virtual tables are intermediate data, without the management of these virtual tables, after the parent lineage graph is split and the datasets are obtained from the source databases, the lack of management of these intermediate datasets will cause the generation of dataset M and dataset N to become disconnected, making it impossible to reconstruct the original parent lineage graph and thus unable to generate the desired dataset result.
[0100] This allows for the processing and splitting of different lineage diagram components based on whether they are generated from a single or multiple database tables. This ensures that the corresponding lineage diagram components in the database are maintained, while also guaranteeing the automatic creation of a virtual database based on virtual tables and fields even without a source database. This provides a guarantee for adapting to the SQL syntax corresponding to any database type.
[0101] The allocation module 40 is used to assign corresponding unique identification information to each sub-data lineage map and to record the process and generation relationship between each sub-data lineage map;
[0102] While generating sub-data lineage maps, the system will automatically assign a unique identification information to each sub-data lineage map and record the process and generation relationship between each sub-data lineage map; so as to facilitate the subsequent generation of new target data lineage maps.
[0103] Script generation module 50 is used to generate data query scripts based on subdata lineage diagrams and database types;
[0104] The sub-data lineage diagram contains the relationships between database names, tables, and fields, ensuring that there are corresponding paths for querying each other. Therefore, appropriate data query scripts can be determined based on this relationship. Furthermore, the data virtualization management system stores relevant information during database integration, allowing for the estimation of the database type. Thus, by determining the corresponding script content in the script library based on the sub-data lineage diagram and database type, the appropriate data query script can be obtained. Alternatively, the appropriate script syntax can be selected from the script library based on the data lineage diagram and database type. This allows for automatic adaptation to the SQL syntax corresponding to any database type, enabling cross-database data association and analysis, while reducing the requirements for developers.
[0105] Query module 60 is used to retrieve the corresponding source dataset by querying a data query script;
[0106] Based on the generated data query scripts, the corresponding databases can be connected to generate the corresponding source datasets. Furthermore, the query scripts generated based on the lineage diagrams of each subdatabase can be connected to the corresponding databases to achieve cross-database and cross-database type queries and calls. This eliminates the need for developers to write corresponding data query scripts to adapt to different database types, reducing the requirements for developers.
[0107] The association module 70 is used to first associate the unique identification information with the source dataset, then replace the sub-data lineage map with the source dataset according to the association relationship, and finally generate the target data lineage map based on the process and generation relationship between each sub-data lineage map.
[0108] Module 80 is used to obtain the target dataset based on the parent data lineage map and the source dataset.
[0109] By associating assigned unique identification information with source datasets to generate relationships, the unique identification information can be used to link corresponding source datasets. The source datasets are then used to replace the sub-data lineage diagrams. Finally, the target data lineage diagram is generated based on the flow and generation relationships between each sub-data lineage diagram. This achieves the updating of the overall data lineage diagram, ensuring that subsequent data retrieval can be achieved using the overall target data lineage diagram and source datasets, resulting in high efficiency.
[0110] As can be seen from the above description, the present invention achieves the following technical effects:
[0111] In this embodiment, a data retrieval method based on data virtualization is adopted. This involves receiving data dictionary information from different databases; constructing a data retrieval model based on the data dictionary information; generating a parent data lineage graph based on the data retrieval model; generating at least one sub-data lineage graph corresponding to each database based on the parent data lineage graph; assigning corresponding unique identification information to each sub-data lineage graph and recording the flow and generation relationships between each sub-data lineage graph; generating a data query script based on the sub-data lineage graphs and database types; retrieving the corresponding source dataset through the data query script; first associating the unique identification information with the source dataset; then replacing the sub-data lineage graphs with the source dataset according to the association relationship; finally generating a target data lineage graph based on the flow and generation relationships between each sub-data lineage graph; and finally obtaining the target dataset based on the target data lineage graph and the source dataset. This achieves the goal of automatically adapting to the SQL syntax corresponding to any database type, thereby realizing cross-database data association retrieval analysis and reducing the technical requirements for developers. This solves the technical problem of not being able to implement cross-database data association retrieval analysis and having high requirements for data developers.
[0112] Obviously, those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps as a single integrated circuit module. Thus, the present invention is not limited to any particular hardware and software combination.
[0113] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A data retrieval method based on data virtualization, characterized in that, include: Receive data dictionary information from different databases; Data dictionary information received from different databases includes: The data virtualization manager interfaces with multiple different types of databases to obtain data dictionary information from these databases. The data dictionary information includes at least the database name, table name, field name, field-related information, conditions, and domain. A data retrieval model is constructed based on the data dictionary information, and a parent data lineage diagram is generated based on the data retrieval model; Generate at least one child data lineage map corresponding to each database based on the parent data lineage map; Assign a unique identification information to each sub-data lineage graph and record the process and generation relationship between each sub-data lineage graph; Generate a data query script based on the subdata lineage diagram and database type; The corresponding source dataset is obtained by querying using a data query script; First, the unique identification information is associated with the source dataset. Then, the sub-data lineage map is replaced with the source dataset according to the association relationship. Finally, the target data lineage map is generated based on the process and generation relationship between each sub-data lineage map. The target dataset is obtained based on the pedigree map of the target data and the source dataset.
2. The data retrieval method according to claim 1, characterized in that, A data retrieval model is constructed based on the data dictionary information, and a parent data lineage diagram is generated based on the data retrieval model, including: The data retrieval model is obtained by using icons or lines to represent the calling relationships of one or more databases, data tables, fields, field-related information, conditions, and domains in the data dictionary information; First, create a virtual library. Then, generate virtual tables and virtual fields for the intermediate datasets in the data call model corresponding to the virtual library. Finally, generate the parent data lineage diagram based on the icons and lines of the data call model, the set conditions, and the association rules.
3. The data retrieval method according to claim 1, characterized in that, Generating at least one child data lineage map corresponding to each database based on the parent data lineage map includes: For a parent data lineage graph generated from data tables of a single database, the parent data lineage graph is used as the child data lineage graph of that database; For a parent data lineage graph generated from data tables of multiple databases, the parent data lineage graph is split into at least one sub-data lineage graph corresponding to a virtual library or an entity library.
4. The data retrieval method according to claim 1, characterized in that, Data query scripts generated based on subdata lineage diagrams and database types include: Based on the subdata lineage diagram and database type, the corresponding script content is determined in the script library to obtain the corresponding data query script.
5. A data retrieval device based on data virtualization, characterized in that, include: The receiving module is used to receive data dictionary information from different databases; The receiving module includes: The data virtualization manager interfaces with multiple different types of databases to obtain data dictionary information from these databases. The data dictionary information includes at least the database name, table name, field name, field-related information, conditions, and domain. The module is used to build a data retrieval model based on the data dictionary information and generate a parent data lineage map based on the data retrieval model. The splitting module is used to generate target data lineage maps corresponding to each database based on the parent data lineage map; The allocation module is used to assign corresponding unique identification information to each sub-data lineage map and record the process and generation relationship between each sub-data lineage map; The script generation module is used to generate data query scripts based on subdata lineage diagrams and database types; The query module is used to retrieve the corresponding source dataset by querying data query scripts; The association module is used to first associate the unique identification information with the source dataset, then replace the sub-data lineage map with the source dataset according to the association relationship, and finally generate the target data lineage map based on the process and generation relationship between each sub-data lineage map. The acquisition module is used to obtain the target dataset based on the parent data lineage map and the source dataset.
6. The data retrieval device according to claim 5, characterized in that, The build module includes: The data retrieval model is obtained by using icons or lines to represent the calling relationships of one or more databases, data tables, fields, field-related information, conditions, and domains in the data dictionary information; First, create a virtual library. Then, generate virtual tables and virtual fields for the intermediate datasets in the data call model corresponding to the virtual library. Finally, generate the parent data lineage diagram based on the icons and lines of the data call model, the set conditions, and the association rules.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the data retrieval method based on data virtualization as described in any one of claims 1 to 4 when it is run.
8. A server, comprising: A memory and a processor, characterized in that the memory stores a computer program, wherein the processor is configured to run the computer program to perform the data retrieval method based on data virtualization as described in any one of claims 1 to 4.