Cross-database adaptation compatible processing method and system based on multiple data sources

Abstract

The invention relates to a cross-database adaptation compatible processing method and system based on multiple data sources, and belongs to the technical field of database access. The method comprises the steps of obtaining a source database, and obtaining general format data through a general format conversion model according to the source database; obtaining a global view through packaging and merging according to the general format data; the upper-layer application generates a query request according to the global view; obtaining an identifiable request through a cross-library operation middleware model according to the query request; and the source database obtains data according to the identifiable request to obtain a data operation result. And cross-database adaptive compatible processing based on multiple data sources is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of database access, and in particular relates to a cross-database adaptation and compatibility processing method and system based on multiple data sources. Background Art

[0002] The most important component of today's computer systems is the database system. At the same time, data is also the core of the enterprise. Different business systems need to interact with each other more frequently. However, the database management systems provided by different manufacturers lead to incompatible and cumbersome operations between databases. How to conveniently and efficiently transfer this data between heterogeneous databases and improve the adaptability and compatibility across databases has become an important issue in practical applications.

[0003] The federated database system can realize data access between multiple data sources, but it is not suitable for the integration of too large heterogeneous data sources and its scalability is poor; data warehouse technology is to establish a data storage warehouse at one level and periodically migrate the data updates in the data source to the data warehouse, but due to its frequent updates, it is not suitable for data processing between multiple heterogeneous databases; middleware technology is to use middleware to provide a global query platform to realize the call of data from heterogeneous data sources, but its data access efficiency needs further optimization. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a cross-database adaptation and compatibility processing method and system based on multiple data sources.

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] A cross-database adaptation and compatibility processing method based on multiple data sources, comprising:

[0007] Acquire a source database, and obtain universal format data according to the source database through a universal format conversion model;

[0008] Obtaining a global view by encapsulating and merging the common format data;

[0009] The upper layer application generates a query request according to the global view;

[0010] Obtaining an identifiable request through a cross-database operation middleware model according to the query request;

[0011] The source database acquires data according to the identifiable request to obtain a data manipulation result.

[0012] Preferably, obtaining the general format data through a general format conversion model according to the source database includes:

[0013] Acquire source format data, and perform data cleaning through ECL according to the source format data to obtain clean data;

[0014] Preset an intermediate library, and obtain a storage intermediate library by storing the clean data in the intermediate library;

[0015] The data is obtained from the storage intermediate library to obtain clean storage data, and the common format data is obtained by converting the clean storage data through the TL component.

[0016] Preferably, obtaining a recognizable request through a cross-database operation middleware model according to the query request includes:

[0017] Preset recognition rules and grammar rules, Lex recognizes the query request according to the recognition rules to obtain recognized words;

[0018] Yacc generates a syntax tree according to the recognized word through the syntax rules;

[0019] Obtaining a query plan tree through a rule conversion model according to the syntax tree;

[0020] Obtain an optimized query plan tree through digital dictionary optimization according to the query plan tree, wherein the optimized query plan tree carries an operation type label, and the operation type label includes a unary operation and a binary operation;

[0021] Obtaining a completely optimized query plan tree through in-database optimization judgment according to the optimized query plan tree;

[0022] Obtaining a decomposed query plan tree by decomposing a model according to the fully optimized query plan tree;

[0023] The identifiable request is obtained by converting the decomposed query plan tree through a Wrapper wrapper.

[0024] Preferably, the Lex is a lexical analysis generator, which can recognize words according to the recognition rules to obtain the recognized words; the Yacc is a grammatical analysis generator, which can generate the grammar tree from the words generated by the Lex according to the grammar rules.

[0025] Preferably, obtaining a query plan tree through a rule conversion model according to the syntax tree comprises:

[0026] Obtain a child node, where the child node is a child node of Fromt in the syntax tree, and obtain an aggregation parameter by performing a Cartesian product operation on the child node;

[0027] Obtaining a selection parameter through an aggregation operation according to the aggregation parameter;

[0028] Obtain projection parameters through a SELECT operation according to the selection parameters;

[0029] The query plan tree is obtained by performing a projection operation according to the projection parameters.

[0030] Preferably, the optimization judgment within the database includes judging whether the operation type label is a unary operation, if yes, the unary operation object is a leaf node; if no, the operation is moved downward.

[0031] Preferably, obtaining a decomposed query plan tree by decomposing a model according to the fully optimized query plan tree comprises:

[0032] Obtaining a leaf node of the fully optimized query plan tree, the leaf node carrying a source database type label, and obtaining a traversal success tree through post-order traversal judgment according to the leaf node and the source database type label, the traversal success tree carrying the number of leaves;

[0033] The post-order traversal judgment includes judging whether the database type labels are the same, if yes, continuing the traversal to obtain the traversal success tree; if no, stopping the traversal;

[0034] Obtain the maximum subtree that has been successfully traversed by sorting the number of leaves;

[0035] The successfully traversed maximum subtree is saved to obtain a remaining fully optimized query plan tree, the remaining fully optimized query plan tree carries a leaf quantity label, and the decomposed query plan tree is obtained by continuing traversal and judgment according to the remaining fully optimized query plan tree and the leaf quantity label;

[0036] The continued traversal judgment includes judging whether the leaf quantity label is 0, if yes, obtaining the decomposed query plan tree; if no, continuing the post-order traversal.

[0037] A cross-database adaptation and compatibility processing system based on multiple data sources, including a data conversion module, a data processing module, a request generation module, a request conversion module, and a request identification module, including:

[0038] The data conversion module is used to obtain a source database and obtain general format data according to the source database through a general format conversion model;

[0039] The data processing module is used to obtain a global view by encapsulating and merging the common format data;

[0040] The request generation module is used for an upper layer application to generate a query request according to the global view;

[0041] The request conversion module is used to obtain a recognizable request through a cross-database operation middleware model according to the query request;

[0042] The request identification module is used for the source database to obtain data according to the identifiable request to obtain data manipulation results.

[0043] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned cross-database adaptation and compatibility processing method based on multiple data sources is implemented.

[0044] A storage medium containing computer executable instructions, wherein the computer executable instructions are used to execute the above-mentioned cross-database adaptation and compatibility processing method based on multiple data sources when executed by a computer processor.

[0045] The beneficial effects of the present invention are:

[0046] (1) The ECL component is used to clean the data, the middleware is used to store the data, and the TL component is used to convert the data. This completely separates data cleaning and data conversion, which has strong stability. At the same time, the cleaned data is saved in the middleware. The data conversion performed by the TL component does not need to directly extract data from multiple heterogeneous data sources, which improves the efficiency of data conversion and increases the adaptability and compatibility across databases.

[0047] (2) The query plan is optimized in the middleware through the digital dictionary to obtain an optimized query plan tree, thereby improving data access efficiency. At the same time, the original optimization capabilities of various database management systems are utilized to optimize the optimized query plan tree in the database to obtain a fully optimized query plan tree, which can reduce the optimization complexity and further improve data access efficiency.

[0048] (3) By decomposing the fully optimized query plan tree into small query plan trees, and then passing the decomposed query plan trees to the Wrapper wrapper, the Wrapper wrapper is reassembled and encapsulated until it reaches the database language that the source database can recognize. This enhances the parallel execution capability during the query process and improves cross-database compatibility and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0050] Figure 1 The present invention provides a cross-database adaptation and compatibility processing method based on multiple data sources. DETAILED DESCRIPTION

[0051] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0052] See also Figure 1 , a cross-database adaptation and compatibility processing method based on multiple data sources, comprising:

[0053] S1: Acquire a source database, and obtain universal format data according to the source database through a universal format conversion model;

[0054] S2: obtaining a global view by encapsulating and merging the common format data;

[0055] S3: The upper layer application generates a query request according to the global view;

[0056] S4: Obtaining an identifiable request through a cross-database operation middleware model according to the query request;

[0057] S5: The source database acquires data according to the identifiable request to obtain a data manipulation result.

[0058] Specifically, the source database is a heterogeneous database including data sources.

[0059] In step S1, the general format conversion model includes:

[0060] S101: Acquire source format data, where the source format data represents data in the source database, and perform data cleaning through ECL according to the source format data to obtain clean data;

[0061] S102: Preset an intermediate library, and obtain a storage intermediate library by storing the clean data in the intermediate library;

[0062] S103: Acquire data from the storage intermediate library to obtain clean storage data, and convert the clean storage data through the TL component to obtain the general format data.

[0063] In this embodiment, the system includes a data source layer, a cross-database operation middleware layer, and an upper application layer. The data source layer is a heterogeneous multi-database system on the entire database environment; the cross-database operation middleware layer packages the source data of various heterogeneous database systems into common format data, and merges the source data of different database systems; the upper application layer generates a query request based on the merged data, and the cross-database operation middleware layer converts the query request into a request that the database can recognize.

[0064] In step S4, the cross-database operation middleware model includes:

[0065] S401: Preset recognition rules and grammar rules, and Lex recognizes the query request according to the recognition rules to obtain recognized words;

[0066] S402: Yacc generates a syntax tree according to the recognized word using the syntax rules;

[0067] S403: Obtaining a query plan tree through a rule conversion model according to the syntax tree;

[0068] S404: Obtain an optimized query plan tree by optimizing the query plan tree through a digital dictionary, wherein the optimized query plan tree carries an operation type label, and the operation type label includes a unary operation and a binary operation;

[0069] S405: Obtaining a completely optimized query plan tree through in-database optimization judgment according to the optimized query plan tree;

[0070] S406: Obtaining a decomposed query plan tree by decomposing a model according to the fully optimized query plan tree;

[0071] S407: Obtain the identifiable request through Wrapper conversion according to the decomposed query plan tree.

[0072] In this embodiment, the query request is written based on a custom SQL-like statement based on the SQL-92 standard.

[0073] Specifically, the Lex is a lexical analysis generator, which can recognize words according to the recognition rules to obtain the recognized words.

[0074] Specifically, the Yacc is a syntax analysis generator, which can generate the syntax tree through the grammar rules from the words generated by Lex.

[0075] In step S403, the rule conversion model includes:

[0076] S403-1: Obtain a child node, where the child node is a child node of Fromt in the syntax tree, and obtain an aggregation parameter by performing a Cartesian product operation on the child node;

[0077] S403-2: Obtaining selection parameters through aggregation operation according to the aggregation parameters;

[0078] S403-3: Obtain projection parameters through a SELECT operation according to the selection parameters;

[0079] S403-4: Obtain the query plan tree through a projection operation according to the projection parameters.

[0080] In step S405, the in-database optimization judgment includes judging whether the operation type label is a unary operation, and if so, the unary operation object is a leaf node;

[0081] If not, move the operation down.

[0082] In step S406, the decomposition model includes:

[0083] S406-1: Obtain leaf nodes of the fully optimized query plan tree, the leaf nodes carrying source database type labels, and obtain a traversal success tree through post-order traversal according to the leaf nodes and the source database type labels, the traversal success tree carrying the number of leaves;

[0084] S406-2: Obtain the maximum subtree that has been successfully traversed by sorting the number of leaves;

[0085] S406-3: Save the successfully traversed maximum subtree, obtain the remaining fully optimized query plan tree, the remaining fully optimized query plan tree carries a leaf number label, and obtain the decomposed query plan tree by continuing traversal based on the remaining fully optimized query plan tree and the leaf number label.

[0086] Specifically, the post-order traversal judgment includes judging whether the database type labels are the same, and if so, continuing the traversal to obtain the traversal success tree;

[0087] If not, stop traversing.

[0088] Specifically, the continued traversal judgment includes judging whether the leaf quantity label is 0, and if so, obtaining the decomposed query plan tree; if not, continuing the post-order traversal.

[0089] In step S407, converting the decomposed query plan tree through a Wrapper to obtain the recognizable request includes the cross-database operation middleware passing the decomposed query plan tree to the Wrapper; and the Wrapper converting the decomposed query plan tree into the recognizable request that can be understood by the source database system.

[0090] In this embodiment, the source database is a relational database, the decomposed query plan tree is converted into an SQL statement through the Wrapper wrapper, and the relational database obtains the data manipulation result by executing the SQL statement.

[0091] A cross-database adaptation and compatibility processing system based on multiple data sources, including a data conversion module, a data processing module, a request generation module, a request conversion module, and a request identification module, including:

[0092] The data conversion module is used to obtain a source database and obtain general format data according to the source database through a general format conversion model;

[0093] The data processing module is used to obtain a global view by encapsulating and merging the common format data;

[0094] The request generation module is used for an upper layer application to generate a query request according to the global view;

[0095] The request conversion module is used to obtain a recognizable request through a cross-database operation middleware model according to the query request;

[0096] The request identification module is used for the source database to obtain data according to the identifiable request to obtain data manipulation results.

[0097] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device.

[0098] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0099] The program code included on the computer readable medium can be transmitted with any appropriate medium, including but not limited to wireless, electric wire, optical cable, RF, etc., or any suitable combination of the above. The computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or their combinations, and the programming language includes object-oriented programming languages-such as Java, Smalltalk, C++, and also includes conventional procedural programming languages-such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0100] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A cross-database adaptation and compatibility processing method based on multiple data sources, characterized in that: The following steps are involved: Acquire a source database, and obtain universal format data according to the source database through a universal format conversion model; Obtaining a global view by encapsulating and merging the common format data; The upper layer application generates a query request according to the global view; Obtaining an identifiable request through a cross-database operation middleware model according to the query request; The source database acquires data according to the identifiable request to obtain a data manipulation result.

2. The cross-database adaptation and compatibility processing method based on multiple data sources according to claim 1 is characterized in that: The obtaining of the general format data through the general format conversion model according to the source database comprises: Acquire source format data, and perform data cleaning through ECL according to the source format data to obtain clean data; Preset an intermediate library, and obtain a storage intermediate library by storing the clean data in the intermediate library; The data is obtained from the storage intermediate library to obtain clean storage data, and the common format data is obtained by converting the clean storage data through the TL component.

3. The cross-database adaptation and compatibility processing method based on multiple data sources according to claim 1 is characterized in that: The obtaining of a recognizable request through a cross-database operation middleware model according to the query request comprises: Preset recognition rules and grammar rules, Lex recognizes the query request according to the recognition rules to obtain recognized words; Yacc generates a syntax tree according to the recognized word through the syntax rules; Obtaining a query plan tree through a rule conversion model according to the syntax tree; Obtain an optimized query plan tree through digital dictionary optimization according to the query plan tree, wherein the optimized query plan tree carries an operation type label, and the operation type label includes a unary operation and a binary operation; Obtaining a completely optimized query plan tree through in-database optimization judgment according to the optimized query plan tree; Obtaining a decomposed query plan tree by decomposing a model according to the fully optimized query plan tree; The identifiable request is obtained by converting the decomposed query plan tree through a Wrapper wrapper.

4. The cross-database adaptation and compatibility processing method based on multiple data sources according to claim 3 is characterized in that: The Lex is a lexical analysis generator, which can recognize words according to the recognition rules to obtain the recognized words; the Yacc is a grammatical analysis generator, which can generate the grammar tree from the words generated by the Lex according to the grammar rules.

5. The cross-database adaptation and compatibility processing method based on multiple data sources according to claim 3 is characterized in that: The obtaining of the query plan tree by converting the model according to the syntax tree through the rule comprises: Obtain a child node, where the child node is a child node of Fromt in the syntax tree, and obtain an aggregation parameter by performing a Cartesian product operation on the child node; Obtaining a selection parameter through an aggregation operation according to the aggregation parameter; Obtain projection parameters through a SELECT operation according to the selection parameters; The query plan tree is obtained by performing a projection operation according to the projection parameters.

6. The cross-database adaptation and compatibility processing method based on multiple data sources according to claim 3 is characterized in that: The in-database optimization judgment includes judging whether the operation type label is a unary operation, if yes, the unary operation object is a leaf node; if no, the operation is moved downward.

7. The cross-database adaptation and compatibility processing method based on multiple data sources according to claim 3 is characterized in that: The step of obtaining a decomposed query plan tree by decomposing a model according to the fully optimized query plan tree comprises: Obtaining a leaf node of the fully optimized query plan tree, the leaf node carrying a source database type label, and obtaining a traversal success tree through post-order traversal judgment according to the leaf node and the source database type label, the traversal success tree carrying the number of leaves; The post-order traversal judgment includes judging whether the database type labels are the same, if yes, continuing the traversal to obtain the traversal success tree; if no, stopping the traversal; Obtain the maximum subtree that has been successfully traversed by sorting the number of leaves; The successfully traversed maximum subtree is saved to obtain a remaining fully optimized query plan tree, the remaining fully optimized query plan tree carries a leaf quantity label, and the decomposed query plan tree is obtained by continuing traversal and judgment according to the remaining fully optimized query plan tree and the leaf quantity label; The continued traversal judgment includes judging whether the leaf quantity label is 0, if yes, obtaining the decomposed query plan tree; if no, continuing the post-order traversal.

8. A cross-database adaptation and compatibility processing system based on multiple data sources, comprising a data conversion module, a data processing module, a request generation module, a request conversion module, and a request identification module, characterized in that: include: The data conversion module is used to obtain a source database and obtain general format data according to the source database through a general format conversion model; The data processing module is used to obtain a global view by encapsulating and merging the common format data; The request generation module is used for an upper layer application to generate a query request according to the global view; The request conversion module is used to obtain a recognizable request through a cross-database operation middleware model according to the query request; The request identification module is used for the source database to obtain data according to the identifiable request to obtain data manipulation results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the cross-database adaptation and compatibility processing method based on multiple data sources is implemented as described in any one of claims 1-7.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute the cross-database adaptation and compatibility processing method based on multiple data sources as described in any one of claims 1 to 7 when executed by a computer processor.

Images