Incremental data acquisition method and device, storage medium and program product

Abstract

The invention provides an incremental data acquisition method and device, a storage medium and a program product, and the method comprises the steps: determining the name of a target PDB of to-be-acquired incremental data by responding to an incremental data acquisition instruction for acquiring any PDB; according to the name of the target PDB, the incremental SQL data of the target PDB is extracted from the redo log of the source end CDB corresponding to the target PDB, so that useless incremental SQL data can be prevented from being obtained; according to the incremental SQL data of the target PDB, the meta-information of the target data table related to the incremental SQL data is obtained, the incremental data of the target PDB comprises the incremental SQL data and the meta-information of the target data table, only the target PDB needs to be connected, the meta-information of the target data table is obtained from the target PDB, the efficiency of obtaining the incremental data of the specified PDB is improved, and then the PDB incremental data synchronization efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular, to an incremental data acquisition method, device, storage medium, and program product. Background Art

[0002] A multi-tenant container database (Oracle Multitenant Container Database, abbreviated as CDB) can accommodate one or more pluggable databases (Pluggable Database, abbreviated as PDB). This feature allows multiple PDBs to be created and maintained in the CDB. Each PDB exists independently in the CDB. When a PDB is used alone, it is no different from an ordinary database.

[0003] Currently, when implementing incremental data synchronization for an arbitrarily specified PDB in a CDB, the efficiency of obtaining incremental data for the specified PDB is low, resulting in low efficiency of PDB incremental data synchronization. How to improve the efficiency of obtaining PDB incremental data is the key to improving the efficiency of PDB incremental data synchronization. Summary of the Invention

[0004] This application provides an incremental data acquisition method, device, storage medium, and program product to improve the efficiency of obtaining PDB incremental data, thereby improving the efficiency of PDB incremental data synchronization.

[0005] In a first aspect, this application provides an incremental data acquisition method, including:

[0006] Responding to an incremental data acquisition instruction for obtaining any pluggable database PDB, determining the name of the target PDB for which incremental data is to be obtained;

[0007] According to the name of the target PDB, extracting incremental SQL data of the target PDB from the redo log of the source CDB corresponding to the target PDB, where the incremental SQL data includes SQL statements executed in the target PDB;

[0008] According to the incremental SQL data of the target PDB, obtaining meta-information of the target data tables involved in the incremental SQL data, where the incremental data of the target PDB includes the incremental SQL data and the meta-information of the target data tables.

[0009] In a second aspect, this application provides an incremental data acquisition device, including:

[0010] A target PDB determination module, configured to respond to an incremental data acquisition instruction for obtaining any pluggable database PDB, and determine the name of the target PDB for which incremental data is to be obtained;

[0011] An incremental SQL data acquisition module, configured to extract the incremental SQL data of the target PDB from the redo log of the source CDB corresponding to the target PDB according to the name of the target PDB, where the incremental SQL data includes SQL statements executed for the target PDB;

[0012] A meta-information acquisition module, configured to acquire the meta-information of the target data table involved in the incremental SQL data according to the incremental SQL data of the target PDB, where the incremental data of the target PDB includes the incremental SQL data and the meta-information of the target data table.

[0013] In a third aspect, the present application provides an electronic device, including: a memory, a processor, and a transceiver;

[0014] The memory stores computer-executable instructions;

[0015] The processor executes the computer-executable instructions stored in the memory to implement the method as described in the first aspect.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium, where computer-executable instructions are stored in the computer-readable storage medium, and when the computer-executable instructions are executed by a processor, they are used to implement the method as described in the first aspect.

[0017] In a fifth aspect, the present application provides a computer program product, including a computer program, where when the computer program is executed by a processor, it implements the method as described in the first aspect.

[0018] The present application provides an incremental data acquisition method, device, storage medium, and program product. The method determines the name of the target PDB for which incremental data is to be acquired by responding to an incremental data acquisition instruction for any pluggable database PDB; extracts the incremental SQL data of the target PDB from the redo log of the source CDB corresponding to the target PDB according to the name of the target PDB, where the incremental SQL data includes SQL statements executed for the target PDB; obtains the meta-information of the target data tables involved in the incremental SQL data according to the incremental SQL data of the target PDB. The incremental data of the target PDB includes the incremental SQL data and the meta-information of the target data tables, enabling the acquisition of only the incremental SQL data corresponding to the target PDB according to the name of the target PDB when acquiring the incremental data of the target PDB, and avoiding the acquisition of useless incremental SQL data of the remaining PDBs that do not need to be synchronized; when obtaining the meta-information of the target data tables involved in the incremental SQL data, only the target PDB needs to be connected to obtain the meta-information of the target data tables; compared with the solution of first acquiring the incremental SQL data of all PDBs and the meta-information of the involved data tables and then filtering out the incremental data of the target PDB, it can avoid the acquisition of useless incremental SQL data and reduce the number of PDB connection switches, improving the efficiency of acquiring the incremental data of the specified PDB, and thus improving the incremental data synchronization efficiency of the PDB. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are incorporated herein and constitute a part of this specification, showing embodiments consistent with the present application and, together with the specification, are used to explain the principles of the present application.

[0020] Figure 1 It is a schematic diagram of the architecture of the incremental data synchronization system provided by an embodiment of the present application;

[0021] Figure 2 It is a schematic flowchart of an incremental data acquisition method provided by an embodiment of the present application;

[0022] Figure 3 It is a schematic flowchart of a method for extracting the incremental SQL data of the target PDB provided by an embodiment of the present application;

[0023] Figure 4 It is a schematic flowchart of a multi-threaded collaboration for acquiring the incremental data of the target PDB provided by an embodiment of the present application;

[0024] Figure 5 It is an example diagram of the process framework for incremental data acquisition provided by an embodiment of the present application;

[0025] Figure 6 It is a schematic diagram of the structure of an incremental data acquisition device provided by an embodiment of the present application;

[0026] Figure 7 The structural schematic diagram of an electronic device provided by an embodiment of the present application.

[0027] Through the above-mentioned drawings, the specific embodiments of the present application have been shown, and more detailed descriptions will be provided later. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. Detailed implementation manners

[0028] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numerals in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0029] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0030] It should be noted that in the embodiments of the present application, some industry-existing solutions such as certain software, components, models, etc. may be mentioned. They should be considered exemplary, and their purpose is only to illustrate the feasibility in the implementation of the technical solution of the present application, but it does not mean that the applicant has already or necessarily used this solution.

[0031] First, the nouns involved in the present application are explained:

[0032] Multitenant Container Database (Oracle Multitenant Container Database, abbreviated as CDB): Also known as an insertable database, it refers to a database that can accommodate one or more Pluggable Databases (abbreviated as PDB). Multiple databases can be created and maintained in a CDB. The databases created in a CDB are called PDBs. Each PDB exists independently in the CDB. When using a PDB alone, it has no difference from an ordinary database.

[0033] Java Database Connectivity (JDBC): A Java API (Application Programming Interface) for executing Structured Query Language (SQL) statements, which can provide a unified access interface for multiple databases. JDBC includes a set of classes and interfaces written in the Java language, and its main functions are: establishing a connection with the database, sending database operation statements, and processing results.

[0034] System Change Number (SCN): A unique number used in Oracle databases to identify each change in the database. It can identify each committed transaction, ensuring that each transaction can be uniquely identified. SCN is a logical identifier of a timestamp, used to track and manage database transactions, identify the timing and version of each committed transaction, and ensure the consistency and recovery ability of the database.

[0035] Transaction: A logical unit composed of a series of database operations. These operations are either all executed successfully or none of them are executed. If any one operation fails, all operations will not be applied, and the database will be restored to the state before the transaction started. For example, transactions such as user registration and order deletion.

[0036] Logminer: A tool in Oracle databases used to read the archived logs and online logs of Oracle databases to understand the operation history and data changes of Oracle databases. It can extract the information in the redo logs and parse it into corresponding SQL statements.

[0037] Figure 1 This is a schematic diagram of the architecture of the incremental data synchronization system provided by the embodiments of this application. As Figure 1 shown, the incremental data synchronization system includes: a source database, a data synchronization tool, and a target database. Among them, the data synchronization tool includes a source synchronization software and a target synchronization software.

[0038] Among them, the source database refers to the database where the data to be synchronized is currently located. The target database refers to the target database to which the data is to be synchronized. During data synchronization, the data in the source database is synchronized to the target database.

[0039] The process of real-time data synchronization using a data synchronization tool is divided into three stages: The first stage is to perform an initial load of the existing data to obtain the basis for data synchronization; the second stage is to perform incremental data synchronization based on the synchronization basis established by the initial data load; the third stage is to periodically compare and verify the source data and target data of the data synchronization to confirm that no data is lost during the data synchronization process. Among them, the second stage and the third stage will be in a long-term parallel state.

[0040] When performing incremental data synchronization, the incremental data to be synchronized is extracted from the source database through the source-side synchronization software. The incremental data can cover the incremental data generated by the operations performed on the source database since the last data synchronization, including but not limited to the incremental data of new, update, or delete operations. The extraction of incremental data is usually to reflect the latest state in the source database, rather than a full copy.

[0041] Among them, the incremental data includes the SQL statements executed on the source database and the meta-information of the target data tables involved in the SQL statements.

[0042] The source-side synchronization software transfers the incremental data of the source database to the target-side synchronization software. The target-side synchronization software executes the corresponding SQL statements on the target database according to the incremental data to ensure data consistency between the target database and the source database.

[0043] In the data synchronization scenario where the source database and the target database are of the same type of database, by executing the SQL statements in the incremental data extracted from the source database in the target database, the incremental data synchronization between the same type of databases can be achieved.

[0044] In the data synchronization scenario where the source database and the target database are of different types of databases, the SQL statements in the incremental data extracted from the source database cannot be directly executed in the target database. The source-side synchronization software is also responsible for encapsulating the incremental data extracted from the source database into a preset data structure. Among them, the preset data structure is a custom data format that can be recognized and understood by both the source-side synchronization software and the target-side synchronization software.

[0045] The target-side synchronization software is responsible for converting the incremental data encapsulated into the preset data structure received from the source-side synchronization software into SQL statements executable on the target database. Further, by executing the converted SQL statements in the target database, the incremental data synchronization between the source database and the target database of different types can be achieved.

[0046] In some example scenarios, the data synchronization tool may further include a filtering module, which is configured to filter and transform the incremental data extracted by the source - side synchronization software according to preset filtering rules. For example, it converts the name of a data table in the source - side database into the table name of the corresponding data table in the target - side database, realizing the mapping of table names from the source - side database to the target - side database. For example, it filters out SQL statements corresponding to specified operations (such as query operations), as these operations do not change the data within the data table.

[0047] In a multi - tenant container database (CDB), one or more pluggable databases (PDBs) can be created and maintained. Each PDB exists independently within the CDB. When a PDB is used alone, it has no difference from a normal database. Currently, when implementing the synchronization of incremental data of any specified PDB in the source - side CDB to the corresponding PDB in the target - side CDB, the Logminer tool is usually used to obtain the incremental data of all PDBs (it is necessary to switch PDB connections multiple times to obtain the data table meta - information in each PDB), and then filter out the incremental data of the specified PDB to be synchronized. The efficiency of obtaining the incremental data of the specified PDB is low, resulting in low efficiency of PDB incremental data synchronization. How to improve the efficiency of obtaining PDB incremental data is the key to improving the efficiency of PDB incremental data synchronization.

[0048] To solve the above - mentioned technical problems, an embodiment of the present application proposes an incremental data acquisition method. The method determines the name of the target PDB for which incremental data is to be acquired in response to an incremental data acquisition instruction for obtaining incremental data of any pluggable database (PDB); according to the name of the target PDB, extracts the incremental SQL data of the target PDB from the redo log of the source - side CDB corresponding to the target PDB, where the incremental SQL data includes SQL statements executed for the target PDB; according to the incremental SQL data of the target PDB, obtains the meta - information of the target data tables involved in the incremental SQL data. The incremental data of the target PDB includes the incremental SQL data and the meta - information of the target data tables, so that when obtaining the incremental data of the target PDB, it is possible to obtain only the incremental SQL data corresponding to the target PDB according to the name of the target PDB, avoiding the acquisition of useless incremental SQL data of the remaining PDBs that do not need to be synchronized; when obtaining the meta - information of the target data tables involved in the incremental SQL data, it is only necessary to connect to the target PDB and obtain the meta - information of the target data tables from the target PDB; compared with the solution of first obtaining the incremental SQL data of all PDBs and the meta - information of the involved data tables and then filtering out the incremental data of the target PDB, it can avoid the acquisition of useless incremental SQL data and reduce the number of PDB connection switches, improving the efficiency of obtaining the incremental data of the specified PDB, and thus improving the efficiency of PDB incremental data synchronization.

[0049] The following uses specific embodiments to elaborate in detail on the technical solution of this application and how the technical solution of this application solves the above technical problems. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below in conjunction with the accompanying drawings.

[0050] Figure 2 It is a schematic flowchart of an incremental data acquisition method provided by an embodiment of this application. The execution subject of this method can be an electronic device for implementing incremental data acquisition, which is implemented in a software and hardware manner, and can specifically be a server deployed locally or in the cloud. As Figure 2 shown, the specific steps of this method are as follows:

[0051] S201. In response to an incremental data acquisition instruction for obtaining any pluggable database PDB, determine the name of the target PDB for which incremental data is to be obtained.

[0052] Among them, the incremental data acquisition instruction for any pluggable database PDB refers to the incremental data acquisition instruction sent to the source-side synchronization software when the incremental data synchronization function of the data synchronization tool is selected and triggered for execution. This incremental data acquisition instruction contains the name of the target PDB to be synchronously incrementally, or the target PDB to be synchronously incrementally is stored in a configuration file.

[0053] Exemplarily, when a user needs to implement incremental data synchronization from any PDB in the source-side CDB to the corresponding PDB in the target-side CDB, an incremental data synchronization instruction for this PDB is sent to the data synchronization tool. In response to receiving the incremental data synchronization instruction for this PDB, the data synchronization tool sends an incremental data acquisition instruction for this PDB to the source-side synchronization software. In response to receiving the incremental data acquisition instruction for this PDB, the source-side synchronization software extracts the name of the PDB for which incremental data is to be obtained from the incremental data acquisition instruction for this PDB to obtain the name of the target PDB; or, the source-side synchronization software reads the name of the specified PDB to be data-synchronized from the configuration file as the name of the target PDB.

[0054] When specifying the target PDB to be data-synchronized, the user can fill in or adjust the name of the specified PDB to be data-synchronized in the configuration file, or fill in or select the name of the specified PDB to be data-synchronized in the visual interface provided by the data synchronization tool, and the data synchronization tool stores the name of the specified PDB in the configuration file.

[0055] S202. According to the name of the target PDB, extract the incremental SQL data of the target PDB from the redo log of the source-side CDB corresponding to the target PDB, and the incremental SQL data includes the SQL statements executed in the target PDB.

[0056] Among them, the redo log of the source CDB records the logs of modification operations on all PDBs in the source CDB, including but not limited to the type of operation, time (such as SCN), executed SQL statements, relevant information of the SQL statements, etc. The modification operations include insert, update, delete, etc.

[0057] Through database redo log analysis tools (such as Logminer, Flashback, etc.), the redo log of the source CDB can be analyzed to obtain the SQL statements executed in all PCBs in the redo log and the relevant information of the SQL statements.

[0058] In this embodiment, the name of the target PDB is used as the input parameter of the redo log analysis tool to specify the target PDB for which incremental SQL data needs to be obtained, so that the redo log analysis tool analyzes the redo log of the source CDB and only returns the SQL statements executed in the target PDB and the relevant information of the SQL statements.

[0059] In addition, in order to obtain the incremental SQL data of the target PDB, a time interval is specified for the redo log analysis tool, so that the redo log analysis tool analyzes the redo log within this time interval in the source CDB and only returns the SQL statements executed in the target PDB within this time interval and the relevant information of the SQL statements, and obtains the incremental SQL data of the target PDB within this time interval. Among them, the time interval can be determined by specifying the start time and end time, or the time interval can also be an SCN interval determined by specifying the start SCN and end SCN.

[0060] Among them, the incremental SQL data of the target PDB can include the SQL statements executed in the target PDB and the relevant information of the SQL statements. The relevant information of the SQL statements can include but not limited to: operation type, transaction information of the target PDB (such as transaction identifier, SCN, timestamp, etc.) and the name of the target PDB.

[0061] S203. According to the incremental SQL data of the target PDB, obtain the meta-information of the target data tables involved in the incremental SQL data. The incremental data of the target PDB includes incremental SQL data and the meta-information of the target data tables.

[0062] After obtaining the incremental SQL data of the target PDB, in this step, according to the incremental SQL data, extract the names of the target data tables from the SQL statements, and query the meta-information of the target data tables from the target PDB according to the names of the target data tables.

[0063] Among them, the meta-information of the data table is also called the metadata of the data table, which refers to the information about the data table itself and describes the structure and attributes of the table. The meta-information of the data table usually includes: table name, column name, data type corresponding to the column (such as integer, string, date, etc.), constraints, indexes, etc.

[0064] Furthermore, by encapsulating the incremental SQL data of the target PDB and the meta-information of the target data table according to a preset data structure, the incremental data of the target PDB can be obtained.

[0065] Among them, the preset data structure is a data structure that the target-side database can support, which can be selected or set according to the needs of actual applications, and this embodiment does not limit this.

[0066] In this embodiment, encapsulating the incremental SQL data of the target PDB and the meta-information of the target data table into a data structure that the target-side database can support can ensure that the data structure of the incremental data of the target PDB meets the requirements of the target database and can be executed and stored in the target database.

[0067] In the embodiment of the present application, by responding to the instruction to obtain the incremental data of any pluggable database PDB, the name of the target PDB for which the incremental data is to be obtained is determined; according to the name of the target PDB, the incremental SQL data of the target PDB is extracted from the redo log of the source-side CDB corresponding to the target PDB, and the incremental SQL data includes the SQL statements executed for the target PDB; according to the incremental SQL data of the target PDB, the meta-information of the target data table involved in the incremental SQL data is obtained. The incremental data of the target PDB includes the incremental SQL data and the meta-information of the target data table, so that when obtaining the incremental data of the target PDB, only the incremental SQL data corresponding to the target PDB can be obtained according to the name of the target PDB, and the useless incremental SQL data of the remaining PDBs that do not need to be synchronized can be avoided; when obtaining the meta-information of the target data table involved in the incremental SQL data, only the target PDB needs to be connected to obtain the meta-information of the target data table from the target PDB; compared with the scheme of first obtaining the incremental SQL data of all PDBs and the meta-information of the involved data tables and then screening out the incremental data of the target PDB, the acquisition of useless incremental SQL data can be avoided, and the number of PDB connection switches can be reduced, improving the efficiency of obtaining the incremental data of the specified PDB, and thus improving the efficiency of PDB incremental data synchronization.

[0068] Figure 3 It is a schematic flowchart of a method for extracting the incremental SQL data of the target PDB provided by the embodiment of the present application. In one implementation, in the foregoing S202, according to the name of the target PDB, the incremental SQL data of the target PDB is extracted from the redo log of the source-side CDB corresponding to the target PDB, asFigure 3 As shown in the figure, the following steps can be specifically adopted to achieve it:

[0069] S301. Determine the SCN interval according to the data synchronization requirements of the target PDB.

[0070] When performing incremental data synchronization, the starting SCN and the ending SCN can be determined according to the data synchronization requirements, obtaining the SCN interval from the starting SCN to the ending SCN, and instructing the data synchronization tool to synchronize the incremental data within this SCN interval.

[0071] Among them, the starting SCN can be the ending SCN of the SCN interval corresponding to the previous data synchronization, and the starting SCN can also be the SCN specified by the user according to the data synchronization requirements. The ending SCN can be the current SCN of the source CDB corresponding to the target PDB (i.e., the latest state of the source CDB), and the ending SCN can also be the SCN specified by the user according to the data synchronization requirements (i.e., a specific SCN to which synchronization is required).

[0072] By using the data synchronization tool to synchronize the incremental data of the target PDB in the source CDB within this SCN interval to the corresponding PDB in the target CDB, the target PDB in the target CDB can be updated to the latest state, ensuring the data consistency between the target PDB in the target CDB and the target PDB in the source CDB.

[0073] S302. Through the redo log analysis tool, create a view according to the SCN interval, where the view contains the SQL statements and related information of the SQL statements in the redo log of the source CDB that fall within the SCN interval.

[0074] In this step, the determined SCN interval is used as the input parameter of the redo log analysis tool, the redo log analysis tool is started, and the redo log analysis tool analyzes the logs in the redo log of the source CDB that fall within the SCN interval to create a view corresponding to the SCN interval. This view contains the SQL statements and related information of the SQL statements in the redo log of the source CDB that fall within the SCN interval.

[0075] S303. Query the incremental SQL data of the target PDB from the view according to the name of the target PDB.

[0076] In this step, according to the name of the target PDB, query the SQL statements and related information of the target PDB from the view corresponding to the SCN interval to obtain the incremental SQL data of the target PDB.

[0077] Exemplarily, taking the use of the redo log analysis tool Logminer as an example, when starting Logminer, specify the SCN range for Logminer. After Logminer starts, analyze the logs in the redo log of the source CDB that fall within the specified SCN range, and create a view corresponding to the SCN range (such as the v$logmnr_contents view).

[0078] Further, use the name of the target PDB as an input parameter to send a query request to Logminer. Through Logminer, query the SQL statements corresponding to the name of the target PDB and the relevant information of the SQL statements from the view v$logmnr_contents corresponding to the SCN range, and the incremental SQL data of the target PDB can be obtained.

[0079] Optionally, after obtaining the SQL statements of the target PDB and the relevant information of the SQL statements, the redo log analysis tool can be closed to release resources in a timely manner.

[0080] It should be noted that in the real-time data synchronization scenario, the above steps S301 - S303 can be continuously repeated to achieve real-time acquisition of the incremental SQL data of the target PDB.

[0081] Exemplarily, in the real-time data synchronization scenario, each time incremental data synchronization is performed, the end SCN of the SCN range corresponding to the previous data synchronization is used as the start SCN, and the SCN obtained by adding a preset step size to the start SCN is used as the end SCN; determine the SCN range corresponding to the current data synchronization according to the start SCN and the end SCN. Among them, the preset step size can be set according to the actual application needs, and this embodiment does not limit it. For example, the preset step size can be set to 5.

[0082] Use the SCN range corresponding to the current data synchronization as the input parameter of the redo log analysis tool, start the redo log analysis tool, and analyze the logs in the redo log of the source CDB that fall within the SCN range corresponding to the current data synchronization through the redo log analysis tool, and create a view corresponding to the SCN range corresponding to the current data synchronization. This view contains the SQL statements in the redo log of the source CDB that fall within the SCN range corresponding to the current data synchronization and the relevant information of the SQL statements.

[0083] According to the name of the target PDB, query the SQL statements related to the target PDB and the relevant information of the SQL statements from the view corresponding to the SCN range corresponding to the current data synchronization, and obtain the incremental SQL data corresponding to the target PDB in the current data synchronization.

[0084] Further, according to the incremental SQL data corresponding to the target PDB in this data synchronization, obtain the meta-information of the target data table involved in the incremental SQL data, and then the incremental data of the target PDB in this data synchronization can be obtained. Based on the incremental data of the target PDB in this data synchronization, the incremental data synchronization of the target PDB can be achieved.

[0085] By continuously repeating the above process of incremental data synchronization, the real-time incremental data synchronization of the target PDB can be achieved.

[0086] It should be noted that in any incremental data synchronization process, if the SCN obtained by adding the preset step size to the starting SCN is greater than the current SCN of the source CDB corresponding to the target PDB, then the current SCN of the source CDB corresponding to the target PDB is used as the ending SCN.

[0087] In this embodiment, the current system change number SCN of the target CDB corresponding to the target PDB is used as the starting SCN, and the current SCN of the source CDB is used as the ending SCN to determine the SCN interval corresponding to the incremental data; through the redo log analysis tool, a view containing SQL statements and related information of the SQL statements is created according to the SCN interval, and the incremental SQL data of the target PDB is queried from the view according to the name of the target PDB, so that the incremental SQL data of the target PDB to be synchronized can be accurately extracted, avoiding obtaining useless incremental SQL data of other PDBs and improving the efficiency of obtaining the incremental data of the target PDB.

[0088] Based on the above embodiment, in an optional embodiment, the acquisition of the incremental data of the target PDB can be completed through multi-thread cooperation. Figure 4 This is a schematic flow diagram of a multi-thread cooperation for obtaining the incremental data of the target PDB provided by the embodiment of the present application. As Figure 4 shown, the acquisition of the incremental data of the target PDB through multi-thread cooperation can be specifically implemented by the following steps:

[0089] S401. Create an Extractor thread, create a Reader thread through the Extractor thread, and create a Parser thread through the Reader thread.

[0090] In the Oracle database, during the instantiation phase of data loading, the source synchronization software creates an Extractor thread, creates a Reader thread through the Extractor thread, and creates a Parser thread through the Reader thread. Through the cooperation of the Extractor thread, Reader thread, and Parser thread, the acquisition of the incremental data of the target PDB is completed.

[0091] S402. Receive the incremental data acquisition instruction of the target PDB through the Extractor thread, and determine the name of the target PDB for which the incremental data is to be acquired.

[0092] When the incremental data synchronization function of the data synchronization tool is selected and triggered to execute, an incremental data acquisition instruction for the target PDB is sent to the source-side synchronization software. The source-side synchronization software receives the incremental data acquisition instruction of the target PDB through the Extractor thread, and determines the name of the target PDB for which the incremental data is to be acquired.

[0093] Exemplarily, when the user needs to implement the incremental data synchronization of any PDB in the source-side CDB to the corresponding PDB in the target-side CDB, an incremental data synchronization instruction for this PDB is sent to the data synchronization tool. The data synchronization tool, in response to receiving the incremental data synchronization instruction for this PDB, sends an incremental data acquisition instruction for this PDB to the source-side synchronization software. The source-side synchronization software receives the incremental data acquisition instruction for this PDB through the Extractor thread, and queries the name of the PDB for which the incremental data is to be acquired from the incremental data acquisition instruction for this PDB as the name of the target PDB; or, the Extractor thread queries the name of the specified PDB for which the data is to be synchronized from the configuration file as the name of the target PDB. S403. Through the Reader thread, according to the name of the target PDB, extract the incremental SQL data of the target PDB from the redo log of the source-side CDB corresponding to the target PDB, and transmit the incremental SQL data to the Parser thread.

[0094] In this step, when it is necessary to acquire the incremental SQL data of the target PDB, the Reader thread extracts the incremental SQL data of the target PDB from the redo log of the source-side CDB corresponding to the target PDB according to the name of the target PDB.

[0095] Exemplarily, after the Reader thread starts, by continuously adjusting the SCN range and starting the redo log analysis tool (such as Logminer) according to the adjusted SCN range, the redo log analysis tool analyzes the logs in the source-side CDB's redo log that fall within the current SCN range and creates a view corresponding to the current SCN range. Thus, the Reader thread can preload the views corresponding to each SCN range.

[0096] When it is necessary to acquire the incremental SQL data of the target PDB, the Reader thread queries and obtains the incremental SQL data of the target PDB from the created target view according to the name of the target PDB.

[0097] After obtaining the incremental SQL data of the target PDB, the Reader thread transfers the incremental SQL data of the target PDB to the Parser thread.

[0098] Exemplarily, the Reader thread adds the incremental SQL data of the target PDB to the first queue in units of transactions. The Parser thread fetches the incremental SQL data of the target PDB from the first queue in units of transactions. Among them, the SQL statements in the incremental SQL data of the target PDB can be divided into multiple transactions, and each transaction contains a set of SQL statements.

[0099] Specifically, the Reader thread divides the SQL statements and related information of the SQL statements in the incremental SQL data of the target PDB into units of transactions, encapsulates the SQL statements and related information contained in each transaction into a data unit, and adds it to the first queue.

[0100] In addition, the Reader thread and the Parser thread can also adopt other methods of inter-thread data communication, which are not specifically limited here.

[0101] S404. Through the Parser thread, according to the incremental SQL data of the target PDB, obtain the meta-information of the target data tables involved in the incremental SQL data, encapsulate the incremental SQL data and the meta-information of the target data tables into the incremental data of the target PDB, and transfer the incremental data of the target PDB to the Extractor thread.

[0102] The Parser thread takes out the incremental SQL data of the target PDB from the first queue.

[0103] After obtaining the incremental SQL data of the target PDB, through the Parser thread, parse the incremental SQL data to determine the names of the target data tables involved in the incremental SQL data. Further, through the Parser thread, query the meta-information of the target data tables from the target PDB according to the names of the target data tables.

[0104] Exemplarily, the Parser thread can take out the data unit corresponding to one transaction (including SQL statements and related information of the SQL statements) from the first queue each time, parse the SQL statements contained in the transaction, determine the names of the target data tables involved in the transaction, and query the meta-information of the target data tables involved in the transaction from the target PDB.

[0105] Optionally, the Parser thread can also query and obtain the meta-information of all target data tables involved in the incremental SQL data from the target PDB at one time after parsing and determining all the target data tables involved in the incremental SQL data of the target PDB.

[0106] In an optional embodiment, through the Parser thread, the meta-information of the data table that has been obtained can be stored in the cache.

[0107] When it is necessary to obtain the meta-information of any target data table, through the Parser thread, first determine whether the meta-information of the target data table exists in the cache. If the meta-information of the target data table already exists in the cache, the Parser thread directly obtains the meta-information of the target data table from the cache.

[0108] If the meta-information of the target data table does not exist in the cache, the Parser thread queries and obtains the meta-information of the target data table from the target PDB, and stores the obtained meta-information of the target data table in the cache.

[0109] Specifically, when querying and obtaining the meta-information of the target data table from the target PDB, the Parser thread first determines whether the PDB of the current session is the target PDB, that is, determines whether the PDB connected currently is the target PDB.

[0110] Exemplarily, the Parser thread can determine the database connected currently according to the database connection (such as JDBC) in the current session. If the database connected currently is the target PDB, it means that the PDB of the current session is the target PDB. If the database connected currently is not the target PDB, it means that the PDB of the current session is not the target PDB.

[0111] If it is determined that the PDB of the current session is the target PDB, the Parser thread can obtain the meta-information of the target data table from the target PDB through the current database connection (such as JDBC).

[0112] If the PDB of the current session is not the target PDB, the Parser thread switches the PDB of the current session to the target PDB, that is, establishes a database connection (such as JDBC) with the target PDB, and obtains the meta-information of the target data table from the database of the target PDB through the database connection with the target PDB.

[0113] After obtaining the meta-information of any target data table, the Parser thread can store the meta-information of the target data table in the cache, so that when it is necessary to obtain the meta-information of the target data table next time, it can be directly obtained from the cache, thereby saving resources and improving the efficiency of obtaining the meta-information of the target data table.

[0114] In this embodiment, by determining the target PDB, it is judged whether the PDB of the current session is the target PDB. When the PDB of the current session is the target PDB, there is no need to switch the PDB of the current session, and the meta-information of the target data table can be directly obtained from the target PDB through the current database connection. When the PDB of the current session is not the target PDB, the PDB of the current session is switched to the target PDB to establish a database connection with the target PDB, and then the meta-information of the target data table is obtained from the target PDB. During the process of obtaining the meta-information of the target data table in the target PDB, the PDB of the current session is switched at most once, reducing the number of times of switching the PDB, improving the efficiency of obtaining the meta-information of the target data table, and further improving the efficiency of obtaining the incremental data of the target PDB.

[0115] After obtaining the incremental SQL data and the meta-information of the target data table involved, the Parser thread encapsulates the incremental SQL data and the meta-information of the target data table involved according to the preset data structure, and the incremental data of the target PDB can be obtained.

[0116] Among them, the preset data structure is a custom data format, which is a data structure that can be recognized and understood by both the source-side synchronization software and the target-side synchronization software, and can be selected or set according to the actual application needs. This embodiment does not limit this.

[0117] Further, the Parser thread transmits the incremental data of the target PDB to the Extractor thread. Exemplarily, the Parser thread can add the incremental data of the target PDB to the second queue. The Extractor thread can obtain the incremental data of the target PDB from the second queue.

[0118] S405. Return the incremental data of the target PDB through the Extractor thread.

[0119] The Extractor thread takes out the incremental data of the target PDB from the second queue and returns the incremental data of the target PDB to the source-side synchronization software.

[0120] Exemplarily, Figure 5 is an example diagram of the process framework for obtaining incremental data provided by the embodiment of the present application. As Figure 5 shown, in the data loading stage of incremental data synchronization, the source-side synchronization software runs the Extractor thread, starts the Reader thread through the Extractor thread, and the Reader thread starts the Parser thread.

[0121] The Reader thread initializes the PDB parameters for the incremental data to be obtained, that is, obtains the target PDB name. The Reader thread starts a redo log analysis tool (such as Logminer) to create a view, and obtains the incremental SQL data of the target PDB from the view with the target PDB name as the condition. The Reader thread adds the incremental SQL data of the target PDB to the first queue.

[0122] The Parser thread obtains the incremental SQL data of the target PDB from the first queue, parses the incremental SQL data, and determines the name of the target data table involved in the incremental SQL data. The Parser thread determines whether the meta-information of the target data table exists in the cache. If the meta-information of the target data table exists in the cache, the Parser thread reads the meta-information of the target data table from the cache. If the meta-information of the target data table does not exist in the cache, the Parser thread determines whether the PDB of the current session is the target PDB. If the PDB of the current session is the target PDB, the Parser thread obtains the meta-information of the target data table from the target PDB through the currently established JDBC. If the PDB of the current session is not the target PDB, the Parser thread first switches the PDB of the current session to the target PDB, that is, establishes a JDBC with the target PDB, and obtains the meta-information of the target data table from the target PDB through the newly established JDBC.

[0123] The Parser thread encapsulates the incremental SQL data of the target PDB and the meta-information of the target data table involved to obtain the incremental data of the target PDB. The Parser thread returns the incremental data of the target PDB to the Extractor thread. Thus, the source-side synchronization software realizes the acquisition of the incremental data of the target PDB.

[0124] In an optional embodiment, after the source-side synchronization software obtains the incremental data of the target PDB, it sends the incremental data of the target PDB to the filtering module of the data synchronization tool. The filtering module filters and converts the incremental data of the target PDB according to the preset filtering rules to obtain the filtered incremental data of the target PDB.

[0125] For example, the filtering module converts the name of the data table in the source-side database to the table name of the corresponding data table in the target-side database, realizing the mapping of the table name from the source-side database to the target-side database.

[0126] For example, the filtering module filters out the SQL statements corresponding to specified operations (such as query operations), and these operations do not change the data in the data table.

[0127] Among them, the preset filtering rules adopted by the filtering module can be configured according to actual application requirements, and no specific limitation is made here.

[0128] The target - end synchronization software converts the filtered incremental data of the target PDB into executable SQL statements that can be executed in the target - end CDB according to a predefined data structure, and executes these executable SQL statements in the target - end CDB, thereby achieving the incremental data synchronization of the target PDB from the source - end CDB to the target CDB.

[0129] Further, the filtering module sends the filtered incremental data of the target PDB to the target - end synchronization software. The target - end synchronization software converts the filtered incremental data of the target PDB into SQL statements supported by the target - end database according to a predefined data structure, obtains the converted SQL statements, and executes the converted SQL statements in the target - end database, thereby achieving the incremental data synchronization of the target PDB in the source - end database to the target - end database.

[0130] In this embodiment, by the division of labor and cooperation of the Extractor thread, Reader thread, and Parser thread, the incremental data synchronization of the target PDB can be realized, which can further improve the efficiency of incremental data synchronization.

[0131] Figure 6 It is a schematic structural diagram of an incremental data acquisition device provided by an embodiment of the present application. As Figure 6 shown, the incremental data acquisition device 600 includes: a target PDB determination module 601, an incremental SQL data acquisition module 602, and a meta - information acquisition module 603.

[0132] Among them, the target PDB determination module 601 is used to determine the name of the target PDB for which incremental data is to be acquired in response to an instruction to acquire incremental data of any pluggable database PDB.

[0133] The incremental SQL data acquisition module 602 is used to extract the incremental SQL data of the target PDB from the redo log of the source - end CDB corresponding to the target PDB according to the name of the target PDB. The incremental SQL data includes the SQL statements executed in the target PDB.

[0134] The meta - information acquisition module 603 is used to obtain the meta - information of the target data tables involved in the incremental SQL data according to the incremental SQL data of the target PDB. The incremental data of the target PDB includes incremental SQL data and the meta - information of the target data tables.

[0135] In an optional implementation manner, when determining the name of the target PDB for which incremental data is to be acquired, the target PDB determination module 601 is specifically used to:

[0136] Read the name of the specified PDB for which data is to be synchronized from the configuration file as the name of the target PDB.

[0137] In an alternative implementation, when extracting the incremental SQL data of the target PDB from the redo log of the source CDB corresponding to the target PDB according to the name of the target PDB, the incremental SQL data acquisition module 602 is specifically configured to:

[0138] Determine the SCN range according to the data synchronization requirements of the target PDB; create a view through the redo log analysis tool according to the SCN range, where the view includes the SQL statements and related information of the SQL statements in the redo log of the source CDB that fall within the SCN range; query the incremental SQL data of the target PDB from the view according to the name of the target PDB.

[0139] In an alternative implementation, when determining the SCN range according to the data synchronization requirements of the target PDB, the incremental SQL data acquisition module 602 is specifically configured to:

[0140] Determine the SCN range according to the end SCN of the SCN range corresponding to the previous incremental data synchronization and the preset step size.

[0141] In an alternative implementation, after obtaining the meta-information of the target data tables involved in the incremental SQL data according to the incremental SQL data of the target PDB, the meta-information acquisition module 603 is further configured to:

[0142] Package the incremental SQL data and the meta-information of the target data tables according to the preset data structure to obtain the incremental data of the target PDB.

[0143] In an alternative implementation, the target PDB determination module 601, the incremental SQL data acquisition module 602, and the meta-information acquisition module 603 are further configured to:

[0144] Create an Extractor thread, create a Reader thread through the Extractor thread, and create a Parser thread through the Reader thread; receive the incremental data acquisition instruction of the target PDB through the Extractor thread and determine the name of the target PDB for which the incremental data is to be acquired; through the Reader thread, extract the incremental SQL data of the target PDB from the redo log of the source CDB corresponding to the target PDB and transmit the incremental SQL data to the Parser thread; through the Parser thread, obtain the meta-information of the target data tables involved in the incremental SQL data according to the incremental SQL data of the target PDB, package the incremental SQL data and the meta-information of the target data tables into the incremental data of the target PDB, and transmit the incremental data of the target PDB to the Extractor thread; return the incremental data of the target PDB through the Extractor thread.

[0145] In an alternative implementation, when obtaining the meta-information of the target data table involved in the incremental SQL data according to the incremental SQL data of the target PDB, the meta-information obtaining module 603 is specifically configured to:

[0146] Parse the incremental SQL data through a Parser thread to determine the target data table involved in the incremental SQL data; determine whether the PDB of the current session is the target PDB; if the PDB of the current session is not the target PDB, then switch the PDB of the current session to the target PDB, establish a database connection with the target PDB; and obtain the meta-information of the target data table from the target PDB.

[0147] The incremental data acquisition device provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here.

[0148] Figure 7 It is a schematic structural diagram of an electronic device provided in an embodiment of the present application. As Figure 7 shown, the electronic device 700 includes: a memory 701, a processor 702, and a transceiver 703. Among them, the memory 701 is used to store a computer program, and when the processor 702 executes the computer program, the method of any of the above embodiments is implemented. There is a communication link between the memory 701 and the processor 702. Exemplarily, the memory 701, the processor 702, and the transceiver 703 can communicate through a communication bus 704.

[0149] Optionally, the above processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. Combining the steps in the method embodiments disclosed in the present application can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0150] The embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, the method in any of the above method embodiments is implemented.

[0151] The embodiment of the present application further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the method in any of the above method embodiments is implemented.

[0152] It should be noted that in this text, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or apparatus comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or apparatus. Without further limitation, an element defined by the statement "comprising one..." does not exclude the existence of additional identical elements in the process, method, article or apparatus comprising that element.

[0153] The order of the above embodiments of the present application is for description only and does not represent the superiority or inferiority of the embodiments. Additionally, in some of the processes described in the above embodiments and the accompanying drawings, there are multiple operations that occur in a specific order. However, it should be clearly understood that these operations may not be executed in the order in which they appear in this text or may be executed in parallel, and are merely used to distinguish the different operations. The serial numbers themselves do not represent any order of execution. Additionally, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions such as "first" and "second" in this text are used to distinguish different messages, devices, modules, etc., do not represent a sequence, and do not limit that "first" and "second" are of different types. The meaning of "a plurality" is two or more, unless otherwise specifically defined.

[0154] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application that follow the general principles of the present application and include common general knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and embodiments are only to be considered as exemplary, and the true scope and spirit of the present application are pointed out by the following claims. It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for acquiring incremental data, characterized in that: include: In response to an instruction to obtain incremental data of any pluggable database PDB, determine the name of a target PDB for which incremental data is to be obtained; Extracting incremental SQL data of the target PDB from the redo log of the source CDB corresponding to the target PDB according to the name of the target PDB, wherein the incremental SQL data includes SQL statements executed in the target PDB; According to the incremental SQL data of the target PDB, meta information of the target data table involved in the incremental SQL data is obtained, and the incremental data of the target PDB includes the incremental SQL data and meta information of the target data table.

2. The method according to claim 1, characterized in that: Determining the name of the target PDB for the incremental data to be obtained includes: The name of the designated PDB to be synchronized with data is read from the configuration file as the name of the target PDB.

3. The method according to claim 1, characterized in that: The step of extracting incremental SQL data of the target PDB from a redo log of a source CDB corresponding to the target PDB according to the name of the target PDB includes: Determine the SCN interval according to the data synchronization requirements of the target PDB; Using a redo log analysis tool, creating a view according to the SCN interval, wherein the view includes SQL statements and related information of the SQL statements in the redo log of the source CDB that fall within the SCN interval; According to the name of the target PDB, the incremental SQL data of the target PDB is queried from the view.

4. The method according to claim 3, characterized in that Determining the SCN interval according to the data synchronization requirement of the target PDB includes: The SCN interval is determined based on the end SCN of the SCN interval corresponding to the last incremental data synchronization and the preset step size.

5. The method according to claim 1, characterized in that: After obtaining the meta information of the target data table involved in the incremental SQL data according to the incremental SQL data of the target PDB, the method further includes: According to the preset data structure, the incremental SQL data and the meta information of the target data table are encapsulated to obtain the incremental data of the target PDB.

6. The method according to any one of claims 1 to 5, characterized in that Also includes: Create an Extractor thread, create a Reader thread through the Extractor thread, and create a Parser thread through the Reader thread; Receive the incremental data acquisition instruction of the target PDB through the Extractor thread, and determine the name of the target PDB for which the incremental data is to be acquired; Through the Reader thread, according to the name of the target PDB, the incremental SQL data of the target PDB is extracted from the redo log of the source CDB corresponding to the target PDB, and the incremental SQL data is transmitted to the Parser thread; Through the Parser thread, according to the incremental SQL data of the target PDB, the meta information of the target data table involved in the incremental SQL data is obtained, the incremental SQL data and the meta information of the target data table are encapsulated into the incremental data of the target PDB, and the incremental data of the target PDB is transmitted to the Extractor thread; The incremental data of the target PDB is returned through the Extractor thread.

7. The method according to claim 6, characterized in that The acquiring, according to the incremental SQL data of the target PDB, meta information of the target data table involved in the incremental SQL data includes: Parsing the incremental SQL data through the Parser thread to determine the target data table involved in the incremental SQL data; Determine whether the PDB of the current session is the target PDB; If the PDB of the current session is not the target PDB, the PDB of the current session is switched to the target PDB, and a database connection with the target PDB is established; The meta information of the target data table is obtained from the target PDB.

8. An incremental data acquisition device, characterized in that: include: A target PDB determination module, configured to determine the name of a target PDB to be acquired for incremental data in response to an instruction to acquire incremental data of any pluggable database PDB; An incremental SQL data acquisition module is used to extract incremental SQL data of the target PDB from the redo log of the source CDB corresponding to the target PDB according to the name of the target PDB, wherein the incremental SQL data includes SQL statements executed in the target PDB; The meta information acquisition module is used to acquire the meta information of the target data table involved in the incremental SQL data according to the incremental SQL data of the target PDB, wherein the incremental data of the target PDB includes the incremental SQL data and the meta information of the target data table.

9. An electronic device, characterized in that: include: memory, processors, and transceivers; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.

Images