An OceanBase distributed database real-time data synchronization system

Abstract

The present invention discloses a real-time data synchronization system for OceanBase distributed database. The corresponding method includes obtaining table dictionary information of all synchronization objects from the database by using the proxy interface of OceanBase for Oracle database, and obtaining the full amount of data of the synchronization objects based on the table dictionary information; obtaining a first incremental data change amount corresponding to DML operations for adding, deleting, and modifying the full amount of data and DDL operations for modifying the structure of the synchronization objects in all data nodes under the first tenant database instance; and sending at least the table dictionary information, the full amount of data, and the first incremental data change amount to the corresponding target end to achieve data synchronization. The present invention can effectively realize real-time replication and sharing of incremental data.

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and in particular to a real-time data synchronization system for an OceanBase distributed database. Background Art

[0002] Currently, synchronization is captured by creating triggers in the source database. Operations need to be performed on the customer's source database, which is invasive to the source database. For many customers, any operation in the production environment is not allowed. Therefore, it has certain limitations. For the method of transferring the intermediate files analyzed at the source end to the target database, there is a large amount of data to be synchronized, a large amount of resource usage, and it is difficult to obtain the data that users want to obtain in real time from the complex data. Summary of the Invention

[0003] The present invention aims to solve at least one of the technical problems in the related art to some extent.

[0004] To this end, the present invention proposes a real-time data synchronization method for an OceanBase distributed database, which does not require creating triggers in the database, nor additional permissions, and is non-invasive to the database. Therefore, data extraction and data forwarding can be effectively achieved.

[0005] Another object of the present invention is to propose another real-time data synchronization method for an OceanBase distributed database.

[0006] A third object of the present invention is to propose a real-time data synchronization system for an OceanBase distributed database.

[0007] A fourth object of the present invention is to propose another real-time data synchronization system for an OceanBase distributed database.

[0008] To achieve the above object, on the one hand, the present invention proposes a real-time data synchronization method for an OceanBase distributed database, which is applied to the source end. The method includes:

[0009] Using the proxy proxy interface of the OceanBase for Oracle database to obtain the table dictionary information of all synchronization objects from the database, and obtaining the full amount of data of the synchronization objects based on the table dictionary information;

[0010] Obtaining the first incremental data change amount corresponding to the DML operations for adding, deleting, and modifying the full amount of data and the DDL operations for modifying the structure of the synchronization objects included in all data nodes under the first tenant database instance;

[0011] Send at least the table dictionary information, the full amount of data, and the change amount of the first incremental data to the corresponding target end to achieve data synchronization.

[0012] The real-time data synchronization method of the OceanBase distributed database according to the embodiment of the present invention may further have the following additional technical features:

[0013] In an embodiment of the present invention, after obtaining the full amount of data of the synchronization object based on the table dictionary information, the method further includes: converting the full amount of data into full amount binary data through a preset first conversion interface.

[0014] In an embodiment of the present invention, after converting the full amount of data into full amount binary data through a preset first conversion interface, the method further includes:

[0015] Obtain the change amount of the second incremental data corresponding to all data nodes under the second tenant database instance;

[0016] Convert the change amount of the second incremental data into incremental binary data through a preset second conversion interface, and save it in the form of a file on the disk of the source end server.

[0017] To achieve the above object, on the other hand, the present invention proposes a real-time data synchronization method for an OceanBase distributed database, which is applied to the target end. The method includes:

[0018] Receive at least the table dictionary information, the full amount of data, the change amount of the first incremental data, and the change amount of the second incremental data sent by the source end;

[0019] According to the format requirements of the target end database, use the corresponding loading interfaces to sequentially load the received table dictionary information, the full amount of data, the change amount of the first incremental data, and the change amount of the second incremental data to complete real-time data sharing.

[0020] To achieve the above object, on the other hand, the present invention proposes a real-time data synchronization system for an OceanBase distributed database, including:

[0021] A full amount module, configured to obtain the table dictionary information of all synchronization objects from the database by using the proxy proxy interface of the OceanBase for Oracle database, and obtain the full amount of data of the synchronization object based on the table dictionary information;

[0022] An incremental module, configured to obtain the change amount of the first incremental data corresponding to all data nodes under the first tenant database instance, including DML operations for adding, deleting, and modifying the full amount of data and DDL operations for modifying the structure of the synchronization object;

[0023] A sending module, configured to send at least the table dictionary information, the full amount of data, and the change amount of the first incremental data to a corresponding target end to achieve data synchronization.

[0024] To achieve the above object, on the other hand, the present invention provides an OceanBase distributed database real-time data synchronization system, including:

[0025] A receiving module, configured to receive at least the table dictionary information, the full amount of data, the change amount of the first incremental data, and the change amount of the second incremental data sent by a source end;

[0026] A loading module, configured to sequentially load the received table dictionary information, the full amount of data, the change amount of the first incremental data, and the change amount of the second incremental data respectively according to the format requirements of the target end database by using corresponding loading interfaces to complete real-time data sharing.

[0027] In the OceanBase distributed database real-time data synchronization method and system according to the embodiments of the present invention, starting from a certain moment, the full amount of data on the OceanBase for Oracle database and the incremental data starting from this moment are replicated and shared in real time to other databases, which can achieve effective extraction and forwarding of data. By sharing databases, multiple copies of data can be avoided, data redundancy is reduced, and thus storage space is saved.

[0028] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be understood through the practice of the present invention. Description of the Drawings

[0029] The above and / or additional aspects and advantages of the present invention will become apparent and be readily understood from the following description of the embodiments in conjunction with the drawings, where:

[0030] Figure 1 is a flowchart of an OceanBase distributed database real-time data synchronization method according to an embodiment of the present invention;

[0031] Figure 2 is a flowchart of another OceanBase distributed database real-time data synchronization method according to an embodiment of the present invention;

[0032] Figure 3 is a structural diagram of an OceanBase distributed database real-time data synchronization system according to an embodiment of the present invention;

[0033] Figure 4 is a structural diagram of another OceanBase distributed database real-time data synchronization system according to an embodiment of the present invention. Detailed Embodiments

[0034] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0035] In order to enable those skilled in the art of the present technology to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

[0036] A method and system for real-time data synchronization of an OceanBase distributed database according to an embodiment of the present invention will be described below with reference to the accompanying drawings.

[0037] Figure 1 is a flowchart of a method for real-time data synchronization of an OceanBase distributed database according to an embodiment of the present invention. As Figure 1 shown, applied to the source end, the method includes:

[0038] S1, obtaining table dictionary information of all synchronization objects from the database by using the proxy proxy interface of the OceanBase for Oracle database, and obtaining the full amount of data of the synchronization objects based on the table dictionary information.

[0039] It can be understood that although OceanBase is not a direct product or branch of the Oracle database, as a highly scalable distributed database system, OceanBase takes into account the compatibility with traditional database systems (such as Oracle) in its design to simplify the migration and docking process for users. This means that for developers and administrators who are accustomed to using the Oracle database, OceanBase strives to provide a certain degree of syntactic and functional similarity, enabling them to transition relatively smoothly to the distributed database environment. By providing compatibility with Oracle and migration tools, OceanBase enables users from an Oracle background to make a smooth transition while enjoying the many advantages of a distributed database. This allows enterprises to gradually migrate to a more modern and flexible database architecture without sacrificing their original investments.

[0040] It can be understood that the Proxy API generally refers to a service or middleware that provides a unified access point to manage access to backend services or resources, especially network requests such as HTTP requests. The proxy interface can encapsulate complex network communication logics, such as functions like authentication, protocol conversion, load balancing, cache fetching, traffic control, monitoring, security filtering, regional redirection (such as IP proxy), etc.

[0041] Specifically, in the embodiment of the present invention, at the source end, the table dictionary information of all synchronization objects is obtained from the database through the proxy interface of OceanBase for Oracle, and the full amount of data of the synchronization objects is obtained. Through a specific conversion interface, which is a preset first conversion interface in the embodiment of the present invention, it is converted into binary data in a unique format, that is, the full amount binary data of the present invention, and cached in the memory.

[0042] S2. Obtain the first incremental data change amount corresponding to the DML operations that perform addition, deletion, and modification on the full amount of data and the DDL operations that modify the synchronization object structure for all data nodes under the first tenant database instance.

[0043] It can be known that in database operations, DML (Data Manipulation Language) and DDL (Data Definition Language) respectively correspond to the management of data content and the definition and modification of the database structure.

[0044] Specifically, during the source - end incremental analysis, the first incremental data change amount of all data nodes under the first tenant, that is, the OceanBase for Oracle tenant one database instance in the embodiment of the present invention, which includes DML operations that perform addition, deletion, and modification on the full amount of data of the synchronization object and DDL operations that modify the synchronization object structure, can be obtained.

[0045] Furthermore, during the source - end incremental analysis, the second incremental data change amount of all data nodes under the second tenant, that is, the OceanBase for Oracle tenant two database instance in the embodiment of the present invention, can also be obtained. Through a specific conversion interface, which is a preset second conversion interface in the embodiment of the present invention, it is converted into incremental binary data in a unique format.

[0046] Furthermore, after being converted into incremental binary data in a unique format through the specific conversion interface, it is saved in the form of a file on the disk of the source - end server.

[0047] S3. Send at least the table dictionary information, the full amount of data, and the first incremental data change amount to the corresponding target end to achieve data synchronization.

[0048] It is understandable that during the source - end data transmission, the sender transmits all the data and each incremental data to the corresponding target - end.

[0049] Specifically, the source - end server can send table dictionary information, all - volume data, the change amount of the first incremental data, the change amount of the second incremental data, all - volume binary data, and incremental binary data, etc. to the target - end to complete real - time data sharing at the target - end.

[0050] According to the real - time data synchronization method of the OceanBase distributed database in the embodiments of the present invention, through the shared database, multiple copies of data can be avoided, data redundancy is reduced, thus saving storage space. In a shared environment, all users or applications see the same data source, which helps to maintain data consistency, reduce data conflicts and inconsistencies. Multiple users can read or update data simultaneously, promoting collaboration and information flow among teams. Compared with configuring a separate database for each user or application, the shared database can reduce hardware, software, and maintenance costs. Different departments within an enterprise can easily achieve data integration through the shared database, promoting data - driven decision - making. Although shared, the database system usually has a fine - grained permission management mechanism to ensure that each user or application can only access the data within its authorized scope, protecting data security.

[0051] To implement the above - mentioned embodiments, as Figure 2 shown, another real - time data synchronization method of the OceanBase distributed database is also provided in this embodiment, which is applied to the target - end. The method includes:

[0052] S101, receiving at least the table dictionary information, all - volume data, the change amount of the first incremental data, and the change amount of the second incremental data sent by the source - end;

[0053] S102, according to the format requirements of the target - end database, using the corresponding loading interfaces to sequentially load the received table dictionary information, all - volume data, the change amount of the first incremental data, and the change amount of the second incremental data respectively to complete real - time data sharing.

[0054] Specifically, during the target - end data reception, the received data is saved on the disk of the target - end server.

[0055] Furthermore, during the target - end data loading, according to the format requirements of the target database, use the corresponding loading interfaces to sequentially load the received data files, such as table dictionary information, all - volume data, the change amount of the first incremental data, and the change amount of the second incremental data, to complete the real - time data sharing of the OceanBase for Oracle database as the source - end.

[0056] Further, store the table dictionary information, full data, the change amount of the first incremental data, and the change amount of the second incremental data on the disk of the target server.

[0057] According to the real-time data synchronization method of the OceanBase distributed database according to the embodiments of the present invention, the distributed relational database OceanBase for Oracle highly compatible with Oracle features is used as the data source, and the full data on the OceanBase for Oracle database and the incremental data starting from a certain moment are replicated and shared in real time to other databases, improving the efficiency of data sharing.

[0058] To implement the above embodiments, as Figure 3 shown, another OceanBase distributed database real-time data synchronization system 10 is further provided in this embodiment, including:

[0059] A full data module 100, configured to obtain the table dictionary information of all synchronization objects from the database by using the proxy interface of the OceanBase for Oracle database, and obtain the full data of the synchronization objects based on the table dictionary information;

[0060] An incremental data module 200, configured to obtain the change amount of the first incremental data corresponding to the DML operations for adding, deleting, and modifying the full data and the DDL operations for modifying the structure of the synchronization objects included in all data nodes under the first tenant database instance;

[0061] A sending module 300, configured to send at least the table dictionary information, the full data, and the change amount of the first incremental data to the corresponding target end to implement data synchronization.

[0062] Further, the full data module 100 is further configured to:

[0063] Convert the full data into full binary data through a preset first conversion interface.

[0064] Further, the incremental data module 200 is further configured to:

[0065] Obtain the change amount of the second incremental data corresponding to all data nodes under the second tenant database instance;

[0066] Convert the change amount of the second incremental data into incremental binary data through a preset second conversion interface, and save it in the form of a file on the disk of the source server.

[0067] The OceanBase distributed database real-time data synchronization system according to the embodiments of the present invention can avoid multiple copies of data through a shared database, reduce data redundancy, and thus save storage space. In a shared environment, all users or applications see the same data source, which helps to maintain data consistency and reduce data conflicts and inconsistencies. Multiple users can read or update data simultaneously, promoting collaboration and information flow among teams. Compared with configuring a separate database for each user or application, a shared database can reduce hardware, software, and maintenance costs. Different departments within an enterprise can easily achieve data integration through a shared database, promoting data-driven decision-making. Although shared, the database system usually has a fine-grained permission management mechanism to ensure that each user or application can only access the data within its authorized scope, protecting data security.

[0068] To implement the above embodiments, as Figure 4 shown, this embodiment also provides another OceanBase distributed database real-time data synchronization system 20, including:

[0069] A receiving module 30, configured to receive at least the table dictionary information, full amount of data, first incremental data change amount, and second incremental data change amount sent by the source end;

[0070] A loading module 40, configured to sequentially load the received table dictionary information, full amount of data, first incremental data change amount, and second incremental data change amount using corresponding loading interfaces according to the format requirements of the target-end database to complete real-time data sharing.

[0071] Furthermore, the loading module 40 is further configured to:

[0072] Store the table dictionary information, full amount of data, first incremental data change amount, and second incremental data change amount on the disk of the target-end server.

[0073] The OceanBase distributed database real-time data synchronization system according to the embodiments of the present invention uses the distributed relational database OceanBase for Oracle, which is highly compatible with Oracle features, as the data source, and starts real-time replication and sharing of the full amount of data on the OceanBase for Oracle database and the incremental data from a certain moment to other databases, improving the efficiency of data sharing.

[0074] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0075] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of the present invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically and clearly defined.

Claims

1. A real-time data synchronization method for OceanBase distributed database, characterized in that: Applied to the source end, the method comprises: The proxy interface of the OceanBase for Oracle database is used to obtain the table dictionary information of all synchronization objects from the database, and the full data of the synchronization objects is obtained based on the table dictionary information. The full data is converted into full binary data through a preset first conversion interface, and cached in the memory; Obtaining a first incremental data change corresponding to a DML operation that adds, deletes, or modifies the full data and a DDL operation that modifies the synchronization object structure for all data nodes under the first tenant database instance; after converting the full data into full binary data through a preset first conversion interface, obtaining a second incremental data change corresponding to all data nodes under the second tenant database instance, and converting the second incremental data change into incremental binary data through a preset second conversion interface, and saving the data in the form of a file on the disk of the source server; At least the table dictionary information, the full data, the full binary data, the first incremental data change, the second incremental data change and the incremental binary data are sent to the corresponding target end to complete real-time data sharing at the target end; wherein, in a shared environment, all users or applications see the same data source.

2. A real-time data synchronization method for OceanBase distributed database, characterized in that: Applied to the target end, the method includes: At least receiving the table dictionary information, full data, full binary data, first incremental data change, second incremental data change and incremental binary data sent by the source end using the method as claimed in claim 1; According to the format requirements of the target database, the received table dictionary information, the full data, the full binary data, the first incremental data change, the second incremental data change and the incremental binary data are sequentially loaded using the corresponding loading interface to complete real-time data sharing.

3. The method according to claim 2, characterized in that After receiving at least the table dictionary information, the full amount of data, the first incremental data change amount and the second incremental data change amount sent by the source end, the method further includes: The table dictionary information, the full data, the first incremental data change amount, and the second incremental data change amount are stored on the disk of the target server.

4. An OceanBase distributed database real-time data synchronization system, characterized in that: Applied to the source end, the system includes: The full module is used to obtain the table dictionary information of all synchronization objects from the database by using the proxy interface of the OceanBase for Oracle database, obtain the full data of the synchronization objects based on the table dictionary information, convert the full data into full binary data through a preset first conversion interface, and cache it in the memory; An incremental module is used to obtain a first incremental data change corresponding to a DML operation that adds, deletes, or modifies the full data and a DDL operation that modifies the synchronization object structure for all data nodes under the first tenant database instance; after converting the full data into full binary data through a preset first conversion interface, obtain a second incremental data change corresponding to all data nodes under the second tenant database instance, and convert the second incremental data change into incremental binary data through a preset second conversion interface, and save the second incremental data change in the form of a file on the disk of the source server; A sending module is used to send at least the table dictionary information, the full data, the full binary data, the first incremental data change, the second incremental data change and the incremental binary data to the corresponding target end to complete real-time data sharing at the target end; wherein, in a shared environment, all users or applications see the same data source.

5. An OceanBase distributed database real-time data synchronization system, characterized in that: Applied to the target end, the system includes: A receiving module, used to receive at least the table dictionary information, full data, full binary data, first incremental data change, second incremental data change and incremental binary data sent by the method as claimed in claim 1; The loading module is used to sequentially load the received table dictionary information, the full data, the full binary data, the first incremental data change, the second incremental data change and the incremental binary data in accordance with the format requirements of the target database using the corresponding loading interface to complete real-time data sharing.

6. The system according to claim 5, characterized in that Loading modules, also used for: After receiving at least the table dictionary information, the full data, the first incremental data change amount and the second incremental data change amount sent by the source end, the table dictionary information, the full data, the first incremental data change amount and the second incremental data change amount are stored on the disk of the target end server.

Images