Data synchronization method and system, computing device, storage medium and program product

Abstract

The embodiment of the invention provides a data synchronization method and system, computing equipment, a storage medium and a program product, and the data synchronization method comprises the steps that a data synchronization task is received, and the data synchronization task is used for indicating to synchronize data of a source database to a target database; according to the data synchronization task, at least one record identifier is created in the target database, the record identifiers are used for identifying records in the table, and the record identifiers are different from one another; establishing a mapping relationship between the storage address of the data in the source database and each record identifier; and based on the mapping relationship, synchronizing the data of the storage address in the source database to the record position of the identifier corresponding to each record identifier in the target database. The mapping relationship between the storage address of the data in the source database and each record identifier is established, so that synchronization can be carried out based on the mapping relationship when data synchronization is carried out on the source database and the target database, and the accuracy of data synchronization between the source database and the target database is ensured.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a data synchronization method, system, computing device, storage medium, and program product. Background Art

[0002] Data migration between heterogeneous databases is usually divided into two parts: incremental data migration and full (existing) data migration. Incremental data migration obtains transactions (customer projects) submitted by the source database and replays the transactions in the target database. Full data migration migrates the data in the source database before the execution of the corresponding incremental data migration transaction to the target database.

[0003] When migrating data between heterogeneous databases, whether it's a full or incremental migration, network outages, disaster recovery, or other scenarios can cause transactions and full data from the source database to be repeatedly executed and migrated in the target database, leading to duplicate data in the target database. Therefore, a method to improve the accuracy of data synchronization and migration between databases is urgently needed. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a data synchronization method. One or more embodiments of this specification also relate to a data synchronization apparatus, a computing device, a computer-readable storage medium, and a computer program product to address technical deficiencies in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a data synchronization method is provided, including: receiving a data synchronization task, wherein the data synchronization task is used to indicate that data from a source database is synchronized to a target database; creating at least one record identifier in the target database according to the data synchronization task, wherein the record identifier is used to identify records of a table, and each record identifier is different from each other; establishing a mapping relationship between a storage address of data in the source database and each record identifier; and based on the mapping relationship, synchronizing the data at the storage address in the source database to the record position corresponding to each record identifier in the target database.

[0006] According to a second aspect of an embodiment of this specification, a data synchronization device is provided, comprising: a receiving module configured to receive a data synchronization task, wherein the data synchronization task is used to indicate synchronization of data from a source database to a target database; a creating module configured to create at least one record identifier in the target database according to the data synchronization task, wherein the record identifier is used to identify records of a table, and each record identifier is different from each other; an establishing module configured to establish a mapping relationship between a storage address of data in the source database and each record identifier; and a synchronization module configured to synchronize the data at the storage address in the source database to the record position corresponding to each record identifier in the target database based on the mapping relationship.

[0007] According to a third aspect of an embodiment of this specification, a data synchronization system is provided, comprising a client and a server; the client is used to send a data synchronization request; the server is used to receive the data synchronization request, and determine a data synchronization task based on the data synchronization request, wherein the data synchronization task is used to indicate that data from a source database is synchronized to a target database; according to the data synchronization task, at least one record identifier is created in the target database, wherein the record identifier is used to identify records of a table, and each record identifier is different from each other; a mapping relationship between a storage address of data in the source database and each record identifier is established; based on the mapping relationship, the data at the storage address in the source database is synchronized to the record position corresponding to each record identifier in the target database.

[0008] According to the fourth aspect of the embodiments of this specification, a computing device is provided, including: a memory and a processor; the memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which implement the steps of the above-mentioned data synchronization method when executed by the processor.

[0009] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, which implements the steps of the above-mentioned data synchronization method when executed by a processor.

[0010] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the steps of executing the above-mentioned data synchronization method.

[0011] One embodiment of the present specification receives a data synchronization task, wherein the data synchronization task is used to instruct the synchronization of data from a source database to a target database; according to the data synchronization task, at least one record identifier is created in the target database, wherein a record identifier is used to identify a record in a table, and each record identifier is different from each other; a mapping relationship between the storage address of the data in the source database and each record identifier is established; based on the mapping relationship, the data at the storage address in the source database is synchronized to the record position corresponding to each of the record identifiers in the target database. In the case of receiving the data synchronization task, according to the data synchronization task, at least one record identifier is created in the target database, and a mapping relationship between the storage address of the data in the source database and each record identifier is established, so that when synchronizing data between the source database and the target database, data synchronization can be performed based on the mapping relationship, so that even in the event of network interruption, disaster recovery, etc., the data of the source database and the target database can continue to be accurately synchronized based on the mapping relationship after the network is restored and the disaster recovery is completed, thereby avoiding duplicate synchronization of data and ensuring the accuracy of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a schematic diagram of an interaction process under a data synchronization system architecture provided by an embodiment of this specification;

[0013] Figure 2 This is a framework diagram of a data synchronization system provided by an embodiment of this specification;

[0014] Figure 3 This is a flow chart of a data synchronization method provided by one embodiment of this specification;

[0015] Figure 4a This is a flowchart of a data synchronization method according to an embodiment of the present disclosure;

[0016] Figure 4b This is a schematic diagram of asynchronous deletion of hidden columns in a data synchronization method provided by an embodiment of this specification;

[0017] Figure 5 This is a structural diagram of a data synchronization device provided by an embodiment of this specification;

[0018] Figure 6 This is a structural block diagram of a computing device provided by one embodiment of this specification. DETAILED DESCRIPTION

[0019] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0020] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0021] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0022] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0023] First, the terms involved in one or more embodiments of this specification are explained.

[0024] Database synchronization: refers to synchronizing data, table structures, and other information from one database instance to another. This process is generally divided into three parts: structure migration, full (existing) data synchronization, and incremental data synchronization.

[0025] Heterogeneous databases: refers to two databases that are not of the same origin and have different underlying structures. It is impossible to directly apply data from one database to another heterogeneous database.

[0026] Hidden columns: These columns are hidden in a table and can only be viewed under certain circumstances. These columns contain information about the database's attributes (attribute) and structure (schema_infomat ion), but are not displayed when selecting data from the table.

[0027] For data replication within the same database (common in master-slave synchronization), the internal unique number of the log can be used to ensure that table transactions are not redone and cause data inconsistency.

[0028] However, the physical storage methods of data between heterogeneous databases are different, and data cannot be synchronized through physical replication. Instead, data synchronization requires the use of a network. Data migration for heterogeneous databases is usually divided into two parts: incremental data migration and full (existing) data migration. Incremental data migration is to obtain transactions submitted by the source database (customer projects) and replay the transactions in the target database. Full data migration is to migrate the data before the execution of the corresponding transaction of the incremental data migration in the source database to the target database. For data migration between heterogeneous databases, whether it is full data migration or incremental data migration, if there are scenarios such as network interruptions and disaster recovery, it is possible that the transactions and full data of the source database will be repeatedly executed and migrated in the target database, resulting in duplicate data in the target database. Therefore, there is an urgent need for a method to improve the accuracy of data synchronization and migration between databases.

[0029] To solve the above technical problems, this specification provides a data synchronization method. This specification also involves a data synchronization device, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0030] See also Figure 1 , Figure 1 FIG. 1 shows a schematic diagram of an interaction process under a data synchronization system architecture provided by an embodiment of this specification. Figure 1 As shown, the system includes a server 100 and a client 200.

[0031] Client 200: used to send data synchronization requests to the server;

[0032] Server 100: used to receive a data synchronization request, determine a data synchronization task based on the data synchronization request, wherein the data synchronization task is used to indicate that the data of the source database is synchronized to the target database; create at least one record identifier in the target database based on the data synchronization task, wherein the record identifier is used to identify the record in the table, and each record identifier is different from each other; establish a mapping relationship between the storage address of the data in the source database and each record identifier; based on the mapping relationship, synchronize the data at the storage address in the source database to the record position corresponding to each record identifier in the target database.

[0033] Optionally, there is a data synchronization component in the server, which is used to synchronize the data of the source database to the target database based on the mapping relationship.

[0034] By applying the solution implemented in this specification, when a data synchronization task is received, at least one record identifier is created in the target database according to the data synchronization task, and a mapping relationship between the storage address of the data in the source database and each record identifier is established, so that when synchronizing the data between the source database and the target database, data synchronization can be performed based on the mapping relationship. Even in the event of network interruption, disaster recovery, etc., the data between the source database and the target database can continue to be accurately synchronized based on the mapping relationship after the network is restored and the disaster recovery is completed, thereby avoiding duplicate synchronization of data and ensuring the accuracy of data synchronization.

[0035] See also Figure 2 , Figure 2 The following figure shows a framework diagram of a data synchronization system provided by one embodiment of this specification. The system may include a server 100 and multiple clients 200. Multiple clients 200 can establish communication connections through the server 100. In the data synchronization scenario, the server 100 is used to provide data synchronization services between multiple clients 200. Multiple clients 200 can act as senders or receivers and communicate through the server 100.

[0036] Users can interact with the server 100 through the client 200 to receive data sent by other clients 200, or send data to other clients 200. In the data synchronization scenario, the user can issue a data synchronization request to the server 100 through the client 200. The server 100 generates a data synchronization result based on the data synchronization request and pushes the data synchronization result to other clients 200 with which communication has been established.

[0037] The client 200 and the server 100 are connected via a network. The network provides a medium for the communication link between the client 200 and the server 100. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. The data transmitted by the client 200 may need to be encoded, transcoded, compressed, and other processing before being released to the server 100.

[0038] The client 200 can be a browser, an application (APP), or a web application such as Hypertext Markup Language 5 (H5) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application. The client 200 can be based on the software development kit (SDK) of the corresponding service provided by the server, such as developed based on the real-time communication (RTC) SDK. The client 200 can be deployed in an electronic device and needs to rely on the device to run or certain APPs in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0039] The server 100 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers that provide background training to support models used on clients, and servers that process data sent by clients. It should be noted that the server 100 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server for a distributed system, or a server that is combined with a blockchain. The server can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0040] It is worth noting that the data synchronization method provided in the embodiments of this specification is generally executed by the server 100.

[0041] See also Figure 3 , Figure 3 A flow chart of a data synchronization method provided by an embodiment of this specification is shown, which specifically includes the following steps.

[0042] Step 302: Receive a data synchronization task, where the data synchronization task is used to instruct synchronization of data from a source database to a target database.

[0043] Specifically, a data synchronization task is a task that synchronizes data between two databases, instructing the synchronization of data from a source database to a target database. A data synchronization task may also include information such as the target data index and location in the source database. The data in the source database to be synchronized can include transactions, structures, or substantive data within the source database.

[0044] The implementation method of receiving data synchronization tasks can be that the front-end user inputs a data synchronization request at the front-end, the client forwards the data synchronization request to the server, and the server generates a data synchronization task based on the data synchronization request; or the front-end user inputs a data synchronization request at the front-end, the client generates a data synchronization task based on the data synchronization request, and sends the generated data synchronization task to the server, so that the server receives the data synchronization task.

[0045] Optionally, the source database and the target database may be homogeneous databases or heterogeneous databases.

[0046] Step 304: according to the data synchronization task, at least one record identifier is created in the target database, wherein the record identifier is used to identify the record in the table, and each record identifier is different from another.

[0047] Specifically, a record identifier is used to identify a record in a table. A record can be a row or a column. A database table includes at least one record identifier. Each record identifier can constitute a row or a column of data in the table. If each record identifier constitutes a row of data in the table, then one record identifier is used to identify a column in the table; if each record identifier constitutes a column of data in the table, then one record identifier is used to identify a row in the table. For example, each record identifier can constitute a hidden column in the table. The form of a record identifier can be explicit, implicit, etc. The explicit form of the record identifier can be perceived by the front end, while the implicit form of the record identifier cannot be perceived by the front end.

[0048] According to the data synchronization task, there are many ways to create at least one record identifier in the target database. The specific method is selected according to the actual situation and is not limited in this specification.

[0049] A possible implementation of the present specification is that, according to the data synchronization task, creating at least one record identifier in the target database may be triggering an operation of creating at least one record identifier in the target database according to the received data synchronization task, wherein the triggered operation of creating a record identifier in the target database may be creating a hidden record identifier or creating an explicit record identifier, which is pre-set on the server side according to actual conditions and is not limited in this specification.

[0050] Another possible implementation of the present specification is to create at least one record identifier in the target database according to the data synchronization task, which may be to parse the data synchronization task, obtain the target form of the record identifier, and create at least one record identifier of the target form in the target database, wherein the target form refers to the form in which the record identifier is displayed, such as a hidden form, an explicit form, etc.

[0051] Optionally, embodiments of this specification can be applied to a data synchronization solution for a database table in a source database that has neither a primary key nor a unique key. When it is determined based on a data synchronization task that the data table to be synchronized in the source database has neither a primary key nor a unique key, at least one record identifier is created in the target database. The format of the record identifier is as described above. That is, the above steps, based on the data synchronization task, further include the following steps:

[0052] Identify the table structure in the source database based on the data synchronization task;

[0053] In a case where the structure of the table has no primary key and no unique key, at least one record identifier is created in the target database.

[0054] Specifically, a primary key is a candidate key that is selected to uniquely identify a row or column. A table has only one primary key. Unique keys are used to avoid adding duplicate data to a database table.

[0055] Identifying whether the structure of a table in the source database has neither a primary key nor a unique key according to a data synchronization task may be triggering an operation of identifying whether the structure of a table in the source database has neither a primary key nor a unique key according to the data synchronization task. Identifying whether the structure of a table in the source database has neither a primary key nor a unique key may include first identifying whether the table in the source database has neither a unique key nor a primary key. If so, then identifying whether the table in the source database has neither a primary key nor a unique key. If so, confirming that the structure of the table has neither a primary key nor a unique key.

[0056] If the source database table is identified as having neither a primary key nor a unique key, at least one record identifier is created in the target database. If the source database table is not identified as having neither a primary key nor a unique key, at least one record identifier is created in the target database, and data from the source database can be synchronized directly to the target database based on the primary key or unique key in the source database. That is, if the source database table is identified as having neither a primary key nor a unique key, only at least one record identifier can be created in the target database. However, if the source database table has a primary key and / or a unique key, multiple implementation methods are available for synchronizing data from the source database to the target database.

[0057] For example, in the link of migrating from Oracle to PolarDB for Oracle, a large number of customers who use Oracle for data storage often use tables without primary keys and unique keys to store data. The ability to complete consistent synchronization of tables without primary keys and unique keys is the key capability of the link from Oracle to PolarDB for Oracle. During the data synchronization process, if the source database is Oracle, the structure of the table in the source database is determined to be without primary keys and unique keys. Oracle is a database management system, a relational database management system, and PolarDB is a distributed cloud-native database.

[0058] According to the data synchronization task, identify whether the structure of the table in the source database has no primary key and no unique key. If so, create at least one record identifier in the target database. When the structure of the table in the source database has no primary key and no unique key, the processing basis of creating at least one record identifier in the target database can be used to synchronize the data in the source database to the target data, thereby ensuring the smooth execution and accuracy of the synchronization.

[0059] In an optional embodiment of the present specification, the above steps of creating at least one record identifier in the target database according to the data synchronization task include the following steps:

[0060] Parsing data synchronization tasks, the record identifier is obtained in a hidden form. The front-end is unaware of the hidden record identifier.

[0061] At least one first record identifier is created in a hidden form in the target database.

[0062] The implementation method for parsing the data synchronization task and obtaining the form of the record identification as a hidden form can be to extract the form keyword from the information carried by the data synchronization task, and based on the form keyword, obtain the form of the record identification as a hidden form. For example, if the form keyword is hidden, then the form of the record identification is determined to be a hidden form. For another example, if the form keyword is invisible, then the form of the record identification is determined to be a hidden form. It can also be to input the data synchronization task into a form recognition model to obtain the form of the record identification as a hidden form, wherein the form recognition model is trained based on multiple sample data synchronization tasks and corresponding label forms.

[0063] The implementation method of creating at least one first record identifier in a hidden form in the target database can be to define a hidden prefix in the target database to obtain at least one first record identifier in a hidden form in the target database; or it can be to define a hidden attribute in the target database to obtain at least one first record identifier in a hidden form in the target database.

[0064] By parsing the data synchronization task, it is determined that the first record identifier created in the target database is in hidden form, and a hidden first record identifier is created in the target database. This allows the hidden first record identifier to be used to establish a mapping relationship with the source database during the data synchronization phase, thereby achieving accurate data synchronization between databases. The front-end is unaware of the hidden first record identifier, thereby improving the user experience.

[0065] In an optional embodiment of the present specification, after the above steps of synchronizing the data of the storage address in the source database to the record position of each record identifier corresponding to the identifier in the target database based on the mapping relationship, the following steps are further included:

[0066] In response to the received stream switching instruction, the stream switching instruction is executed, and the meta information of the first record identifier is deleted, where the meta information is used to define the structure of the first record identifier in the target database.

[0067] Specifically, a traffic switching instruction is an instruction that switches access traffic from a source database to a target database, which then provides database services. Meta-information is information that defines the structure of the first record identifier in the target database. For example, the meta-information can be a header for the first record identifier. The first record identifier includes both the meta-information and the actual data.

[0068] The implementation method of executing the stream switching instruction and deleting the meta-information of the first record identifier in response to the received stream switching instruction can be to simultaneously execute the stream switching instruction and delete the meta-information of the first record identifier based on the received stream switching instruction; or it can be to execute the stream switching instruction based on the received stream switching instruction and delete the meta-information of the first record identifier without affecting the execution of the stream switching instruction.

[0069] Executing the flow switching instruction switches the access traffic to the source database to the access traffic to the target database. Deleting the metadata of the first record identifier may be accomplished by deleting the structural information corresponding to the first record identifier so that the first record identifier no longer has any effect in the target database.

[0070] Optionally, in response to the received stream switching instruction, the metadata of the first record identifier is deleted, while the actual data of the first record identifier is retained and deleted again when the space is sorted, thereby implementing a step-by-step deletion of the first record identifier by first deleting the metadata and then deleting the actual data, thereby avoiding scenarios such as table locking that affect customer use.

[0071] In response to the received switching instruction, the switching instruction is executed to enable the user to access the target database and delete the metadata of the first record identifier, so that after the switching instruction is executed, the front end can smoothly perform corresponding processing on the target database, ensuring the user's usage experience.

[0072] Optionally, after the meta-information of the first record identifier is deleted, the first record identifier no longer has any effect in the target database. To ensure normal use of the database, the actual data in the first record identifier may be deleted when the table space of the target database is organized. That is, after the above steps of responding to the received stream switching instruction, executing the stream switching instruction, and deleting the meta-information of the first record identifier, the following steps may be further included:

[0073] When the table space of the target database is organized, the first record identifier is deleted.

[0074] When the table space of the target database is sorted out during idle time, the first record identifier is deleted based on the deleted metadata. For example, when the table space of the target database is sorted out during idle time, metadata carrying a deletion mark is identified, and it is determined that the metadata corresponds to the first record identifier, then the first record identifier is deleted.

[0075] Optionally, deleting the first record identifier refers to deleting the first record identifier itself and data other than the meta information.

[0076] When the table space of the target database is organized, the actual data identified by the first record is deleted, thereby avoiding scenarios such as table locking, avoiding blocking the customer's use of the database, and improving the user's experience of using the database.

[0077] The solution of the embodiment of this specification is designed based on hidden columns to ensure that the consistency of tables without primary keys and unique keys can be synchronized without the customer's awareness during the synchronization process. This meets the customer's requirements for data synchronization consistency and guarantees the customer experience. No manual intervention is required, and the migration to O links can be completed directly on the public cloud.

[0078] In an optional embodiment of the present specification, the above steps of creating at least one record identifier in the target database according to the data synchronization task include the following steps:

[0079] Parse the data synchronization task and obtain the record identifier in explicit form. The front-end is aware of the explicit record identifier.

[0080] At least one second record identifier is created in the target database in an explicit form.

[0081] The implementation method for parsing the data synchronization task and obtaining the record identification form in explicit form may be to extract the form keyword from the information carried by the data synchronization task, and based on the form keyword, obtain the record identification form in explicit form. For example, if the form keyword is explicit, then the record identification form is determined to be explicit. For another example, if the form keyword is visible, then the record identification form is determined to be explicit. Alternatively, the data synchronization task may be input into a form recognition model to obtain the record identification form in explicit form.

[0082] The implementation method of creating at least one second record identifier in an explicit form in the target database may be to define an explicit prefix in the target database to obtain at least one second record identifier in an explicit form in the target database; or it may be to define an explicit attribute in the target database to obtain at least one second record identifier in an explicit form in the target database.

[0083] By parsing the data synchronization task, it is determined that at least one second record identifier created in the target database is in explicit form, and at least one second record identifier in explicit form is created in the target database, so that in the data synchronization stage, the at least one second record identifier in explicit form can be used to establish a mapping relationship with the source database to achieve accurate data synchronization between databases.

[0084] In an optional embodiment of the present specification, after synchronizing the data of the source database to the target database based on the mapping relationship in the above steps, the following steps are further included:

[0085] In response to the received stream switching instruction from the front end, executing a database shutdown operation and deleting the second record identifier;

[0086] Start the database and execute the switch command.

[0087] Specifically, the second record identifier corresponds to the meta information and the identifier itself.

[0088] The implementation of executing the database shutdown operation and deleting the second record identifier in response to the received front-end switching instruction may be to trigger the database shutdown operation and delete the second record identifier based on the received switching instruction.

[0089] After deleting the second record identifier, a database startup operation is performed, and an operation of executing a stream switching instruction is performed.

[0090] After receiving the stream switching instruction, the server stores a predetermined procedure, and according to the predetermined procedure, shuts down the database, deletes the second record identifier, restarts the database, and executes the stream switching instruction. Deleting the second record identifier deletes both the metadata corresponding to the second record identifier and the second record identifier itself.

[0091] Since the second record identifier is in an explicit form, in order to prevent the explicit second record identifier from interfering with the user's use of the target database, the database shutdown operation is first performed, and then the database startup operation is performed after deleting the second record identifier, and the stream switching instruction is executed, thereby ensuring that after the stream switching, the target database used by the user is a clean version that does not contain redundant data.

[0092] Step 306: Establish a mapping relationship between the storage address of the data in the source database and each record identifier.

[0093] Specifically, the mapping relationship refers to the mapping relationship between the storage address of the data in the source database and each record identifier in the source database. The storage address of the data and each record identifier are mapped to each other. Through the mapping relationship, the corresponding relationship between the source database and the target database is established.

[0094] The implementation method for establishing the mapping relationship between the storage address of the data in the source database and each record identifier can be to fill the storage address of the data in the source database into each record identifier to obtain the mapping relationship between the storage address of the data in the source database and each record identifier; it can also be to establish a transit mapping unit, in which the corresponding relationship between the storage address of the data in the source database and each record identifier is constructed, and based on the corresponding relationship, the mapping relationship between the storage address of the data in the source database and each record identifier is obtained.

[0095] Among them, the implementation method of filling the storage address of the data in the source database into each record identifier can be to fill the storage address of the row data in the source database into each row identifier; or to fill the storage address of the column data in the source database into each column identifier.

[0096] Before the mapping relationship between the storage address of the data in the source database and each record identifier is established, an operation can be defined based on the target data in the source database to update the storage address of the data in the source database and establish a mapping relationship between the updated storage address and each record identifier.

[0097] Optionally, if the data synchronization task is a full data synchronization task, then when establishing the mapping relationship between the storage address of the data in the source database and each record identifier, the storage address of the data in the source database is sent to each record identifier and filled in to determine the mapping relationship between the storage address of the data in the source database and each record identifier. If the data synchronization task is an incremental data synchronization task, then the storage address of the data in the source database is stored in a document of the desktop storage service, and the storage address in the document of the desktop storage service is sent to each record identifier of the target database for filling in to determine the mapping relationship between the storage address of the data in the source database and each record identifier. If the data synchronization task is a task for synchronizing both full data and incremental data, then full data synchronization can be performed first, and then incremental data synchronization can be performed based on the full data synchronization.

[0098] In the process of storing the storage address of the data in the source database in the document of the desktop storage service, it is necessary to enable the logical replication transparent transmission switch of the hidden column in the source database in advance.

[0099] Optionally, when executing some data definition operations in the source database, the storage address of the data in the source database will be changed, but the target database is unaware of the change in the storage address. Therefore, by obtaining the target data definition operation, the storage address of the data in the source database is updated based on the target data definition operation in the source database, so that the updated storage address is mapped to each record identifier to ensure consistency of data synchronization. That is, before establishing the mapping relationship between the storage address of the data in the source database and each record identifier in the above step, the following steps are also included:

[0100] In response to a target data definition operation, obtaining an updated storage address corresponding to the target data definition operation, wherein the target data definition operation is an operation that affects an address change of a storage address of data in a source database;

[0101] The above steps establish a mapping relationship between the storage address of the data in the source database and each record identifier, including the following steps:

[0102] Based on the updated storage address, update the storage address of the data in the source database;

[0103] Establish a mapping relationship between the storage address of the data in the updated source database and each record identifier.

[0104] Specifically, target data definition operations refer to data definition operations that affect the storage address of data in the source database, such as insert and delete operations. Data definition operations are also called Data Definition Language (DDL) operations.

[0105] There are many ways to implement the acquisition of the updated storage address corresponding to the target data definition operation in response to the target data definition operation. The specific selection is made according to the actual situation and is not limited in this specification.

[0106] In one possible implementation of the present specification, in response to a target data definition operation, obtaining an updated storage address corresponding to the target data definition operation may be to obtain an initial data definition operation, determine the target data definition operation from the initial data definition operation, and obtain an updated storage address corresponding to the target data definition operation.

[0107] Determining the target data definition operation from the initial data definition operation may be determining the data definition operation that meets a preset condition as the target data definition operation, wherein the preset condition is generating an address change impact on the storage address of the data in the source database.

[0108] In another possible implementation of the present specification, in response to the target data definition operation, the updated storage address corresponding to the target data definition operation is obtained. Alternatively, the target data definition operation may be obtained, and based on the target data definition operation, the updated storage address corresponding to the target data definition operation may be inferred.

[0109] Among them, based on the target data definition operation, the implementation method of inferring the updated storage address corresponding to the target data definition operation can be to perform corresponding inference update on the initial storage address based on the target data definition operation to obtain the updated storage address.

[0110] Exemplarily, the target data definition operation is to merge two columns of data in a table, merge the initial storage addresses corresponding to the two columns, obtain a merged storage address, and determine the merged storage address as the updated storage address.

[0111] The implementation method of updating the storage address of the data in the source database based on the updated storage address can be to infer the change of the storage address of the data in the source database based on the updated storage address to obtain the storage address of the data in the updated source database; it can also be based on the updated storage address, perform target data definition operation on the storage address of the data in the source database to obtain the storage address of the data in the updated source database.

[0112] Establish a mapping relationship between the storage address of the data in the updated source database and each record identifier.

[0113] By responding to the target data definition operation, obtaining the updated storage address corresponding to the target data definition operation, updating the storage address of the data in the source database based on the updated storage address, and establishing a mapping relationship between the storage address of the data in the updated source database and each record identifier, the updated storage address caused by the target data definition operation is added to the data synchronized, thereby ensuring the accuracy of the data after data synchronization and the consistency of the data in the source database and the target database.

[0114] In an optional embodiment of the present specification, the above steps of establishing a mapping relationship between the storage address of the data in the source database and each record identifier include the following steps:

[0115] The storage address of the data in the source database is filled into each record identifier, and a mapping relationship between the storage address of the data in the source database and each record identifier is determined.

[0116] Exemplarily, the data in the source database is stored in the form of records, and each record corresponds to a storage address. Therefore, the records in the source database can have a one-to-one correspondence with the record identifiers in the target database table. In this way, according to the correspondence, the storage address of the record in the source database can be filled into the corresponding record identifier of the target database to obtain the mapping relationship between the storage address of the data in the source database and each record identifier.

[0117] By filling the storage address of the data in the source database into each record identifier, a consistency comparison can be performed based on the storage address filled in each record identifier and the storage address of the data in the source database during data synchronization, thereby ensuring the accuracy of data synchronization and the consistency of the synchronized data in the source database and the target database.

[0118] Step 308: Based on the mapping relationship, synchronize the data of the storage address in the source database to the record position of each record identifier corresponding to the identifier in the target database.

[0119] Based on the mapping relationship, the data to be synchronized in the source database is migrated to the target database according to the consistency comparison between the data of the storage address and the record position corresponding to the record identifier, thereby realizing data migration between different databases.

[0120] When receiving a data synchronization task, the embodiments of this specification create at least one record identifier in the target database according to the data synchronization task, and establish a mapping relationship between the storage address of the data in the source database and each record identifier, so that when synchronizing data between the source database and the target database, synchronization can be performed based on the mapping relationship. Even in scenarios such as network interruption and disaster recovery, repeated data synchronization can be avoided based on the mapping relationship, thereby ensuring the accuracy of data synchronization.

[0121] The following combined Figure 4a , taking the application of the data synchronization method provided in this specification in a heterogeneous database as an example, the data synchronization method is further explained. Figure 4a A processing flow chart of a data synchronization method provided in an embodiment of this specification is shown, which specifically includes the following steps.

[0122] Step 402: Receive a data synchronization task for incremental data.

[0123] When there is a need to synchronize incremental data on the front end, the server will receive a data synchronization task for the incremental data.

[0124] Step 404: When the structure of the table in the source database has neither a primary key nor a unique key, a hidden column (pol arSeqRowid) is created in the target database, and this column is set as the unique key of the table in the target database.

[0125] After receiving the data synchronization task, the source database is determined based on the data synchronization task, and the structure of the table in the source database is identified.

[0126] The source database may be an Oracle database.

[0127] Step 406: Obtain the ROWID (the address where the data is physically stored in the source database) of each row of data from the source database, and use the address as the unique key of the source database transaction.

[0128] When the source database does not have a primary key or unique key, the storage address of each row of data in the table in the source database is determined as a unique key based on the unique characteristics of the storage address, so as to synchronize the data in the source database to the target database using the storage address.

[0129] Step 408: Execute a ROWID change in the source database to obtain an updated ROWID, wherein when the ROWID change is caused by a partial DDL, the relevant ROWID change is recorded.

[0130] When there is a change operation on the database, the change operation is usually recorded, and the actual change is not directly made to the database. Therefore, when the database data needs to be synchronized, the ROWID is changed in the source database to obtain the updated ROWID. The updated ROWID is then used for data synchronization to ensure the accuracy of data synchronization.

[0131] Specifically, the ROWID of each row of data in the source database is changed.

[0132] Step 410: Write the updated ROWID in the source database into the hidden column in the target database, and determine the mapping relationship between the updated ROWID and the hidden column.

[0133] According to the correspondence between rows, the updated ROWID corresponding to each row is written into each identifier position in the hidden column of the target database. The hidden column includes multiple identifiers, and each identifier has a corresponding position in the target database.

[0134] Step 412: Based on the mapping relationship, migrate the data to be synchronized with the updated ROWID in the source database to the row position corresponding to the identifier of the hidden column in the target database.

[0135] According to the mapping relationship between the storage address and the hidden column, the correspondence between the row in the source database and the row corresponding to the hidden column is determined. Based on the correspondence, the data to be synchronized with the updated ROWID in the source database is migrated to the row position corresponding to the hidden column in the target database.

[0136] Step 414: Receive the stream switching instruction from the front end, execute the stream switching instruction, delete the metadata of the hidden column, and delete the actual data of the hidden column when arranging the table space in the target database during idle time.

[0137] Deleting both the metadata and actual data of a hidden column at the same time can cause the database to lock the table and require rewriting the data in the database. To avoid this, you can first delete the metadata of the hidden column, and then delete the actual data of the hidden column during idle time. This step-by-step deletion of the hidden column avoids errors such as database table locks and data rewriting.

[0138] To delete the actual data in the hidden column, it is necessary to shut down the database, delete the actual data in the hidden column, and then restart the database.

[0139] See also Figure 4b , shows a schematic diagram of asynchronous deletion of hidden columns in a data synchronization method provided by an embodiment of this specification.

[0140] Hidden columns are not visible on the front end. Figure 4b In the figure, the deletion process of hidden columns is shown. Figure 4b As can be seen, the data corresponding to header 1 is a hidden column, which includes data 1, data 2, and data 3. Header 2 is the common data in the database table, including data 4, data 5, and data 6. The way to delete the metadata of the hidden column can be to mark header 1 as deleted. When idle, clean up the actual data of the hidden column, that is, delete data 1, data 2, and data 3, and determine that the remaining data in the database table is header 2 and the included data 4, data 5, and data 6, to avoid locking the table for a long time.

[0141] By applying the solution implemented in this specification, when a data synchronization task is received, at least one record identifier is created in the target database according to the data synchronization task, and a mapping relationship between the storage address of the data in the source database and each record identifier is established, so that when synchronizing the data between the source database and the target database, data synchronization can be performed based on the mapping relationship. Even in the event of network interruption, disaster recovery, etc., the data between the source database and the target database can continue to be accurately synchronized based on the mapping relationship after the network is restored and the disaster recovery is completed, thereby avoiding duplicate synchronization of data and ensuring the accuracy of data synchronization.

[0142] Corresponding to the above method embodiment, this specification also provides a data synchronization device embodiment, Figure 5 FIG1 shows a schematic diagram of the structure of a data synchronization device provided by an embodiment of this specification. Figure 5 As shown, the device includes:

[0143] The receiving module 502 is configured to receive a data synchronization task, wherein the data synchronization task is used to instruct synchronization of data from a source database to a target database;

[0144] A creation module 504 is configured to create at least one record identifier in the target database according to the data synchronization task, wherein the record identifier is used to identify a record in a table, and each record identifier is different from another;

[0145] Establishing module 506, configured to establish a mapping relationship between the storage address of the data and each record identifier in the source database;

[0146] The synchronization module 508 is configured to synchronize the data of the storage address in the source database to the record position corresponding to each record identifier in the target database based on the mapping relationship.

[0147] Optionally, the creation module 504 is further configured to identify the structure of the table in the source database according to the data synchronization task; and create at least one record identifier in the target database when the structure of the table has no primary key and no unique key.

[0148] Optionally, the creation module 504 is further configured to parse the data synchronization task and obtain the record identifier in a hidden form, wherein the front end is unaware of the hidden form of the record identifier; and create at least one first record identifier in the hidden form in the target database.

[0149] Optionally, the data synchronization device further includes a first deletion module configured to execute the stream switching instruction in response to the received stream switching instruction and delete the metadata of the first record identifier, where the metadata is used to define the structure of the first record identifier in the target database.

[0150] Optionally, the data synchronization device further includes a second deletion module configured to delete the first record identifier when arranging the table space of the target database.

[0151] Optionally, the creation module 504 is further configured to parse the data synchronization task and obtain the record identifier in an explicit form, wherein the front end is aware of the record identifier in the explicit form; and create at least one second record identifier in the explicit form in the target database.

[0152] Optionally, the data synchronization device further includes a third deletion module configured to, in response to the received stream switching instruction from the front end, execute a database shutdown operation and delete the second record identifier; execute a database startup operation and execute the stream switching instruction.

[0153] Optionally, the data synchronization device also includes a response module, which is configured to obtain an updated storage address corresponding to the target data definition operation in response to the target data definition operation, wherein the target data definition operation refers to an operation that affects the storage address of the data in the source database and has an address change; an establishment module 506 is further configured to update the storage address of the data in the source database based on the updated storage address; and establish a mapping relationship between the storage address of the data in the source database and each of the record identifiers after the update.

[0154] Optionally, the establishing module 506 is further configured to fill the storage address of the data in the source database into each of the record identifiers, and determine a mapping relationship between the storage address of the data in the source database and each of the record identifiers.

[0155] By applying the solution implemented in this specification, when a data synchronization task is received, at least one record identifier is created in the target database according to the data synchronization task, and a mapping relationship between the storage address of the data in the source database and each record identifier is established, so that when synchronizing the data between the source database and the target database, data synchronization can be performed based on the mapping relationship. Even in the event of network interruption, disaster recovery, etc., the data between the source database and the target database can continue to be accurately synchronized based on the mapping relationship after the network is restored and the disaster recovery is completed, thereby avoiding duplicate synchronization of data and ensuring the accuracy of data synchronization.

[0156] The above is a schematic scheme of a data synchronization device of this embodiment. It should be noted that the technical scheme of the data synchronization device and the technical scheme of the above-mentioned data synchronization method are of the same concept. For details not described in detail in the technical scheme of the data synchronization device, please refer to the description of the technical scheme of the above-mentioned data synchronization method.

[0157] Figure 6 6 shows a block diagram of a computing device according to an embodiment of the present disclosure. Components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0158] The computing device 600 also includes an access device 640 that enables the computing device 600 to communicate via one or more networks 660. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, or a near field communication (NFC) interface.

[0159] In one embodiment of the present specification, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. Figure 6 The computing device structure block diagram shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0160] Computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 600 may also be a mobile or stationary server.

[0161] The processor 620 is configured to execute the following computer program / instruction, which implements the steps of the above-mentioned data synchronization method when executed by the processor.

[0162] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above-mentioned data synchronization method are of the same concept. For details not described in detail in the technical scheme of the computing device, please refer to the description of the technical scheme of the above-mentioned data synchronization method.

[0163] An embodiment of the present specification further provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned data synchronization method when executed by a processor.

[0164] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of this storage medium and the technical scheme of the above-mentioned data synchronization method are of the same concept. For details not described in detail in the technical scheme of the storage medium, please refer to the description of the technical scheme of the above-mentioned data synchronization method.

[0165] An embodiment of the present specification further provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned data synchronization method when executed by a processor.

[0166] The above is a schematic solution of a computer program product of this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the above-mentioned data synchronization method are based on the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the above-mentioned data synchronization method.

[0167] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0168] The computer instructions include computer program codes, which may be in source code form, object code form, executable files, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0169] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0170] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0171] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data synchronization method, comprising: receiving a data synchronization task, wherein the data synchronization task is used to instruct synchronization of data from a source database to a target database; According to the data synchronization task, at least one record identifier is created in the target database, wherein the record identifier is used to identify a record of a table, and each record identifier is different from another; Establishing a mapping relationship between the storage address of the data in the source database and each of the record identifiers; Based on the mapping relationship, the data of the storage address in the source database is synchronized to the record position corresponding to each record identifier in the target database.

2. The method according to claim 1, wherein creating at least one record identifier in the target database according to the data synchronization task comprises: According to the data synchronization task, identifying the structure of the table in the source database; In a case where the structure of the table has neither a primary key nor a unique key, at least one record identifier is created in the target database.

3. The method according to claim 1 or 2, wherein creating at least one record identifier in the target database according to the data synchronization task comprises: Parsing the data synchronization task, obtaining that the record identifier is in a hidden form, wherein the front end is unaware of the hidden record identifier; At least one first record identification of the hidden form is created in the target database.

4. The method according to claim 3, after synchronizing the data of the storage address in the source database to the record position corresponding to each record identifier in the target database based on the mapping relationship, further comprising: In response to the received stream switching instruction, the stream switching instruction is executed, and the meta information of the first record identifier is deleted, where the meta information is used to define the structure of the first record identifier in the target database.

5. The method according to claim 4, further comprising, after executing the stream switching instruction in response to the received stream switching instruction and deleting the meta-information of the first record identifier: When the table space of the target database is organized, the first record identifier is deleted.

6. The method according to claim 1 or 2, wherein creating at least one record identifier in the target database according to the data synchronization task comprises: Parsing the data synchronization task to obtain the record identifier in an explicit form, wherein the front end is aware of the record identifier in the explicit form; At least one second record identifier in the explicit form is created in the target database.

7. The method according to claim 6, after synchronizing the data of the storage address in the source database to the target database based on the mapping relationship, further comprising: In response to the received stream switching instruction from the front end, executing a database shutdown operation and deleting the second record identifier; Execute the database startup operation and execute the stream switching instruction.

8. The method according to claim 1, before establishing the mapping relationship between the storage address of the data in the source database and each of the record identifiers, further comprising: In response to a target data definition operation, obtaining an updated storage address corresponding to the target data definition operation, wherein the target data definition operation is an operation that affects an address change of a storage address of data in the source database; The establishing of a mapping relationship between the storage address of the data in the source database and each of the record identifiers includes: Based on the updated storage address, updating the storage address of the data in the source database; A mapping relationship between the storage address of the updated data in the source database and each of the record identifiers is established.

9. The method according to claim 1, wherein establishing a mapping relationship between the storage address of the data in the source database and each of the record identifiers comprises: The storage address of the data in the source database is filled into each of the record identifiers, and a mapping relationship between the storage address of the data in the source database and each of the record identifiers is determined.

10. A data synchronization system, comprising a client and a server; The client is used to send a data synchronization request; The server is used to receive the data synchronization request and determine the data synchronization task according to the data synchronization request, wherein: The data synchronization task is used to instruct the synchronization of data from the source database to the target database; according to the data synchronization task, at least one record identifier is created in the target database, wherein the record identifier is used to identify the record of the table, and each record identifier is different from each other; a mapping relationship between the storage address of the data in the source database and each record identifier is established; based on the mapping relationship, the data at the storage address in the source database is synchronized to the record position corresponding to each record identifier in the target database.

11. A computing device comprising: memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the data synchronization method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the data synchronization method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the data synchronization method according to any one of claims 1 to 9.

Images