Data synchronization method and device

Abstract

The present invention relates to the field of financial technology (Fintech) and discloses a data synchronization method and device, in which a first synchronization component obtains a table structure mapping relationship between a first data table in a first stand-alone database and a second data table in a distributed database; the first synchronization component modifies first record information in a first log file of the first stand-alone database based on the table structure mapping relationship to obtain second record information; the first record information records data objects in the first data table to be synchronized to the second data table; the first synchronization component synchronizes the data objects to the second data table based on the second record information. This method achieves the goal of not pausing data synchronization of a stand-alone database that has undergone a table structure change in a grayscale change scenario, but rather can simultaneously implement a data table structure change operation in a distributed database for a stand-alone database that has not undergone a table structure change during the data synchronization process, thereby improving synchronization efficiency in grayscale change scenarios.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of financial technology (Fintech), and in particular to a data synchronization method and device. Background Art

[0002] With the development of computer technology, more and more technologies (such as big data, cloud computing, and blockchain) are being applied in the financial sector, and the traditional financial industry is gradually transforming into FinTech. In the context of big data technology, databases are often needed to store the various data generated by applications.

[0003] Currently, when storing data in a MySQL database, since the MySQL database is a stand-alone database, that is, it can only be stored on one physical server, and the hard disk capacity of a physical server is limited. For example, a common physical server generally has a capacity of 3TB. In industries such as the Internet and finance, with the explosive growth in the number of customers, the capacity of a stand-alone database will reach a bottleneck, that is, it will be unable to store massive amounts of data.

[0004] To address this issue, when a single-server database encounters capacity bottlenecks, its data tables can be split into multiple homogeneous tables (i.e., tables with the same structure) according to specific rules and stored on multiple single-server databases. However, this approach imposes many limitations on applications, such as the inability to perform cross-database join operations and the need to modify distributed transactions.

[0005] Because distributed databases offer unlimited horizontal scalability, meaning there are no capacity bottlenecks, the resulting homogeneous data tables can be aggregated and synchronized to a single distributed database using synchronization components. After switching to a distributed database, applications can treat the distributed database as a single-machine database with unlimited capacity, eliminating any other limitations and providing a very user-friendly experience.

[0006] For example, when storing customer numbers in financial scenarios, the customer number model is generally used to split the massive amount of customer number data and store it in multiple stand-alone databases. That is, the multiple stand-alone databases upstream are generally equipped with isomorphic data tables to store data related to the same research object (such as customer numbers). When making table structure changes to isomorphic data tables in multiple stand-alone databases upstream, in order to minimize the impact on the business, a grayscale change method is used. That is, the table structure of one or several upstream stand-alone databases is changed first. After observing for a few days to confirm that there are no abnormalities, the full surface structure change will be made to the remaining upstream stand-alone databases.

[0007] However, this phased change approach can cause the table structure of the upstream standalone database that has undergone the table structure change to become inconsistent with that of other upstream standalone databases and downstream distributed databases. Because inconsistent table structures can interrupt data synchronization, the traditional approach is to suspend data synchronization for the upstream standalone database that has undergone the table structure change. Data synchronization is only resumed after the other upstream standalone databases have fully completed the table structure change, and the downstream distributed databases are notified to also implement the table structure change.

[0008] In the aforementioned grayscale change scenario, before changing the table structure of other upstream stand-alone databases, data synchronization of the upstream stand-alone databases that have undergone table structure changes needs to be suspended, which obviously reduces the efficiency of data synchronization.

[0009] In summary, there is an urgent need for a method to improve the data synchronization efficiency in grayscale change scenarios. Summary of the Invention

[0010] The present application provides a data synchronization method and device to improve data synchronization efficiency in grayscale change scenarios.

[0011] In a first aspect, an embodiment of the present application provides a data synchronization method, which is suitable for at least one stand-alone database to synchronize data to a distributed database through its own synchronization components; the method includes: a first synchronization component obtains a table structure mapping relationship between a first data table in a first stand-alone database and a second data table in the distributed database; the first synchronization component modifies the first record information in a first log file of the first stand-alone database according to the table structure mapping relationship to obtain second record information; the first record information records the data objects in the first data table to be synchronized to the second data table; the first synchronization component synchronizes the data objects to the second data table according to the second record information.

[0012] Based on this solution, in the grayscale change scenario, after the stand-alone database that has undergone table structure changes updates the table structure of the second data table of the distributed database through its own synchronization component, the stand-alone database that has not yet undergone table structure changes (i.e., the first stand-alone database) can obtain the table structure mapping relationship between its local first data table and the second data table through its own synchronization component (i.e., the first synchronization component). Then, the first synchronization component can modify the first record information in the first log file according to the table structure mapping relationship and form the second record information. Finally, the first synchronization component can change the table structure of the first data table in the first stand-alone database in the distributed database based on the second record information. This method eliminates the need to suspend data synchronization for a stand-alone database that has undergone table structure changes in a grayscale change scenario. Instead, during the data synchronization process for the stand-alone database that has undergone table structure changes, the table structure changes for the stand-alone database that has not undergone table structure changes in the distributed database can be simultaneously implemented. This improves the data synchronization efficiency in grayscale change scenarios. On the other hand, when the table structure of the first data table of the stand-alone database that has not undergone table structure changes is subsequently changed, the synchronization efficiency of the first data table at this time can also be improved.

[0013] In one possible implementation method, the first synchronization component modifies the first record information in the first log file of the first stand-alone database according to the table structure mapping relationship to obtain second record information, including: if in the table structure mapping relationship, the second data table has an added first field relative to the first data table, then the first synchronization component adds the setting information corresponding to the first field to the first record information to obtain the second record information; if in the table structure mapping relationship, the second data table has a second field and a third field whose field positions are changed relative to the first data table, then the first synchronization component adjusts the position of the information corresponding to the second field and the information corresponding to the third field in the first record information to obtain the second record information; if in the table structure mapping relationship, the second data table has a reduced fourth field relative to the first data table, then the first synchronization component deletes the information corresponding to the fourth field in the first record information to obtain the second record information.

[0014] Based on this solution, in the process of the first synchronization component modifying the first record information according to the table structure mapping relationship to obtain the second record information, there may be situations in which fields in the data table are increased, fields are reduced, and fields remain unchanged but the positions are changed. In the case where the second data table has more fields than the first data table, the first synchronization component can adaptively add setting information corresponding to the extra fields to the first record information, thereby obtaining the second record information. In the case where the second data table has fewer fields than the first data table, the first synchronization component can adaptively delete information corresponding to the missing fields from the first record information, thereby obtaining the second record information. In the case where the fields in the second data table are the same as those in the first data table (meaning the number of fields and the contents of each field are the same) but the field positions have changed, because current technology only synchronizes data in the order of fields when synchronizing data tables with the same fields, this will obviously lead to data synchronization anomalies (data and fields do not correspond). Therefore, this solution can establish a binding relationship between two fields located in the same position in the first and second data tables, so that the first synchronization component can adaptively adjust the position of the changed fields in the first record information corresponding to the first data table, so that the data corresponding to each field in the first data table can be correctly synchronized to the second data table in the future without data confusion.

[0015] In one possible implementation method, before the first synchronization component obtains the table structure mapping relationship between the first data table in the first stand-alone database and the second data table in the distributed database, it also includes: the first synchronization component checks the data object according to a check script; the check script includes multiple check items, and each check item is used to determine whether data synchronization operations can be performed on the data object; the first synchronization component determines that the data object meets the requirements of each check item.

[0016] Based on this solution, when synchronizing the data table in a stand-alone database to a distributed database, because the distributed database is incompatible with or has some restrictions on some features, such as the data table must have a primary key and the data table cannot have a foreign key, the current technology requires manual confirmation of each check item before realizing data synchronization from a stand-alone database to a distributed database, in order to confirm in advance whether the distributed database is compatible. Obviously, this method of manually confirming whether data synchronization from a stand-alone database to a distributed database can be achieved is time-consuming and labor-intensive, greatly reducing the efficiency of data synchronization. To this end, this solution allows the first synchronization component to run a pre-configured check script to check the data table, and immediately synchronizes the data after confirming that the requirements of each check item are met. On the one hand, this method can achieve universality; different distributed databases may have different check items, so each check item can be flexibly configured according to actual conditions; on the other hand, it can also realize automated inspection, without the need to manually check each check item one by one, saving labor costs and improving the efficiency of data synchronization.

[0017] In one possible implementation method, the first synchronization component sends registration information to a monitoring module; the first synchronization component receives first information sent by the monitoring module; the first information is used to instruct the first synchronization component to feedback the survival status of the first synchronization component to the monitoring module; the first synchronization component sends second information to the monitoring module; the second information is used to instruct the monitoring module to determine whether to replace the first synchronization component.

[0018] Based on this solution, when multiple stand-alone databases use their own synchronization components to synchronize data between isomorphic data tables and distributed databases, if the synchronization component corresponding to one of the stand-alone databases fails (process abnormality or the server where it is located is down), the data synchronization process corresponding to the stand-alone database will terminate abnormally, which will greatly affect the overall data synchronization efficiency. To this end, this solution uses a monitoring module to monitor the survival status of the first synchronization component, and after monitoring the failure of the first synchronization component, immediately activates an idle synchronization component to replace the first synchronization component, which can make the synchronization component highly available and achieve self-healing of faults; and can avoid the problem of decreased data synchronization efficiency caused by the abnormality of the first synchronization component.

[0019] In one possible implementation method, after the first synchronization component synchronizes the data object to the second data table according to the second record information, it also includes: for any stand-alone database in each stand-alone database, the first synchronization component determines according to the verification script that the synchronization delay between the distributed database and the stand-alone database meets the preset delay requirement; the first synchronization component determines according to the verification script that the distributed database and the stand-alone database have completed data synchronization; the first synchronization component switches the database connected to the application from each stand-alone database to the distributed data based on the application switching script.

[0020] Based on this solution, current technology requires near-real-time data synchronization. SQL (Structured Query Language) additions, deletions, and modifications in the upstream standalone database require a certain amount of time to be transmitted to the downstream distributed database for execution, typically around a second or even longer. To ensure complete data consistency between upstream and downstream, applications typically require a service outage before switching databases. This downtime significantly impacts the customer experience and the 24 / 7 uninterrupted service requirements of businesses. To ensure complete data consistency between upstream and downstream, current technology requires manual verification of downstream data aligning with upstream data, which is time-consuming and labor-intensive. Therefore, the first synchronization component in this solution quickly verifies whether the downstream distributed database has synchronized with the upstream standalone database by running a verification script, eliminating the time-consuming and labor-intensive manual verification process. Furthermore, because this solution can quickly verify data synchronization, the application switching script in this solution can quickly switch the database connected to the application from a standalone database to a distributed database without requiring service downtime, improving the user experience and making it particularly suitable for financial scenarios.

[0021] In one possible implementation method, for any stand-alone database among the stand-alone databases, the first synchronization component determines, according to a verification script, that the synchronization delay between the distributed database and the stand-alone database meets a preset delay requirement, including: the first synchronization component writes a first timestamp to the stand-alone database according to the verification script; the first timestamp is used to represent the moment of writing to the stand-alone database; if the first synchronization component determines that the difference between the second timestamp and the first timestamp meets a set threshold, then it determines that the synchronization delay between the distributed database and the stand-alone database meets the preset delay requirement; the second timestamp is the moment when the first timestamp is synchronized to the distributed database as a data object; the first synchronization component determines, according to the verification script, that data synchronization between the distributed database and the stand-alone database has been completed, including: the first synchronization component sets the stand-alone database to read-only according to the verification script; the first synchronization component determines that data synchronization between the distributed database and the stand-alone database has been completed when the data synchronization record value of the distributed database is consistent with the data synchronization record value of the stand-alone database.

[0022] Based on this solution, this solution can quickly determine whether the downstream distributed database has completed data synchronization with the upstream stand-alone database by running the verification script. Among them, the verification script in this solution includes two aspects. On the one hand, it is necessary to first confirm whether the delay of a single piece of data in synchronizing from the upstream stand-alone database to the downstream distributed data meets the set delay requirements. Only when the delay meets the set delay requirements can the relevant operations in the second aspect be performed; on the other hand, when it is confirmed that the delay meets the set delay requirements, the upstream stand-alone database needs to be set to read-only, and the data synchronization behavior from the upstream stand-alone database to the downstream distributed database continues under read-only conditions, and periodically checks whether the data synchronization record values of the upstream and downstream databases are consistent, and within the timeout protection period, it is determined that the data synchronization record values of the upstream and downstream databases are consistent, which means that data synchronization has been achieved. This solution automatically executes the above two aspects of the verification script for data synchronization, with high inspection efficiency, avoiding the time-consuming and labor-intensive problems of manual inspection.

[0023] In one possible implementation method, the first synchronization component switches the database connected to the application from each stand-alone database to the distributed data based on the application switching script, including: the first synchronization component sets the access IP weight of each stand-alone database set in the proxy gateway from a first value to a second value based on the application switching script, and sets the access IP weight of the distributed database set in the proxy gateway from the second value to the first value; wherein the first value is 100% and the second value is 0.

[0024] Based on this solution, through proxy gateway technology, the access IPs (Internet Protocol, a protocol for interconnecting networks) of upstream stand-alone databases and downstream distributed databases can be unified into a virtual IP (VIP). Before switching the database connected to the application from a stand-alone database to a distributed database, that is, before data synchronization between the downstream distributed database and the upstream stand-alone database is achieved, the access IP weight of the upstream stand-alone database can be set to 100, and the access IP weight of the downstream distributed database can be set to 0. A weight of 0 means that the corresponding IP is removed from the forwarding target at the network layer, that is, at this time, only the data generated by the application will be written to the upstream stand-alone database and not to the downstream distributed database; once the verification script is run to confirm that the downstream distributed database has achieved data synchronization with the upstream stand-alone database, the access IP weight of the downstream distributed database can be set to 100, and the access IP weight of the upstream stand-alone database can be set to 0, that is, at this time, only the data generated by the application will be written to the downstream distributed database and not to the upstream stand-alone database, thereby quickly switching the database connected to the application without stopping the application, improving the user experience and being particularly friendly to financial scenarios.

[0025] In one possible implementation method, before the first synchronization component obtains the table structure mapping relationship between the first data table in the first stand-alone database and the second data table in the distributed database, it also includes: the second synchronization component receives a grayscale release instruction; the second synchronization component is used to synchronize data between a third data table in the second stand-alone database and the second data table, and the third data table and the first data table are isomorphic data tables; the second synchronization component updates the second data table according to the grayscale release instruction.

[0026] Based on this solution, current data synchronization technology for upstream standalone databases and downstream distributed databases involves performing a grayscale change when making table structure changes to homogeneous data tables in each standalone database to minimize business impact. This involves first performing the table structure change on one or a few upstream standalone databases. After a few days of observation to confirm that no anomalies exist, the full table structure change is performed on the remaining upstream standalone databases. This grayscale change scenario can cause the table structure of the upstream standalone database that has undergone the table structure change to become more inconsistent with that of other upstream standalone databases and downstream distributed databases. This inconsistency can interrupt data synchronization. Current technology suspends data synchronization for the upstream standalone database that has undergone the table structure change, resuming data synchronization only after the other upstream standalone databases have fully completed the table structure change, and notifying the downstream distributed databases to also perform the table structure change. Clearly, in this grayscale change solution, the downstream distributed database is the last to undergo the table structure change, significantly reducing the overall efficiency of data synchronization. To this end, this solution proposes that when the table structure of one or several upstream stand-alone databases is first changed, the table structure of the downstream distributed database is also changed at the same time. Therefore, when the table structure of other upstream stand-alone databases is subsequently changed, there is no need to suspend data synchronization between the upstream stand-alone database that has undergone table structure changes and the downstream distributed database, thereby improving the overall efficiency of data synchronization.

[0027] In the second aspect, an embodiment of the present application provides a data synchronization device, which is suitable for at least one stand-alone database to synchronize data to a distributed database through its own synchronization components; the device includes: a table structure mapping relationship acquisition unit, used to obtain the table structure mapping relationship between the first data table in the first stand-alone database and the second data table in the distributed database; a record information update unit, used to modify the first record information in the first log file of the first stand-alone database according to the table structure mapping relationship to obtain second record information; the first record information records the data objects in the first data table to be synchronized to the second data table; a synchronization unit, used to synchronize the data objects to the second data table according to the second record information.

[0028] In one possible implementation method, the record information update unit is specifically used to: if in the table structure mapping relationship, the second data table has an added first field relative to the first data table, then the first synchronization component adds the setting information corresponding to the first field in the first record information to obtain the second record information; if in the table structure mapping relationship, the second data table has a second field and a third field whose field positions are changed relative to the first data table, then the first synchronization component adjusts the position of the information corresponding to the second field and the information corresponding to the third field in the first record information to obtain the second record information; if in the table structure mapping relationship, the second data table has a reduced fourth field relative to the first data table, then the first synchronization component deletes the information corresponding to the fourth field in the first record information to obtain the second record information.

[0029] In one possible implementation method, the device also includes a data synchronization confirmation unit; the data synchronization confirmation unit is used to: check the data object according to a check script; the check script includes multiple check items, each check item is used to determine whether data synchronization operation can be performed on the data object; determine whether the data object meets the requirements of each check item.

[0030] In one possible implementation method, the device also includes a transceiver unit; the transceiver unit is used to: send registration information to the monitoring module; receive first information sent by the monitoring module; the first information is used to instruct the first synchronization component to feedback the survival status of the first synchronization component to the monitoring module; send second information to the monitoring module; the second information is used to instruct the monitoring module to determine whether to replace the first synchronization component.

[0031] In one possible implementation method, the device also includes an application switching unit; the application switching unit is used to: for any stand-alone database among the stand-alone databases, determine according to a verification script whether the synchronization delay between the distributed database and the stand-alone database meets the preset delay requirement; determine according to the verification script that the distributed database and the stand-alone database have completed data synchronization; based on the application switching script, switch the database connected to the application from each stand-alone database to the distributed database.

[0032] In one possible implementation method, the application switching unit is specifically used to: write a first timestamp to the stand-alone database according to a verification script; the first timestamp is used to represent the moment of writing to the stand-alone database; if it is determined that the difference between the second timestamp and the first timestamp meets the set threshold, it is determined that the synchronization delay between the distributed database and the stand-alone database meets the preset delay requirement; the second timestamp is the moment when the first timestamp is synchronized to the distributed database as a data object; according to the verification script, set the stand-alone database to read-only; when the data synchronization record value of the distributed database is consistent with the data synchronization record value of the stand-alone database, it is determined that the distributed database and the stand-alone database have completed data synchronization.

[0033] In one possible implementation method, the application switching unit is specifically used to: based on the application switching script, set the access IP weight of each stand-alone database set in the proxy gateway from a first value to a second value, and set the access IP weight of the distributed database set in the proxy gateway from the second value to the first value; wherein, the first value is 100% and the second value is 0.

[0034] In one possible implementation method, a second synchronization component is also included, which includes a grayscale release instruction receiving unit and a data table updating unit; the grayscale release instruction receiving unit is used to receive the grayscale release instruction; the second synchronization component is used to synchronize data between a third data table and the second data table in a second stand-alone database, and the third data table and the first data table are isomorphic data tables; the data table updating unit is used to update the second data table according to the grayscale release instruction.

[0035] In a third aspect, an embodiment of the present application provides a computing device, including:

[0036] a memory for storing program instructions;

[0037] The processor is used to call the program instructions stored in the memory and execute any implementation method of the first aspect according to the obtained program.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute any implementation method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 A schematic diagram of a data synchronization process from a single-machine database to a distributed database in the prior art;

[0041] Figure 2 A schematic diagram of the working principle of a synchronization component in the prior art during data synchronization;

[0042] Figure 3 A data synchronization method provided in an embodiment of the present application;

[0043] Figure 4 A diagram showing the working principle of a monitoring module (detector) provided in an embodiment of the present application;

[0044] Figure 5 A data synchronization system provided in an embodiment of the present application;

[0045] Figure 6 A schematic diagram of the principle of replacing a synchronization component provided in an embodiment of the present application;

[0046] Figure 7 A diagram of a database connection state before application switching provided in an embodiment of the present application;

[0047] Figure 8 A diagram of the database connection status during application switching provided in an embodiment of the present application;

[0048] Figure 9 A data synchronization device provided in an embodiment of the present application;

[0049] Figure 10 A computing device is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] To make the objectives, technical solutions, and advantages of this application more clear, this application will be further described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0051] Currently, if you need to store data corresponding to a research object in the financial field, you can use a stand-alone database to store it, and the stand-alone database can use a MySQL database. However, when the data corresponding to the research object increases, if the data is only stored in a stand-alone database, capacity bottlenecks are likely to occur. To solve this problem, based on certain storage rules, the data table stored on the stand-alone database for storing the data corresponding to the research object can be split into multiple isomorphic data tables (i.e., the data table structure is the same), and then these multiple isomorphic data tables are stored in a stand-alone database respectively. However, this method has many limitations on applications, such as the inability to perform cross-database join operations, which requires the transformation of distributed transactions.

[0052] Since the capacity of a distributed database is infinitely scalable, that is, there is no bottleneck in capacity, the homogeneous data tables obtained by splitting can be aggregated and synchronized into a distributed database through synchronization components. Figure 1 As shown in FIG, a schematic diagram of the data synchronization process from a single-machine database to a distributed database in the prior art is shown. Since the data in the single-machine database needs to be synchronized to the distributed database, each single-machine database is an upstream database, and the distributed database is a downstream database. One upstream database corresponds to one synchronization component, such as Figure 1 The synchronization components 1, 2, 3, ..., N are used to synchronize data between upstream database 1, upstream database 2, upstream database 3, ..., upstream database N and the downstream distributed database. Figure 2 The figure below is a schematic diagram illustrating the working principle of a synchronization component in the prior art during data synchronization. The synchronization component is responsible for parsing the upstream database's binlog file and replaying and executing SQL statements in the downstream database to achieve data synchronization. The binlog is a binary log file that records internal MySQL database changes (only data modifications are recorded).

[0053] After data synchronization is achieved, the database connected to the application can be switched from the upstream standalone database to the downstream distributed database. After switching to the distributed database, the application can treat the distributed database as a single-machine database with unlimited capacity, without any other restrictions, which is obviously very application-friendly.

[0054] In the process of synchronizing data from the upstream stand-alone database to the downstream distributed data, if it is required to change the table structure of the isomorphic data tables in the upstream stand-alone database at the same time, in order to minimize the impact on the business, a grayscale change method can be adopted. That is, the table structure of one or several upstream stand-alone databases is changed first. After observing for a few days to confirm that there are no abnormalities, the full surface structure change of the remaining upstream stand-alone databases is performed.

[0055] The aforementioned grayscale change method can cause the table structure of the upstream standalone database that has undergone the table structure change to be inconsistent with that of other upstream standalone databases with isomorphic data tables, as well as the table structure of downstream distributed databases. Because inconsistent table structures can interrupt data synchronization, the current technology suspends data synchronization for the upstream standalone database that has undergone the table structure change. Data synchronization is resumed only after the other upstream standalone databases with isomorphic data tables have fully completed the table structure change. This notifies the downstream distributed databases to also implement the table structure change.

[0056] Obviously, the current grayscale change technology requires suspending data synchronization of the upstream stand-alone database that has undergone table structure changes in order to change the table structure of the isomorphic data table of other upstream stand-alone databases with isomorphic data tables. This obviously greatly reduces the efficiency of data synchronization.

[0057] In order to solve the above technical problems, the embodiment of the present application provides a data synchronization method, which is applicable to at least one stand-alone database to synchronize data to a distributed database through its own synchronization components, such as Figure 3 As shown, the method includes the following steps:

[0058] Step 301: A first synchronization component obtains a table structure mapping relationship between a first data table in a first stand-alone database and a second data table in the distributed database.

[0059] Before this step, optionally, a second synchronization component receives a grayscale release instruction; the second synchronization component is used to synchronize data between a third data table and the second data table in a second stand-alone database, and the third data table and the first data table are isomorphic data tables; the second synchronization component updates the second data table according to the grayscale release instruction.

[0060] For example, you can Figure 1The architecture shown illustrates this solution. Before making table structure changes to isomorphic data tables, upstream standalone databases 1, 2, 3, ..., and N share isomorphic data tables with the downstream distributed database for the same research object. These isomorphic data tables are denoted as the second data table in the downstream distributed database. Assume that this grayscale release uses upstream standalone database 1 as the test object. This means that the grayscale release first modifies the table structure of the isomorphic data tables in upstream standalone database 1. Then, based on the observation of the data synchronization effect of standalone database 1 after the modified table structure over a period of time, it can be determined whether to make full table structure changes to upstream standalone databases 2, 3, ..., and N. Based on this assumption, the isomorphic data table can be denoted as the third data table in upstream standalone database 1 and the first data table in any of the upstream standalone databases: 2, 3, ..., and N. Therefore, in this grayscale release, when the table structure of the third data table in the upstream stand-alone database 1 is changed first, Figure 1 The synchronization component 1 shown can be used as a second synchronization component to receive the grayscale release instruction initiated by the user. The grayscale release instruction includes information for indicating changes in the table structure. For example, the information on the table structure change can be the addition of fields, reduction of fields, and position adjustment of existing fields in the third data table relative to the second data table. Thus, when the synchronization component 1 receives the grayscale release instruction, it can update the second data table in the downstream distributed database according to the grayscale release instruction. After the synchronization component 1 updates the table structure of the second data table, the synchronization component 2, synchronization component 3... synchronization component N, which are respectively responsible for data synchronization between the stand-alone database 2, stand-alone database 3... stand-alone database N and the distributed database, can be used as a first synchronization component, and the first synchronization component can obtain the table structure mapping relationship between the first data table and the second data table.

[0061] The following is a specific example to illustrate this.

[0062] Assume there are three upstream standalone databases, namely upstream standalone database B, upstream standalone database C, and upstream standalone database D. The isomorphic data tables of these three upstream standalone databases for the same research object are denoted as table A. In addition, synchronization components B', C', and D' are used to synchronize data from upstream standalone database B, upstream standalone database C, and upstream standalone database D to the downstream distributed database. Then:

[0063] First, the initial state of the table structure of each upstream stand-alone database regarding table D is:

[0064] The structure of table A in the upstream stand-alone database B is:

[0065] x int

[0066] y int

[0067] z int

[0068] The structure of table A in the upstream stand-alone database C is:

[0069] x int

[0070] y int

[0071] z int

[0072] The structure of table A in the upstream stand-alone database D is:

[0073] x int

[0074] y int

[0075] z int

[0076] That is, table A of the three upstream stand-alone databases contains field x, field y, and field z in sequence.

[0077] Next, assume that this grayscale release uses the upstream stand-alone database B as the test object, that is, this grayscale release first changes the table structure of data table A in the upstream stand-alone database B; and, we can further assume that the change in the table structure is to add a field m, then the specific grayscale release instruction can be expressed as: add m varchar(100)default 100 field to table A in the upstream stand-alone database B (alter table B add column m varchar(100)default 100). Then the synchronization component B' can change the structure of table A in the upstream stand-alone database B, while the synchronization component C' will not change the structure of table A in the upstream stand-alone database C, and the synchronization component D' will not change the structure of table A in the upstream stand-alone database D. Therefore:

[0078] The structure of table A in the upstream stand-alone database B is:

[0079] x int

[0080] y int

[0081] z int

[0082] m varchar(100)default'pad'

[0083] The structure of table A in the upstream stand-alone database C is:

[0084] x int

[0085] y int

[0086] z int

[0087] The structure of table A in the upstream stand-alone database D is:

[0088] x int

[0089] y int

[0090] z int

[0091] Among them, when synchronization component B' adds a field m to table A of the upstream stand-alone database B, synchronization component B' will directly execute the addition of m varchar(100) default 100 field (alter table A add column m varchar(100) default 100) in the downstream distributed database. During the data synchronization process, each synchronization component will record its own operation information on the data table in a log file. For example, in the context of a MySQL database, each operation information can be recorded as a POS point, and the log file can be a binlog file. The grayscale change process is no exception. Therefore, after the table structure of table A of the upstream stand-alone database B is changed, for the stand-alone database C and the stand-alone database D that have not yet changed their table structure, synchronization component C' and synchronization component D' can respectively obtain the differences in table structure between the table A they are responsible for and the table A in the downstream distributed database. Therefore, synchronization component C' can obtain the table structure mapping relationship between table A in the upstream stand-alone database C and the downstream distributed database. Similarly, synchronization component D can obtain the same table structure mapping relationship.

[0092] In step 302, the first synchronization component modifies the first record information in the first log file of the first stand-alone database according to the table structure mapping relationship to obtain second record information; the first record information records the data objects in the first data table to be synchronized to the second data table.

[0093] In a stand-alone database, there can be multiple data tables at the same time. Among them, for any data table in the stand-alone database, there can be a corresponding record information, which can be used to record the user's operation information on the data table one by one. Therefore, during the data synchronization process, by replaying the record information in the distributed database, data synchronization between the stand-alone database and the distributed database can be achieved; in addition, each record information in the same stand-alone database can be stored in a log file together. For example, suppose there are three data tables in a stand-alone database, let them be data table 1, data table 2 and data table 3 respectively, then each user's operation information on data table 1 can be recorded in record information 1, each user's operation information on data table 2 can be recorded in record information 2, and each user's operation information on data table 3 can be recorded in record information 3, and record information 1, record information 2 and record information 3 are all stored in the log file corresponding to the stand-alone database. Therefore, in this step, after the first synchronization component obtains the table structure mapping relationship, it can search for the first record information corresponding to the first data table in the first log file corresponding to the first stand-alone database, and modify the first record information according to the obtained table structure mapping relationship, thereby generating updated first record information, that is, second record information.

[0094] Step 303: The first synchronization component synchronizes the data object to the second data table according to the second record information.

[0095] In this step, since the second record information has been generated in step 302, the update operation of the table structure of the first data table prepared by the user can be executed in the distributed database by replaying the second record information in the distributed database.

[0096] For example, in the case where the aforementioned upstream stand-alone databases B, C, and D each have an isomorphic data table A, when synchronization component B' adds a field m to table A of the upstream stand-alone database B, synchronization component C' parses the record information corresponding to data table A and discovers that the upstream and downstream table structures are inconsistent—there is an added field. In this case, the data of field m can be added to the record information corresponding to data table A and filled with the default value "pad." For example, if the SQL statement is: insert into A(x,y,z)values(1,1,1), the SQL statement can be converted to: insertinto A(x,y,z,m)values(1,1,1,'pad'). Synchronization component D' also follows the same processing logic as synchronization component C' and will not be repeated here.

[0097] In this solution, after the second synchronization component implements the update of the table structure of the second data table in the distributed database of the third data table in the second stand-alone database according to the grayscale release instruction, the first synchronization component can modify the first record information in the first log file corresponding to the first stand-alone database based on the table structure mapping relationship between the first data table and the second data table in the first stand-alone database, and generate second record information, so that by replaying the second record information in the distributed database, the table structure of the first data table can be updated in the distributed database in the same manner as the third data table. That is, in the grayscale release technology provided by this solution, when the table structure change is executed on one or several stand-alone databases upstream, the table structure change can be executed in the distributed database at the same time. In addition, the table structure of the isomorphic data table in the distributed database of other stand-alone databases upstream that are not included in the scope of this grayscale change can be changed in the same manner, thereby avoiding the problem described in the background technology that in the grayscale release scenario, the data synchronization behavior of the stand-alone database that has undergone table structure change needs to be interrupted, thereby improving the efficiency of data synchronization.

[0098] The following will explain some of the above steps in detail with examples.

[0099] In one implementation of the above-mentioned step 302, the first synchronization component modifies the first record information in the first log file of the first stand-alone database according to the table structure mapping relationship to obtain second record information, including: if in the table structure mapping relationship, the second data table has an added first field relative to the first data table, then the first synchronization component adds the setting information corresponding to the first field to the first record information to obtain the second record information; if in the table structure mapping relationship, the second data table has a second field and a third field whose field positions are changed relative to the first data table, then the first synchronization component adjusts the position of the information corresponding to the second field and the information corresponding to the third field in the first record information to obtain the second record information; if in the table structure mapping relationship, the second data table has a reduced fourth field relative to the first data table, then the first synchronization component deletes the information corresponding to the fourth field in the first record information to obtain the second record information.

[0100] In the above example, the upstream standalone databases B, C, and D each have an isomorphic data table A, and table A in standalone database B is used as the test object for this grayscale release. During this grayscale release, the types of table structure changes performed on table A in standalone database B can be categorized into three types: adding fields, reducing fields, and adjusting the position of existing fields compared to the isomorphic data tables in the distributed database. The following discusses the grayscale release instructions for each of these three types:

[0101] In case 1, the grayscale release instruction is to add a field.

[0102] For details, please refer to the above examples and will not be repeated here.

[0103] In case 2, the grayscale release instruction this time is to reduce the number of fields.

[0104] Then, after synchronization component B' deletes one or several user-specified fields from data table A in stand-alone database B, it will record the deleted fields in the record information and delete the same fields from the distributed database's data table that is isomorphic to itself (stand-alone database B). Then, after synchronization component C' parses the record information corresponding to data table A and finds that the upstream and downstream table structures are inconsistent—there is a reduction in fields—synchronization component C' will delete the same fields from the distributed database's data table that is isomorphic to itself (stand-alone database C). Synchronization component D' also follows the same processing logic as synchronization component C' and will not be repeated here.

[0105] In case 3, the grayscale release instruction is to adjust the position of the existing fields.

[0106] Since the log file corresponding to the MySQL database, namely binlog, generally only records the data changes of a certain table in the order of fields, but does not record the corresponding field names. For example:

[0107] When a piece of data is written to table A in the upstream stand-alone database B, the SQL statement may be: insert intoA values('123','xxxx','yyy'). The binlog will display the following:

[0108] Column 1 123

[0109] Column 2 xxxx

[0110] Column 3 yyy

[0111] When a piece of data is written to table A in the upstream stand-alone database C, the SQL statement may be: insert intoA values('456,'zzzz,'kkkkk), then the binlog will display the following:

[0112] Column 1 456

[0113] Column 2zzzz

[0114] Column 3 kkkkk

[0115] Assuming that this grayscale release process does not involve changes to the table structure of the downstream distributed database, then when synchronizing data, you only need to write the above two records directly to the downstream.

[0116] However, if this grayscale release requires adjusting the field order of table A in the upstream stand-alone database B, for example, swapping the order of columns 2 and 3 in table A in the upstream stand-alone database B, then:

[0117] When a piece of data is written to table A in the upstream stand-alone database B, the SQL statement may be: insert intoA values('123','xxxx','yyy'). The binlog will display the following:

[0118] Column 1 123

[0119] Column 2 yyy #(original column 3)

[0120] Column 3 xxxx #(original column 2)

[0121] When a piece of data is written to table A in the upstream stand-alone database C, the SQL statement may be: insert intoA values('456,'zzzz,'kkkkk), then the binlog will display the following:

[0122] Column 1 456

[0123] Column 2zzzz

[0124] Column 3 kkkkk

[0125] At this time, because the downstream distributed database and the upstream stand-alone database B still maintain the original order of columns 1, 2, and 3 for the isomorphic data table - data table A, if the synchronization component B' still writes to the downstream table according to the original order, confusion will occur, that is, the data in upstream column 3 will be written to downstream column 2, and the data in upstream column 2 will be written to downstream column 3. To this end, this solution maintains a mapping relationship between the upstream and downstream fields, for example, the maintenance relationship can be stored in a local file. For example, the maintenance relationship can be:

[0126] Initial state:

[0127] Relationship 1 (field mapping relationship between upstream stand-alone database B and downstream distributed database regarding isomorphic data table - data table A):

[0128] Upstream column 1----Downstream column 1

[0129] Upstream column 2----Downstream column 2

[0130] Upstream row 3----Downstream row 3

[0131] Relationship 2 (the field mapping relationship between the upstream stand-alone database C and the downstream distributed database regarding the isomorphic data table - data table A):

[0132] Upstream column 1----Downstream column 1

[0133] Upstream column 2----Downstream column 2

[0134] Upstream row 3----Downstream row 3

[0135] Then, if the order of data table A in the upstream stand-alone database B is adjusted, relationship 1 can be modified at the same time:

[0136] Relationship 1 (field mapping relationship between upstream stand-alone database B and downstream distributed database regarding isomorphic data table - data table A):

[0137] Upstream column 1----Downstream column 1

[0138] Upstream column 3----Downstream column 2

[0139] Upstream column 2----Downstream column 3

[0140] Relationship 2 (the field mapping relationship between the upstream stand-alone database C and the downstream distributed database regarding the isomorphic data table - data table A):

[0141] Upstream column 1----Downstream column 1

[0142] Upstream column 2----Downstream column 2

[0143] Upstream row 3----Downstream row 3

[0144] With this mapping relationship maintained, when synchronization component B' changes the table structure of the downstream distributed database, the operation information on data table A will be recorded in the binlog file corresponding to stand-alone database B and the record information corresponding to data table A. After synchronization component C' parses the binlog, it can find the corresponding field based on the mapping relationship and write data, thus achieving accurate data synchronization.

[0145] In one implementation of the above-mentioned step 301, before the first synchronization component obtains the table structure mapping relationship between the first data table in the first stand-alone database and the second data table in the distributed database, it also includes: the first synchronization component checks the data object according to the check script; the check script includes multiple check items, and each check item is used to determine whether data synchronization operation can be performed on the data object; the first synchronization component determines that the data object meets the requirements of each check item.

[0146] The previous examples discussed methods for improving data synchronization efficiency. However, before implementing data synchronization, it is necessary to check the homogeneous data tables of the upstream standalone database and the downstream distributed database for the table structure changes to confirm whether data synchronization is possible. Because distributed databases may be incompatible with or have restrictions on some features, checking homogeneous data tables before executing data synchronization is the basis for achieving data synchronization. If incompatibilities are detected, adaptive modifications must be made before data synchronization is implemented.

[0147] Currently, checking isomorphic tables in upstream standalone databases and downstream distributed databases is done manually. This requires checking at least the following: 1. The table must have a primary key; 2. The table cannot have foreign keys; 3. The table cannot have triggers; 4. The table cannot have views; 5. The table cannot have functions; 6. The table cannot have partitions; and 7. The table character set must be utf8 or utfmb. Clearly, manually checking all of these items is extremely time-consuming and labor-intensive.

[0148] To this end, the embodiment of the present application can implement a quick check of isomorphic data tables before data synchronization by performing configurable management on the above-mentioned manual inspection solution.

[0149] The specific implementation method may be:

[0150] The check script requires a configuration file as input, which includes the following configuration items:

[0151]

[0152] check_scripts_dir indicates the directory where the check scripts are located; check_items indicates the specific check script. To add a new check item, simply place the check script in the check_scripts_dir directory and configure the script name in check_items.

[0153] Expect is 0, which means that if the script returns a value of 0, the check passes, and if the return value is not 0, the check fails. For example, it can be expressed in pseudo code:

[0154] For example, for item1:check_primary.sh script:

[0155] If check primary key yes (if the check table has a primary key):

[0156] Return 0

[0157] If check primary key no (if the check table does not have a primary key):

[0158] Return 1

[0159] If check meet other error(If you encounter other errors):

[0160] Return 1

[0161] You can see that the above script only returns two values: 0 and 1.

[0162] The above configuration-based checking of homogeneous data tables can have at least the following two benefits:

[0163] First, it can achieve universality. Different distributed databases may have different check items, so you can flexibly configure the check script to the configuration file and place the check script in the check directory according to the actual situation.

[0164] Second, the inspection can be automated, and there is no need to manually check each inspection item one by one.

[0165] against Figure 1In the architecture shown, during data synchronization, if one of the synchronization components experiences an exception or its server crashes, the data synchronization process between the stand-alone database and the distributed database, which is managed by that synchronization component, will terminate abnormally, significantly affecting the efficiency of data synchronization. This adverse effect can even cause business interruption if multiple synchronization components fail simultaneously.

[0166] To this end, the embodiments of the present application may provide the following solutions:

[0167] The first synchronization component sends registration information to the monitoring module; the first synchronization component receives the first information sent by the monitoring module; the first information is used to instruct the first synchronization component to feedback the survival status of the first synchronization component to the monitoring module; the first synchronization component sends second information to the monitoring module; the second information is used to instruct the monitoring module to determine whether to replace the first synchronization component.

[0168] As an example, etcd can be used as a monitoring module (detector) in the embodiments of this application. etcd is a highly available open source component, so the monitoring module itself has high availability characteristics. Generally, at least three nodes are required, and the failure of any node will not affect the normal operation of the monitoring module. The working principle of etcd's underlying raft algorithm to ensure high availability is as follows:

[0169] Each etcd cluster consists of three members, each of which is an independently running etcd instance. Under normal operation, one member in the cluster serves as the leader, and the remaining members are followers. The leader synchronizes logs with the followers to ensure that each member has a copy of the data. The leader also periodically sends heartbeat messages to all members. If a follower does not receive a heartbeat within the specified time, the cluster will re-elect a new leader to ensure high availability.

[0170] refer to Figure 4 , which is a working principle diagram of a monitoring module (detector) provided in an embodiment of the present application. The detector is used to detect whether a working synchronization component is alive. If it is not alive, an idle synchronization component will be selected to replace it, thereby ensuring the high availability of the synchronization component.

[0171] exist Figure 4 In the example, the same component 1 is used as the first synchronization component for explanation:

[0172] Step 1: Synchronize component 1 to register with etcd.

[0173] In step 2, synchronization component 1 receives the registration success information sent by etcd, and at the same time receives the timeout information sent by etcd. For example, the timeout period can be set to 60 seconds. The timeout information is used to instruct synchronization component 1 to send a heartbeat to itself (referring to etcd) at most once within 60 seconds, so that etcd can determine whether synchronization component 1 needs to be replaced by other idle synchronization components.

[0174] Step 3: Synchronization component 1 continues to report its own liveness to etcd.

[0175] In step 4, synchronization component 1 is replaced by another idle synchronization component selected by etcd because it fails to send a heartbeat to etcd for more than 60 seconds or etcd receives a non-survival heartbeat within the timeout period.

[0176] refer to Figure 5 , is a data synchronization system provided in an embodiment of the present application. Figure 1 Come and see, Figure 5 A new monitoring module (detector) and at least one idle synchronization component have been added. Figure 5 This solution divides synchronization components into two roles: active and idle. Therefore, if a working synchronization component fails due to a process anomaly or the server it resides on crashes, the monitoring module (detector) will select an idle component to replace the abnormal one. This approach ensures high availability and self-healing of synchronization components.

[0177] When the monitoring module (detector) replaces a faulty synchronization component with an idle one, synchronization must resume from the previous synchronization node, the faulty POS point. Therefore, during data synchronization, the monitoring module (detector) records each POS point that has undergone data synchronization in a downstream distributed database. Once the replacement synchronization component is activated, it can read the POS point from the downstream distributed database and resume synchronization.

[0178] During data synchronization, the synchronization component needs to parse the binlog file corresponding to the upstream stand-alone database and execute the parsed SQL statements in the downstream distributed database. During this process, if the monitoring module detects an abnormality in the synchronization component, it will select an idle synchronization component from the idle synchronization component cluster to replace it. At the same time, the monitoring module reads the POS point from the downstream distributed database, and then the idle synchronization component that has been upgraded to the working state will continue synchronization from this POS point. Figure 6 , is a schematic diagram of a principle for replacing a synchronization component provided by an embodiment of the present application, which includes the following steps:

[0179] Step 1: The monitoring module detects that the synchronization component 1 of the stand-alone database 1 serving the upstream is abnormal.

[0180] Step 2: The monitoring module confirms that the data synchronization process managed by the synchronization component 1 is interrupted.

[0181] Step 3: The monitoring module selects a replacement for the synchronization component 1 from the idle synchronization component cluster based on a preset rule, and the replacement is set to be the idle synchronization component 1.

[0182] Step 4: The monitoring module promotes the idle synchronization component 1 to a working synchronization component.

[0183] Step 5: The monitoring module reads the POS point of the synchronization component 1 from the downstream distributed database as POS X.

[0184] Step 6: The monitoring module sends a message to the idle synchronization component 1, where the message is used to instruct the idle synchronization component 1 to continue synchronization from pos X.

[0185] Step 7: The idle synchronization component 1 that has been upgraded to the working state parses from pos X and plays back the SQL statement obtained by parsing in the downstream distributed database.

[0186] In one implementation of the above step 303, after the first synchronization component synchronizes the data object to the second data table according to the second record information, it also includes: for any stand-alone database in each stand-alone database, the first synchronization component determines according to the verification script that the synchronization delay between the distributed database and the stand-alone database meets the preset delay requirement; the first synchronization component determines according to the verification script that the distributed database and the stand-alone database have completed data synchronization; the first synchronization component switches the database connected to the application from each stand-alone database to the distributed data based on the application switching script.

[0187] In current data synchronization technology, after ensuring upstream and downstream data consistency, a time point needs to be selected to switch the database connected to the application from the upstream stand-alone database to the downstream distributed database. Application switching generally includes the following steps:

[0188] Step 1: Stop the application.

[0189] Step 2: The user manually confirms that the downstream data has caught up with the upstream data.

[0190] Step 3: Change the application's database configuration from a stand-alone database to a distributed database and start the application.

[0191] However, through analysis of the above steps, we can find that the current application switching technology has at least the following two shortcomings:

[0192] Disadvantage 1: Because data synchronization is practically real-time, SQL statements for additions, deletions, and modifications in the upstream database take time to be transmitted to the downstream database for execution, typically around a second or even longer. To ensure complete consistency between upstream and downstream data, the application typically needs to be shut down before the switchover. This downtime significantly impacts the user experience in scenarios requiring 24 / 7 uninterrupted service, such as in the financial sector.

[0193] Disadvantage 2: Manual confirmation is required to ensure that downstream data matches upstream data. This manual confirmation is obviously time-consuming and labor-intensive.

[0194] To this end, the embodiments of the present application improve the above-mentioned technical solution for application switching, including using an automated method to confirm whether the downstream data is on par with the upstream data, thereby improving the efficiency of checking the consistency of upstream and downstream data, and after confirming that the upstream and downstream data are synchronized, the database connected to the application can be quickly switched from the upstream stand-alone database to the downstream distributed database by running the application switching script.

[0195] In order to address the above-mentioned shortcoming 2, the embodiment of the present application includes two aspects in the process of using an automated method to confirm whether the downstream data has caught up with the upstream data:

[0196] The first aspect is to confirm whether the time it takes for each piece of data to be synchronized from the upstream stand-alone database to the downstream distributed database meets the preset latency.

[0197] The second aspect: confirm that the downstream distributed database has completed the synchronization of all data with the upstream stand-alone database.

[0198] With respect to the first aspect above, the embodiments of the present application may be implemented in the following manner:

[0199] For any stand-alone database among the stand-alone databases, the first synchronization component determines, according to the verification script, that the synchronization delay between the distributed database and the stand-alone database meets the preset delay requirement, including: the first synchronization component writes a first timestamp to the stand-alone database according to the verification script; the first timestamp is used to represent the moment of writing to the stand-alone database; if the first synchronization component determines that the difference between the second timestamp and the first timestamp meets the set threshold, then it is determined that the synchronization delay between the distributed database and the stand-alone database meets the preset delay requirement; the second timestamp is the moment when the first timestamp is synchronized to the distributed database as a data object.

[0200] Regarding the second aspect above, the embodiments of the present application can be implemented in the following manner:

[0201] The first synchronization component determines that the distributed database and the stand-alone database have completed data synchronization according to the verification script, including: the first synchronization component sets the stand-alone database to read-only according to the verification script; the first synchronization component determines that the distributed database and the stand-alone database have completed data synchronization when the data synchronization record value of the distributed database is consistent with the data synchronization record value of the stand-alone database.

[0202] For example, the following example illustrates the automated method provided by the present invention to determine whether downstream data has caught up with upstream data:

[0203] In step 1, a timestamp of the current time is written to the upstream standalone database every second. Assume it is ts. For example, if ts is 2020-09-28 16:07:30, the binlog will be transmitted to the downstream distributed database and the timestamp will also be written to the downstream distributed database.

[0204] Step 2: Continuously query the current time in the downstream distributed database every 1 second and compare it with ts. For example, if the current time queried is 2020-09-28 16:07:32, it can be seen that the upstream and downstream delay is 2 seconds.

[0205] In step 3, based on a pre-configured threshold, for example, if the upstream and downstream delay is no more than 10 seconds, the delay is considered to be minimal and step 4 can be continued. Otherwise, steps 1 and 2 are repeated until the delay is less than 10 seconds. The threshold of 10 seconds is the preset delay requirement and can be set by those skilled in the art based on actual needs.

[0206] In step 4, the upstream stand-alone database is set to read-only. This means that the stand-alone database cannot be modified. If the application continues to write data to the stand-alone database, it will receive a read-only error. At the same time, an automated verification script continuously checks the GTID information of the upstream and downstream databases every 1 second. If the GTIDs are equal, the downstream data is considered to have caught up with the upstream data.

[0207] In this embodiment, a timeout protection process can also be configured. Specifically, during the process of the automated verification script checking the equality of the upstream and downstream GTID information, a preset timeout period can be set. For example, if the upstream and downstream GTIDs are still unequal after 20 seconds, the switch process can be considered to have taken too long, and the upstream stand-alone database will be reset to readable and writable, terminating the application switch process. The timeout period of 20 seconds can be customized by those skilled in the art based on actual needs.

[0208] During the application switching process, the issue of confirming whether the downstream data has caught up with the upstream data can be addressed through the automated method provided by the embodiments of the present application. This allows for quick confirmation of data synchronization between upstream and downstream, saving time and effort. Furthermore, based on the automated technology provided by the embodiments of the present application that allows for quick confirmation of data synchronization between upstream and downstream, the embodiments of the present application can also implement switching of the database connected to the application from an upstream stand-alone database to a downstream distributed database based on the application switching script.

[0209] In certain embodiments of the present application, the first synchronization component switches the database connected to the application from each stand-alone database to the distributed data based on the application switching script, including: the first synchronization component sets the access IP weight of each stand-alone database set in the proxy gateway from a first value to a second value based on the application switching script, and sets the access IP weight of the distributed database set in the proxy gateway from the second value to the first value; wherein the first value is 100% and the second value is 0.

[0210] For example, in an embodiment of the present application, proxy gateway technology can be used to add the access IP addresses of the downstream distributed database and the upstream stand-alone database to one VIP.

[0211] refer to Figure 7, which is a database connection status diagram before application switching provided by an embodiment of the present application. Before application switching, the weight of the access IP of the upstream stand-alone database can be adjusted to 100. If the weight is 100, the PROXY will send all application requests to the access IP and adjust the weight of the access IP of the downstream distributed database to 0. A weight of 0 means that the corresponding IP is removed from the forwarding target at the network layer. In this way, the application can connect to the IP1 of the upstream stand-alone database through VIP IP1.

[0212] refer to Figure 8 , which is a database connection status diagram during application switching provided by an embodiment of the present application. Among them, based on the automation technology provided by the embodiment of the present application that can quickly confirm that upstream and downstream have completed data synchronization, when it is confirmed that upstream and downstream have completed data synchronization, the weight of the access IP of the upstream stand-alone database on the PROXY is set to 0 by running the automated application switching script, and the weight of the access IP of the downstream distributed database is set to 100. At this time, all application requests will be written to the downstream distributed database, and the switching is completed. In this way, the switching can be completed without stopping the application.

[0213] Based on the same concept, the embodiment of the present application provides a data synchronization device suitable for synchronizing data from at least one stand-alone database to a distributed database through respective synchronization components; Figure 9 As shown, the device includes a table structure mapping relationship acquisition unit 901, a record information update unit 902 and a synchronization unit 903;

[0214] The table structure mapping relationship acquisition unit 901 is configured to acquire a table structure mapping relationship between a first data table in a first stand-alone database and a second data table in the distributed database.

[0215] The record information updating unit 902 is used to modify the first record information in the first log file of the first stand-alone database according to the table structure mapping relationship to obtain second record information; the first record information records the data objects in the first data table to be synchronized to the second data table.

[0216] The synchronization unit 903 is configured to synchronize the data object to the second data table according to the second record information.

[0217] Further, for the device, the record information update unit 902 is specifically used to: if in the table structure mapping relationship, the second data table has an added first field relative to the first data table, then the first synchronization component adds the setting information corresponding to the first field in the first record information to obtain the second record information; if in the table structure mapping relationship, the second data table has a second field and a third field whose field positions are changed relative to the first data table, then the first synchronization component adjusts the position of the information corresponding to the second field and the information corresponding to the third field in the first record information to obtain the second record information; if in the table structure mapping relationship, the second data table has a reduced fourth field relative to the first data table, then the first synchronization component deletes the information corresponding to the fourth field in the first record information to obtain the second record information.

[0218] Furthermore, the device also includes a data synchronization confirmation unit 904; the data synchronization confirmation unit 904 is used to: check the data object according to the check script; the check script includes multiple check items, each check item is used to determine whether data synchronization operation can be performed on the data object; determine whether the data object meets the requirements of each check item.

[0219] Furthermore, the device also includes a transceiver unit 905; the transceiver unit 905 is used to: send registration information to the monitoring module; receive first information sent by the monitoring module; the first information is used to instruct the first synchronization component to feedback the survival status of the first synchronization component to the monitoring module; send second information to the monitoring module; the second information is used to instruct the monitoring module to determine whether to replace the first synchronization component.

[0220] Furthermore, the device also includes an application switching unit 906; the application switching unit 906 is used to: for any stand-alone database among the stand-alone databases, determine according to the verification script whether the synchronization delay between the distributed database and the stand-alone database meets the preset delay requirement; determine according to the verification script that the distributed database and the stand-alone database have completed data synchronization; based on the application switching script, switch the database connected to the application from each stand-alone database to the distributed database.

[0221] Further, for the device, an application switching unit 906 is specifically used to: write a first timestamp to the stand-alone database according to a verification script; the first timestamp is used to represent the moment of writing to the stand-alone database; if it is determined that the difference between the second timestamp and the first timestamp meets the set threshold, it is determined that the synchronization delay between the distributed database and the stand-alone database meets the preset delay requirement; the second timestamp is the moment when the first timestamp is synchronized to the distributed database as a data object; according to the verification script, set the stand-alone database to read-only; when the data synchronization record value of the distributed database is consistent with the data synchronization record value of the stand-alone database, it is determined that the distributed database and the stand-alone database have completed data synchronization.

[0222] Furthermore, for the device, the application switching unit 906 is specifically used to: based on the application switching script, set the access IP weight of each stand-alone database set in the proxy gateway from the first value to the second value, and set the access IP weight of the distributed database set in the proxy gateway from the second value to the first value; wherein, the first value is 100% and the second value is 0.

[0223] Furthermore, the device also includes a second synchronization component, which includes a grayscale release instruction receiving unit and a data table update unit; the grayscale release instruction receiving unit is used to receive the grayscale release instruction; the second synchronization component is used to synchronize data between the third data table and the second data table in the second stand-alone database, and the third data table and the first data table are isomorphic data tables; the data table update unit is used to update the second data table according to the grayscale release instruction.

[0224] The present application also provides a computing device, which may be a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant (PDA), etc. The computing device may include a central processing unit (CPU), a memory, input / output devices, etc. The input device may include a keyboard, a mouse, a touch screen, etc. The output device may include a display device, such as a liquid crystal display (LCD), a cathode ray tube (CRT), etc.

[0225] The memory may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor with program instructions and data stored in the memory. In an embodiment of the present application, the memory may be used to store program instructions of the data synchronization method;

[0226] The processor is used to call the program instructions stored in the memory and execute the data synchronization method according to the obtained program.

[0227] like Figure 10 FIG. 1 is a schematic diagram of a computing device provided in an embodiment of the present application, wherein the computing device includes:

[0228] Processor 1001, memory 1002, transceiver 1003, bus interface 1004; wherein the processor 1001, memory 1002 and transceiver 1003 are connected via bus 1005;

[0229] The processor 1001 is configured to read the program in the memory 1002 and execute the above-mentioned data synchronization method;

[0230] Processor 1001 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. It may also be a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0231] The memory 1002 is used to store one or more executable programs and can store data used by the processor 1001 when performing operations.

[0232] Specifically, the program may include program code, which includes computer operating instructions. Memory 1002 may include volatile memory, such as random-access memory (RAM); memory 1002 may also include non-volatile memory, such as flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); and memory 1002 may also include a combination of the aforementioned types of memory.

[0233] The memory 1002 stores the following elements, executable modules or data structures, or a subset or an extension thereof:

[0234] Operation instructions: include various operation instructions, used to implement various operations.

[0235] Operating system: includes various system programs used to implement various basic services and process hardware-based tasks.

[0236] The bus 1005 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0237] The bus interface 1004 may be a wired communication access port, a wireless bus interface, or a combination thereof. The wired bus interface may be, for example, an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless bus interface may be a WLAN interface.

[0238] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute a data synchronization method.

[0239] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0240] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0241] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0242] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0243] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0244] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A data synchronization method, characterized in that: Applicable to synchronizing data from at least one stand-alone database to a distributed database through respective synchronization components; the method includes: The first synchronization component obtains a table structure mapping relationship between the first data table in the first stand-alone database and the second data table in the distributed database; The first synchronization component modifies the first record information in the first log file of the first stand-alone database according to the table structure mapping relationship to obtain second record information; the first record information records the data objects in the first data table to be synchronized to the second data table; The first synchronization component synchronizes the data object to the second data table according to the second record information; The first synchronization component modifies the first record information in the first log file of the first stand-alone database according to the table structure mapping relationship to obtain second record information, including: If, in the table structure mapping relationship, the second data table has an additional first field relative to the first data table, the first synchronization component adds setting information corresponding to the first field to the first record information to obtain the second record information; If, in the table structure mapping relationship, the second data table has a second field and a third field whose field positions are changed relative to those in the first data table, the first synchronization component adjusts the positions of the information corresponding to the second field and the information corresponding to the third field in the first record information to obtain the second record information; If, in the table structure mapping relationship, the second data table has a reduced fourth field compared to the first data table, the first synchronization component deletes information corresponding to the fourth field in the first record information to obtain the second record information.

2. The method according to claim 1, wherein Before the first synchronization component obtains the table structure mapping relationship between the first data table in the first stand-alone database and the second data table in the distributed database, the method further includes: The first synchronization component checks the data object according to a check script; the check script includes a plurality of check items, each check item is used to determine whether a data synchronization operation can be performed on the data object; The first synchronization component determines that the data object meets the requirements of each check item.

3. The method according to claim 1, wherein The method further comprises: The first synchronization component sends registration information to the monitoring module; The first synchronization component receives the first information sent by the monitoring module; the first information is used to instruct the first synchronization component to feedback the survival status of the first synchronization component to the monitoring module; The first synchronization component sends second information to the monitoring module; the second information is used to instruct the monitoring module to determine whether to replace the first synchronization component.

4. The method according to claim 1, wherein After the first synchronization component synchronizes the data object to the second data table according to the second record information, the method further includes: For any stand-alone database among the stand-alone databases, the first synchronization component determines, according to a verification script, that a synchronization delay between the distributed database and the stand-alone database meets a preset delay requirement; The first synchronization component determines, according to the verification script, that data synchronization between the distributed database and the stand-alone database has been completed; The first synchronization component switches the database connected to the application from each stand-alone database to the distributed database based on the application switching script.

5. The method according to claim 4, wherein For any stand-alone database among the stand-alone databases, the first synchronization component determines, according to the verification script, that the synchronization delay between the distributed database and the stand-alone database meets the preset delay requirement, including: The first synchronization component writes a first timestamp to the stand-alone database according to the verification script; the first timestamp is used to represent the time when the data is written into the stand-alone database; If the first synchronization component determines that the difference between the second timestamp and the first timestamp meets the set threshold, it is determined that the synchronization delay between the distributed database and the stand-alone database meets the preset delay requirement; the second timestamp is the time when the first timestamp is synchronized to the distributed database as a data object; The first synchronization component determines, according to the verification script, that data synchronization between the distributed database and the stand-alone database has been completed, including: The first synchronization component sets the stand-alone database to read-only according to the verification script; The first synchronization component determines that data synchronization between the distributed database and the stand-alone database has been completed when the data synchronization record value of the distributed database is consistent with the data synchronization record value of the stand-alone database.

6. The method according to claim 4, wherein The first synchronization component switches the database connected to the application from each stand-alone database to the distributed database based on the application switching script, including: The first synchronization component sets the access IP weight of each stand-alone database set in the proxy gateway from a first value to a second value based on the application switching script, and sets the access IP weight of the distributed database set in the proxy gateway from the second value to the first value; The first value is 100% and the second value is 0.

7. The method according to any one of claims 1 to 6, wherein: Before the first synchronization component obtains the table structure mapping relationship between the first data table in the first stand-alone database and the second data table in the distributed database, the method further includes: A second synchronization component receives a grayscale release instruction; the second synchronization component is used to synchronize data between a third data table and the second data table in a second stand-alone database, where the third data table and the first data table are isomorphic data tables; The second synchronization component updates the second data table according to the grayscale release instruction.

8. A data synchronization device, characterized in that: Applicable to at least one stand-alone database synchronizing data to a distributed database through its own synchronization components; The device comprises: A table structure mapping relationship acquisition unit, configured to acquire a table structure mapping relationship between a first data table in a first stand-alone database and a second data table in the distributed database; a record information updating unit, configured to modify first record information in a first log file of the first stand-alone database according to the table structure mapping relationship to obtain second record information; the first record information records data objects in the first data table to be synchronized to the second data table; a synchronization unit, configured to synchronize the data object to the second data table according to the second record information; The record information updating unit is specifically used to: if in the table structure mapping relationship, the second data table has an added first field relative to the first data table, then the first synchronization component adds the setting information corresponding to the first field in the first record information to obtain the second record information; if in the table structure mapping relationship, the second data table has a second field and a third field whose field positions are changed relative to the first data table, then the first synchronization component adjusts the position of the information corresponding to the second field and the information corresponding to the third field in the first record information to obtain the second record information; if in the table structure mapping relationship, the second data table has a reduced fourth field relative to the first data table, then the first synchronization component deletes the information corresponding to the fourth field in the first record information to obtain the second record information.

9. A computer-readable storage medium, characterized in that The storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images