System and method for realizing data synchronization processing based on relational database and distributed database, processor and readable storage medium

Abstract

The invention relates to a system for realizing data synchronization processing based on a relational database and a distributed database, comprising a webpage front-end client through which a synchronization task is operated; the relational database is used for storing and managing various clearing business data; the distributed database is used for storing and managing credential clearing business data; the business middleware is used for receiving an operation task of the webpage front-end client, preparing a synchronization temporary table according to configuration, generating a synchronization instruction and sending the synchronization instruction to the synchronization execution machine; and the synchronous execution unit comprises a plurality of synchronous execution machines and is used for receiving the synchronous task parameters, converting the synchronous task parameters into input configuration files and starting a synchronous program. The system and method for realizing data synchronization processing based on the relational database and the distributed database, the processor and the computer readable storage medium are flexible in configuration, high in performance and easy to expand, meet the data synchronization requirement of mass data exchange between the databases, provide matched backup and check functions, and improve the data synchronization efficiency. And the security and the accuracy of the synchronous data are ensured.

Description

Technical Field

[0001] The present invention relates to the field of the financial industry, and particularly to the field of data synchronization. Specifically, it refers to a system, method, processor and computer-readable storage medium for implementing data synchronization processing based on a relational database and a distributed database. Background Art

[0002] In recent years, the localization and security controllability strategy of the IT systems in the financial industry has been accelerating. Especially the clearing system, which is the core business function of securities firms, has become the focus of localization transformation. The sub-database and sub-table strategy of traditional commercial databases faces challenges in processing a large amount of customer data, while domestic distributed databases can meet the requirements in terms of reliability, performance, ease of maintenance, etc. However, the following problems need to be addressed during the transformation process: (1) design challenges brought by inconsistent database types; (2) large and complex data volume; (3) high requirements for timeliness, stability and accuracy.

[0003] Please refer to Figure 1 , the existing clearing system is gradually transitioning to localization. One is the production system that is currently in operation and uses a relational database. The other is the Xinchuang centralized clearing platform that uses a distributed relational database. The relational database is designed with a sub-database and sub-table structure, and all customer data is scattered to each business node (sub-database) of centralized clearing according to certain rules. Each business table in the database is divided into several tables with the same structure (sub-tables), and the node number is used as the suffix of the sub-table name. All sub-tables are scattered in several sub-databases. The sub-database and sub-table design can avoid the problems of insufficient processing capacity of a single table in a single-body database and slow high-concurrency response, and can easily expand the system capacity by adding more database instances. The architecture of the distributed database performs sharding at the bottom layer, and the management and scheduling of sharded data are managed by the database in the cluster, which is transparent to the application layer, and the application layer does not need to manually specify sub-databases and sub-tables according to rules. It is now required to synchronize the data in the relational database table to the table in the distributed database before daily clearing, and then reverse-synchronize the data in the distributed database table back to the relational database after clearing.

[0004] However, since the application system designs the relational database with a sub-database and sub-table structure, it brings difficulties to data migration. It cannot establish a one-to-one correspondence with the tables in the distributed database, and existing data synchronization tools cannot be directly used. Summary of the Invention

[0005] The object of the present invention is to overcome the above-mentioned disadvantages of the prior art, and provide a system, method, processor and computer-readable storage medium for implementing data synchronization processing based on a relational database and a distributed database, which meet flexible configuration, high performance and easy expansion.

[0006] To achieve the above object, the system, method, processor and computer-readable storage medium for realizing data synchronization processing based on a relational database and a distributed database of the present invention are as follows:

[0007] The system for realizing data synchronization processing based on a relational database and a distributed database, the main features of which are that the system includes:

[0008] A web front-end client for operating synchronization tasks through the web front-end client;

[0009] A relational database for storing and managing various types of clearing business data;

[0010] A distributed database for storing and managing Xinchuang clearing business data;

[0011] A service middleware, connected to the web front-end client, the relational database and the distributed database, for receiving the operation tasks of the web front-end client, preparing a synchronization temporary table according to pre-configurations, generating a synchronization instruction and sending it to the synchronization executor, checking the synchronization data, and overwriting the synchronization data;

[0012] A synchronization executor group, including multiple synchronization executors, connected to the web front-end client, the relational database, the distributed database and the service middleware, for receiving synchronization task parameters, converting them into input configuration files, starting a synchronization program, and reading data from the data source end and writing it to the data destination end.

[0013] Preferably, the relational database and the distributed database synchronize their respective table data to each other at different time nodes, synchronizing from either end to the other end.

[0014] Preferably, in the direction of synchronizing from the relational database to the distributed database, the system automatically calculates and generates the corresponding relationship between the sub-tables in the relational database and the sharding keys in the distributed database according to rules, that is, 10 sub-tables of the relational database correspond to 1 sharding key of the distributed database, and the 10 sub-tables of the relational database and 1 sharding key of the distributed database with the corresponding relationship are used as a group of synchronization tasks.

[0015] Preferably, in the direction of synchronizing from the distributed database to the relational database, the distributed database has an additional table associated with the synchronization table for calculating the sub-table values of the current data in the relational database.

[0016] Preferably, the synchronization executor synchronizes between the two databases, and has the following synchronization operation modes:

[0017] Full table synchronization mode, synchronizing all the data in the source table to the target table, and cleaning the data in the target table in advance;

[0018] Incremental synchronization mode, according to the timestamp or other fields that can identify incremental data, filter out the data that needs to be synchronized;

[0019] Query synchronization mode, configure the query SQL statement, and synchronize the data queried by this statement to the target table.

[0020] Preferably, the process design of the business middleware for processing data synchronization tasks has the following steps:

[0021] (1) Back up the current data of the target table to be synchronized, and automatically generate the table structure of the synchronization temporary table according to the table structure of the table to be synchronized, as the target table for data synchronization, and store the synchronization data in this temporary table;

[0022] (2) Execute the synchronization task, synchronize the data from the synchronization source end to the temporary table prepared in step (1) at the synchronization destination end. The business middleware prepares the synchronization task parameters sent by the web front-end client, converts them into the required input file format, and sends them to the synchronization executor; after the synchronization task is completed, record the task status and the amount of synchronized data into the scheduling system, and verify whether the number of data in the synchronization temporary table is consistent with the number synchronized by the synchronization executor to ensure the correctness of the synchronization process;

[0023] (3) Execute the verification logic, verify the validity and accuracy of the synchronized data, and ensure the correctness of the data generated at the synchronization source end;

[0024] (4) Overwrite the verified synchronization data in the temporary table to the formal table at the synchronization destination end.

[0025] Preferably, the system dynamically adds or deletes data sources and target tables in the synchronization task parameters, and adds and expands synchronization executors.

[0026] The method for implementing data synchronization processing based on the above system between a relational database and a distributed database, its main feature is that the method includes the following steps:

[0027] (1) The web front-end client initiates a group of synchronization task start instructions;

[0028] (2) The business middleware calculates synchronization parameters according to the incoming parameters of each group of synchronization tasks;

[0029] (3) Allocate execution tasks to the synchronization executor according to the load algorithm, convert the task parameters into the input file format required by the synchronization executor, send them to the synchronization executor, and call the script of the synchronization executor through remote login with the aggregated synchronization parameters;

[0030] (4) The synchronization execution machine invokes the script, replaces the placeholders in the synchronization program input parameter template file with the incoming parameters, and starts the synchronization program to execute the synchronization task.

[0031] The device for realizing data synchronization processing based on a relational database and a distributed database is characterized in that the device comprises:

[0032] A processor configured to execute computer-executable instructions;

[0033] A memory storing one or more computer-executable instructions, and when the computer-executable instructions are executed by the processor, each step of the method for realizing data synchronization processing based on a relational database and a distributed database as described above is realized.

[0034] The processor for realizing data synchronization processing based on a relational database and a distributed database is characterized in that the processor is configured to execute computer-executable instructions, and when the computer-executable instructions are executed by the processor, each step of the method for realizing data synchronization processing based on a relational database and a distributed database as described above is realized.

[0035] The computer-readable storage medium is characterized in that a computer program is stored thereon, and the computer program can be executed by a processor to realize each step of the method for realizing data synchronization processing based on a relational database and a distributed database as described above.

[0036] By adopting the system, method, processor and computer-readable storage medium for realizing data synchronization processing based on a relational database and a distributed database of the present invention, the problem of difficult synchronization caused by different design structures between a sharded and partitioned database and a single-body database is solved. It is characterized by being flexibly configurable, having high performance, being easy to expand, being able to meet the data synchronization requirements for large-volume data exchange between databases, and providing supporting backup and verification functions to ensure the security and accuracy of synchronized data. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the synchronization table relationship between an existing relational database and a distributed database.

[0038] Figure 2 It is a schematic diagram of the system for realizing data synchronization processing based on a relational database and a distributed database of the present invention.

[0039] Figure 3 It is a flowchart of the method for realizing data synchronization processing based on a relational database and a distributed database. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] In order to more clearly describe the technical content of the present invention, the following will be further described in conjunction with specific embodiments.

[0041] The system for realizing data synchronization processing based on a relational database and a distributed database of the present invention includes:

[0042] A web front-end client for operating synchronization tasks through the web front-end client;

[0043] A relational database for storing and managing various types of clearing business data;

[0044] A distributed database for storing and managing Xinchuang clearing business data;

[0045] A business middleware connected to the web front-end client, the relational database, and the distributed database, for receiving operation tasks from the web front-end client, preparing a synchronization temporary table according to pre-configurations, generating a synchronization instruction and sending it to a synchronization executor, checking synchronization data, and overwriting synchronization data;

[0046] A synchronization executor group including multiple synchronization executors, connected to the web front-end client, the relational database, the distributed database, and the business middleware, for receiving synchronization task parameters, converting them into input configuration files, starting a synchronization program, and reading data from a data source end and writing it to a data destination end.

[0047] As a preferred embodiment of the present invention, the relational database and the distributed database synchronize the table data processed by their respective business systems to each other at different time nodes, synchronizing from any one party to the other.

[0048] As a preferred embodiment of the present invention, in the direction of synchronizing from the relational database to the distributed database, the system automatically calculates and generates the corresponding relationship between the sub-tables in the relational database and the sharding keys in the distributed database according to rules, that is, 10 sub-tables in the relational database correspond to one sharding key of one table in the distributed database. The 10 sub-tables in the relational database and one sharding key of one table in the distributed database with the corresponding relationship are used as a group of synchronization tasks, and this mapping relationship can be modified according to actual needs to adapt to specific situations.

[0049] As a preferred embodiment of the present invention, in the direction of synchronizing from the distributed database to the relational database, the distributed database has an additional table associated with the synchronization table for calculating the corresponding sub-table value of the current data's sharding key in the relational database.

[0050] As a preferred embodiment of the present invention, the synchronization executor synchronizes between the two databases and has the following synchronization operation modes:

[0051] Full-table synchronization mode synchronizes all data in the source table to the target table and clears the data in the target table in advance;

[0052] Incremental synchronization mode filters out the data to be synchronized according to the timestamp or other fields that can identify incremental data;

[0053] Query synchronization mode configures a query SQL statement and synchronizes the data retrieved by this statement to the target table.

[0054] As a preferred embodiment of the present invention, the process design of the business middleware for processing data synchronization tasks has the following steps:

[0055] (1) Back up the current data of the table to be synchronized, and automatically generate the table structure of the synchronization temporary table according to the table structure of the table to be synchronized as the target table for data synchronization, and store the synchronization data in this temporary table;

[0056] (2) Execute the synchronization task, synchronize the data at the synchronization source end to the temporary table prepared in step (1) at the synchronization destination end. The business middleware prepares the synchronization task parameters sent by the web front-end client, converts them into the required input file format, and sends them to the synchronization executor; after the synchronization task is completed, record the task status and the amount of synchronized data in the scheduling system, and verify whether the number of data in the synchronization temporary table is consistent with the number synchronized by the synchronization executor to ensure the correctness of the synchronization process;

[0057] (3) Execute the verification logic to verify the validity and accuracy of the synchronized data to ensure the correctness of the data generated at the synchronization source end;

[0058] (4) Overwrite the verified synchronization data in the temporary table to the formal table at the synchronization target end.

[0059] This process makes data synchronization secure. After synchronization, the verification logic can be executed. If problems are found in the data, it can prevent problematic data from directly entering the formal table. If the incorrect synchronized data has been overwritten into the formal table, the data can also be restored from the backup table.

[0060] As a preferred embodiment of the present invention, the system dynamically adds or deletes data sources and target tables in the synchronization task parameters and adds and expands synchronization executors.

[0061] The system according to claim 1 of the present invention implements a method for data synchronization processing based on a relational database and a distributed database, and the method includes the following steps:

[0062] (1) The web front-end client initiates a set of synchronization task start instructions;

[0063] (2) The business middleware calculates synchronization parameters based on the input parameters of each group of synchronization tasks;

[0064] (3) Allocate execution tasks to the synchronization execution machines according to the load algorithm, convert the task parameters into the input file format required by the synchronization execution machines, send them to the synchronization execution machines, and call the summarized synchronization parameters to the scripts of the synchronization execution machines through remote login;

[0065] (4) The synchronization execution machine calls the script, replaces the placeholders in the synchronization program input parameter template file with the input parameters, and starts the synchronization program to execute the synchronization task.

[0066] As a preferred embodiment of the present invention, the method further includes the step of the business middleware backing up the current data of the table to be synchronized before the synchronization task, and specifically includes the following operation steps:

[0067] Generate the table structure of the synchronization temporary table according to the table structure of the table to be synchronized, use it as the target table for data synchronization, store data in the target table, and initialize the status of the synchronization task.

[0068] As a preferred embodiment of the present invention, the method further includes the following steps after (4):

[0069] The business middleware verifies whether the quantity of data in the synchronization temporary table is correct. If the verification is correct, the data in the synchronization temporary table is overwritten into the synchronization target table.

[0070] The device for realizing data synchronization processing based on a relational database and a distributed database of the present invention, wherein the device includes:

[0071] A processor configured to execute computer-executable instructions;

[0072] A memory storing one or more computer-executable instructions, and when the computer-executable instructions are executed by the processor, each step of the method for realizing data synchronization processing based on a relational database and a distributed database as described above is realized.

[0073] The processor for realizing data synchronization processing based on a relational database and a distributed database of the present invention, wherein the processor is configured to execute computer-executable instructions, and when the computer-executable instructions are executed by the processor, each step of the method for realizing data synchronization processing based on a relational database and a distributed database as described above is realized.

[0074] The computer-readable storage medium of the present invention, on which a computer program is stored, and the computer program can be executed by a processor to realize each step of the method for realizing data synchronization processing based on a relational database and a distributed database as described above.

[0075] In the specific implementation of the present invention, it is used to synchronize service data between two sets of securities clearing system databases. Through flexible configuration, it provides the function of synchronizing all or specified data in different directions between the two system databases. Data sources and synchronization execution machines can be dynamically added for expansion, and parameters such as synchronization direction, synchronization table mode, synchronization conditions, and data deletion conditions can be configured. The system components include: a task scheduling component; a service middleware; a synchronization execution machine; a relational database, and a distributed database.

[0076] A data synchronization method, device, equipment, and storage medium based on database sharding and table partitioning design for synchronizing service data between a database with sharding and table partitioning design and a distributed database. The data synchronization system includes:

[0077] A web front-end client through which an operator can operate synchronization tasks;

[0078] A service middleware that is used to receive front-end operation tasks, prepare a synchronization temporary table according to pre-configurations, generate a synchronization instruction and send it to the synchronization execution machine, check synchronization data, overwrite synchronization data, etc.;

[0079] A relational database for storing and managing various types of clearing service data;

[0080] A distributed database, which is used together with the relational database to store and manage Xinchuang clearing service data;

[0081] A synchronization execution machine that is used to receive a synchronization instruction, convert the synchronization instruction into an input configuration file of a synchronization program, start the synchronization program, and enable it to read data from the data source end and write it to the data destination end. The data source end can be a relational database or a distributed database.

[0082] The synchronization system can also back up the existing data in the table to be synchronized before the synchronization task starts and create a synchronization temporary table as the target table for data synchronization to store data.

[0083] The synchronization system converts the synchronization instruction generated according to the configuration into the input file format required by the synchronization program to drive the synchronization program to perform the data synchronization task.

[0084] After the synchronization task is completed, the synchronization system checks whether the actual number of synchronized data is consistent with the reported number of synchronized data. If the synchronization task is successful, it writes the data in the synchronization temporary table into the target table to complete the synchronization.

[0085] The synchronization system provides dynamic configuration of synchronization tasks, dynamic addition of data sources and synchronization execution machines, configuration of synchronization direction, synchronization table mode, synchronization conditions, and data deletion conditions.

[0086] The synchronization system provides data synchronization functions in two directions.

[0087] Before the synchronization is implemented, the synchronization system backs up the original data and prepares a temporary table for the synchronized data to be stored.

[0088] After the synchronization is implemented, the synchronization system verifies the synchronized data to prevent undetected errors in the synchronized data.

[0089] The synchronization system provides full-table synchronization and synchronization modes based on the results of query statements.

[0090] As shown in the appendix Figure 2 A synchronization entry for the web front-end is established to complete the display and operation interface of the synchronization task, allowing operators to perform data synchronization tasks step by step.

[0091] The relational database is designed as a sharded database with sharded tables, a distributed database, and the table data of each needs to be synchronized to the other at different time nodes.

[0092] The business middleware completes the preparatory tasks before synchronization: backing up the current data of the table to be synchronized; generating the table structure of the synchronization temporary table according to the table structure of the table to be synchronized; initializing the status of the synchronization task.

[0093] The business middleware prepares the parameters of the synchronization task sent from the front-end, generates synchronization parameters according to the task parameters: sharded database address, node range of sharded tables, table names of source and target tables, column names of source and target tables, etc., and sends the parameters to the synchronization execution machine.

[0094] The business middleware completes the tasks after synchronization: verifying whether the number of data in the synchronization temporary table is correct; if the verification is error-free, overwriting the data in the synchronization temporary table into the synchronization target table.

[0095] It is recommended to deploy a group of synchronization execution machines, deploy the synchronization program, as well as task configuration and startup scripts. The synchronization execution machine receives the synchronization task parameters, converts them into the input file format of the synchronization program, and starts the synchronization program to complete the synchronization task.

[0096] Specifically, the synchronization process:

[0097] 1. The web front-end client initiates a set of synchronization task start instructions;

[0098] 2. Calculate the specific location of the sharded database, the table names of the sharded tables, all column names of the tables, etc. as synchronization parameters according to the incoming parameters of each group of synchronization tasks;

[0099] 3. Allocate the execution task to one of the synchronization execution clusters according to the load algorithm, and call the bash script on the synchronization execution machine through remote login with the aggregated synchronization parameters;

[0100] 4. The synchronization executor calls the bash script, replaces the placeholders in the synchronization program parameter template file with the passed-in parameters, and then starts the synchronization program to execute the synchronization task.

[0101] In particular, for the synchronization from a relational database to a distributed database:

[0102] To fully utilize the parallel performance of the synchronization executor and the distributed database, the synchronization tasks of all sharded tables are divided into several groups according to the sharding rules of the distributed database. In database design, the sharding keys of 10 sharded tables in the relational database correspond to 1 table in the distributed database. That is to say, the data of 10 sharded tables in the relational database will be aggregated into the sharding key of 1 table in the distributed database. Therefore, the 10-to-1 tables are used as a group of synchronization tasks.

[0103] For the synchronization from a distributed database to a relational database: Since the number of shards in the distributed database is less than the number of sharded tables in the distributed database and the data mapping relationship is 1 to 10, an additional table needs to be provided to be associated with the synchronization table to calculate the sharding value of the current data in the relational database.

[0104] Synchronization has three modes:

[0105] (1) Full table synchronization mode, which synchronizes all data in the source table to the target table. Correspondingly, the data in the target table needs to be cleared in advance.

[0106] (2) Incremental synchronization mode, which filters out the data to be synchronized according to the timestamp or other fields that can identify incremental data.

[0107] (3) Query synchronization mode, which configures the query SQL statement and synchronizes the data queried by this statement to the target table.

[0108] The present invention can dynamically add or delete data sources and target tables in the configuration item, add and expand the synchronization executor, and has the ability to dynamically expand or shrink the data synchronization range and the ability to dynamically expand the synchronization efficiency when the execution performance is insufficient.

[0109] For the specific implementation solution of this embodiment, reference can be made to the relevant descriptions in the above embodiments, which will not be elaborated here.

[0110] It can be understood that the same or similar parts in the above embodiments can be referred to each other, and the content not detailed in some embodiments can be referred to the same or similar content in other embodiments.

[0111] It should be noted that in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be construed as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "a plurality of" refers to at least two.

[0112] Any process or method description shown in a flowchart or described otherwise herein can be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a specific logical function or process. The scope of the preferred embodiments of the present invention includes additional implementations where functions may be executed in a substantially simultaneous manner or in an order opposite to that shown or discussed, according to the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0113] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0114] Those of ordinary skill in the art of the present technology can understand that all or part of the steps carried by the method of the above embodiments can be completed by instructing relevant hardware through a program, and the corresponding program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0115] In addition, each functional unit in various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0116] The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disk, etc.

[0117] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0118] The system, method, processor, and computer-readable storage medium for implementing data synchronization processing based on a relational database and a distributed database of the present invention solve the problem of difficult synchronization caused by different design structures between sharded databases and single-body databases. Its characteristics lie in flexible configuration, high performance, and easy expansion, being able to meet the data synchronization requirements for large-volume data exchange between databases, and providing supporting backup and verification functions to ensure the security and accuracy of synchronized data.

[0119] In this specification, the present invention has been described with reference to its specific embodiments. However, it is obvious that various modifications and transformations can still be made without departing from the spirit and scope of the present invention. Therefore, the specification and the drawings should be regarded as illustrative rather than restrictive.

Claims

1. A system for realizing data synchronization processing based on relational database and distributed database, characterized in that: The system comprises: Web front-end client, used to operate synchronization tasks through the web front-end client; Relational database, used to store and manage various types of clearing business data; Distributed database, used to store and manage the clearing business data of Xinchuang; The business middleware is connected to the web front-end client, the relational database and the distributed database, and is used to receive the operation tasks of the web front-end client, prepare the synchronization temporary table according to the pre-configuration, generate the synchronization instruction and send it to the synchronization execution machine, check the synchronization data, and overwrite the synchronization data; The synchronous execution group includes multiple synchronous execution machines, which are connected to the web front-end client, relational database, distributed database and business middleware, and are used to receive synchronization task parameters, convert them into input configuration files, start the synchronization program, read data from the data source end and write them to the data destination end.

2. The system for realizing data synchronization processing based on relational database and distributed database according to claim 1, characterized in that: The relational database and the distributed database synchronize the table data processed by their respective business systems to each other at different time points, and synchronize from either party to the other party.

3. The system for realizing data synchronization based on relational database and distributed database according to claim 1, characterized in that: In the direction of synchronization from the relational database to the distributed database, the system automatically calculates and generates the correspondence between the shard tables in the relational database and the shard keys in the distributed database based on the rules, that is, 10 shard tables in the relational database correspond to one shard key of one table in the distributed database, and the 10 shard tables in the relational database with the corresponding relationship and one shard key of one table in the distributed database are regarded as a set of synchronization tasks.

4. The system for realizing data synchronization based on relational database and distributed database according to claim 1, characterized in that: In the direction of synchronization of the distributed database to the relational database, the distributed database has an additional table associated with the synchronization table for calculating the sharding value corresponding to the sharding key of the current data in the relational database.

5. The system for realizing data synchronization based on relational database and distributed database according to claim 1, characterized in that: The synchronization execution machine synchronizes between two databases, and has the following synchronization operation modes: Full table synchronization mode synchronizes all data in the source table to the target table and cleans up the data in the target table in advance; Incremental synchronization mode, filter out the data to be synchronized based on timestamps or other fields that can identify incremental data; Query synchronization mode, configure the query SQL statement, and synchronize the data queried by this statement to the target table.

6. The system for realizing data synchronization based on relational database and distributed database according to claim 1, characterized in that: The process design of the business middleware in processing data synchronization tasks has the following steps: (1) Back up the current data of the table to be synchronized, automatically generate the table structure of the synchronization temporary table according to the table structure of the synchronized table, and use it as the target table for data synchronization. Store the synchronization data in this temporary table; (2) Execute the synchronization task, synchronize the data of the synchronization source end to the temporary table prepared in step (1) of the synchronization destination end, the business middleware prepares the synchronization task parameters sent by the web front-end client, converts them into the required input file format according to the synchronization task parameters, and sends them to the synchronization execution machine; after the synchronization task is completed, the task status and the amount of synchronized data are recorded in the scheduling system, and the number of data in the synchronization temporary table is verified to be consistent with the number synchronized by the synchronization execution machine to ensure the correctness of the synchronization process; (3) Execute the verification logic to check the validity and accuracy of the synchronized data to ensure the correctness of the data generated by the synchronization source; (4) Overwrite the verified synchronization data in the temporary table to the formal table on the synchronization target end.

7. The system for realizing data synchronization based on relational database and distributed database according to claim 6, characterized in that: The system dynamically adds or deletes data sources and target tables in synchronization task parameters, and adds and expands synchronization execution machines.

8. A method for implementing data synchronization processing based on a relational database and a distributed database using the system of claim 1, characterized in that: The method comprises the following implementation steps: (1) The web front-end client initiates a set of synchronization task start instructions; (2) The business middleware calculates synchronization parameters based on the input parameters of each group of synchronization tasks; (3) Allocate execution tasks to synchronous execution machines according to the load algorithm, convert task parameters into the input file format required by the synchronous execution machine, send it to the synchronous execution machine, and call the script of the synchronous execution machine with the summarized synchronization parameters through remote login; (4) The synchronization execution machine calls the script and replaces the placeholders in the synchronization program input template file with the input parameters, and starts the synchronization program to execute the synchronization task.

9. A device for realizing data synchronization processing based on a relational database and a distributed database, characterized in that: The device comprises: a processor configured to execute computer-executable instructions; The memory stores one or more computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the method for realizing data synchronization processing based on a relational database and a distributed database as described in any one of claim 8 are implemented.

10. A processor for implementing data synchronization processing based on a relational database and a distributed database, characterized in that: The processor is configured to execute computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the method for realizing data synchronization processing based on a relational database and a distributed database as described in any one of claim 8 are implemented.

11. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the various steps of the method for realizing data synchronization processing based on a relational database and a distributed database as described in any one of claim 8.

Images