Data synchronization method and system

Abstract

The present application discloses a data synchronization method and system. Relating to the field of financial technology, the method comprises: periodically acquiring transaction data from the application server of the target business system at a first frequency and storing it in an open source database of the distributed system; acquiring business parameter data from the database of the target business system at a second frequency and storing it in an in-memory database of the distributed system; and synchronizing the changed data to the open source database and / or in-memory database based on the changed data in the database when the transaction of the target business system changes. This application solves the problem of data discrepancies between the HBase open source database and Redis in-memory database of the managed distributed system and the Oracle database of the traditional business system in the related art, and the need to ensure data integrity before distributed computing processing in the distributed system.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and specifically, to a data synchronization method and system. Background Art

[0002] In recent years, the capital market has developed rapidly, and the asset custody market has also grown rapidly. The amount of data processed by custody business has increased exponentially. Institutions have invested a lot of manpower in the research and development of distributed systems. This system is based on distributed computing and large memory processing, breaking through the bottleneck of performance and capacity to meet the requirements of business processing efficiency.

[0003] HBase (Hadoop Database) is a distributed, column-oriented, open-source database that supports massive storage, extreme scalability, and high concurrency. It's ideal for storing massive amounts of daily market data and managed portfolio trading data. Redis (Remote Dictionary Server) is an in-memory database with fast access speeds, support for various data structures, data backup, and transactions. It's ideal for storing parameter data frequently used during computations. Therefore, the distributed system uses an HBase + Redis model for data storage, with market and transaction data stored in the distributed HBase database and parameter data in the in-memory Redis database. However, due to the company's existing internal management systems, business system applications are deployed on application servers (using middleware such as WAS and Tomcat) and Oracle Database (a data block management system). Manual maintenance of parameter data and new business data by business personnel is still stored in Oracle, making data integrity in the managed distributed system a major challenge.

[0004] As for the problem of how to ensure the data integrity of the managed distributed system in related technologies, no effective solution has been proposed so far. Summary of the Invention

[0005] The main purpose of this application is to provide a data synchronization method and system to solve the problem of data differences between the HBase open source database and Redis memory database of the hosted distributed system and the Oracle database of the traditional business system in the related technology, and the need to ensure data integrity before distributed computing processing in the distributed system.

[0006] To achieve the above-mentioned objectives, according to one aspect of the present application, a data synchronization method is provided. The method comprises: periodically acquiring transaction data from an application server of a target business system at a first frequency and storing the data in an open-source database of a distributed system; acquiring business parameter data from a database of the target business system at a second frequency and storing the data in an in-memory database of the distributed system, wherein the first frequency is higher than the second frequency; and synchronizing the changed data in the database to the open-source database and / or the in-memory database when a transaction of the target business system changes, based on the changed data in the database.

[0007] Optionally, synchronizing the changed data to the open source database and / or the in-memory database based on the changed data in the database includes: creating a trigger under the preset path of the target business system; when the transaction of the target business system changes, automatically triggering the trigger to change the changed data to generate change data in a preset format, wherein the changed data includes changed transaction data and / or changed business parameter data, and the change data includes first change data generated based on the changed transaction data, and / or second change data generated based on the changed business parameter data; based on the first change data, synchronizing the changed transaction data to the open source database, and / or, based on the second change data, synchronizing the changed business parameter data to the in-memory database.

[0008] Optionally, before synchronizing the changed transaction data to the open source database according to the first change data, and / or synchronizing the changed business parameter data to the in-memory database according to the second change data, the method further includes: setting a scheduled scanning task through the server cluster of the distributed system to perform a scheduled scanning of the log of the change data; when the first change data is scanned, executing the step of synchronizing the changed transaction data to the open source database according to the first change data, and / or, when the second change data is scanned, executing the step of synchronizing the changed business parameter data to the in-memory database according to the second change data.

[0009] Optionally, according to the second change data, synchronizing the changed business parameter data to the in-memory database includes: obtaining the second serial number of the changed business parameter data according to the log of the second change data; searching for the corresponding business parameter data from the open source database according to the second serial number; and synchronizing the business parameter data according to the changed business parameter data.

[0010] Optionally, synchronizing the changed transaction data to the open source database based on the first change data includes: obtaining the first serial number of the changed transaction data based on the log of the first change data; searching for corresponding transaction data from the open source database based on the first serial number; and synchronizing the transaction data based on the primary key of the changed transaction data.

[0011] Optionally, synchronizing the transaction data according to the primary key of the changed transaction data includes: when the changed transaction data has a primary key, constructing an insert object set of the open source database according to the primary key value; when the changed transaction data does not have a primary key, constructing an insert object set of the open source database according to the row key of the changed transaction data in the target business system; based on the insert object set, inserting the insert object set into the open source database to synchronize the transaction data of the open source database.

[0012] Optionally, constructing the insert object set of the open source database according to the primary key value includes: building a delete and add queue for the same table; offsetting the same new and deleted records according to the primary key value to construct the insert object set; constructing the insert object set of the open source database according to the row key of the target business system of the changed transaction data includes: merging the same records according to the row key to construct the insert object set.

[0013] To achieve the above-mentioned objectives, according to another aspect of the present application, a data synchronization device is provided. The device comprises: an open source database storage module configured to periodically obtain transaction data from an application server of a target business system at a first frequency and store the data in an open source database of a distributed system; an in-memory database storage module configured to obtain business parameter data from a database of the target business system at a second frequency and store the data in an in-memory database of the distributed system, wherein the first frequency is higher than the second frequency; and a synchronization module configured to synchronize the changed data in the database with the open source database and / or the in-memory database when a transaction of the target business system changes.

[0014] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the storage medium is used to store a program, wherein the program executes any one of the above-mentioned data synchronization methods.

[0015] According to another aspect of the present application, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the data synchronization methods described above.

[0016] Through this application, the following steps are adopted: transaction data is obtained from the application server of the target business system at a first frequency and stored in the open source database of the distributed system; business parameter data is obtained from the database of the target business system at a second frequency and stored in the memory database of the distributed system, wherein the first frequency is higher than the second frequency; in the case of changes in the transactions of the target business system, the changed data is synchronized to the open source database and / or the memory database according to the changed data in the database, and the transaction data and business parameter data of the target business system are synchronized to different types of databases respectively by adopting a fixed frequency acquisition method, and separate data incremental synchronization is adopted for different databases according to the changes in the target business system data, thereby achieving the purpose of synchronizing the data of the target business system to the distributed system database and realizing the technical effect of improving the integrity and reliability of the distributed system data. Through this application, the problem of data differences between the HBase open source database and Redis memory database of the hosted distributed system and the Oracle database of the traditional business system in the related technology and the need to ensure data integrity before distributed computing processing in the distributed system is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0018] Figure 1 This is a flow chart of a data synchronization method provided according to an embodiment of the present application;

[0019] Figure 2 Schematic diagram of a data synchronization module according to an embodiment of the present application;

[0020] Figure 3 This is a schematic diagram of a data synchronization process according to an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of a data synchronization device provided according to an embodiment of the present application;

[0022] Figure 5 This is a schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0027] The present invention will be described below in conjunction with preferred implementation steps. Figure 1 This is a flow chart of a data synchronization method provided according to an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0028] Step S101: acquiring transaction data from an application server of a target business system at a regular interval according to a first frequency, and storing the data in an open source database of a distributed system;

[0029] Step S102, obtaining service parameter data from a database of the target service system according to a second frequency, and storing the data in a memory database of the distributed system, wherein the first frequency is higher than the second frequency;

[0030] Step S103 , when the transaction of the target business system changes, the changed data is synchronized to the open source database and / or the in-memory database according to the changed data in the database.

[0031] Through the above steps, transaction data is periodically obtained from the application server of the target business system at a first frequency and stored in the open source database of the distributed system; business parameter data is obtained from the database of the target business system at a second frequency and stored in the memory database of the distributed system, wherein the first frequency is higher than the second frequency; in the case of changes in the transactions of the target business system, the changed data is synchronized to the open source database and / or the memory database according to the changed data in the database, and the transaction data and business parameter data of the target business system are synchronized to different types of databases respectively by adopting a fixed frequency acquisition method, and separate data incremental synchronization is adopted for different databases according to changes in the target business system data, thereby achieving the purpose of synchronizing the data of the target business system of the hosted distributed system to the target business system data distributed system database, and realizing the technical effect of improving the integrity and reliability of the distributed system data. Through this application, the problem of data differences between the HBase open source database and Redis memory database of the hosted distributed system and the Oracle database of the traditional business system in the related technology is solved, and the problem of ensuring data integrity before distributed computing processing in the distributed system is solved.

[0032] The above-mentioned target business system may refer to a business management system, which is provided with an Oracle database, and the business transaction data and transaction parameter data of the business management system are stored in the above-mentioned Oracle database. The Oracle database requires business personnel to manually maintain parameter data and manually add new business data. The above-mentioned distributed system may be a hosted distributed system, which can store business transaction data and corresponding parameter data in a distributed manner. The above-mentioned distributed system may contain its own database, including the above-mentioned HBase open source database and Redis memory database. The distributed system data storage adopts the HBase+Redis mode. For example, market data and transaction data are stored in the distributed database HBase, and business parameter data is stored in the memory database Redis. The transaction data in the Oracle database in the above-mentioned target business system can be synchronously stored in the HBase open source database in the distributed system, and the business parameter data in the Oracle database can be synchronously stored in the Redis memory database to achieve data backup.

[0033] In a hosted distributed system, the target business system can synchronize transaction data with large amounts of data in the database to an open source database at a first frequency, or synchronize frequently called business parameter data in the database to a memory database with fast access speed at a second frequency, thereby distributing the data of the target business system according to data access requirements, thereby improving data access efficiency. The first frequency and the second frequency can refer to two different paths or triggering methods for synchronizing data to the open source database and the memory database of the distributed system. The transaction data can be synchronized to the open source database at the first frequency, and the business parameter data can be synchronized to the memory database at the second frequency. The first frequency can be equal to the second frequency.

[0034] The above-mentioned target business system can also update the transaction data that has changed due to operations in its own database to the open source database of the hosted distributed system, and update the business parameter data that has changed due to operations in the target business system to the memory database of the hosted distributed system. The above-mentioned operations may refer to operations such as deletion, modification, and addition of data by business personnel. By synchronizing the data in the above-mentioned target business system database to the hosted distributed system database, it avoids the situation where the changed data in the traditional system is always stored in the business system database, and the integrity of the distributed system data cannot be guaranteed.

[0035] By adopting a fixed frequency method to synchronize the transaction data and business parameter data of the target business system to different types of databases respectively, and using separate data synchronization for changes in the target business system data, the purpose of synchronizing the hosted distributed system data to the target business system data is achieved, and the technical effect of improving the integrity and reliability of the distributed system data is realized.

[0036] Optionally, synchronizing the changed data to the open source database and / or the in-memory database based on the changed data in the database includes: creating a trigger under the preset path of the target business system; when the transaction of the target business system changes, automatically triggering the trigger to convert the changed data into change data in a preset format, wherein the changed data includes the changed transaction data and / or the changed business parameter data, and the change data includes first change data generated based on the changed transaction data, and / or second change data generated based on the changed business parameter data; synchronizing the changed transaction data to the open source database based on the first change data, and / or synchronizing the changed business parameter data to the in-memory database based on the second change data.

[0037] If the transaction data table and business parameter data table in the business system database change, the changes may be caused by manual maintenance of the data table by business personnel. The above changes may refer to the addition, deletion, modification, etc. of the data table. The business system can synchronize the changed transaction data and business parameter data to the open source database and in-memory database respectively.

[0038] When triggering incremental synchronization, a trigger for monitoring data changes can be created under the preset path of the target business system. When the trigger detects a data change, the corresponding incremental synchronization process is automatically triggered. The above-mentioned target business system can create separate triggers for changes in transaction data and business parameter data, respectively, to store the transaction data and business parameter data in the target business system in the open source database and in-memory database of the distributed system, respectively. The above-mentioned changed data includes changed transaction data and / or changed business parameter data. Changes in the above-mentioned transaction data and business parameter data can be monitored by the above-mentioned triggers. The change data in the preset format generated includes first change data generated based on the changed transaction data, and / or second change data generated based on the changed business parameter data.

[0039] If the data table of the business system database changes, a trigger can be automatically triggered to generate change data in a preset format for the changed transaction data or parameter data. The changed data can be scanned in a scheduled or manually set manner, and the changed transaction data can be synchronized to the open source database, or the changed business parameter data can be synchronized to the in-memory database for data synchronization.

[0040] Specifically, if the data in the Oracle business table of the business database changes, the TRIGGER trigger is triggered to generate JSON (JavaScript Object Notation, programming language JavaScript object notation format change data), and then the cluster DRIVER job scans the change data log table to determine whether there is data to be synchronized. If not, the cluster DRIVER (i.e., cluster server) continues to scan the change data log table. If so, data synchronization is performed. It should be noted that the change data of transaction data and business parameter data are synchronized to the open source database HBase and the in-memory database Redis respectively.

[0041] By adopting the incremental synchronization method, the purpose of synchronizing the changed business transaction data and business parameter data to the database is achieved, and the technical effect of improving the system storage efficiency is achieved.

[0042] Optionally, before synchronizing the changed transaction data to the open source database according to the first change data, and / or synchronizing the changed business parameter data to the in-memory database according to the second change data, the method further includes: setting a scheduled scanning task through the server cluster of the distributed system to perform a scheduled scanning of the log of the change data; when the first change data is scanned, executing the step of synchronizing the changed transaction data to the open source database according to the first change data, and / or, when the second change data is scanned, executing the step of synchronizing the changed business parameter data to the in-memory database according to the second change data.

[0043] Before synchronizing the changed transaction data to the open source database based on the first change data, and / or synchronizing the changed business parameter data to the in-memory database based on the second change data, the server cluster of the distributed system can scan the log of the change data record in the preset format generated by the above trigger, including scanning both the log of the above first change data and the log of the second change data. The server cluster of the above distributed system can periodically scan the log of the change data by setting a scheduled scanning task. When the first change data is scanned, the changed transaction data is synchronized to the open source database based on the first change data, and / or when the second change data is scanned, the changed business parameter data is synchronized to the in-memory database based on the second change data.

[0044] By setting a timed log of data changes in the Redis memory database, the goal of timely synchronization of changed data is achieved, and the technical effect of improving the timeliness of synchronization of changed data is realized.

[0045] Optionally, synchronizing the changed business parameter data to the in-memory database based on the second change data includes: obtaining the second serial number of the changed business parameter data based on the log of the second change data; searching for the corresponding business parameter data from the open source database based on the second serial number; and synchronizing the business parameter data based on the changed business parameter data.

[0046] The changed business parameter data is synchronized to the in-memory database according to the second change data. The second change data log of the above-mentioned business parameter data may include a second serial number. The above-mentioned second serial number may be a serial number that has an identification function for the business parameter data. The business parameter data corresponding to the second change data log can be searched in the in-memory database through the above-mentioned second serial number. The business parameter data can be found through the second serial number, and the changed business parameter data can be synchronized to the in-memory database.

[0047] Specifically, when performing incremental synchronization of the Redis database, the job on the cluster server periodically scans the Redis memory database synchronization data table. The cluster server determines whether there is data to be synchronized. If not, it continues to scan. If so, it continues. The cluster server obtains the data sequence number of the Redis memory database to be synchronized (i.e., the second sequence number mentioned above) and updates the synchronization status. The data is written to the Redis memory database to complete the incremental synchronization of the data in the Redis memory database. By searching for the changed business parameter data in the memory database through the second sequence number, the purpose of accurately locating the changed business parameter data in the memory database is achieved, and the technical effect of improving the accuracy of business parameter data synchronization is realized.

[0048] Optionally, synchronizing the changed transaction data to the open source database based on the first change data includes: obtaining the first serial number of the changed transaction data based on the log of the first change data; searching for corresponding transaction data from the open source database based on the first serial number; and synchronizing the transaction data based on the primary key of the changed transaction data.

[0049] The changed transaction data is synchronized to the open source database according to the first change data. The first change data log of the above-mentioned transaction data may include a first serial number. The above-mentioned first serial number may be a serial number that has an identification function for the transaction data. The transaction data corresponding to the first change data log can be searched in the open source database through the above-mentioned first serial number. The transaction data can be found through the first serial number. The changed transaction data can be synchronized to the open source database according to the primary key of the changed transaction data.

[0050] Specifically, the HBase incremental synchronization module is entered, and the job on the cluster server periodically scans the HBase synchronization data table. The cluster server determines whether there is data to be synchronized. If not, the scanning process is continued. If so, the cluster server proceeds to the next step. The cluster server can obtain the data sequence number of the HBase open source database to be synchronized (i.e., the first sequence number mentioned above) and update the synchronization status. The cluster server determines whether the data to be synchronized contains a primary key. If so, a deletion and addition queue is established for the same table. The cluster server can offset the same addition and deletion records based on the primary key.

[0051] By searching for changed transaction data in the open source database through the first serial number, the purpose of accurately locating the changed transaction data in the memory database is achieved, and the technical effect of improving the accuracy of transaction data synchronization is realized.

[0052] Optionally, synchronizing transaction data based on the primary key of the changed transaction data includes: when the changed transaction data has a primary key, constructing an insert object set of the open source database based on the primary key value; when the changed transaction data does not have a primary key, constructing an insert object set of the open source database based on the row key of the changed transaction data in the target business system; based on the insert object set, inserting the insert object set into the open source database to synchronize the transaction data of the open source database.

[0053] During the synchronization of transaction data based on the primary key of the changed transaction data, an insert object set can also be constructed. The target business system can insert the changed data into the open source database or in-memory database through the constructed insert object set. The above insert object set can refer to a set of commands for performing database insert, modification, deletion and other operations. If the changed transaction data has a primary key, the insert object set of the open source database can be constructed based on the set primary key value that uniquely identifies the record. If the changed transaction data does not have a primary key, the insert object set of the open source database can be constructed based on the row key that the database automatically establishes for each table that uniquely identifies a record. Based on the insert object set, the changed data can be synchronized to the open source database.

[0054] Specifically, the cluster server determines whether the data to be synchronized contains a primary key. If so, it creates a delete and add queue for the same table. Based on the primary key, the cluster server offsets the identical new and deleted records and constructs a PUT object set (i.e., the insert object set described above) for storage. If the data does not contain a primary key, it merges identical records based on the row key (i.e., the row key described above) and constructs a PUT object set for storage.

[0055] By constructing an insert object collection to synchronize the changed data to the open source database, the purpose of synchronizing the changed transaction data is achieved, and the technical effect of improving the synchronization speed of the changed data is realized.

[0056] Optionally, constructing an insert object set of an open source database based on a primary key value includes: building a delete and add queue for the same table; offsetting identical new and deleted records according to the primary key value to construct an insert object set; constructing an insert object set of an open source database based on a row key of the changed transaction data in the target business system includes: merging identical records according to the row key to construct an insert object set.

[0057] When constructing an insert object set of an open source database based on the primary key value, you can also construct a delete and add queue for the same table. You can merge the same records through the constructed delete and add queue. If there is a primary key in the same two tables, you can offset the same new and deleted records according to the primary key value that uniquely identifies the record to construct an insert object set. If there is no primary key in the same two tables, you can merge the same records according to the row key value that uniquely identifies the record in the record table to construct an insert object set.

[0058] Before constructing the insert object set, the same new and deleted records of the same table are merged, thereby reducing the data redundancy of the insert object set and achieving the technical effect of improving the data simplicity of the insert object set.

[0059] It should be noted that this embodiment also provides an optional implementation method, which is described in detail below.

[0060] This implementation provides a solution for ensuring data integrity in a managed distributed system. The overall concept is as follows:

[0061] Managed distributed system data synchronization first ensures that the full data in the HBase open source database and Redis in-memory database is basically consistent with the business system Oracle database. Then, when data changes in the business system due to manual operations by business personnel, the changed data can be synchronized to the HBase open source database and Redis in-memory database in real time. Distributed systems are prohibited from initiating calculations before synchronization is completed.

[0062] Figure 2 This is a schematic diagram of a data synchronization module according to an embodiment of the present application, wherein the specific modules are as follows: Figure 2 As shown: the present invention includes four modules, namely HBase open source database full synchronization module, Redis memory database full synchronization module, HBase open source database incremental synchronization module, and Redis memory database incremental synchronization module.

[0063] HBase Open Source Database Full Synchronization Module: The HBase open source database primarily stores transaction data. After the market closes each trading day, the system automatically initiates data reception upon receiving transaction files from exchanges, registration companies, management companies, and other institutions. This module builds upon the existing data reception module in the business system by adding a branch to handle data import into the HBase open source database. This template is deployed on a separate server, designated the Data Receiving Server, and runs two scheduled jobs. The first scans the control table every 5 seconds, pulling files from the WAS server via FTP (File Transfer Protocol) to a designated directory on the Data Receiving Server. The second scans the receiving control table every 5 seconds and inserts the pending file data into the HBase open source database. This allows for data consistency comparisons with synchronized data from Oracle.

[0064] Redis in-memory database full synchronization module: The Redis in-memory database mainly stores business parameter data. The full parameter data of Oracle is synchronized to the Redis in-memory database once every morning.

[0065] HBase open-source database incremental synchronization module: When business personnel manually maintain transaction data, the business data table data in Oracle changes. This data needs to be synchronized from Oracle to the HBase open-source database table. An INSERT, UPDATE, or DELETE trigger is created. The trigger name begins with H_ and serializes and stores the records before and after the change in JSON (JavaScript Object Notation). The cluster driver (i.e., the cluster server) uses a scheduled job to scan the log table for data synchronization. Specifically:

[0066] 1) The Oracle business table data changes.

[0067] 2) Trigger TRIGGER to generate change data in JSON format.

[0068] 3) The cluster DRIVER job scans the change data log table.

[0069] 4) Determine whether there is data to be synchronized. If not, return to 3) to continue scanning. If so, continue.

[0070] 5) Determine whether the data to be synchronized has a primary key. If so, for the same table, build a delete and add queue, offset the same new and deleted records by primary key value, and construct a PUT object set (that is, the above-mentioned insert object set); if the primary key does not exist, merge the same records by rowid to construct a PUT object set.

[0071] 6) Submit the data in batches to the HBase open source database, and the synchronization is complete.

[0072] Redis In-Memory Database Incremental Synchronization Module 4: When business personnel manually maintain parameter data, the parameter table data in Oracle changes. This data needs to be synchronized from Oracle to the Redis in-memory database table. INSERT, UPDATE, and DELETE triggers are created, with the trigger name beginning with R_. The records before and after the change are serialized and stored in JSON format. A scheduled job in the cluster driver scans the log table to synchronize the data. The specific steps are similar to those in the HBase open source database incremental synchronization module 3 and are not repeated here.

[0073] Figure 3 This is a schematic diagram of the data synchronization process provided by the embodiment of the present application. The specific process is as follows Figure 3 As shown:

[0074] Step S301: The data receiving server obtains transaction and market data files.

[0075] Step S302: The data receiving server directly enters the file data into the HBase open source database.

[0076] Step S303: Daily full data synchronization of the Redis memory database, which is the above-mentioned Redis memory database full synchronization module 2.

[0077] Step S304: Entering the HBase open source database incremental synchronization module 3, the job on the cluster server regularly scans the HBase open source database synchronization data table.

[0078] Step S305: On the cluster server, determine whether there is data to be synchronized. If not, return to S303 to continue scanning. If yes, continue.

[0079] Step S306: The cluster server obtains the sequence number of the HBase open source database data to be synchronized and updates the synchronization status.

[0080] Step S307: The cluster server determines whether the data to be synchronized contains a primary key.

[0081] Step S308: If there is a primary key, the newly added queue is deleted from the same table construction.

[0082] Step S309: The cluster server offsets the same added and deleted records based on the primary key, and constructs a PUT object set to prepare for storage.

[0083] Step S310: If there is no primary key, merge the same records according to the rowkey (i.e. the above row key) and construct a PUT object set to prepare for storage.

[0084] Step S311: Submit data in batches to the HBase open source database to complete incremental synchronization of HBase open source database data.

[0085] Step S312: Enter the Redis memory database incremental synchronization module, and the job on the cluster server regularly scans the Redis memory database synchronization data table.

[0086] Step S313: The cluster server determines whether there is data to be synchronized. If not, it returns to S312 to continue scanning. If yes, it continues.

[0087] Step S314: The cluster server obtains the sequence number of the Redis memory database data to be synchronized and updates the synchronization status.

[0088] Step S315: The data is written into the Redis memory database, completing the incremental synchronization of the Redis memory database data.

[0089] This implementation addresses the problem of data discrepancies between the hosted distributed system and the bank's traditional business system, and the need to ensure data integrity before distributed computing processing. A data synchronization solution is proposed, namely, real-time synchronization of the HBase open source database, the Redis in-memory database, and the Oracle database to ensure that all data is complete and the calculation results are reliable during the distributed system's computing processing.

[0090] The key to this implementation is to solve the data synchronization problem between the hosted distributed system and the traditional business system. While the distributed system provides high-performance computing, it also ensures data integrity and accuracy, improves business processing capabilities, and enhances the service capabilities of the custodial bank.

[0091] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0092] The present application also provides a data synchronization device. It should be noted that the data synchronization device of the present application can be used to execute the data synchronization method provided in the present application. The following introduces a data synchronization device provided in the present application.

[0093] Figure 4 Schematic diagram of a data synchronization device according to an embodiment of the present application. Figure 4 As shown, the device includes an open source database storage module 40, a memory database storage module 42, and a synchronization module 44. The device will be described in detail below.

[0094] The open source database storage module 40 is used to periodically obtain transaction data from the application server of the target business system at a first frequency and store it in the open source database of the distributed system; the memory database storage module 42 is connected to the above-mentioned open source database storage module 40, and is used to obtain business parameter data from the database of the target business system at a second frequency and store it in the memory database of the distributed system, wherein the first frequency is higher than the second frequency; the synchronization module 44 is connected to the above-mentioned memory database storage module 42, and is used to synchronize the changed data to the open source database and / or the memory database according to the changed data in the database when the transaction of the target business system changes.

[0095] The embodiment of the present application provides a data synchronization device, which periodically obtains transaction data from the application server of the target business system at a first frequency and stores it in the open source database of the distributed system; obtains business parameter data from the database of the target business system at a second frequency and stores it in the memory database of the distributed system, wherein the first frequency is higher than the second frequency; in the case of changes in the transactions of the target business system, the changed data is synchronized to the open source database and / or the memory database according to the changed data in the database, and the transaction data and business parameter data of the target business system are synchronized to different types of databases respectively by adopting a fixed frequency acquisition method, and separate data incremental synchronization is adopted for different databases according to changes in the target business system data, thereby achieving the purpose of synchronizing the data of the target business system to the distributed system database, and realizing the technical effect of improving the integrity and reliability of the distributed system data. Through this application, the problem of data differences between the HBase open source database and Redis memory database of the hosted distributed system and the Oracle database of the traditional business system in the related technology and the need to ensure data integrity before distributed computing processing in the distributed system is solved.

[0096] The data synchronization device includes a processor and a memory. The above-mentioned open source database storage module 40, memory database storage module 42, synchronization module 44, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0097] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured. Kernel parameters can be adjusted to address data discrepancies between the HBase open-source database and Redis in-memory database used in managed distributed systems and the Oracle database used in traditional business systems. This addresses the need to ensure data integrity before distributed computing processing in distributed systems.

[0098] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0099] An embodiment of the present invention provides a computer-readable storage medium having a program stored thereon, which implements the data synchronization method when executed by a processor.

[0100] An embodiment of the present invention provides a processor, which is used to run a program, wherein the data synchronization method is executed when the program is running.

[0101] like Figure 5 As shown, an embodiment of the present invention provides an electronic device, the electronic device 50 includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the following steps are implemented: transaction data is periodically acquired from an application server of a target business system at a first frequency, and stored in an open source database of a distributed system; business parameter data is acquired from a database of the target business system at a second frequency, and stored in a memory database of the distributed system, wherein the first frequency is higher than the second frequency; when transactions of the target business system change, the changed data is synchronized to the open source database and / or the memory database according to the changed data in the database.

[0102] Optionally, synchronizing the changed data to the open source database and / or the in-memory database based on the changed data in the database includes: creating a trigger under the preset path of the target business system; when the transaction of the target business system changes, automatically triggering the trigger to convert the changed data into change data in a preset format, wherein the changed data includes the changed transaction data and / or the changed business parameter data, and the change data includes first change data generated based on the changed transaction data, and / or second change data generated based on the changed business parameter data; synchronizing the changed transaction data to the open source database based on the first change data, and / or synchronizing the changed business parameter data to the in-memory database based on the second change data.

[0103] Optionally, before synchronizing the changed transaction data to the open source database according to the first change data, and / or synchronizing the changed business parameter data to the in-memory database according to the second change data, the method further includes: setting a scheduled scanning task through the server cluster of the distributed system to perform a scheduled scanning of the log of the change data; when the first change data is scanned, executing the step of synchronizing the changed transaction data to the open source database according to the first change data, and / or, when the second change data is scanned, executing the step of synchronizing the changed business parameter data to the in-memory database according to the second change data.

[0104] Optionally, synchronizing the changed business parameter data to the in-memory database based on the second change data includes: obtaining the second serial number of the changed business parameter data based on the log of the second change data; searching for the corresponding business parameter data from the open source database based on the second serial number; and synchronizing the business parameter data based on the changed business parameter data.

[0105] Optionally, synchronizing the changed transaction data to the open source database based on the first change data includes: obtaining the first serial number of the changed transaction data based on the log of the first change data; searching for corresponding transaction data from the open source database based on the first serial number; and synchronizing the transaction data based on the primary key of the changed transaction data.

[0106] Optionally, synchronizing transaction data based on the primary key of the changed transaction data includes: when the changed transaction data has a primary key, constructing an insert object set of the open source database based on the primary key value; when the changed transaction data does not have a primary key, constructing an insert object set of the open source database based on the row key of the changed transaction data in the target business system; based on the insert object set, inserting the insert object set into the open source database to synchronize the transaction data of the open source database.

[0107] Optionally, constructing an insert object set of an open source database based on a primary key value includes: building a delete and add queue for the same table; offsetting identical new and deleted records according to the primary key value to construct an insert object set; constructing an insert object set of an open source database based on a row key of the changed transaction data in the target business system includes: merging identical records according to the row key to construct an insert object set.

[0108] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0109] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program initialized with the following method steps: periodically obtaining transaction data from an application server of a target business system at a first frequency and storing it in an open source database of a distributed system; obtaining business parameter data from a database of the target business system at a second frequency and storing it in an in-memory database of the distributed system, wherein the first frequency is higher than the second frequency; and in the event that transactions of the target business system change, synchronizing the changed data to the open source database and / or the in-memory database based on the changed data in the database.

[0110] Optionally, synchronizing the changed data to the open source database and / or the in-memory database based on the changed data in the database includes: creating a trigger under the preset path of the target business system; when the transaction of the target business system changes, automatically triggering the trigger to convert the changed data into change data in a preset format, wherein the changed data includes the changed transaction data and / or the changed business parameter data, and the change data includes first change data generated based on the changed transaction data, and / or second change data generated based on the changed business parameter data; synchronizing the changed transaction data to the open source database based on the first change data, and / or synchronizing the changed business parameter data to the in-memory database based on the second change data.

[0111] Optionally, before synchronizing the changed transaction data to the open source database according to the first change data, and / or synchronizing the changed business parameter data to the in-memory database according to the second change data, the method further includes: setting a scheduled scanning task through the server cluster of the distributed system to perform a scheduled scanning of the log of the change data; when the first change data is scanned, executing the step of synchronizing the changed transaction data to the open source database according to the first change data, and / or, when the second change data is scanned, executing the step of synchronizing the changed business parameter data to the in-memory database according to the second change data.

[0112] Optionally, synchronizing the changed business parameter data to the in-memory database based on the second change data includes: obtaining the second serial number of the changed business parameter data based on the log of the second change data; searching for the corresponding business parameter data from the open source database based on the second serial number; and synchronizing the business parameter data based on the changed business parameter data.

[0113] Optionally, synchronizing the changed transaction data to the open source database based on the first change data includes: obtaining the first serial number of the changed transaction data based on the log of the first change data; searching for corresponding transaction data from the open source database based on the first serial number; and synchronizing the transaction data based on the primary key of the changed transaction data.

[0114] Optionally, synchronizing transaction data based on the primary key of the changed transaction data includes: when the changed transaction data has a primary key, constructing an insert object set of the open source database based on the primary key value; when the changed transaction data does not have a primary key, constructing an insert object set of the open source database based on the row key of the changed transaction data in the target business system; based on the insert object set, inserting the insert object set into the open source database to synchronize the transaction data of the open source database.

[0115] Optionally, constructing an insert object set of an open source database based on a primary key value includes: building a delete and add queue for the same table; offsetting identical new and deleted records according to the primary key value to construct an insert object set; constructing an insert object set of an open source database based on a row key of the changed transaction data in the target business system includes: merging identical records according to the row key to construct an insert object set.

[0116] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0117] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0118] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0119] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0120] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0121] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0122] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0123] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0124] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A data synchronization method, characterized in that: include: periodically acquiring transaction data from an application server of a target business system according to a first frequency, and storing the data in an open source database of a distributed system; acquiring business parameter data from a database of the target business system at a second frequency, and storing the data in a memory database of the distributed system, wherein the first frequency is higher than the second frequency, and the business parameter data refers to parameter data frequently called by the target business system during storage and calculation; When a transaction in the target business system changes, a trigger is created under a preset path of the target business system, and based on the trigger and the changed data in the database, the changed data is synchronized to the open source database and / or the in-memory database.

2. The method according to claim 1, characterized in that Synchronizing the changed data to the open source database and / or the in-memory database according to the changed data in the database includes: When a transaction of the target business system changes, the trigger is automatically triggered to generate change data in a preset format based on the changed data, wherein the changed data includes changed transaction data and / or changed business parameter data, and the change data includes first change data generated based on the changed transaction data and / or second change data generated based on the changed business parameter data; According to the first change data, the changed transaction data is synchronized to the open source database, and / or, according to the second change data, the changed business parameter data is synchronized to the in-memory database.

3. The method according to claim 2, characterized in that Before synchronizing the changed transaction data to the open source database according to the first change data, and / or synchronizing the changed business parameter data to the in-memory database according to the second change data, the method further includes: Setting a scheduled scanning task through the server cluster of the distributed system to regularly scan the log of the changed data; When the first change data is scanned, the step of synchronizing the changed transaction data to the open source database according to the first change data is executed, and / or when the second change data is scanned, the step of synchronizing the changed business parameter data to the in-memory database according to the second change data is executed.

4. The method according to claim 2, characterized in that Synchronizing the changed business parameter data to the in-memory database according to the second change data includes: Obtaining a second serial number of the changed business parameter data according to the log of the second change data; searching the open source database for corresponding service parameter data according to the second serial number; The service parameter data is synchronized according to the changed service parameter data.

5. The method according to claim 2, characterized in that Synchronizing the changed transaction data to the open source database according to the first change data includes: Obtaining, according to the log of the first change data, a first serial number of the changed transaction data; searching for corresponding transaction data from the open source database according to the first serial number; The transaction data is synchronized according to the primary key of the changed transaction data.

6. The method according to claim 5, characterized in that Synchronizing the transaction data according to the primary key of the changed transaction data includes: In the case where the changed transaction data has a primary key, constructing an insert object set of the open source database according to the primary key value; In a case where the changed transaction data does not have a primary key, constructing an insert object set of the open source database according to the row key of the changed transaction data in the target business system; Based on the insertion object set, the insertion object set is inserted into the open source database, and transaction data of the open source database is synchronized.

7. The method according to claim 6, characterized in that Constructing the insert object set of the open source database according to the primary key value includes: For the same table, build a delete and add queue; Constructing the insertion object set according to the primary key value offsetting the same new and deleted records; Constructing an insert object set of the open source database according to the row key of the changed transaction data in the target business system includes: The same records are merged according to the row keys to construct the insert object set.

8. A data synchronization device, characterized in that: include: an open source database storage module, configured to periodically obtain transaction data from an application server of a target business system at a first frequency and store the data in an open source database of a distributed system; an in-memory database storage module, configured to obtain business parameter data from a database of the target business system at a second frequency and store the data in the in-memory database of the distributed system, wherein the first frequency is higher than the second frequency, and the business parameter data refers to parameter data frequently called by the target business system during storage and calculation; A synchronization module is used to create a trigger under a preset path of the target business system when a transaction of the target business system changes, and synchronize the changed data to the open source database and / or the in-memory database based on the trigger and the changed data in the database.

9. A computer-readable storage medium, characterized in that The storage medium is used to store a program, wherein the program executes the data synchronization method according to any one of claims 1 to 7 when running.

10. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data synchronization method described in any one of claims 1 to 7.

Images