A data synchronization method and device based on real-time transaction
By introducing an independent liquidation library in the fund sales system, using polling timing and batch asynchronous synchronization strategies, the problem of conflict between real-time transactions and batch clearing data is solved, 24-hour uninterrupted transactions and efficient clearing are achieved, and user experience is improved and Liquidation efficiency.
Patent Information
- Application Number
- CN202311776248.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-21
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2043-12-21
AI Technical Summary
The existing fund sales system is prone to data conflicts during real-time transactions and batch clearing, resulting in interruption of transaction services and inability to provide 7*24-hour uninterrupted services, reducing user experience and clearing efficiency.
Using the data synchronization method based on real-time transactions, by introducing a second database (independent clearing library) into the system, using polling timing synchronization strategy and batch asynchronous incremental synchronization strategy, real-time detection of liquidation status, avoiding data conflicts, and ensuring liquidation period Supports real-time transactions without interruption.
It realizes synchronization between transaction data and clearing data, supports 24-hour uninterrupted transactions, improves user experience, and ensures that the transaction data and clearing confirmation data meet business expectations, improving the efficiency of clearing services.
Smart Images

Figure CN117648381B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data synchronization, and in particular to a data synchronization method and device based on real-time transactions. Background Art
[0002] The fund sales system needs to accept investors' fund trading applications in real time (real-time trading), and also needs to serve as a clearing system to confirm the batch clearing of daily fund trading data. The user's position data needs to be modified during both real-time trading and batch clearing confirmation (for example, if a user initiates a share redemption, it will trigger a share freeze, and if the batch clearing is successful in the previous day's redemption confirmation, it needs to be unfrozen and the confirmed shares will be deducted), which will lead to data conflicts. In order to avoid this conflict, the industry generally reserves a certain time window for suspending trading services, and restarts trading after completing the clearing work.
[0003] At present, although batch clearing is generally carried out at night when user transactions are not active, it is still impossible to provide users with 7*24 hours of uninterrupted service. Users cannot trade during the period when the trading service is closed. The clearing service efficiency is low, which reduces the user experience. It is also impossible to guarantee that the transaction data and the data after the clearing confirmation are completed meet the business expectations. Summary of the invention
[0004] The present invention provides a data synchronization method and device based on real-time transactions, which realizes synchronization of transaction data and clearing data, avoids conflicts between real-time transactions and batch clearing data, ensures uninterrupted support of real-time transactions during clearing, and improves user experience.
[0005] In order to solve the above technical problems, an embodiment of the present invention provides a data synchronization method based on real-time transactions, including:
[0006] Writing data generated by real-time transactions in the current system into the first database, detecting the clearing processing status in real time, and obtaining clearing status parameters;
[0007] When the clearing state parameter is in a non-clearing state, the incremental data of the first database is synchronized to the second database through a polling timing synchronization strategy;
[0008] When the liquidation state parameter is the liquidation state, or when the liquidation state parameter is converted from the non-liquidation state to the liquidation state, a liquidation operation is performed on the second database, and the data change row corresponding to the data table of the liquidation operation is recorded in real time and stored in the table;
[0009] When the liquidation state parameter is converted from the liquidation state to the non-liquidation state, the liquidation result data is obtained, and based on the liquidation result data, the liquidation result data in the second database is synchronized to the first database through a batch asynchronous incremental synchronization strategy.
[0010] In implementing the embodiment of the present invention, the data generated by real-time transactions in the current system are written into the first database, the clearing processing status is detected in real time, and the clearing status parameters are obtained; when the clearing status parameters are in a non-clearing state, the incremental data of the first database is synchronized to the second database through a polling timing synchronization strategy; when the clearing status parameters are in a clearing state, or when the clearing status parameters are converted from a non-clearing state to a clearing state, a clearing operation is performed on the second database, and the data change row storage table corresponding to the data table of the clearing operation is recorded in real time; when the clearing status parameters are converted from a clearing state to a non-clearing state, the clearing result data is obtained, and based on the clearing result data, the clearing result data in the second database is synchronized to the first database through a batch asynchronous incremental synchronization strategy. In the current system, transaction data and clearing data are synchronized, and corresponding synchronization and clearing operations are performed under different clearing states to avoid conflicts between real-time transactions and batch clearing data, ensure that real-time transactions are supported without interruption during clearing, support 24-hour uninterrupted transactions, and effectively improve user experience.
[0011] As a preferred solution, the data change rows corresponding to the data table of the real-time record clearing operation are stored in the table, specifically:
[0012] Add corresponding row storage tables to the data tables that need to be processed during the liquidation period; the table structure of the row storage table includes user number, operation type, original table primary key field, synchronization status, liquidation date and the fields changed in the original table;
[0013] When performing a clearing operation on the current data table in the second database, the current row storage table corresponding to the current data table is obtained in real time, the operation type corresponding to the clearing operation is synchronously inserted into the current row storage table, and the current change data is stored in the field corresponding to the current row storage table to obtain the data change row storage table corresponding to the current data table.
[0014] As a preferred solution, obtain the liquidation result data, specifically:
[0015] When all clearing operations on the second database are completed, the data change row storage table corresponding to the data table of all clearing operations is obtained to obtain the total data change row storage table;
[0016] All data change rows are stored in the table as clearing result data.
[0017] As a preferred solution, the incremental data of the first database is synchronized to the second database through a polling timing synchronization strategy, specifically:
[0018] Obtain the timestamp of the first database updated by the database operation during the real-time transaction, obtain the first timestamp, obtain the current timestamp of the second database, and obtain the second timestamp;
[0019] When the first timestamp is greater than the timestamp of the second database, the field data in the first database that is inconsistent with the second database is updated to the second database by polling.
[0020] As a preferred solution, based on the liquidation result data, the liquidation result data in the second database is synchronized to the first database through a batch asynchronous incremental synchronization strategy, specifically:
[0021] Based on the table of all data changes, the user numbers whose data changed during the liquidation period are counted to obtain all the changed user numbers;
[0022] All changed user numbers are sorted, and based on a preset batch number, all sorted changed user numbers are batched to obtain batch numbers corresponding to all changed user numbers;
[0023] Create a clearing custom table and store the user number, batch number and processing status corresponding to all changed user numbers in the clearing custom table;
[0024] Based on the clearing custom table, all data change rows in the second database are synchronized in batches to the first database.
[0025] As a preferred solution, based on the clearing custom table, all data change rows in the second database are synchronized to the first database in batches, specifically:
[0026] Based on the clearing custom table, the data tables corresponding to the data change row storage table in the current batch are synchronized in sequence in one database transaction, and the clearing operation types of all user numbers in the current batch are updated to the second database using the incremental synchronization method; wherein the clearing operation types include insert operations, update operations and delete operations.
[0027] As a preferred solution, the clearing operation types of all user numbers in the current batch are updated to the second database using an incremental synchronization method, specifically:
[0028] Obtain the clearing operation type of the current user number in the current batch in sequence, and use the clearing operation type of the current user number as the current clearing operation type;
[0029] When the current liquidation operation type is an insert operation, the first original data corresponding to the first database is found according to the original table primary key field in the table storing all data change rows, and the first original data is inserted into the data table corresponding to the first database;
[0030] When the current liquidation operation type is an update operation, the second original data corresponding to the first database is found according to the original table primary key field in the table of all data change rows, and the corresponding fields of the second original data are updated in the same incremental manner;
[0031] When the current liquidation operation type is a delete operation, the third original data corresponding to the first database is found out according to the original table primary key field in the table of all data change rows, and the third original data in the first database is deleted.
[0032] In order to solve the same technical problem, an embodiment of the present invention further provides a data synchronization processing device based on real-time transactions, including: the data synchronization processing device is used to implement a data synchronization method based on real-time transactions, including: a real-time transaction detection module, a polling timing synchronization module, a parallel processing module and an asynchronous incremental synchronization module;
[0033] The real-time transaction detection module is used to write the data generated by the real-time transaction in the current system into the first database, detect the clearing processing status in real time, and obtain the clearing status parameters;
[0034] The polling timing synchronization module is used to synchronize the incremental data of the first database to the second database through the polling timing synchronization strategy when the clearing state parameter is a non-clearing state;
[0035] The parallel processing module is used to perform a clearing operation on the second database when the clearing state parameter is in the clearing state, or when the clearing state parameter is converted from a non-clearing state to a clearing state, and to record in real time the data change rows corresponding to the data table of the clearing operation;
[0036] The asynchronous incremental synchronization module is used to obtain the liquidation result data when the liquidation state parameter is converted from the liquidation state to the non-liquidation state, and based on the liquidation result data, synchronize the liquidation result data in the second database to the first database through a batch asynchronous incremental synchronization strategy.
[0037] In order to solve the same technical problem, an embodiment of the present invention further provides a computer device, including a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, a data synchronization method based on real-time transactions is implemented.
[0038] In order to solve the same technical problem, an embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which implements a data synchronization method based on real-time transactions when the computer program is executed by a processor.
[0039] Compared with the prior art, the present invention has the advantages of:
[0040] (1) Based on the first database (trading database), the second database (independent clearing database) is introduced. Based on the polling timing synchronization strategy and batch asynchronous incremental synchronization strategy for data synchronization between the two databases, it is ensured that real-time trading is supported 24 hours a day during the clearing period without interruption, trading services are not interrupted, and are not affected by the system's clearing operations, thereby improving user experience.
[0041] (2) Through the batch asynchronous incremental synchronization strategy, based on numerical incremental updates, we avoid data synchronization results that do not meet business expectations due to direct overwriting, ensure that transaction data and clearing confirmation are completed in accordance with business expectations, and ensure that the efficiency of clearing services does not decrease.
[0042] Other features and advantages of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 : A flow chart of an embodiment of a data synchronization method based on real-time transactions provided by the present invention;
[0044] Figure 2 : A data synchronization mechanism diagram of an embodiment of a data synchronization method based on real-time transactions provided by the present invention;
[0045] Figure 3 : A structural diagram of an embodiment of a data synchronization processing device based on real-time transactions provided by the present invention. DETAILED DESCRIPTION
[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0047] Embodiment 1
[0048] Please refer to Figure 1 , is a flow chart of a data synchronization method based on real-time transactions provided by an embodiment of the present invention. The data synchronization method includes steps 101 to 104, each of which is as follows:
[0049] Step 101: Write the data generated by the real-time transactions in the current system into the first database, detect the clearing processing status in real time, and obtain the clearing status parameters.
[0050] In this embodiment, the current system is the execution subject of the present invention. The current system is a comprehensive system that undertakes both real-time transactions and transaction data clearing, such as a fund sales system. The first database is the transaction database of the system, which is used to store transaction data after users conduct transactions in the system. The second database is the clearing database of the system, which is used for system clearing processing. Exemplarily, when the current system is a fund sales system, the first database is the transaction database, and the second database is the clearing database, a specific mechanism is used to synchronize data between the transaction database and the clearing database, such as the data synchronization mechanism. Figure 2As shown, the current clearing state of the current system is marked by a system parameter, namely, the clearing state parameter, which includes 0 and 1, 0 represents the non-clearing state, and 1 represents the clearing state. The non-clearing period is the period when the clearing state parameter is 0, and the clearing period is the period when the clearing state parameter is 1. The real-time transaction data is updated to the transaction library through the transaction background, and the data in the clearing library is cleared through the clearing background.
[0051] For example, when the current system is a fund sales system, the first database is a transaction database, and the second database is a clearing database, in order to improve the disadvantage that the fund sales system cannot perform transactions and clearing at the same time, the original database of the fund sales system is defined as a transaction database, and a database specifically used for clearing is copied based on the transaction database, namely the clearing database. When the transaction system in the fund sales system accepts real-time transactions from users, the data is stored in the transaction database. Users query data from the transaction database for the latest results. The clearing system in the fund sales system performs data clearing processing based on the clearing database. The clearing database is used for data processing during the clearing period to ensure that transactions and clearing do not interfere with each other.
[0052] Step 102: When the clearing state parameter is a non-clearing state, the incremental data of the first database is synchronized to the second database through a polling timing synchronization strategy.
[0053] In this embodiment, exemplarily, when the current system is a fund sales system, the first database is a transaction database, and the second database is a clearing database, when the clearing system in the fund sales system does not perform clearing processing, the clearing status is 0, that is, the clearing status parameter is a non-clearing status. At this time, the data generated by real-time transactions are all written into the transaction database, and continuously synchronized to the clearing database through an asynchronous mechanism. The asynchronous synchronization mechanism from the transaction database to the clearing database adopts a polling timing synchronization strategy, and synchronizes the incremental data of the transaction database to the clearing database through a polling program at a regular interval.
[0054] It should be noted that during the non-clearing period, since there will be no data changes in the clearing system, when the data in the trading library changes, it can be directly updated to the clearing library based on the changed data in the trading library. There are two specific situations that need to be considered. The first is the newly added data in the trading library, and the second is the updated data in the trading library. The data synchronization processing method of the polling timing synchronization strategy is used to implement the polling timing synchronization strategy using the poller and ActiveMQ.
[0055] Optionally, step 102 is specifically as follows: obtain the timestamp of the first database updated by the database operation during the real-time transaction, obtain the first timestamp, obtain the current timestamp of the second database, and obtain the second timestamp; when the first timestamp is greater than the timestamp of the second database, update the field data in the first database that is inconsistent with the second database to the second database by polling.
[0056] In this embodiment, illustratively, when the first database is a trading library and the second database is a clearing library, a non-empty timestamp field is added to each table. When processing real-time transactions, all database DML operations must update this field to the current system timestamp (systimestamp). If the data exists in both the trading library and the clearing library, but the timestamp of the trading library is greater than the timestamp of the clearing library, that is, the first timestamp is greater than the second timestamp, it means that an update has occurred in the trading library, and all inconsistent fields can be updated to the clearing library. Taking the trequest-trading application form as an example, trade is the trading library, liq is the clearing library, the primary key of trequest is vc_requestno-application number, there are three business fields, namely column1, column2 and column3, the timestamp field is ts_lastmodify, and the synchronization sql of the polling program is implemented as follows:
[0057] Merge into liq.trequest t1
[0058] using(select*from trade.trequest)t2
[0059] on(t1.vc_requestno=t2.vc_requestno)
[0060] when matched then
[0061] update set t1.column1=t2.column1,t1.column2=t2.column2,t1.column3=t2.column3 where t1.ts_lastmodify>t2.ts_lastmodify
[0062] when not matched then
[0063] insert(vc_requestno,column1,column2,column3,ts_lastmodify)values(t2.vc_requestno,t2.column1,t2.column2,t2.column3,t2.ts_lastmodify);
[0064] Step 103: When the liquidation state parameter is the liquidation state, or when the liquidation state parameter is converted from the non-liquidation state to the liquidation state, a liquidation operation is performed on the second database, and the data change row corresponding to the data table of the liquidation operation is recorded in real time and stored in the table.
[0065] In this embodiment, exemplarily, when the current system is a fund sales system, the first database is a transaction database, and the second database is a clearing database, when the clearing system in the fund sales system starts clearing, the clearing status parameter is first marked as 1, that is, the clearing status parameter is the clearing status, or the clearing status parameter is converted from a non-clearing status to a clearing status. At this time, real-time transactions will continue to be written to the transaction database, but the polling program determines that it has entered the clearing period based on the clearing status parameter, and therefore suspends synchronization of the transaction database to the clearing database, that is, stops synchronizing the incremental data of the first database to the second database.
[0066] Optionally, a data table corresponding to a data table that records clearing operations in real time stores data changes, specifically:
[0067] A corresponding row storage table is added to the data table that needs to be processed during the liquidation period; the table structure of the row storage table includes the user number, operation type, original table primary key field, synchronization status, liquidation date and the field of the original table change; when performing a liquidation operation on the current data table in the second database, the current row storage table corresponding to the current data table is obtained in real time, the operation type corresponding to the liquidation operation is synchronously inserted into the current row storage table, and the current change data is stored in the corresponding field of the current row storage table, so as to obtain the data change row storage table corresponding to the current data table.
[0068] In this embodiment, for example, when the first database is a transaction database and the second database is a liquidation database, a corresponding xxx_delta table (row storage table) is added to the data table that needs to be processed during the liquidation period, and the table structure includes the following fields (vc_custno-user number, c_opttype-operation type, original table primary key field, c_dealstatus-synchronization status, vc_liqdate-liquidation date, and other fields of the original table that change during the liquidation period). When a liquidation operation such as insert, update, or delete is performed on a data table during the liquidation process, a record with c_opttype of INSERT, UPDATE, or DELETE is synchronously inserted into the corresponding delta table, and other corresponding fields are recorded.
[0069] It should be noted that the delta table is a row-stored table attached to the column-stored table. If the delta table is enabled when creating a column-stored table, in some cases, the data inserted into the column-stored table will also be saved in row-stored form.
[0070] Step 104: when the liquidation status parameter is converted from the liquidation status to the non-liquidation status, the liquidation result data is obtained, and based on the liquidation result data, the liquidation result data in the second database is synchronized to the first database through batch asynchronous incremental synchronization.
[0071] In this embodiment, for example, when the current system is a fund sales system, the first database is a transaction database, and the second database is a clearing database, when the clearing system ends clearing, that is, the clearing state parameter is converted from the clearing state to the non-clearing state, the clearing result data during the clearing period is synchronized back to the transaction database by the method of batch asynchronous incremental synchronization strategy. Then the clearing state parameter is re-marked as 0. At this time, the polling program recognizes that the clearing system has settled the clearing based on the state parameter, and then restarts the timing synchronization program to perform the polling timing synchronization strategy.
[0072] Optionally, obtain liquidation result data, specifically:
[0073] When all clearing operations on the second database are completed, the data change row storage table corresponding to the data tables of all clearing operations is obtained to obtain the total data change row storage table; the total data change row storage table is used as the clearing result data.
[0074] In this embodiment, exemplarily, when the first database is a transaction database and the second database is a clearing database, when the clearing is completed, the records in the delta table (a table storing all data changes) represent all data changes during the clearing period, and the data is synchronized to the transaction database based on the delta table.
[0075] It should be noted that from the perspective of a single user, the update of the liquidation result data should ensure atomicity, otherwise the subscription application may not be confirmed successfully, but the share has been increased. In order to ensure atomicity, all data of the same user must be submitted in the same database transaction to be synchronized back to the trading library. The system adopts batch multi-threaded processing by user number, and each batch is submitted once after all the table data of the user is synchronized. The number of users processed in each batch should not be too many or too few. If the number of users in a batch is too small, too many submissions will reduce the synchronization efficiency; if the number of users in a batch is too large, the processing time of a batch is too long. When users are conducting real-time transactions at the same time, there will be a long time of lag. Therefore, through the batch asynchronous incremental synchronization strategy, batch processing is carried out to avoid lag.
[0076] Optionally, based on the liquidation result data, the liquidation result data in the second database is synchronized to the first database through a batch asynchronous incremental synchronization strategy, specifically including steps 1041 to 1044, each step is as follows:
[0077] Step 1041: Based on the table of all data changes, count the user numbers whose data changed during the clearing period and obtain all changed user numbers.
[0078] In this embodiment, users whose data has been updated during the clearing period are collected based on the delta table (a table storing all data change rows) corresponding to all data tables. Taking tstaticshsare_delta as an example, the following SQL is used to collect statistics on user numbers.
[0079] Merge into tliqcustom t1
[0080] using(select distinct a.vc_custno from liq.tstaticshare_delta a)t2
[0081] on(t1.vc_custno=t2.vc_custno)
[0082] when not matched then
[0083] insert(vc_custno,vc_dealstatus,vc_syncbatchno)values
[0084] (t2.vc_custno,'0','-');
[0085] Step 1042: sort all the changed user numbers, and divide all the sorted changed user numbers into batches based on a preset batch quantity to obtain batch numbers corresponding to all the changed user numbers.
[0086] In this embodiment, all user numbers (all changed user numbers) are divided into batches and sorted by user numbers. For example, if the number of batches is 50, 50 users are taken as one batch, and the batch number is updated to vc_syncbatchno batch number.
[0087] Step 1043: Create a clearing customization table, and store the user numbers, batch numbers, and processing statuses corresponding to all changed user numbers in the clearing customization table.
[0088] In this embodiment, a tliqcustom table (transaction customization table) is created to store the user ID whose data has been changed. The table contains vc_custno user ID, vc_syncbatchno batch number and c_dealstatus processing status.
[0089] Step 1044: Based on the clearing custom table, synchronize all data change row storage tables in the second database to the first database in batches.
[0090] In this embodiment, batch data synchronization is performed based on the tliqcustom table (transaction customization table). Each data table is synchronized in turn in one database transaction. Each data table update is based on the corresponding delta table. The INSERT insertion, UPDATE update and DELETE deletion operations of all user numbers in the batch are updated to the transaction library according to the incremental synchronization method.
[0091] Optionally, step 1044 is specifically as follows: based on the clearing custom table, the data tables corresponding to the data change row storage tables in the current batch are synchronized in sequence in one database transaction, and the clearing operation types of all user numbers in the current batch are updated to the second database using an incremental synchronization method; wherein the clearing operation types include insert operations, update operations, and delete operations.
[0092] Optionally, the clearing operation types of all user numbers in the current batch are updated to the second database using an incremental synchronization method, specifically including steps S1 to S4, each of which is as follows:
[0093] S1: sequentially obtain the clearing operation type of the current user number in the current batch, and use the clearing operation type of the current user number as the current clearing operation type.
[0094] S2: When the current liquidation operation type is an insert operation, the first original data corresponding to the first database is found according to the original table primary key field in the table storing all data change rows, and the first original data is inserted into the data table corresponding to the first database.
[0095] In this embodiment, for the insert operation (INSERT operation), the original data is found according to the primary key in the delta table, and the original data is inserted into the corresponding table of the transaction library.
[0096] S3: When the current clearing operation type is an update operation, the second original data corresponding to the first database is found according to the original table primary key field in the table of all data change rows, and the corresponding fields of the second original data are updated in the same incremental manner.
[0097] In this embodiment, for the update operation (UPDATE operation), the numerical value that needs to be updated during the clearing process has the same field in the delta table, which is used to record the incremental value relative to the original value (for example, the original value is 10, and it is updated to 30, then the corresponding field in the delta table is recorded as 20). After the clearing is completed, the original data of the transaction library is found according to the primary key of the delta table, and the corresponding fields of the original data are updated with the same incremental update.
[0098] S4: When the current liquidation operation type is a delete operation, the third original data corresponding to the first database is found according to the original table primary key field in the table of all data change rows, and the third original data in the first database is deleted.
[0099] In this embodiment, for the deletion operation (DELETE operation), the original data is found according to the primary key in the delta table, and the original data in the transaction database is also deleted.
[0100] In implementing the embodiment of the present invention, the data generated by real-time transactions in the current system are written into the first database, the clearing processing status is detected in real time, and the clearing status parameters are obtained; when the clearing status parameters are in a non-clearing state, the incremental data of the first database is synchronized to the second database through a polling timing synchronization strategy; when the clearing status parameters are in a clearing state, or when the clearing status parameters are converted from a non-clearing state to a clearing state, a clearing operation is performed on the second database, and the data change row storage table corresponding to the data table of the clearing operation is recorded in real time; when the clearing status parameters are converted from a clearing state to a non-clearing state, the clearing result data is obtained, and based on the clearing result data, the clearing result data in the second database is synchronized to the first database through a batch asynchronous incremental synchronization strategy. In the current system, transaction data and clearing data are synchronized, and corresponding synchronization and clearing operations are performed under different clearing states to avoid conflicts between real-time transactions and batch clearing data, ensure that real-time transactions are supported without interruption during clearing, support 24-hour uninterrupted transactions, and effectively improve user experience.
[0101] Compared with the prior art, the present invention has the advantages of:
[0102] (1) Based on the first database (trading database), the second database (independent clearing database) is introduced. Based on the polling timing synchronization strategy and batch asynchronous incremental synchronization strategy for data synchronization between the two databases, it is ensured that real-time trading is supported 24 hours a day during the clearing period without interruption, trading services are not interrupted, and are not affected by the system's clearing operations, thereby improving user experience.
[0103] (2) Through the batch asynchronous incremental synchronization strategy, based on numerical incremental updates, we avoid data synchronization results that do not meet business expectations due to direct overwriting, ensure that transaction data and clearing confirmation are completed in accordance with business expectations, and ensure that the efficiency of clearing services does not decrease.
[0104] Embodiment 2
[0105] Accordingly, see Figure 3 , Figure 3 1 is a schematic diagram of the structure of a second embodiment of a data synchronization processing device based on real-time transactions provided by the present invention. Figure 3As shown, the data synchronization processing device is used to implement a data synchronization method based on real-time transactions. The data synchronization processing device based on real-time transactions includes a real-time transaction detection module 301, a polling timing synchronization module 302, a parallel processing module 303 and an asynchronous incremental synchronization module 304;
[0106] The real-time transaction detection module 301 is used to write the data generated by the real-time transaction in the current system into the first database, detect the clearing processing status in real time, and obtain the clearing status parameters;
[0107] The polling timing synchronization module 302 is used to synchronize the incremental data of the first database to the second database through the polling timing synchronization strategy when the clearing state parameter is a non-clearing state;
[0108] The parallel processing module 303 is used to perform a clearing operation on the second database when the clearing state parameter is in the clearing state, or when the clearing state parameter is converted from the non-clearing state to the clearing state, and to record in real time the data change rows corresponding to the data table of the clearing operation;
[0109] The asynchronous incremental synchronization module 304 is used to obtain the liquidation result data when the liquidation state parameter is converted from the liquidation state to the non-liquidation state, and based on the liquidation result data, synchronize the liquidation result data in the second database to the first database through a batch asynchronous incremental synchronization strategy.
[0110] In implementing the embodiment of the present invention, the data generated by real-time transactions in the current system are written into the first database, the clearing processing status is detected in real time, and the clearing status parameters are obtained; when the clearing status parameters are in a non-clearing state, the incremental data of the first database is synchronized to the second database through a polling timing synchronization strategy; when the clearing status parameters are in a clearing state, or when the clearing status parameters are converted from a non-clearing state to a clearing state, a clearing operation is performed on the second database, and the data change row storage table corresponding to the data table of the clearing operation is recorded in real time; when the clearing status parameters are converted from a clearing state to a non-clearing state, the clearing result data is obtained, and based on the clearing result data, the clearing result data in the second database is synchronized to the first database through a batch asynchronous incremental synchronization strategy. In the current system, transaction data and clearing data are synchronized, and corresponding synchronization and clearing operations are performed under different clearing states to avoid conflicts between real-time transactions and batch clearing data, ensure that real-time transactions are supported without interruption during clearing, support 24-hour uninterrupted transactions, and effectively improve user experience.
[0111] The above-mentioned data synchronization processing device based on real-time transactions can implement a data synchronization processing method based on real-time transactions in the above-mentioned method embodiment. The options in the above-mentioned method embodiment are also applicable to this embodiment and will not be described in detail here. The rest of the contents of the embodiment of this application can refer to the contents of the above-mentioned method embodiment, and will not be repeated in this embodiment.
[0112] In addition, an embodiment of the present application further provides a computer device, which includes a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the steps in any of the above method embodiments are implemented.
[0113] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.
[0114] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.
Claims
1. A data synchronization method based on real-time transactions, characterized in that: include: Writing data generated by real-time transactions in the current system into the first database, detecting the clearing processing status in real time, and obtaining clearing status parameters; When the liquidation state parameter is a non-liquidation state, synchronizing the incremental data of the first database to the second database through a polling timing synchronization strategy; When the liquidation state parameter is the liquidation state, or when the liquidation state parameter is converted from the non-liquidation state to the liquidation state, a liquidation operation is performed on the second database, and a row storage table corresponding to a data table of the liquidation operation is recorded in real time, specifically: a corresponding row storage table is added to the data table to be processed during the liquidation period; the table structure of the row storage table includes a user number, an operation type, a primary key field of the original table, a synchronization state, a liquidation date, and a field changed in the original table; When performing a clearing operation on the current data table in the second database, a current row storage table corresponding to the current data table is obtained in real time, an operation type corresponding to the clearing operation is synchronously inserted into the current row storage table, and current change data is stored in a field corresponding to the current row storage table, so as to obtain a data change row storage table corresponding to the current data table; When the liquidation state parameter is converted from the liquidation state to the non-liquidation state, liquidation result data is obtained, and based on the liquidation result data, the liquidation result data in the second database is synchronized to the first database through a batch asynchronous incremental synchronization strategy; wherein, the obtaining of the liquidation result data is specifically: when all liquidation operations on the second database are completed, the data change row storage table corresponding to the data table of all liquidation operations is obtained to obtain the row storage table of all data changes; and the row storage table of all data changes is used as the liquidation result data.
2. The data synchronization method based on real-time transactions according to claim 1, characterized in that: The step of synchronizing the incremental data of the first database to the second database through the polling timing synchronization strategy is specifically as follows: Obtaining a timestamp of the first database updated by a database operation during a real-time transaction, obtaining a first timestamp, obtaining a current timestamp of the second database, and obtaining a second timestamp; When the first timestamp is greater than the timestamp of the second database, the field data in the first database that is inconsistent with the second database is updated to the second database by polling.
3. The data synchronization method based on real-time transactions according to claim 1, characterized in that: Based on the liquidation result data, synchronizing the liquidation result data in the second database to the first database through a batch asynchronous incremental synchronization strategy is specifically: Based on the data change row storage table, the user numbers whose data changed during the liquidation period are counted to obtain all changed user numbers; Sorting all the changed user numbers, and dividing all the sorted changed user numbers into batches based on a preset batch number, to obtain batch numbers corresponding to all the changed user numbers; Creating a clearing customization table, and storing the user numbers, the batch numbers, and the processing statuses corresponding to all the changed user numbers in the clearing customization table; Based on the clearing custom table, all the data change row storage tables in the second database are synchronized in batches to the first database.
4. The data synchronization method based on real-time transactions according to claim 3, characterized in that: Based on the clearing custom table, synchronizing all the data change row storage tables in the second database to the first database in batches is specifically as follows: Based on the clearing custom table, the data tables corresponding to the data change row storage table in the current batch are synchronized in sequence in one database transaction, and the clearing operation types of all user numbers in the current batch are updated to the second database using an incremental synchronization method; wherein the clearing operation types include insert operations, update operations, and delete operations.
5. The data synchronization method based on real-time transactions according to claim 4, characterized in that: The updating of the liquidation operation types of all user numbers in the current batch to the second database using an incremental synchronization method is specifically as follows: Sequentially obtain the clearing operation type of the current user number in the current batch, and use the clearing operation type of the current user number as the current clearing operation type; When the current clearing operation type is the insert operation, finding the first original data corresponding to the first database according to the original table primary key field in the all data change row storage table, and inserting the first original data into the data table corresponding to the first database; When the current liquidation operation type is the update operation, finding the second original data corresponding to the first database according to the original table primary key field in the all data change row storage table, and performing the same incremental update on the corresponding field of the second original data; When the current liquidation operation type is the deletion operation, the third original data corresponding to the first database is found out according to the original table primary key field in the table storing all data changes, and the third original data in the first database is deleted.
6. A data synchronization processing device based on real-time transactions, characterized in that: The data synchronization processing device is used to implement the data synchronization method based on real-time transactions as described in any one of claims 1 to 5, comprising: a real-time transaction detection module, a polling timing synchronization module, a parallel processing module and an asynchronous incremental synchronization module; The real-time transaction detection module is used to write the data generated by the real-time transaction in the current system into the first database, detect the clearing processing status in real time, and obtain the clearing status parameters; The polling timing synchronization module is used to synchronize the incremental data of the first database to the second database through the polling timing synchronization strategy when the clearing state parameter is a non-clearing state; The parallel processing module is used to perform a clearing operation on the second database when the clearing state parameter is a clearing state, or when the clearing state parameter is converted from the non-clearing state to the clearing state, and record in real time a data change row storage table corresponding to the data table of the clearing operation, specifically: adding a corresponding row storage table to the data table to be processed during the clearing period; the table structure of the row storage table includes a user number, an operation type, a primary key field of the original table, a synchronization state, a clearing date, and a field of change in the original table; when a clearing operation is performed on the current data table in the second database, a current row storage table corresponding to the current data table is obtained in real time, the operation type corresponding to the clearing operation is synchronously inserted into the current row storage table, and the current change data is stored in the field corresponding to the current row storage table, and the data change row storage table corresponding to the current data table is obtained; The asynchronous incremental synchronization module is used to obtain liquidation result data when the liquidation state parameter is converted from the liquidation state to the non-liquidation state, and based on the liquidation result data, synchronize the liquidation result data in the second database to the first database through a batch asynchronous incremental synchronization strategy; wherein, the obtaining of the liquidation result data is specifically: when all liquidation operations on the second database are completed, the data change row storage table corresponding to the data table of all liquidation operations is obtained to obtain the total data change row storage table; the total data change row storage table is used as the liquidation result data.
7. A computer device, characterized in that: It comprises a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the real-time transaction-based data synchronization method as claimed in any one of claims 1 to 5 is implemented.
8. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the data synchronization method based on real-time transactions as described in any one of claims 1 to 5.
