A processing method and device based on distributed batch processing

Abstract

The present invention discloses a processing method and device based on decentralized batch processing, and relates to the field of computer technology. A specific implementation of the method includes: performing online transaction processing on account data; performing decentralized batch processing on account data, and performing online transaction processing on account data during decentralized batch processing; performing online daily switching, synchronizing data snapshots used for online transactions and data snapshots used for batch processing, and performing online transaction processing on account data during data synchronization; performing batch daily switching, and unloading data based on the data snapshots used for batch processing. This implementation can solve the technical problem that batch processing takes a long time and affects user experience.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a processing method and device based on distributed batch processing. Background Art

[0002] As competition in the financial market intensifies, in order to attract more customers, banks must provide customers with more convenient all-weather services, allowing customers to trade anytime and anywhere. For the bank's back-end system, 24 / 7 stable operation is essential.

[0003] The traditional transaction mode of the banking system is usually divided into online transactions during the day and batch processing at night. Since batch processing takes a lot of time, banks usually start batch processing at a certain time at night (such as 22:00 Beijing time) to centrally process all data in batches, assuming that the day has ended. During this period, users can no longer initiate transactions, or they can initiate transactions but the default is that the transaction data is for the next day. During batch processing, online services need to be suspended for a long time, which seriously affects the availability of the system, brings great inconvenience to customers, and causes a bad user experience. In some business systems, it may also cause financial losses or credit problems to customers.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0005] 1) Batch processing is centralized, the time window is difficult to control, and it takes a long time;

[0006] 2) The system pressure is high, affecting system availability;

[0007] 3) Failure to complete the batch run in time will affect the downstream dependent systems and peripheral systems, which may cause delays in data reception or even failure of reception in the downstream systems. This is especially true when the data volume is large, such as during holidays, when traffic bursts occur, and data congestion may occur easily.

[0008] 4) During batch processing, online services need to be suspended for a long time, during which time users cannot conduct transactions, affecting the user experience. Summary of the invention

[0009] In view of this, an embodiment of the present invention provides a processing method and device based on decentralized batch processing to solve the technical problem that batch processing takes a long time and affects the user experience.

[0010] To achieve the above object, according to one aspect of an embodiment of the present invention, a processing method based on decentralized batch processing is provided, comprising:

[0011] Perform online transaction processing on account data;

[0012] Perform decentralized batch processing on account data, and during the decentralized batch processing, perform online transaction processing on the account data;

[0013] Perform online daily switching, synchronize the data snapshots used for online trading and the data snapshots used for batch processing, and perform online transaction processing on account data during data synchronization;

[0014] Perform batch daily switching and unload data based on the data snapshot used for batch processing.

[0015] Optionally, online transaction processing is performed on the account data, including:

[0016] Receive online transaction requests for accounts;

[0017] Performing online transaction processing on the account data of the account based on the data snapshot used for online transactions;

[0018] Set the processing status of the account to re-run batch, and update the account date of the account to the online date;

[0019] The data snapshot used by the online transaction is updated based on the processing result of the online transaction.

[0020] Optionally, decentralized batch processing of account data is performed, including:

[0021] Accounts whose transaction dates are less than or equal to the batch date are screened out, and the account data of each of the accounts is batch processed.

[0022] Optionally, batch processing is performed on the account data of each of the accounts, including:

[0023] For each account, determine whether the processing status of the account is a re-run batch;

[0024] If yes, synchronize the data snapshot used for online trading to the data snapshot used for batch processing, then perform batch processing based on the data snapshot used for batch processing, set the processing status of the account to batch run, and update the data snapshot used for batch processing based on the batch processing result;

[0025] If not, batch processing is performed based on the data snapshot used for the batch processing, the processing status of the account is set to batch run, and the data snapshot used for the batch processing is updated based on the batch processing result.

[0026] Optionally, synchronize the data snapshots used by online transactions and the data snapshots used by batch processing, including:

[0027] For each account, determine whether the processing status of the account is batch completed;

[0028] If yes, synchronizing the data snapshot used for batch processing to the data snapshot used for online trading, and setting the processing status of the account to synchronized;

[0029] If not, batch processing is performed on the account data of the account, and then the data snapshot used for the batch processing is synchronized with the data snapshot used for the online transaction, and the processing status of the account is set to synchronized.

[0030] Optionally, during data synchronization, online transaction processing is performed on the account data, including:

[0031] During data synchronization, receive online transaction requests from accounts;

[0032] Determining whether the processing status of the account is synchronized;

[0033] If yes, then perform online transaction processing on the account data of the account based on the data snapshot used for online transaction, set the processing status of the account to re-run batch, update the account date of the account to the online date, and update the data snapshot used for online transaction based on the processing result of online transaction;

[0034] If not, synchronize the data snapshot used for online transactions and the data snapshot used for batch processing, then perform online transaction processing on the account data of the account based on the data snapshot used for online transactions, set the processing status of the account to re-run batch, update the account date of the account to the online date, and update the data snapshot used for online transactions based on the processing result of the online transaction.

[0035] Optionally, unload data based on the data snapshot used for batch processing, including:

[0036] Unload data to data warehouses or downstream systems based on the data snapshot used for batch processing.

[0037] In addition, according to another aspect of an embodiment of the present invention, a processing device based on decentralized batch processing is provided, comprising:

[0038] An online processing module for performing online transaction processing on account data;

[0039] A batch processing module, for performing decentralized batch processing on the account data, and performing online transaction processing on the account data during the decentralized batch processing;

[0040] The synchronization module is used to perform online daily switching, synchronize the data snapshots used for online transactions and the data snapshots used for batch processing, and perform online transaction processing on account data during data synchronization;

[0041] The data unloading module is used to perform batch daily switching and unload data based on the data snapshot used for batch processing.

[0042] Optionally, the online processing module is further used for:

[0043] Receive online transaction requests for accounts;

[0044] Performing online transaction processing on the account data of the account based on the data snapshot used for online transactions;

[0045] Set the processing status of the account to re-run batch, and update the account date of the account to the online date;

[0046] The data snapshot used by the online transaction is updated based on the processing result of the online transaction.

[0047] Optionally, the batch processing module is further used for:

[0048] Accounts whose transaction dates are less than or equal to the batch date are screened out, and the account data of each of the accounts is batch processed.

[0049] Optionally, the batch processing module is further used for:

[0050] For each account, determine whether the processing status of the account is a re-run batch;

[0051] If yes, synchronize the data snapshot used for online trading to the data snapshot used for batch processing, then perform batch processing based on the data snapshot used for batch processing, set the processing status of the account to batch run, and update the data snapshot used for batch processing based on the batch processing result;

[0052] If not, batch processing is performed based on the data snapshot used for the batch processing, the processing status of the account is set to batch run, and the data snapshot used for the batch processing is updated based on the batch processing result.

[0053] Optionally, the synchronization module is further used for:

[0054] For each account, determine whether the processing status of the account is batch completed;

[0055] If yes, synchronizing the data snapshot used for batch processing to the data snapshot used for online trading, and setting the processing status of the account to synchronized;

[0056] If not, batch processing is performed on the account data of the account, and then the data snapshot used for the batch processing is synchronized with the data snapshot used for the online transaction, and the processing status of the account is set to synchronized.

[0057] Optionally, the synchronization module is further used for:

[0058] During data synchronization, receive online transaction requests from accounts;

[0059] Determining whether the processing status of the account is synchronized;

[0060] If yes, then perform online transaction processing on the account data of the account based on the data snapshot used for online transaction, set the processing status of the account to re-run batch, update the account date of the account to the online date, and update the data snapshot used for online transaction based on the processing result of online transaction;

[0061] If not, synchronize the data snapshot used for online transactions and the data snapshot used for batch processing, then perform online transaction processing on the account data of the account based on the data snapshot used for online transactions, set the processing status of the account to re-run batch, update the account date of the account to the online date, and update the data snapshot used for online transactions based on the processing result of the online transaction.

[0062] Optionally, the data unloading module is further used for:

[0063] Unload data to data warehouses or downstream systems based on the data snapshot used for batch processing.

[0064] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, including:

[0065] one or more processors;

[0066] a storage device for storing one or more programs,

[0067] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any one of the above embodiments.

[0068] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in any one of the above embodiments is implemented.

[0069] One embodiment of the above invention has the following advantages or beneficial effects: because the technical means of distributing batch processing of account data before online daily switching and performing online transaction processing of account data during distributing batch processing is adopted, the technical problem of batch processing taking a long time in the prior art that affects user experience is overcome. The embodiment of the present invention is based on data synchronization and isolation technology, which can make batch processing and online transactions independent of each other and quickly synchronize data after online daily switching. Moreover, the present invention can combine business and system characteristics to effectively screen out data that can be distributed in batches in advance, thereby erasing the obvious boundary between online and batch.

[0070] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0072] Figure 1 is a schematic diagram of the main flow of a processing method based on decentralized batch processing according to an embodiment of the present invention;

[0073] Figure 2 is a schematic diagram of the main process of a processing method based on decentralized batch processing according to a reference embodiment of the present invention;

[0074] Figure 3 is a schematic diagram of the main flow of a processing method based on decentralized batch processing according to another reference embodiment of the present invention;

[0075] Figure 4 is a schematic diagram of the main flow of a processing method based on decentralized batch processing according to yet another reference embodiment of the present invention;

[0076] Figure 5 is a schematic diagram of the main flow of a processing method based on decentralized batch processing according to yet another reference embodiment of the present invention;

[0077] Figure 6 is a schematic diagram of main modules of a processing device based on decentralized batch processing according to an embodiment of the present invention;

[0078] Figure 7 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0079] Figure 8 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0081] Figure 1 Schematic diagram of the main process of the processing method based on decentralized batch processing according to an embodiment of the present invention. Figure 1 As shown, the processing method based on decentralized batch processing may include:

[0082] Step 101, perform online transaction processing on account data.

[0083] An account can initiate an online transaction request. After receiving the online transaction request, the system will perform online transaction processing on the account data of the account.

[0084] Optionally, step 101 may include: receiving an online transaction request from an account; performing online transaction processing on the account data of the account based on the data snapshot used for online transactions; setting the processing status of the account to rerun batch, and updating the account date of the account to the online date; and updating the data snapshot used for online transactions based on the processing result of the online transaction. On day T, when the system receives an online transaction request initiated by an account, it performs online transaction processing based on the O-file data. After the online transaction processing is completed, the processing status of the account is set to rerun batch, and the account date of the account is updated to the online date as a screening condition for subsequent batch processing. Moreover, after completing the online transaction processing, the system updates the O-file data based on the processing result of the online transaction to ensure that the O-file data is the latest.

[0085] In the embodiments of the present invention, file O (Online) refers to a data snapshot used for online transactions; file B (Batch) refers to a data snapshot used for batch processing; online date refers to a system date used for online transactions; batch date refers to a system date used for batch processing; account date refers to the date on which the most recent online account transaction of an account is recorded; account processing status includes: synchronized (initial status), batch run, and re-batch run; wherein synchronized indicates that the data in file B and file O are consistent; batch run indicates that the batch processing on day T has been completed; and re-batch runs indicates that the batch processing on day T needs to be reprocessed.

[0086] Step 102, performing decentralized batch processing on the account data, and during the decentralized batch processing, performing online transaction processing on the account data.

[0087] The system will initiate several batch processes in a decentralized manner depending on the situation. Each batch is processed based on the B-file data and processed vertically based on the account dimension. Optionally, decentralized batch processing is performed on the account data, including: screening out accounts whose account date is less than or equal to the batch date, and batch processing is performed on the account data of each of the accounts. In an embodiment of the present invention, the account date of the account is used as a screening condition to screen out accounts whose account date is less than or equal to the batch date, and batch processing is performed separately for each of these screened accounts. For a certain account during decentralized batch processing, since batch processing and online processing are based on different data files, online financial transactions are allowed to occur without any blockage or impact.

[0088] The core idea of ​​the embodiment of the present invention is to decentralize and advance the centralized batch processing. It is crucial to choose which data to batch process and when to batch process. Data should be selected according to the system and business characteristics. The selection criteria may include: 1. Clearly define the data that can be batch processed in advance. Taking the account transaction system as an example, after some financial transactions occur, it is often necessary to transfer the transaction flow to the downstream system. For this part of the data, it can be extracted from the batch processing and changed to an online asynchronous push mode, such as asynchronous push through the MQ message middleware, and provide night file services to fill in the failed messages during the day to achieve double insurance; 2. According to the business characteristics, some data is likely to not have account transactions during the day, and batch processing in advance is likely not to cause re-running of batches, so the benefits of batch processing of data in advance will be more optimistic. Taking the loan business as an example, there is a high probability that no financial transactions will occur on non-repayment days for accounts that are not overdue, and this part of the data can be batch processed in advance.

[0089] According to the above standards, the data whose synchronized account date is less than or equal to the batch date is selected, and then the data volume and batch processing time point are determined according to the system's busyness, and each account is batch processed vertically. For the re-run batch data, secondary batch processing is performed at a certain time interval according to the transaction time. In an embodiment of the present invention, most of the data is dispersed in this stage for batch processing, and only a small amount of data needs to be processed after the online daily cut. Therefore, based on different data snapshots, the embodiment of the present invention uses advance batches and dispersed batches instead of centralized batch processing, screens data that meets the standards, and performs batch processing at the appropriate time.

[0090] Optionally, the account data of each of the accounts is batch processed, including: for each account, determining whether the processing status of the account is a re-run batch; if so, synchronizing the data snapshot used for online transactions to the data snapshot used for batch processing, and then performing batch processing based on the data snapshot used for batch processing, setting the processing status of the account to batch run, and updating the data snapshot used for batch processing based on the batch processing result; if not, performing batch processing based on the data snapshot used for batch processing, setting the processing status of the account to batch run, and updating the data snapshot used for batch processing based on the batch processing result. For accounts that are re-run batches, first synchronize the data corresponding to file O to file B, and then perform batch processing based on file B, and after the batch processing is completed, set the processing status of the account to batch run. For accounts that are not re-run batches, batch processing is directly performed based on file B, and after the batch processing is completed, set the processing status of the account to batch run.

[0091] The embodiment of the present invention no longer adopts a centralized batch processing mode, but adopts an advance and decentralized approach based on different data snapshots, which can effectively solve the technical problems existing in the prior art and provide users with all-weather uninterrupted online services.

[0092] Step 103, perform online daily switching, synchronize the data snapshot used for online transactions and the data snapshot used for batch processing, and perform online transaction processing on the account data during data synchronization.

[0093] Optionally, after the natural day is switched (after 00:00), an online day switch can be performed, and during the data synchronization period, the online date is switched to T+1 day, and the batch date is still T day. During this period, the data of file B and file O are synchronized. It should be pointed out that the online day switch can be set as needed, for example, it can be performed at 22:00, 23:00, or 00:00.

[0094] Optionally, synchronizing the data snapshot used for online transactions and the data snapshot used for batch processing includes: for each account, determining whether the processing status of the account is batched; if so, synchronizing the data snapshot used for batch processing to the data snapshot used for online transactions, and setting the processing status of the account to synchronized; if not, batch processing the account data of the account, and then synchronizing the data snapshot used for batch processing to the data snapshot used for online transactions, and setting the processing status of the account to synchronized. This stage is mainly used for synchronizing B-file data with O-file, and batch processing of a small amount of data that has not been batched or re-batched. During data synchronization, batch running is triggered for data that has not been batched, and data synchronization is performed after the batch running is completed.

[0095] Optionally, during data synchronization, online transaction processing is performed on account data, including: during data synchronization, receiving an online transaction request for an account; determining whether the processing status of the account is synchronized; if so, performing online transaction processing on the account data of the account based on the data snapshot used for online transactions, setting the processing status of the account to rerun batch, updating the account date of the account to the online date, and updating the data snapshot used for online transactions based on the processing result of the online transactions; if not, synchronizing the data snapshot used for online transactions and the data snapshot used for batch processing, and then performing online transaction processing on the account data of the account based on the data snapshot used for online transactions, setting the processing status of the account to rerun batch, updating the account date of the account to the online date, and updating the data snapshot used for online transactions based on the processing result of the online transactions. Online transactions occurring at this stage belong to T+1 day. First determine whether the data has been synchronized. For accounts that have not been synchronized, trigger data synchronization first. After the synchronization is completed, online transaction processing is performed based on the O file. The account date is updated to the online date (i.e. T+1 day), and the processing status of the account is updated to re-run batch, which is used as the screening condition for the dispersed batch processing on T+1 day.

[0096] Step 104, perform batch daily switching, and unload data based on the data snapshot used for batch processing.

[0097] Optionally, step 104 may include: unloading data to a data warehouse or downstream system based on the data snapshot used for batch processing. After completing data synchronization, batch daily switching is performed to fully enter the second day's trading phase, and the data unloading to the data warehouse or downstream system is completed at the same time. After all accounts have completed data synchronization, batch daily switching is performed, the batch date is switched to T+1 day, and the system is completely switched to T+1 day. After entering the data download phase, data is unloaded to the data warehouse or downstream system based on file B. Due to data isolation, online transactions on T+1 day will not be affected at all.

[0098] The embodiments of the present invention can avoid the time consumption caused by too much data volume by decentralized processing instead of centralized processing, and ensure that all accounts are successfully batched before the day cut. By decentralized batching, since some accounts have been batch processed, data congestion can be avoided at a centralized point in time. The system has to process a large amount of business every day, and these business batches will not all be accumulated at the same time point for centralized batch processing, which can reduce system pressure, improve system availability, and ensure that downstream dependent systems can operate smoothly after batching without causing data delays.

[0099] According to the various embodiments described above, it can be seen that the embodiments of the present invention solve the technical problem in the prior art that batch processing takes a long time and affects the user experience by distributing batch processing of account data before online daily switching, and performing online transaction processing on account data during the distributed batch processing. The embodiments of the present invention are based on data synchronization and isolation technology, which can make batch processing and online transactions independent of each other, and quickly synchronize data after online daily switching. Moreover, the present invention can combine business and system characteristics to effectively filter out data that can be distributed in batches in advance, thereby erasing the obvious boundary between online and batch.

[0100] Therefore, the embodiments of the present invention solve the technical problem of needing to suspend online services for a long time during batch processing, provide account-level batch processing, and perform vertical batch processing. Even if there are online transactions at the day-cut time in the evening, a single-account batch run will be triggered first. The batch run ensures data consistency, and a single account can ensure timely system response. The process is transparent to users, achieving true 7*24 service (referring to the system providing services uninterruptedly or tending to be uninterrupted), providing users with 7*24 hours of all-weather uninterrupted service, and improving user experience.

[0101] Figure 2 Schematic diagram of the main process of a processing method based on decentralized batch processing according to a reference embodiment of the present invention. As another embodiment of the present invention, Figure 2 As shown, the step of batch processing the account data of each of the accounts may include:

[0102] Step 201, determine whether the processing status of the account is a re-run batch; if so, execute step 202; if not, execute step 203.

[0103] Step 202, synchronizing the data snapshot used for online transactions with the data snapshot used for batch processing.

[0104] Step 203: performing batch processing based on the data snapshot used for the batch processing.

[0105] Step 204, set the processing status of the account to batch run.

[0106] Step 205: Update the data snapshot used for the batch processing based on the batch processing result.

[0107] In addition, the specific implementation content of the processing method based on decentralized batch processing in a reference embodiment of the present invention has been described in detail in the processing method based on decentralized batch processing described above, so the repeated content will not be described here.

[0108] Figure 3Schematic diagram of the main process of a processing method based on distributed batch processing according to another reference embodiment of the present invention. As another embodiment of the present invention, Figure 3 As shown, the steps of data synchronization may include:

[0109] Step 301, determine whether the processing status of the account is batch run; if not, execute step 302; if so, execute step 303.

[0110] Step 302: batch process the account data of the account.

[0111] Step 303: Synchronize the data snapshot used for batch processing with the data snapshot used for online trading.

[0112] Step 304, setting the processing status of the account to synchronized;

[0113] In addition, the specific implementation content of the processing method based on distributed batch processing in another reference embodiment of the present invention has been described in detail in the processing method based on distributed batch processing described above, so the repeated content will not be described here.

[0114] Figure 4 Schematic diagram of the main flow of a processing method based on distributed batch processing according to another reference embodiment of the present invention. As another embodiment of the present invention, Figure 4 As shown, the steps of performing online transaction processing on account data may include:

[0115] Step 401, receiving an online transaction request from an account.

[0116] Step 402 , determining whether the processing status of the account is synchronized; if not, executing step 403 ; if yes, executing step 404 .

[0117] Step 403, synchronizing the data snapshot used for online transactions and the data snapshot used for batch processing.

[0118] Step 404, performing online transaction processing on the account data of the account based on the data snapshot used for online transaction.

[0119] Step 405, setting the processing status of the account to re-run batch, and updating the account date of the account to the online date.

[0120] Step 406: Update the data snapshot used for the online transaction based on the processing result of the online transaction.

[0121] In addition, the specific implementation content of the processing method based on distributed batch processing in another reference embodiment of the present invention has been described in detail in the processing method based on distributed batch processing described above, so the repeated content will not be described here.

[0122] Figure 5 Schematic diagram of the main flow of a processing method based on distributed batch processing according to another reference embodiment of the present invention. As another embodiment of the present invention, Figure 5 As shown, the steps of performing online transaction processing on account data may include:

[0123] When the system receives a financial transaction for a certain account, it performs online transaction processing based on the O file data. After the online transaction processing is completed, the processing status of the account is set to re-run batch, and the account date of the account is updated to the online date as a screening condition for subsequent batch processing. Moreover, after the online transaction processing is completed, the system updates the O file data based on the processing results of the online transaction to ensure that the O file data is the latest.

[0124] Then, decentralized batch processing begins. Specifically, the system will initiate several decentralized batch processing depending on the situation. Each batch is processed based on the data in file B, and vertical processing is performed based on the account dimension. For accounts that need to be re-run, the data corresponding to file O is first synchronized to file B, and then batch processing is performed based on file B. After the batch processing is completed, the processing status of the account is set to batch run. For accounts that are not re-run, batch processing is performed directly based on file B. After the batch processing is completed, the processing status of the account is set to batch run. During decentralized batch processing, account data can still be processed for online transactions. For a certain account during decentralized batch processing, since batch processing and online processing are based on different data files, online financial transactions are allowed to occur without any blockage or impact.

[0125] Next, when the online day cut-off point is reached, the online day is switched and the data synchronization period begins. The online date is switched to T+1 day, and the batch date is still T day. During this period, the data of files B and O are synchronized. This stage is mainly used to synchronize file B data with file O, and to batch process a small amount of data that has not been batched or re-batched. During data synchronization, the batch run is triggered for data that has not been batched, and data synchronization is performed after the batch run is completed. The online transactions that occur in this stage belong to T+1 day. First, determine whether the data has been synchronized. For accounts that have not been synchronized, data synchronization is triggered first. After the synchronization is completed, online transaction processing is performed based on file O, and the account date is updated to the online date (i.e., T+1 day), and the account processing status is updated to re-batch, which is used as a screening condition for the decentralized batch processing of T+1 day.

[0126] Finally, when the batch day cut point is reached, the batch day switch is performed. After all accounts have completed data synchronization, the batch day cut is performed, the batch date is switched to T+1 day, and the system is completely switched to T+1 day. After entering the data download stage, the data is unloaded to the data warehouse or downstream system based on file B. Due to data isolation, the online transaction on T+1 day will not be affected at all.

[0127] In addition, the specific implementation content of the processing method based on distributed batch processing in another reference embodiment of the present invention has been described in detail in the processing method based on distributed batch processing described above, so the repeated content will not be described here.

[0128] Figure 6 is a schematic diagram of main modules of a processing device based on decentralized batch processing according to an embodiment of the present invention, such as Figure 6 As shown, the processing device 600 based on decentralized batch processing includes an online processing module 601, a batch processing module 602, a synchronization module 603 and an unloading module 604; wherein, the online processing module 601 is used to perform online transaction processing on account data; the batch processing module 602 is used to perform decentralized batch processing on account data, and during the decentralized batch processing, perform online transaction processing on the account data; the synchronization module 603 is used to perform online daily switching, synchronize the data snapshot used for online transactions and the data snapshot used for batch processing, and during the data synchronization, perform online transaction processing on the account data; the unloading module 604 is used to perform batch daily switching, and unload data based on the data snapshot used for batch processing.

[0129] Optionally, the online processing module 601 is further used for:

[0130] Receive online transaction requests for accounts;

[0131] Performing online transaction processing on the account data of the account based on the data snapshot used for online transactions;

[0132] Set the processing status of the account to re-run batch, and update the account date of the account to the online date;

[0133] The data snapshot used by the online transaction is updated based on the processing result of the online transaction.

[0134] Optionally, the batch processing module 602 is further used for:

[0135] Accounts whose transaction dates are less than or equal to the batch date are screened out, and the account data of each of the accounts is batch processed.

[0136] Optionally, the batch processing module 602 is further used for:

[0137] For each account, determine whether the processing status of the account is a re-run batch;

[0138] If yes, synchronize the data snapshot used for online trading to the data snapshot used for batch processing, then perform batch processing based on the data snapshot used for batch processing, set the processing status of the account to batch run, and update the data snapshot used for batch processing based on the batch processing result;

[0139] If not, batch processing is performed based on the data snapshot used for the batch processing, the processing status of the account is set to batch run, and the data snapshot used for the batch processing is updated based on the batch processing result.

[0140] Optionally, the synchronization module 603 is further used for:

[0141] For each account, determine whether the processing status of the account is batch completed;

[0142] If yes, synchronizing the data snapshot used for batch processing to the data snapshot used for online trading, and setting the processing status of the account to synchronized;

[0143] If not, batch processing is performed on the account data of the account, and then the data snapshot used for the batch processing is synchronized with the data snapshot used for the online transaction, and the processing status of the account is set to synchronized.

[0144] Optionally, the synchronization module 603 is further used for:

[0145] During data synchronization, receive online transaction requests from accounts;

[0146] Determining whether the processing status of the account is synchronized;

[0147] If yes, then perform online transaction processing on the account data of the account based on the data snapshot used for online transaction, set the processing status of the account to re-run batch, update the account date of the account to the online date, and update the data snapshot used for online transaction based on the processing result of online transaction;

[0148] If not, synchronize the data snapshot used for online transactions and the data snapshot used for batch processing, then perform online transaction processing on the account data of the account based on the data snapshot used for online transactions, set the processing status of the account to re-run batch, update the account date of the account to the online date, and update the data snapshot used for online transactions based on the processing result of the online transaction.

[0149] Optionally, the data unloading module 604 is further used for:

[0150] Unload data to data warehouses or downstream systems based on the data snapshot used for batch processing.

[0151] According to the various embodiments described above, it can be seen that the embodiments of the present invention solve the technical problem in the prior art that batch processing takes a long time and affects the user experience by distributing batch processing of account data before online daily switching, and performing online transaction processing on account data during the distributed batch processing. The embodiments of the present invention are based on data synchronization and isolation technology, which can make batch processing and online transactions independent of each other, and quickly synchronize data after online daily switching. Moreover, the present invention can combine business and system characteristics to effectively filter out data that can be distributed in batches in advance, thereby erasing the obvious boundary between online and batch.

[0152] It should be noted that the specific implementation content of the processing device based on distributed batch processing in the present invention has been described in detail in the processing method based on distributed batch processing described above, so the repeated content will not be described again here.

[0153] Figure 7 An exemplary system architecture 700 is shown to which a processing method based on decentralized batch processing or a processing device based on decentralized batch processing according to an embodiment of the present invention can be applied.

[0154] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, 703, network 704 and server 705. Network 704 is used to provide a medium for communication links between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0155] Users can use terminal devices 701, 702, and 703 to interact with server 705 through network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0156] The terminal devices 701 , 702 , and 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0157] Server 705 may be a server that provides various services, such as a backend management server (for example only) that provides support for shopping websites browsed by users using terminal devices 701, 702, and 703. The backend management server may analyze and process the received data such as item information query requests, and feed back the processing results to the terminal device.

[0158] It should be noted that the processing method based on distributed batch processing provided in the embodiment of the present invention is generally executed by the server 705 , and accordingly, the processing device based on distributed batch processing is generally arranged in the server 705 .

[0159] It should be understood that Figure 7 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0160] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system 800 of a terminal device suitable for implementing an embodiment of the present invention. Figure 8 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0161] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0162] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0163] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present invention are executed.

[0164] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0165] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0166] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes an online processing module, a batch processing module, a synchronization module and an unloading module, wherein the names of these modules do not constitute a limitation on the modules themselves in some cases.

[0167] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device implements the following methods: performing online transaction processing on account data; performing decentralized batch processing on account data, and performing online transaction processing on account data during decentralized batch processing; performing online daily switching, synchronizing the data snapshot used for online trading and the data snapshot used for batch processing, and performing online transaction processing on account data during data synchronization; performing batch daily switching, and unloading data based on the data snapshot used for batch processing.

[0168] According to the technical solution of the embodiment of the present invention, the technical means of distributing batch processing of account data before online daily switching and performing online transaction processing of account data during the distributing batch processing are adopted, so the technical problem of batch processing taking a long time in the prior art that affects the user experience is overcome. The embodiment of the present invention is based on data synchronization and isolation technology, which can make batch processing and online transactions independent of each other, and quickly synchronize data after online daily switching. Moreover, the present invention can combine business and system characteristics, and can effectively filter out data that can be distributed in batches in advance, thereby erasing the obvious boundary between online and batch.

[0169] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A processing method based on distributed batch processing, characterized in that: include: Perform online transaction processing on account data; Perform decentralized batch processing on account data, and during the decentralized batch processing, perform online transaction processing on the account data; Perform online daily switching, synchronize the data snapshots used for online trading and the data snapshots used for batch processing, and perform online transaction processing on account data during data synchronization; Perform batch daily switching and unload data based on the data snapshot used for batch processing; Decentralized batch processing of account data, including: Screen out accounts whose transaction date is less than or equal to the batch date, and perform batch processing on the account data of each of the accounts; Batch processing is performed on the account data of each of the accounts, including: For each account, determine whether the processing status of the account is a re-run batch; wherein the re-run batch indicates a batch that needs to be reprocessed; If yes, synchronize the data snapshot used for online trading to the data snapshot used for batch processing, then perform batch processing based on the data snapshot used for batch processing, set the processing status of the account to batch run, and update the data snapshot used for batch processing based on the batch processing result; If not, batch processing is performed based on the data snapshot used for the batch processing, the processing status of the account is set to batch run, and the data snapshot used for the batch processing is updated based on the batch processing result; Synchronizes data snapshots used by online transactions and batch processing, including: For each account, determine whether the processing status of the account is batch completed; If yes, synchronizing the data snapshot used for batch processing to the data snapshot used for online trading, and setting the processing status of the account to synchronized; If not, batch processing is performed on the account data of the account, and then the data snapshot used for the batch processing is synchronized with the data snapshot used for the online transaction, and the processing status of the account is set to synchronized.

2. The method according to claim 1, characterized in that Perform online transaction processing on account data, including: Receive online transaction requests for accounts; Performing online transaction processing on the account data of the account based on the data snapshot used for online transactions; Set the processing status of the account to re-run batch, and update the account date of the account to the online date; The data snapshot used by the online transaction is updated based on the processing result of the online transaction.

3. The method according to claim 1, characterized in that During data synchronization, online transaction processing is performed on account data, including: During data synchronization, receive online transaction requests from accounts; Determining whether the processing status of the account is synchronized; If yes, then perform online transaction processing on the account data of the account based on the data snapshot used for online transaction, set the processing status of the account to re-run batch, update the account date of the account to the online date, and update the data snapshot used for online transaction based on the processing result of online transaction; If not, synchronize the data snapshot used for online transactions and the data snapshot used for batch processing, then perform online transaction processing on the account data of the account based on the data snapshot used for online transactions, set the processing status of the account to re-run batch, update the account date of the account to the online date, and update the data snapshot used for online transactions based on the processing result of the online transaction.

4. The method according to claim 1, characterized in that: Unload data based on the data snapshot used by batch processing, including: Unload data to data warehouses or downstream systems based on the data snapshot used for batch processing.

5. A processing device based on distributed batch processing, characterized in that: include: An online processing module for performing online transaction processing on account data; A batch processing module, for performing decentralized batch processing on the account data, and performing online transaction processing on the account data during the decentralized batch processing; The synchronization module is used to perform online daily switching, synchronize the data snapshots used for online transactions and the data snapshots used for batch processing, and perform online transaction processing on account data during data synchronization; The data unloading module is used for batch daily switching and unloading data based on the data snapshot used for batch processing; The batch processing module is also used to: Screen out accounts whose transaction date is less than or equal to the batch date, and perform batch processing on the account data of each of the accounts; The batch processing module is also used to: For each account, determine whether the processing status of the account is a re-run batch; wherein the re-run batch indicates a batch that needs to be reprocessed; If yes, synchronize the data snapshot used for online trading to the data snapshot used for batch processing, then perform batch processing based on the data snapshot used for batch processing, set the processing status of the account to batch run, and update the data snapshot used for batch processing based on the batch processing result; If not, batch processing is performed based on the data snapshot used for the batch processing, the processing status of the account is set to batch run, and the data snapshot used for the batch processing is updated based on the batch processing result; The synchronization module is also used for: For each account, determine whether the processing status of the account is batch completed; If yes, synchronizing the data snapshot used for batch processing to the data snapshot used for online trading, and setting the processing status of the account to synchronized; If not, batch processing is performed on the account data of the account, and then the data snapshot used for the batch processing is synchronized with the data snapshot used for the online transaction, and the processing status of the account is set to synchronized.

6. The device according to claim 5, characterized in that The online processing module is also used for: Receive online transaction requests for accounts; Performing online transaction processing on the account data of the account based on the data snapshot used for online transactions; Set the processing status of the account to re-run batch, and update the account date of the account to the online date; The data snapshot used by the online transaction is updated based on the processing result of the online transaction.

7. The device according to claim 5, characterized in that The synchronization module is also used for: During data synchronization, receive online transaction requests from accounts; Determining whether the processing status of the account is synchronized; If yes, then perform online transaction processing on the account data of the account based on the data snapshot used for online transaction, set the processing status of the account to re-run batch, update the account date of the account to the online date, and update the data snapshot used for online transaction based on the processing result of online transaction; If not, synchronize the data snapshot used for online transactions and the data snapshot used for batch processing, then perform online transaction processing on the account data of the account based on the data snapshot used for online transactions, set the processing status of the account to re-run batch, update the account date of the account to the online date, and update the data snapshot used for online transactions based on the processing result of the online transaction.

8. The device according to claim 5, characterized in that The data unloading module is also used for: Unload data to data warehouses or downstream systems based on the data snapshot used for batch processing.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images