Multi-batch concurrent data clearing method, device and computer program product

Abstract

The present application discloses a multi-batch concurrent data clearing method, device and computer program product, which relates to the field of automated clearing technology. The method includes: receiving a data file to be split, and performing multi-dimensional data analysis on the data file according to a preset split setting rule; writing the data file after the multi-dimensional data analysis into the corresponding partition of the data table to obtain multiple batches of data to be cleared; reading batch automation configuration information and performing concurrent clearing on the data to be cleared according to the batch automation configuration information. The data is split into multiple batches that are easier to manage and process through preset split setting rules, thereby improving the efficiency of data clearing; the batch automation configuration information flexibly adjusts the processing flow of each batch to achieve fine and efficient control of the clearing process.

Description

Technical Field

[0001] The present application relates to the field of automated clearing technology, and in particular to a method, device and computer program product for clearing multiple batches of concurrent data. Background Art

[0002] After receiving data files from wealth management subsidiaries or fund companies (hereinafter referred to as TA), bank sales agencies need to process the data in a centralized manner. The existing processing method usually checks and records the transaction records in the data files together according to the same account number, and relies on manual operations in multiple links. The execution steps are usually not adjustable, and there is no ability to set processing steps according to different batches. The flexibility is poor, and there is a lack of effective data splitting and concurrent processing mechanisms. It is very difficult to process customer transaction records of tens of millions of transactions.

[0003] Therefore, how to flexibly clear large amounts of transaction data and improve the efficiency of data clearing has become an urgent problem to be solved in this application.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a multi-batch concurrent data clearing method, device and computer program product, aiming to solve the technical problem of how to flexibly clear large quantities of transaction data and improve the efficiency of data clearing.

[0006] To achieve the above purpose, the present application proposes a multi-batch concurrent data clearing method, the method comprising:

[0007] Receiving a data file to be split, and performing multi-dimensional data analysis on the data file according to a preset split setting rule;

[0008] Write the data files after multi-dimensional data analysis into the corresponding partitions of the data table to obtain multiple batches of data to be cleared;

[0009] Batch automation configuration information is read and concurrent clearing of the data to be cleared is performed according to the batch automation configuration information.

[0010] In one embodiment, before the step of receiving the data file to be split and performing multi-dimensional data analysis on the data file according to the preset split setting rule, the step further includes:

[0011] Acquire actual business scenario information and determine business dimensions, and configure split setting rules based on the actual business scenario information and the business dimensions.

[0012] In one embodiment, the step of reading batch automation configuration information and concurrently clearing the to-be-cleared data according to the batch automation configuration information includes:

[0013] Read batch automation configuration information and query historical processing time and processing quantity to calculate processing efficiency;

[0014] Predict the time required for concurrent liquidation based on the processing efficiency;

[0015] The data to be cleared is cleared concurrently according to the batch automation configuration information, and the time required for the concurrent clearing is visualized during the concurrent clearing process.

[0016] In one embodiment, the step of reading batch automation configuration information and concurrently clearing the to-be-cleared data according to the batch automation configuration information includes:

[0017] Reading batch automation configuration information, wherein the batch automation configuration information includes batch execution step configuration information;

[0018] The number of concurrent steps is set according to the batch execution step configuration information, and the data to be cleared is cleared concurrently by setting the number of concurrent steps.

[0019] In one embodiment, the step of reading batch automation configuration information and concurrently clearing the to-be-cleared data according to the batch automation configuration information includes:

[0020] Reading batch automation configuration information, wherein the batch automation configuration information includes batch dependency configuration information;

[0021] Determine the dependency relationship of the data to be cleared according to the batch dependency configuration information, and perform concurrent clearing on the data to be cleared with reference to the dependency relationship.

[0022] In one embodiment, the step of reading batch automation configuration information and concurrently clearing the to-be-cleared data according to the batch automation configuration information includes:

[0023] Reading batch automation configuration information, wherein the batch automation configuration includes step start time configuration information;

[0024] The data to be cleared is cleared concurrently according to the step start time configuration information.

[0025] In one embodiment, the step of reading batch automation configuration information and concurrently clearing the to-be-cleared data according to the batch automation configuration information includes:

[0026] Reading batch automation configuration information, wherein the batch automation configuration information includes an exception retry mechanism;

[0027] Performing concurrent clearing of the data to be cleared, and monitoring abnormalities in the concurrent clearing process;

[0028] When an abnormal situation is detected, the abnormal retry mechanism is triggered, and abnormal diagnosis is initiated for the data to be cleared according to the abnormal retry mechanism to obtain a diagnosis result;

[0029] Retry and concurrently clear the data to be cleared according to the diagnosis result.

[0030] In one embodiment, after the step of reading the batch automation configuration information and concurrently clearing the to-be-cleared data according to the batch automation configuration information, the step further includes:

[0031] Record concurrent liquidation results;

[0032] When the concurrent liquidation result is a liquidation failure, an alarm mechanism is triggered, and manual intervention is requested through the alarm mechanism.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes a multi-batch concurrent data clearing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the multi-batch concurrent data clearing method as described above.

[0034] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the multi-batch concurrent data clearing method described above.

[0035] One or more technical solutions proposed in this application have at least the following technical effects:

[0036] Receive the data file to be split, and perform multi-dimensional data analysis on the data file according to the preset split setting rules, write the data file after the multi-dimensional data analysis into different partitions of the data table, and realize batch management of data, providing a basis for subsequent concurrent clearing. Split the data into multiple batches that are easier to manage and process according to the preset split setting rules. Write the data file after the multi-dimensional data analysis into different partitions of the data table so that each batch of data is independent and can be processed in parallel, thereby improving the efficiency of data clearing; read the batch automation configuration information and perform concurrent clearing on the data to be cleared according to the batch automation configuration information. According to the batch automation configuration information, the processing flow of each batch can be flexibly adjusted, realizing fine control of the clearing process. Concurrent clearing further improves the processing speed and completes the clearing of large amounts of data in a shorter time. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 This is a flowchart of the first embodiment of the multi-batch concurrent data clearing method of the present application;

[0040] Figure 2 This is a flow chart of the second embodiment of the multi-batch concurrent data clearing method of the present application;

[0041] Figure 3 This is an example diagram of the estimated processing time in batch step processing;

[0042] Figure 4 This is an example diagram of parallel execution control of batch steps in this application;

[0043] Figure 5 This is an example diagram of the dependency relationship processing of different batch steps in this application;

[0044] Figure 6 Example diagram of setting processing time for step start time in batch;

[0045] Figure 7 Flow chart of the fourth embodiment of the method for clearing multiple batches of concurrent data in this application

[0046] Figure 8This is a schematic diagram of the module structure of a multi-batch concurrent data clearing device according to an embodiment of the present application;

[0047] Fig. 9 Schematic diagram of the device structure of the hardware operating environment involved in the multi-batch concurrent data clearing method in the embodiment of the present application.

[0048] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0050] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0051] The main solution of the embodiment of the present application is: receiving the data file to be split, and performing multi-dimensional data analysis on the data file according to the preset split setting rules. Among them, the preset split setting rules can be flexibly adjusted, and can be controllably implemented based on the dimensions of the business, such as splitting and setting by the wealth management subsidiary dimension, by the product dimension, by the business type dimension, by the file type dimension, etc.; writing the data file after multi-dimensional data analysis into the corresponding partition of the data table to obtain multiple batches of data to be cleared; reading the batch automation configuration information and performing concurrent clearing of the data to be cleared according to the batch automation configuration information. The batch automation configuration information allows the operation and maintenance personnel to make corresponding settings according to the actual business needs, such as the batch execution steps, batch start time, specific step start time, number of parallel operations, number of abnormal retries, etc.

[0052] In this embodiment, for ease of description, the following description is made by taking the identification of a multi-batch concurrent data clearing system as the execution subject.

[0053] The embodiment of the present application takes into account that after a bank agency receives a data file from a wealth management subsidiary or fund company (hereinafter referred to as TA), it needs to process the data in a centralized manner. The existing processing method usually performs data inspection and bookkeeping on the transaction records in the data file according to the same account number, and relies on manual operations in multiple links. The execution steps are usually not adjustable, and there is no ability to set processing steps according to different batches. The flexibility is poor, and there is a lack of effective data splitting and concurrent processing mechanisms. It is very difficult to process customer transaction records of tens of millions of transactions.

[0054] Therefore, the present application provides a solution, which receives a data file to be split, performs multi-dimensional data analysis on the data file according to preset split setting rules, writes the data file after multi-dimensional data analysis into different partitions of the data table, realizes batch management of data, and provides a basis for subsequent concurrent clearing. The data is split into multiple batches that are easier to manage and process through preset split setting rules. Writing the data file after multi-dimensional data analysis into different partitions of the data table makes the data of each batch independent and can be processed in parallel, thereby improving the efficiency of data clearing; reading the batch automation configuration information and performing concurrent clearing on the data to be cleared according to the batch automation configuration information. According to the batch automation configuration information, the processing flow of each batch can be flexibly adjusted, thereby realizing fine control of the clearing process. Concurrent clearing further improves the processing speed and completes the clearing of large amounts of data in a shorter time.

[0055] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a multi-batch concurrent data clearing system, etc. The multi-batch concurrent data clearing system includes a data splitting module and a batch processing automation control module. The following takes the multi-batch concurrent data clearing system as an example to illustrate this embodiment and the following embodiments.

[0056] Based on this, the present application embodiment provides a multi-batch concurrent data clearing method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the multi-batch concurrent data clearing method of the present application.

[0057] In this embodiment, the multi-batch concurrent data clearing method includes steps S10 to S30:

[0058] Step S10, receiving a data file to be split, and performing multi-dimensional data analysis on the data file according to a preset splitting setting rule;

[0059] The multi-batch concurrent data clearing system receives a large number of clearing data files from different sources (such as banks, financial institutions, etc.). These clearing data files contain various transaction data, customer information, account balances, etc., which need to be processed by the multi-batch concurrent data clearing system for clearing.

[0060] Preset splitting rules mean that before receiving the data files to be split, the multi-batch concurrent data clearing system has preset the splitting dimensions and rules according to business needs. Splitting rules determine how to split a large amount of data into smaller, manageable batches. For example, you can split based on dimensions such as transaction type, transaction amount, transaction time, or customer category.

[0061] Multi-dimensional data analysis refers to analyzing the received data files according to the preset splitting setting rules, and splitting the data into multiple small batches according to the splitting setting dimensions. This step involves data cleaning, format conversion, and data verification to ensure the accuracy and consistency of the data.

[0062] Step S20, writing the data file after multi-dimensional data analysis into the corresponding partition of the data table to obtain multiple batches of data to be cleared;

[0063] The multi-batch concurrent data clearing system maintains data tables internally to store the clearing data after parsing and splitting. These data tables are usually divided into multiple partitions, each partition corresponds to one or more data batches. This partitioning strategy helps optimize data query and processing performance. The multi-batch concurrent data clearing system maintains data tables internally to store the clearing data after parsing and splitting. These data tables are usually divided into multiple partitions, each partition corresponds to one or more data batches. This partitioning strategy helps optimize data query and processing performance.

[0064] After writing to the data table, the multi-batch concurrent data clearing system obtains multiple batches of data to be cleared. These batches of data can be processed in parallel or serially, depending on the settings of the batch automation configuration.

[0065] Step S30 , reading batch automation configuration information and concurrently clearing the data to be cleared according to the batch automation configuration information.

[0066] Before processing the data to be cleared, the multi-batch concurrent data clearing system needs to read the batch automation configuration information. The automation configuration information defines key information such as the processing steps, processing order, parallelism, and exception handling strategy for each batch.

[0067] The multi-batch concurrent data clearing system processes each batch of data to be cleared according to the read batch automation configuration information, executes specific clearing logic, updates the database status, and records the processing log.

[0068] Since the batch automation configuration information is adjustable, the multi-batch concurrent data clearing system can flexibly adjust the processing flow according to changes in business needs. For example, you can add new processing steps, modify the order or parallelism of existing steps, define new exception handling strategies, etc. This flexibility and scalability enables the multi-batch concurrent data clearing system to adapt to the ever-changing market environment and business needs.

[0069] This embodiment provides a multi-batch concurrent data clearing method, which receives a data file to be split, and performs multi-dimensional data analysis on the data file according to a preset split setting rule, and writes the data file after the multi-dimensional data analysis into different partitions of the data table, thereby realizing batch management of data and providing a basis for subsequent concurrent clearing. The data is split into multiple batches that are easier to manage and process through preset split setting rules. Writing the data file after the multi-dimensional data analysis into different partitions of the data table makes the data of each batch independent and can be processed in parallel, thereby improving the efficiency of data clearing; reading the batch automation configuration information and performing concurrent clearing on the data to be cleared according to the batch automation configuration information. According to the batch automation configuration information, the processing flow of each batch can be flexibly adjusted, thereby realizing fine control of the clearing process. Concurrent clearing further improves the processing speed and completes the clearing of large amounts of data in a shorter time.

[0070] Based on the first embodiment of the present application, the second embodiment of the present application is proposed. In the second embodiment of the present application, the same or similar contents as those of the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following.

[0071] On this basis, please refer to Figure 2 Before step S10, the multi-batch concurrent data clearing method further includes step S01:

[0072] Step S01, obtaining actual business scenario information and determining business dimensions, and configuring splitting setting rules according to the actual business scenario information and the business dimensions.

[0073] Gain in-depth understanding of the actual business scenario information that needs to be processed or supported. The actual business scenario information includes but is not limited to business type, data volume, transaction frequency, business priority, etc. The actual business scenario information is the basis for subsequent data processing and system configuration. By collecting and analyzing the actual business scenario information, it can ensure that the clearing plan can closely meet the actual business needs and improve the accuracy and efficiency of clearing.

[0074] After obtaining the actual business scenario information, it is necessary to determine the business dimensions, that is, according to which standards or characteristics to divide and process data. Business dimensions include wealth management subsidiary dimensions, product dimensions, business type dimensions, file type dimensions, etc. By reasonably determining the business dimensions, accurate data segmentation and efficient processing can be achieved.

[0075] It should be noted that when determining the business dimension, we should not only consider the current business needs, but also foresee the development trend and possible expansion direction. Therefore, a multi-level and multi-dimensional business dimension system can be constructed to meet the business needs and analysis needs at different levels.

[0076] Based on the actual business scenario information and the determined business dimensions, the splitting setting rules of the data can be flexibly configured. The splitting setting rules include but are not limited to the splitting dimensions, the number of batches, the amount of data in each batch, etc. By flexibly configuring the splitting setting rules, flexible and concurrent processing of data can be achieved, improving the clearing capacity and efficiency of the multi-batch concurrent data clearing system.

[0077] As the business develops and changes, the actual business scenario information and business dimensions may also need to be continuously adjusted and optimized. Therefore, configuring split setting rules should be considered as an ongoing iterative process rather than a one-time task. By regularly reviewing and updating business scenario information and business dimensions, you can ensure that data processing and system configuration are always in sync with business needs.

[0078] In this embodiment, by obtaining and analyzing information from actual business scenarios, it is possible to ensure that data processing is highly consistent with actual business needs, thereby avoiding business problems caused by improper configuration or processing mismatch. After determining the business dimensions, the split setting rules can be flexibly configured based on the business dimensions to split a large amount of data into multiple batches and multi-dimensional data for processing. This splitting method can optimize the data processing process, improve processing efficiency, and shorten processing time. Flexible configuration of split setting rules means that the system can be dynamically adjusted according to different business scenarios and needs without the need for large-scale transformation or upgrade of the system. This enhances the flexibility and adaptability of the system, enabling it to quickly adapt to new business scenarios or changes.

[0079] Based on the first embodiment and / or the second embodiment of the present application, the third embodiment of the present application is proposed. In the third embodiment of the present application, the same or similar contents as those of the above-mentioned first embodiment and / or the second embodiment can be referred to the above introduction, and will not be repeated in the following.

[0080] In a feasible implementation of this embodiment, the step S30 of reading batch automation configuration information and concurrently clearing the to-be-cleared data according to the batch automation configuration information may include steps S311 to S313:

[0081] Step S311, read batch automation configuration information and query historical processing time and processing quantity to calculate processing efficiency;

[0082] The following references Figure 3 , Figure 3 This is an example of the estimated processing time in batch step processing, such as Figure 3 As shown, the multi-batch concurrent data clearing system first reads the batch automation configuration. The batch automation configuration information includes key information such as batch execution steps, start time, and concurrent quantity. These settings ensure that the clearing process can be flexibly adjusted according to business needs.

[0083] Furthermore, the multi-batch concurrent data clearing system queries the historical processing time and processing quantity to calculate the processing efficiency, that is, the amount of data that the system can process per unit time. Based on the historical processing time and processing quantity, the average processing efficiency of each step is calculated, which is an important basis for predicting the time required for concurrent clearing.

[0084] Step S312, predicting the time required for concurrent liquidation according to the processing efficiency;

[0085] The time required for concurrent liquidation is predicted based on the calculated processing efficiency. This prediction is based on the current amount of data to be liquidated and the processing efficiency, which helps operation and maintenance personnel understand the liquidation progress and expected completion time.

[0086] Step S313, performing concurrent clearing on the data to be cleared according to the batch automation configuration information, and visualizing the time required for the concurrent clearing during the concurrent clearing process.

[0087] According to the batch automation configuration information, the data to be cleared is cleared concurrently. Concurrent clearing means that multiple batches can be cleared at the same time, greatly improving the processing efficiency. During the concurrent clearing process, the multi-batch concurrent data clearing system visualizes the time required for concurrent clearing. This function enables operation and maintenance personnel to monitor the clearing progress in real time and discover and solve potential problems in a timely manner.

[0088] In this embodiment, by querying the historical processing time and processing quantity to calculate the processing efficiency, the multi-batch concurrent data clearing system can have a more accurate estimate of the processing time of the current batch based on past experience, which helps to optimize resource allocation and avoid unnecessary waiting, thereby improving the overall processing efficiency.

[0089] Additionally, in another feasible implementation of this embodiment, the step S30 of reading batch automation configuration information and concurrently clearing the to-be-cleared data according to the batch automation configuration information may include steps S321 to S322:

[0090] Step S321, reading batch automation configuration information, wherein the batch automation configuration information includes batch execution step configuration information;

[0091] The batch execution step configuration information is one of the core contents of the batch automation configuration information, which defines in detail the specific steps and their order that need to be executed for each batch. For example, in the wealth management business, the batch execution steps may include wealth management data inspection, wealth management share processing, wealth management system fund processing, etc. The configuration of these steps allows the system to be personalized according to business needs, ensuring that the clearing process is both in line with business rules and can be executed efficiently.

[0092] By setting the number of concurrent steps, the multi-batch concurrent data clearing system can process multiple batches of clearing tasks at the same time. For example, the multi-batch concurrent data clearing system can be configured to execute the same steps in parallel between multiple batches, or to set dependencies between steps in different batches to achieve serial or parallel clearing processes. This concurrent processing capability can significantly improve clearing efficiency and shorten processing time.

[0093] The following references Figure 4 , Figure 4 This is an example diagram of parallel execution control of batch steps in this application, such as Figure 4 As shown in the figure, by setting the number of concurrent steps to 1, when multiple batches are executed concurrently, only one batch of step 2 is being executed, and the steps 2 in the other two batches are waiting in line. This scenario can alleviate the processing pressure of the system. For example, when step 2 is executed, it may cause the system memory or CPU to increase, so the number of concurrent steps can be controlled.

[0094] Step S322, setting the number of concurrent steps according to the batch execution step configuration information, and concurrently clearing the data to be cleared by setting the number of concurrent steps.

[0095] In this embodiment, the setting of the number of concurrent steps not only improves the liquidation efficiency, but also increases the flexibility of the process. The operation and maintenance personnel can dynamically adjust the number of concurrent steps according to actual business needs and system processing capabilities to optimize the liquidation process.

[0096] Additionally, in another feasible implementation of this embodiment, the step S30 of reading batch automation configuration information and concurrently clearing the to-be-cleared data according to the batch automation configuration information may include steps S331 to S332:

[0097] Step S331, reading batch automation configuration information, wherein the batch automation configuration information includes batch dependency configuration;

[0098] Batch automation configuration information contains a series of rules and parameters that each batch in the clearing process needs to follow. In particular, "batch automation configuration information" includes not only the configuration information of the batch execution steps (i.e., which specific processing steps need to be executed for each batch), but also the batch dependency configuration information.

[0099] Batch dependency configuration information is an important embodiment of the flexibility and accuracy of batch automation processes. It allows operation and maintenance personnel to define dependencies between different batches, ensuring that during the data clearing process, some batches must wait for other batches to complete specific steps before they can continue to execute. This dependency setting is particularly important for business scenarios with data association or sequence.

[0100] Step S332: Determine the dependency relationship of the data to be cleared according to the batch dependency configuration information, and perform concurrent clearing on the data to be cleared with reference to the dependency relationship.

[0101] Specifically, the multi-batch concurrent data clearing system will check whether there are other batches or steps that need to be waited for before each batch executes its processing steps based on the batch dependency configuration information. If there is a dependency, the multi-batch concurrent data clearing system will adjust the concurrent clearing strategy accordingly to ensure that the dependent batches or steps are executed first.

[0102] For example, suppose there are two batches A and B, where step 2 of batch B depends on step 2 of batch A. Then, during the concurrent clearing process, the multi-batch concurrent data clearing system will ensure that step 2 of batch A is completed before allowing step 2 of batch B to start. This dependency processing avoids conflicts and errors in the data clearing process and improves the accuracy and efficiency of clearing.

[0103] The following references Figure 5 , Figure 5 This is an example diagram of the dependency relationship processing of different batch steps in this application. Figure 5 As shown in the figure, set step 2 of batch b to depend on step 2 of batch a. Then when batch b executes step 2, it will check whether step 2 in batch a has been completed. If not, step 2 of batch b will wait until step 2 in batch a has been completed. This scenario can be used to set when there is data dependency in the split batches, or to execute the batches in serial mode.

[0104] In this embodiment, the data dependency between different batches is clarified by reading the batch dependency configuration information. The setting of such dependency enables the clearing process to proceed in the order of business logic, avoiding clearing errors caused by incorrect data sequence. Based on the understanding of batch dependency, the batches and steps of concurrent clearing are more accurately controlled, which helps to balance the system load and ensure that the clearing process is evenly distributed among multiple batches, thereby improving the overall clearing efficiency.

[0105] Additionally, in a feasible implementation of this embodiment, the step S30 of reading batch automation configuration information and concurrently clearing the to-be-cleared data according to the batch automation configuration information may include steps S341 to S342:

[0106] Step S341, reading batch automation configuration information, wherein the batch automation configuration information includes step start time configuration information;

[0107] Get the start time of each batch step by reading the information in the configuration file or database. These start times may be fixed (such as a specific time of day) or triggered based on other conditions (such as after the previous step is completed). The step start time configuration information usually includes the batch ID, step number, and start time (specific to hours, minutes, or even seconds).

[0108] Step S342: concurrently clear the data to be cleared according to the step start time configuration information.

[0109] Specifically, the multi-batch concurrent data clearing system checks whether the current time has reached the start time of a batch step. If the start time is reached, the system triggers the execution of the step and calls the corresponding clearing processing unit to process the data to be cleared. If there are multiple batch steps with the same or similar start times, the multi-batch concurrent data clearing system will execute these steps concurrently according to the setting of the number of concurrent steps.

[0110] like Figure 6 As shown, Figure 6 An example diagram of setting up processing for step start times in a batch. Figure 6 The start time of step 2 of batch b is set. Step 2 will only start to execute when the current time reaches the start time, otherwise it will keep waiting. This effectively controls the start time of the business. For example, if the large transaction time of the bank system is 9 am, the start time of the relevant fund processing steps can be set to start after 9 am.

[0111] In this embodiment, by accurately controlling the start time of the steps, the liquidation process is ensured to be executed at the optimal time, thereby improving the efficiency of liquidation. The step start time configuration allows personalized settings according to business needs, thereby enhancing the flexibility of the liquidation process. By accurately controlling the execution time and sequence of the liquidation steps, human intervention and errors are reduced, thereby improving the accuracy of liquidation.

[0112] Additionally, in a feasible implementation of this embodiment, the step S30 of reading batch automation configuration information and concurrently clearing the to-be-cleared data according to the batch automation configuration information may include steps S351 to S354:

[0113] Step S351, reading batch automation configuration information, wherein the batch automation configuration information includes an abnormal retry mechanism;

[0114] The multi-batch concurrent data clearing system reads the configuration information about the abnormal retry from the preset batch automation configuration. The configuration information of the abnormal retry usually includes the number of abnormal retries, the interval time of the retries, the conditions for triggering the retry, etc.

[0115] Step S352, concurrently clearing the data to be cleared, and monitoring the process of concurrent clearing for abnormalities;

[0116] The multi-batch concurrent data clearing system performs concurrent clearing of the data to be cleared according to the previously read batch execution step configuration. Concurrent clearing can significantly improve clearing efficiency and shorten clearing time. At the same time, the concurrent clearing process is monitored in real time to detect whether there are abnormal situations, such as data errors, some communication interruptions, occasional database anomalies and other systemic errors.

[0117] Step S353, when an abnormal situation is detected, the abnormal retry mechanism is triggered, and abnormal diagnosis is initiated for the data to be cleared according to the abnormal retry mechanism to obtain a diagnosis result;

[0118] When an abnormality is detected during the concurrent liquidation process, the abnormal retry mechanism is triggered immediately. According to the abnormal retry mechanism, the abnormality is diagnosed to determine the cause of the abnormality and possible solutions. The diagnosis process includes checking data files, system logs, network status, etc. to obtain detailed information about the abnormality.

[0119] Step S354: retry concurrent clearing of the data to be cleared according to the diagnosis result.

[0120] After obtaining the diagnosis results, the abnormality is retried in combination with the diagnosis results. The retry process includes re-reading the data file, re-executing the liquidation steps, adjusting the system parameters, etc. During the retry process, the liquidation process continues to be monitored to ensure that the abnormality has been resolved and the liquidation can proceed smoothly.

[0121] If the number of retries reaches the preset upper limit and the exception is still not resolved, the multi-batch concurrent data clearing system records the error information and notifies the operation and maintenance personnel to handle it manually.

[0122] In this embodiment, by reading the abnormal retry mechanism in the batch automation configuration, automatic monitoring, diagnosis and retry processing of abnormal situations in the concurrent liquidation process are achieved, which significantly improves the stability and success rate of liquidation and reduces the need for manual intervention.

[0123] Based on the above embodiments of the present application, a fourth embodiment of the present application is proposed. In the fourth embodiment of the present application, the same or similar contents as those of the above embodiments can be referred to the above introduction, and will not be described in detail later.

[0124] On this basis, please refer to Figure 7 , Figure 7 This is a schematic flow chart of the fourth embodiment provided in this application.

[0125] In this embodiment, after the step S30 of reading the batch automation configuration information and concurrently clearing the data to be cleared according to the batch automation configuration information, the steps S40 to S50 are also included:

[0126] Step S40, recording the concurrent liquidation results;

[0127] The multi-batch concurrent data clearing system records various data of each batch during the processing process, including but not limited to the number of processing, processing steps, amount of funds processed, estimated completion time, actual completion time, and whether there are any abnormal circumstances.

[0128] After each step of each batch is completed, the batch automation control module of the multi-batch concurrent data clearing system will register the execution results of the step, including key information such as start time, end time, and processing quantity, and write this information into the data table. In this way, the processing results of each batch will be recorded in detail.

[0129] Step S50: When the concurrent liquidation result is a liquidation failure, an alarm mechanism is triggered, and manual intervention is requested through the alarm mechanism.

[0130] When a certain step of a batch fails, the batch automation control module will immediately recognize the anomaly and trigger the alarm mechanism. The alarm mechanism will send the fault information to the preset operation and maintenance personnel through SMS, phone or other communication methods. After receiving the alarm notification, the operation and maintenance personnel can log in to the system to view detailed fault information, including the failed steps, error causes, operation reminders, etc. Based on this information, the operation and maintenance personnel can perform manual processing, such as retrying steps, adjusting configurations, or performing other necessary repair operations.

[0131] In this embodiment, by recording the liquidation results in detail and triggering the alarm mechanism in time when the liquidation fails, the reliability and maintainability of the liquidation process are ensured, various abnormal situations in the liquidation process are effectively dealt with, and the smooth progress of the liquidation process is ensured.

[0132] This application also provides a multi-batch concurrent data clearing device, please refer to Figure 8 , the multi-batch concurrent data clearing device comprises:

[0133] The data splitting module 10 is used to receive the data files to be split, and perform multi-dimensional data analysis on the data files according to the preset splitting setting rules, and write the data files after the multi-dimensional data analysis into the corresponding partitions of the data table to obtain multiple batches of data to be cleared;

[0134] The batch processing automation control module 20 is used to read the batch automation configuration information and perform concurrent clearing on the data to be cleared according to the batch automation configuration information.

[0135] The multi-batch concurrent data clearing device provided by the present application adopts the multi-batch concurrent data clearing method in the above-mentioned embodiment, which can solve the technical problem of multi-batch concurrent data clearing. Compared with the prior art, the beneficial effects of the multi-batch concurrent data clearing device provided by the present application are the same as the beneficial effects of the multi-batch concurrent data clearing method provided by the above-mentioned embodiment, and other technical features in the multi-batch concurrent data clearing device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0136] The present application provides a multi-batch concurrent data clearing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multi-batch concurrent data clearing method in the above-mentioned embodiment one.

[0137] Reference below Fig. 9 , which shows a schematic diagram of the structure of a multi-batch concurrent data clearing device suitable for implementing the embodiment of the present application. The multi-batch concurrent data clearing device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The multi-batch concurrent data clearing device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0138] like Fig. 9As shown, the multi-batch concurrent data clearing device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the multi-batch concurrent data clearing device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the multi-batch concurrent data clearing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a multi-batch concurrent data clearing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0139] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0140] The multi-batch concurrent data clearing device provided by the present application adopts the multi-batch concurrent data clearing method in the above embodiment, which can solve the technical problem of multi-batch concurrent data clearing. Compared with the prior art, the beneficial effects of the multi-batch concurrent data clearing device provided by the present application are the same as the beneficial effects of the multi-batch concurrent data clearing method provided by the above embodiment, and the other technical features in the multi-batch concurrent data clearing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0141] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0142] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0143] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, the computer-readable program instructions being used to execute the multi-batch concurrent data clearing method in the above-mentioned embodiment.

[0144] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0145] The computer-readable storage medium may be included in the multi-batch concurrent data clearing device; or may exist independently without being assembled into the multi-batch concurrent data clearing device.

[0146] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a multi-batch concurrent data clearing device, the multi-batch concurrent data clearing device: receives the data file to be split, and performs multi-dimensional data analysis on the data file according to a preset split setting rule; writes the data file after the multi-dimensional data analysis into the corresponding partition of the data table to obtain multiple batches of data to be cleared; reads batch automation configuration information and concurrently clears the data to be cleared according to the batch automation configuration information.

[0147] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0148] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0149] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0150] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned multi-batch concurrent data clearing method, and can solve the technical problem of multi-batch concurrent data clearing. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the multi-batch concurrent data clearing method provided in the above-mentioned embodiment, and will not be repeated here.

[0151] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned multi-batch concurrent data clearing method when executed by a processor.

[0152] The computer program product provided by this application can solve the technical problem of multiple batches of concurrent data clearing. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the multiple batches of concurrent data clearing method provided by the above embodiment, which will not be repeated here.

[0153] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A multi-batch concurrent data clearing method, characterized in that: The method comprises: Receiving a data file to be split, and performing multi-dimensional data analysis on the data file according to a preset split setting rule; Write the data files after multi-dimensional data analysis into the corresponding partitions of the data table to obtain multiple batches of data to be cleared; Batch automation configuration information is read and concurrent clearing of the data to be cleared is performed according to the batch automation configuration information.

2. The method according to claim 1, characterized in that Before the step of receiving the data file to be split and performing multi-dimensional data analysis on the data file according to the preset split setting rule, the step further includes: Acquire actual business scenario information and determine business dimensions, and configure split setting rules based on the actual business scenario information and the business dimensions.

3. The method according to claim 1, characterized in that The step of reading batch automation configuration information and concurrently clearing the data to be cleared according to the batch automation configuration information includes: Read batch automation configuration information and query historical processing time and processing quantity to calculate processing efficiency; Predict the time required for concurrent liquidation based on the processing efficiency; The data to be cleared is cleared concurrently according to the batch automation configuration information, and the time required for the concurrent clearing is visualized during the concurrent clearing process.

4. The method according to claim 1, characterized in that The step of reading batch automation configuration information and concurrently clearing the data to be cleared according to the batch automation configuration information includes: Reading batch automation configuration information, wherein the batch automation configuration information includes batch execution step configuration information; The number of concurrent steps is set according to the batch execution step configuration information, and the data to be cleared is cleared concurrently by setting the number of concurrent steps.

5. The method according to claim 1, characterized in that The step of reading batch automation configuration information and concurrently clearing the data to be cleared according to the batch automation configuration information includes: Reading batch automation configuration information, wherein the batch automation configuration information includes batch dependency configuration information; Determine the dependency relationship of the data to be cleared according to the batch dependency configuration information, and perform concurrent clearing on the data to be cleared with reference to the dependency relationship.

6. The method according to claim 1, characterized in that The step of reading batch automation configuration information and concurrently clearing the data to be cleared according to the batch automation configuration information includes: Reading batch automation configuration information, wherein the batch automation configuration includes step start time configuration information; The data to be cleared is cleared concurrently according to the step start time configuration information.

7. The method according to claim 1, characterized in that The step of reading batch automation configuration information and concurrently clearing the data to be cleared according to the batch automation configuration information includes: Reading batch automation configuration information, wherein the batch automation configuration information includes an exception retry mechanism; Performing concurrent clearing of the data to be cleared, and monitoring abnormalities in the concurrent clearing process; When an abnormal situation is detected, the abnormal retry mechanism is triggered, and abnormal diagnosis is initiated for the data to be cleared according to the abnormal retry mechanism to obtain a diagnosis result; Retry and concurrently clear the data to be cleared according to the diagnosis result.

8. The method according to claim 1, characterized in that After the step of reading the batch automation configuration information and concurrently clearing the data to be cleared according to the batch automation configuration information, the following step further includes: Record concurrent liquidation results; When the concurrent liquidation result is a liquidation failure, an alarm mechanism is triggered, and manual intervention is requested through the alarm mechanism.

9. A multi-batch concurrent data clearing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the multi-batch concurrent data clearing method according to any one of claims 1 to 8.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the multi-batch concurrent data clearing method according to any one of claims 1 to 8 are implemented.

Images