Implementation method and device capable of intercepting batch large file data splitting and timed execution

Abstract

The invention discloses a batch large file data splitting and timed execution interceptable implementation method and device, and the device comprises a timed splitting rule management module which is used for setting and maintaining a splitting rule of a file, and providing a rule selection interface when a large data file is imported; after receiving the file / request, the data processing module performs multi-thread file splitting on a big data file according to a splitting rule, generates a plurality of sub-files, constructs a delay task in a to-be-executed state and stores the delay task in a database, and performs multi-thread processing and updates the execution state under distributed task scheduling; and the task job scheduling service module queries a to-be-executed delay task according to a set execution frequency, and calls the data processing module to perform multi-thread processing when the delay task reaches execution time. According to the method, the delay task is automatically generated through the configuration rule, and the system automatically processes and maintains operations such as splitting, execution and interception of the file, so that manual intervention is reduced, the misoperation risk is reduced, and the processing accuracy and timeliness are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to the technology of data splitting, timed execution and interceptable control of large batch files. Background Art

[0002] In today's digital age, many industries, especially banking practices, frequently face the need to quickly process large amounts of data, such as sending batch text messages. Traditional processing methods usually use batch files for one-time real-time or scheduled processing.

[0003] However, this approach has many drawbacks. On the one hand, in the process of processing large amounts of data, the system is prone to excessive memory usage, which seriously affects system performance and stability; on the other hand, once errors are found in file data, it is difficult to effectively intervene in real time in the data processing process for data that has not yet been processed, making it impossible to effectively control related business risks.

[0004] In addition, most of the existing technologies rely on manual operations. For example, when processing part of the data in a large file, the data needs to be manually split and imported, and then manually wait for the appropriate time before the next round of processing. This not only leads to low overall processing efficiency, but also easily causes problems due to errors in manual splitting operations, and the execution processing time is difficult to control accurately. It is impossible to automatically generate delayed tasks through configuration rules like the present invention, and the system cannot automatically process and maintain file splitting, execution, and interception operations. There are obvious deficiencies in automation, visualization, and standardized management. Summary of the invention

[0005] The present invention provides a method for implementing batch large file data splitting and scheduled execution that can be intercepted, comprising:

[0006] Step 101, when a large data file is requested to be processed, a corresponding splitting rule is selected, and the parameters of the splitting rule include execution time and execution interval;

[0007] Step 102, splitting the big data file into multiple sub-files according to the splitting rule, and setting the initial state of each sub-file to be executed;

[0008] Step 103, calculating the execution time of each delayed task according to the processing time of the sub-files and the batch processing interval;

[0009] According to the execution time and execution interval of the splitting rules, the execution time of each sub-file is calculated, and a delayed task to be executed is generated. Each split sub-file can be regarded as an independent task.

[0010] Save delayed tasks to the database;

[0011] Step 104, according to the preset execution frequency, query the delayed task records whose execution time is less than or equal to the current time and whose status is to be executed, obtain the delayed task information and update its record status to being executed;

[0012] Use multithreading to process delayed tasks to be executed, and update the status according to the execution status: successful execution or failed execution;

[0013] Step 105: When any of the above steps is executed, any of the following actions of this step may be inserted:

[0014] For delayed tasks that are to be executed but have not yet been scheduled for execution, you can pause, cancel (intercept), or execute them immediately;

[0015] For suspended tasks, you can resume execution;

[0016] For failed tasks, they can be retried after analysis.

[0017] Furthermore, the method further includes step 106, analyzing and monitoring the execution status of delayed tasks, wherein the analysis and monitoring dimensions include:

[0018] Task execution status, number of processed file records, and execution status of each server.

[0019] Multi-dimensional analysis and monitoring of delayed task execution status enable business personnel to clearly understand the overall execution and flow of batch files, thereby improving management efficiency and user experience.

[0020] Furthermore, the task execution status includes: pending, executing, paused, cancelled, successfully executed, and failed to execute;

[0021] The number of processed file records includes the total number, the number of successes, and the number of failures;

[0022] The execution status of each server includes: the number of tasks, the number of successful tasks, the number of failed tasks, the total execution time of the entire batch, the execution time of each sub-batch, and the average execution time.

[0023] It provides a variety of operation functions for delayed tasks, such as immediate execution, pause, cancel, resume execution, retry, etc., which can meet the needs of different business scenarios and enhance the flexibility and controllability of business processing.

[0024] The present invention also provides a device for batch large file data splitting and scheduled execution that can be intercepted, including: a scheduled splitting rule management module, a data processing module, and a task job scheduling service module.

[0025] Scheduled splitting rule management module: used to set and maintain file splitting rules, and provide a rule selection interface when importing big data files. The parameters of the splitting rules include execution time and execution interval;

[0026] The data processing module is used for data processing and execution services. After receiving the file / request, the data processing module performs multi-threaded file splitting on the big data file according to the splitting rules to generate multiple sub-files. According to the processing time of the sub-files and the batch processing interval, the execution time of each delayed task is calculated, and the delayed tasks in the to-be-executed state are constructed and saved to the database. After that, multi-threaded processing is performed under distributed task scheduling and the execution status is updated.

[0027] Task job scheduling service module: queries the delayed tasks to be executed according to the set execution frequency. When the delayed tasks reach the execution time, the data processing module is called for multi-threaded processing.

[0028] Furthermore, it also includes a task management module for querying the status of delayed tasks and providing an interface for pausing, canceling, immediately executing, resuming execution, and retrying delayed tasks.

[0029] Furthermore, it also includes a monitoring and analysis module, which analyzes and monitors the execution status of delayed tasks from multiple dimensions, including statistics and analysis of dimensions such as task execution status, number of processed file records, and execution status of each server, and provides a visual display interface.

[0030] Furthermore, the splitting rule is a splitting rule selected by the client, browser, or API when the big data file is processed by the client, browser, or API request.

[0031] Furthermore, when the big data file is processed by the client, browser, or API request, the splitting rule is a splitting rule selected by the data processing platform based on a preset strategy.

[0032] Furthermore, the parameters of the splitting rule also include: first batch execution time, first batch execution interval, first batch execution quantity, remaining batch interval, and remaining batch number.

[0033] Furthermore, the parameters of the splitting rule also include: first batch execution time, first batch execution interval, first batch execution quantity, remaining batch interval, and remaining batch number;

[0034] The sub-file of the first batch execution quantity calculates the execution time of each delayed task in the first batch execution based on the first batch execution time and the first batch execution interval;

[0035] The sub-file of the remaining number of processing times calculates the execution time of the delayed tasks of each remaining batch based on the execution time of the first batch, the execution interval of the first batch, and the interval of the remaining batches.

[0036] Beneficial effects:

[0037] It effectively solves problems such as high system memory usage, long single execution time, and long client waiting time, and improves system performance and processing efficiency.

[0038] By configuring rules, delayed tasks are automatically generated and the system automatically handles and maintains file splitting, execution, interception and other operations, reducing manual intervention, lowering the risk of misoperation, and improving the accuracy and timeliness of processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to better understand the above and other purposes, features, advantages and functions of the present invention, reference may be made to the embodiments shown in the accompanying drawings. The same reference numerals in the accompanying drawings refer to the same components. It should be understood by those skilled in the art that the accompanying drawings are intended to schematically illustrate the preferred embodiments of the present invention and have no limiting effect on the scope of the present invention, and the components in the drawings are not drawn to scale.

[0040] Figure 1 Schematic diagram of the splitting and processing of big data files by the client and data processing module.

[0041] Figure 2 A schematic diagram of rule settings for the timing split rule management module.

[0042] Figure 3 This is a schematic diagram of the delayed task execution processing by the task job scheduling service module.

[0043] Figure 4 This is a schematic diagram of the delayed task status operation of the task management module and the monitoring and analysis module.

[0044] Figure 5 The present invention is a flow chart of a method for batch large file data splitting and scheduled execution that can be intercepted. Figure 6 It is the status table after the big data file is split according to the splitting rules. DETAILED DESCRIPTION

[0045] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0046] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0047] In order to at least partially solve one or more of the above problems and other potential problems, an embodiment of the present disclosure proposes a device for intercepting the scheduled execution of batch large file data splitting.

[0048] like Figure 1 , Figure 2 , Figure 3 As shown, a device for batch large file data splitting and scheduled execution that can be intercepted includes: a scheduled splitting rule management module, a data processing module, and a task job scheduling service module.

[0049] Scheduled splitting rule management module: used to set and maintain file splitting rules, and provide a rule selection interface when importing big data files. The parameters of the splitting rules include execution time, execution interval, the number of first batch executions, the number of remaining batches, etc. Specifically, the execution time includes: the first batch execution time, and the execution interval includes: the first batch execution interval and the remaining batch interval;

[0050] The data processing module is used for data processing and execution services. When the client, browser, or API imports a big data file, the splitting rule is selected through the timing splitting rule management module. After receiving the file / request, the data processing module performs multi-threaded file splitting on the big data file according to the splitting rule to generate multiple sub-files. According to the processing time of the sub-files and the batch processing interval, the execution time of each delayed task is calculated, and the delayed tasks in the to-be-executed state are constructed and saved to the database. After that, multi-threaded processing is performed under distributed task scheduling and the execution status is updated.

[0051] Task job scheduling service module: query the delayed tasks to be executed according to the set execution frequency, for example, once per minute. When the delayed task reaches the execution time, call the data processing module for multi-threaded processing;

[0052] like Figure 4 As shown, the device may also include a task management module and / or a monitoring and analysis module, wherein the task management module is used to query the status of delayed tasks and provide an interface for pausing, canceling, immediately executing, resuming execution, retrying, and other operations on delayed tasks.

[0053] The monitoring and analysis module analyzes and monitors the execution status of delayed tasks from multiple dimensions, including statistics and analysis of task execution status, number of processed file records, execution status of each server, and provides a visual display interface.

[0054] like Figure 5 As shown, a method for implementing batch large file data splitting and scheduled execution that can be intercepted is applied to the above-mentioned device example. The method includes:

[0055] Step 100: The client requests to call the big data file processing, and the timing splitting rule management module provides a splitting rule selection interface;

[0056] Step 101, the client selects a split rule. The parameters of the split rule include execution time, execution interval, the number of first batch executions = 2000, the number of remaining batches 7800, and the number of first batch executions = 2. Specifically, the execution time includes: the first batch execution time 22:00, and the execution interval includes: the first batch execution interval is 10 minutes, and the remaining batch interval is 8 minutes;

[0057] Step 102, split the big data file into 10 sub-files according to the splitting rule, and the initial state of each sub-file is set to be executed;

[0058] Step 103, the number of files FN executed in the first batch is 2 (serial numbers 001 and 002), and the execution time of the delayed task corresponding to the 001 subfile of the first batch is calculated to be 22:00, and the execution time of the delayed task corresponding to the 002 subfile is calculated to be 22:10 according to the first batch execution time FT of 22:00 and the first batch execution interval FS of 10 minutes;

[0059] The sub-file of the remaining batch processing quantity (LN=8) calculates the execution time LT of each remaining batch of delayed tasks based on the first batch execution time, the first batch execution interval FS=10 minutes, and the remaining batch interval LS=8 minutes;

[0060] LT(i)=FT+(FN-1)*FS+LS*i

[0061] i is the i-th one in the remaining batches;

[0062] Generate a deferred task in the pending state. Each split sub-file can be regarded as an independent task.

[0063] Save delayed tasks to the database;

[0064] like Figure 6 As shown in the figure, the status table of the big data file after it is split according to the splitting rules.

[0065] Step 104, according to a preset execution frequency (e.g., 1 time / minute), query the delayed task records whose execution time is less than or equal to the current time and whose status is to be executed, obtain the delayed task information and update its record status to being executed;

[0066] Use multithreading to process delayed tasks to be executed, and update the status according to the execution status: successful execution or failed execution;

[0067] Based on the above method, step 106 may be further included to analyze and monitor the execution status of delayed tasks, and the analysis and monitoring dimensions include:

[0068] Task execution status, number of processed file records, and execution status of each server.

[0069] Multi-dimensional analysis and monitoring of delayed task execution status enable business personnel to clearly understand the overall execution and flow of batch files, thereby improving management efficiency and user experience.

[0070] The above task execution status includes but is not limited to: pending, executing, paused, canceled, executed successfully, and executed failed;

[0071] The number of processed file records includes: total number, success number, and failure number;

[0072] The execution status of each server includes: number of tasks, number of successful tasks, number of failed tasks, total execution time of the entire batch, execution time of each sub-batch, and average execution time.

[0073] It provides a variety of operation functions for delayed tasks, such as immediate execution, pause, cancel, resume execution, retry, etc., which can meet the needs of different business scenarios and enhance the flexibility and controllability of business processing.

[0074] Beneficial effects:

[0075] It effectively solves problems such as high system memory usage, long single execution time, and long client waiting time, and improves system performance and processing efficiency.

[0076] By configuring rules, delayed tasks are automatically generated and the system automatically handles and maintains file splitting, execution, interception and other operations, reducing manual intervention, lowering the risk of misoperation, and improving the accuracy and timeliness of processing.

Claims

1. A method for implementing batch large file data splitting with scheduled execution and interception, characterized in that: include: Step 101, when a large data file is requested to be processed, a corresponding splitting rule is selected, and the parameters of the splitting rule include execution time and execution interval; Step 102, splitting the big data file into multiple sub-files according to the splitting rule, and setting the initial state of each sub-file to be executed; Step 103, calculating the execution time of each delayed task according to the processing time of the sub-files and the batch processing interval; According to the execution time and execution interval of the splitting rules, the execution time of each sub-file is calculated, and a delayed task to be executed is generated. Each split sub-file can be regarded as an independent task. Save delayed tasks to the database; Step 104, according to the preset execution frequency, query the delayed task records whose execution time is less than or equal to the current time and whose status is to be executed, obtain the delayed task information and update its record status to being executed; Use multithreading to process delayed tasks to be executed, and update the status according to the execution status: successful execution or failed execution; Step 105: When any of the above steps is executed, any of the following actions of this step may be inserted: For delayed tasks that are to be executed but have not yet been scheduled for execution, you can pause, cancel (intercept), or execute them immediately; For suspended tasks, you can resume execution; For failed tasks, they can be retried after analysis.

2. A method for implementing batch large file data splitting with scheduled execution and interception as claimed in claim 1, characterized in that: The step 106 is also included, analyzing and monitoring the execution status of the delayed tasks, and the analysis and monitoring dimensions include: Task execution status, number of processed file records, and execution status of each server.

3. A method for implementing batch large file data splitting with scheduled execution and interception as claimed in claim 1, characterized in that: The task execution status includes: pending, executing, paused, canceled, executed successfully, and executed failed; The number of processed file records includes the total number, the number of successes, and the number of failures; The execution status of each server includes: the number of tasks, the number of successful tasks, the number of failed tasks, the total execution time of the entire batch, the execution time of each sub-batch, and the average execution time; It provides a variety of operation functions for delayed tasks, such as immediate execution, pause, cancel, resume execution, retry, etc., which can meet the needs of different business scenarios and enhance the flexibility and controllability of business processing.

4. A device for batch large file data splitting and scheduled execution that can be intercepted, characterized in that: include: Timing splitting rule management module, data processing module, task job scheduling service module; Scheduled splitting rule management module: used to set and maintain file splitting rules, and provide a rule selection interface when importing big data files. The parameters of the splitting rules include execution time and execution interval; The data processing module is used for data processing and execution services. After receiving the file / request, the data processing module performs multi-threaded file splitting on the big data file according to the splitting rules to generate multiple sub-files. According to the processing time of the sub-files and the batch processing interval, the execution time of each delayed task is calculated, and the delayed tasks in the to-be-executed state are constructed and saved to the database. After that, multi-threaded processing is performed under distributed task scheduling and the execution status is updated. Task job scheduling service module: queries the delayed tasks to be executed according to the set execution frequency. When the delayed tasks reach the execution time, the data processing module is called for multi-threaded processing.

5. The device for batch large file data splitting and scheduled execution interceptable as claimed in claim 4, characterized in that: It also includes a task management module, which is used to query the status of delayed tasks and provide an interface for pausing, canceling, immediately executing, resuming execution, and retrying delayed tasks.

6. The device for batch large file data splitting and scheduled execution interceptable as claimed in claim 4, characterized in that: It also includes a monitoring and analysis module, which analyzes and monitors the execution status of delayed tasks from multiple dimensions, including statistics and analysis of dimensions such as task execution status, number of processed file records, and execution status of each server, and provides a visual display interface.

7. A method or device for implementing batch large file data splitting with scheduled execution and interception as claimed in any one of claims 1 to 6, characterized in that: The splitting rule is a splitting rule selected by the client, browser, or API when the big data file is called and processed by the client, browser, or API request.

8. A method or device for implementing batch large file data splitting with scheduled execution and interception as claimed in any one of claims 1 to 6, characterized in that: The splitting rules are splitting rules selected by the data processing platform according to a preset strategy when the big data file is called and processed by the client, browser, or API request.

9. A method or device for implementing batch large file data splitting with scheduled execution and interception as claimed in any one of claims 1 to 6, characterized in that: The parameters of the splitting rule also include: first batch execution time, first batch execution interval, first batch execution quantity, remaining batch interval, and remaining batch number.

10. A method or device for implementing batch large file data splitting with scheduled execution and interception as claimed in any one of claims 1 to 6, characterized in that: The parameters of the splitting rule also include: first batch execution time, first batch execution interval, first batch execution quantity, remaining batch interval, and remaining batch number; The sub-file of the first batch execution quantity calculates the execution time of each delayed task in the first batch execution based on the first batch execution time and the first batch execution interval; The sub-file of the remaining number of processing times calculates the execution time of the delayed tasks of each remaining batch based on the execution time of the first batch, the execution interval of the first batch, and the interval of the remaining batches.

Images