A batch job processing method and apparatus

Abstract

This application provides a batch job processing method and apparatus, which can be used in the financial field or other fields. The method includes: updating pre-obtained conflict relationship data when there are idle executors; traversing the waiting queue and placing batch jobs that do not conflict with currently running batch jobs into the ready queue according to the conflict relationship data; selecting the batch job with the longest predicted running time from the ready queue and running it; monitoring the lock resource status during the process, and if the lock resource requested by the running batch job is already occupied, causing the batch job with the requested lock resource to fail immediately and performing a rollback operation to release the occupied resources, thereby improving resource utilization efficiency. At the same time, it avoids batch jobs that conflict with the resources required by the running batch job from entering the ready queue, reducing the probability of conflict and thus reducing the probability of running batch jobs being rerun, preventing resource waste and avoiding the loss of business data.

Description

Technical Field

[0001] This application relates to the financial field, specifically to the field of batch processing, and particularly to a batch processing method and apparatus. Background Technology

[0002] Batch jobs refer to a method of processing business data in batches to improve data processing efficiency. An application that processes a set of data is called a job. To improve system processing capacity, business software systems often need to use batch jobs to process a set of data centrally. The system can use a batch framework to automatically trigger the execution of batch jobs at fixed times or fixed time intervals.

[0003] Batch jobs strictly require transaction atomicity and consistency, meaning that transactions are initiated at the beginning of the job and released only when the job ends. Because batch jobs process large amounts of data across a wide range, often involving access to and updating multiple tables and records within those tables, data resource locking and release issues arise. During concurrent or parallel execution of jobs, the data being accessed remains locked. When multiple jobs attempt to access the same data, prolonged waits due to the inability to acquire lock resources can lead to timeouts and failures.

[0004] In existing technologies, common batch processing frameworks provide some support for rerunning failed jobs. For example, open-source or self-developed batch schedulers like Spring Batch typically support configuring the number of automatic reruns for batch jobs, i.e., the number of times a job will automatically trigger re-execution when it fails. However, because jobs involved in resource conflicts do not fail immediately, but only throw a lock wait timeout exception if they still cannot acquire the required resources after a certain waiting period, the job will still occupy locked data resources, as well as executor memory and computing resources during the waiting period. Furthermore, in the case of lock resource conflicts, whether rerunning immediately or automatically rerunning after a certain waiting period, failure is still possible. Existing technologies cannot determine whether a failed job should be rerun at a given moment. Moreover, a job will stop rerunning after multiple reruns, which may lead to missing or inaccurate business data. Therefore, existing automatic rerun mechanisms for failed jobs suffer from low accuracy, inefficiency, and lack of flexibility in handling resource conflict situations. Summary of the Invention

[0005] In view of the problems in the prior art, the embodiments of this application provide a batch processing method and apparatus, which can at least partially solve the problems existing in the prior art.

[0006] Firstly, this application provides a batch job processing method, including:

[0007] When an idle executor exists, update the pre-obtained conflict relationship data; the conflict relationship data includes the conflict relationships between the batch jobs in the waiting queue and the resources required by the running batch jobs;

[0008] Traverse the waiting queue and, based on the conflict relationship data, place batch jobs in the waiting queue that do not conflict with the currently running batch jobs into the ready queue;

[0009] Select the batch job with the longest predicted execution time from the ready queue and run it;

[0010] During operation, the lock resource status is monitored. If the lock resource requested by the running batch job is already occupied, the batch job whose lock resource is occupied will fail immediately and a rollback operation will be performed.

[0011] This also includes:

[0012] When a new batch job enters the waiting queue, update the pre-obtained conflict relationship data;

[0013] Traverse the waiting queue and, based on the conflict relationship data, place batch jobs in the waiting queue that do not conflict with the currently running batch jobs into the ready queue.

[0014] Prior to updating the pre-obtained conflict relationship data, the following steps are also included:

[0015] Describe the batch job using the operations to be performed and the resources required for each operation;

[0016] Use the job launcher to start the batch jobs described, generate instances of the started batch jobs, and put them into the waiting queue;

[0017] Conflict relationship data is generated based on the descriptions of batch jobs in the waiting queue, the descriptions of batch jobs currently running, and the predicted runtimes of each type of operation obtained in advance.

[0018] The step of generating conflict relationship data based on the description results of batch jobs in the waiting queue, the description results of currently running batch jobs, and the pre-obtained predicted running times of each type of operation includes:

[0019] The operation to identify resource conflicts between batch jobs in the waiting queue and currently running batch jobs is performed.

[0020] A first interval time is obtained based on the resource conflict operation, the predicted running time of each type of operation, and the description results of the batch jobs in the waiting queue. The first interval time is the time required to execute the batch jobs in the waiting queue up to the resource conflict operation.

[0021] Obtain the current operation of the running batch job, and obtain the second interval time based on the predicted running time of each type of operation and the description result of the running batch job. The second interval time is the time required for the running batch job to complete execution.

[0022] If the first interval is less than the second interval, conflict relationship data between the batch jobs in the waiting queue and the batch jobs that are currently running is generated.

[0023] The step of obtaining the first interval time based on the resource conflict operation, the predicted running time of each type of operation, and the description results of the batch jobs in the waiting queue includes:

[0024] Based on the description of the batch jobs in the waiting queue, the pre-operations required for the batch jobs in the waiting queue to be executed before the resource conflict operation are obtained;

[0025] The predicted runtime of each preceding operation is obtained based on the category of each preceding operation and the predicted runtime of each category of operation.

[0026] The first interval time is obtained based on the predicted running time of each preceding operation.

[0027] The step of obtaining the current operation of the running batch job and obtaining the second interval time based on the predicted running time of each type of operation and the description result of the running batch job includes:

[0028] The predicted runtime of the current operation is obtained based on the category of the current operation and the predicted runtime of each category of operation.

[0029] Based on the description of the currently running batch job and the current operation, the operations that the currently running batch job has not yet executed are obtained;

[0030] The predicted runtime of each unexecuted operation is obtained based on the category of each unexecuted operation and the predicted runtime of each category of operation;

[0031] The second interval time is obtained based on the predicted runtime of the current operation and the predicted runtime of each of the operations that have not yet been executed.

[0032] This also includes:

[0033] The predicted runtime of each batch job is obtained based on the description results of each batch job and the predicted runtime of each type of operation obtained in advance.

[0034] This also includes:

[0035] Obtain historical execution time data for each type of operation;

[0036] Based on the historical execution time data, obtain the number of operations of each category that were completed and the number that failed to be completed within different preset time periods;

[0037] The probability of each type of operation being completed within each preset time length is calculated sequentially in ascending order of preset time length. The probability of each type of operation being completed within each preset time length is calculated based on the probability of completion within the previous preset time length and the number of operations completed and the number of operations that failed within the corresponding preset time period.

[0038] The minimum time length at which the probability of completion is greater than the preset probability is selected as the predicted execution time for each type of operation.

[0039] The process of monitoring lock resource status during operation, and if the lock resource requested by the batch job is already occupied, causing the batch job to fail immediately and performing a rollback operation, further includes:

[0040] Determine whether the number of retries for a failed batch job has reached a preset maximum. If the maximum number has been reached, skip the batch job. If the maximum number has not been reached, increment the number of retries and place the batch job in a waiting queue.

[0041] Secondly, this application provides a batch processing apparatus, comprising:

[0042] The conflict relationship update unit is used to update the pre-obtained conflict relationship data when there is an idle executor; the conflict relationship data includes the conflict relationship between the batch jobs in the waiting queue and the resources required by the batch jobs that are running.

[0043] The ready conversion unit is used to traverse the waiting queue and, based on the conflict relationship data, put batch jobs in the waiting queue that do not conflict with the batch jobs that are currently running into the ready queue.

[0044] The job execution unit is used to select the batch job with the longest predicted execution time from the ready queue for execution;

[0045] The running status monitoring unit is used to monitor the lock resource status during operation. If the lock resource requested by the running batch job is already occupied, the batch job whose lock resource is occupied will fail immediately and a rollback operation will be performed.

[0046] Thirdly, this application provides a computer electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any of the above embodiments.

[0047] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described in any of the above embodiments.

[0048] Fifthly, this application provides a computer program product, which includes a computer program that, when executed by a processor, implements the method described in any of the above embodiments.

[0049] The batch job processing method and apparatus provided in this application update pre-obtained conflict relationship data when idle executors exist; traverse the waiting queue and, based on the conflict relationship data, place batch jobs in the waiting queue that do not conflict with currently running batch jobs into the ready queue; select the batch job with the longest predicted running time from the ready queue for execution; during execution, monitor the lock resource status, and if the lock resource requested by a running batch job is already occupied, immediately cause the batch job with the occupied lock resource to fail and perform a rollback operation to release the occupied resources, thereby saving resources and improving resource utilization efficiency. Simultaneously, it prevents batch jobs whose required resources conflict with those of running batch jobs from entering the ready queue, reducing the probability of conflicts and thus reducing the probability of running batch jobs rerunning, further improving resource utilization efficiency while avoiding the loss of business data. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart of a batch job processing method provided in an embodiment of this application;

[0052] Figure 2 This is a flowchart of a batch job processing method provided in an embodiment of this application;

[0053] Figure 3 This is a flowchart of a batch job processing method provided in an embodiment of this application;

[0054] Figure 4 This is a flowchart of a batch job processing method provided in an embodiment of this application;

[0055] Figure 5 This is a flowchart of a batch job processing method provided in an embodiment of this application;

[0056] Figure 6 This is a flowchart of a batch job processing method provided in an embodiment of this application;

[0057] Figure 7 This is a flowchart of a batch job processing method provided in an embodiment of this application;

[0058] Figure 8 This is a flowchart of a batch job processing method provided in an embodiment of this application;

[0059] Figure 9 This is a schematic diagram of the structure of a batch processing device provided in an embodiment of this application;

[0060] Figure 10 This is a schematic diagram of the structure of a batch processing device provided in an embodiment of this application;

[0061] Figure 11 This is a schematic diagram of the structure of a batch processing device provided in an embodiment of this application;

[0062] Figure 12 This is a schematic diagram of the structure of a batch processing device provided in an embodiment of this application;

[0063] Figure 13 This is a schematic diagram of the structure of a batch processing device provided in an embodiment of this application;

[0064] Figure 14 This is a schematic diagram of the structure of a batch processing device provided in an embodiment of this application;

[0065] Figure 15 This is a schematic diagram of the structure of a batch processing device provided in an embodiment of this application;

[0066] Figure 16 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other.

[0068] It should be noted that the customer information mining method and apparatus disclosed in this application can be used in the field of financial technology, or in any field other than financial technology. The application field of the batch processing method and apparatus disclosed in this application is not limited.

[0069] The following describes the specific implementation process of the batch job processing method provided in this embodiment of the invention, using a server as the execution subject as an example.

[0070] Figure 1 This is a flowchart of a batch job processing method provided in an embodiment of this application, such as... Figure 1 As shown in the embodiments of this application, the batch job processing method includes:

[0071] S101: When there is an idle executor, update the pre-obtained conflict relationship data; the conflict relationship data contains the conflict relationships between the batch jobs in the waiting queue and the resources required by the batch jobs that are running.

[0072] Specifically, since the total number of batch jobs is often greater than the number of executors, idle executors often come from the resources released by batch jobs that have completed or failed. At this time, the batch jobs in the running state change, so it is necessary to update the conflict relationship between the batch jobs in the waiting queue and the resources required by the batch jobs that are running.

[0073] S102: Traverse the waiting queue and, based on the conflict relationship data, put batch jobs in the waiting queue that do not conflict with the currently running batch jobs into the ready queue.

[0074] Specifically, after the conflict relationship data is updated, there may be new batch jobs in the waiting queue that do not conflict with the currently running batch jobs. The server traverses the waiting queue, puts the batch jobs that do not conflict with the currently running batch jobs into the ready queue, and removes them from the waiting queue.

[0075] S103: Select the batch job with the longest predicted running time from the ready queue and run it;

[0076] Specifically, since long-running jobs often require more resources and are more likely to conflict with other jobs, prioritizing the execution of the batch job with the longest predicted runtime can prevent other jobs that conflict with it from entering the ready queue, thus avoiding conflicts with the longest-running job and improving the success rate of completing resource-intensive and conflict-prone batch jobs. When there are idle executors on the server, the server allocates the idle executor to the batch job with the longest predicted runtime in the ready queue, causing the batch job allocated to the executor to change from the ready state to the running state and then remove it from the ready queue.

[0077] S104: During operation, monitor the lock resource status. If the lock resource requested by the running batch job is already occupied, the batch job whose lock resource is occupied will fail immediately and a rollback operation will be performed.

[0078] Specifically, during operation, the server monitors the occupancy of lock resources in the system. When a batch job in the running state requests a lock resource, if the lock resource is already occupied, it will not wait, but will immediately fail, release the occupied resources, and perform a rollback operation to restore the state before the current operation.

[0079] The batch job processing method provided in this application updates pre-obtained conflict relationship data when idle executors exist; iterates through the waiting queue and, based on the conflict relationship data, places batch jobs in the waiting queue that do not conflict with currently running batch jobs into the ready queue; it selects the batch job with the longest predicted execution time from the ready queue for execution; during execution, it monitors the lock resource status, and if the lock resource requested by a running batch job is already occupied, the batch job requesting the occupied lock resource immediately fails and performs a rollback operation to release the occupied resource, thereby saving resource usage and improving resource utilization efficiency. Simultaneously, it prevents batch jobs whose required resources conflict with those of running batch jobs from entering the ready queue, reducing the probability of conflicts and thus reducing the probability of running batch jobs rerunning. This further improves resource utilization efficiency while avoiding the loss of business data.

[0080] Figure 2 This is a flowchart of a batch job processing method provided in an embodiment of this application, such as... Figure 2 As shown, based on the above embodiments, the batch job processing method provided in this application further includes:

[0081] S201: When a new batch job enters the waiting queue, update the pre-obtained conflict relationship data;

[0082] Specifically, when a new job enters the waiting queue, the batch jobs in the waiting queue change, thus requiring an update of the conflict relationships between the batch jobs in the waiting queue and the resources required by the currently running batch jobs. In one embodiment, the original conflict relationship data can be retained, and the conflict relationships between the newly entered batch jobs and the resources required by the currently running batch jobs can be added to the original conflict relationship data.

[0083] S202: Traverse the waiting queue and, based on the conflict relationship data, add batch jobs that do not conflict with the currently running batch jobs to the ready queue.

[0084] Specifically, among the newly entered batch jobs in the waiting queue, there may be batch jobs that do not conflict with the currently running batch jobs. Therefore, after the conflict relationship data is updated, the server iterates through the waiting queue, adds batch jobs that do not conflict with the currently running batch jobs to the ready queue, and removes them from the waiting queue. In another embodiment, the server iterates through the newly entered batch jobs in the waiting queue and adds batch jobs that do not conflict with the currently running batch jobs to the ready queue.

[0085] The batch job processing method provided in this application updates the pre-obtained conflict relationship data when a new batch job enters the waiting queue; iterates through the waiting queue and, based on the conflict relationship data, places batch jobs in the waiting queue that do not conflict with the currently running batch jobs into the ready queue, ensuring that the conflict relationship data is up-to-date and that batch jobs in the waiting queue that do not conflict with the currently running batch jobs can enter the ready queue in a timely manner.

[0086] Figure 3 This is a flowchart of a batch job processing method provided in an embodiment of this application, such as... Figure 3 As shown, based on the above embodiments, the batch job processing method provided in this application, before S101, further includes:

[0087] S301: Describe the batch job using the operations to be performed and the resources required for each operation;

[0088] Specifically, a batch job often consists of multiple operations executed sequentially. Besides executor resources, batch job execution also requires resources needed to perform various operations, such as I / O resources and lock resources. Since the successful execution of a batch job is closely related to the availability of the necessary resources, batch jobs are described using the operations to be executed and the resources required for each operation.

[0089] S302: Use the job initiator to start the batch job described, generate instances of the started batch job and put them into the waiting queue;

[0090] Specifically, each batch job includes a job initiator, which is used to start, pause, abandon, or terminate the batch job. The server periodically calls the job initiator to start the batch job as described, and creates instances of the started batch jobs, placing them in a waiting queue. By setting different start times for different types of batch jobs, different types of batch jobs can be placed in the waiting queue at different times, thus ensuring that different types of batch jobs are completed in a fixed order. This allows each type of batch job to acquire only the system resources required for execution without waiting for the completion of preceding tasks.

[0091] S303: Generate conflict relationship data based on the description results of batch jobs in the waiting queue, the description results of batch jobs currently running, and the predicted running times of each type of operation obtained in advance.

[0092] Specifically, based on the description results of batch jobs in the waiting queue, the description results of batch jobs currently running, and the predicted running time of each type of operation obtained in advance, it is possible to determine whether there are resource conflict operations between jobs, and whether resource conflict operations will be executed at the same time, thereby determining whether resource conflicts will occur between batch jobs in the waiting queue and batch jobs currently running, and obtaining conflict relationship data.

[0093] The batch job processing method provided in this application describes the batch jobs by utilizing the operations to be performed and the resources required by each operation; it starts the described batch jobs using a job initiator, generates instances of the started batch jobs, and places them in a waiting queue; it generates conflict relationship data based on the description results of the batch jobs in the waiting queue, the description results of the batch jobs currently running, and the predicted running times of each type of operation obtained in advance. This achieves a scientific description of batch jobs and advance prediction of conflicts, reduces the probability of conflicts, avoids the waste of resources caused by multiple failures and reruns, reduces the number of reruns, and improves the probability of job completion and resource utilization efficiency.

[0094] In one embodiment, such as Figure 4 As shown, S303 includes:

[0095] S401: Retrieves resource conflict operations between batch jobs in the waiting queue and currently running batch jobs;

[0096] Specifically, a conflict can be determined by finding the intersection of the resources required by each operation. If the intersection is empty, there is no conflict; if the intersection is not empty, there is a conflict. If resource conflicting operations exist, S402 is executed; if resource conflicting operations do not exist, it means that there is no conflict between the batch job in the waiting queue and the batch job that is currently running.

[0097] For example, a batch of jobs in the waiting queue involves sequential operations A / B / C / D, with each operation requiring a set of resources {a} / {b} / {c} / {d} respectively. A batch of jobs currently running involves sequential operations E / C / F / G, with each operation requiring a set of resources {e} / {c} / {f} / {g} respectively. Thus, we can obtain a resource conflict operation C, which involves conflicting resources c.

[0098] S402: Based on the predicted running time of the resource conflict operation, each type of operation, and the description results of the batch jobs in the waiting queue, the first interval time is the time required to execute the batch jobs in the waiting queue until the resource conflict operation.

[0099] Specifically, based on the predicted running time of resource conflict operations, various types of operations, and the description results of batch jobs in the waiting queue, the time required for batch jobs in the waiting queue to run to the resource conflict operation is obtained, that is, the time interval between the start of running and the issuance of a request for conflicting resources, which is used as the first interval time.

[0100] In one embodiment, such as Figure 5 As shown, S402 includes:

[0101] S501: Based on the description of the batch jobs in the waiting queue, obtain the pre-operations required for the batch jobs in the waiting queue to proceed before the resource conflict operation;

[0102] Specifically, the operations that batch jobs in the waiting queue need to perform before executing resource conflict operations are called pre-processing operations.

[0103] For example, if a batch of jobs in a waiting queue involves sequential operations A / B / C / D, and there is a resource conflict operation C, then the preceding operations include A and B.

[0104] S502: Obtain the predicted runtime of each preceding operation based on the category of each preceding operation and the predicted runtime of each category of operation;

[0105] Specifically, the server obtains the category to which each preceding operation belongs, such as query operation, storage operation, etc., and uses the predicted runtime of the category as the predicted runtime of the corresponding preceding operation.

[0106] S503: Obtain the first interval time based on the predicted running time of each preceding operation.

[0107] Specifically, the server uses the sum of the predicted runtimes of each preceding operation as the first interval.

[0108] The batch job processing method provided in this application obtains the pre-operations required for the batch jobs in the waiting queue to execute before the resource conflict operation, based on the description results of the batch jobs in the waiting queue; and obtains the predicted execution time of each pre-operation based on the category of each pre-operation and the predicted execution time of each category of operations, so as to obtain the first interval time. This method obtains the time required for a batch job with a conflict in the waiting queue to request the conflicting resource from the start of execution in a relatively accurate manner, providing a basis for the subsequent judgment of the conflict relationship.

[0109] S403: Obtain the current operation of the running batch job, and obtain the second interval time based on the predicted running time of each type of operation and the description result of the running batch job. The second interval time is the time required for the running batch job to complete execution.

[0110] Specifically, since batch jobs are submitted uniformly after all operations are completed and the occupied resources are released, it is necessary to obtain the time required for the running job to go from its current state to the end of its run based on the resource conflict operations, the predicted running time of each type of operation, and the description results of the batch jobs currently running, as the second interval time.

[0111] In one embodiment, such as Figure 6 As shown, S403 includes:

[0112] S601: Obtain the predicted runtime of the current operation based on the category of the current operation and the predicted runtime of each category of operations;

[0113] Specifically, the server obtains the operation category to which the current operation belongs and uses the predicted runtime of that category as the predicted runtime of the current operation.

[0114] S602: Based on the description of the currently running batch job and the current operations, obtain the operations that the currently running batch job has not yet executed;

[0115] Specifically, the server retrieves the operations that have not yet been executed in the running batch jobs, that is, the operations that still need to be executed after the current operation is completed.

[0116] S603: Obtain the predicted runtime of each unexecuted operation based on the category of each unexecuted operation and the predicted runtime of each category of operations;

[0117] Specifically, the server retrieves the categories to which each operation that has not yet been executed belongs, such as query operations and storage operations, and uses the predicted runtime of the category as the predicted runtime of the corresponding operation.

[0118] S604: The second interval time is obtained based on the predicted runtime of the current operation and the predicted runtime of each operation that has not yet been executed.

[0119] Specifically, if the batch jobs in the waiting queue have low urgency, the predicted runtime of the current operation and the sum of the predicted runtimes of all unexecuted operations can be used as the second interval to reduce computation and save resources. In another embodiment, if the batch jobs in the waiting queue have high urgency, the running time of the current operation can be obtained, and the sum of the predicted runtimes of the current operation and the predicted runtimes of all unexecuted operations minus the running time of the current operation can be used as the second interval to obtain a more accurate second interval. This allows for a more precise determination of whether there will be a conflict between the batch jobs in the waiting queue and the currently running batch jobs, enabling the batch jobs in the waiting queue to enter the ready queue as soon as possible.

[0120] The batch job processing method provided in this application obtains the predicted execution time of the current operation based on the category of the current operation and the predicted execution time of each category of operation; obtains the operations that have not yet been executed in the running batch job based on the description results of the running batch job and the current operation; and obtains the predicted execution time of each operation that has not yet been executed based on the category of each operation that has not yet been executed and the predicted execution time of each category of operation, thereby obtaining a second interval time. This method obtains the time required for the running batch job to release the occupied resources after execution is completed in a relatively accurate manner, providing a basis for subsequent judgment of conflict relationships.

[0121] S404: If the first interval is less than the second interval, generate conflict relationship data between batch jobs in the waiting queue and batch jobs that are currently running.

[0122] Specifically, if the first interval is greater than the second interval, it means that even if the batch jobs in the waiting queue run immediately, when they reach the point of requesting a conflicting resource, the batch job that is currently running that is occupying the resource has already finished running and released the conflicting resource. The resource will not cause a substantial conflict, so there is no conflict between the batch jobs in the waiting queue and the batch jobs that are currently running.

[0123] If the first interval is less than the second interval, it indicates that there is a possibility that when a batch job in the waiting queue requests a conflicting resource, that resource is still occupied by a running batch job, causing the batch job in the waiting queue to fail. Therefore, conflict relationship data between batch jobs in the waiting queue and running batch jobs is generated to avoid batch jobs in the waiting queue and running batch jobs running at the same time, which would lead to failure and unnecessary waste of resources.

[0124] The conflict relationship data includes batch job identifiers of batch jobs in the waiting queues with conflicts and batch jobs that are currently running. In addition, it may also include resource conflict operations and the resources required by resource conflict operations, etc. This application does not limit this.

[0125] The batch job processing method provided in this application obtains resource conflict operations between batch jobs in the waiting queue and currently running batch jobs; obtains a first interval time based on the resource conflict operations, the predicted running time of each type of operation, and the description results of the batch jobs in the waiting queue; obtains the current operation of the currently running batch jobs, and obtains a second interval time based on the predicted running time of each type of operation and the description results of the currently running batch jobs; compares the first interval time and the second interval time to obtain conflict relationship data. While ensuring the accuracy of conflict prediction and reducing the probability of conflict, it avoids the situation where jobs cannot enter the ready queue as long as there is a conflict. It accurately predicts the resource occupation time, ensures that jobs in the waiting queue can enter the ready queue as soon as possible, and improves the probability of job completion and job execution efficiency.

[0126] exist Figure 3 Based on the embodiments, the batch job processing method provided in this application further includes:

[0127] The predicted runtime of each batch job is obtained based on the description results of each batch job and the predicted runtime of each type of operation obtained in advance.

[0128] Specifically, the server obtains the operations contained in each batch job based on the description results of each batch job, and uses the predicted runtime of the category to which each operation belongs as the predicted runtime of the corresponding operation, and uses the sum of the predicted runtimes of all operations in a batch job as the predicted runtime of the batch.

[0129] The batch job processing method provided in this application obtains the predicted running time of each batch job based on the description results of each batch job and the predicted running time of each type of operation obtained in advance, and provides a sorting basis for the allocation of actuators.

[0130] Figure 7 This is a flowchart of a batch job processing method provided in an embodiment of this application, such as... Figure 7 As shown, based on the above embodiments, the batch job processing method provided in this application further includes:

[0131] S701: Obtain historical execution time data for each type of operation;

[0132] Specifically, historical execution time data includes the time elapsed from start to finish or from start to failure for each type of operation in history.

[0133] S702: Based on historical execution time data, obtain the number of operations of each category that were completed and the number that failed to be completed within different preset time periods;

[0134] Specifically, since not all batch jobs complete successfully, a certain percentage will fail due to resource constraints causing timeouts or other reasons. Therefore, predictions cannot be made solely based on the execution time of completed batch jobs. Conversely, failed jobs cannot be simply considered completed jobs, as this would lead to an underestimation of the predicted execution time, potentially failing to prevent conflicts. The server uses historical execution time data to obtain the number of completed and failed operations for each category within different preset time periods.

[0135] For example, the historical execution data of a certain type of operation is shown in Table 1:

[0136] Table 1 Historical Execution Data of a Certain Type of Operation

[0137]

[0138]

[0139] The preset time length is set in seconds, and the number of operations of this type that are completed, failed, and are currently being executed within 1-6 seconds are shown in Table 2.

[0140] Table 2. Statistical table of execution status of a certain type of operation within 1-6 seconds.

[0141]

[0142] S703: Calculate the completion probability of each type of operation within each preset time length in ascending order of preset time length; the completion probability of each type of operation within each preset time length is calculated based on the completion probability of the previous preset time length and the number of completed operations and the number of failed operations within the corresponding preset time period.

[0143] Specifically, the probability of a class of operations completing within each preset time period can be calculated using the following formula:

[0144]

[0145] N t =N t-1 -C t-1 -D t-1 (2)

[0146] P t =1-S t (3)

[0147] Among them, S t Let P be the probability that an operation is still being executed after a preset time length t. t Let N be the probability that a class of operations will complete within a preset time length t. t D represents the number of operations that have not been completed and have not failed at the start of the t-th preset time period. t C represents the number of operations completed within the t-th preset time period. t Let S0 be the number of operations that failed to execute within the t-th preset time period. Where S0 = 1, C0 = 0, and D0 = 0.

[0148] Taking the data in Table 2 as an example, the probability of this type of operation completing within 1-6 seconds can be obtained as shown in Table 3:

[0149] Table 3. Probability of a certain type of operation being completed within 1-6 seconds.

[0150]

[0151] S704: Select the minimum time length at which the probability of completion is greater than the preset probability as the predicted execution time for each type of operation.

[0152] Specifically, the server selects the minimum time length during which the probability of execution completion is greater than a preset probability, i.e., the time length during which the probability of execution completion first reaches the preset probability, as the predicted execution time for each type of operation. Taking the data in Table 3 as an example, if the preset probability is 0.5, then the predicted execution time for this type is 5 seconds.

[0153] The batch job processing method provided in this application obtains the number of operations completed and the number of operations that failed to complete in different preset time periods based on historical execution time data; calculates the completion probability of each type of operation in each preset time period in ascending order of preset time length; selects the minimum time length in which the completion probability is greater than the preset probability as the predicted execution time of each type of operation. While obtaining the predicted time length, it fully considers the impact of execution failure on the operation execution time length and the difference between operation execution failure and execution completion, thereby improving the accuracy of the predicted execution time of each operation.

[0154] Based on the above embodiments, the batch processing method provided in this application further includes, after S104:

[0155] Determine if the number of retries for a failed batch job has reached the preset maximum. If it has, skip the batch job; otherwise, increment the retries count and add the batch job to the waiting queue.

[0156] Specifically, the server determines whether the number of retries for a failed batch job has reached the preset maximum. If the maximum has been reached, the server calls the job launcher for that batch job to abandon it, allowing the batch job to be skipped. A suitable job launcher, such as JobLauncher or JobOperator, can be selected based on the actual situation; this application does not impose any restrictions on this. If the maximum has not been reached, the number of retries for the batch job is incremented by one, and it is placed in a waiting queue to await the next retrieval.

[0157] In one embodiment, when a batch job reaches the maximum number of retries but still fails to run successfully, the batch job is skipped and will not be rerun. To prevent data loss, the relevant information of the skipped batch job can be stored in the database for administrators to view and process, thereby further ensuring data integrity.

[0158] In addition, the success rate of batch jobs can be regularly analyzed. If the maximum number of retries is set too low, it may lead to a high failure rate, increasing the workload of subsequent tasks. Therefore, the maximum number of retries can be adjusted periodically based on the success rate. If the success rate is lower than the preset success rate, the maximum number of retries should be increased to ensure the probability of successful completion of batch jobs. Similarly, when the success rate is high enough, the maximum number of retries can be appropriately reduced to avoid individual batch jobs having to be repeatedly retried.

[0159] The batch job processing method provided in this application determines whether the number of reruns for failed batch jobs has reached a preset maximum. If the maximum number has been reached, the batch job is skipped; if the maximum number has not been reached, the number of reruns is incremented by one, and the batch job is placed in a waiting queue. This prevents the batch job from being repeatedly rerun, thus avoiding resource waste and improving resource utilization efficiency.

[0160] The following uses a batch database operation job as an example to illustrate the specific implementation process of the batch job processing method provided in this application embodiment.

[0161] Figure 8 This is a flowchart illustrating a batch job processing method provided in an embodiment of this application, as shown below. Figure 8 As shown, based on the above embodiments, the batch job processing method provided in this application further includes:

[0162] S801: Describe the batch job using the operations to be performed and the resources required for each operation;

[0163] Specifically, because databases need to maintain data consistency, multiple batch jobs can be allowed to perform query operations on a single record simultaneously. However, while a batch job is querying a record, it is not allowed to modify that record. Similarly, when one batch job modifies a record, other batch jobs are not allowed to access or modify that record. This type of lock, which allows multiple batch jobs to perform queries but not modifications, is called a shared lock; the lock required for modification operations is called an exclusive lock. Therefore, database batch job operations require not only executor resources but also lock resources. When a batch job wants to acquire a shared lock to perform a query operation on a record, it also needs to check whether the exclusive lock for that record is occupied. If the exclusive lock is not occupied, it means that no other batch job is modifying that record, and the shared lock for that record can be acquired. Therefore, batch jobs are described using the exclusive locks required by each operation within the batch job.

[0164] S802: Use the job initiator to start the batch job described, generate instances of the started batch job and put them into the waiting queue;

[0165] Specifically, the server periodically calls JobLauncher to start the batch jobs described, and generates instances of the already started batch jobs, placing them in the waiting queue.

[0166] S803: Generate conflict relationship data based on the description results of batch jobs in the waiting queue, the description results of batch jobs currently running, and the predicted running time of each type of operation obtained in advance;

[0167] Specifically, based on the description results of batch jobs in the waiting queue and the description results of batch jobs currently running, it is determined whether the operations in the batch jobs in the waiting queue and the operations in the batch jobs currently running involve the same exclusive lock. Furthermore, based on the predicted running time of each type of operation, it is determined whether operations requiring the same exclusive lock will be executed at the same time, thus obtaining conflict relationship data.

[0168] S804: When an idle executor exists, update the pre-obtained conflict relationship data; the conflict relationship data contains the conflict relationships between the batch jobs in the waiting queue and the resources required by the batch jobs that are currently running;

[0169] Specifically, when there are idle executors, the batch jobs in the running state change. Therefore, it is necessary to update the conflict relationship between the batch jobs in the waiting queue and the resources required by the running batch jobs.

[0170] S805: Traverse the waiting queue and, based on the conflict relationship data, add batch jobs that do not conflict with the currently running batch jobs to the ready queue.

[0171] Specifically, the server iterates through the waiting queue, adds batch jobs that do not conflict with currently running batch jobs to the ready queue, and removes them from the waiting queue.

[0172] S806: Select the batch job with the longest predicted execution time from the ready queue and run it;

[0173] Specifically, when there are idle executors in the server, the server assigns the idle executor to the batch job with the longest predicted running time in the ready queue, so that the batch job assigned to the executor changes from the ready state to the running state and is removed from the ready queue.

[0174] S807: During operation, monitor the lock resource status. If the lock resource requested by the running batch job is already occupied, the batch job whose lock resource is occupied will fail immediately and a rollback operation will be performed.

[0175] Specifically, during operation, the server monitors the occupancy of lock resources in the system. When a batch job in the running state requests a lock resource, if the lock resource is already occupied, it will not wait, but will immediately fail, release the occupied resources, and perform a rollback operation to restore the state before the current operation.

[0176] S808: Determine whether the number of retries for a failed batch job has reached the preset maximum. If the maximum has been reached, skip the batch job; otherwise, increment the number of retries and add the batch job to the waiting queue.

[0177] Specifically, the server determines whether the number of retries for a failed batch job has reached the preset maximum. If it has, the server calls the JobLauncher of the batch job to mark the task execution status of the batch job as failed (FAILED), allowing the batch job to be skipped. If the maximum number of retries has not been reached, the server increments the number of retries for the batch job and places it in a waiting queue to wait for the next retrieval.

[0178] The batch job processing method provided in this application updates pre-obtained conflict relationship data when idle executors exist; iterates through the waiting queue and, based on the conflict relationship data, places batch jobs in the waiting queue that do not conflict with currently running batch jobs into the ready queue; it selects the batch job with the longest predicted execution time from the ready queue for execution; during execution, it monitors the lock resource status, and if the lock resource requested by a running batch job is already occupied, the batch job requesting the occupied lock resource immediately fails and performs a rollback operation to release the occupied resource, thereby saving resource usage and improving resource utilization efficiency. Simultaneously, it prevents batch jobs whose required resources conflict with those of running batch jobs from entering the ready queue, reducing the probability of conflicts and thus reducing the probability of running batch jobs rerunning. This further improves resource utilization efficiency while avoiding the loss of business data.

[0179] Based on the same inventive concept, this application also provides a batch job processing apparatus, which can be used to implement the methods described in the above embodiments, as described in the following embodiments. Since the principle of the batch job processing apparatus in solving the problem is similar to that of the batch job processing method, the implementation of the batch job processing apparatus can refer to the implementation of the method based on software performance benchmarks, and will not be repeated. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0180] Figure 9 This is a schematic diagram of the structure of a batch processing device provided in an embodiment of this application, as shown below. Figure 9 As shown, the batch processing apparatus provided in this application includes:

[0181] The conflict relationship update unit 910 is used to update the pre-obtained conflict relationship data when there is an idle executor; the conflict relationship data contains the conflict relationships between the batch jobs in the waiting queue and the resources required by the batch jobs that are running.

[0182] Specifically, since the total number of batch jobs is often greater than the number of executors, idle executors often come from the resources released by batch jobs that have completed or failed. At this time, the batch jobs in the running state change, so the conflict relationship update unit 910 needs to update the conflict relationship between the batch jobs in the waiting queue and the resources required by the running batch jobs.

[0183] The ready conversion unit 920 is used to traverse the waiting queue and put batch jobs that do not conflict with the currently running batch jobs into the ready queue according to the conflict relationship data.

[0184] Specifically, after the conflict relationship data is updated, there may be new batch jobs in the waiting queue that do not conflict with the currently running batch jobs. The ready conversion unit 920 traverses the waiting queue, puts the batch jobs in the waiting queue that do not conflict with the currently running batch jobs into the ready queue, and removes them from the waiting queue.

[0185] The job execution unit 930 is used to select the batch job with the longest predicted execution time from the ready queue for execution;

[0186] Specifically, since long-running jobs often require more resources and are more likely to conflict with other jobs, prioritizing the execution of the batch job with the longest predicted runtime can prevent other jobs that conflict with that job from entering the ready queue, thus avoiding conflicts with the longest-running job and improving the success rate of completing batch jobs that consume more resources and have a higher probability of conflict. When there are idle executors on the server, the job execution unit 930 allocates the idle executor to the batch job with the longest predicted runtime in the ready queue, causing the batch job allocated to the executor to change from the ready state to the running state, and then removes it from the ready queue.

[0187] The running status monitoring unit 940 is used to monitor the lock resource status during operation. If the lock resource requested by the running batch job is already occupied, the batch job whose lock resource is occupied will fail immediately and a rollback operation will be performed.

[0188] Specifically, during operation, the running status monitoring unit 940 monitors the occupancy of lock resources in the system. When a batch job in the running state requests a lock resource, if the lock resource is already occupied, it will no longer wait, causing the batch job to fail immediately, releasing the occupied resources, and performing a rollback operation to restore the state before the current operation.

[0189] The batch job processing device provided in this application, through the conflict relationship update unit 910, the ready conversion unit 920, the job execution unit 930, and the running status monitoring unit 940, releases occupied resources in a timely manner, thereby saving resources and improving resource utilization efficiency. At the same time, it avoids batch jobs that conflict with the required resources from entering the ready queue, reducing the probability of conflicts and thus reducing the probability of running batch jobs being rerun. This further improves resource utilization efficiency while avoiding the loss of business data.

[0190] Figure 10 This is a schematic diagram of the structure of a batch processing device provided in an embodiment of this application, as shown below. Figure 10 As shown, based on the above embodiments, the batch processing apparatus provided in this application further includes:

[0191] The second conflict relationship update unit 1010 is used to update the pre-obtained conflict relationship data when a new batch job enters the waiting queue.

[0192] The second ready conversion unit 1020 is used to traverse the waiting queue and, based on the conflict relationship data, put batch jobs that do not conflict with the currently running batch jobs into the ready queue.

[0193] The batch job processing device provided in this application ensures that the conflict relationship data is up-to-date through the second conflict relationship update unit 1010 and the second ready conversion unit 1020, so that batch jobs that do not conflict with the batch jobs that are running in the waiting queue can enter the ready queue in a timely manner.

[0194] Figure 11 This is a schematic diagram of the structure of a batch processing device provided in an embodiment of this application, as shown below. Figure 11 As shown, based on the above embodiments, the batch processing apparatus provided in this application further includes:

[0195] Batch job description unit 1110 is used to describe batch jobs using the operations to be performed and the resources required for each operation;

[0196] The job instance generation unit 1120 is used to start the batch jobs described by the job initiator, generate instances of the started batch jobs, and put them into the waiting queue.

[0197] The conflict relationship generation unit 1130 is used to generate conflict relationship data based on the description results of batch jobs in the waiting queue, the description results of batch jobs currently running, and the predicted running time of each type of operation obtained in advance.

[0198] The batch job processing device provided in this application, through the batch job description unit 1110, job instance generation unit 1120 and conflict relationship generation unit 1130, realizes the scientific description of batch jobs and the early prediction of conflicts, reduces the probability of conflicts, avoids the waste of resources caused by multiple failures and reruns, reduces the number of reruns, and improves the probability of job completion and resource utilization efficiency.

[0199] Figure 12 This is a schematic diagram of the structure of a batch processing device provided in an embodiment of this application, as shown below. Figure 12 As shown, based on the above embodiments, the conflict relationship generation unit 1130 in the batch job processing apparatus provided in this application further includes:

[0200] The resource conflict operation acquisition module 1131 is used to acquire resource conflict operations between batch jobs in the waiting queue and batch jobs that are currently running.

[0201] The first interval time calculation module 1132 is used to obtain the first interval time based on the predicted running time of the resource conflict operation, the operation of each type, and the description results of the batch jobs in the waiting queue. The first interval time is the time required to execute the batch jobs in the waiting queue until the resource conflict operation.

[0202] The second interval time calculation module 1133 is used to obtain the current operation of the running batch job, and obtain the second interval time based on the predicted running time of each type of operation and the description result of the running batch job. The second interval time is the time required for the running batch job to complete execution.

[0203] The conflict relationship data generation module 1134 is used to generate conflict relationship data between batch jobs in the waiting queue and batch jobs that are running, if the first interval time is less than the second interval time.

[0204] The batch job processing device provided in this application, through the resource conflict operation acquisition module 1131, the first interval time calculation module 1132, the second interval time calculation module 1133, and the conflict relationship data generation module 1134, ensures the accuracy of conflict prediction, reduces the probability of conflict, and avoids the situation where jobs cannot enter the ready queue as long as there is a conflict. It accurately predicts the resource occupation time, ensures that jobs in the waiting queue can enter the ready queue as soon as possible, and improves the probability of job completion and job execution efficiency.

[0205] Figure 13 This is a schematic diagram of the structure of a batch processing device provided in an embodiment of this application, as shown below. Figure 13 As shown, in Figure 11 Based on the embodiments, the batch processing apparatus provided in this application further includes:

[0206] The batch job prediction runtime calculation unit 1310 is used to obtain the predicted runtime of each batch job based on the description results of each batch job and the predicted runtime of each type of operation obtained in advance.

[0207] The batch processing apparatus provided in this application obtains the predicted running time of each batch job through the batch job predicted running time calculation unit 1310, and provides a sorting basis for the allocation of actuators.

[0208] Figure 14 This is a schematic diagram of the structure of a batch processing device provided in an embodiment of this application, as shown below. Figure 14As shown, based on the above embodiments, the batch processing apparatus provided in this application further includes:

[0209] Historical execution time data acquisition unit 1410 is used to acquire historical execution time data for various types of operations;

[0210] The quantity statistics unit 1420 is used to obtain the number of operations of each category that were completed and the number that failed to be completed within different preset time periods based on historical execution time data.

[0211] The completion probability calculation unit 1430 is used to calculate the completion probability of various operations within each preset time length in ascending order of preset time length; the completion probability of various operations within each preset time length is calculated based on the completion probability within the previous preset time length and the number of completed operations and the number of failed operations within the corresponding preset time period.

[0212] The predicted execution time acquisition unit 1440 is used to select the minimum time length in which the probability of completion is greater than a preset probability as the predicted execution time of each type of operation.

[0213] The batch processing device provided in this application, through the historical execution time data acquisition unit 1410, the quantity statistics unit 1420, the execution completion probability calculation unit 1430, and the predicted execution time acquisition unit 1440, fully considers the impact of execution failure on the operation execution time and the difference between operation execution failure and execution completion while obtaining the predicted time length, thereby improving the accuracy of the predicted execution time of each operation.

[0214] Figure 15 This is a schematic diagram of the structure of a batch processing device provided in an embodiment of this application, as shown below. Figure 15 As shown, based on the above embodiments, the batch processing apparatus provided in this application further includes:

[0215] The rerun judgment unit 1510 is used to determine whether the number of reruns for a failed batch job has reached the preset maximum number. If the maximum number has been reached, the batch job is skipped; if the maximum number has not been reached, the number of reruns is incremented by one, and the batch job is placed in the waiting queue.

[0216] The batch job processing method provided in this application, through the rerun judgment unit 1510, prevents batch jobs from being repeatedly rerun, thus avoiding resource waste and improving resource utilization efficiency.

[0217] Figure 16 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, as shown below. Figure 16As shown, the electronic device may include a processor 1601, a communications interface 1602, a memory 1603, and a communication bus 1604. The processor 1601, communications interface 1602, and memory 1603 communicate with each other via the communication bus 1604. The processor 1601 can call logical instructions in the memory 1603 to execute the following methods: when an idle executor exists, update pre-obtained conflict relationship data; traverse the waiting queue and, based on the conflict relationship data, place batch jobs in the waiting queue that do not conflict with currently running batch jobs into the ready queue; select the batch job with the longest predicted execution time from the ready queue and run it; during execution, monitor the lock resource status, and if the lock resource requested by a running batch job is already occupied, immediately fail the batch job whose lock resource is occupied and perform a rollback operation.

[0218] Furthermore, the logical instructions in the aforementioned memory 1603 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0219] This embodiment discloses a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided in the above-described method embodiments, such as: when there is an idle executor, updating pre-obtained conflict relationship data; traversing the waiting queue and, according to the conflict relationship data, placing batch jobs in the waiting queue that do not conflict with currently running batch jobs into the ready queue; selecting the batch job with the longest predicted running time from the ready queue for execution; and monitoring the lock resource status during execution. If the lock resource requested by the running batch job is already occupied, the batch job whose lock resource is occupied will immediately fail and a rollback operation will be performed.

[0220] This embodiment provides a computer-readable storage medium storing a computer program that causes the computer to execute the methods provided in the above-described method embodiments. For example, the methods include: updating pre-obtained conflict relationship data when an idle executor exists; traversing the waiting queue and, based on the conflict relationship data, placing batch jobs in the waiting queue that do not conflict with currently running batch jobs into a ready queue; selecting the batch job with the longest predicted running time from the ready queue for execution; monitoring the lock resource status during execution, and if the lock resource requested by a running batch job is already occupied, immediately causing the batch job with the occupied lock resource to fail and performing a rollback operation.

[0221] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0222] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0223] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0224] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0225] In the description of this specification, the references to terms such as "an embodiment," "a specific embodiment," "some embodiments," "for example," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0226] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of processing batch jobs, characterized by, include: When an idle executor exists, update the pre-obtained conflict relationship data; The conflict relationship data includes the conflict relationships between batch jobs in the waiting queue and the resources required by the batch jobs that are currently running; Traverse the waiting queue and, based on the conflict relationship data, place batch jobs in the waiting queue that do not conflict with the currently running batch jobs into the ready queue; Select the batch job with the longest predicted execution time from the ready queue and run it; During operation, the lock resource status is monitored. If the lock resource requested by the running batch job is already occupied, the batch job whose lock resource is occupied will fail immediately and a rollback operation will be performed. Before updating the pre-obtained conflict relationship data, the following is also included: Describe the batch job using the operations to be performed and the resources required for each operation; Use the job launcher to start the batch jobs described, generate instances of the started batch jobs, and put them into the waiting queue; Conflict relationship data is generated based on the description results of batch jobs in the waiting queue, the description results of batch jobs currently running, and the predicted running time of each type of operation obtained in advance. During the process of monitoring the lock resource status, if the lock resource requested by the batch job is already occupied, the batch job will immediately fail and a rollback operation will be performed. The process also includes: Determine whether the number of retries for a failed batch job has reached a preset maximum. If the maximum number has been reached, skip the batch job. If the maximum number has not been reached, increment the number of retries and place the batch job in a waiting queue.

2. The batch job processing method of claim 1, wherein, Also includes: When a new batch job enters the waiting queue, update the pre-obtained conflict relationship data; Traverse the waiting queue and, based on the conflict relationship data, place batch jobs in the waiting queue that do not conflict with the currently running batch jobs into the ready queue.

3. The batch job processing method of claim 1, wherein, The generation of conflict relationship data based on the descriptions of batch jobs in the waiting queue, the descriptions of currently running batch jobs, and the pre-obtained predicted running times for each type of operation includes: The operation to identify resource conflicts between batch jobs in the waiting queue and currently running batch jobs is performed. A first interval time is obtained based on the resource conflict operation, the predicted running time of each type of operation, and the description results of the batch jobs in the waiting queue. The first interval time is the time required to execute the batch jobs in the waiting queue up to the resource conflict operation. Obtain the current operation of the running batch job, and obtain the second interval time based on the predicted running time of each type of operation and the description result of the running batch job. The second interval time is the time required for the running batch job to complete execution. If the first interval is less than the second interval, conflict relationship data between the batch jobs in the waiting queue and the batch jobs that are currently running is generated.

4. The batch job processing method of claim 3, wherein, The step of obtaining the first interval time based on the resource conflict operation, the predicted runtime of each type of operation, and the description results of the batch jobs in the waiting queue includes: Based on the description of the batch jobs in the waiting queue, the pre-operations required for the batch jobs in the waiting queue to be executed before the resource conflict operation are obtained; The predicted runtime of each preceding operation is obtained based on the category of each preceding operation and the predicted runtime of each category of operation. The first interval time is obtained based on the predicted running time of each preceding operation.

5. The batch job processing method of claim 3, wherein, The step of obtaining the current operation of the running batch job and obtaining the second interval time based on the predicted execution time of each type of operation and the description result of the running batch job includes: The predicted runtime of the current operation is obtained based on the category of the current operation and the predicted runtime of each category of operation. Based on the description of the currently running batch job and the current operation, the operations that the currently running batch job has not yet executed are obtained; The predicted runtime of each unexecuted operation is obtained based on the category of each unexecuted operation and the predicted runtime of each category of operation; The second interval time is obtained based on the predicted runtime of the current operation and the predicted runtime of each of the operations that have not yet been executed.

6. The batch job processing method of claim 1, wherein, Also includes: The predicted runtime of each batch job is obtained based on the description results of each batch job and the predicted runtime of each type of operation obtained in advance.

7. The batch job processing method of claim 1, wherein, Also includes: Obtain historical execution time data for each type of operation; Based on the historical execution time data, obtain the number of operations of each category that were completed and the number that failed to be completed within different preset time periods; The probability of each type of operation being completed within each preset time length is calculated sequentially in ascending order of preset time length. The probability of completing each type of operation within each preset time period is calculated based on the probability of completion within the previous preset time period and the number of completed and failed operations within the corresponding preset time period. The minimum time length at which the probability of completion is greater than the preset probability is selected as the predicted execution time for each type of operation.

8. A batch job processing apparatus characterized by comprising: include: The conflict relationship update unit is used to update the pre-obtained conflict relationship data when there is an idle executor. The conflict relationship data includes the conflict relationships between batch jobs in the waiting queue and the resources required by the batch jobs that are currently running; The ready conversion unit is used to traverse the waiting queue and, based on the conflict relationship data, put batch jobs in the waiting queue that do not conflict with the batch jobs that are currently running into the ready queue. The job execution unit is used to select the batch job with the longest predicted execution time from the ready queue for execution; The running status monitoring unit is used to monitor the lock resource status during operation. If the lock resource requested by the running batch job is already occupied, the batch job whose lock resource is occupied will fail immediately and a rollback operation will be performed. The batch processing device further includes: A batch job description unit is used to describe a batch job using the operations to be performed and the resources required for each operation. The job instance generation unit is used to start the batch jobs described by the job starter, generate instances of the started batch jobs, and put them into the waiting queue. The conflict relationship generation unit is used to generate conflict relationship data based on the description results of batch jobs in the waiting queue, the description results of batch jobs currently running, and the predicted running time of each type of operation obtained in advance. The device further includes a rerun judgment unit, used to determine whether the number of reruns for the failed batch job has reached a preset maximum number. If the maximum number has been reached, the batch job is skipped; if the maximum number has not been reached, the number of reruns is incremented by one, and the batch job is placed in a waiting queue.

9. A computer electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

11. A computer program product, characterised in that, The computer program product includes a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images