Batch sub-job processing method, device, equipment, medium and program product

Abstract

This application provides a batch sub-job processing method, apparatus, equipment, medium, and program product, relating to the field of big data technology. It includes: acquiring historical processing data and pending data for batch sub-jobs to determine the longest and average processing time for a single record processing path, and accordingly determining the initial allocation amount for each sub-job in the batch; dividing the pending data into multiple data ranges; allocating each data range to multiple sub-jobs according to the initial allocation amount, and storing unallocated pending data ranges; monitoring the execution process of each sub-job to determine its completion status and current resource usage; and reclaiming or retrieving the stored unallocated pending data ranges based on the completion status and current resource usage of each sub-job, and reallocating the data ranges of each sub-job; thereby achieving the effect of improving sub-job processing speed and enhancing resource utilization.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a method, apparatus, equipment, medium and program product for processing batch sub-jobs. Background Technology

[0002] In batch processing scenarios within financial institutions, data processing typically involves the centralized processing of business records. For example, a bank's interest payment process requires completing multiple stages. Each stage is broken down into multiple sub-jobs for parallel processing to improve overall efficiency. However, due to the complexity of the business logic in banking systems, the processing paths and time consumption for different data records vary significantly. Therefore, developing an efficient batch sub-job processing method has become a promising application direction for improving processing efficiency.

[0003] In existing technologies, batch subjob processing methods primarily utilize pre-allocated processors to statically distribute data ranges among subjobs. After the batch job console starts the job, the pre-allocated processor calculates the total amount of data to be processed and distributes the data range evenly according to the preset number of subjobs. For example, 1 million data records are evenly distributed among 10 subjobs, with each subjob processing 100,000 data records. The job executor processes the data based on the allocated boundary values ​​and reports the status back to the console upon completion. The core principle is static equalization, assuming that all data records take the same processing time and the number of subjobs is fixed.

[0004] However, existing technical solutions suffer from the "weakest link" effect due to the mismatch between the average data range and the actual processing time, and lack dynamic adjustment capabilities. They cannot effectively recover and redistribute unprocessed data when increasing or decreasing the number of sub-jobs. At the same time, resource utilization is low, and sub-jobs that are completed quickly cannot acquire new tasks. Affected by the above factors, existing technologies have technical problems such as slow sub-job processing speed and low resource utilization. Summary of the Invention

[0005] This application provides a method, apparatus, equipment, medium, and program product for processing batch sub-jobs, in order to solve the technical problems of slow sub-job processing speed and low resource utilization in the prior art.

[0006] Firstly, this application provides a method for processing batch sub-jobs, including:

[0007] Retrieve historical processing data and pending data for batch subjobs;

[0008] Based on historical processing data, determine the longest and average processing time for a single record processing path;

[0009] Determine the initial allocation of each sub-job in the batch sub-job based on the longest and average time.

[0010] Divide the data to be processed into multiple data ranges;

[0011] Based on the initial allocation, each data range is assigned to multiple sub-jobs, and unassigned data ranges to be processed are stored.

[0012] Monitor the execution process of each subjob to determine its completion status and current resource usage.

[0013] Based on the completion status and current resource usage of each subjob, the stored but unallocated data range to be processed is reclaimed or retrieved, and the data range of each subjob is reallocated.

[0014] Secondly, this application provides a batch sub-job processing apparatus, comprising:

[0015] The acquisition module is used to acquire historical processing data and pending data for batch sub-jobs;

[0016] The first processing module is used to determine the longest and average processing time for a single record processing path based on historical processing data.

[0017] The second processing module is used to determine the initial allocation of each sub-job in the batch sub-job based on the longest and average time consumption.

[0018] The partitioning module is used to divide the data to be processed into multiple data ranges;

[0019] The third processing module is used to allocate each data range to multiple sub-jobs according to the initial allocation amount, and to store the unallocated data ranges to be processed.

[0020] The monitoring module is used to monitor the execution process of each sub-job in order to determine the completion status and current resource usage of each sub-job.

[0021] The fourth processing module is used to reclaim or retrieve the stored, unallocated data range to be processed based on the completion status and current resource usage of each sub-job, and to reallocate the data range of each sub-job.

[0022] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0023] The memory stores the instructions that the computer executes;

[0024] The processor executes computer execution instructions stored in memory, causing the processor to perform the method described in the first aspect above.

[0025] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect above.

[0026] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0027] This application provides a method, apparatus, device, medium, and program product for processing batch sub-jobs. By acquiring historical processing data and pending data of batch sub-jobs, and determining the longest and average processing time of a single record processing path based on historical data, the initial allocation of each sub-job is determined accordingly, making the initial allocation more in line with actual processing capacity and reducing waiting caused by unreasonable allocation. The pending data is divided into multiple data ranges and allocated to sub-jobs, while unallocated data is stored to avoid overload or under-allocation of the initial allocation. The execution process of each sub-job is monitored, and the unallocated pending data ranges are reclaimed or retrieved and reallocated according to the completion status and current resource occupancy, so that resources are fully utilized and sub-jobs have continuous task processing, thereby achieving the technical effect of improving the processing speed of sub-jobs and enhancing resource utilization. Attached Figure Description

[0028] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0029] Figure 1 This application provides a schematic diagram of an application data processing system architecture.

[0030] Figure 2 A flowchart illustrating a batch sub-job processing method provided in this application embodiment. Figure 1 ;

[0031] Figure 3 A flowchart illustrating a batch sub-job processing method provided in this application embodiment. Figure 2 ;

[0032] Figure 4 A flowchart illustrating a batch sub-job processing method provided in this application embodiment. Figure 3 ;

[0033] Figure 5 A flowchart illustrating a batch sub-job processing method provided in this application embodiment. Figure 4 ;

[0034] Figure 6A flowchart illustrating a batch sub-job processing method provided in this application embodiment. Figure 5 ;

[0035] Figure 7 A schematic diagram of the structure of a batch sub-job processing system provided in an embodiment of this application;

[0036] Figure 8 A schematic diagram of a batch sub-job processing device provided in an embodiment of this application;

[0037] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0038] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0039] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0040] It should be noted that the batch sub-job processing method, apparatus, equipment, medium and program products provided in this application can be used in the field of big data technology, or in any field other than big data technology. The application fields of the batch sub-job processing method, apparatus, equipment, storage medium and products in this application are not limited.

[0041] Existing technical solutions suffer from the "weakest link" effect due to the mismatch between the average data range and the actual processing time. Furthermore, they lack dynamic adjustment capabilities, making it impossible to effectively recover and redistribute unprocessed data when increasing or decreasing the number of sub-jobs. Additionally, they have low resource utilization, and rapidly completed sub-jobs cannot acquire new tasks. As a result of these factors, existing technologies suffer from slow sub-job processing speed and low resource utilization.

[0042] To address the aforementioned issues, this application provides a method, apparatus, device, medium, and program product for processing batch sub-jobs. By acquiring historical processing data and pending data for batch sub-jobs, and determining the longest and average processing time for a single record processing path based on historical data, the initial allocation for each sub-job is determined, making the initial allocation more aligned with actual processing capacity and reducing waiting time caused by unreasonable allocation. The pending data is divided into multiple data ranges and allocated to sub-jobs, with unallocated data stored to avoid initial allocation overload or under-allocation. The execution process of each sub-job is monitored, and based on completion status and current resource usage, unallocated pending data ranges are reclaimed or retrieved and reallocated, ensuring full resource utilization and continuous task processing for sub-jobs. This overall achieves the technical effect of improving sub-job processing speed and enhancing resource utilization.

[0043] Figure 1 This is a schematic diagram of an application data processing system architecture provided in an embodiment of this application. The application data processing system can be a computer device. Figure 1 As shown, the above architecture includes at least one of a data acquisition device 101, a processing device 102, and a display device 103.

[0044] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the architecture of the application data processing system. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0045] In the specific implementation process, the data acquisition device 101 may include an input / output interface or a communication interface, and the data acquisition device 101 can be connected to the processing device through the input / output interface or the communication interface.

[0046] The processing device 102 can acquire historical processing data and pending data of batch sub-jobs, determine the longest and average processing time of a single record based on the historical data, and determine the initial allocation of sub-jobs accordingly; divide the pending data into multiple ranges and allocate them, and store the unallocated pending data ranges; monitor the execution status and resource usage of sub-jobs, dynamically reclaim or retrieve the unallocated pending data ranges and reallocate them, thereby optimizing processing speed and resource utilization.

[0047] The display device 103 can also be a touch screen or the screen of a terminal device, used to receive user commands while displaying the above-mentioned content, so as to realize interaction with the user.

[0048] It should be understood that the aforementioned processing device can be implemented by a processor reading instructions from memory and executing those instructions, or it can be implemented by a chip circuit.

[0049] Furthermore, the network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0050] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0051] Figure 2 A flowchart illustrating a batch sub-job processing method provided in this application embodiment. Figure 1 ,like Figure 2 As shown in the embodiments of this application, the batch sub-job processing method includes:

[0052] S201. Obtain historical processing data and pending data for batch sub-jobs.

[0053] Retrieve historical processing data and pending data for batch subjobs. Historical processing data consists of records generated during past batch subjob processing, while pending data is the batch subjob data that needs to be processed currently.

[0054] S202. Based on historical processing data, determine the longest processing time and the average processing time for a single record processing path.

[0055] In one possible implementation, before determining the longest and average processing time of a single record processing path based on historical processing data, the method further includes: filtering the historical processing data according to a preset filtering rule to obtain filtered historical processing data, wherein the filtered historical processing data includes at least the path time data of each path in the single record processing process.

[0056] Based on historical processing data, the longest and average processing times for a single record's processing path are determined. However, before determining the processing times, the historical processing data needs to be filtered according to preset filtering rules. The filtered historical processing data includes the path processing time data for each path during the processing of a single record. The longest and average processing times are calculated based on the filtered data.

[0057] S203. Based on the longest and average time consumption, determine the initial allocation of each sub-job in the batch sub-jobs.

[0058] By combining the longest and average processing times, calculate the initial amount of data that each subjob needs to process, i.e., the initial allocation amount.

[0059] S204. Divide the data to be processed into multiple data ranges.

[0060] In one possible implementation, the data to be processed is divided into multiple data ranges, including: dividing the data to be processed into N data ranges based on the number of currently available sub-jobs.

[0061] In this embodiment, the value of N is matched with the number of currently available sub-jobs; the initial data volume of each data range is determined based on the initial allocation, and the total amount of each data range does not exceed the total data volume of the data to be processed.

[0062] Based on the number of available sub-jobs, the data to be processed is divided into N data ranges, where N matches the number of sub-jobs. The initial data volume of each data range is based on the initial allocation, and the total amount does not exceed the total amount of data to be processed.

[0063] S205. Based on the initial allocation, allocate each data range to multiple sub-jobs and store the unallocated data ranges to be processed.

[0064] The data range is allocated to multiple subjobs according to the initial allocation, and the remaining unallocated data range to be processed is stored.

[0065] S206. Monitor the execution process of each subjob to determine the completion status and current resource usage of each subjob.

[0066] Monitor the execution process of each subjob to determine its completion status and current resource usage. During subjob execution, monitor its completion status and current resource usage in real time.

[0067] S207. Based on the completion status and current resource usage of each sub-job, reclaim or retrieve the stored but unallocated data range to be processed, and reallocate the data range of each sub-job.

[0068] Based on the completion status and resource usage of subjobs, unprocessed data ranges are reclaimed or stored unallocated data ranges are retrieved. At the same time, based on the completion status and current resource usage of each subjob, the data range of each subjob is reallocated.

[0069] This embodiment provides a batch sub-job processing method. By acquiring historical processing data and pending data of batch sub-jobs, the method determines the longest and average processing time of a single record processing path based on historical data, and determines the initial allocation amount for each sub-job accordingly. This makes the initial allocation more in line with the actual processing capacity and reduces waiting caused by unreasonable allocation. The pending data is divided into multiple data ranges and allocated to sub-jobs, while unallocated data is stored to avoid initial allocation overload or under-allocation. The execution process of each sub-job is monitored, and unallocated pending data ranges are reclaimed or retrieved and reallocated according to the completion status and current resource usage. This ensures full utilization of resources and continuous task processing for sub-jobs, thereby achieving the overall technical effect of improving sub-job processing speed and enhancing resource utilization.

[0070] Optional, Figure 3 A flowchart illustrating a batch sub-job processing method provided in this application embodiment. Figure 2 ,like Figure 3 As shown, this embodiment, based on the above embodiments, provides a detailed explanation of the process for determining the initial allocation of each sub-job, including:

[0071] S301. Get the number of currently available subjobs.

[0072] The system retrieves the number of currently available subjobs and reads the maximum number of subjobs (n) that the job needs to be split into. This maximum number of subjobs (n) is the number of currently available subjobs. Upon receiving the job pre-allocation instruction, the system determines the total number of subjobs that the system can process concurrently. This number determines the number of parts of the data to be processed and the basis for subsequent load balancing.

[0073] S302. Based on the number of jobs, the longest time taken, and the average time taken, determine the initial allocation of each sub-job using a preset algorithm.

[0074] Based on the number of jobs, the longest processing time, and the average processing time, the initial allocation of each sub-job is determined using a preset algorithm. The job allocator calculates the total amount of data N to be processed by the batch jobs, reads the upper limit n of sub-jobs, and obtains the longest processing time L for a single record. max (Longest path) and average time L ave (Mean path). Using the preset formula: N1=N*L ave / (n*L max Calculate the initial allocation of N1 (the amount to be processed in each sub-job) and round N1 down. Of the data to be processed, the first N1*n portion is divided into n parts, each N1 part, and the boundary values ​​S1 to S2 are calculated. n The remaining (Sn, Smax] range is stored as unallocated data. If the calculated value is lower than the set minimum value, the minimum value is taken to ensure reasonable allocation.

[0075] This embodiment provides a batch sub-job processing method that accurately obtains the number of available sub-jobs and calculates the initial allocation based on historical latency data, making the data partitioning more consistent with actual processing capacity. It avoids allocation chaos caused by unclear factor job quantities and uses the longest and average latency to balance the load, preventing some sub-jobs from overloading. The mechanism combining pre-allocation and unallocated ranges provides a basis for subsequent dynamic adjustments, improving overall processing efficiency and resource utilization.

[0076] Optional, Figure 4 A flowchart illustrating a batch sub-job processing method provided in this application embodiment. Figure 3 ,like Figure 4 As shown, this embodiment elaborates on a process for reclaiming or retrieving unallocated data ranges based on the above embodiments, including:

[0077] S401. Determine the changes in the number of sub-tasks based on the completion status of each sub-task.

[0078] The system monitors the completion status of subjobs during execution to determine whether the number of subjobs has changed.

[0079] S402. When the number of sub-jobs decreases, reclaim and store the unallocated unprocessed data range corresponding to the reduced sub-jobs.

[0080] When the number of sub-jobs decreases, the unallocated unprocessed data range corresponding to the reduced sub-jobs is reclaimed and stored. The system sends a stop command to the sub-jobs that need to be reduced, the sub-jobs stop data processing, and inform the corresponding storage unit of the unprocessed data range. After receiving the unprocessed data range, the storage unit stores it in the unallocated data range and notifies the system. This process ensures that unprocessed data is reclaimed and stored for subsequent reallocation, avoiding data loss or duplicate processing.

[0081] S403. When the number of sub-jobs increases, based on the current resource usage, retrieve the stored unallocated data range to be processed and allocate the stored unallocated data range to the data range of the newly added sub-jobs.

[0082] For example, when the number of sub-jobs increases, based on the current resource usage, the stored unallocated unprocessed data range is retrieved and allocated to the data range of the newly added sub-job. The newly added sub-job requests data from the storage unit, and the storage unit calculates the total unallocated data N', combined with the current sub-job limit n' and the longest time elapsed per record L. max Average time L ave Calculate the single job allocation quantity N1' (N1'=N'L)ave / (n'L max Then round down. If the data size of the first interval is less than or equal to N1', then allocate directly; otherwise, split the first interval, allocate N1', and update the boundary.

[0083] This embodiment provides a method for processing batch sub-jobs. By dynamically adjusting the number of jobs through real-time monitoring of job status, it reclaims unprocessed data to prevent loss when jobs are reduced, and accurately allocates unallocated data based on historical processing times when jobs are increased. This mechanism enables elastic resource scaling, balances the load of each sub-job, and improves system throughput. Dynamic reallocation reduces idle waiting time, optimizes memory and computing resource utilization, and ensures the high efficiency and stability of batch sub-job processing.

[0084] Optional, Figure 5 A flowchart illustrating a batch sub-job processing method provided in this application embodiment. Figure 4 ,like Figure 5 As shown, this embodiment, based on the above embodiments, provides a supplementary explanation of another process for reclaiming or retrieving unallocated data ranges to be processed, including:

[0085] S501. Determine the completion progress of each sub-task based on its completion status.

[0086] The system determines the completion progress of each sub-job based on its completion status. The system monitors the execution process of each sub-job in real time, comparing the amount of data processed with the total allocated data range. When the amount of data processed reaches 100% of the allocated total, the completion progress is considered complete. If the processing speed is faster than expected, the system marks it as completed ahead of schedule.

[0087] S502. When any subjob completes the allocated data range ahead of schedule, receive the completion request sent by the subjob that has completed the allocation ahead of schedule, and determine whether there is a stored unallocated data range to be processed based on the completion request.

[0088] When any subjob completes its allocated data range ahead of schedule, the system receives a completion request from that subjob and, based on the request, determines whether there are any stored, unallocated data ranges awaiting processing. The subjob that completed its allocation ahead of schedule sends a completion request to the management module, which then queries the storage area. If the total amount of unallocated data in the storage area is greater than zero, it is determined that there are unallocated data ranges awaiting processing.

[0089] S503. If it exists, retrieve the stored unallocated data range to be processed and allocate the stored unallocated data range to the data range of the sub-job that has been allocated in advance.

[0090] If it exists, retrieve the stored unallocated data range to be processed and allocate it to the data range of the pre-allocated subjob. Retrieve the unallocated data and calculate the computational cost N1' (N1'=N'L) according to the reallocation algorithm. ave / (n'L max The calculated data range boundary is assigned to the subjob.

[0091] S504. When any subjob is in a paused state, obtain and store the unprocessed data range corresponding to the paused subjob.

[0092] When any subjob enters a paused state, the system acquires and stores the unprocessed data range corresponding to the paused subjob. If the system detects that a subjob has entered a paused state due to an exception or external instruction, it immediately acquires the current processing breakpoint. The portion from the breakpoint to the end of the original data range is taken as the unprocessed data range and sent to the storage unit for backup.

[0093] S505. Based on the current resource usage, retrieve the unprocessed data range and allocate the processed data range to the data range of other sub-jobs that are not in a paused state.

[0094] Based on the current resource usage, the system retrieves the unprocessed data range and distributes the processed data range to the data ranges of other sub-jobs that are not currently paused. The system assesses the load of active sub-jobs; if the resource utilization rate is below a threshold, it retrieves the stored unprocessed data. The data ranges are then split according to a load balancing strategy and allocated to idle or low-load sub-jobs, updating their data range boundaries to maintain the overall system processing capacity at a high level.

[0095] This embodiment provides a batch sub-job processing method that achieves immediate reclamation and precise reallocation of unallocated data by real-time monitoring of completion progress and pause status. Sub-jobs completed ahead of schedule can continue to receive tasks, avoiding resource idleness, while data from paused jobs is securely stored and transferred to prevent data loss. Dynamically adjusting the allocation strategy makes the system load more balanced, significantly improving batch processing throughput and resource utilization, shortening the overall sub-job processing time, and thus increasing the processing speed of sub-jobs.

[0096] Optional, Figure 6 A flowchart illustrating a batch sub-job processing method provided in this application embodiment. Figure 5 ,like Figure 6 As shown, this embodiment, based on the above embodiments, details the specific process of reallocating the data range of each sub-job, including:

[0097] S601. Determine the processing speed of each sub-job based on its completion status and current resource usage.

[0098] The processing speed of each subjob is determined based on its completion status and current resource usage. The system collects the amount of data processed and the time taken by each subjob in real time, and calculates the amount of data processed per unit time as the processing speed by combining this data with the current CPU and memory resource usage.

[0099] Optionally, the system uses a sliding window to count the number of processed single records within a recent period, and divides this number by the processing time to obtain the real-time processing speed. Simultaneously, it monitors current resource usage. If excessive resource usage leads to a decrease in speed, the speed value needs to be corrected to ensure accurate assessment and provide a reliable basis for subsequent adjustments.

[0100] S602. Based on the processing speed of each sub-job, determine whether it is necessary to reallocate the data range of each sub-job.

[0101] Based on the processing speed of each subjob, determine whether the data range of each subjob needs to be reallocated. Compare the calculated processing speed with a preset speed threshold; if the speed is consistently higher or lower than the threshold, it is determined that reallocation is necessary.

[0102] Specifically, the preset values ​​are set with upper and lower limits based on the longest and average historical processing times. If the processing speed exceeds the upper limit, it indicates that the capacity is sufficient; if it falls below the lower limit, it indicates that the load is too heavy. The system periodically polls the speed values ​​of all active sub-jobs, and only triggers the reallocation process when the deviation exceeds a certain percentage and persists for a certain period of time, thus avoiding frequent adjustments.

[0103] S603. When the processing speed of a subjob is higher than the preset value, increase the data range allocation of the subjob.

[0104] When the processing speed of a subjob exceeds a preset value, the data range allocated to the subjob is increased. Data is retrieved from the stored unallocated data range, and the amount of data to be added is calculated based on the ratio of the current processing speed to the average speed, and then merged into the data range of the atomic job.

[0105] S604. When the processing speed of a subjob is lower than the preset value, reduce the data range allocation of the subjob.

[0106] When the processing speed of a subjob is lower than a preset value, the data range allocated to the subjob is reduced. The unprocessed portion of the atomic job's data range is separated and stored back into the storage area as a new unallocated data range to be processed, and the data range boundary of the subjob is reduced.

[0107] This embodiment provides a batch sub-job processing method that calculates processing speed in real time and dynamically adjusts data range to ensure load distribution matches actual capacity. High-speed jobs receive more data to avoid idle time, while low-speed jobs are less burdened to prevent congestion. This mechanism eliminates the imbalance of static allocation, improves the overall system throughput, and ensures that batch processing jobs can still run efficiently and stably under resource fluctuations.

[0108] For ease of understanding, Figure 7 This is a schematic diagram of the structure of a batch sub-job processing system provided in an embodiment of this application, as shown below. Figure 7 As shown, the database stores historical processing data and pending data for batch subjobs; the batch job console starts the job process, sends pre-allocation instructions, and monitors the completion status and resource usage of subjobs; the job allocator determines the longest and average time consumption per record based on historical data, calculates the initial allocation amount, divides the data range, allocates data, and re-allocates data based on status changes; the buffer stores the unallocated pending data range and job allocation information; batch job executors 1 to N execute subjobs, process the allocated data range, and provide feedback on the completion status.

[0109] Figure 8 This is a schematic diagram of a batch sub-job processing apparatus provided in an embodiment of this application. The apparatus in this embodiment can be in the form of software and / or hardware. Figure 8 As shown in the embodiment of this application, a batch sub-job processing device 800 is provided. The device includes: an acquisition module 801, a first processing module 802, a second processing module 803, a division module 804, a third processing module 805, a monitoring module 806, and a fourth processing module 807.

[0110] The acquisition module 801 is used to acquire historical processing data and pending data of batch sub-jobs;

[0111] The first processing module 802 is used to determine the longest and average processing time for a single record processing path based on historical processing data.

[0112] The second processing module 803 is used to determine the initial allocation of each sub-job in the batch sub-job based on the longest time and the average time.

[0113] The partitioning module 804 is used to divide the data to be processed into multiple data ranges;

[0114] The third processing module 805 is used to allocate each data range to multiple sub-jobs according to the initial allocation amount, and to store the unallocated data ranges to be processed.

[0115] The monitoring module 806 is used to monitor the execution process of each sub-job in order to determine the completion status and current resource usage of each sub-job.

[0116] The fourth processing module 807 is used to reclaim or retrieve the stored unallocated data range to be processed based on the completion status and current resource usage of each sub-job, and to reallocate the data range of each sub-job.

[0117] In one possible implementation, the first processing module 802 is further configured to:

[0118] Based on preset filtering rules, historical processing data is filtered to obtain filtered historical processing data. The filtered historical processing data includes at least the path time data for each path in a single record processing process.

[0119] In one possible implementation, the second processing module 803 is further configured to:

[0120] Get the number of currently available subjobs;

[0121] The initial allocation of each sub-job is determined by a preset algorithm based on the number of jobs, the longest duration, and the average duration.

[0122] In one possible implementation, the partitioning module 804 is further configured to:

[0123] Based on the number of currently available sub-jobs, the data to be processed is divided into N data ranges, where the value of N matches the number of currently available sub-jobs; the initial data volume of each data range is determined based on the initial allocation, and the total amount of each data range does not exceed the total amount of data to be processed.

[0124] In one possible implementation, the fourth processing module 807 is further configured to:

[0125] Based on the completion status of each sub-task, determine the changes in the number of sub-tasks;

[0126] When the number of sub-jobs decreases, the unallocated unprocessed data range corresponding to the reduced sub-jobs is reclaimed and stored;

[0127] When the number of subjobs increases, based on the current resource usage, the stored unallocated unprocessed data range is retrieved and allocated to the data range of the newly added subjob.

[0128] In one possible implementation, the fourth processing module 807 is further configured to:

[0129] Determine the completion progress of each sub-task based on its completion status;

[0130] When any subjob completes the allocated data range ahead of schedule, receive the completion request sent by the subjob that has completed the allocation ahead of schedule;

[0131] Based on the completion request, determine whether there is currently a stored range of unallocated pending data;

[0132] If it exists, retrieve the stored unallocated data range to be processed, and allocate the stored unallocated data range to the data range of the sub-job that has been allocated in advance.

[0133] In one possible implementation, the fourth processing module 807 is further configured to:

[0134] Determine the completion progress of each sub-task based on its completion status;

[0135] When any subjob is paused, retrieve and store the range of unprocessed data corresponding to the paused subjob;

[0136] Based on the current resource usage, retrieve the unprocessed data range and allocate the processed data range to the data range of other sub-jobs that are not in a paused state.

[0137] In one possible implementation, the fourth processing module 807 is further configured to:

[0138] The processing speed of each sub-job is determined based on its completion status and current resource usage.

[0139] Based on the processing speed of each sub-job, determine whether it is necessary to reallocate the data range of each sub-job;

[0140] When the processing speed of a subjob is higher than the preset value, increase the data range allocated to the subjob.

[0141] When the processing speed of a subjob is lower than the preset value, reduce the data range allocated to the subjob.

[0142] This embodiment provides a batch sub-job processing device that can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0143] Figure 9 This is a schematic diagram of the structure of an electronic device provided in this application. Figure 9 As shown, the electronic device 900 provided in this embodiment includes at least one processor 901 and a memory 902. Optionally, the device 900 further includes a communication component 903. The processor 901, memory 902, and communication component 903 are connected via a bus.

[0144] In a specific implementation, at least one processor 901 executes computer execution instructions stored in memory 902, causing at least one processor 901 to perform the above-described method.

[0145] The specific implementation process of processor 901 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0146] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0147] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0148] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0149] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0150] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0151] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0152] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0153] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0156] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0157] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0158] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for processing batch sub-jobs, characterized in that, include: Retrieve historical processing data and pending data for batch subjobs; Based on the historical processing data, determine the longest and average processing time for a single record processing path; Based on the longest time consumption and the average time consumption, determine the initial allocation amount for each sub-job in the batch sub-jobs; The data to be processed is divided into multiple data ranges; Based on the initial allocation amount, each data range is allocated to multiple sub-jobs, and unallocated data ranges to be processed are stored. Monitor the execution process of each sub-job to determine the completion status and current resource usage of each sub-job; Based on the completion status of each sub-job and the current resource usage, the stored unallocated unprocessed data range is reclaimed or retrieved, and the data range of each sub-job is reallocated.

2. The method according to claim 1, characterized in that, Before determining the longest and average processing time for a single record processing path based on the historical processing data, the method further includes: The historical processing data is filtered according to preset filtering rules to obtain filtered historical processing data, wherein the filtered historical processing data includes at least the path time data of each path in a single record processing process.

3. The method according to claim 2, characterized in that, The step of determining the initial allocation of each sub-job in the batch sub-jobs based on the longest time consumption and the average time consumption includes: Get the number of currently available subjobs; Based on the number of jobs, the longest time taken, and the average time taken, the initial allocation of each sub-job is determined by a preset algorithm.

4. The method according to claim 3, characterized in that, The step of dividing the data to be processed into multiple data ranges includes: Based on the number of currently available sub-jobs, the data to be processed is divided into N data ranges, wherein the value of N matches the number of currently available sub-jobs; the initial data volume of each data range is determined based on the initial allocation amount, and the total amount of each data range does not exceed the total data volume of the data to be processed.

5. The method according to claim 4, characterized in that, The step of reclaiming or retrieving the unallocated range of data to be processed based on the completion status of each sub-job and the current resource usage includes: Based on the completion status of each sub-job, determine the change in the number of sub-jobs; When the number of sub-jobs decreases, the unallocated unprocessed data range corresponding to the reduced sub-jobs is reclaimed and stored; When the number of sub-jobs increases, based on the current resource usage, the stored unallocated unprocessed data range is retrieved and allocated to the data range of the newly added sub-job.

6. The method according to claim 5, characterized in that, The step of reclaiming or retrieving the unallocated range of data to be processed based on the completion status of each sub-job and the current resource usage also includes: The completion progress of each sub-job is determined based on its completion status. When any of the sub-jobs completes the allocated data range ahead of schedule, a completion request is received from the sub-job that has completed the allocation ahead of schedule; Based on the completion request, determine whether there is currently a stored range of unallocated pending data; If it exists, retrieve the stored unallocated data range to be processed, and allocate the stored unallocated data range to the data range of the pre-allocated sub-job.

7. The method according to claim 6, characterized in that, The step of reclaiming or retrieving the unallocated range of data to be processed based on the completion status of each sub-job and the current resource usage also includes: The completion progress of each sub-job is determined based on its completion status. When any of the sub-jobs is in a paused state, the unprocessed data range corresponding to the paused sub-job is obtained and stored; Based on the current resource usage, retrieve the unprocessed data range and allocate the processed data range to the data range of other sub-jobs that have not experienced the paused state.

8. The method according to claim 7, characterized in that, The reallocation of the data range for each sub-job includes: The processing speed of each sub-job is determined based on its completion status and current resource usage. Based on the processing speed of each sub-job, determine whether it is necessary to reallocate the data range of each sub-job; When the processing speed of the sub-job is higher than a preset value, the data range allocation of the sub-job is increased; When the processing speed of the sub-job is lower than a preset value, the data range allocated to the sub-job is reduced.

9. A batch sub-job processing device, characterized in that, include: The acquisition module is used to acquire historical processing data and pending data for batch sub-jobs; The first processing module is used to determine the longest and average processing time for a single record processing path based on the historical processing data. The second processing module is used to determine the initial allocation amount of each sub-job in the batch sub-job based on the longest time consumption and the average time consumption; A partitioning module is used to divide the data to be processed into multiple data ranges; The third processing module is used to allocate each data range to multiple sub-jobs according to the initial allocation amount, and to store the unallocated data ranges to be processed. The monitoring module is used to monitor the execution process of each sub-job to determine the completion status and current resource usage of each sub-job. The fourth processing module is used to reclaim or retrieve the stored unallocated data range to be processed based on the completion status of each sub-job and the current resource usage, and to reallocate the data range of each sub-job.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images