Batch job processing method and apparatus, storage medium, and electronic device

Abstract

The application discloses a batch job processing method and device, a storage medium and an electronic equipment. It relates to the field of financial technology. The method comprises the following steps: obtaining a plurality of batch jobs to be processed; performing time consumption prediction on each batch job according to the code characteristics of each batch job, and obtaining time consumption data corresponding to each batch job; calculating the load data of each server according to the time consumption data corresponding to each batch job; determining the target server corresponding to each batch job according to the load data of each server, and distributing each batch job to the target server corresponding to each batch job. Through the application, the problem of low resource utilization rate of the server in processing batch jobs in the related art is solved.

Description

Technical Field

[0001] This application relates to the field of financial technology, and more specifically, to a batch processing method and apparatus, storage medium and electronic device. Background Technology

[0002] For distributed applications, load balancing capability is a crucial factor affecting server resource utilization. Load balancing algorithms in related technologies primarily allocate resources based on the current resource occupancy of each server when processing batch jobs. However, the resource usage of batch jobs during execution is highly uncertain, leading to low server resource utilization.

[0003] There is currently no effective solution to the problem of low server resource utilization when processing batch jobs in related technologies. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, storage medium, and electronic device for processing batch jobs, in order to solve the problem of low server resource utilization in the related art when processing batch jobs.

[0005] To achieve the above objectives, according to one aspect of this application, a method for processing batch jobs is provided. The method includes: acquiring multiple batch jobs to be processed; predicting the time consumption of each batch job based on its code characteristics to obtain time consumption data corresponding to each batch job; calculating the load data of each server based on the time consumption data of each batch job; determining the target server corresponding to each batch job based on the load data of each server; and assigning each batch job to the target server corresponding to each batch job.

[0006] Furthermore, based on the code characteristics of each batch job, the time consumption prediction for each batch job is performed to obtain the time consumption data corresponding to each batch job. This includes: comparing the number of runs of each batch job with a preset threshold; if the number of runs of the batch job is greater than or equal to the preset threshold, the time consumption prediction is performed based on the historical run data of the batch job to obtain the time consumption data corresponding to the batch job; if the number of runs of the batch job is less than the preset threshold, a target sample set corresponding to the batch job is determined, and the time consumption prediction is performed based on the target sample set to obtain the time consumption data corresponding to the batch job. The target sample set consists of multiple target historical batch job samples, which are determined based on the similarity between historical batch job samples and batch jobs.

[0007] Further, determining the target sample set corresponding to the batch job includes: calculating the similarity between the batch job and multiple historical batch job samples to obtain the calculation results; sorting the multiple historical batch job samples according to the calculation results to obtain the sorting results; and determining the target sample set from the multiple historical batch job samples according to the sorting results.

[0008] Furthermore, similarity calculations are performed on the batch job and multiple historical batch job samples. The calculation results include: integrating the code content of each historical batch job sample and the code content of the batch job to obtain integrated code content of each historical batch job sample and the integrated code content of the batch job; performing structural formatting on the integrated code content of each historical batch job sample and the integrated code content of the batch job to obtain formatted code content of each historical batch job sample and the formatted code content of the batch job; performing parameter replacement on the formatted code content of each historical batch job sample and the formatted code content of the batch job to obtain processed code content of each historical batch job sample and the processed code content of the batch job; performing text comparison on the processed code content of each historical batch job sample and the processed code content of the batch job to obtain comparison results, and determining the calculation results based on the comparison results.

[0009] Furthermore, based on the comparison results, the calculation results are determined as follows: the number of lines of code that are identical to the code content of each processed historical batch job sample in the statistical comparison results; and the calculation results are obtained based on the number of identical lines of code and the number of lines of code in the code content of each processed historical batch job sample.

[0010] Furthermore, based on the target sample set, the time consumption prediction is performed to obtain the time consumption data corresponding to the batch job, including: obtaining the first time consumption data of each target historical batch job sample; calculating the single-line predicted time consumption data corresponding to the batch job based on the first time consumption data of each target historical batch job sample, the similarity corresponding to each target historical batch job sample, and the number of samples in the target sample set, wherein the first time consumption data is used to characterize the average time consumption of a single line of data; calculating the number of lines of code for the batch job; calculating the time consumption data corresponding to the batch job based on the single-line predicted time consumption data and the number of lines of code for the batch job.

[0011] Furthermore, based on the load data of each server, the target server for each batch job is determined by comparing the load data of each server and selecting the server with the smallest load data as the target server.

[0012] To achieve the above objectives, according to another aspect of this application, a batch job processing apparatus is provided. The apparatus includes: an acquisition unit, configured to acquire multiple batch jobs to be processed, predict the time consumption of each batch job based on the code characteristics of each batch job, and obtain time consumption data corresponding to each batch job; a first processing unit, configured to calculate the load data of each server based on the time consumption data corresponding to each batch job; and a second processing unit, configured to determine the target server corresponding to each batch job based on the load data of each server, and allocate each batch job to the target server corresponding to each batch job.

[0013] Furthermore, the acquisition unit includes: a first processing subunit, used to compare the number of runs of each batch job with a preset threshold; a second processing subunit, used to predict the time consumption based on the historical running data of the batch job when the number of runs of the batch job is greater than or equal to the preset threshold, to obtain the time consumption data corresponding to the batch job; and a third processing subunit, used to determine the target sample set corresponding to the batch job when the number of runs of the batch job is less than the preset threshold, and to predict the time consumption based on the target sample set, to obtain the time consumption data corresponding to the batch job, wherein the target sample set consists of multiple target historical batch job samples, and the target historical batch job samples are determined based on the similarity between the historical batch job samples and the batch job.

[0014] Furthermore, the third processing subunit includes: a first processing module for calculating the similarity between the batch job and multiple historical batch job samples to obtain the calculation result; a second processing module for sorting the multiple historical batch job samples according to the calculation result to obtain the sorting result; and a third processing module for determining the target sample set from the multiple historical batch job samples according to the sorting result.

[0015] Further, the first processing module includes: a first processing submodule, used to integrate the code content of each historical batch job sample and the code content of the batch job respectively, to obtain the integrated code content of each historical batch job sample and the integrated code content of the batch job; a second processing submodule, used to perform structure formatting on the integrated code content of each historical batch job sample and the integrated code content of the batch job respectively, to obtain the formatted code content of each historical batch job sample and the formatted code content of the batch job respectively; a third processing submodule, used to perform parameter replacement on the formatted code content of each historical batch job sample and the formatted code content of the batch job respectively, to obtain the processed code content of each historical batch job sample and the processed code content of the batch job respectively; and a fourth processing submodule, used to perform text comparison on the processed code content of each historical batch job sample and the processed code content of the batch job, to obtain the comparison result, and to determine the calculation result based on the comparison result.

[0016] Furthermore, the fourth processing submodule includes: a fifth processing submodule, used to count the number of lines of code in the code content of each processed historical batch job sample that are identical to the code content of the processed batch job in the comparison results; and a sixth processing submodule, used to calculate the result based on the number of identical lines of code and the number of lines of code in the code content of each processed historical batch job sample.

[0017] Furthermore, the third processing subunit also includes: an acquisition subunit, used to acquire the first time consumption data of each target historical batch job sample, and calculate the single-line predicted time consumption data corresponding to the batch job based on the first time consumption data of each target historical batch job sample, the similarity corresponding to each target historical batch job sample, and the number of samples in the target sample set, wherein the first time consumption data is used to characterize the average time consumption of a single line of data; and a calculation subunit, used to calculate the number of lines of code for the batch job, and calculate the time consumption data corresponding to the batch job based on the single-line predicted time consumption data and the number of lines of code for the batch job.

[0018] Furthermore, the second processing unit includes a determination subunit, used to compare the load data of each server and select the server with the smallest load data as the target server.

[0019] According to another aspect of the present invention, an electronic device is also provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the program, when running, executes a batch job processing method of any of the above-mentioned methods.

[0020] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a processing method for batch operations of any of the above-mentioned methods is provided when the program is running, which controls the device where the storage medium is located to perform the batch operations.

[0021] In this embodiment, the following steps are employed: acquiring multiple batch jobs to be processed; predicting the time consumption of each batch job based on its code characteristics to obtain the time consumption data corresponding to each batch job; calculating the load data of each server based on the time consumption data of each batch job; determining the target server corresponding to each batch job based on the load data of each server; and allocating each batch job to the target server corresponding to each batch job. This solves the technical problem of low server resource utilization when processing batch jobs in related technologies. In this solution, by analyzing the code characteristics of batch jobs to predict time consumption, dynamically calculating server load, and optimizing job allocation, the job allocation process becomes more accurate, effectively improving server resource utilization. Attached Figure Description

[0022] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0023] Figure 1 A hardware block diagram of a computer terminal for implementing a batch job processing method is shown.

[0024] Figure 2 This is a flowchart of a batch job processing method provided according to an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of a batch processing apparatus according to an embodiment of this application;

[0026] Figure 4 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding access points are provided for users to choose to authorize or refuse. For example, interfaces are set up between this system and relevant users or organizations, providing users with corresponding access points to choose to agree to or refuse automated decision-making results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.

[0030] Example 1

[0031] According to an embodiment of this application, a method embodiment for processing batch jobs is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a batch job processing method is shown. Figure 1As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0033] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0034] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the batch job processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the batch job processing method described above. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0035] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0036] The display may be a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0037] Under the aforementioned operating environment, this application provides the following: Figure 2 The batch job processing method is shown. Figure 2 This is a flowchart of a batch job processing method according to Embodiment 1 of this application. The batch job processing method includes:

[0038] Step S201: Obtain multiple batch jobs to be processed, predict the time consumption of each batch job based on the code characteristics of each batch job, and obtain the time consumption data corresponding to each batch job.

[0039] Step S202: Calculate the load data of each server based on the time consumption data corresponding to each batch job;

[0040] Step S203: Based on the load data of each server, determine the target server corresponding to each batch job, and assign each batch job to the target server corresponding to each batch job.

[0041] Optionally, when allocating batch jobs, the job allocation server can allocate them to appropriate servers based on the estimated execution time and the standard deviation of the estimated execution time. First, multiple batch jobs to be processed are obtained. Then, the execution time of each batch job is predicted based on its code characteristics, resulting in the execution time data for each batch job. For example, for batch jobs with a high number of runs, the execution time can be predicted based on historical execution data; for batch jobs with a low number of runs, the execution time can be predicted based on the execution data of similar samples.

[0042] Optionally, after obtaining the time consumption data corresponding to each batch job, the load data of each server can be calculated based on the time consumption data of each batch job. For example, firstly, the estimated total time consumption of jobs already assigned to the server and not yet completed is divided by the number of threads configured on the server to obtain the remaining processing time for each thread on the server, denoted as the first time. Then, the time since the last job assigned to the server is calculated, denoted as the second time. Then, the first time is subtracted from the second time to obtain the server's immediate available processing time, which is the time when each thread on the server can theoretically start processing a new job immediately. Then, the server's immediate available processing time is summed with the time consumption data corresponding to the batch jobs to obtain the server's load data.

[0043] Optionally, in the batch job processing method provided in the embodiments of this application, determining the target server corresponding to each batch job based on the load data of each server includes: comparing the load data of each server and selecting the server with the smallest load data as the target server.

[0044] In an optional embodiment, the load data of each server is compared, and the server with the smallest load data is selected as the target server, i.e., it is preferentially assigned to the server with the smallest current load.

[0045] In summary, by analyzing the code characteristics of batch jobs to predict processing time, dynamically calculating server load, and optimizing job allocation, the job allocation process becomes more accurate, effectively improving server resource utilization. This solves the technical problem of low server resource utilization when processing batch jobs in related technologies.

[0046] Optionally, in the batch job processing method provided in this application embodiment, predicting the time consumption of each batch job based on the code characteristics of each batch job to obtain the time consumption data corresponding to each batch job includes: comparing the number of runs of each batch job with a preset threshold; if the number of runs of the batch job is greater than or equal to the preset threshold, predicting the time consumption based on the historical run data of the batch job to obtain the time consumption data corresponding to the batch job; if the number of runs of the batch job is less than the preset threshold, determining the target sample set corresponding to the batch job, and predicting the time consumption based on the target sample set to obtain the time consumption data corresponding to the batch job, wherein the target sample set consists of multiple target historical batch job samples, and the target historical batch job samples are determined based on the similarity between the historical batch job samples and the batch job.

[0047] In an optional embodiment, the number of runs for each batch job is compared with a preset run count threshold (i.e., the preset threshold). For batch jobs with a large number of runs, i.e., when the number of runs for a batch job is greater than or equal to the preset run count threshold, the time consumption can be predicted based on the historical run data of the batch job to obtain the time consumption data corresponding to the batch job. For batch jobs with a small number of runs, i.e., when the number of runs for a batch job is less than the preset run count threshold, it is necessary to determine the target sample set corresponding to the batch job and predict the time consumption based on the target sample set to obtain the time consumption data corresponding to the batch job.

[0048] Optionally, statistical analysis is performed on previously executed existing jobs to filter out batch jobs whose execution frequency and number exceed a certain threshold (e.g., jobs that have been running in the production environment for more than 6 months and are executed more than 30 times per month). The code file of each existing job is used as a sample to calculate the average time (Xcf) and standard deviation (Scf) for processing a single business file, and the average time (Xcr) and standard deviation (Scr) for processing a single line of data. For example, historical execution data of existing jobs can be obtained, including job ID, start time, end time, file size, and total number of lines processed. The execution time for each run is calculated based on the start and end times. The collected execution times for each run are summed and then divided by the number of runs to obtain the average time (Xcf) for processing a single business file. Then, the difference between each execution time and the average time (Xcf) is calculated, the squares of these differences are calculated, and the average of all squared differences is calculated as the variance. The square root of the variance is the standard deviation (Scf). For example, divide the time taken for each run by the total number of rows to get the time taken for a single row of data. Add up all the estimated times taken for a single row of data and divide by the number of runs to get the average time taken for a single row of data, Xcr. Then calculate the difference between the time taken for each single row of data and the average time taken, Xcr. Then calculate the square mean of these differences and finally take the square root to get the standard deviation, Scr.

[0049] Optionally, the code of existing jobs is parsed, recording the list and order of methods called in each branch to obtain sample code blocks for subsequent similarity comparison. During code parsing, method calls are traced until a non-self-developed code level is reached. Specifically, when parsing the code of a batch job, if a method encountered is self-developed (characterized by the package header of the method's class containing the company name abbreviation), the parsing continues deeper until a method provided by an external vendor, open-source library, or other third party is reached.

[0050] By intelligently differentiating the operational maturity of batch jobs and combining historical data with code similarity analysis, accurate time consumption prediction and optimized resource allocation are achieved, effectively improving system processing efficiency and server resource utilization.

[0051] Optionally, in the batch job processing method provided in the embodiments of this application, determining the target sample set corresponding to the batch job includes: performing similarity calculation on the batch job and multiple historical batch job samples to obtain the calculation result; sorting the multiple historical batch job samples according to the calculation result to obtain the sorting result; and determining the target sample set from the multiple historical batch job samples according to the sorting result.

[0052] In an optional embodiment, during the process of determining the target sample set corresponding to the batch job, the similarity between the batch job and multiple historical batch job samples is calculated to obtain the calculation result, that is, the similarity value between the batch job and each historical batch job sample. Then, based on the calculation result, the multiple historical batch job samples are sorted to obtain the sorting result. Then, based on the sorting result, the target sample set is determined from the multiple historical batch job samples. For example, the top n samples with the highest similarity are selected to form the target sample set.

[0053] By calculating and sorting the similarity between batch jobs and historical samples, a target sample set is selected for time prediction, which can effectively improve the accuracy of time prediction for new jobs, optimize job scheduling, and ensure efficient use of resources.

[0054] Optionally, in the batch job processing method provided in this application embodiment, the similarity calculation of the batch job with multiple historical batch job samples to obtain the calculation result includes: integrating the code content of each historical batch job sample and the code content of the batch job to obtain the integrated code content of each historical batch job sample and the integrated code content of the batch job; performing structural formatting on the integrated code content of each historical batch job sample and the integrated code content of the batch job to obtain the formatted code content of each historical batch job sample and the formatted code content of the batch job; performing parameter replacement on the formatted code content of each historical batch job sample and the formatted code content of the batch job to obtain the processed code content of each historical batch job sample and the processed code content of the batch job; performing text comparison on the processed code content of each historical batch job sample and the processed code content of the batch job to obtain the comparison result, and determining the calculation result based on the comparison result.

[0055] Optionally, in the batch job processing method provided in this application embodiment, determining the calculation result based on the comparison result includes: counting the number of lines of code that are the same as the code content of each processed historical batch job sample in the comparison result; and calculating the result based on the number of lines of code that are the same and the number of lines of code in the code content of each processed historical batch job sample.

[0056] In an optional embodiment, when a new job is put into production, there is a lack of sufficient actual runs to statistically analyze the time consumption. It is necessary to calculate and find highly similar samples to predict the time consumption based on similarity. First, the code content of each historical batch job sample and the code content of the batch job are integrated to obtain the integrated code content of each historical batch job sample and the integrated code content of the batch job. For example, the code content of the new batch job is code A, and the code content of the historical batch job sample is code B. The code blocks A and B are integrated separately, that is, according to the code syntax, all methods within the code are integrated into one method. For example, code block A has three methods a, b, and c, where a is the entry point, a calls b, and b calls c. First, the line of code in b that calls c is completely replaced with the code in method c. Then, the line of code in a that calls b is completely replaced with the code in method b, resulting in the integrated code content.

[0057] In an optional embodiment, the code content of each integrated historical batch job sample and the code content of the integrated batch job are respectively subjected to structure formatting to obtain the formatted code content of each historical batch job sample and the formatted code content of the batch job. For example, the code blocks A and B are structure formatted to ensure that each line of code has only one method or conditional statement, so that the format of both sides is consistent and easy to compare. For example, "String dateStr=(new Date()).toString()" is changed to two lines: the first line is "Date date=new Date()", and the second line is "String dateStr = date.toString()".

[0058] In an optional embodiment, parameter replacement processing is performed on the code content of each formatted historical batch job sample and the code content of the formatted batch job, respectively, to obtain the processed code content of each historical batch job sample and the processed batch job. For example, all parameters in the code blocks A and B are abstracted, that is, the parameters are replaced with a specific string.

[0059] In an optional embodiment, redundant code (such as constant declarations and other lines of code that do not contain actual execution methods) is removed from A and B. Then, a text comparison is performed between the code content of each processed historical batch job sample and the code content of the processed batch job to determine whether each line of text is the same, thus obtaining the comparison result. The number of lines of code that are identical between the code content of each processed historical batch job sample and the code content of the processed batch job is then counted. A similarity value is calculated based on this number of identical lines and the total number of lines in the code content of each processed historical batch job sample. For example, the similarity value is obtained by dividing the number of identical lines in A and B by the total number of lines in A.

[0060] By integrating, formatting, and standardizing the code of batch jobs and historical samples, and then conducting detailed text comparison analysis, the similarity between jobs was effectively quantified, providing a basis for time prediction and resource optimization allocation.

[0061] Optionally, in the batch job processing method provided in this application embodiment, the time consumption prediction based on the target sample set to obtain the time consumption data corresponding to the batch job includes: obtaining the first time consumption data of each target historical batch job sample, calculating the single-line predicted time consumption data corresponding to the batch job based on the first time consumption data of each target historical batch job sample, the similarity corresponding to each target historical batch job sample, and the number of samples in the target sample set, wherein the first time consumption data is used to characterize the average time consumption of a single line of data; calculating the number of lines of code for the batch job, calculating the time consumption data corresponding to the batch job based on the single-line predicted time consumption data and the number of lines of code for the batch job.

[0062] In an optional embodiment, the first time consumption data Xcri for each target historical batch job sample is first obtained, which is the average time consumption Xcr of a single row of data statistically analyzed from the i-th sample. Based on the first time consumption data Xcri for each target historical batch job sample, the similarity Si (similarity of the i-th sample) corresponding to each target historical batch job sample, and the number of samples n in the target sample set, the single row predicted time consumption data Xcy corresponding to the batch job is calculated, as follows:

[0063]

[0064] In an optional embodiment, the number of lines of code Cl for the batch job is calculated based on the predicted single-line time consumption data Xcy corresponding to the batch job and the number of lines of code Cl for the batch job, to obtain the total time consumption data Xcy corresponding to the batch job. For example, when using streaming to read a file, the total file length is divided by the length of the second line of the file to obtain the estimated length Cl (i.e., the number of lines of code), and the time consumption data corresponding to the batch job is... The formula is as follows:

[0065]

[0066] in, It is the value with the lowest similarity among n samples. This is the timeout period for a single task.

[0067] The batch job processing method provided in this application includes the following steps: acquiring multiple batch jobs to be processed; predicting the time consumption of each batch job based on its code characteristics to obtain the time consumption data corresponding to each batch job; calculating the load data of each server based on the time consumption data of each batch job; determining the target server corresponding to each batch job based on the load data of each server; and allocating each batch job to the target server corresponding to each batch job. This solves the technical problem of low server resource utilization when processing batch jobs in related technologies. In this solution, by analyzing the code characteristics of batch jobs to predict time consumption, dynamically calculating server load, and optimizing job allocation, the job allocation process becomes more accurate, effectively improving server resource utilization.

[0068] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0069] Example 2

[0070] This application also provides a batch job processing apparatus. It should be noted that the batch job processing apparatus of this application can be used to execute the batch job processing method provided in this application. The batch job processing apparatus provided in this application will be described below.

[0071] According to an embodiment of this application, a batch processing apparatus for implementing the above-described batch processing method is also provided, such as... Figure 3 As shown, the device includes: an acquisition unit 301, a first processing unit 302, and a second processing unit 303.

[0072] The acquisition unit 301 is used to acquire multiple batch jobs to be processed, predict the time consumption of each batch job based on the code characteristics of each batch job, and obtain the time consumption data corresponding to each batch job.

[0073] The first processing unit 302 is used to calculate the load data of each server based on the time consumption data corresponding to each batch job;

[0074] The second processing unit 303 is used to determine the target server corresponding to each batch job based on the load data of each server, and to allocate each batch job to the target server corresponding to each batch job.

[0075] The batch job processing apparatus provided in this application embodiment acquires multiple batch jobs to be processed through an acquisition unit 301, performs time prediction on each batch job based on the code characteristics of each batch job, and obtains the time data corresponding to each batch job; a first processing unit 302 calculates the load data of each server based on the time data corresponding to each batch job; and a second processing unit 303 determines the target server corresponding to each batch job based on the load data of each server, and assigns each batch job to the target server corresponding to each batch job.

[0076] Optionally, in the batch job processing apparatus provided in this application embodiment, the acquisition unit includes: a first processing subunit, used to compare the number of runs of each batch job with a preset threshold; a second processing subunit, used to predict the time consumption based on the historical running data of the batch job when the number of runs of the batch job is greater than or equal to the preset threshold, and obtain the time consumption data corresponding to the batch job; and a third processing subunit, used to determine the target sample set corresponding to the batch job when the number of runs of the batch job is less than the preset threshold, and to predict the time consumption based on the target sample set, and obtain the time consumption data corresponding to the batch job, wherein the target sample set consists of multiple target historical batch job samples, and the target historical batch job samples are determined based on the similarity between the historical batch job samples and the batch job.

[0077] Optionally, in the batch job processing apparatus provided in the embodiments of this application, the third processing subunit includes: a first processing module, used to perform similarity calculation on the batch job and multiple historical batch job samples to obtain a calculation result; a second processing module, used to sort the multiple historical batch job samples according to the calculation result to obtain a sorting result; and a third processing module, used to determine a target sample set from the multiple historical batch job samples according to the sorting result.

[0078] Optionally, in the batch job processing apparatus provided in this application embodiment, the first processing module includes: a first processing submodule, used to integrate the code content of each historical batch job sample and the code content of the batch job respectively, to obtain the integrated code content of each historical batch job sample and the integrated code content of the batch job; a second processing submodule, used to perform structure formatting processing on the integrated code content of each historical batch job sample and the integrated code content of the batch job respectively, to obtain the formatted code content of each historical batch job sample and the formatted code content of the batch job respectively; a third processing submodule, used to perform parameter replacement processing on the formatted code content of each historical batch job sample and the formatted code content of the batch job respectively, to obtain the processed code content of each historical batch job sample and the processed code content of the batch job respectively; and a fourth processing submodule, used to perform text comparison on the processed code content of each historical batch job sample and the processed code content of the batch job, to obtain the comparison result, and to determine the calculation result based on the comparison result.

[0079] Optionally, in the batch job processing apparatus provided in this application embodiment, the fourth processing submodule includes: a fifth processing submodule, used to count the number of lines of code that are the same as the code content of each processed historical batch job sample in the comparison results; and a sixth processing submodule, used to calculate based on the same number of lines of code and the number of lines of code in the code content of each processed historical batch job sample to obtain a calculation result.

[0080] Optionally, in the batch job processing apparatus provided in this application embodiment, the third processing subunit further includes: an acquisition subunit, configured to acquire first time consumption data for each target historical batch job sample, and calculate based on the first time consumption data for each target historical batch job sample, the similarity corresponding to each target historical batch job sample, and the number of samples in the target sample set to obtain single-line predicted time consumption data corresponding to the batch job, wherein the first time consumption data is used to characterize the average time consumption of single-line data; and a calculation subunit, configured to calculate the number of lines of code for the batch job, and calculate based on the single-line predicted time consumption data corresponding to the batch job and the number of lines of code for the batch job to obtain the time consumption data corresponding to the batch job.

[0081] Optionally, in the batch processing apparatus provided in the embodiments of this application, the second processing unit includes: a determining subunit, used to compare the load data of each server and select the server with the smallest load data as the target server.

[0082] It should be noted that the acquisition unit 301, the first processing unit 302, and the second processing unit 303 mentioned above correspond to steps S201 to S203 in Embodiment 1. The three units and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above units can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0083] Example 3

[0084] Embodiments of this application may provide an electronic device. Figure 4 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 4 As shown, the electronic device may include: one or more ( Figure 4 (Only one is shown) Processor 402, memory 404, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0085] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0086] The processor can access information and applications stored in memory via a transmission device to perform the following steps: acquire multiple batch jobs to be processed; predict the time consumption of each batch job based on the code characteristics of each batch job to obtain the time consumption data corresponding to each batch job; calculate the load data of each server based on the time consumption data corresponding to each batch job; determine the target server corresponding to each batch job based on the load data of each server, and assign each batch job to the target server corresponding to each batch job.

[0087] The processor can access information and applications stored in memory via a transmission device to perform the following steps: comparing the number of runs for each batch job with a preset threshold; if the number of runs for a batch job is greater than or equal to the preset threshold, predicting the time consumption based on the historical running data of the batch job to obtain the time consumption data corresponding to the batch job; if the number of runs for a batch job is less than the preset threshold, determining the target sample set corresponding to the batch job, and predicting the time consumption based on the target sample set to obtain the time consumption data corresponding to the batch job, wherein the target sample set consists of multiple target historical batch job samples, and the target historical batch job samples are determined based on the similarity between the historical batch job samples and the batch job.

[0088] The processor can access the information and application programs stored in the memory via the transmission device to perform the following steps: calculate the similarity between the batch job and multiple historical batch job samples to obtain the calculation results; sort the multiple historical batch job samples according to the calculation results to obtain the sorting results; and determine the target sample set from the multiple historical batch job samples according to the sorting results.

[0089] The processor can access the information and application programs stored in the memory via a transmission device to perform the following steps: Integrate the code content of each historical batch job sample and the code content of the batch job, resulting in integrated code content for each historical batch job sample and the integrated code content of the batch job; perform structure formatting on the integrated code content of each historical batch job sample and the integrated code content of the batch job, resulting in formatted code content for each historical batch job sample and the formatted code content of the batch job; perform parameter replacement on the formatted code content of each historical batch job sample and the formatted code content of the batch job, resulting in processed code content for each historical batch job sample and the processed code content of the batch job; perform text comparison on the processed code content of each historical batch job sample and the processed code content of the batch job, obtain the comparison result, and determine the calculation result based on the comparison result.

[0090] The processor can access the information and application programs stored in the memory via the transmission device to perform the following steps: 1) Count the number of lines of code in the code content of each processed historical batch job sample that are identical to the code content of the processed batch job in the comparison results; 2) Calculate the result based on the number of identical lines of code and the number of lines of code in the code content of each processed historical batch job sample.

[0091] The processor can access the information and application stored in the memory via the transmission device to perform the following steps: obtain the first time consumption data for each target historical batch job sample; calculate the single-line predicted time consumption data corresponding to the batch job based on the first time consumption data for each target historical batch job sample, the similarity corresponding to each target historical batch job sample, and the number of samples in the target sample set; wherein the first time consumption data is used to characterize the average time consumption of a single line of data; calculate the number of lines of code for the batch job; calculate the time consumption data corresponding to the batch job based on the single-line predicted time consumption data and the number of lines of code for the batch job.

[0092] The processor can access information and applications stored in memory via a transmission device to perform the following steps: compare the load data of each server and select the server with the smallest load data as the target server.

[0093] Those skilled in the art will understand that Figure 4 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 4 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 4 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 4 The different configurations shown.

[0094] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0095] Example 4

[0096] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the batch job processing method provided in Embodiment 1.

[0097] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0098] This application also provides a computer program product, which, when executed on a data processing device, is suitable for performing processing method steps for batch jobs.

[0099] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0100] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0101] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0102] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0103] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0104] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0105] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for processing batch jobs, characterized in that, include: Multiple batch jobs to be processed are obtained, and the time consumption of each batch job is predicted based on the code characteristics of each batch job to obtain the time consumption data corresponding to each batch job. Based on the time consumption data corresponding to each batch job, calculate the load data of each server; Based on the load data of each server, the target server corresponding to each batch job is determined, and each batch job is assigned to the target server corresponding to each batch job.

2. The method according to claim 1, characterized in that, Based on the code characteristics of each batch job, the time consumption of each batch job is predicted, and the time consumption data corresponding to each batch job is obtained, including: The number of times each batch job is run is compared with a preset threshold; If the number of times the batch job is run is greater than or equal to the preset threshold, the time consumption is predicted based on the historical running data of the batch job to obtain the time consumption data corresponding to the batch job. If the number of times the batch job is run is less than the preset threshold, a target sample set corresponding to the batch job is determined, and the time consumption is predicted based on the target sample set to obtain the time consumption data corresponding to the batch job. The target sample set consists of multiple target historical batch job samples, and the target historical batch job samples are determined based on the similarity between the historical batch job samples and the batch job.

3. The method according to claim 2, characterized in that, Determining the target sample set corresponding to the batch job includes: The similarity between the batch job and multiple historical batch job samples is calculated to obtain the calculation results; Based on the calculation results, the multiple historical batch job samples are sorted to obtain the sorting results; Based on the sorting results, the target sample set is determined from the plurality of historical batch job samples.

4. The method according to claim 3, characterized in that, The similarity between the batch job and multiple historical batch job samples is calculated, and the calculation results include: The code content of each historical batch job sample and the code content of the batch job are integrated to obtain the integrated code content of each historical batch job sample and the integrated code content of the batch job. The code content of each integrated historical batch job sample and the code content of the integrated batch job are respectively subjected to structure formatting processing to obtain the formatted code content of each historical batch job sample and the formatted code content of the batch job. Parameter replacement processing is performed on the code content of each formatted historical batch job sample and the code content of the formatted batch job respectively to obtain the processed code content of each historical batch job sample and the processed code content of the batch job. The code content of each processed historical batch job sample and the code content of the processed batch job are compared by text to obtain the comparison result, and the calculation result is determined based on the comparison result.

5. The method according to claim 4, characterized in that, Based on the comparison results, the calculation results are determined to include: The number of lines of code in the comparison results that are identical to the code content of the processed batch job in each historical batch job sample is counted. The calculation result is obtained by calculating based on the same number of lines of code and the number of lines of code in the code content of each historical batch job sample after processing.

6. The method according to claim 2, characterized in that, Based on the target sample set, the time consumption prediction is performed to obtain the time consumption data corresponding to the batch job, including: The first time consumption data of each target historical batch job sample is obtained. Based on the first time consumption data of each target historical batch job sample, the similarity corresponding to each target historical batch job sample, and the number of samples in the target sample set, the single-line predicted time consumption data corresponding to the batch job is calculated. The first time consumption data is used to characterize the average time consumption of a single line of data. The number of lines of code for the batch job is calculated, and the time consumption data corresponding to the batch job is obtained by calculating the single-line predicted time consumption data and the number of lines of code for the batch job.

7. The method according to claim 1, characterized in that, Based on the load data of each server, the target server corresponding to each batch job is determined as follows: The load data of each server is compared, and the server with the smallest load data is selected as the target server.

8. A batch processing device, characterized in that, include: The acquisition unit is used to acquire multiple batch jobs to be processed, predict the time consumption of each batch job based on the code characteristics of each batch job, and obtain the time consumption data corresponding to each batch job. The first processing unit is used to calculate the load data of each server based on the time consumption data corresponding to each batch job; The second processing unit is used to determine the target server corresponding to each batch job based on the load data of each server, and to allocate each batch job to the target server corresponding to each batch job.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the batch job processing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the batch job processing method according to any one of claims 1 to 7.

Images