Batch task processing method, device, computer equipment and storage medium

Abstract

The present application relates to a batch task processing method, device, computer equipment, storage medium and computer program product. The method includes: obtaining multi-dimensional system load parameters; inputting the system load parameters into a preset system time-consuming regression model, converting the preset system time-consuming regression model into a batch task slicing expression; the preset system time-consuming regression model is trained based on system load sample data, task slicing sample data and system time-consuming sample data; solving the batch task slicing expression to obtain the number of task slicing; and slicing and parallel processing the batch tasks according to the number of task slicing. The entire scheme reasonably allocates batch tasks according to the multi-dimensional system load parameters and the preset system time-consuming regression model, slices the batch tasks according to the number of task slicing, and allocates threads corresponding to the number of slicing to process the sliced ​​batch tasks, thereby improving the efficiency of batch task processing.

Description

Technical Field

[0001] The present application relates to the technical field of resource scheduling, and in particular to a batch task processing method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of computer technology, all industries have entered the era of digital transformation, and the transaction volume of various business systems has increased day by day. In this context, how to improve the efficiency of batch tasks and the scheduling of batch tasks have become particularly important.

[0003] Some current methods mainly process batch tasks in a concurrent manner. The concurrent processing method is relatively fixed, and the same concurrent processing is performed on different batch tasks, which cannot adapt to different batch task processing scenarios.

[0004] Therefore, the current batch task processing method cannot perform reasonable task allocation, resulting in low efficiency of batch task processing. Summary of the invention

[0005] Based on this, it is necessary to provide a batch task processing method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve processing efficiency in response to the above technical problems.

[0006] In a first aspect, the present application provides a batch task processing method. The method comprises:

[0007] Obtain multi-dimensional system load parameters;

[0008] Input the system load parameters into a preset system time-consuming regression model, and convert the preset system time-consuming regression model into a batch task slicing expression; the preset system time-consuming regression model is trained based on system load sample data, task slicing sample data, and system time-consuming sample data;

[0009] Solve the batch task sharding expression to obtain the number of task shards;

[0010] Batch tasks are divided into shards and processed in parallel according to the number of task shards.

[0011] In one embodiment, solving the batch task slice expression to obtain the number of task slices includes: deriving the batch task slice expression to obtain a task slice derivative expression; and solving and determining the number of task slices corresponding to the minimum system time consumption according to the task slice derivative expression.

[0012] In one of the embodiments, the generation process of the preset system time consumption regression model includes: constructing an initial system time consumption regression model based on system resource parameters; the system resource parameters include system load parameters and task slice parameters; obtaining sample data, and fitting the initial system time consumption regression model based on the sample data and using the least squares method to obtain the preset system time consumption regression model, and the sample data includes system load sample data, task slice sample data and system time consumption sample data.

[0013] In one of the embodiments, sample data is obtained, and an initial system time regression model is fitted based on the sample data and using the least squares method to obtain a preset system time regression model, including: identifying the data type dimension included in the system resource parameters; determining the number of required sample data based on the data type dimension; obtaining sample data from historical data based on the number of required sample data; and fitting the initial system time regression model based on the sample data and the least squares method to obtain a preset system time regression model.

[0014] In one of the embodiments, the initial system time consumption regression model is fitted according to the sample data and the least squares method to obtain the preset system time consumption regression model, including: fitting the initial system time consumption regression model according to the sample data and the least squares method to obtain the linear relationship between the system time consumption and the system resource parameters; according to the linear relationship between the system time consumption and the system resource parameters, constructing the preset system time consumption regression model.

[0015] In one of the embodiments, the initial system time consumption regression model is fitted according to sample data and the least squares method to obtain the linear relationship between the system time consumption and the system resource parameters, including: constructing the objective function of the system resource parameters and the system time consumption in the initial system time consumption regression model according to the least squares method; solving the objective function according to the sample data to determine the coefficients of the system resource parameters that minimize the error of the objective function; and obtaining the linear relationship between the system time consumption and the system resource parameters according to the coefficients of the system resource parameters.

[0016] In a second aspect, the present application also provides a batch task processing device. The device comprises:

[0017] An acquisition module is used to obtain system load parameters in multiple dimensions;

[0018] An input module is used to input the system load parameters into a preset system time-consuming regression model, and convert the preset system time-consuming regression model into a batch task slicing expression; the preset system time-consuming regression model is trained based on system load sample data, task slicing sample data, and system time-consuming sample data;

[0019] The sharding module is used to solve the batch task sharding expression to obtain the number of task shards;

[0020] The processing module is used to perform parallel processing of batch tasks in slices according to the number of task slices.

[0021] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0022] Obtain multi-dimensional system load parameters;

[0023] Input the system load parameters into a preset system time-consuming regression model, and convert the preset system time-consuming regression model into a batch task slicing expression; the preset system time-consuming regression model is trained based on system load sample data, task slicing sample data, and system time-consuming sample data;

[0024] Solve the batch task sharding expression to obtain the number of task shards;

[0025] Batch tasks are divided into shards and processed in parallel according to the number of task shards.

[0026] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0027] Obtain multi-dimensional system load parameters;

[0028] Input the system load parameters into a preset system time-consuming regression model, and convert the preset system time-consuming regression model into a batch task slicing expression; the preset system time-consuming regression model is trained based on system load sample data, task slicing sample data, and system time-consuming sample data;

[0029] Solve the batch task sharding expression to obtain the number of task shards;

[0030] Batch tasks are divided into shards and processed in parallel according to the number of task shards.

[0031] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0032] Obtain multi-dimensional system load parameters;

[0033] Input the system load parameters into a preset system time-consuming regression model, and convert the preset system time-consuming regression model into a batch task slicing expression; the preset system time-consuming regression model is trained based on system load sample data, task slicing sample data, and system time-consuming sample data;

[0034] Solve the batch task sharding expression to obtain the number of task shards;

[0035] Batch tasks are divided into shards and processed in parallel according to the number of task shards.

[0036] The above-mentioned batch task processing method, device, computer equipment, storage medium and computer program product obtain multi-dimensional system load parameters; input the system load parameters into a preset system time-consuming regression model, and convert the preset system time-consuming regression model into a batch task slicing expression; the preset system time-consuming regression model is trained based on system load sample data, task slicing sample data and system time-consuming sample data; the batch task slicing expression is solved to obtain the number of task slicings; and the batch tasks are sliced ​​and processed in parallel according to the number of task slicings. The entire scheme reasonably allocates batch tasks according to multi-dimensional system load parameters and the preset system time-consuming regression model, slices the batch tasks according to the number of task slicings, and then allocates a corresponding number of threads according to the number of task slicings to process the sliced ​​batch tasks, thereby improving the efficiency of batch task processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 An application environment diagram of a batch task processing method in an embodiment;

[0038] Figure 2 A schematic diagram of a process flow of a batch task processing method in one embodiment;

[0039] Figure 3 A schematic flow chart of a step of determining a coefficient of a system resource parameter in one embodiment;

[0040] Figure 4 A schematic diagram of a flow chart of a batch task processing method in another embodiment;

[0041] Figure 5 is a structural block diagram of a batch task processing device in an embodiment;

[0042] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] Generally speaking, the factors that affect server performance are: CPU, memory, disk I / O bandwidth, and network I / O bandwidth. These four system resources are interdependent. Excessive load on any one system resource can cause bottlenecks in other system resources.

[0045] System resources are independent of each other, but also interdependent. System resources are positively correlated with system load capacity, but there is also an upper limit. If a certain system resource is configured more and other resources are configured less, as the system load increases, the system resource with less will become the bottleneck of the system load.

[0046] The regression algorithm uses a huge amount of observation data and mathematical statistics methods to try to establish a regression function relationship between the dependent variable and the independent variable. When there is only a dependent variable and one independent variable, it is called a univariate regression; when two or more independent variables are involved, it becomes a multivariate regression; in addition, according to whether the function expression between the independent variable and the dependent variable is linear or nonlinear, it is divided into linear regression and nonlinear regression.

[0047] The nonlinearity of the function expression between the independent variable and the dependent variable is reflected in the fact that the exponent of at least one variable is not 1 (power function, exponential function, logarithmic function, S function, etc.). It is worth noting that some nonlinear functions have a direct transformation relationship with linear functions. Therefore, some nonlinear regression can be converted into linear regression to solve nonlinear regression problems; some nonlinear regression cannot be converted into linear regression but can be converted into polynomial regression.

[0048] The time consumption of batch tasks is positively correlated with system resources. Polynomial regression can also be used to fit the relationship model between system resources and batch time consumption. Batch tasks are composed of different batch steps. In order to improve the execution efficiency of batch tasks, batch steps are divided into multiple slices for execution. One thread of the system executes one slice, thereby further improving the execution efficiency of batch tasks. The more slices there are, the higher the efficiency. However, the number of slices cannot be increased indefinitely and must be allocated according to the current remaining resources of the system.

[0049] Therefore, the present application fits the relationship model between system resources and the time consumed by the system to process batch tasks, and then calculates the number of shards according to the current system load, and then divides the batch tasks into shards according to the number of task shards, and then allocates a corresponding number of threads according to the number of task shards to process the sharded batch tasks, thereby improving the efficiency of batch task processing.

[0050] The batch task processing method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 sends a batch task processing request to the server, and the server 104 listens and responds to the batch task processing request sent by the terminal 102, and obtains multi-dimensional system load parameters; the system load parameters are input into the preset system time-consuming regression model, and the preset system time-consuming regression model is converted into a batch task slicing expression; the preset system time-consuming regression model is trained based on the system load sample data, the task slicing sample data and the system time-consuming sample data; the batch task slicing expression is solved to obtain the number of task slicing; and the batch tasks are sliced ​​and processed in parallel according to the number of task slicing. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0051] In one embodiment, Figure 2 As shown, a batch task processing method is provided, which is applied to Figure 1 Taking the server 104 in the example as an example, the following steps are included:

[0052] Step 202: Acquire multi-dimensional system load parameters.

[0053] The system load parameter is a parameter related to system resources, and the system load parameter represents the available resources of the system. Further, the system load parameter includes the current CPU usage, remaining available memory, remaining available disk IO bandwidth, remaining network IO bandwidth, etc.

[0054] Specifically, the server listens to and responds to the batch task processing request sent by the terminal, parses the batch task processing request, and obtains the batch task information. The server obtains the system load parameters at the current moment, and further obtains the CPU usage, remaining available memory, remaining available disk I / O bandwidth, and remaining network I / O bandwidth at the current moment.

[0055] Step 204 , input the system load parameter into a preset system time-consuming regression model, and convert the preset system time-consuming regression model into a batch task partition expression.

[0056] Among them, the preset system time consumption regression model is a polynomial fitting function of the relationship between system time consumption and system resource parameters, and the system resource parameters include system load parameters and task slicing parameters. The preset system time consumption regression model is trained based on system load sample data, task slicing sample data and system time consumption sample data. The system load sample data includes CPU usage, remaining available memory, remaining available disk I / O bandwidth, and remaining network I / O bandwidth. The preset system time consumption regression model is a multivariate polynomial regression model between CPU usage, remaining available memory, remaining available disk I / O bandwidth, remaining network I / O bandwidth, number of task slicing and system time consumption.

[0057] Specifically, the server inputs the current CPU usage, remaining available memory, remaining available disk I / O bandwidth, and remaining network I / O bandwidth into a preset system time regression model, and converts the preset system time regression model from a multivariate polynomial to a unary batch task partitioning expression.

[0058] Step 206, solving the batch task slicing expression to obtain the number of task slicings.

[0059] The number of task shards refers to the number of subtasks into which a batch task is divided.

[0060] Specifically, the server solves the task slicing derivative expression to obtain the extreme point data of the task slicing derivative expression, that is, the number of task slicing corresponding to the minimum system time consumption. The number of task slicing corresponding to the minimum system time consumption is the optimal number of slicing. In actual application, after the server solves the task slicing derivative expression to obtain the number of slicing, it compares the number of slicing with the number of CPU cores. If the number of CPU cores is less than the number of slicing, the number of CPU cores is used as the number of task slicing, that is, the optimal number of slicing. If the number of CPU cores is greater than or equal to the number of slicing, the number of slicing is used as the number of task slicing, that is, the optimal number of slicing.

[0061] Step 208: divide the batch task into slices and process them in parallel according to the number of task slices.

[0062] Specifically, the server divides the batch task into subtasks corresponding to the number of task slices, generates and calls the number of threads corresponding to the number of task slices to perform parallel processing on the sliced ​​subtasks, at which time the system consumes the least time.

[0063] In the above batch task processing method, multi-dimensional system load parameters are obtained; the system load parameters are input into a preset system time-consuming regression model, and the preset system time-consuming regression model is converted into a batch task slicing expression; the preset system time-consuming regression model is trained based on system load sample data, task slicing sample data, and system time-consuming sample data; the batch task slicing expression is solved to obtain the number of task slicings; and the batch tasks are sliced ​​and processed in parallel according to the number of task slicings. The entire solution reasonably allocates batch tasks according to multi-dimensional system load parameters (such as the current CPU usage, remaining available memory, remaining available disk I / O bandwidth, and remaining network I / O bandwidth) and the preset system time-consuming regression model, slices the batch tasks according to the number of task slicings, and then allocates a corresponding number of threads according to the number of task slicings to process the sliced ​​batch tasks, thereby improving the efficiency of batch task processing.

[0064] In an optional embodiment, solving the batch task slice expression to obtain the number of task slices includes: deriving the batch task slice expression to obtain a task slice derivative expression; and solving and determining the number of task slices corresponding to the minimum system time consumption based on the task slice derivative expression.

[0065] The task slice expression is a univariate equation about the relationship between the number of task slices and the system time consumption.

[0066] Specifically, the server derives the unary batch task slicing expression to obtain the task slicing derivative expression, assigns the result value of the task slicing derivative expression to zero, solves and rounds the result to obtain the number of task slicing corresponding to the minimum system time consumption.

[0067] The machine configuration, environment and other factors of a system are the same. Therefore, the preset system time-consuming regression model corresponding to a system is the same. Therefore, when deploying batch tasks in the same system, it is only necessary to adjust the number of shards based on the fitted preset system time-consuming regression model, that is, the relationship between the number of system shards and the current system load, as well as the current system load data. The server can adaptively shard the batch tasks according to the optimal number of shards, generate and call the task shards to allocate the corresponding number of threads to perform parallel processing of the sharded subtasks, thereby improving the efficiency of batch execution tasks.

[0068] In an optional embodiment, multiple servers collaboratively process a batch task in a distributed manner. Among the servers that collaboratively process batch tasks, one server is a master server and the other servers are slave servers. The master server, as a master device, obtains the system load parameters of all servers that collaboratively process batch tasks and the preset system time regression model corresponding to the server, that is, the master server obtains the system load parameters of the master server and the system load parameters of other slave servers, as well as the preset system time regression model corresponding to each server, calculates the number of task slices corresponding to each server, adds the number of task slices of each server to obtain the total number of task slices, multiplies the ratio of the number of task slices of each server to the total number of task slices by the total number of batch tasks, obtains the number of tasks of the server, and allocates the task data of the corresponding number of tasks to the server. For example, there are three servers A, B, and C that collaboratively process a batch task in a distributed manner. The system load parameters of the three servers A, B, and C and the corresponding preset system time-consuming regression models are obtained. According to the system load parameters of server A and the preset system time-consuming regression model corresponding to server A, the number of shards of task A is obtained; according to the system load parameters of server B and the preset system time-consuming regression model corresponding to server B, the number of shards of task B is obtained; according to the system load parameters of server C and the preset system time-consuming regression model corresponding to server C, the number of shards of task C is obtained. The number of shards of task A, the number of shards of task B, and the number of shards of task C are added together to obtain the total number of task shards, the number of shards of task A / total number of task shards*total number of batch tasks=the number of A tasks, the number of shards of task B / total number of task shards*total number of batch tasks=the number of B tasks, and the number of shards of task C / total number of task shards*total number of batch tasks=the number of C tasks.

[0069] In an optional embodiment, the generation process of the preset system time-consuming regression model includes: constructing an initial system time-consuming regression model according to system resource parameters; acquiring sample data, and fitting the initial system time-consuming regression model based on the sample data and using the least squares method to obtain the preset system time-consuming regression model.

[0070] Among them, the sample data is obtained through a large number of tests, recording the system under different CPU usage, remaining available memory, remaining available disk I / O bandwidth, remaining network I / O bandwidth, and the number of allocated shards. The execution time of batch tasks is observed, and historical operation data is obtained, and sample data is obtained from the historical operation data. The sample data includes system load sample data, task shard sample data, and system time consumption sample data, that is, a set of data including the current CPU usage, remaining available memory, remaining available disk IO bandwidth, remaining network IO bandwidth, number of task shards, and the time it takes for the system to process batch tasks.

[0071] The initial system time consumption regression model is a polynomial function that expresses the relationship between CPU usage, remaining available memory, remaining available disk IO bandwidth, remaining network IO bandwidth, the number of task shards, and the time consumption of the system to process batch tasks. The polynomial coefficients are unknown.

[0072] Specifically, the server constructs an initial system time regression model based on CPU usage, remaining available memory, remaining available disk IO bandwidth, remaining network IO bandwidth, number of task shards, and the time it takes for the system to process batch tasks, namely a five-variable five-order polynomial regression model. The initial system time regression model is simulated as follows:

[0073]

[0074] y is the time it takes for the system to process batch tasks. There are five factors in total that affect the time it takes to process batch tasks, namely: CPU usage, remaining available memory, remaining available disk I / O bandwidth, remaining network I / O bandwidth, and the number of allocated shards. This polynomial reflects the dependency and relative independence between various resources.

[0075] The coefficients of each polynomial in the initial system time-consuming regression model are fitted according to the least squares method, so that the error between the data value in the fitted system time-consuming regression model and the sample data value is minimized, and a fitted preset system time-consuming regression model is obtained.

[0076] Due to the differences in machine configuration, environment and other factors of each system, the polynomial relationship function corresponding to the preset system time-consuming regression model is different. Therefore, when deploying batch tasks in different systems, the polynomial relationship function of the system must be fitted. The polynomial relationship functions of different systems are not universal. After the fitting is successful, the batch task adaptively calculates the number of task shards before executing each batch task. Calculating the number of shards takes a certain amount of time, but it is negligible compared to the total execution time of the batch task.

[0077] In an optional embodiment, sample data is obtained, and an initial system time regression model is fitted based on the sample data and using the least squares method to obtain a preset system time regression model, including: identifying the data type dimension included in the system resource parameters; determining the number of required sample data based on the data type dimension; obtaining sample data from historical data based on the number of required sample data; and fitting the initial system time regression model based on the sample data and the least squares method to obtain a preset system time regression model.

[0078] Among them, the data type dimension is the number of parameter types in the system resource parameters. In this embodiment, the system resource parameters include the current CPU usage, remaining available memory, remaining available disk IO bandwidth, remaining network IO bandwidth, and the number of task shards, so the data type dimension is 5.

[0079] Specifically, the number of samples is determined according to the number of system resource parameters, and the expression for determining the number of samples is: b represents the number of system resource parameters. The number of system resource parameters is input into the sample determination expression for calculation to obtain the number of required sample data. Sample data of the required number of sample data is obtained from historical operation data. Each set of CPU usage, remaining available memory, remaining available disk IO bandwidth, remaining network IO bandwidth, number of task shards and system processing batch task time data in the sample data is input into the initial system time regression model to fit the polynomial coefficients. When the fitting error reaches the preset threshold, the preset system time regression model is obtained.

[0080] The least squares algorithm fits the polynomial through R+1 sets of different sample data and calculates the coefficient a of the polynomial i , so that the sum of squares of the difference between the value of the fitting curve, that is, the initial system time-consuming regression model in the R+1 group of data and the known function value is minimized, and a fitted polynomial regression equation is obtained, that is, the preset system time-consuming regression model.

[0081] In an optional embodiment, the initial system time consumption regression model is fitted according to the sample data and the least squares method to obtain the preset system time consumption regression model, including: fitting the initial system time consumption regression model according to the sample data and the least squares method to obtain the linear relationship between the system time consumption and the system resource parameters; according to the linear relationship between the system time consumption and the system resource parameters, constructing the preset system time consumption regression model.

[0082] Specifically, the sample data is substituted into the fitting function, the sum of the squares of the differences between the fitting function value and the system time consumption in the sample data is calculated, and the polynomial coefficients of the fitting function that minimizes the sum of the squares of the differences between the fitting function value and the system time consumption in the sample data are solved, that is, the linear relationship between the system time consumption and the system resource parameters is obtained, and the polynomial coefficients are substituted into the fitting function to obtain the fitted polynomial regression equation, that is, the preset system time consumption regression model.

[0083] In an optional embodiment, the initial system time consumption regression model is fitted according to sample data and the least squares method to obtain the linear relationship between the system time consumption and the system resource parameters, including: constructing the objective function of the system resource parameters and the system time consumption in the initial system time consumption regression model according to the least squares method; solving the objective function according to the sample data to determine the coefficients of the system resource parameters that minimize the error of the objective function; and obtaining the linear relationship between the system time consumption and the system resource parameters according to the coefficients of the system resource parameters.

[0084] Specifically, according to the least squares method, the square sum of the deviations between the sample data and the fitting curve function value of the initial system time-consuming regression model is constructed, that is, the loss function. The objective function is as follows:

[0085]

[0086] Where P n (x1, x2, x3, x4, x5) is the fitting curve, y j is the known function value, x1 is the CPU usage, x2 is the remaining available memory, x3 is the remaining available disk I / O bandwidth, x4 is the remaining network I / O bandwidth, and x5 is the number of task shards.

[0087] Formula (1) has a total of Term, the product of each variable of the polynomial (i.e. the product between x1 and x2, x3, x4, x5, i5,...i1, i from 0-5, there are 252 products) use υ i The coefficient of each term is represented by α i If we express it as

[0088]

[0089] According to the principle of least squares algorithm, the equations can be listed as follows:

[0090]

[0091] Find the coefficients a0, a1....a of the quintic polynomial q-1 , where q = 252, so 252 coefficients need to be calculated, and at least 252 sample points are needed to solve the equation group j represents the first group of known data, y j Indicates the execution time of batch steps under group data.

[0092] Through a large number of tests, we recorded the execution time of batch tasks under different CPU usage, remaining available memory, remaining available disk I / O bandwidth, remaining network I / O bandwidth, and server load conditions with different task shards, and obtained a large amount of sample data.

[0093] By fitting a multivariate polynomial with a large amount of sample data and fitting the polynomial using the least squares method, we can fit the relationship between y and x1, x2, x3, x4, and x5.

[0094] The quintic polynomial of degree 5 requires at least 252 sets of sample data (x 1,0 , x 2,0 , x 3,0 , x 4,0 , x 5,0, y0), (x 1,1 , x 2,1 , x 3,1 , x 4,1 , x 5,1 ,y1).......(x 1,R , x 2,R , x 3,R , x 4,R , x 5,R , y5). Where R≧252.

[0095] Through the sample points, all υ can be obtained ij ,like Figure 3 As shown, the calculation process is as follows:

[0096] (1) Let i = 0;

[0097] (2) Set i1, i2, ... i5 = 0;

[0098] (3) Assign j1, j2, …j5 the value of 5;

[0099] (4) Order For ij ;

[0100] (5) Order

[0101] (6) Let i5+1=i5;

[0102] (7) Judgement i m With j m The size relationship of

[0103] (8) When i m ≤j m , return to step (4);

[0104] (9) When i m >jm, find the first one that satisfies i from i4,i3,i2,i1 R ≤j R Condition R;

[0105] (10) Determine the relationship between R and 1;

[0106] (11) If R≤1, 252 υ ij Calculation completed;

[0107] (12) If R>1, let i R+1 =j R ;

[0108] (13) Let i R+1 ,i R+2 …i m = 0, and calculate jR+1 ,j R+2 …j m value, and return to step (4).

[0109] When υ is calculated i Coefficients, that is, the coefficients of system resource parameters, and then fitting the polynomial P n (x1, x2, x3, x4, x5) successfully obtains the linear relationship between the system time consumption and the system resource parameters. According to the linear relationship between the system time consumption and the system resource parameters, the preset system time consumption regression model is obtained.

[0110] In order to facilitate understanding of the technical solution provided in the embodiment of the present application, the batch task processing method provided in the embodiment of the present application is briefly described with a complete batch task processing process:

[0111] (1) Obtain the system's current CPU usage x1, remaining available memory x2, remaining available disk I / O bandwidth x3, and remaining network I / O bandwidth x4.

[0112] (2) Substitute the system's current CPU usage x1, remaining available memory x2, remaining available disk I / O bandwidth x3, and remaining network I / O bandwidth x4 into the fitted preset system time-consuming regression model and convert it into the batch task partitioning expression P(x1, x2, x3, x4, x5).

[0113] (3) The preset system time-consuming regression model is converted into a batch task partitioning expression, that is, P(x1, x2, x3, x4, x5) is converted into a univariate equation P(x5) about x5.

[0114] (4) Deriving the batch task partition expression, we obtain the task partition derivative expression. That is, we derive P(x5) to obtain the task partition derivative expression P′(x5).

[0115] (5) Assign the result value of the task slice derivative expression to zero, solve and round the result to obtain the number of task slices corresponding to the minimum system time consumption. Let P′(x5)=0, calculate x5, and round it. At this time, x5 is the number of task slices.

[0116] (6) Divide the batch task into subtasks corresponding to the number of task slices, generate and call threads corresponding to the number of task slices to perform parallel processing on the subtasks after the slices.

[0117] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0118] Based on the same inventive concept, the embodiment of the present application also provides a batch task processing device for implementing the batch task processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more batch task processing device embodiments provided below can refer to the limitations of the batch task processing method above, and will not be repeated here.

[0119] In one embodiment, Figure 5 As shown, a batch task processing device is provided, including: an acquisition module 502, an input module 504, a slicing module 506 and a processing module 508, wherein:

[0120] The acquisition module 502 is used to acquire multi-dimensional system load parameters.

[0121] Input module 504 is used to input system load parameters into a preset system time-consuming regression model and convert the preset system time-consuming regression model into a batch task slicing expression; the preset system time-consuming regression model is trained based on system load sample data, task slicing sample data and system time-consuming sample data.

[0122] The slicing module 506 is used to solve the batch task slicing expression to obtain the number of task slicings.

[0123] The processing module 508 is used to perform parallel processing on batch tasks in slices according to the number of task slices.

[0124] In an optional embodiment, the slicing module 506 is further used to derive the batch task slicing expression to obtain a task slicing derivative expression; and solve and determine the number of task slicing corresponding to the minimum system time consumption according to the task slicing derivative expression.

[0125] In an optional embodiment, the batch task processing device also includes a training module for constructing an initial system time consumption regression model based on system resource parameters; the system resource parameters include system load parameters and task slice parameters; obtaining sample data, and fitting the initial system time consumption regression model based on the sample data and using the least squares method to obtain a preset system time consumption regression model, wherein the sample data includes system load sample data, task slice sample data and system time consumption sample data.

[0126] In an optional embodiment, the training module is also used to identify the data type dimension included in the system resource parameters; determine the number of required sample data based on the data type dimension; obtain sample data from historical data based on the number of required sample data; fit the initial system time regression model based on the sample data and the least squares method to obtain a preset system time regression model.

[0127] In an optional embodiment, the training module is also used to fit the initial system time consumption regression model based on sample data and least squares method to obtain the linear relationship between system time consumption and system resource parameters; based on the linear relationship between system time consumption and system resource parameters, a preset system time consumption regression model is constructed.

[0128] In an optional embodiment, a training module is used to construct an objective function of system resource parameters and system time consumption in an initial system time consumption regression model based on the least squares method; solve the objective function based on sample data to determine the coefficients of the system resource parameters that minimize the error of the objective function; and obtain the linear relationship between the system time consumption and the system resource parameters based on the coefficients of the system resource parameters.

[0129] Each module in the batch task processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0130] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store a preset system time-consuming regression model. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a batch task processing method is implemented.

[0131] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0132] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0133] Obtain multi-dimensional system load parameters;

[0134] Input the system load parameters into a preset system time-consuming regression model, and convert the preset system time-consuming regression model into a batch task slicing expression; the preset system time-consuming regression model is trained based on system load sample data, task slicing sample data, and system time-consuming sample data;

[0135] Solve the batch task sharding expression to obtain the number of task shards;

[0136] Batch tasks are divided into shards and processed in parallel according to the number of task shards.

[0137] In one embodiment, when the processor executes the computer program, the following steps are also implemented: solving the batch task slice expression to obtain the number of task slices, including: deriving the batch task slice expression to obtain a task slice derivative expression; and solving and determining the number of task slices corresponding to the minimum system time consumption based on the task slice derivative expression.

[0138] In one embodiment, the processor also implements the following steps when executing the computer program: the generation process of the preset system time consumption regression model includes: constructing an initial system time consumption regression model according to system resource parameters; the system resource parameters include system load parameters and task slice parameters; obtaining sample data, and fitting the initial system time consumption regression model based on the sample data and using the least squares method to obtain a preset system time consumption regression model, and the sample data includes system load sample data, task slice sample data and system time consumption sample data.

[0139] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining sample data, fitting an initial system time regression model based on the sample data and using the least squares method to obtain a preset system time regression model, including: identifying the data type dimension contained in the system resource parameters; determining the number of required sample data based on the data type dimension; obtaining sample data from historical data based on the number of required sample data; fitting the initial system time regression model based on the sample data and the least squares method to obtain a preset system time regression model.

[0140] In one embodiment, the processor also implements the following steps when executing the computer program: fitting the initial system time consumption regression model according to the sample data and the least squares method to obtain the preset system time consumption regression model, including: fitting the initial system time consumption regression model according to the sample data and the least squares method to obtain the linear relationship between the system time consumption and the system resource parameters; constructing the preset system time consumption regression model according to the linear relationship between the system time consumption and the system resource parameters.

[0141] In one embodiment, the processor also implements the following steps when executing the computer program: fitting the initial system time consumption regression model according to sample data and the least squares method to obtain the linear relationship between the system time consumption and the system resource parameters includes: constructing the objective function of the system resource parameters and the system time consumption in the initial system time consumption regression model according to the least squares method; solving the objective function according to the sample data to determine the coefficients of the system resource parameters that minimize the error of the objective function; and obtaining the linear relationship between the system time consumption and the system resource parameters according to the coefficients of the system resource parameters.

[0142] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0143] Obtain multi-dimensional system load parameters;

[0144] Input the system load parameters into a preset system time-consuming regression model, and convert the preset system time-consuming regression model into a batch task slicing expression; the preset system time-consuming regression model is trained based on system load sample data, task slicing sample data, and system time-consuming sample data;

[0145] Solve the batch task sharding expression to obtain the number of task shards;

[0146] Batch tasks are divided into shards and processed in parallel according to the number of task shards.

[0147] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: solving the batch task slice expression to obtain the number of task slices, including: deriving the batch task slice expression to obtain a task slice derivative expression; and solving and determining the number of task slices corresponding to the minimum system time consumption based on the task slice derivative expression.

[0148] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the generation process of the preset system time consumption regression model includes: constructing an initial system time consumption regression model according to system resource parameters; the system resource parameters include system load parameters and task slice parameters; obtaining sample data, and fitting the initial system time consumption regression model based on the sample data and using the least squares method to obtain a preset system time consumption regression model, and the sample data includes system load sample data, task slice sample data and system time consumption sample data.

[0149] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining sample data, fitting an initial system time regression model based on the sample data and using the least squares method to obtain a preset system time regression model, including: identifying the data type dimension contained in the system resource parameters; determining the number of required sample data based on the data type dimension; obtaining sample data from historical data based on the number of required sample data; fitting the initial system time regression model based on the sample data and the least squares method to obtain a preset system time regression model.

[0150] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: fitting the initial system time consumption regression model according to the sample data and the least squares method to obtain the preset system time consumption regression model, including: fitting the initial system time consumption regression model according to the sample data and the least squares method to obtain the linear relationship between the system time consumption and the system resource parameters; constructing the preset system time consumption regression model according to the linear relationship between the system time consumption and the system resource parameters.

[0151] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: fitting the initial system time consumption regression model according to sample data and the least squares method to obtain the linear relationship between the system time consumption and the system resource parameters includes: constructing the objective function of the system resource parameters and the system time consumption in the initial system time consumption regression model according to the least squares method; solving the objective function according to the sample data to determine the coefficients of the system resource parameters that minimize the error of the objective function; and obtaining the linear relationship between the system time consumption and the system resource parameters according to the coefficients of the system resource parameters.

[0152] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0153] Obtain multi-dimensional system load parameters;

[0154] Input the system load parameters into a preset system time-consuming regression model, and convert the preset system time-consuming regression model into a batch task slicing expression; the preset system time-consuming regression model is trained based on system load sample data, task slicing sample data, and system time-consuming sample data;

[0155] Solve the batch task sharding expression to obtain the number of task shards;

[0156] Batch tasks are divided into shards and processed in parallel according to the number of task shards.

[0157] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: solving the batch task slice expression to obtain the number of task slices, including: deriving the batch task slice expression to obtain a task slice derivative expression; and solving and determining the number of task slices corresponding to the minimum system time consumption based on the task slice derivative expression.

[0158] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the generation process of the preset system time consumption regression model includes: constructing an initial system time consumption regression model according to system resource parameters; the system resource parameters include system load parameters and task slice parameters; obtaining sample data, and fitting the initial system time consumption regression model based on the sample data and using the least squares method to obtain a preset system time consumption regression model, and the sample data includes system load sample data, task slice sample data and system time consumption sample data.

[0159] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining sample data, fitting an initial system time regression model based on the sample data and using the least squares method to obtain a preset system time regression model, including: identifying the data type dimension contained in the system resource parameters; determining the number of required sample data based on the data type dimension; obtaining sample data from historical data based on the number of required sample data; fitting the initial system time regression model based on the sample data and the least squares method to obtain a preset system time regression model.

[0160] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: fitting the initial system time consumption regression model according to the sample data and the least squares method to obtain the preset system time consumption regression model, including: fitting the initial system time consumption regression model according to the sample data and the least squares method to obtain the linear relationship between the system time consumption and the system resource parameters; constructing the preset system time consumption regression model according to the linear relationship between the system time consumption and the system resource parameters.

[0161] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: fitting the initial system time consumption regression model according to sample data and the least squares method to obtain the linear relationship between the system time consumption and the system resource parameters includes: constructing the objective function of the system resource parameters and the system time consumption in the initial system time consumption regression model according to the least squares method; solving the objective function according to the sample data to determine the coefficients of the system resource parameters that minimize the error of the objective function; and obtaining the linear relationship between the system time consumption and the system resource parameters according to the coefficients of the system resource parameters.

[0162] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0163] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0164] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A batch task processing method, characterized in that: The method comprises: Obtain multi-dimensional system load parameters; the system load parameters represent the available resources of the system; the system load parameters include CPU usage, remaining available memory, remaining available disk IO bandwidth, and remaining network IO bandwidth; The system load parameters are input into a preset system time-consuming regression model, and the preset system time-consuming regression model is converted into a batch task slicing expression; the preset system time-consuming regression model is trained based on system load sample data, task slicing sample data and system time-consuming sample data; the system load sample data includes CPU usage, remaining available memory, remaining available disk I / O bandwidth, and remaining network I / O bandwidth; the preset system time-consuming regression model is a multivariate polynomial regression model between CPU usage, remaining available memory, remaining available disk I / O bandwidth, remaining network I / O bandwidth, number of task slicing and system time consumption; the generation process of the preset system time-consuming regression model includes: constructing an initial system time-consuming regression model according to system resource parameters; the system resource parameters include system load parameters and task slicing parameters; identifying the data type dimension contained in the system resource parameters; determining the number of required sample data according to the data type dimension; obtaining sample data from historical data according to the number of required sample data; fitting the initial system time-consuming regression model according to the sample data and the least squares method to obtain the preset system time-consuming regression model; Solving the batch task slice expression to obtain the number of task slices; solving the batch task slice expression to obtain the number of task slices, including: deriving the batch task slice expression to obtain a task slice derivative expression; solving and determining the number of task slices corresponding to the minimum system time consumption according to the task slice derivative expression; wherein, after solving the task slice derivative expression to obtain the number of slices, comparing the number of slices with the number of CPU cores, if the number of CPU cores is less than the number of slices, using the number of CPU cores as the number of task slices, and if the number of CPU cores is greater than or equal to the number of slices, using the number of slices as the number of task slices; The batch tasks are sliced ​​and processed in parallel according to the number of task slices.

2. The method according to claim 1, characterized in that The sample data includes system load sample data, task slice sample data and system time consumption sample data.

3. The method according to claim 1, characterized in that The fitting of the initial system time-consuming regression model according to the sample data and the least squares method to obtain a preset system time-consuming regression model includes: Fitting the initial system time consumption regression model according to the sample data and the least square method to obtain a linear relationship between the system time consumption and the system resource parameters; According to the linear relationship between the system time consumption and the system resource parameters, a preset system time consumption regression model is constructed.

4. The method according to claim 3, characterized in that The fitting of the initial system time consumption regression model according to the sample data and the least square method to obtain the linear relationship between the system time consumption and the system resource parameter comprises: According to the least squares method, construct an objective function of system resource parameters and system time consumption in the initial system time consumption regression model; Solving the objective function according to the sample data to determine the coefficients of the system resource parameters that minimize the error of the objective function; According to the coefficient of the system resource parameter, a linear relationship between the system time consumption and the system resource parameter is obtained.

5. The method according to claim 4, characterized in that The objective function of system resource parameters and system time consumption in the initial system time consumption regression model is constructed according to the least squares method, including: According to the least square method, the sum of squares of deviations between the sample data and the fitting curve function value of the initial system time-consuming regression model is constructed.

6. A batch task processing device, characterized in that: The device comprises: An acquisition module is used to acquire multi-dimensional system load parameters; the system load parameters represent the available resources of the system; the system load parameters include CPU usage, remaining available memory, remaining available disk IO bandwidth, and remaining network IO bandwidth; An input module is used to input the system load parameters into a preset system time-consuming regression model, and convert the preset system time-consuming regression model into a batch task slicing expression; the preset system time-consuming regression model is trained based on system load sample data, task slicing sample data, and system time-consuming sample data; the system load sample data includes CPU usage, remaining available memory, remaining available disk I / O bandwidth, and remaining network I / O bandwidth; the preset system time-consuming regression model is a multivariate polynomial regression model between CPU usage, remaining available memory, remaining available disk I / O bandwidth, remaining network I / O bandwidth, number of task slicing, and system time consumption; A slicing module is used to solve the batch task slicing expression to obtain the number of task slicings; solving the batch task slicing expression to obtain the number of task slicings includes: deriving the batch task slicing expression to obtain a task slicing derivative expression; solving and determining the number of task slicings corresponding to the minimum system time consumption according to the task slicing derivative expression; wherein, after solving the task slicing derivative expression to obtain the number of slicings, the number of slicings is compared with the number of CPU cores, if the number of CPU cores is less than the number of slicings, the number of CPU cores is used as the number of task slicings, and if the number of CPU cores is greater than or equal to the number of slicings, the number of slicings is used as the number of task slicings; A processing module, used for performing parallel processing of the batch tasks in slices according to the number of task slices; The batch task processing device also includes a training module, which is used to construct an initial system time-consuming regression model based on system resource parameters; the system resource parameters include system load parameters and task slice parameters; identify the data type dimensions contained in the system resource parameters; determine the number of required sample data based on the data type dimensions; obtain sample data from historical data based on the number of required sample data; fit the initial system time-consuming regression model based on the sample data and the least squares method to obtain a preset system time-consuming regression model.

7. The device according to claim 6, characterized in that The training module is also used to fit the initial system time consumption regression model according to sample data and least squares method to obtain the linear relationship between system time consumption and system resource parameters; based on the linear relationship between system time consumption and system resource parameters, a preset system time consumption regression model is constructed.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images