A batch task processing method and system based on traffic distribution

Abstract

The present invention belongs to the technical field of data processing and traffic distribution, and proposes a batch task processing method and system based on traffic distribution, specifically: initialize the batch task processing scenario, which includes a load balancing operator, and then obtain the processing feature group of each processing node from the batch task processing scenario, and then calculate the distribution efficiency for each processing node according to the processing feature group, and finally calculate the imbalanced effective state weight based on each distribution efficiency obtained in the batch task processing scenario and send it to the scheduling node. The risk of balanced performance collapse caused by large differences in server computing power and performance in distributed systems is effectively quantified. By introducing the data in the traffic distribution technology for analysis, tasks and data are dynamically distributed to the most suitable processing nodes or servers, so the efficiency of batch task processing can still be improved when the task processing volume is even larger.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing and traffic distribution, and in particular relates to a batch task processing method and system based on traffic distribution. Background Art

[0002] Batch task processing can shorten task processing time and effectively improve efficiency by performing parallel computing on multiple servers. The traditional method is to establish a server cluster and use a static allocation strategy to assign requests to different servers for processing. However, with the rapid development of the Internet and big data technologies, especially driven by emerging technologies such as cloud computing, the Internet of Things, and artificial intelligence, the amount of data is facing explosive growth, and the demand for computing power is increasing day by day.

[0003] In order to meet the needs of big data, enterprises now use distributed systems to deploy servers in different regions, so as to have powerful computing power to process big data and realize batch task processing. However, due to the long construction cycle of distributed systems, in order to meet market business needs, it is usually necessary to update the server. In addition, the different years of service of the server will cause the performance differences between servers to change. The task allocation method for batch task processing is usually based on the load balancing operator for task matching, which easily ignores the differences in server computing power or performance, resulting in a balanced performance collapse phenomenon. The balanced performance collapse phenomenon refers to the problem of efficiency not increasing but decreasing in the process of task advancement after the load balancing control batch task processing allocation; especially in the current popular big data application scenarios, the amount of data in batch task processing is even larger, and this balanced performance collapse phenomenon will be more prominent. Therefore, there is an urgent need for a batch task processing method and system based on traffic distribution to effectively solve and optimize this problem. Summary of the invention

[0004] The purpose of the present invention is to propose a batch task processing method and system based on traffic distribution to solve one or more technical problems existing in the prior art and at least provide a beneficial choice or create conditions.

[0005] In order to achieve the above object, according to one aspect of the present invention, a batch task processing method based on traffic distribution is provided, and the method comprises the following steps:

[0006] S100, initializing the batch task processing scenario, which includes the load balancing operator;

[0007] S200, obtaining a processing feature group of each processing node from a batch task processing scenario;

[0008] S300, calculating the distribution efficiency for each processing node according to the processing feature group;

[0009] S400, calculating the imbalanced efficiency weight in combination with various distribution efficiencies obtained in the batch task processing scenario and sending it to the scheduling node.

[0010] Further, in step S100, a batch task processing scenario is initialized, wherein the method including a load balancing operator is as follows: a distributed system is built in the batch task processing scenario, wherein the distributed system is composed of a number of servers; the servers responsible for batch task processing in the distributed system are used as processing nodes, wherein the servers for batch task processing are computing nodes that execute batch processing jobs; the servers responsible for managing and allocating tasks in the distributed system are used as scheduling nodes, wherein the scheduling nodes are preset with load balancing operators.

[0011] Further, in step S200, the method for obtaining the processing feature group of each processing node from the batch task processing scenario is: define any batch task being completely executed as an execution unit; the batch task includes batch processing data, the batch processing data is divided into a plurality of sub-processing data by a load balancing operator, and each sub-processing data has a one-to-one correspondence with a processing node; the ratio of the data size of the sub-processing data obtained by a processing node to the data size of the batch processing data is the ratio to be distributed;

[0012] The time spent on executing the batch job in the processing node is recorded as the execution feedback time; the minimum execution feedback time of each processing node under an execution unit is obtained and recorded as the delay base, and the difference between the execution feedback time and the delay base is recorded as the delay distance DEP_S of the execution unit, then the execution delay degree Pcss_DEP is: Pcss_DEP=e DEP_S ; The binary group consisting of the proportion to be distributed and the execution delay corresponding to any execution unit is recorded as a processing feature group.

[0013] Furthermore, in step S300, the method for calculating the distribution efficiency for each processing node according to the processing feature group is as follows: ROAM is the monitoring period, ROAM∈[3,10] hours; suppose that n execution units are input in the current ROAM period, there are m processing nodes, and then an m×n-order two-dimensional processing feature matrix Tri(i, j) is obtained; the ratio of the input data to be distributed to the jth execution unit of the i-th processing node is Ratio(i, j), which is abbreviated as R i,j , and the matrix formed by them is ∑R i,j ,∑R i,j The largest element in is denoted as RMX; the execution delay of the jth execution unit of the i-th processing node is Pcss_DEP(i,j), abbreviated as P i,j , and the matrix formed by them is ∑P i,j， ∑P i,jThe largest element in is recorded as PMX, so the element corresponding to the two-dimensional processing feature matrix Tri(i,j) is (P i,j , R i,j );

[0014] ∑P i,j The elements in are sorted from small to large, and the upper quartile is recorded as the high extension value Pc(75%), the lower quartile is recorded as the low extension value Pc(25%), and the value between Pc(25%) and Pc(75%) is recorded as the medium extension value;

[0015] If P i,j <Pc(25%), the distribution efficiency is less affected by the execution delay and is considered to be only related to the proportion to be distributed Ratio(i,j). It is known that for the jth processing unit, ,Data processing was performed using CLR central logarithmic transformation, ,R j The geometric mean of j , get the logarithmic distribution ratio ;

[0016] Calculate the sub-distribution efficiency of the i-th processing node corresponding to the j-th execution unit ;

[0017] If Pc(25%)≤P i,j ≤Pc(75%), the distribution efficiency is affected by both execution delay and distribution efficiency. First, unify the dimensions of the two and normalize them. , , calculate the sub-distribution efficiency of the i-th processing node corresponding to the j-th execution unit ;

[0018] If P i,j >Pc(75%), the distribution efficiency is greatly affected by the execution delay, and is less affected by the proportion of distribution to be distributed Ratio(i,j). From the Markov property of the execution feedback time, it can be seen that its execution delay is affected by the execution delay of the previous execution unit P i,j-1 , and calculate the sub-distribution efficiency of the i-th processing node corresponding to the j-th execution unit. ;

[0019] Calculate the distribution efficiency Di_Eff = mean obtained by the i-th processing node ROAM {Di_Eff i}.

[0020] Since there is a phenomenon of missing execution delay data classification in the calculation process of the above distribution efficiency, it will lead to overfitting problems in the quantification of distribution efficiency, so that some execution units that do not obviously show the phenomenon of balanced performance collapse are misclassified, thereby further reducing the stability of the processing node screening function. This problem will be more obvious in particular when the processing capabilities of various devices in the distributed system are relatively different. In order to better solve this problem and eliminate the overfitting phenomenon, the present invention proposes a more preferred solution as follows:

[0021] Preferably, in step S300, the method of calculating the distribution efficiency for each processing node according to the processing feature group can be replaced by: assuming a time period as the monitoring period ROAM, ROAM∈[3,10] hours; assuming that a total of n execution units are input in the current ROAM period, there are m processing nodes, and then an m×n-order two-dimensional processing feature matrix Tri(i, j) is obtained, and the ratio of the input data to be distributed to the jth execution unit of the i-th processing node is Ratio(i, j), which is abbreviated as R i,j ;The execution delay of the jth execution unit of the i-th processing node is Pcss_DEP(i,j), abbreviated as P i,j , and the matrix formed by them is ∑P i,j , so the element corresponding to the two-dimensional processing feature matrix Tri(i,j) is (P i,j ,R i,j ), the distribution efficiency matrix Di_Eff(i,j) corresponds one to one with Tri(i,j);

[0022] ∑P i,j The elements in are sorted from small to large, and the upper quartile is recorded as the high extension value Pc(75%), the lower quartile is recorded as the low extension value Pc(25%), and the value between Pc(25%) and Pc(75%) is recorded as the medium extension value;

[0023] For the jth execution unit, record its indicative function , if the execution unit satisfies , then it is defined as a screening execution unit, and the jth execution unit is eliminated; the execution unit obtained after the elimination operation is recorded as a screening execution unit, and its number is recorded as k; the corresponding column of the screening execution unit is intercepted from Tri (i, j) to obtain an m×k-order two-dimensional efficient processing feature matrix HTri (i, k);

[0024] For the i-th processing node in the efficient processing feature matrix, the average of the proportions to be distributed corresponding to the k screening execution units is the coefficient to be distributed AVERatio i ;

[0025] Arrange HTri(i, k) in ascending order according to the coefficients to be distributed, and record its maximum value as the dense coefficient to be distributed MX_AVERatio, and its corresponding node is the dense processing node q; its minimum value is the sparse coefficient to be distributed MN_AVERatio, and its corresponding node is the sparse processing node s. The processing nodes other than the dense processing nodes and the sparse processing nodes are recorded as interval processing nodes;

[0026] The distribution feature vectors corresponding to the proportion of dense processing nodes q and sparse processing nodes s to be distributed are LSR q =(R q1 , R q2 ,…,R qk ),LSR s =(R s1 , R s2 ,…,R sk ),

[0027] Any interval processing node and LSR q and LSR s The Euclidean distance of the Euclidean distance is denoted as D_LSR q and D_LSR s , if D_LSR q ≤D_LSR s Then the processing node is defined as belonging to a dense data group, otherwise the processing node is defined as belonging to a sparse data group;

[0028] Calculate the execution effectiveness Ex_Eff of the kth screening execution unit in the efficient processing feature matrix HTri(i,k) k , Ex_Eff k =|P i,k -P i-1,k | / P i,k , when the value of i-1 is 0, then Ex_Eff k The value is 1;

[0029] When a processing node belongs to a sparse data group, it is considered that the data volume is small and the mutual influence of the cooperation efficiency between the processing nodes is small. The distribution efficiency corresponding to the i-th processing node is calculated. ; Where SUM(AVERatio) represents the sum of the coefficients to be distributed of all processing nodes, and the accumulation variable of the accumulation function is k;

[0030] When a processing node belongs to a dense data group, it is considered that the data volume is large and the coordination efficiency between the processing nodes has a great impact on each other. The distribution efficiency corresponding to the i-th processing node is calculated. .

[0031] Further, in step S400, the method of calculating the imbalanced effectiveness weight in combination with each distribution efficiency obtained in the batch task processing scenario and sending it to the scheduling node is: sorting the obtained distribution efficiency in descending order to obtain the descending sequence number DEC_RATING corresponding to each processing node; presetting a time period TL, TL∈[1,3] hours; recording the average distribution efficiency of all processing nodes in the TL period as the evaluation factor; taking any processing node as the current processing node, if the distribution efficiency under the current execution unit is greater than the evaluation factor, the current execution unit is defined as a type of utility unit, and the imbalanced effectiveness weight IEWt is calculated: ; k1 is the cumulative variable, n_k1 is the number of utility units of the current processing node in the TL period, Di_Eff k1 and DEC_RATING k1 They represent the distribution efficiency and descending sequence number of the k1th first-class utility unit respectively, and exp() is an exponential function with the natural constant e as the base; the imbalanced efficiency weights of each processing node are merged into a sequence and sent to the scheduling node.

[0032] Preferably, all undefined variables in the present invention, if not clearly defined, can be manually set thresholds.

[0033] The present invention also provides a batch task processing system based on traffic distribution, the batch task processing system based on traffic distribution includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, the processor implements the steps in the batch task processing method based on traffic distribution when executing the computer program, the batch task processing system based on traffic distribution can be run in computing devices such as desktop computers, notebook computers, PDAs, and cloud data centers, and the executable system may include, but is not limited to, a processor, a memory, and a server cluster, and the processor executes the computer program to run in the following system units:

[0034] The scenario construction unit is used to initialize the batch task processing scenario, which includes the load balancing operator;

[0035] A data acquisition unit, used to obtain a processing feature group of each processing node from a batch task processing scenario;

[0036] A distribution efficiency calculation unit, used to calculate the distribution efficiency of each processing node according to the processing feature group;

[0037] The processing node control unit is used to calculate the imbalanced efficiency weight based on the various distribution efficiencies obtained in the batch task processing scenario and send it to the scheduling node.

[0038] The beneficial effects of the present invention are as follows: the present invention provides a batch task processing method and system based on traffic distribution, which effectively quantifies the risk of balanced performance collapse caused by large differences in server computing power and performance in distributed systems. By introducing data from traffic distribution technology for analysis, tasks and data are dynamically distributed to the most suitable processing nodes or servers, so that the efficiency of batch task processing can still be improved when the task processing volume is even larger, effectively reducing the risk of efficiency reduction in the task advancement process based on load balancing control batch task processing allocation, and improving the execution efficiency of batch task processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and other features of the present invention will become more obvious by describing in detail the embodiments shown in the accompanying drawings. The same reference numerals in the accompanying drawings of the present invention represent the same or similar elements. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work. In the accompanying drawings:

[0040] Figure 1 Shown is a flow chart of a batch task processing method based on traffic distribution;

[0041] Figure 2 The figure shows a structure diagram of a batch task processing system based on traffic distribution. DETAILED DESCRIPTION

[0042] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0043] like Figure 1 The flowchart of a batch task processing method based on traffic distribution is shown below. Figure 1 A method for batch task processing based on traffic distribution according to an embodiment of the present invention is described below. The method comprises the following steps:

[0044] S100, initializing the batch task processing scenario, which includes the load balancing operator;

[0045] S200, obtaining a processing feature group of each processing node from a batch task processing scenario;

[0046] S300, calculating the distribution efficiency of each processing node according to the processing feature group;

[0047] S400, calculating the imbalanced efficiency weight in combination with various distribution efficiencies obtained in the batch task processing scenario and sending it to the scheduling node.

[0048] Further, in step S100, a batch task processing scenario is initialized, wherein the method including a load balancing operator is as follows: a distributed system is built in the batch task processing scenario, wherein the distributed system is composed of a number of servers; the servers responsible for batch task processing in the distributed system are used as processing nodes, wherein the servers for batch task processing are computing nodes that execute batch processing jobs; the servers responsible for managing and allocating tasks in the distributed system are used as scheduling nodes, wherein the scheduling nodes are preset with load balancing operators.

[0049] After the scheduling node obtains a batch processing task, it will use the load balancing operator to split the batch processing task into batch sub-processing tasks according to the data size ratio, and distribute the sub-batch processing tasks to the corresponding processing nodes for operation;

[0050] Further, in step S200, the method for obtaining the processing feature group of each processing node from the batch task processing scenario is: define any batch task being completely executed as an execution unit; the batch task includes batch processing data, the batch processing data is divided into a plurality of sub-processing data by a load balancing operator, and each sub-processing data has a one-to-one correspondence with a processing node; the ratio of the data size of the sub-processing data obtained by a processing node to the data size of the batch processing data is the ratio to be distributed;

[0051] The time spent on executing the batch job in the processing node is recorded as the execution feedback time; the minimum execution feedback time of each processing node under an execution unit is obtained and recorded as the delay base, and the difference between the execution feedback time and the delay base is recorded as the delay distance DEP_S of the execution unit, then the execution delay degree Pcss_DEP is: Pcss_DEP=e DEP_S ; The binary group consisting of the proportion to be distributed and the execution delay corresponding to any execution unit is recorded as a processing feature group.

[0052] Batch processing data is text, voice, video, image or other structured data; by default, all processing nodes are non-redundant and maintain normal operation.

[0053] The time taken by any processing node to execute a batch job refers to the time period from the moment when the scheduling node starts to send sub-processing data to the processing node to the moment when the scheduling node obtains all processing results from the processing node. The unit of feedback time is minutes.

[0054] Furthermore, in step S300, the method for calculating the distribution efficiency for each processing node according to the processing feature group is as follows: ROAM is the monitoring period, ROAM∈[3,10] hours; suppose that n execution units are input in the current ROAM period, there are m processing nodes, and then an m×n-order two-dimensional processing feature matrix Tri(i, j) is obtained; the ratio of the input data to be distributed to the jth execution unit of the i-th processing node is Ratio(i, j), which is abbreviated as R i,j , and the matrix formed by them is ∑R i,j ,∑R i,j The largest element in is denoted as RMX; the execution delay of the jth execution unit of the i-th processing node is Pcss_DEP(i,j), abbreviated as P i,j , and the matrix formed by them is ∑P i,j， ∑P i,j The largest element in is recorded as PMX, so the element corresponding to the two-dimensional processing feature matrix Tri(i,j) is (P i,j , R i,j ), the distribution efficiency matrix Di_Eff(i,j) has a one-to-one correspondence with Tri(i,j);

[0055] ∑P i,j The elements in are sorted from small to large, and the upper quartile is recorded as the high extension value Pc(75%), the lower quartile is recorded as the low extension value Pc(25%), and the value between Pc(25%) and Pc(75%) is recorded as the medium extension value;

[0056] If P i,j <Pc(25%), the distribution efficiency is less affected by the execution delay and is considered to be only related to the proportion to be distributed Ratio(i,j). It is known that for the jth processing unit, ,Data processing was performed using CLR central logarithmic transformation, ,R j The geometric mean of j , get the logarithmic distribution ratio );

[0057] Calculate the sub-distribution efficiency of the i-th processing node corresponding to the j-th execution unit ;

[0058] R _j The geometric mean is calculated as: ; exp() is an exponential function with the natural constant e as the base;

[0059] If Pc(25%)≤P i,j ≤Pc(75%), the distribution efficiency is affected by both execution delay and distribution efficiency. First, unify the dimensions of the two and normalize them. , , calculate the sub-distribution efficiency of the i-th processing node corresponding to the j-th execution unit ;

[0060] If P i,j >Pc(75%), the distribution efficiency is greatly affected by the execution delay, and is less affected by the proportion of distribution to be distributed Ratio(i,j). From the Markov property of the execution feedback time, it can be seen that its execution delay is affected by the execution delay of the previous execution unit P i,j-1 , and calculate the sub-distribution efficiency of the i-th processing node corresponding to the j-th execution unit. ;

[0061] Calculate the distribution efficiency Di_Eff = mean obtained by the i-th processing node ROAM {Di_Eff i}, mean{} is the average function, mean ROAM {Di_Eff i} represents the average value of each sub-distribution efficiency during the ROAM period.

[0062] Where j-1 refers to the first execution unit searched in the reverse time direction by the jth execution unit;

[0063] Preferably, in step S300, the method of calculating the distribution efficiency for each processing node according to the processing feature group can be replaced by: assuming a time period as the monitoring period ROAM, ROAM∈[3,10] hours; assuming that a total of n execution units are input in the current ROAM period, there are m processing nodes, and then an m×n-order two-dimensional processing feature matrix Tri(i, j) is obtained, and the ratio of the input data to be distributed to the jth execution unit of the i-th processing node is Ratio(i, j), which is abbreviated as R i,j ;The execution delay of the jth execution unit of the i-th processing node is Pcss_DEP(i,j), abbreviated as P i,j , and the matrix formed by them is ∑P i,j , so the element corresponding to the two-dimensional processing feature matrix Tri(i,j) is (P i,j ,R i,j ), the distribution efficiency matrix Di_Eff(i,j) corresponds one to one with Tri(i,j);

[0064] ∑P i,j The elements in are sorted from small to large, and the upper quartile is recorded as the high extension value Pc(75%), the lower quartile is recorded as the low extension value Pc(25%), and the value between Pc(25%) and Pc(75%) is recorded as the medium extension value;

[0065] For the jth execution unit, record its indicative function , if the execution unit satisfies , then it is defined as a screening execution unit, and the jth execution unit is eliminated; the execution unit obtained after the elimination operation is recorded as a screening execution unit, and its number is recorded as k; the corresponding column of the screening execution unit is intercepted from Tri (i, j) to obtain an m×k-order two-dimensional efficient processing feature matrix HTri (i, k);

[0066] For the i-th processing node in the efficient processing feature matrix, the average of the proportions to be distributed corresponding to the k screening execution units is the coefficient to be distributed AVERatio i ;

[0067] Arrange HTri(i, k) in ascending order according to the coefficients to be distributed, and record its maximum value as the dense coefficient to be distributed MX_AVERatio, and its corresponding node is the dense processing node q; its minimum value is the sparse coefficient to be distributed MN_AVERatio, and its corresponding node is the sparse processing node s. The processing nodes other than the dense processing nodes and the sparse processing nodes are recorded as interval processing nodes;

[0068] The distribution feature vectors corresponding to the proportion of dense processing nodes q and sparse processing nodes s to be distributed are LSR q =(R q1 , R q2 ,…,R qk ),LSR s =(R s1 , R s2 ,…,R sk ),

[0069] Any interval processing node and LSR q and LSR s The Euclidean distance of the Euclidean distance is denoted as D_LSR q and D_LSR s , if D_LSR q ≤D_LSR s Then the processing node is defined as belonging to a dense data group, otherwise the processing node is defined as belonging to a sparse data group;

[0070] Calculate the execution effectiveness Ex_Eff of the kth screening execution unit in the efficient processing feature matrix HTri(i,k) k , Ex_Eff k =|P i,k -P i-1,k | / P i,k , when the value of i-1 is 0, then Ex_Eff k The value is 1;

[0071] When a processing node belongs to a sparse data group, it is considered that the data volume is small and the mutual influence of the cooperation efficiency between the processing nodes is small. The distribution efficiency corresponding to the i-th processing node is calculated. ; Where SUM(AVERatio) represents the sum of the coefficients to be distributed of all processing nodes, and the accumulation variable of the accumulation function is k;

[0072] When a processing node belongs to a dense data group, it is considered that the data volume is large and the coordination efficiency between the processing nodes has a great impact on each other. The distribution efficiency corresponding to the i-th processing node is calculated. .

[0073] The efficient processing feature matrix HTri(i, k) is a matrix obtained by intercepting the corresponding columns of each screening execution unit in the two-dimensional processing feature matrix Tri(i, j); q1 , R q2 , and R qk are the proportions of data to be distributed under the first, second, and kth screening execution units under the dense processing node, R s1 , R s2 , and R sk are the proportions to be distributed under the first, second, and kth screening execution units under the sparse processing node, respectively;

[0074] Dense processing nodes and sparse processing nodes belong to the dense data group and sparse data group respectively by default;

[0075] Where P i-1,k is the execution delay of the kth screening execution unit under the i-1th processing node after HTri(i, k) is arranged in ascending order according to the coefficients to be distributed;

[0076] Further, in step S400, the method of calculating the imbalanced effectiveness weight in combination with each distribution efficiency obtained in the batch task processing scenario and sending it to the scheduling node is: sorting the obtained distribution efficiency in descending order to obtain the descending sequence number DEC_RATING corresponding to each processing node; presetting a time period TL, TL∈[1,3] hours; recording the average distribution efficiency of all processing nodes in the TL period as the evaluation factor; taking any processing node as the current processing node, if the distribution efficiency under the current execution unit is greater than the evaluation factor, the current execution unit is defined as a type of utility unit, and the imbalanced effectiveness weight IEWt is calculated: ; k1 is the cumulative variable, n_k1 is the number of utility units of the current processing node in the TL period, Di_Eff k1 and DEC_RATING k1They represent the distribution efficiency and descending sequence number of the k1th first-class utility unit respectively, and exp() is an exponential function with the natural constant e as the base; the imbalanced efficiency weights of each processing node are merged into a sequence and sent to the scheduling node.

[0077] The more the imbalanced effective state weight value deviates from the average value of the sequence, the greater the probability that the balancing performance collapse will occur during the load balancing control of its scheduling node. The corresponding task weight in the load balancing control will be increased or decreased. If the imbalanced effective state weight value is much larger than the average value, the task weight will be decreased. If the imbalanced effective state weight value is much smaller than the average value, the task weight will be increased. The increase or decrease ratio needs to be defined in combination with the actual load balancing control algorithm.

[0078] An embodiment of the present invention provides a batch task processing system based on traffic distribution, such as Figure 2 Shown is a structural diagram of a batch task processing system based on traffic distribution of the present invention. A batch task processing system based on traffic distribution of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned batch task processing method embodiment based on traffic distribution are implemented.

[0079] The system comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to run in the following units of the system:

[0080] The scenario construction unit is used to initialize the batch task processing scenario, which includes the load balancing operator;

[0081] A data acquisition unit, used to obtain a processing feature group of each processing node from a batch task processing scenario;

[0082] A distribution efficiency calculation unit, used to calculate the distribution efficiency of each processing node according to the processing feature group;

[0083] The processing node control unit is used to calculate the imbalanced efficiency weight based on the various distribution efficiencies obtained in the batch task processing scenario and send it to the scheduling node.

[0084] The batch task processing system based on traffic distribution can be run in computing devices such as desktop computers, laptops, PDAs and cloud servers. The batch task processing system based on traffic distribution can run systems including, but not limited to, processors and memories. Those skilled in the art will appreciate that the example is merely an example of a batch task processing system based on traffic distribution, and does not constitute a limitation on a batch task processing system based on traffic distribution. It can include more or fewer components than the example, or a combination of certain components, or different components. For example, the batch task processing system based on traffic distribution can also include input and output devices, network access devices, buses, etc.

[0085] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the batch task processing system based on traffic distribution, and uses various interfaces and lines to connect the various parts of the entire batch task processing system based on traffic distribution.

[0086] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the batch task processing system based on traffic distribution by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0087] Although the description of the present invention has been quite detailed and has been described in particular with respect to several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, so as to effectively cover the intended scope of the present invention. In addition, the present invention is described above with the embodiments foreseeable by the inventors, and its purpose is to provide a useful description, and those non-substantial changes to the present invention that are not currently foreseen may still represent equivalent changes of the present invention.

Claims

1. A batch task processing method based on traffic distribution, characterized in that: The method comprises the following steps: S100, initializing the batch task processing scenario, which includes the load balancing operator; S200, obtaining a processing feature group of each processing node from a batch task processing scenario; S300, calculating the distribution efficiency of each processing node according to the processing feature group; S400, calculating the imbalanced efficiency weight based on each distribution efficiency obtained in the batch task processing scenario and sending it to the scheduling node; In step S300, the method for calculating the distribution efficiency of each processing node according to the processing feature group is as follows: ROAM is the monitoring period, ROAM∈[3,10] hours; suppose that n execution units are input in the current ROAM period, there are m processing nodes, and then an m×n-order two-dimensional processing feature matrix Tri(i, j) is obtained; the ratio of the input data to be distributed by the jth execution unit of the i-th processing node is Ratio(i, j), abbreviated as R i,j , and the matrix formed by them is ∑R i,j ,∑R i,j The largest element in is denoted as RMX; the execution delay of the jth execution unit of the i-th processing node is Pcss_DEP(i,j), abbreviated as P i,j , and the matrix formed by them is ∑P i,j， ∑P i,j The largest element in is recorded as PMX, so the element corresponding to the two-dimensional processing feature matrix Tri(i,j) is (P i,j , R i,j ); ∑P i,j The elements in are sorted from small to large, and the upper quartile is recorded as the high extension value Pc(75%), the lower quartile is recorded as the low extension value Pc(25%), and the value between Pc(25%) and Pc(75%) is recorded as the medium extension value; If P i,j <Pc(25%), the distribution efficiency is less affected by the execution delay and is considered to be only related to the proportion to be distributed Ratio(i,j). It is known that for the jth processing unit, ,Data processing was performed using CLR central logarithmic transformation, ,R j The geometric mean of j , logarithmic distribution ratio ; Calculate the sub-distribution efficiency of the i-th processing node corresponding to the j-th execution unit ; If Pc(25%)≤P i,j ≤Pc(75%), the distribution efficiency is affected by both execution delay and distribution efficiency. First, unify the dimensions of the two and normalize them. , ; Calculate the sub-distribution efficiency of the i-th processing node corresponding to the j-th execution unit ; If P i,j >Pc(75%), the distribution efficiency is greatly affected by the execution delay, and is less affected by the proportion of distribution to be distributed Ratio(i,j). From the Markov property of the execution feedback time, it can be seen that its execution delay is affected by the execution delay of the previous execution unit P i,j-1 The influence of the sub-distribution efficiency of the i-th processing node on the j-th execution unit ; Calculate the distribution efficiency Di_Eff = mean obtained by the i-th processing node ROAM {Di_Eff i }.

2. A batch task processing method based on traffic distribution according to claim 1, characterized in that: In step S100, a batch task processing scenario is initialized, wherein the method including a load balancing operator is as follows: a distributed system is built in the batch task processing scenario, wherein the distributed system is composed of a number of servers; the server responsible for batch task processing in the distributed system is used as a processing node, wherein the server for batch task processing is a computing node that executes batch processing jobs; the server responsible for managing and allocating tasks in the distributed system is used as a scheduling node, wherein the scheduling node is preset with a load balancing operator.

3. A batch task processing method based on traffic distribution according to claim 1, characterized in that: In step S200, the method for obtaining the processing feature group of each processing node from the batch task processing scenario is as follows: define any batch task being completely executed as an execution unit; the batch task includes batch processing data, the batch processing data is divided into a plurality of sub-processing data by a load balancing operator, and each sub-processing data has a one-to-one correspondence with a processing node; the ratio of the data size of the sub-processing data obtained by a processing node to the data size of the batch processing data is the ratio to be distributed; The time spent on executing the batch job in the processing node is recorded as the execution feedback time; Get the minimum execution feedback time of each processing node under an execution unit and record it as the delay base. The difference between the execution feedback time and the delay base is recorded as the delay distance DEP_S of the execution unit. Then the execution delay degree Pcss_DEP is: Pcss_DEP=e DEP_S ; The binary group consisting of the proportion to be distributed and the execution delay corresponding to any execution unit is recorded as a processing feature group.

4. The method for batch task processing based on traffic distribution according to claim 1, characterized in that: In step S300, the method of calculating the distribution efficiency of each processing node according to the processing feature group can be replaced by: for the j-th execution unit, record its indicative function , if the execution unit satisfies , then it is defined as a screening execution unit, and the jth execution unit is eliminated; the execution unit obtained after the elimination operation is recorded as a screening execution unit, and its number is recorded as k; the corresponding column of the screening execution unit is intercepted from Tri (i, j) to obtain an m×k-order two-dimensional efficient processing feature matrix HTri (i, k); For the i-th processing node in the efficient processing feature matrix, the average of the proportions to be distributed corresponding to the k screening execution units is the coefficient to be distributed AVERatio i ; Arrange HTri(i, k) in ascending order according to the coefficients to be distributed, and record its maximum value as the dense coefficient to be distributed MX_AVERatio, and its corresponding node as the dense processing node q; its minimum value is the sparse coefficient to be distributed MN_AVERatio, and its corresponding node is the sparse processing node s. The processing nodes other than the dense processing nodes and the sparse processing nodes are recorded as interval processing nodes; The distribution feature vectors corresponding to the proportion of dense processing nodes q and sparse processing nodes s to be distributed are LSR q =(R q1 , R q2 ,…,R qk ),LSR s =(R s1 , R s2 ,…,R sk ), Any interval processing node and LSR q and LSR s The Euclidean distance of the Euclidean distance is denoted as D_LSR q and D_LSR s , if D_LSR q ≤D_LSR s Then the processing node is defined as belonging to a dense data group, otherwise the processing node is defined as belonging to a sparse data group; Calculate the execution effectiveness Ex_Eff of the kth screening execution unit in the efficient processing feature matrix HTri(i,k) k , Ex_Eff k =|P i,k -P i-1,k | / P i,k , when the value of i-1 is 0, then Ex_Eff k The value is 1; When a processing node belongs to a sparse data group, it is considered that the amount of data is small and the mutual influence of the cooperation efficiency between the processing nodes is small. The distribution efficiency corresponding to the i-th processing node is calculated. ; Where SUM(AVERatio) represents the sum of the coefficients to be distributed of all processing nodes, and the accumulation variable of the accumulation function is k; When a processing node belongs to a dense data group, it is considered that the data volume is large and the coordination efficiency between the processing nodes has a great impact on each other. The distribution efficiency corresponding to the i-th processing node is calculated. .

5. The method for batch task processing based on traffic distribution according to claim 1, characterized in that: In step S400, the method of calculating the imbalanced effectiveness weight in combination with each distribution efficiency obtained in the batch task processing scenario and sending it to the scheduling node is: sorting the obtained distribution efficiency in descending order to obtain the descending sequence number DEC_RATING corresponding to each processing node; presetting a time period TL, TL∈[1,3] hours; recording the average distribution efficiency of all processing nodes in the TL period as the evaluation factor; taking any processing node as the current processing node, if the distribution efficiency under the current execution unit is greater than the evaluation factor, the current execution unit is defined as a type of utility unit, and calculating the imbalanced effectiveness weight IEWt: ; k1 is the cumulative variable, n_k1 is the number of utility units of the current processing node in the TL period, Di_Eff k1 and DEC_RATING k1 They represent the distribution efficiency and descending sequence number of the k1th first-class utility unit respectively, and exp() is an exponential function with the natural constant e as the base; the imbalanced efficiency weights of each processing node are merged into a sequence and sent to the scheduling node.

6. A batch task processing system based on traffic distribution, characterized in that: The system includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a batch task processing method based on traffic distribution described in any one of claims 1 to 5 are implemented. The system runs on a desktop computer, a laptop computer, a PDA, and a computing device in a cloud data center.

Images