Scheduling algorithm for distributed computing under limited storage and computing resources

Abstract

The invention belongs to the technical field of distributed computing, discloses a scheduling strategy method for distributed computing under limited storage and computing resources, and optimizes the performance of task scheduling without changing the architecture of a whole application system. And the utilization rate of the application system, the web server and the database server is improved. Based on the architecture of a traditional application system, the execution efficiency of the whole timed task can be improved on the premise that the architecture does not need to be changed at all. When the timed task is executed, the application server is not singly utilized, but the application server and the database server are fully utilized. The invention aims to solve the problem of task scheduling performance by taking a traditional relational database as a data source; the method can be suitable for the architecture of the traditional application, and the performance of task scheduling can be optimized without any architecture change.

Description

Technical Field

[0001] The present invention belongs to the field of distributed computing technology, and in particular, relates to a scheduling strategy method and system for distributed computing under limited storage and computing resources. Background Art

[0002] There are many batch jobs in the applications in the production environment of many industries, which are used to organize, summarize, analyze and generate data reports for daily data. However, due to the planning of the company's system and the selection of technology, many domestic companies' systems still use traditional relational databases for data storage, and the resources of database servers and application servers are limited. After years of operation and personnel problems, it is impossible to change the data storage to or add new big data technologies such as hbase, hive, and hadoop. Moreover, although the above technologies can improve the efficiency of batch processing, they cannot guarantee the acid of transactions, so relational databases cannot be eliminated. After introducing new components, it is necessary to consider the synchronization and data consistency between relational databases and big data components and many difficulties. Therefore, in this case without adding any hardware resources, it is necessary to find a reasonable strategy to ensure the load balancing and execution time of the cluster in a multi-task scenario.

[0003] Currently, since the transmission of data from the database to the application server and the transmission of data from the application server to the database will generate a large amount of network IO, it is necessary to find a strategy to improve data locality. The system can effectively alleviate the communication volume of the network in the cluster and reduce the occurrence of network congestion by making full use of the locality of data.

[0004] In the current execution mode, when the application server is processing data, the database server is completely idle, resulting in a large amount of resource waste and low resource utilization. Therefore, a reasonable strategy is found to send some computing programs to the database server for task execution, which can not only reduce network IO and improve data locality, but also improve the resource utilization of the entire cluster.

[0005] Improve data locality through resource prefetching for optimization. "Research and Optimization of Locality in Hadoop Job Scheduling_Chen Ruofei" proposes a resource prefetching-based approach to balance locality and load balancing. When server F1 that meets the locality of task T1 is executing other tasks T2, and the remaining time of task T2 is greater than the transmission time of the data required by task T1, task T1 will be assigned to other server F2, and the data will be transferred from F1 to F2, and then task T2 will be executed on F2. The core idea of ​​this strategy is to give priority to satisfying the data locality of the task. If it cannot be satisfied, the data of non-local tasks will be migrated while the task is calculated, which not only ensures load balancing, but also guarantees the locality of each task as much as possible.

[0006] At present, there have been many studies on task scheduling. Whether it is improving data locality, reducing IO, or optimizing algorithms, the research direction has effectively improved the efficiency of task execution, reduced energy consumption, and improved the load balance of the cluster. However, in the current research on scheduling algorithms, such as the optimization of various artificial intelligence algorithms (particle swarm algorithm, simulated annealing algorithm, genetic algorithm, etc.), the scheduling scenarios adopted, whether heterogeneous or homogeneous, are all in the case of many-to-one tasks and execution servers, that is, one task involves one server, and one server can execute multiple tasks. In the research on improving data locality, many optimization methods have been proposed for the scenario where one task involves multiple servers, such as calculating the data size corresponding to the key generated by the Hadoop MapReduce model in the map phase to allocate tasks to the reduce phase, which improves the efficiency of the entire task execution and load balance. However, most of these studies are based on the Hadoop big data computing framework. If the data cannot be migrated to the HDFS file system in the actual production project, it will be difficult to use these scheduling strategies.

[0007] Through the above analysis, the problems and defects of the prior art are as follows:

[0008] (1) The existing technology cannot solve the problem of optimizing the performance of task scheduling models that use traditional relational databases as data sources for data storage.

[0009] (2) Existing technologies are unable to optimize the architecture of most traditional application systems. Summary of the invention

[0010] In view of the problems existing in the prior art, the present invention provides a scheduling strategy method for distributed computing under limited storage and computing resources.

[0011] The present invention is implemented in this way: a scheduling strategy method for distributed computing under limited storage and computing resources includes:

[0012] Step 1, generate a task matrix T;

[0013] Data volume -> rdc, computational volume -> rcc, required memory size -> rnm, output data volume -> rodc;

[0014] Step 2, generate an application server resource matrix Y;

[0015] Number of cpu cores -> acn, processing rate of each core -> acv, memory size -> an, network transmission rate -> nv;

[0016] Step 3, database server resource matrix D;

[0017] Network transmission rate -> nv, number of cpu cores -> dpcn, processing rate of each core -> dbcv, memory size - dbn, hard disk read rate - dbdv;

[0018] Step 4, classify the type of the task. If the rdc data volume of the task < rcc computational volume, define the task as a Cpu-intensive task; otherwise, define it as an IO-intensive task;

[0019] Step 5, determine the servers that can be allocated currently. For cpu-intensive tasks, first judge whether the application server can execute in parallel under the current situation;

[0020] Calculate the parallelism of the application server Pac = acn * (rcc / acv + (rdc + rodc) / nv) / ((rdc + rodc) / nv), that is, the number of cores * (cpu time + IO time) / IO time. For IO-intensive tasks, first judge whether it can execute concurrently on the database server;

[0021] Calculate the parallelism PaO = acn * (rcc / acv + (rdc + rodc) / dbdv) / ((rdc + rodc) / dbdv),

[0022] If the calculated parallelism >= the number of tasks allocated to this server tn + 1, then this server can be regarded as a candidate server and added to the task candidate server list tsl;

[0023] Step 6, if tsl is not empty, calculate which server has the shortest execution time to allocate the task. At this time, take the longest task execution duration as the total duration Ect of the server execution;

[0024] Step 7, if tsl is empty, calculate the execution duration of the task on all servers and allocate the task to this server first;

[0025] Step 8, generate allocation plans for the tasks, ts1, ts2, ts3, and prioritize the allocation plans according to steps 5, 6, and 7, and allocate them in order of priority;

[0026] Step 9, obtain the task from the task matrix and repeat steps 4, 5, 6, 7, and 8 until a solution is obtained. If the set time threshold maxT is met, the solution is output. If not, the above steps are repeated.

[0027] Furthermore, the task execution duration includes the time consumption calculation of a single task and the overall time consumption calculation.

[0028] Further, the time consumption of the single task is calculated as follows:

[0029] When a task is processed, the required data is not pulled from the hard disk to the memory at once, but pulled and processed in batches. The amount of data in each batch is the memory size required for the task.

[0030] The number of pulls is the (data volume / required memory size) of the task. The time consumed in each batch is (cpu computing rate*computation volume / number of pulls)+data volume*IO rate / number of pulls. According to the formula of pipeline instructions, the calculation formula for the time consumption of a single task is:

[0031] (number of pulls - 1) * max ((cpu computing rate * computing amount / number of pulls), data volume * IO rate / number of pulls) + (cpu computing rate * computing amount / number of pulls) + data volume * IO rate / number of pulls, that is, the total time to complete a task plus (number of times - 1) * the time of the longest instruction, where the IO rate is the hard disk read rate of the database server when calculation moves to data, and is the network transmission rate when data moves to calculation.

[0032] Further, the time consumption of the single task is calculated as follows:

[0033] The time consumed by a single server is the longest execution time of the task on that server.

[0034] The longest time of the cluster is the longest single server time.

[0035] Another object of the present invention is to provide a scheduling strategy system for distributed computing under limited storage and computing resources, including:

[0036] A generation module, used to generate a task matrix T; data volume -> rdc, computational volume -> rcc, required memory size -> rnm, output data volume -> rodc; generate an application server resource matrix Y; number of cpu cores -> acn, processing rate of each core -> acv, memory size -> an, network transmission rate -> nv;

[0037] A database module, used for a database server resource matrix D; network transmission rate -> nv, number of cpu cores -> dpcn, processing rate of each core -> dbcv, memory size - dbn, hard disk read rate - dbdv;

[0038] A partitioning module, used to partition the type of this task. If the rdc data volume of the task < rcc computational volume, then define this task as a Cpu-intensive task, otherwise define it as an IO-intensive task;

[0039] A judgment module, used to judge the servers that can be allocated currently. For Cpu-intensive tasks, first judge whether the application server can execute in parallel under the current situation; calculate the parallelism Pac of the application server = acn * (rcc / acv + (rdc + rodc) / nv) / ((rdc + rodc) / nv), that is, number of cores * (cpu time + IO time) / IO time. For IO-intensive tasks, first judge whether it can execute concurrently on the database server;

[0040] Calculate the parallelism PaO = acn * (rcc / acv + (rdc + rodc) / dbdv) / ((rdc + rodc) / dbdv),

[0041] If the calculated parallelism >= the number of tasks tn + 1 allocated to this server, then this server can be regarded as a candidate server and added to the task candidate server list tsl;

[0042] If tsl is not empty, then calculate which server to allocate the task to with the shortest execution time. At this time, use the longest task execution duration as the total execution duration Ect of the server;

[0043] If tsl is empty, then calculate the execution duration of this task on all servers and allocate the task to this server first;

[0044] An allocation module, used to generate allocation schemes ts1, ts2, ts3 for the task and sort the allocation schemes by priority and allocate them in the order of priority;

[0045] An output module, used to obtain the task repetition steps from the task matrix until a scheme is obtained. If the set time threshold maxT is met, then output the scheme. If not, repeat the above steps.

[0046] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the scheduling strategy method for distributed computing under limited storage and computing resources.

[0047] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the scheduling strategy method for distributed computing under limited storage and computing resources.

[0048] Another object of the present invention is to provide an information data processing terminal, which is used to implement the scheduling strategy system for distributed computing under limited storage and computing resources.

[0049] In combination with the above technical solutions and the technical problems solved, please analyze the advantages and positive effects of the technical solutions to be protected by the present invention from the following aspects:

[0050] First, in view of the technical problems existing in the above-mentioned prior art, some creative technical effects are brought about after solving the problems. The specific description is as follows:

[0051] The performance of task scheduling is optimized without changing the architecture of the entire application system.

[0052] Improved the utilization of application systems, web servers and database servers.

[0053] Based on the architecture of traditional application systems, the efficiency of the entire scheduled task execution can be improved without making any changes to the architecture.

[0054] When scheduled tasks are executed, the application server is no longer used alone, but the application server and database server are fully utilized.

[0055] The prior art cannot solve the problem of optimizing the performance of a task scheduling model that uses a traditional relational database as a data source for data storage. The present invention aims to solve the problem of optimizing the performance of task scheduling that uses a traditional relational database as a data source.

[0056] The prior art cannot optimize the architecture of most traditional application systems, that is, the architecture in which the web server is used as the application layer and the database server is used as the data storage layer. The present invention can be applied to the architecture of such traditional applications and can optimize the performance of task scheduling without making any architectural changes.

[0057] The scheduling strategy of distributed computing under limited storage and computing resources is an important research field. In actual production and scientific research, the execution efficiency of scheduled tasks directly affects the production efficiency and scientific research results. The focus of the present invention is on the selection of strategies for the mutual movement of calculations and data. Reasonable movement strategies can improve the execution efficiency and performance of scheduled tasks, thereby helping to complete tasks more accurately and quickly. The present invention studies the strategies for the mutual movement of calculations and data on the efficiency of scheduled tasks.

[0058] The present invention calculates the impact of the mutual movement of calculation and data on the efficiency of the scheduled task, including network bandwidth, hard disk read and write rate, CPU execution speed, memory size, and the characteristics of the task itself (IO intensive and CPU intensive). When the task is CPU intensive, data should be used to move to calculation first, and when the task is IO intensive, calculation should be moved to data first. The optimization model of the scheduled task uses a mathematical model to describe the execution process of the scheduled task. The data movement model describes the data movement process, and the calculation model represents the calculation process. The present invention analyzes and evaluates the impact of calculation and data transmission efficiency on the efficiency of the scheduled task. Transmitting data is a key part of task execution, and transmitting data on a computer network can significantly reduce the task execution time, but there will also be some additional costs during the data transmission process, such as network transmission time, network transmission cost, etc. These costs must also be taken into account. Not only that, storing data locally instead of executing it on a remote computer can also significantly improve execution efficiency.

[0059] The present invention proposes some strategies for directly evaluating the execution efficiency of scheduled tasks, such as bandwidth-aware scheduling and real-time optimization algorithms. The basic idea is to select appropriate load and data movement schemes according to different task types and environments to improve the execution efficiency of tasks. For example, bandwidth-aware scheduling can improve the execution efficiency of tasks when using appropriate network communication bandwidth and performing task allocation. The real-time optimization algorithm monitors, analyzes and optimizes tasks during task execution to maximize the execution efficiency of tasks. The present invention summarizes its research results and conclusions.

[0060] The selection of the efficiency of scheduled tasks based on the movement of computation and data is a comprehensive and complex process that requires the integration and analysis of different factors. When selecting, the impact of the transmission of data and the computation process on the execution efficiency is equally important. The optimization of these factors can improve the execution efficiency and performance of the task. In a specific environment, the use of appropriate load and data movement strategies can further improve the execution efficiency and performance of the task.

[0061] The present invention studies the effect of computation and data movement on the efficiency of scheduled tasks, carefully analyzes the factors that affect execution efficiency and performance, and proposes some appropriate strategies, which have important application value in actual production and scientific research.

[0062] Second, whether the technical solution of the present invention solves the technical problems that people have been eager to solve but have never been able to solve successfully:

[0063] At present, many technical solutions to improve the efficiency of scheduled task processing require a large amount of server resources as support. This solution can effectively optimize the efficiency of scheduled task processing of the system with the existing limited resources.

[0064] Currently, many technical solutions for scheduled tasks need to rely on new technical components including but not limited to hadoop, spark, etc. For the transformation of traditional applications, the scheduling of the entire system needs to be migrated to new technical components, which is costly, high-risk and low-return. This solution can improve the execution efficiency of scheduled tasks by optimizing the scheduling algorithm of scheduled tasks under the existing architecture.

[0065] Traditional tax systems usually process a large amount of data from upstream systems at night, and relevant business personnel will use it as soon as they go to work the next day. However, due to the large amount of work and data, the work is often not completed during the day, causing business personnel to wait for the work to be completed before they can proceed to the next step. After the optimization of this solution, the execution efficiency of the entire system work has been improved, and all scheduled tasks can be guaranteed to be completed before going to work every day, solving the pain point of business personnel having to wait for the work to be completed during working hours. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a flow chart of a scheduling strategy method for distributed computing under limited storage and computing resources provided by an embodiment of the present invention.

[0067] Figure 2 It is a structural block diagram of a scheduling strategy system for distributed computing under limited storage and computing resources provided by an embodiment of the present invention.

[0068] Figure 3 It is a flow chart of a scheduling algorithm for distributed computing under limited storage and computing resources provided by an embodiment of the present invention.

[0069] Figure 4 This is a diagram of the IO data volume and CPU computing volume of each task provided by an embodiment of the present invention.

[0070] Figure 5 This is a scheduled job scheduling distribution diagram provided by an embodiment of the present invention.

[0071] Figure 6It is the effect diagram of task efficiency improvement provided by the embodiments of the present invention.

[0072] Figure 7 It is the time comparison diagram (unit: h) of each task before and after optimization provided by the embodiments of the present invention. Specific embodiments

[0073] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

[0074] As Figure 1 shown, a scheduling strategy method for distributed computing under limited storage and computing resources provided by the embodiments of the present invention includes the following steps:

[0075] S101, generate a task matrix T;

[0076] Data volume -> rdc, computing volume -> rcc, required memory size -> rnm, output data volume -> rodc;

[0077] S102, generate an application server resource matrix Y;

[0078] Number of cpu cores -> acn, processing rate of each core -> acv, memory size -> an, network transmission rate -> nv;

[0079] S103, database server resource matrix D;

[0080] Network transmission rate -> nv, number of cpu cores -> dpcn, processing rate of each core -> dbcv, memory size - dbn, hard disk read rate - dbdv;

[0081] S104, classify the type of the task. If the rdc data volume of the task < rcc computing volume, define the task as a Cpu-intensive task; otherwise, define it as an IO-intensive task;

[0082] S105, determine the servers that can be allocated currently. For cpu-intensive tasks, first judge whether the application server can be executed in parallel under the current situation;

[0083] Calculate the parallelism Pac of the application server = acn * (rcc / acv + (rdc + rodc) / nv) / ((rdc + rodc) / nv), that is, the number of cores * (cpu time + IO time) / IO time. For IO-intensive tasks, first judge whether they can be executed concurrently on the database server;

[0084] The computing parallelism PaO = acn * (rcc / acv + (rdc + rodc) / dbdv) / ((rdc + rodc) / dbdv),

[0085] If the calculated parallelism >= the number of tasks tn + 1 assigned to the server, then the server can be regarded as a candidate server and added to the task candidate server list tsl;

[0086] S106, if tsl is not empty, calculate which server the task will be assigned to with the shortest execution time, and at this time, use the longest task execution duration as the total execution duration Ect of the server;

[0087] S107, if tsl is empty, calculate the execution duration of the task on all servers and assign the task to this server first;

[0088] S108, generate an allocation plan for the task, ts1, ts2, ts3, and sort the allocation plan according to the priorities in steps S105, S106, and S107, and allocate according to the priority order;

[0089] S109, obtain the steps of repeating S104, S105, S106, S107, and S108 from the task matrix until a plan is obtained. If the set time threshold maxT is satisfied, output the plan; if not, repeat the above steps.

[0090] The core of this method is to model distributed computing tasks and server resources as a task matrix \(T\) and resource matrices \(Y, D\). The task matrix \(T\) contains the key attributes of tasks, including data volume (\(rdc\)), computing volume (\(rcc\)), required memory size (\(rnm\)), and output data volume (\(rodc\)). The application server resource matrix \(Y\) and database server resource matrix \(D\) are used to describe the key resource parameters of the server, such as the number of CPU cores, processing rate per core, memory size, and network transmission rate. By mapping task requirements to resource capabilities, the matching degree between tasks and resources can be systematically analyzed.

[0091] The method divides tasks into CPU-intensive tasks and IO-intensive tasks by comparing the data volume (\(rdc\)) and computing volume (\(rcc\)) of tasks. If \(rdc < rcc\), the task is defined as a CPU-intensive task and is preferentially assigned to an application server with stronger computing performance; otherwise, it is defined as an IO-intensive task and is preferentially assigned to a database server more suitable for processing large data volumes. This type of division is the core of the task scheduling strategy and directly affects the allocation priority and execution efficiency of tasks.

[0092] For each task, select possible candidate servers according to its type. For CPU-intensive tasks, prioritize whether the application server can achieve parallel processing under the current resources and calculate the degree of parallelism \(Pac\). If \(Pac\) is greater than the current number of tasks (\(tn+1\)), add the server to the task candidate server list \(tsl\). Similarly, for IO-intensive tasks, calculate the degree of parallelism \(PaO\) of the database server. This step dynamically selects servers by calculating the degree of parallelism of resources to ensure optimal resource utilization.

[0093] When the candidate server list \(tsl\) is not empty, the server with the shortest task execution time is selected from it. At this time, by calculating the total execution time \(Ect\) of each candidate server, the solution with the shortest execution time of the longest task is selected to achieve task load balancing. If \(tsl\) is empty, the task will find a server with the shortest execution time among all servers for allocation, so as to ensure that the task can be executed smoothly.

[0094] The task scheduling system generates allocation plans through iterative optimization, including server priority sorting and task allocation order optimization. For each task, the system executes steps S104 to S108 in the candidate servers and allocation plans, and dynamically updates the allocation results. If the maximum execution time threshold of the task \(maxT\) is met, the current optimal plan is output; otherwise, the task scheduling process is re-iterated to further optimize the allocation strategy.

[0095] The entire scheduling strategy adjusts the task and resource matching strategy in real time by looping and iterating the unassigned tasks in the task matrix, making full use of the limited computing and storage resources of the system. Through candidate server screening, parallelism calculation, and optimization calculation of execution time, it ensures that task scheduling meets global performance indicators. At the same time, the system will dynamically adjust the allocation strategy according to the actual resource usage during task execution, improving the overall efficiency and task responsiveness of the distributed computing system.

[0096] The task execution duration provided by the embodiment of the present invention includes the time consumption calculation of a single task and the overall time consumption calculation.

[0097] The time consumption calculation of a single task provided by the embodiment of the present invention is as follows:

[0098] When a task is processed, the required data is not pulled from the hard disk to the memory at once, but pulled and processed in batches. The amount of data in each batch is the memory size required for the task.

[0099] The number of pulls is (data volume / required memory size) of the task. The time consumption within each batch is (cpu computing rate * amount of computation / number of pulls) + data volume * IO rate / number of pulls. According to the formula of pipeline instructions, the formula for the time consumption of a single task is:

[0100] (Number of pulls - 1) * max((cpu computing rate * amount of computation / number of pulls), data volume * IO rate / number of pulls) + (cpu computing rate * amount of computation / number of pulls) + data volume * IO rate / number of pulls, that is, the total time consumption to complete a task plus (number of times - 1) * the time consumption of the instruction with the longest time consumption. Among them, the IO rate is the hard disk read rate of the database server when calculating the data movement to the calculation, and the network transmission rate when the data moves to the calculation.

[0101] The time consumption calculation of a single task provided by the embodiment of the present invention:

[0102] The time consumption of a single server is the task with the longest execution time on this server.

[0103] The longest time consumption of the cluster is the single server with the longest time consumption.

[0104] As Figure 2 shown, a scheduling strategy system for distributed computing under limited storage and computing resources provided by the embodiment of the present invention includes:

[0105] A generation module, used to generate a task matrix T; data volume -> rdc, amount of computation -> rcc, required memory size -> rnm, output data volume -> rodc; generate an application server resource matrix Y; cpu core number -> acn, processing rate of each core -> acv, memory size -> an, network transmission rate -> nv;

[0106] A database module, used for the database server resource matrix D; network transmission rate -> nv, cpu core number -> dpcn, processing rate of each core -> dbcv, memory size - dbn, hard disk read rate - dbdv;

[0107] A partitioning module, used to partition the type of the task. If the rdc data volume of the task < rcc amount of computation, then define the task as a Cpu-intensive task, otherwise define it as an IO-intensive task;

[0108] The judgment module is used to judge the servers that can be currently allocated. For CPU-intensive tasks, it is prioritized to judge whether the application server can be executed in parallel under the current situation; the parallelism of the application server is calculated as Pac = acn*(rcc / acv+(rdc+rodc) / nv) / ((rdc+rodc) / nv), which is the number of cores*(cpu time+IO time) / IO time. For IO-intensive tasks, it is prioritized to judge whether concurrent execution can be performed on the database server;

[0109] Calculate the parallelism PaO = acn*(rcc / acv+(rdc+rodc) / dbdv) / ((rdc+rodc) / dbdv),

[0110] If the calculated parallelism is greater than or equal to the number of tasks assigned to the server tn+1, the server can be considered as a candidate server and added to the candidate server list tsl of the task;

[0111] If tsl is not empty, calculate which server has the shortest execution time when assigning the task to it. In this case, the longest task execution time is taken as the total execution time Ect of the server.

[0112] If tsl is empty, the execution time of the task on all servers is calculated, and the task is assigned to the server first;

[0113] The allocation module is used to generate allocation schemes for tasks, ts1, ts2, ts3, and prioritize the allocation schemes and allocate them in order of priority;

[0114] The output module is used to obtain tasks from the task matrix and repeat S10 until a solution is obtained. If the set time threshold maxT is met, the solution is output; if not, the above S10 is repeated.

[0115] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the scheduling strategy method for distributed computing under limited storage and computing resources.

[0116] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the scheduling strategy method for distributed computing under limited storage and computing resources.

[0117] Another object of the present invention is to provide an information data processing terminal, which is used to implement the scheduling strategy system for distributed computing under limited storage and computing resources.

[0118] The present invention is specifically implemented:

[0119] The complete technical solution provided by the present invention (inventive solution)

[0120] 1. Define the indicators

[0121] Task (amount of data, amount of computation, required memory size).

[0122] Application server (number of CPU cores, processing speed of each core, memory size, network transmission speed).

[0123] Database server (network transmission rate, number of CPU cores, processing speed of each core, memory size, hard disk read rate).

[0124] 2. Time calculation of a single task

[0125] 1. Generally speaking, when processing a task, the required data is not pulled from the hard disk to the memory at once, but pulled and processed in batches. The amount of data in each batch is the memory size required for the task.

[0126] 2. The number of pulls is the (data volume / required memory size) of the task. The time consumed in each batch is (cpu computing rate*computation volume / number of pulls) + data volume*IO rate / number of pulls. According to the formula of pipeline instructions, the time consumption of a single task is calculated as (number of pulls-1)*max((cpu computing rate*computation volume / number of pulls), data volume*IO rate / number of pulls)+(cpu computing rate*computation volume / number of pulls)+data volume*IO rate / number of pulls, that is, the total time consumption to complete a task plus (number of times-1)*the time consumption of the longest instruction, where the IO rate is the hard disk read rate of the database server when calculation moves to data, and is the network transmission rate when data moves to calculation.

[0127] 3. Overall time calculation

[0128] 1. The time consumed by a single server is the longest execution time of the task on that server.

[0129] 2. The longest time consumed by the cluster is the longest single server time consumed.

[0130] like Figure 3 As shown, 4. Algorithm implementation

[0131] S201, establishing two matrices, namely a task matrix of resources required for the task and a resource matrix of the server.

[0132] S202, assigning tasks one by one to each server, finding an assignment with the shortest total execution time and then assigning it to the server.

[0133] S203, two situations are considered during allocation: one is that the data is executed on the application server, which requires network IO from the database to the application server; the other is that the data is executed directly on the database, which does not require network IO.

[0134] S204, when allocating, the simulated annealing algorithm is used for inspiration to improve the performance and convergence speed of the algorithm, mainly using the idea of ​​using the evaluation function to calculate the increment and accepting new solutions, and taking the accepted new solutions as the priority for the next search.

[0135] 2.4 Engineering Examples Provided by the Invention

[0136] Ningbo Bank, full tax management system, night data processing, after using this solution, the overall execution time of all scheduled tasks has been significantly shortened, and the utilization rate of application servers and database servers has been significantly improved.

[0137] 1. Generate task matrix T (100M, 1G, 200M, 50M

[0138] 1G, 1G, 1G, 200M,

[0139] 20M, 10M, 15M, 40M,

[0140] 300M, 1.5G, 800M, 400M,

[0141] 210M, 400M, 300M, 40M,

[0142] 2G, 16G, 512M, 4G,

[0143] 500M, 8G, 2G, 800M,

[0144] 3G, 3G, 1G, 3G)

[0145] 2. Generate application server resource matrix Y

[0146] (4, 2.8GHZ, 8G, 100M / S4, 2.8GHZ, 8G, 100M / S).

[0147] 3. Generate database server resource matrix (100M / S, 8, 2.8GHZ, 16G, 256M / s).

[0148] 4. Get a task (100M, 1G, 200M, 50M) from the task matrix T and define its type as CPU intensive.

[0149] 5. The maximum parallelism of the application server is (1.05 / 0.05)*4=84. If 84 is greater than 1, it can be allocated to the server, and the total time is 1.65s.

[0150] 6. There are three ways to complete this task: t1Y1, t1Y2, and t1D1. Based on the result of step 5, search for t1Y1 first.

[0151] 7. Take out the second task t2 (1G, 1G, 1G, 200M) from type T and define its type as IO intensive.

[0152] 8. The maximum parallelism of the database server is 8, and the tasks can be assigned to this server with a total time of 5.05.

[0153] 9. This task also has three ways: t2Y1, t2Y2, and t2D1. Based on the result of step 8, search t2D1 first.

[0154] 10. Repeat steps 4, 5, 6, 7, 8, and 9 until a scheduling solution that meets the maximum duration threshold is obtained.

[0155] Scheduling algorithm classification and analysis:

[0156] 1. Optimize by reducing IO

[0157] At present, the execution process of batch processing tasks is completed in three steps: reading data, processing data, and storing results. Usually, data processing is performed by the program deployed on the application server. If the data to be processed is different from the server where the program is deployed, the application server is A, and the server where the required data is located is B, then when reading the data, server B needs to read the file content into the kernel space through DMA, and the CPU will copy the data in the kernel space into the user space. After the user program reads the file data from the kernel space, it will write the data into the write buffer of the socket connection established by A and B in the kernel space, and then

[0158] DMA copies the data in the write buffer into the network card and then transmits it to server A. Server A needs to use DMA to read the data from the network card of server A to the read buffer of the socket connection established by A and B in the kernel space of server A. The CPU then copies the data to the user space. Only then can the program on server A actually start processing the data.

[0159] In the above process, the data reading stage consumes a lot of resources and performance, so reducing IO has also become an important research direction for optimizing scheduled task scheduling.

[0160] 1.1 Reduce IO consumption through zero copy

[0161] Zero copy is an effective way to reduce network IO time consumption in network transmission. It mainly improves the efficiency of the entire data transmission by reducing the number of times data is copied in memory. There are currently two ways to implement zero copy. MMAP is mainly implemented through memory mapping. It points the user space directly to the kernel space. When the user process needs to obtain a file, DMA first reads the data into the kernel space and then copies it to the user space. MMAP directly saves the copy from the kernel space to the user space. The kernel space corresponding to the file content can be directly copied to the kernel space corresponding to the socket connection. This saves one copy and improves the efficiency of the entire transmission. Send File marks the user space where the file is located. It does not need to be copied from the kernel space where the file is located to the kernel space where the socket is located. Instead, DMA directly copies the data from the kernel space where the file is located into the network card according to the mark, which is further improved in efficiency compared to MMAP.

[0162] The advantage of zero copy is that it requires less modification to the system. The entire system architecture layout does not need to be changed to achieve significant improvement. However, zero copy only increases the transmission speed, thereby reducing IO time. It does not fundamentally solve the performance consumption caused by data transmission between servers. Although the IO transmission speed becomes faster, the total amount of IO data does not decrease. In scenarios with large data volumes, the improvement effect is relatively limited.

[0163] 1.2 Optimizing by Improving Data Locality

[0164] Data locality means that when scheduling tasks, tasks are directly assigned to the server where the data is located for calculation. That is, if the data required by task T is on server B, then task T is executed on server B. This method can eliminate the performance and time consumed by network IO, thereby improving the execution efficiency of the entire scheduled task. However, if the data required by multiple tasks is on the same server, load imbalance is likely to occur. There are many related studies to balance the relationship between data locality and load balancing.

[0165] 1.2.1 Resource Prefetching

[0166] "Research and Optimization of Locality in Hadoop Job Scheduling_Chen Ruofei" proposes a resource prefetching-based approach to balance locality and load balancing. When server F1 that meets the locality of task T1 is executing other tasks T2, and the remaining time of task T2 is greater than the transmission time of the data required by task T1, task T1 will be assigned to other server F2, and the data will be transferred from F1 to F2, and then task T2 will be executed on F2. The core idea of ​​this strategy is to give priority to satisfying the data locality of the task. If it cannot be satisfied, the data of non-local tasks will be migrated while the task is calculated, which not only ensures load balancing, but also guarantees the locality of each task as much as possible.

[0167] 1.2.2 Search Algorithms through Artificial Intelligence

[0168] At present, there are many studies that combine artificial intelligence search algorithms to comprehensively consider factors such as data locality and load balancing to improve the efficiency of task execution. "Research and Implementation of Hadoop Job Scheduling Algorithms in Heterogeneous Environments_Tian Ye" uses a hybrid genetic cuckoo algorithm to comprehensively consider factors such as cpu, data locality, and load balancing in heterogeneous environments to perform task scheduling. "Research on Dynamic Job Scheduling Based on Hadoop Heterogeneous Clusters_Chen Jinglin" uses genetic algorithms to schedule tasks in heterogeneous environments through CPU, memory, locality, and load balancing. "Research on Load-Balanced Hadoop Platform Scheduling Algorithms_Yang Mingchen" uses a genetic algorithm with a competition strategy to improve the load balancing of task scheduling.

[0169] 1.2.3 Optimization according to transmission distance

[0170] "Research on Data Locality Load Balancing Strategy under MapReduce Model_Chen Yi" proposes a data transmission strategy when the task cannot meet data locality. The article proposes that if the task cannot meet data locality, it should be prioritized to ensure that the task and data are on the server of the same rack, because the transmission speed of servers in the same rack is much different than that of servers in different racks. The core idea of ​​the article is to prioritize data locality during scheduling. If the server needs to execute too many tasks and cannot guarantee data locality, it is necessary to consider the data transmission distance as short as possible to reduce IO consumption.

[0171] 2. Scheduling algorithm optimization

[0172] There are many ways to optimize the scheduling algorithm, such as optimizing formulas, parameters, and steps. You can also combine multiple algorithms organically and use the advantages of each algorithm to make the algorithm more adaptable to the required scheduling scenario. In the relevant research on algorithm optimization, it can be found that the use of various artificial intelligence algorithms, such as particle swarm optimization and genetic algorithm, can take into account multiple dimensions at the same time, and can comprehensively consider multiple aspects such as CPU, memory, network, and data scale to find the optimal solution for task scheduling. From the current research, the optimization of scheduling algorithms is mainly divided into the following aspects.

[0173] 2.1 Convergence Optimization

[0174] "Research on Cloud Task Scheduling Strategy Based on Improved Simulated Annealing Algorithm_Sun Chenhui" optimizes the cooling parameters to improve the convergence speed of the algorithm while ensuring the accuracy of the algorithm. "Cloud Service Task Scheduling Algorithm Based on Cauchy Mutation Particle Swarm_Bi Yang" improves the convergence accuracy by improving the initial population, adjusting the inertia weight, introducing Cauchy mutation particles and changing the speed formula. "Cloud Data Center Energy Consumption Optimization Task Scheduling Strategy Based on Improved Particle Swarm Algorithm_Liu Chenwei" improves the convergence accuracy of the particle swarm algorithm by optimizing the fitness function and using chaos mapping to generate the initial cluster. "Application of Improved Particle Swarm Optimization Algorithm in Cloud Computing Task Scheduling_Wang Ting" combines simulated annealing fusion algorithm and hunger game to improve the convergence speed of the particle swarm algorithm.

[0175] 2.2 Local Optimal Solution Optimization

[0176] "Research on Cloud Task Scheduling Strategy Based on Improved Simulated Annealing Algorithm_Sun Chenhui" mentioned how to solve the problem of simulated annealing algorithm falling into local optimal solution by optimizing temperature update function. The article mentioned memory annealing algorithm and tempering annealing method.

[0177] To jump out of the local optimal solution, and proposed an optimization method for oscillating cooling parameters to improve the global search capability. When combined with the immune algorithm, a method of retaining inferior solutions with a certain probability is proposed to avoid the problem of traditional immune algorithms falling into local optimal solutions. "Cloud Computing Task Scheduling Based on Exchange Improved Particle Swarm Algorithm_Chen Junren" ensures the diversity of particles by introducing adaptive adjustment probability, reducing the probability of the algorithm falling into the local optimal solution. "Cloud Computing Task Scheduling Method Based on Improved Genetic Algorithm_Wang Hongjie" ensures the richness of the initial population by calculating the chromosome distance, thereby increasing the possibility of the genetic algorithm obtaining the global optimal solution.

[0178] 2.3 Optimizing task execution efficiency

[0179] "Cloud Computing Task Scheduling Strategy Based on Improved Differential Evolution Algorithm_Lin Tao" proposed an improved differential evolution algorithm based on Levy distribution adaptive differential evolution to improve the energy consumption and completion time of tasks. "HADOOP Adaptive Task Scheduling Algorithm Based on Node Set Computing Power Difference_Zhu Jie" proposed an adaptive algorithm to improve the execution efficiency of tasks by calculating the difference in node capabilities and the identification of lagging tasks and the dynamic changes in node performance during operation.

[0180] 3. Summary

[0181] At present, there have been many studies on task scheduling. Whether it is improving data locality, reducing IO, or optimizing algorithms, the research direction has effectively improved the efficiency of task execution, reduced energy consumption, and improved the load balance of the cluster. However, in the current research on scheduling algorithms, such as the optimization of various artificial intelligence algorithms (particle swarm algorithm, simulated annealing algorithm, genetic algorithm, etc.), the scheduling scenarios adopted, whether heterogeneous or homogeneous, are all in the case of many-to-one tasks and execution servers, that is, one task involves one server, and one server can execute multiple tasks. In the research on improving data locality, many optimization methods have been proposed for the scenario where one task involves multiple servers, such as calculating the data size corresponding to the key generated by the map phase of the Hadoop MapReduce model to allocate tasks in the reduce phase, which improves the efficiency of the entire task execution and load balance. However, most of these studies are based on the Hadoop big data computing framework. If the data cannot be migrated to the HDFS file system in the actual production project, it will be difficult to use these scheduling strategies. At present, most production projects still use traditional relational databases as data storage, with MySQL and Oracle being the most typical. No matter what scheduling framework is used for the scheduled tasks in these projects, the process of task execution is to establish a TCP connection between the application server and the database server. The application server uses SQL statements to let the database server transfer the data required for the task to the application server. The application server executes the processing logic of the corresponding task, and then writes it back to the database through DML statements. In this process, a task will involve two servers, one is the application server and the other is the database server. The application server is responsible for executing the program instructions of the scheduled task, and the database server is responsible for providing the data required for the task. If the data required for the task is large, then the entire task will consume a lot of time on the network IO. In this case, you can consider uploading the calculation program to the database server and directly executing the program by the database server, that is, the calculation moves to the data. However, if all tasks use the calculation to move to the data, it will inevitably cause the idle resource utilization of the application server to be low. Therefore, the focus of task scheduling under limited resources is to study how to solve the balance problem between the movement of calculation and data.

[0182] 1. Specific application fields or related products of the present invention.

[0183] Ningbo Bank Full Tax Management System

[0184] 2. Relevant evidence of the technical effects obtained by the embodiments of the present invention.

[0185] After optimizing the scheduling algorithm, the various tasks of Ningbo Bank's full tax management system were more reasonably allocated to the server for execution.

[0186] (1) Customer information synchronization task. This task needs to parse the full amount of customer data files pushed by the upstream every day. There are 90 million data items that need to be processed every day. Before algorithm optimization, the entire task execution process is that the application server pulls data from the database server to the application server through network IO. After simple processing, the application server writes the data back to the database server through network IO. The two network IOs result in a very long time consumption. After algorithm optimization, the execution of the entire task is directly assigned to the database server, which effectively reduces the IO consumption of the task and improves the utilization rate of the server resources of the entire system. This task is the most effective when using this solution. The entire task has been reduced from 1 hour to 15 minutes.

[0187] (2) This solution effectively allocates tasks that require a large amount of IO to the server where the data is located, and allocates tasks that require a large amount of computing but relatively small amounts of data to the application server, thereby improving the execution efficiency of the entire scheduled task and the utilization of resources.

[0188] (3) Tasks such as synchronization of invoice flow, account balance sheet, and sales item detail sheet, which have large data volumes but relatively simple calculation logic, were precisely assigned to the database server, increasing efficiency by 2 times, 3 times, and 2.7 times, respectively.

[0189] (4) Jobs with relatively small data volumes and relatively complex calculation logic, such as the VAT analysis table generation job, stamp tax analysis table generation job, and stamp tax declaration form generation job, were assigned to the application server, increasing efficiency by 50%, 62%, and 58%, respectively.

[0190] After algorithm optimization, the overall execution efficiency of tasks has increased by about 2 times, solving the pain point that scheduled tasks have not been completed by daytime work. This allows the entire system to be completed before work, and relevant business personnel can use them immediately without having to wait for scheduled tasks to be completed during work.

[0191] Figure 4 The amount of IO data and CPU computing power of each task;

[0192] Figure 5 Scheduled job scheduling distribution diagram;

[0193] Figure 6 Mission efficiency improvement effect diagram;

[0194] Figure 7 Time comparison chart before and after optimization of each task (unit: h).

[0195] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0196] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A scheduling strategy method for distributed computing under limited storage and computing resources, characterized in that: It includes the following steps: Step 1, generate a task matrix T; Data volume -> rdc, computing volume -> rcc, required memory size -> rnm, output data volume -> rodc; Step 2, generate an application server resource matrix Y; Number of cpu cores -> acn, processing rate of each core -> acv, memory size -> an, network transmission rate -> nv; Step 3, database server resource matrix D; Network transmission rate -> nv, number of cpu cores -> dpcn, processing rate of each core -> dbcv, memory size - dbn, hard disk read rate - dbdv; Step 4, classify the type of the task. If the rdc data volume of the task < rcc computing volume, then define the task as a Cpu-intensive task, otherwise define it as an IO-intensive task; Step 5, determine the servers that can be allocated currently. For cpu-intensive tasks, first determine whether the application server can be executed in parallel under the current situation; Calculate the parallelism Pac of the application server = acn * (rcc / acv + (rdc + rodc) / nv) / ((rdc + rodc) / nv), that is, the number of cores * (cpu time + IO time) / IO time. For IO-intensive tasks, first determine whether they can be executed concurrently on the database server; Calculate the parallelism PaO = acn * (rcc / acv + (rdc + rodc) / dbdv) / ((rdc + rodc) / dbdv), If the calculated parallelism >= the number of tasks tn + 1 allocated to the server, then the server can be regarded as a candidate server and added to the task candidate server list tsl; Step 6, if tsl is not empty, then calculate which server the task will be allocated to with the shortest execution time. At this time, take the longest task execution duration as the total duration Ect of the server execution; Step 7, if tsl is empty, then calculate the execution duration of the task on all servers and allocate the task to this server first; Step 8, generate an allocation plan for the task, ts1, ts2, ts3, and sort the allocation plan according to the priorities of Step 5, Step 6, and Step 7, and allocate according to the priority order; Step 9, obtain tasks from the task matrix and repeat steps 4, 5, 6, 7, and 8 until a plan is obtained. If the set time threshold maxT is met, then output the plan. If not, repeat the above steps.

2. The scheduling strategy method for distributed computing under limited storage and computing resources as claimed in claim 1, characterized in that: The task execution duration includes the time-consuming calculation of a single task and the overall time-consuming calculation.

3. The scheduling strategy method for distributed computing under limited storage and computing resources as claimed in claim 2, characterized in that: The time-consuming calculation of a single task: When a task is being processed, it does not pull all the required data from the hard disk to the memory at one time, but pulls and processes it in batches. The data volume of each batch is the memory size required for the task; The number of pulls is (data volume / required memory size) of the task. The time-consuming within each batch is (cpu calculation rate * computing volume / number of pulls) + data volume * IO rate / number of pulls. According to the formula of pipeline instructions, the formula for calculating the time-consuming of a single task is: (Pull count - 1) * max((CPU computing rate * computing volume / pull count), data volume * IO rate / pull count) + (CPU computing rate * computing volume / pull count) + data volume * IO rate / pull count, that is, the total time taken to complete a task plus (number of times - 1) * the time taken by the longest instruction. Here, the IO rate is the hard disk read rate of the database server when calculating data movement to computing, and the network transmission rate when data moves to computing.

4. The scheduling strategy method for distributed computing under limited storage and computing resources as claimed in claim 2, characterized in that: Calculation of the time taken for a single task: The time taken for a single server is the task with the longest execution time on that server. The longest time taken for the cluster is the single server with the longest time taken.

5. A scheduling strategy system for distributed computing under limited storage and computing resources that implements the scheduling strategy method for distributed computing under limited storage and computing resources as described in any one of claims 1 to 4, characterized in that: The scheduling strategy system for distributed computing under limited storage and computing resources includes: A generation module for generating a task matrix T; data volume -> rdc, computing volume -> rcc, required memory size -> rnm, output data volume -> rodc; generating an application server resource matrix Y; number of CPU cores -> acn, processing rate of each core -> acv, memory size -> an, network transmission rate -> nv; A database module for the database server resource matrix D; network transmission rate -> nv, number of CPU cores -> dpcn, processing rate of each core -> dbcv, memory size - dbn, hard disk read rate - dbdv; A partitioning module for partitioning the type of the task. If the rdc data volume of the task < rcc computing volume, the task is defined as a CPU-intensive task, otherwise it is defined as an IO-intensive task; A judgment module for judging the servers that can be allocated currently. For CPU-intensive tasks, it first judges whether the application server can be executed in parallel under the current situation; calculates the parallelism of the application server Pac = acn * (rcc / acv + (rdc + rodc) / nv) / ((rdc + rodc) / nv), that is, the number of cores * (CPU time + IO time) / IO time. For IO-intensive tasks, it first judges whether concurrent execution can be carried out on the database server; Calculates the parallelism PaO = acn * (rcc / acv + (rdc + rodc) / dbdv) / ((rdc + rodc) / dbdv), If the calculated parallelism >= the number of tasks allocated to this server tn + 1, then this server can be regarded as a candidate server and added to the task candidate server list tsl; If tsl is not empty, calculate which server the task will be allocated to with the shortest execution time. At this time, the longest task execution duration is used as the total execution duration Ect of the server; If tsl is empty, calculate the execution duration of the task on all servers and allocate the task to this server first; An allocation module for generating an allocation plan for the task, ts1, ts2, ts3, and sorting the allocation plan by priority and allocating it in the order of priority; The output module is used to obtain the task repetition steps from the task matrix until a solution is obtained. If the set time threshold maxT is met, the solution is output; if not, the above steps are repeated.

6. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the scheduling strategy method for distributed computing under limited storage and computing resources as described in any one of claims 1-4.

7. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the scheduling strategy method for distributed computing under limited storage and computing resources as described in any one of claims 1 to 4.

8. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the scheduling strategy system for distributed computing under limited storage and computing resources as described in claim 5.

Images