A scheduling algorithm for distributed computing under limited storage and computing resources

Abstract

The application belongs to the technical field of distributed computing, and discloses a scheduling strategy method for distributed computing under limited storage and computing resources. The application optimizes the performance of task scheduling without changing the architecture of the whole application system. The utilization of the application system, the web server and the database server is improved. Based on the architecture of the traditional application system, the efficiency of the whole timing task execution can be improved without changing the architecture. During the timing task execution, the application server is no longer used alone, and the application server and the database server are fully utilized. The application aims to solve the performance of task scheduling with the traditional relational database as a data source. The application can be applied to the architecture of the traditional application, and the performance of task scheduling can be optimized without changing the architecture.

Description

Technical Field

[0001] This invention belongs to the field of distributed computing technology, and in particular relates to a scheduling strategy method and system for distributed computing under limited storage and computing resources. Background Technology

[0002] In many industry production environments, batch jobs exist to process, summarize, and analyze daily data to generate reports. However, due to system planning and technology selection, many domestic companies still use traditional relational databases for data storage, and the resources of database servers and application servers are limited. After years of operation and manpower constraints, it's not feasible to replace or add new big data technologies such as HBase, Hive, and Hadoop. Furthermore, while these technologies can improve batch processing efficiency, they cannot guarantee transaction ACID, so relational databases cannot be phased out. Introducing new components presents challenges in synchronizing relational databases with big data components and ensuring data consistency. Therefore, without increasing hardware resources, a reasonable strategy is needed to ensure cluster load balancing and execution time in multi-tasking scenarios.

[0003] Currently, the transmission of data from the database to the application server and the return of data from the application server to the database generate a large amount of network I / O. Therefore, a strategy to improve data locality is needed. By fully utilizing data locality, the system can effectively alleviate network traffic within the cluster and reduce network congestion.

[0004] The current execution method leaves the database server completely idle while the application server processes data, resulting in a significant waste of resources and low resource utilization. Therefore, it is necessary to find a reasonable strategy to send some computing programs to the database server for task execution, which can reduce network I / O, improve data locality, and also improve the resource utilization of the entire cluster.

[0005] The paper "Research and Optimization of Hadoop Job Scheduling Locality" by Chen Ruofei proposes a resource prefetching-based approach to balance locality and load balancing. When server F1, which satisfies the locality requirement of task T1, is executing another task T2, and the remaining time for task T2 is greater than the data transfer time required by task T1, then task T1 is assigned to another server F2, and the data is transferred from F1 to F2. Then, task T2 is executed on F2. The core idea of ​​this strategy is to prioritize satisfying the data locality of tasks. If this cannot be satisfied, the computation of tasks continues while data from non-local tasks is migrated, thus ensuring both load balancing and maximizing the locality of each task.

[0006] There is already a wealth of research on task scheduling. Whether it's improving data locality, reducing I / O, or optimizing algorithms, these studies have effectively improved task execution efficiency, reduced energy consumption, and enhanced cluster load balancing. However, current research on scheduling algorithms, such as the optimization of various artificial intelligence algorithms (particle swarm optimization, simulated annealing, genetic algorithms, etc.), uses scheduling scenarios, whether heterogeneous or homogeneous, where there is a many-to-one relationship between tasks and execution servers—meaning one task involves one server, and one server can execute multiple tasks. Research on improving data locality has proposed many optimization methods for scenarios where one task involves multiple servers. For example, by calculating the data size corresponding to the key generated in the map phase of Hadoop's MapReduce model to allocate tasks in the reduce phase, this improves the overall task execution efficiency and load balancing. However, most of these studies are based on Hadoop, a big data computing framework. If data cannot be migrated to the HDFS file system in actual production projects, these scheduling strategies will be difficult to use.

[0007] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0008] (1) Existing technologies cannot solve the performance optimization of task scheduling models that use traditional relational databases as data sources for data storage.

[0009] (2) Existing technologies cannot optimize the architecture of most traditional application systems. Summary of the Invention

[0010] To address the problems existing in the prior art, this invention provides a scheduling strategy method for distributed computing under limited storage and computing resources.

[0011] This invention is implemented as follows: a scheduling strategy method for distributed computing under limited storage and computing resources includes:

[0012] Step 1, generate the task matrix T;

[0013] Data volume -> rdc, computational volume -> rcc, required memory size -> rnm, output data volume -> rodc;

[0014] Step 2, generate the application server resource matrix Y;

[0015] Number of cpu cores -> acn, processing rate per core -> acv, memory size -> an, network transmission rate -> nv;

[0016] Step 3, database server resource matrix D;

[0017] Network transmission rate -> nv, number of cpu cores -> dpcn, processing rate per core -> dbcv, memory size - dbn, hard disk read rate - dbdv;

[0018] Step 4, classify the type of the task. If the rdc data volume of the task < rcc computational volume, define the task as a Cpu-intensive task; otherwise, define it as an IO-intensive task;

[0019] Step 5, determine the servers that can be allocated currently. For cpu-intensive tasks, first judge whether the application server can be executed in parallel under the current situation;

[0020] Calculate the parallelism Pac of the application server = acn * (rcc / acv + (rdc + rodc) / nv) / ((rdc + rodc) / nv), that is, number of cores * (cpu time + IO time) / IO time. For IO-intensive tasks, first judge whether it can be executed concurrently on the database server;

[0021] Calculate the parallelism PaO = acn * (rcc / acv + (rdc + rodc) / dbdv) / ((rdc + rodc) / dbdv),

[0022] If the calculated parallelism >= the number of tasks tn + 1 allocated to this server, then this server can be regarded as a candidate server and added to the task candidate server list tsl;

[0023] Step 6, if tsl is not empty, calculate which server to allocate the task to with the shortest execution time, and at this time, use the longest task execution duration as the total execution duration Ect of the server;

[0024] Step 7, if tsl is empty, calculate the execution duration of the task on all servers, and allocate the task to this server first;

[0025] Step 8: Generate task allocation schemes ts1, ts2, and ts3, and prioritize the allocation schemes according to steps 5, 6, and 7, and allocate tasks in priority order.

[0026] Step 9: Obtain tasks from the task matrix and repeat steps 4, 5, 6, 7, and 8 until a solution is obtained. If the set time threshold maxT is met, output the solution; otherwise, repeat the above steps.

[0027] Furthermore, the task execution time includes the time calculation for a single task and the overall time calculation.

[0028] Furthermore, the time consumption calculation for the individual task:

[0029] When a task is being processed, the required data is not fetched from the hard drive into memory all at once, but is fetched and processed in batches. The amount of data in each batch is the amount of memory required by the task.

[0030] The number of fetches is equal to the task's (data volume / required memory size). The time consumed within each batch is (CPU calculation speed * calculation volume / number of fetches) + data volume * IO speed / number of fetches. According to the pipeline instruction formula, the formula for calculating the time consumed by a single task is:

[0031] (Number of fetches - 1) * max((CPU calculation speed * computational load / number of fetches), data volume * I / O rate / number of fetches) + (CPU calculation speed * computational load / number of fetches) + data volume * I / O rate / number of fetches, which is the total time to complete a task plus (number of fetches - 1) * the time of the longest instruction. The I / O rate is the hard disk read rate of the database server when moving from computation to data, and the network transmission rate when moving from data to computation.

[0032] Furthermore, the time consumption calculation for the individual task:

[0033] The time taken by a single server is the longest task executed by that server.

[0034] The longest execution time in the cluster is the longest execution time of a single server.

[0035] Another object of the present invention is to provide a scheduling strategy system for distributed computing under limited storage and computing resources, comprising:

[0036] A generation module for generating a task matrix T; data volume -> rdc, computing volume -> rcc, required memory size -> rnm, output data volume -> rodc; generating an application server resource matrix Y; number of cpu cores -> acn, processing rate of each core -> acv, memory size -> an, network transmission rate -> nv;

[0037] A database module for the database server resource matrix D; network transmission rate -> nv, number of cpu cores -> dpcn, processing rate of each core -> dbcv, memory size - dbn, hard disk read rate - dbdv;

[0038] A partitioning module for partitioning the type of the task. If the rdc data volume of the task < rcc computing volume, the task is defined as a Cpu-intensive task; otherwise, it is defined as an IO-intensive task;

[0039] A judgment module for judging the servers that can be allocated currently. For Cpu-intensive tasks, it first judges whether the application server can execute in parallel under the current situation; calculating the parallelism Pac of the application server = acn * (rcc / acv + (rdc + rodc) / nv) / ((rdc + rodc) / nv), that is, the number of cores * (cpu time + IO time) / IO time. For IO-intensive tasks, it first judges whether it can execute concurrently on the database server;

[0040] Calculating the parallelism PaO = acn * (rcc / acv + (rdc + rodc) / dbdv) / ((rdc + rodc) / dbdv),

[0041] If the calculated parallelism >= the number of tasks tn + 1 allocated to this server, then this server can be regarded as a candidate server and added to the task candidate server list tsl;

[0042] If tsl is not empty, calculate which server to allocate the task to with the shortest execution time. At this time, take the longest task execution duration as the total execution duration Ect of the server;

[0043] If tsl is empty, calculate the execution duration of the task on all servers and allocate the task to this server first;

[0044] An allocation module for generating allocation schemes ts1, ts2, ts3 for the task and sorting the allocation schemes by priority and allocating them in the order of priority;

[0045] An output module for obtaining the task repetition steps from the task matrix until a scheme is obtained. If the set time threshold maxT is met, output the scheme; if not, repeat the above steps.

[0046] Another object of the present invention is to provide a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps of the scheduling strategy method for distributed computing under limited storage and computing resources.

[0047] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the scheduling strategy method for distributed computing under limited storage and computing resources.

[0048] Another objective of this invention is to provide an information data processing terminal, which is used to implement a scheduling strategy system for distributed computing under limited storage and computing resources.

[0049] Based on the above technical solutions and the technical problems solved, please analyze the advantages and positive effects of the technical solution to be protected by this invention from the following aspects:

[0050] First, regarding the technical problems existing in the above-mentioned prior art, the creative technical effects resulting from solving these problems are described in detail below:

[0051] The performance of task scheduling was optimized without changing the architecture of the entire application system.

[0052] It improved the utilization of application systems, web servers, and database servers.

[0053] Based on the traditional application system architecture, the efficiency of the entire scheduled task execution can be improved without making any changes to the architecture.

[0054] When scheduled tasks are executed, the application server is no longer used alone; instead, both the application server and the database server are fully utilized.

[0055] Existing technologies cannot solve the performance optimization of task scheduling models that use traditional relational databases as data sources for data storage. This invention aims to solve the performance problem of task scheduling that uses traditional relational databases as data sources.

[0056] Existing technologies cannot optimize the architecture of most traditional application systems, namely, an architecture where a web server acts as the application layer and a database server acts as the data storage layer. This invention is applicable to such traditional application architectures, optimizing task scheduling performance without requiring any architectural changes.

[0057] Scheduling strategies for distributed computing under limited storage and computing resources is an important research area. In practical production and scientific research, the execution efficiency of scheduled tasks directly affects production efficiency and scientific research results. This invention focuses on selecting strategies for the mutual movement of computation and data. A reasonable movement strategy can improve the execution efficiency and performance of scheduled tasks, thereby helping to complete tasks more accurately and quickly. This invention studies strategies for the mutual movement of computation and data to improve the efficiency of scheduled tasks.

[0058] This invention examines the impact of computation and data movement on the efficiency of scheduled tasks, including network bandwidth, hard disk read / write speed, CPU execution speed, memory size, and the characteristics of the task itself (IO-intensive and CPU-intensive). When a task is CPU-intensive, data should be prioritized for computation, while when a task is IO-intensive, computation should be prioritized for data. The optimization model for scheduled tasks uses a mathematical model to describe the execution process. The data movement model describes the data movement process, and the computation model represents the computation process. This invention analyzes and evaluates the impact of computation and data transmission efficiency on the efficiency of scheduled tasks. Data transmission is a critical part of task execution, and transmitting data over a computer network can significantly reduce task execution time. However, data transmission also incurs additional costs, such as network transmission time and costs. These costs must also be considered. Furthermore, storing data locally, rather than executing it on a remote computer, can also significantly improve execution efficiency.

[0059] This invention proposes several strategies for directly evaluating the execution efficiency of scheduled tasks, such as bandwidth-aware scheduling and real-time optimization algorithms. The basic idea is to select appropriate load and data movement schemes based on different task types and environments, thereby improving task execution efficiency. For example, bandwidth-aware scheduling can improve task execution efficiency by utilizing appropriate network communication bandwidth and allocating tasks accordingly. Real-time optimization algorithms monitor, analyze, and optimize tasks during execution to maximize their efficiency. This invention summarizes its research results and conclusions.

[0060] Choosing the optimal approach for scheduled tasks based on the inter-task efficiency of computation and data movement is a comprehensive and complex process that requires the integration and analysis of various factors. In this selection process, the impact of data transfer and computation on execution efficiency is equally important; optimizing these factors can improve task efficiency and performance. Under specific circumstances, employing appropriate load and data movement strategies can further enhance task efficiency and performance.

[0061] This invention studies the efficiency of timed tasks by considering the interaction between computation and data movement, carefully analyzes the factors affecting execution efficiency and performance, and proposes some suitable strategies, which have important application value in actual production and scientific research.

[0062] Second, does the technical solution of this invention solve a technical problem that people have long desired to solve but have never been able to successfully address?

[0063] Many current technical solutions for improving the efficiency of scheduled task processing require substantial server resources. This solution, however, can effectively optimize the efficiency of scheduled task processing within existing limited resources.

[0064] Currently, many scheduled task solutions rely on new technical components, including but not limited to Hadoop and Spark. For the transformation of traditional applications, migrating the entire system's scheduling to new technical components is costly, risky, and has low returns. This solution can improve the execution efficiency of scheduled tasks by optimizing the scheduling algorithm within the existing architecture.

[0065] Traditional tax systems typically process large amounts of data from upstream systems at night, which is then used by relevant business personnel as soon as they start work the next day. However, due to the large volume of tasks and data, these tasks are often not completed by the start of the workday, forcing business personnel to wait for them to finish before proceeding to the next step. This optimized solution improves the overall system efficiency, ensuring that all scheduled tasks are completed before the start of the workday each day, thus resolving the pain point of business personnel having to wait for tasks to finish during work hours. Attached Figure Description

[0066] Figure 1 This is a flowchart of a scheduling strategy method for distributed computing under limited storage and computing resources provided in an embodiment of the present invention.

[0067] Figure 2 This is a system structure diagram of a scheduling strategy for distributed computing under limited storage and computing resources provided in an embodiment of the present invention.

[0068] Figure 3 This is a flowchart of a scheduling algorithm for distributed computing under limited storage and computing resources provided in an embodiment of the present invention.

[0069] Figure 4 This is a diagram showing the amount of I / O data and CPU computation for each task provided in the embodiments of the present invention.

[0070] Figure 5 This is a timed job scheduling distribution diagram provided in an embodiment of the present invention.

[0071] Figure 6It is the effect diagram of task efficiency improvement provided by the embodiments of the present invention.

[0072] Figure 7 It is the time comparison chart (unit: h) of each task before and after optimization provided by the embodiments of the present invention. Detailed implementation manners

[0073] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0074] As Figure 1 shown, a scheduling strategy method for distributed computing under limited storage and computing resources provided by the embodiments of the present invention includes the following steps:

[0075] S101, generate a task matrix T;

[0076] Data volume -> rdc, computing amount -> rcc, required memory size -> rnm, output data volume -> rodc;

[0077] S102, generate an application server resource matrix Y;

[0078] Number of cpu cores -> acn, processing rate of each core -> acv, memory size -> an, network transmission rate -> nv;

[0079] S103, database server resource matrix D;

[0080] Network transmission rate -> nv, number of cpu cores -> dpcn, processing rate of each core -> dbcv, memory size - dbn, hard disk read rate - dbdv;

[0081] S104, classify the type of the task. If the rdc data volume of the task < rcc computing amount, define the task as a Cpu-intensive task; otherwise, define it as an IO-intensive task;

[0082] S105, determine the servers that can be allocated currently. For cpu-intensive tasks, first judge whether the application server can be executed in parallel under the current situation;

[0083] Calculate the parallelism Pac of the application server = acn * (rcc / acv + (rdc + rodc) / nv) / ((rdc + rodc) / nv), that is, number of cores * (cpu time + IO time) / IO time. For IO-intensive tasks, first judge whether it can be executed concurrently on the database server;

[0084] The computing parallelism PaO = acn * (rcc / acv + (rdc + rodc) / dbdv) / ((rdc + rodc) / dbdv),

[0085] If the calculated parallelism >= the number of tasks tn + 1 allocated to this server, then this server can be regarded as a candidate server and added to the task candidate server list tsl;

[0086] S106, if tsl is not empty, then calculate which server the task will be assigned to with the shortest execution time, and at this time, use the longest task execution duration as the total execution duration Ect of the server;

[0087] S107, if tsl is empty, then calculate the execution duration of the task on all servers and preferentially assign the task to this server;

[0088] S108, generate an allocation plan for the task, ts1, ts2, ts3, and sort the allocation plan according to the priorities in steps S105, S106, and S107, and allocate according to the priority order;

[0089] S109, obtain the steps of repeating S104, S105, S106, S107, and S108 from the task matrix until a plan is obtained. If the set time threshold maxT is satisfied, then output the plan. If not, repeat the above steps.

[0090] The core of this method is to model the distributed computing tasks and server resources as a task matrix \(T\) and resource matrices \(Y, D\). The task matrix \(T\) contains the key attributes of the tasks, including the data volume (\(rdc\)), computing volume (\(rcc\)), required memory size (\(rnm\)), and output data volume (\(rodc\)). The application server resource matrix \(Y\) and the database server resource matrix \(D\) are used to describe the key resource parameters of the servers, such as the number of CPU cores, the processing rate per core, the memory size, and the network transmission rate. By mapping the task requirements to the resource capabilities, the matching degree between the tasks and resources can be systematically analyzed.

[0091] The method divides the tasks into CPU-intensive tasks and IO-intensive tasks by comparing the data volume (\(rdc\)) and computing volume (\(rcc\)) of the tasks. If \(rdc < rcc\), the task is defined as a CPU-intensive task and is preferentially assigned to an application server with stronger computing performance; otherwise, it is defined as an IO-intensive task and is preferentially assigned to a database server more suitable for processing large data volumes. This type of division is the core of the task scheduling strategy and directly affects the allocation priority and execution efficiency of the tasks.

[0092] For each task, potential candidate servers are selected based on its type. For CPU-intensive tasks, it is first determined whether the application server can achieve parallel processing under the current resources, and the degree of parallelism (Pac) is calculated. If Pac is greater than the current number of tasks (tn+1), the server is added to the task candidate server list (tsl). Similarly, for I / O-intensive tasks, the degree of parallelism (PaO) of the database server is calculated. This step dynamically selects servers by calculating the degree of parallelism of resources, ensuring optimal resource utilization.

[0093] When the candidate server list (tsl) is not empty, the server with the shortest task execution time is selected from it. At this point, the total execution time (Ect) of each candidate server is calculated, and the solution with the longest task execution time and the shortest execution time is selected to achieve task load balancing. If (tsl) is empty, the task will be assigned to a server with the shortest execution time among all servers, thus ensuring that the task can be executed smoothly.

[0094] The task scheduling system generates allocation schemes through iterative optimization, including server priority ranking and task allocation order optimization. For each task, the system executes steps S104 to S108 among candidate servers and allocation schemes, dynamically updating the allocation results. If the maximum execution time threshold (maxT) of the task is met, the current optimal scheme is output; otherwise, the task scheduling process is iterated again to further optimize the allocation strategy.

[0095] The entire scheduling strategy iterates through the unassigned tasks in the task matrix, adjusting the task-resource matching strategy in real time to fully utilize the system's limited computing and storage resources. Through candidate server selection, parallelism calculation, and execution time optimization, it ensures that task scheduling meets global performance metrics. Simultaneously, the system dynamically adjusts the allocation strategy based on actual resource usage during task execution, improving the overall efficiency and task responsiveness of the distributed computing system.

[0096] The task execution time provided in this embodiment of the invention includes the time calculation of a single task and the overall time calculation.

[0097] The time consumption calculation for a single task provided in this embodiment of the invention:

[0098] When a task is being processed, the required data is not fetched from the hard drive into memory all at once, but is fetched and processed in batches. The amount of data in each batch is the amount of memory required by the task.

[0099] The number of pulls is (data volume / required memory size) for the task. The time consumption within each batch is (cpu computing rate * amount of computation / number of pulls) + data volume * IO rate / number of pulls. According to the formula of pipeline instructions, the formula for the time consumption of a single task is:

[0100] (Number of pulls - 1) * max((cpu computing rate * amount of computation / number of pulls), data volume * IO rate / number of pulls) + (cpu computing rate * amount of computation / number of pulls) + data volume * IO rate / number of pulls, that is, the total time consumption to complete a task plus (number - 1) * the time consumption of the instruction with the longest time. Among them, the IO rate is the hard disk read rate of the database server when calculating the data movement, and the network transmission rate when the data moves to the calculation.

[0101] The calculation of the time consumption of a single task provided by the embodiment of the present invention:

[0102] The time consumption of a single server is the task with the longest execution time on this server.

[0103] The longest time consumption of the cluster is the single server with the longest time consumption.

[0104] Such as Figure 2 As shown, a scheduling strategy system for distributed computing under limited storage and computing resources provided by the embodiment of the present invention includes:

[0105] A generation module, used to generate a task matrix T; data volume -> rdc, amount of computation -> rcc, required memory size -> rnm, output data volume -> rodc; generate an application server resource matrix Y; cpu core number -> acn, processing rate of each core -> acv, memory size -> an, network transmission rate -> nv;

[0106] A database module, used for the database server resource matrix D; network transmission rate -> nv, cpu core number -> dpcn, processing rate of each core -> dbcv, memory size - dbn, hard disk read rate - dbdv;

[0107] A partitioning module, used to partition the type of the task. If the rdc data volume of the task < rcc amount of computation, then define the task as a Cpu-intensive task, otherwise define it as an IO-intensive task;

[0108] The judgment module is used to determine the available servers for allocation. For CPU-intensive tasks, it first determines whether the application server can perform parallel execution under the current conditions. It calculates the parallelism of the application server, Pac = acn * (rcc / acv + (rdc + rodc) / nv) / ((rdc + rodc) / nv), which is the number of cores * (CPU time + IO time) / IO time. For IO-intensive tasks, it first determines whether they can be executed concurrently on the database server.

[0109] The parallelism PaO is calculated as follows: PaO = acn * (rcc / acv + (rdc + rodc) / dbdv) / ((rdc + rodc) / dbdv).

[0110] If the calculated parallelism is greater than or equal to the number of tasks tn+1 assigned to the server, then the server can be considered a candidate server and added to the candidate server list tsl for that task.

[0111] If tsl is not empty, calculate which server to assign the task to with the shortest execution time, and use the longest task execution time as the total server execution time Ect.

[0112] If the tsl is empty, calculate the execution time of the task across all servers and prioritize assigning the task to that server.

[0113] The allocation module is used to generate allocation schemes (ts1, ts2, ts3) for tasks, prioritize the allocation schemes, and allocate tasks according to priority.

[0114] The output module is used to obtain the task repeat S10 from the task matrix until a solution is obtained. If the set time threshold maxT is met, the solution is output; otherwise, the above S10 is repeated.

[0115] Another object of the present invention is to provide a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps of the scheduling strategy method for distributed computing under limited storage and computing resources.

[0116] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the scheduling strategy method for distributed computing under limited storage and computing resources.

[0117] Another objective of this invention is to provide an information data processing terminal, which is used to implement a scheduling strategy system for distributed computing under limited storage and computing resources.

[0118] Specific implementation of the present invention:

[0119] The complete technical solution (invention solution) provided by this invention

[0120] I. Defining Indicators

[0121] Task (data volume, computational load, required memory size).

[0122] Application server (number of CPU cores, processing speed per core, memory size, network transmission rate).

[0123] Database server (network transmission rate, number of CPU cores, processing speed per core, memory size, hard disk read speed).

[0124] II. Time Calculation for a Single Task

[0125] 1. Generally speaking, when processing a task, the required data is not fetched from the hard drive into memory all at once, but rather fetched and processed in batches. The amount of data in each batch is the memory size required by the task.

[0126] 2. The number of fetches is the (data volume / required memory size) of the task. The time consumed in each batch is (CPU calculation speed * calculation volume / number of fetches) + data volume * IO rate / number of fetches. According to the pipeline instruction formula, the calculation formula for the time consumed by a single task is (number of fetches - 1) * max((CPU calculation speed * calculation volume / number of fetches), data volume * IO rate / number of fetches) + (CPU calculation speed * calculation volume / number of fetches) + data volume * IO rate / number of fetches. That is, the total time consumed to complete a task plus (number of fetches - 1) * the time consumed by the longest instruction. The IO rate is the hard disk read rate of the database server when moving from computation to data, and the network transmission rate when moving from data to computation.

[0127] III. Overall Time Consumption Calculation

[0128] 1. The time taken by a single server is the longest task executed by that server.

[0129] 2. The longest execution time in the cluster is the longest execution time of a single server.

[0130] like Figure 3 As shown in section IV, Algorithm Implementation

[0131] S201. Establish two matrices: the task matrix for the resources required by the task and the server resource matrix.

[0132] S202: Distribute tasks one by one to various servers, find the allocation with the shortest total execution time, and then allocate it to that server.

[0133] S203 considers two scenarios during allocation: one is execution on the application server, which requires network I / O from the database to the application server; the other is execution directly on the database, which does not require network I / O.

[0134] S204 uses simulated annealing as a heuristic during allocation to improve the algorithm's performance and convergence speed. It mainly uses the idea of ​​calculating the increment using the evaluation function and accepting new solutions, prioritizing the accepted new solutions for the next search step.

[0135] 2.4 Engineering Examples Provided by the Invention

[0136] Ningbo Bank's tax management system, which processes data at night, has seen a significant reduction in the overall execution time of all scheduled tasks and a substantial improvement in the utilization of application servers and database servers after implementing this solution.

[0137] 1. Generate task matrix T(100M, 1G, 200M, 50M)

[0138] 1G, 1G, 1G, 200M

[0139] 20M, 10M, 15M, 40M

[0140] 300M, 1.5G, 800M, 400M

[0141] 210M, 400M, 300M, 40M

[0142] 2G, 16G, 512M, 4G

[0143] 500M, 8G, 2G, 800M

[0144] 3G, 3G, 1G, 3G)

[0145] 2. Generate application server resource matrix Y

[0146] (4, 2.8GHZ, 8G, 100M / S4, 2.8GHZ, 8G, 100M / S).

[0147] 3. Generate the database server resource matrix (100M / S, 8, 2.8GHZ, 16G, 256M / s).

[0148] 4. Obtain a task (100M, 1G, 200M, 50M) from the task matrix T and define its type as CPU intensive.

[0149] 5. The maximum parallelism of the application server is (1.05 / 0.05)*4=84. Since 84 is greater than 1, it can be allocated to this server, and the total time is 1.65s.

[0150] 6. There are 3 ways to perform this task: t1Y1, t1Y2, and t1D1. Based on the result of step 5, prioritize searching for t1Y1.

[0151] 7. Take the second task t2 (1G, 1G, 1G, 200M) from T and define its type as IO intensive.

[0152] 8. The maximum parallelism of the database server is 8. Tasks can be assigned to this server, with a total time of 5.05.

[0153] 9. This task also has 3 methods: t2Y1, t2Y2, and t2D1. Based on the result of step 8, t2D1 is searched first.

[0154] 10. Repeat steps 4, 5, 6, 7, 8, and 9 until a scheduling scheme that satisfies the longest duration threshold is obtained.

[0155] Scheduling Algorithm Classification and Analysis:

[0156] 1. Optimize by reducing I / O

[0157] Currently, the execution flow of batch processing tasks consists of three steps: reading data, processing data, and storing results. Typically, data processing is performed by a program deployed on the application server. If the server hosting the data differs from the application server (e.g., application server A, data server B), then server B needs to read the file content into kernel space via DMA. The CPU then copies the data from kernel space into user space. After the user program reads the file data from kernel space, it writes the data to the write buffer of the socket connection established between A and B in kernel space. Then...

[0158] DMA copies the data written to the buffer into the network card and then transmits it to server A. Server A needs to use DMA to read the data from its network card into the read buffer of the socket connection established between A and B in the kernel space on server A. Then the CPU copies the data to user space. Only then can the program on server A truly begin data processing.

[0159] The data reading phase in the above process consumes a lot of resources and performance, so reducing I / O has become an important research direction for optimizing scheduled tasks.

[0160] 1.1 Reduce I / O consumption through zero-copy

[0161] Zero-copy is an effective way to reduce network I / O time in network transmission. It mainly improves the efficiency of overall data transmission by reducing the number of times data is copied in memory. Currently, there are two implementations of zero-copy: MMAP and Send File. MMAP primarily uses memory mapping, which directly points user space to kernel space. When a user process needs to retrieve a file, the DMA first reads the data into kernel space and then copies it back to user space. MMAP eliminates this kernel-to-user-space copy, directly copying the file content from the kernel space corresponding to the socket connection, thus improving overall transmission efficiency. Send File, on the other hand, marks the user space containing the file. Instead of copying from the kernel space containing the file to the kernel space containing the socket, the DMA directly copies the data from the kernel space containing the file to the network card based on the mark, further improving efficiency compared to MMAP.

[0162] The advantage of zero-copy is that it requires minimal system modifications, and significant improvements can be achieved without any changes to the overall system architecture. However, zero-copy only increases the transmission speed, thereby reducing IO time, without fundamentally solving the performance overhead caused by data transmission between servers. Although the IO transmission speed is faster, the total amount of IO data is not reduced, and the improvement is relatively limited in scenarios with large data volumes.

[0163] 1.2 Optimization by improving data locality

[0164] Data locality refers to assigning tasks directly to the server where the data resides during task scheduling. In other words, if task T requires data on server B, then task T is executed on server B. This eliminates the performance and time consumed by network I / O, thereby improving the overall efficiency of scheduled tasks. However, if multiple tasks require data on the same server, load imbalance can easily occur. Currently, much research is being conducted to balance the relationship between data locality and load balancing.

[0165] 1.2.1 Resource Prefetching

[0166] The paper "Research and Optimization of Hadoop Job Scheduling Locality" by Chen Ruofei proposes a resource prefetching-based approach to balance locality and load balancing. When server F1, which satisfies the locality requirement of task T1, is executing another task T2, and the remaining time of task T2 is greater than the data transmission time required by task T1, then task T1 is assigned to another server F2, and the data is transmitted from F1 to F2. Then, task T2 is executed on F2. The core idea of ​​this strategy is to prioritize the data locality of tasks. If it cannot be satisfied, the computation of tasks is carried out while the data of non-local tasks is migrated, thus ensuring both load balancing and maximizing the locality of each task.

[0167] 1.2.2 Using artificial intelligence search algorithms

[0168] Numerous studies have incorporated artificial intelligence search algorithms to comprehensively consider factors such as data locality and load balancing to improve task execution efficiency. "Research and Implementation of Hadoop Job Scheduling Algorithms in Heterogeneous Environments (by Tian Ye)" uses a hybrid genetic cuckoo algorithm to comprehensively consider factors such as CPU, data locality, and load balancing in heterogeneous environments for task scheduling. "Research on Dynamic Job Scheduling Based on Hadoop Heterogeneous Clusters (by Chen Jinglin)" uses a genetic algorithm to schedule tasks in heterogeneous environments based on CPU, memory, locality, and load balancing. "Research on Load Balancing Hadoop Platform Scheduling Algorithms (by Yang Mingchen)" uses a competition-strategy genetic algorithm to improve the load balancing of task scheduling.

[0169] 1.2.3 Optimization based on transmission distance

[0170] The paper, "Research on Data Locality Load Balancing Strategy under MapReduce Model_Chen Yi," proposes a data transfer strategy when task locality cannot be guaranteed. The paper suggests that if task locality cannot be satisfied, priority should be given to ensuring that tasks and data reside on servers within the same rack, as the transmission speed differs significantly between servers on the same rack compared to those on different racks. The core idea is to prioritize data locality during scheduling. If data locality cannot be guaranteed due to the large number of tasks the server needs to execute, the data transmission distance should be minimized to reduce I / O consumption.

[0171] 2. Scheduling Algorithm Optimization

[0172] There are various methods for optimizing scheduling algorithms, such as optimizing formulas, parameters, and steps. Multiple algorithms can also be organically combined, leveraging their respective strengths to make the algorithm more adaptable to the required scheduling scenario. Research on algorithm optimization reveals that various artificial intelligence algorithms, such as particle swarm optimization and genetic algorithms, can simultaneously consider multiple dimensions, comprehensively taking into account factors such as CPU, memory, network, and data scale to find the optimal task scheduling solution. Current research indicates that the optimization of scheduling algorithms mainly falls into the following categories.

[0173] 2.1 Convergence Optimization

[0174] "Research on Cloud Task Scheduling Strategy Based on Improved Simulated Annealing Algorithm (Sun Chenhui)" improves the convergence speed of the algorithm while maintaining its accuracy by optimizing cooling parameters. "Cloud Service Task Scheduling Algorithm Based on Cauchy Mutant Particle Swarm Optimization (Bi Yang)" improves convergence accuracy by improving the initial population, adjusting inertia weights, introducing Cauchy mutant particles, and changing the velocity formula. "Cloud Data Center Energy Consumption Optimization Task Scheduling Strategy Based on Improved Particle Swarm Optimization Algorithm (Liu Chenwei)" improves the convergence accuracy of the particle swarm optimization algorithm by optimizing the fitness function and using chaotic mapping to generate the initial cluster. "Application of Improved Particle Swarm Optimization Algorithm in Cloud Computing Task Scheduling (Wang Ting)" integrates simulated annealing fusion algorithm and the Hunger Games algorithm to improve the convergence speed of the particle swarm optimization algorithm.

[0175] 2.2 Local Optimal Solution Optimization

[0176] The paper, "Research on Cloud Task Scheduling Strategy Based on Improved Simulated Annealing Algorithm (Sun Chenhui)," discusses how to solve the problem of simulated annealing algorithm getting trapped in local optima by optimizing the temperature update function. The paper mentions memory-based annealing algorithms and tempering annealing methods.

[0177] To escape local optima, an oscillating cooling parameter optimization method is proposed to improve global search capability. When combined with the immune algorithm, a method is proposed to retain inferior solutions with a certain probability to avoid the problem of traditional immune algorithms getting trapped in local optima. "Cloud Computing Task Scheduling Based on Improved Particle Swarm Optimization Algorithm (Chen Junren)" introduces an adaptive adjustment probability to ensure particle diversity and reduce the probability of the algorithm falling into local optima. "Cloud Computing Task Scheduling Method Based on Improved Genetic Algorithm (Wang Hongjie)" ensures the richness of the initial population by calculating chromosome distance, thereby increasing the probability of the genetic algorithm obtaining a global optimum.

[0178] 2.3 Optimization of Task Execution Efficiency

[0179] "A Cloud Computing Task Scheduling Strategy Based on an Improved Differential Evolution Algorithm (Lin Tao)" proposes an improved differential evolution algorithm based on the Levy distribution, which improves task energy consumption and completion time. "A Hadoop Adaptive Task Scheduling Algorithm Based on Node Set Computational Capability Differences (Zhu Jie)" proposes an adaptive algorithm that improves task execution efficiency by calculating node capability differences, identifying lagging tasks, and considering the dynamic changes in node performance during operation.

[0180] 3. Summary

[0181] There is already a wealth of research on task scheduling. Whether it's improving data locality, reducing I / O, or optimizing algorithms, these studies have effectively improved task execution efficiency, reduced energy consumption, and enhanced cluster load balancing. However, current research on scheduling algorithms, such as the optimization of various artificial intelligence algorithms (particle swarm optimization, simulated annealing, genetic algorithms, etc.), uses scheduling scenarios, whether heterogeneous or homogeneous, where there is a many-to-one relationship between tasks and execution servers—that is, one task involves one server, and one server can execute multiple tasks. Research on improving data locality has proposed many optimization methods for scenarios where one task involves multiple servers. For example, by calculating the data size corresponding to the key generated in the map phase of Hadoop's MapReduce model to allocate tasks in the reduce phase, this improves the overall efficiency and load balancing of task execution. However, most of these studies are based on Hadoop, a big data computing framework. If data cannot be migrated to the HDFS file system in actual production projects, these scheduling strategies will be difficult to use. Currently, most production projects still use traditional relational databases for data storage, with MySQL and Oracle being the most typical. In these projects, regardless of the scheduling framework used, the execution flow of scheduled tasks involves the application server establishing a TCP connection with the database server. The application server uses SQL statements to instruct the database server to transfer the data required by the task to the application server. The application server then executes the corresponding task's processing logic and writes the data back to the database using DML statements. In this process, a single task involves two servers: the application server and the database server. The application server executes the scheduled task's program instructions, while the database server provides the necessary data. If the task requires a large amount of data, the entire task will consume significant time on network I / O. In such cases, uploading the computation program to the database server for direct execution (i.e., computation moving to data) could be considered. However, if all tasks rely solely on computation-to-data movement, it will inevitably lead to low utilization of application server resources. Therefore, the key to task scheduling under limited resources is to study how to balance the movement of computation and data.

[0182] I. Specific application areas or related products of this invention.

[0183] Ningbo Bank's Full Tax Management System

[0184] II. Evidence related to the technical effects obtained by the embodiments of the present invention.

[0185] After optimizing the scheduling algorithm, the various tasks of Ningbo Bank's tax management system were more rationally allocated to the server for execution.

[0186] (1) Customer information synchronization task: This task requires parsing the full customer data file pushed by the upstream every day. There are 90 million data entries to be processed every day. Before the algorithm optimization, the entire task execution process was that the application server pulled the data from the database server to the application server through network IO, and the application server processed it simply and then wrote the data back to the database server through network IO. The two network IOs caused a very long time consumption. After the algorithm optimization, the execution of the entire task was directly allocated to the database server, which effectively reduced the IO consumption of the task and improved the utilization rate of the server resources of the entire system. This solution is the most typical for this task. The entire task used to take 1 hour to execute every day has now been reduced to 15 minutes.

[0187] (2) This solution effectively distributes tasks requiring a large amount of I / O to the server where the data resides, while distributing tasks requiring a large amount of computation but with relatively small data volumes to the application server, thereby improving the overall execution efficiency of the scheduled tasks and the utilization rate of resources.

[0188] (3) Tasks with large data volumes, such as the task of synchronizing the invoice flow, the task of synchronizing the account balance sheet, and the task of synchronizing the sales details sheet, but with relatively simple calculation logic, were accurately assigned to the database server, which improved efficiency by 2 times, 3 times, and 2.7 times respectively.

[0189] (4) The data volume of jobs such as the value-added tax analysis table generation job, the stamp duty analysis table generation job, and the stamp duty declaration form generation job, which are relatively small and have more complex calculation logic, were assigned to the application server, and the efficiency was improved by 50%, 62%, and 58%, respectively.

[0190] After algorithm optimization, the overall execution efficiency of the task has been improved by about 2 times, solving the pain point that scheduled tasks have not been completed by the start of the workday. This allows all the tasks in the system to be completed before the start of the workday, so that relevant business personnel can use them immediately without having to wait for scheduled tasks to be completed during the workday.

[0191] Figure 4 The amount of I / O data and CPU computation for each task;

[0192] Figure 5 Scheduled job distribution diagram;

[0193] Figure 6 Image showing the effect of improved task efficiency;

[0194] Figure 7 Comparison chart of time before and after optimization for each task (unit: hours).

[0195] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0196] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A scheduling strategy method for distributed computing under limited storage and computing resources, characterized in that, It includes the following steps: Step 1, generate the task matrix T; Data volume -> rdc, computational volume -> rcc, required memory size -> rnm, output data volume -> rodc; Step 2, generate the application server resource matrix Y; Number of CPU cores -> acn, processing rate of each core -> acv, memory size -> an, network transmission rate -> nv; Step 3, database server resource matrix D; Network transmission rate -> nv, number of CPU cores -> dpcn, processing rate of each core -> dbcv, memory size - dbn, hard disk read rate - dbdv; Step 4, classify the type of the task. If the rdc data volume of the task < rcc computational volume, then define the task as a CPU-intensive task, otherwise define it as an IO-intensive task; Step 5, determine the servers that can be allocated currently. For CPU-intensive tasks, first judge whether the application server can execute in parallel under the current situation; Calculate the parallelism Pac of the application server = acn * (rcc / acv + (rdc + rodc) / nv) / ((rdc + rodc) / nv), that is, the number of cores * (CPU time + IO time) / IO time. For IO-intensive tasks, first judge whether it can execute concurrently on the database server; Calculate the parallelism PaO = acn * (rcc / acv + (rdc + rodc) / dbdv) / ((rdc + rodc) / dbdv), If the calculated parallelism >= the number of tasks tn + 1 allocated to this server, then this server can be regarded as a candidate server and added to the task candidate server list tsl; Step 6, if tsl is not empty, then calculate which server the task is allocated to with the shortest execution time. At this time, take the longest task execution duration as the total duration Ect of the server execution; Step 7, if tsl is empty, then calculate the execution duration of the task on all servers, and allocate the task to the server with the shortest execution time first; Step 8, generate an allocation plan for the task, ts1, ts2, ts3, and sort the allocation plan according to the priorities in Step 5, Step 6, and Step 7, and allocate according to the priority order; Step 9, obtain the task from the task matrix and repeat Steps 4, 5, 6, 7, and 8 until a plan is obtained. If the set time threshold maxT is satisfied, output the plan. If not, repeat the above steps.

2. The scheduling strategy method for distributed computing under limited storage and computing resources as described in claim 1, characterized in that, The task execution duration includes the time-consuming calculation of a single task and the overall time-consuming calculation.

3. The scheduling strategy method for distributed computing under limited storage and computing resources as described in claim 2, characterized in that, The time-consuming calculation of a single task: When a task is processed, it does not pull all the required data from the hard disk to the memory at one time, but pulls and processes it in batches. The data volume of each batch is the memory size required by the task; The number of pulls is (data volume / required memory size) of the task. The time-consuming within each batch is (CPU calculation rate * computational volume / number of pulls) + data volume * IO rate / number of pulls. According to the formula of pipeline instructions, the calculation formula for the time-consuming of a single task is: ((Pull count - 1) * max((CPU computing rate * Computation volume / Pull count), Data volume * IO rate / Pull count)) + (CPU computing rate * Computation volume / Pull count) + Data volume * IO rate / Pull count, that is, the total time taken to complete a task plus (Number of times - 1) * Time taken for the longest instruction, where the IO rate is the hard disk read rate of the database server when calculating data movement, and the network transmission rate when data moves to the calculation.

4. The scheduling strategy method for distributed computing under limited storage and computing resources as described in claim 2, characterized in that, Calculation of the time taken for a single task: The time taken for a single server is the task with the longest execution time on that server; The longest time taken for the cluster is the single server with the longest time taken.

5. A scheduling strategy system for distributed computing under limited storage and computing resources, implementing the scheduling strategy method for distributed computing under limited storage and computing resources as described in any one of claims 1-4, characterized in that, The scheduling policy system for distributed computing under limited storage and computing resources includes: A generation module for generating a task matrix T; Data volume -> rdc, Computation volume -> rcc, Required memory size -> rnm, Output data volume -> rodc; Generating an application server resource matrix Y; CPU core count -> acn, Processing rate of each core -> acv, Memory size -> an, Network transmission rate -> nv; A database module for the database server resource matrix D; Network transmission rate -> nv, CPU core count -> dpcn, Processing rate of each core -> dbcv, Memory size - dbn, Hard disk read rate - dbdv; A partitioning module for partitioning the type of the task. If the rdc data volume of the task < rcc computation volume, the task is defined as a CPU-intensive task; otherwise, it is defined as an IO-intensive task; A judgment module for judging the servers that can be allocated currently. For CPU-intensive tasks, it first judges whether the application server can execute in parallel under the current situation; Calculate the parallelism of the application server Pac = acn * (rcc / acv + (rdc + rodc) / nv) / ((rdc + rodc) / nv), that is, Core count * (CPU time + IO time) / IO time. For IO-intensive tasks, it first judges whether concurrent execution can be carried out on the database server; Calculate the parallelism PaO = acn * (rcc / acv + (rdc + rodc) / dbdv) / ((rdc + rodc) / dbdv), If the calculated parallelism >= the number of tasks allocated to this server tn + 1, then this server can be regarded as a candidate server and added to the task candidate server list tsl; If tsl is not empty, calculate which server the task will be allocated to with the shortest execution time. At this time, the longest task execution duration is used as the total execution duration Ect of the server; If tsl is empty, calculate the execution duration of the task on all servers and allocate the task to this server first; An allocation module for generating an allocation plan for the task, ts1, ts2, ts3, and sorting the allocation plan by priority and allocating it in the order of priority; The output module is used to obtain the task repetition steps from the task matrix until a solution is obtained. If the set time threshold maxT is met, the solution is output; otherwise, the above steps are repeated.

6. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the scheduling strategy method for distributed computing under limited storage and computing resources as described in any one of claims 1-4.

7. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the scheduling strategy method for distributed computing under limited storage and computing resources as described in any one of claims 1-4.

8. An information data processing terminal, characterized in that, The information data processing terminal is used to implement the scheduling strategy system for distributed computing under limited storage and computing resources as described in claim 5.

Images