Adaptive scheduling system and method for big data analysis task
By using an adaptive scheduling system to evaluate the fitness of tasks and execution nodes in real time, calculate the response time matrix, and generate a preference list, the problem of unbalanced machine load in traditional scheduling algorithms is solved, achieving efficient task execution and resource utilization.
Patent Information
- Application Number
- CN202511252535.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-03
- Publication Date
- 2025-12-16
AI Technical Summary
Traditional big data analytics task scheduling algorithms cannot effectively balance machine load, resulting in increased task waiting time, uneven server load, and low task processing efficiency.
An adaptive scheduling system is provided, including task management, execution node management, scheduling analysis, and data migration modules. It evaluates the adaptability of tasks and execution nodes in real time, calculates the response time matrix, generates a task-execution node preference list, and performs data migration when necessary to optimize resource utilization.
It improved the success rate and efficiency of task execution, reduced the overall response time, optimized the utilization of system resources, and enhanced the stability and real-time performance of the system.
Smart Images

Figure CN121143971A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of data management, and in particular relates to an adaptive scheduling system and method for big data analysis tasks. Background Technology
[0002] With the development of big data technology, big data analytics technology has emerged. By scheduling and managing resources for various data-intensive tasks, it can improve task processing efficiency, optimize resource allocation, and meet the high requirements for data processing capabilities.
[0003] In traditional technologies, some classic scheduling algorithms are often used for scheduling and resource management of big data analysis tasks. The First Come First Served (FCFS) scheduling algorithm processes tasks according to the order in which they arrive, with the first task to arrive being executed first. The Shortest Job First (SJF) algorithm selects the task with the shortest execution time for priority processing. The Round Robin (RR) algorithm allocates an equal amount of CPU time slices to each task in a round-robin manner.
[0004] However, the above methods have many problems. The FCFS algorithm will increase the task waiting time and does not take into account the differences in task size, which will not effectively balance the machine load and reduce data locality. Although the SJF algorithm can reduce the execution time, it will cause uneven server load and make it difficult to meet the service level agreement. The RR algorithm will produce a high average waiting time, which will affect the task processing efficiency. Summary of the Invention
[0005] Therefore, it is necessary to provide an adaptive scheduling system and method for big data analysis tasks that can comprehensively consider task requirements, machine resources, and data storage conditions, effectively reduce task response time, increase throughput, and optimize machine resource utilization, in order to address the above-mentioned technical problems.
[0006] In a first aspect, this application provides an adaptive scheduling system for big data analysis tasks, comprising:
[0007] The task management module is used to receive online tasks and assign unique identifiers to them to form a set of tasks to be completed; online tasks include task attribute data, required dataset, and arrival time;
[0008] The execution node management module is used to acquire the resources of the execution nodes in real time and evaluate the suitability of online tasks with the execution nodes based on the resources; the resources include the datasets stored on the execution nodes.
[0009] The scheduling analysis module is used to calculate the response time matrix of each online task in the task set to be completed on different execution nodes based on fitness, and generate a task-execution node preference list.
[0010] The scheduling decision module is used to generate scheduling decisions based on the order of the task-execution node preference list; the scheduling decisions include online tasks and their corresponding target execution nodes;
[0011] The data migration module is used to migrate the data source corresponding to the required dataset to the dataset stored on the target execution node when the required dataset for an online task is not stored on the target execution node.
[0012] In one embodiment, the execution node management module includes a processor management unit, a storage capacity management unit, and a load management unit;
[0013] The processor management unit is used to calculate the processor utilization of each online task on the execution node; if the processor utilization is greater than 1, the fitness is determined to be unable to handle the task; if the processor utilization is less than or equal to 1, the fitness is determined to be able to handle the task.
[0014] The storage capacity management unit is used to process the stored content according to a preset storage processing mode when the remaining storage capacity in the resources is insufficient to meet the data set requirements.
[0015] The load management unit checks the load value in the resources. If the load is greater than a preset threshold, the fitness is determined to be unable to handle the task; if the load is less than the preset threshold, the fitness is determined to be able to handle the task.
[0016] The storage content is processed according to a preset storage processing mode, including:
[0017] When the remaining storage capacity is insufficient to store the required dataset, the dataset stored on some execution nodes is migrated according to the data replication strategy and task dependencies.
[0018] If, after migrating some of the execution nodes' stored datasets, the remaining storage capacity is still insufficient to store the required datasets, the fitness is determined to be "unable to support the task"; if the remaining storage capacity is sufficient to store the required datasets, the fitness is determined to be "able to support the task".
[0019] In one embodiment, the scheduling analysis module includes a response time calculation module and a list generation module;
[0020] The response time calculation module is used to calculate the response time matrix when the fitness of the online task and the execution node is sufficient to handle the task.
[0021] The list generation module is used for:
[0022] Based on the response time matrix, tasks are arranged in ascending order of response time to generate a task-execution node preference list. The task-execution node preference list includes online tasks, their corresponding execution nodes, and response times.
[0023] The response time matrix can be obtained using the following formula:
[0024] RT ij =WT ij +TET ij =ST ij +Δ ij +DMT ij +DAT ij +ET ij
[0025] Among them, RT ij WT represents response time; i represents the online task; j represents the execution node; WT ij For waiting time; TET ij Total execution time; ST ij For scheduling time; Δ ij For delay time; DMT ij Data migration time; DAT ij For data access time; ET ij Execution time.
[0026] In one embodiment, migrating the data source corresponding to the required dataset to the dataset stored on the target execution node includes:
[0027] The following formula can be used to calculate the data migration time from the data source corresponding to the required dataset to the dataset stored on the target execution node:
[0028]
[0029] Among them, DMT ij For online tasks t i Multiple required datasets are not located on the target execution node m j Total data migration time; For a single desired dataset, from the execution node m that stores the desired dataset l Migrate to target execution node m j Time; r l For execution node m l Read speed; w j For execution node m l write speed; b lj For execution node m l and target execution node m j Connection bandwidth between; v k Required dataset size; M k A collection of execution nodes for storing the required dataset; This is the desired dataset set.
[0030] In one embodiment, the adaptive scheduling system for big data analytics tasks also includes a dataset management module;
[0031] The dataset management module is used to generate a dataset-execution node allocation matrix based on the historical task-execution node preference list and historical scheduling decisions, and to update the datasets stored in the execution nodes based on the dataset-execution node allocation matrix.
[0032] In one embodiment, the adaptive scheduling system for big data analytics tasks further includes a state update module;
[0033] The status update module is used to update the set of tasks to be completed, the resources of the execution nodes, and the task-execution node preference list according to the scheduling decision, until the task-execution node preference list is empty.
[0034] Secondly, this application also provides an adaptive scheduling method for big data analysis tasks, including:
[0035] Retrieve online tasks and generate a set of tasks to be completed based on them; the set of tasks to be completed includes a unique identifier and the corresponding online task; the online task includes task attribute data, required dataset, and arrival time;
[0036] Obtain the execution node and its corresponding resources;
[0037] Based on the execution node and the corresponding resources, calculate the response time of each online task in the task set to be completed on different execution nodes to obtain the response time matrix.
[0038] Based on the response times in the response time matrix arranged in ascending order, a task-execution node preference list is obtained;
[0039] Based on the task-execution node preference list, online tasks are assigned to execution nodes, generating scheduling decisions; the scheduling decisions include online tasks and their corresponding target execution nodes;
[0040] When the required dataset for an online task is not stored in the target execution node, the data source corresponding to the required dataset is migrated to the dataset stored in the target execution node.
[0041] In one embodiment, migrating the data source corresponding to the required dataset to the dataset stored on the target execution node includes:
[0042] Obtain the storage information of the required dataset; the storage information includes the execution nodes where the dataset is stored and the connection paths between the stored execution nodes and the target execution node;
[0043] The data migration time is calculated based on the storage situation; the data migration time is the time it takes for the storage execution node to migrate the required dataset to the target execution node.
[0044] Select the corresponding connection path with the shortest data migration time, and copy the data source to the dataset stored on the target execution node according to the data replication strategy; the replication instruction creates the required dataset copy in the dataset stored on the target execution node.
[0045] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the adaptive scheduling method for any of the above-mentioned big data analysis tasks.
[0046] Fourthly, this application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the adaptive scheduling method for any of the above-mentioned big data analysis tasks.
[0047] The aforementioned adaptive scheduling system and method for big data analytics tasks utilizes a data-driven task scheduling mechanism to ensure that each task is processed in an orderly manner, avoiding task omissions or chaos. It can adaptively allocate resources and balance load based on the actual status of execution nodes, including stored datasets, and rationally select suitable nodes for task execution, improving the success rate and efficiency of task execution. By optimizing multi-source data scheduling based on response time, calculating the response time matrix and generating a preference list, it provides a quantitative basis for matching tasks with execution nodes. This makes scheduling decisions more scientific, prioritizing task-execution node combinations with shorter response times, effectively reducing the overall task response time and improving system real-time performance. The concurrent task scheduling method for large-scale nodes optimizes system resource utilization through orderly allocation, improving system throughput and enabling the system to handle more tasks simultaneously. Data migration is performed when the dataset required by the task is not stored on the target execution node, supporting a low-latency migration mechanism for data localization, ensuring the continuity of task execution. By migrating the data source to the target node, data localization is achieved, reducing data access latency and thus improving task execution efficiency. Attached Figure Description
[0048] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0049] Figure 1This is a structural diagram of the adaptive scheduling system for big data analysis tasks according to the present invention.
[0050] Figure 2 This is a flowchart illustrating the adaptive scheduling method for big data analysis tasks according to the present invention.
[0051] Figure 3 This is a step-by-step flowchart for step S206. Detailed Implementation
[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0053] In one embodiment, such as Figure 1 As shown, an adaptive scheduling system for big data analysis tasks is provided. This embodiment illustrates the system's application to a terminal. It is understood that this system can also be applied to a server, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. This embodiment includes:
[0054] The task management module is used to receive online tasks and assign unique identifiers to them to form a set of tasks to be completed. Online tasks include task attribute data, required dataset, and arrival time.
[0055] Indicatively, the task management module serves as the interface between the entire system and external task inputs. Online tasks entering the system include task attribute data such as task content, instruction length, required memory, and processor performance. The required dataset includes the required dataset content and size. Arrival time refers to the time an online task arrives at the system, serving as a crucial benchmark for measuring the entire task execution cycle. Furthermore, to achieve effective management of numerous tasks, the task management module assigns a unique identifier to each received online task. Throughout the task's lifecycle, the system accurately identifies and tracks tasks using these identifiers. All received online tasks are integrated into a set of tasks to be completed based on their assigned unique identifiers. In subsequent processing, the system uses this set of tasks to be completed, combined with information provided by other modules, to schedule and process the tasks.
[0056] The execution node management module is used to obtain the resources of the execution nodes in real time and evaluate the suitability of online tasks with the execution nodes based on the resources; the resources include the datasets stored on the execution nodes.
[0057] To illustrate, the execution node management module continuously collects resource information from each execution node. Resources include the datasets stored on the execution node, and, for example, computing resources such as the number of CPU (processor) cores, the performance per core (MIPS), memory capacity, storage capacity, and network bandwidth. The performance of each core determines the speed at which the execution node processes task instructions; memory capacity limits the amount of data that can be loaded and processed simultaneously; storage capacity affects the scale of data that can be stored; and network bandwidth relates to the speed of data migration and transmission. By acquiring this resource information in real time, the module can accurately grasp the real-time status of each execution node.
[0058] After acquiring the resource information of the execution nodes, the execution node management module uses big data analytics to evaluate the suitability of online tasks with the execution nodes based on the resources. For example, for the dataset required by an online task, the module checks whether the execution node already stores it. If the execution node contains all or part of the dataset required by the online task, it has an advantage in data access, reducing data migration time. Regarding computing resources, the module calculates the CPU utilization, memory, storage capacity, and load of the task on that execution node. The execution node management module provides a suitability evaluation result for each online task and each execution node, providing crucial information for subsequent task scheduling.
[0059] The scheduling analysis module is used to calculate the response time matrix of each online task in the task set to be completed on different execution nodes based on fitness, and generate a task-execution node preference list.
[0060] After obtaining the fitness evaluation results of each online task in the task set and different execution nodes, the scheduling analysis module begins to calculate the response time matrix. Specifically, the response time of the online task on the execution node is only calculated when the fitness of the online task and the execution node is considered suitable. Furthermore, for each task, it iterates through all execution nodes. When calculating the response time of a task on a particular execution node, several key factors are considered, including waiting time, data migration time, data access time, and execution time. Waiting time depends on the task's waiting order in the task queue and the time it takes for resources to meet the requirements. Data migration time refers to the time required to migrate data from other storage nodes if the dataset needed by the task is not stored on that execution node. Data access time refers to the time it takes for the online task to read the required dataset locally on the execution node. Execution time refers to the processing time of the online task's instruction length on the execution node. These time factors are summed to obtain the response time of the online task on that execution node. The response time of each task is calculated on all execution nodes that meet the fitness requirements, forming a response time matrix. Each element in the matrix represents the response time of a task on a specific execution node.
[0061] After obtaining the response time matrix, the scheduling analysis module generates a task-execution node preference list based on the data in the matrix. Illustratively, the task-execution node preference list is generated based on the principle of shortest response time, matching each task with the execution node with the shortest response time and arranging them in ascending order of response time. Furthermore, online tasks and their corresponding execution nodes that appear earlier in the list have higher priority during task scheduling and are selected for execution first. For example, the task-execution node preference list is a three-dimensional array containing online tasks, execution nodes, and response times.
[0062] The scheduling decision module is used to generate scheduling decisions based on the order of the task-execution node preference list; the scheduling decisions include online tasks and their corresponding target execution nodes.
[0063] After obtaining the task-execution node preference list, the scheduling decision module processes them sequentially according to a predefined order. For example, the order in the list is based on the task's response time on different execution nodes, from shortest to longest. A combination listed earlier indicates that the task will have the shortest response time on that execution node. Starting from the beginning of the list, the scheduling decision module selects the first task-execution node combination, identifies the online task within it as the online task to be executed, and determines the corresponding execution node as the target execution node, thus generating a scheduling decision.
[0064] Furthermore, during the generation of scheduling decisions, the scheduling decision module also considers the real-time status of the system, such as the current load of execution nodes and changes in the task queue. For example, if the scheduling process detects that the load of the target execution node is suddenly too high, which may affect the execution efficiency of the task, the scheduling decision module will adjust according to a preset strategy, such as temporarily skipping the combination and selecting the next combination with a shorter response time and a relatively reasonable load on the execution nodes. In this way, the scheduling decision module can ensure that the generated scheduling decisions are based on the optimal execution time of the task and can adapt to the dynamic changes of the system, ensuring the efficient and stable operation of the entire task scheduling system.
[0065] The data migration module is used to migrate the data source corresponding to the required dataset to the dataset stored on the target execution node when the required dataset for an online task is not stored on the target execution node.
[0066] To illustrate, the data migration module checks whether the dataset required by the online task is already stored on the target execution node. If the required dataset is not stored on the target execution node, the data migration module will start the data migration process. Specifically, the data migration module locates the node or storage device where the data source of the required dataset is distributed, and then selects the optimal data migration path and method based on factors such as network conditions, the read speed of the data source node, the write speed of the target execution node, and the size of the dataset.
[0067] Furthermore, during the data migration process, to ensure data integrity and accuracy, the module performs data verification and error handling. For example, if a network failure or data transmission error occurs during migration, the data migration module will attempt to retransmit the data or take other recovery measures. Illustratively, the module also considers the storage capacity and structure of the target execution node to ensure that the migrated data can be correctly stored on the target execution node and will not affect the storage and use of other data on the target execution node.
[0068] In the aforementioned adaptive scheduling system for big data analytics tasks, the modules collaborate to enable dynamic self-adaptation. The execution node management module monitors resource changes in real time, while the scheduling module dynamically adjusts scheduling decisions based on resource changes and task status. When system load changes or task requirements evolve, the system can quickly adapt, ensuring tasks are always rationally scheduled and executed efficiently, thus enhancing system stability and reliability.
[0069] In one embodiment, the execution node management module includes a processor management unit, a storage capacity management unit, and a load management unit.
[0070] The processor management unit is used to calculate the processor utilization of each online task on the execution node; if the processor utilization is greater than 1, the fitness is determined to be unable to handle the task; if the processor utilization is less than or equal to 1, the fitness is determined to be able to handle the task.
[0071] The processor management unit is primarily responsible for evaluating whether the processors of the execution nodes can handle online tasks. Illustratively, the processor management unit calculates the processor utilization of each online task on the execution nodes. Processor utilization is calculated based on the processor resources required by the task and the currently available processor resources on the execution node. If the calculated processor utilization is greater than 1, it means that the processor resources required by the task exceed the processing capacity of the execution node; in this case, the execution node cannot handle the task, and its fitness is determined as "unable to handle the task." Conversely, if the processor utilization is less than or equal to 1, it indicates that the processors of the execution node have sufficient capacity to handle the task, and its fitness is "able to handle the task."
[0072] The storage capacity management unit is used to process the stored content according to a preset storage processing mode when the remaining storage capacity in the resources is insufficient to meet the storage requirements of the dataset.
[0073] The storage content is processed according to a preset storage processing mode, including:
[0074] When the remaining storage capacity is insufficient to store the required dataset, the dataset stored on some execution nodes is migrated according to the data replication strategy and task dependencies.
[0075] If, after migrating some of the execution nodes' stored datasets, the remaining storage capacity is still insufficient to store the required datasets, the fitness is determined to be "unable to support the task"; if the remaining storage capacity is sufficient to store the required datasets, the fitness is determined to be "able to support the task".
[0076] The storage capacity management unit focuses on the storage resources of the execution nodes. When matching online tasks, the storage capacity management unit checks whether the remaining storage capacity of the execution nodes is sufficient to meet the requirements of the dataset needed for the task. If the remaining storage capacity is insufficient, the storage capacity management unit processes the stored content according to a preset storage processing mode. For example, based on the data replication strategy and task dependencies, the storage capacity management unit migrates a portion of the dataset stored in the execution node to free up storage space. If, after these operations, the remaining storage capacity still cannot meet the task requirements, then the fitness of the execution node for that task is determined to be insufficient; if the requirements are met, the fitness is determined to be sufficient to handle the task.
[0077] The load management unit checks the load value in the resources. If the load is greater than a preset threshold, the fitness is determined to be unable to handle the task; if the load is less than the preset threshold, the fitness is determined to be able to handle the task.
[0078] The load management unit checks the load value of the execution nodes, which reflects the current workload of the execution nodes. The system presets a load threshold. When the load of an execution node exceeds this threshold, it indicates that the node is operating under high load and can no longer effectively handle new tasks. In this case, the fitness is determined to be unable to handle tasks. If the load does not exceed the preset threshold, it indicates that the execution node is capable of handling new tasks, and the fitness is determined to be able to handle tasks.
[0079] In one embodiment, the scheduling analysis module includes a response time calculation module and a list generation module.
[0080] The response time calculation module is used to calculate the response time matrix when the fitness of the online task and the execution node is sufficient to handle the task.
[0081] To illustrate, for each online task i that can be carried by execution node j, it iterates through all suitable execution nodes to calculate the response time RT. ij Calculate the waiting time (WT). ij By scheduling time ST ij and delay time Δ ij Composition. Scheduling time ST ij The delay time Δ depends on the queuing order of the tasks in the waiting queue. ij Total Execution Time (TET) refers to the time when the execution node's resources are sufficient to execute a task, but it must wait for other tasks to complete to obtain sufficient resources. This time is calculated based on the estimated completion time of existing tasks on the execution node. ij Covering Data Migration Time (DMT) ij Data Access Time (DAT) ij and execution time ET ij Data migration time (DAT) ij The calculation must consider factors such as data storage location, source node read speed, target execution node write speed, and network bandwidth between nodes. If the data is already local, the migration time is 0. Data access time (DAT) ij This is the time it takes for the task to read the required dataset locally on the execution node, and it is related to the dataset size and the execution node's reading speed. Execution Time ET ij The response time (RT) is determined by the task's instruction length and the processing power of the execution node. By comprehensively calculating these time factors, the response time (RT) of each task on different execution nodes is obtained. ij This leads to the formation of a response time matrix, where each element corresponds to the response time of a task on a specific execution node.
[0082] The list generation module is used for:
[0083] Based on the response time matrix, tasks are arranged in ascending order of response time to generate a task-execution node preference list. The task-execution node preference list includes online tasks, their corresponding execution nodes, and response times.
[0084] The list generation module takes a response time matrix as input. It arranges the elements in the matrix in ascending order of response time. During the arrangement process, each task, its corresponding execution node, and the calculated response time are combined into a triplet, forming a task-execution node preference list. Task-execution node combinations that appear earlier in the list have shorter response times, meaning they have higher priority during task scheduling and are more suitable for being selected for execution first. This provides a clear priority order for task allocation for the subsequent scheduling decision module.
[0085] The response time matrix can be obtained using the following formula:
[0086] RT ij =WT ij +TET ij =ST ij +Δ ij +DMT ij +DAT ij +ET ij
[0087] Among them, RT ij WT represents response time; i represents the online task; j represents the execution node; WT ij For waiting time; TET ij Total execution time; ST ij For scheduling time; Δ ij For delay time; DMT ij Data migration time; DAT ij For data access time; ET ij Execution time.
[0088] In one embodiment, migrating the data source corresponding to the required dataset to the dataset stored on the target execution node includes:
[0089] The following formula can be used to calculate the data migration time from the data source corresponding to the required dataset to the dataset stored on the target execution node:
[0090]
[0091] Among them, DMT ij For online tasks t i Multiple required datasets are not located on the target execution node m j Total data migration time; For a single desired dataset, from the execution node m that stores the desired dataset l Migrate to target execution node m j Time; r l For execution node m l Read speed; w j For execution node m l write speed; b lj For execution node m l and target execution node m j Connection bandwidth between; v k Required dataset size; M k A collection of execution nodes for storing the required dataset; This is the desired dataset set.
[0092] In one embodiment, the adaptive scheduling system for big data analytics tasks also includes a dataset management module.
[0093] The dataset management module is used to generate a dataset-execution node allocation matrix based on the historical task-execution node preference list and historical scheduling decisions, and to update the datasets stored in the execution nodes based on the dataset-execution node allocation matrix.
[0094] The dataset management module relies on historical task-execution node preference lists and historical scheduling decision information accumulated during past system operations. This includes, illustratively, the allocation of each task to its corresponding execution node and related dataset information involved in task execution. The module conducts in-depth analysis of the historical task-execution node preference list and historical scheduling decisions to identify the correlation between each dataset and execution node. Specifically, the dataset management module statistically analyzes which datasets were frequently assigned to which execution nodes in past task executions. For example, if dataset A is frequently assigned to execution node 1 in multiple task schedulings, the corresponding element values for dataset A and execution node 1 in the dataset-execution node allocation matrix will increase accordingly, thus constructing a matrix reflecting the probability or frequency of dataset allocation on different execution nodes. Furthermore, based on the generated dataset-execution node allocation matrix, the dataset management module optimizes and updates the datasets stored on the execution nodes. For datasets that show a high correlation with a particular execution node in the dataset-execution node allocation matrix but are not currently stored on that execution node, the module will consider migrating them to that execution node to improve the local availability of data during future task executions and reduce the time overhead of data migration. Conversely, for datasets with extremely low correlation to a particular execution node that are currently stored on that execution node, the module may choose to delete them or migrate them to other more suitable nodes based on the actual storage situation and system policies to free up storage space on the execution node and improve the utilization efficiency of storage resources.
[0095] In one embodiment, the adaptive scheduling system for big data analytics tasks further includes a state update module;
[0096] The status update module is used to update the set of tasks to be completed, the resources of the execution nodes, and the task-execution node preference list according to the scheduling decision, until the task-execution node preference list is empty.
[0097] As an illustration, when the system executes a scheduling decision and assigns certain online tasks to the corresponding target execution nodes, the status update module removes these assigned tasks from the task set to be completed, ensuring that the task set to be completed always accurately reflects the status of tasks that have not yet been assigned. For example, if the scheduling decision assigns task A to execution node X, the status update module will immediately delete task A from the task set to be completed, avoiding duplicate processing of already assigned tasks.
[0098] To illustrate, the resource usage of an execution node changes after it accepts a task. The status update module monitors and updates the resource information of the execution node in real time. For processor resources, it updates the available CPU resources of the execution node based on the processor utilization of the task; for storage resources, it adjusts the remaining storage capacity of the execution node based on the storage status of the dataset required by the task; for network resources, it updates the network bandwidth usage status based on data migration and transmission, etc. For example, if task B runs on execution node Y and consumes a certain amount of memory and CPU time, the status update module will correspondingly reduce the available memory and CPU time of execution node Y so that the resource status of the execution node can be accurately considered in subsequent scheduling decisions.
[0099] As an illustration, the task-execution node preference list is generated based on factors such as the response time of tasks on different execution nodes. After a task is assigned and executed, its related response time and other information may change, or the response time of other tasks on that execution node may also change due to changes in the node's resource status. Therefore, the status update module recalculates the response time of tasks on different execution nodes based on the new task execution status and execution node resource status, thereby updating the task-execution node preference list. For example, task C originally had a long response time on execution node Z, but because execution node Z released some resources after executing other tasks, the recalculated response time of task C became shorter, and the status update module adjusted the position of task C in the task-execution node preference list accordingly.
[0100] The status update module continues to perform the above update operations until the task-execution node preference list is empty, which means that all tasks have been successfully assigned and executed. During this process, the system continuously adjusts dynamically based on task execution status and resource changes to achieve efficient task scheduling and rational resource utilization. Furthermore, upon receiving a new online task, the status update module updates the set of tasks awaiting completion.
[0101] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0102] Based on the same inventive concept, this application also provides an adaptive scheduling method for big data analysis tasks to implement the adaptive scheduling system for big data analysis tasks described above. The solution provided by this method is similar to the implementation scheme described in the above system. Therefore, the specific limitations in one or more embodiments of the adaptive scheduling method for big data analysis tasks provided below can be found in the limitations of the adaptive scheduling system for big data analysis tasks described above, and will not be repeated here.
[0103] In one exemplary embodiment, such as Figure 2 As shown, an adaptive scheduling method for big data analysis tasks is provided, including:
[0104] S201. Obtain online tasks and generate a set of tasks to be completed based on the online tasks; the set of tasks to be completed includes a unique identifier and the corresponding online task; the online task includes task attribute data, required dataset and arrival time.
[0105] The system receives online tasks from external data sources, such as user submissions and transmissions from other business systems. Each online task contains key information such as task attribute data, required dataset, and arrival time. The system assigns a unique identifier to each acquired online task, and uses this identifier to integrate all online tasks into a set of tasks to be completed, facilitating subsequent unified management and scheduling.
[0106] S202, Obtain the execution node and its corresponding resources.
[0107] The system monitors the running status of execution nodes in real time, acquiring resource information for each node, including processor performance, storage capacity, network bandwidth, and the datasets currently stored on the node. This real-time acquisition of execution node resources provides fundamental data for assessing the compatibility between tasks and nodes.
[0108] S203. Based on the execution node and the corresponding resources, calculate the response time of each online task in the task set to be completed on different execution nodes to obtain the response time matrix.
[0109] For each online task in the task set to be completed, all execution nodes are traversed, and the response time of the task on that execution node is calculated based on the overall fitness. By performing this calculation on each task on different execution nodes, a response time matrix is formed, and the elements in the matrix correspond to the response time of each task on a specific execution node.
[0110] S204. Arrange the response times in the response time matrix in ascending order to obtain the task-execution node preference list.
[0111] Based on the response time matrix, the system arranges the response times of each task combined with different execution nodes in ascending order. The tasks, their corresponding execution nodes, and response times are then integrated into an ordered task-execution node preference list. Combinations listed earlier in the list have shorter response times, meaning they have higher priority during task scheduling.
[0112] S205. Based on the task-execution node preference list, assign online tasks to execution nodes and generate scheduling decisions; the scheduling decisions include online tasks and their corresponding target execution nodes.
[0113] Online tasks are assigned to their corresponding execution nodes in the order of the task-execution node preference list. During the assignment process, the resource status of the execution nodes is fully considered to ensure that tasks can be executed smoothly on the target execution nodes. Each time a task assignment is completed, a scheduling decision is generated, which clarifies the online task and its corresponding target execution node.
[0114] S206. When the required dataset for an online task is not stored in the target execution node, migrate the data source corresponding to the required dataset to the dataset stored in the target execution node.
[0115] After task allocation, the system checks whether the dataset required by the online task is stored on the target execution node. If the required dataset is not stored on the target execution node, the system initiates a data migration process. The system locates the data source of the required dataset and, based on factors such as network conditions, the read speed of the data source node, the write speed of the target execution node, and the size of the dataset, selects the optimal data migration path and method to migrate the data source to the target execution node's storage dataset, ensuring that the required data can be obtained promptly during task execution.
[0116] In one embodiment, such as Figure 3 As shown, the data source corresponding to the required dataset is migrated to the dataset stored on the target execution node, including:
[0117] S301. Obtain the storage status of the required dataset; the storage status includes the execution node of the storage and the connection path between the execution node of the storage and the target execution node.
[0118] During task scheduling, when the target execution node is identified and it is found that the dataset required by the online task is not stored on that node, the storage status of the required dataset is actively obtained. This includes querying all execution nodes that store the dataset, as well as the connection path information between these storage nodes and the target execution node. The connection path information covers key data such as network topology and bandwidth between nodes. By obtaining the storage location and connection path of the dataset, a basis is provided for the reasonable planning of data migration.
[0119] S302. Calculate the data migration time based on the storage situation; the data migration time is the time it takes for the storage execution node to migrate the required dataset to the target execution node.
[0120] Based on the obtained dataset storage information, for each execution node storing the dataset, the time to migrate the required dataset to the target execution node is calculated. The data migration time calculation comprehensively considers multiple factors, including the dataset size, the read speed of the storage node, the write speed of the target node, and the bandwidth of the connection path. Specifically, the data migration time is directly proportional to the dataset size and inversely proportional to the read speed of the storage node, the write speed of the target node, and the bandwidth of the connection path. By accurately calculating the data migration time for each possible migration path, quantitative support is provided for selecting the optimal migration scheme.
[0121] S303. Select the corresponding connection path with the shortest data migration time, and copy the data source to the data set stored on the target execution node according to the data replication strategy; the replication instruction creates the required dataset copy in the data set stored on the target execution node.
[0122] The calculated data migration times for each connection path are compared, and the connection path with the shortest data migration time is selected as the optimal migration path. After determining the optimal path, the data source is copied from the storage node to the storage dataset on the target execution node according to a pre-defined data replication strategy. For example, the data replication strategy may include full replication, incremental replication, etc. The essence of the replication operation is to create a copy of the required dataset in the storage dataset on the target execution node to ensure that the task can quickly access the required data during execution, reducing data retrieval latency and improving task execution efficiency.
[0123] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.
[0124] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.
[0125] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0126] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.
Claims
1. An adaptive scheduling system for big data analysis tasks, characterized in that, include: The task management module is used to receive online tasks and assign unique identifiers to the online tasks to form a set of tasks to be completed; The online task includes task attribute data, required dataset, and arrival time; The execution node management module is used to acquire the resources of the execution node in real time and evaluate the suitability of the online task with the execution node based on the resources; the resources include the dataset stored by the execution node; The scheduling analysis module is used to calculate the response time matrix of each online task in the task set to be completed on different execution nodes based on the fitness, and generate a task-execution node preference list. The scheduling decision module is used to generate scheduling decisions based on the order of the task-execution node preference list; The scheduling decision includes online tasks and corresponding target execution nodes; The data migration module is used to migrate the data source corresponding to the required dataset to the dataset stored on the target execution node when the required dataset for the online task is not stored on the target execution node.
2. The system according to claim 1, characterized in that: The execution node management module includes a processor management unit, a storage capacity management unit, and a load management unit; The processor management unit is used to calculate the processor utilization of each online task on the execution node; if the processor utilization is greater than 1, the fitness is determined to be unable to handle the task; if the processor utilization is less than or equal to 1, the fitness is determined to be able to handle the task. The storage capacity management unit is used to process the stored content according to a preset storage processing mode when the remaining storage capacity in the resources is insufficient to store the required dataset. The load management unit is used to check the load value in the resource. If the load is greater than a preset threshold, the fitness is determined to be unable to bear the task. If the load is less than a preset threshold, the fitness level is considered to be able to handle the task. The step of processing the stored content according to a preset storage processing mode includes: When the remaining storage capacity is insufficient to store the required dataset, the dataset stored on some of the execution nodes is migrated according to the data replication strategy and task dependencies. If, after migrating some of the datasets stored on the execution nodes, the remaining storage capacity is still insufficient to store the required dataset, the fitness is determined to be unable to support the task; if the remaining storage capacity is sufficient to store the required dataset, the fitness is determined to be able to support the task.
3. The system according to claim 2, characterized in that: The scheduling analysis module includes a response time calculation module and a list generation module; The response time calculation module is used to calculate the response time matrix when the fitness of the online task and the execution node is sufficient to handle the task. The list generation module is used for: Based on the response time matrix, the task-execution node preference list is generated by arranging the response times in ascending order; the task-execution node preference list includes online tasks, corresponding execution nodes, and response times. The response time matrix can be obtained using the following formula: RT ij =WT ij +TET ij =ST ij +Δ ij +DMT ij +THAT ij +ET ij Among them, RT ij WT represents the response time; i represents the online task; j represents the execution node; WT ij Waiting time; TET ij Total execution time; ST ij For scheduling time; Δ ij For delay time; DMT ij Data migration time; DAT ij For data access time; ET ij Execution time.
4. The system according to claim 3, characterized in that, The step of migrating the data source corresponding to the required dataset to the dataset stored on the target execution node includes: The data migration time for migrating the data source corresponding to the required dataset to the dataset stored in the target execution node can be obtained using the following formula: Among them, DMT ij For the online task t i The multiple required datasets are not located on the target execution node m j The total data migration time mentioned above; For a single desired dataset, from the execution node m that stores the desired dataset l Migrate to the target execution node m j Time; r l For execution node m l Read speed; w j For execution node m l write speed; b lj For execution node m l and target execution node m j The connection bandwidth between them; v k Required dataset size; M k A collection of execution nodes for storing the required dataset; This is the desired dataset set.
5. The system according to claim 3, characterized in that: The adaptive scheduling system for big data analysis tasks also includes a dataset management module; The dataset management module is used to generate a dataset-execution node allocation matrix based on the historical task-execution node preference list and the historical scheduling decisions, and to update the dataset stored in the execution node based on the dataset-execution node allocation matrix.
6. The system according to claim 1, characterized in that: The adaptive scheduling system for the big data analysis task also includes a status update module; The status update module is used to update the set of tasks to be completed, the resources of the execution nodes, and the task-execution node preference list according to the scheduling decision, until the task-execution node preference list is empty.
7. An adaptive scheduling method for big data analysis tasks, characterized in that, The method includes: Obtain online tasks and generate a set of tasks to be completed based on the online tasks; the set of tasks to be completed includes a unique identifier and a corresponding online task; the online task includes task attribute data, required dataset, and arrival time; Obtain the execution node and its corresponding resources; Based on the execution node and the corresponding resource, calculate the response time of each online task in the task set to be completed on different execution nodes to obtain a response time matrix; Based on the response times in the response time matrix arranged in ascending order, a task-execution node preference list is obtained; Based on the task-execution node preference list, the online tasks are assigned to execution nodes, generating a scheduling decision; the scheduling decision includes the online tasks and their corresponding target execution nodes. When the required dataset for the online task is not stored in the target execution node, the data source corresponding to the required dataset is migrated to the dataset stored in the target execution node.
8. The method according to claim 7, characterized in that, The step of migrating the data source corresponding to the required dataset to the dataset stored on the target execution node includes: Obtain the storage status of the required dataset; the storage status includes the stored execution nodes and the connection paths between the stored execution nodes and the target execution node; The data migration time is calculated based on the storage situation; the data migration time is the time it takes for the execution node in the storage to migrate the required dataset to the target execution node. Select the connection path with the shortest data migration time, and copy the data source to the dataset stored on the target execution node according to the data replication strategy; the replication instruction creates the required dataset copy in the dataset stored on the target execution node.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 7 to 8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 7 to 8.
