Resource scheduling method and device for YARN task, electronic equipment and storage medium
By using GPT algorithm models to predict resource requirements and formulate scheduling strategies in YARN clusters, the problem that existing technology cannot cope with complex and changing tasks and resource requirements is solved, and resource utilization and task execution efficiency are improved.
Patent Information
- Application Number
- CN202411256962.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-09
- Publication Date
- 2025-02-14
AI Technical Summary
The existing technology cannot effectively respond to complex and changeable tasks and resource requirements, resulting in uneven resource allocation and low utilization rates, and even trigger performance bottlenecks during peak periods, limiting the efficiency and potential of big data processing and analysis.
By obtaining the relevant data of the YARN cluster, the training set and test set are obtained, the GPT algorithm model is trained using self-supervised learning, the model is optimized to predict the resource requirements of the target task, and the resource scheduling strategy is formulated based on the resource requirements and cluster resource status, and YARN is called The cluster performs target tasks.
The resource scheduling strategy is optimized, the efficiency and resource utilization of target tasks in the cloud computing environment are improved, the waiting time and execution time are reduced, and the efficiency and performance of task execution are improved.
Smart Images
Figure CN119440799A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of cloud computing and task management, and in particular to a resource scheduling method, device, electronic device and storage medium for YARN tasks. Background Art
[0002] With the rapid development of cloud computing technology, YARN, as the cornerstone of the Hadoop ecosystem, shoulders the mission of efficient scheduling and management of cluster resources, ensuring that resources are allocated on demand and optimizing processing performance. Its importance is self-evident and plays a key role in promoting the development of big data processing and cloud computing applications.
[0003] However, faced with increasingly complex and ever-changing business scenarios and task requirements, the traditional YARN resource scheduling algorithm has gradually revealed its limitations due to its inherent static rules and heuristic methods. Existing technologies find it difficult to accurately capture and dynamically respond to subtle changes in resource supply and demand, which in turn leads to uneven resource allocation and low utilization, and even causes performance bottlenecks during peak hours, limiting the efficiency and potential of big data processing and analysis.
[0004] Therefore, there is an urgent need for a resource scheduling method for YARN tasks that can cope with complex and changeable tasks and resource requirements, thereby improving the efficiency and flexibility of resource scheduling. Summary of the invention
[0005] The embodiments of the present invention provide a resource scheduling method for YARN tasks to solve the problems that the prior art cannot cope with complex and changeable tasks and resource requirements, and has uneven resource allocation and low utilization. The technical solution is as follows:
[0006] According to one aspect of the present invention, a resource scheduling method for YARN tasks, the method comprising: obtaining relevant data of a YARN cluster, processing the relevant data to obtain a training set and a test set; the relevant data comprises task data, resource usage data and status data; training a GPT algorithm model using the training set in a self-supervised learning manner, evaluating the GPT algorithm model using the test set and optimizing the GPT algorithm model according to the evaluation result; obtaining feature information of a target task, obtaining resource requirements of the target task according to the feature information through the GPT algorithm model; the feature information comprises task type, data size and task dependency; formulating a resource scheduling strategy according to the resource requirements and the resource status of the YARN cluster, and calling the YARN cluster to execute the target task according to the scheduling strategy; the resource scheduling strategy comprises task priority, and the allocation order and quantity of resources.
[0007] In one embodiment, the relevant data is processed to obtain a training set and a test set by the following steps: the relevant data is cleaned to remove noise and outliers, and the relevant data is formatted so that the format of the relevant data meets the requirements of the GPT algorithm model; the key features of the relevant data are extracted, and the processed relevant data and key features are used as the training set and the test set; the key features include task type, data size, and the correlation between tasks.
[0008] In one of the embodiments, obtaining the resource requirement of the target task according to the characteristic information through the GPT algorithm model is achieved by the following steps: inputting the characteristic information into the GPT algorithm model to predict the required resource amount of the target task; the required resource amount includes CPU and memory; setting the priority, resource allocation order and quantity of the target task according to the priority of the target task, task association relationship and required resource amount and using them as the resource requirement of the target task.
[0009] In one of the embodiments, formulating a resource scheduling strategy according to the resource demand and the resource status of the YARN cluster is achieved by the following steps: obtaining the resource usage and resource availability of the YARN cluster; the resource usage includes the total amount, allocated amount and free amount of each resource; formulating a resource scheduling strategy for the target task according to the resource demand of the target task and the resource usage and resource availability of the YARN cluster so that the allocation of resources can meet the demand of the target task and there is no waste of resources.
[0010] In one of the embodiments, calling the YARN cluster to execute the target task according to the scheduling strategy is achieved by the following steps: calling the YARN cluster to execute the target task according to the scheduling strategy, recording the execution process and execution result of the target task and detecting whether it is executed normally; if an error occurs during task execution, timely adjusting and optimizing the scheduling strategy to execute the target task.
[0011] In one of the embodiments, after executing the target task, the following steps are also included: continuously observing the resource usage of the YARN cluster and the execution status of the target task, wherein the execution status includes the execution progress and resource usage of the target task; and updating the GPT algorithm model using the resource usage of the YARN cluster and the execution status of the target task.
[0012] In one of the embodiments, the method further includes the following steps: constructing and storing a tree-like task graph based on the characteristic relationship of the target task; the root node of the tree-like task represents a task without a preceding dependency; before executing the target task, ensuring that the dependency relationships of each task are satisfied, executing the target task in the order of the tree-like task graph, and updating the node status after the target task is executed.
[0013] According to one aspect of the present invention, a resource scheduling device for YARN tasks includes: a data set preparation module, used to obtain relevant data of the YARN cluster, and process the relevant data to obtain a training set and a test set; a model training module, used to train a GPT algorithm model using the training set by self-supervised learning, and use the test set to evaluate the GPT algorithm model and optimize the GPT algorithm model according to the evaluation result; a resource demand analysis module, used to obtain feature information of a target task, and obtain the resource demand of the target task according to the feature information through the GPT algorithm model; a resource scheduling module, used to formulate a resource scheduling strategy according to the resource demand and the resource status of the YARN cluster, and call the YARN cluster to execute the target task according to the scheduling strategy.
[0014] According to one aspect of the present invention, an electronic device includes at least one processor and at least one memory, wherein the memory stores computer-readable instructions; the computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the resource scheduling method for YARN tasks as described above.
[0015] According to one aspect of the present invention, a storage medium stores computer-readable instructions thereon, wherein the computer-readable instructions are executed by one or more processors to implement the resource scheduling method for YARN tasks as described above.
[0016] The beneficial effects brought by the technical solution provided by the present invention are:
[0017] In the above technical scheme, the present invention first obtains relevant data of the YARN cluster, processes the relevant data to obtain a training set and a test set, trains the GPT algorithm model using the training set by self-supervised learning, evaluates the GPT algorithm model using the test set and optimizes the GPT algorithm model according to the evaluation result, obtains characteristic information of the target task, obtains the resource requirement of the target task according to the characteristic information through the GPT algorithm model, formulates a resource scheduling strategy according to the resource requirement and the resource status of the YARN cluster, calls the YARN cluster to execute the target task according to the scheduling strategy, and optimizes the resource scheduling strategy by combining YARN cluster resource management, data processing, self-supervised learning and the GPT algorithm model, improves the efficiency of target task execution and resource utilization in the cloud computing environment, can formulate a more reasonable resource scheduling strategy, helps to reduce task waiting time and execution time, and improves the efficiency and performance of task execution, thereby effectively solving the problems that the existing technology cannot cope with complex and changeable tasks and resource requirements, uneven resource allocation, and low and insufficient utilization. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in describing the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative work.
[0019] Figure 1 is a flowchart of a resource scheduling method for a YARN task according to an exemplary embodiment;
[0020] Figure 2 It is a schematic diagram of the resource scheduling method of YARN tasks in an application scenario;
[0021] Figure 3 This is a schematic diagram of the resource scheduling method for YARN tasks in another application scenario;
[0022] Figure 4 This is a schematic diagram of the resource scheduling method for YARN tasks in another application scenario;
[0023] Figure 5 yes Figure 4 A schematic diagram of constructing a task relationship diagram in the corresponding application scenario;
[0024] Figure 6 is a block diagram of a resource scheduling device for a YARN task according to an exemplary embodiment;
[0025] Figure 7 is a hardware structure diagram of an electronic device according to an exemplary embodiment;
[0026] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION
[0027] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.
[0028] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure refers to the presence of the features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.
[0029] The present invention provides a resource scheduling method for YARN tasks. By combining YARN cluster resource management, data processing, self-supervised learning and GPT algorithm model, the resource scheduling strategy is optimized, and the problem that the prior art cannot cope with complex and changeable tasks and resource requirements, uneven resource allocation and low utilization is solved. The resource scheduling method for YARN tasks is applicable to a resource scheduling device for YARN tasks, and the resource scheduling device for YARN tasks can be an electronic device. The resource scheduling method for YARN tasks in the embodiment of the present invention can be applied to a variety of scenarios, such as resource scheduling.
[0030] See also Figure 1 An embodiment of the present invention provides a resource scheduling method for a YARN task, which is applicable to an electronic device.
[0031] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.
[0032] like Figure 1 As shown, the method may include the following steps:
[0033] Step 110, obtain relevant data of the YARN cluster, and process the relevant data to obtain a training set and a test set.
[0034] Among them, relevant data includes task data, resource usage data and status data.
[0035] In one possible implementation, the relevant data is cleaned to remove noise and outliers, the relevant data is formatted so that the format of the relevant data meets the requirements of the GPT algorithm model, the key features of the relevant data are extracted, and the processed relevant data and key features are used as training sets and test sets.
[0036] Among them, the key features include task type, data size, and the relationship between tasks.
[0037] Step 130, train the GPT algorithm model using the training set through self-supervised learning, evaluate the GPT algorithm model using the test set, and optimize the GPT algorithm model based on the evaluation results.
[0038] Step 150, obtain characteristic information of the target task, and obtain the resource requirements of the target task according to the characteristic information through the GPT algorithm model.
[0039] The characteristic information includes task type, data size and task dependency.
[0040] In one possible implementation, feature information is input into the GPT algorithm model to predict the required resource amount of the target task, and the priority, resource allocation order and quantity of the target task are set according to the priority, task association and required resource amount of the target task and used as the resource requirement of the target task.
[0041] The required resources include CPU and memory.
[0042] Step 170: formulate a resource scheduling strategy according to resource requirements and the resource status of the YARN cluster, and call the YARN cluster to execute the target task according to the scheduling strategy.
[0043] Among them, the resource scheduling strategy includes the priority of the task, the allocation order and quantity of resources.
[0044] In one possible implementation, the resource usage and resource availability of the YARN cluster are obtained, and a resource scheduling strategy is formulated for the target task according to the resource requirements of the target task and the resource usage and resource availability of the YARN cluster so that resource allocation can meet the requirements of the target task without wasting resources.
[0045] The resource usage includes the total amount, allocated amount and free amount of each resource.
[0046] In one possible implementation, the YARN cluster is called to execute the target task according to the scheduling strategy, the execution process and results of the target task are recorded, and whether it is executed normally is detected. If an error occurs during task execution, the scheduling strategy is adjusted and optimized in time to execute the target task.
[0047] In one possible implementation, after executing the target task, the resource usage of the YARN cluster and the execution status of the target task are continuously observed, and the GPT algorithm model is updated using the resource usage of the YARN cluster and the execution status of the target task.
[0048] Among them, the execution status includes the execution progress of the target tasks and the resource usage.
[0049] In one possible implementation, a tree-like task graph is constructed and stored based on the characteristic relationships of the target tasks. Before executing the target task, it is determined that the dependencies of each task are satisfied. The target task is executed in the order of the tree-like task graph. After the target task is executed, the node status is updated.
[0050] Among them, the root node of the tree task represents a task without preceding dependencies.
[0051] Through the above process, the present invention first obtains relevant data of the YARN cluster, processes the relevant data to obtain a training set and a test set, uses the training set to train the GPT algorithm model in a self-supervised learning manner, uses the test set to evaluate the GPT algorithm model and optimizes the GPT algorithm model according to the evaluation result, obtains characteristic information of the target task, obtains the resource requirement of the target task according to the characteristic information through the GPT algorithm model, formulates a resource scheduling strategy according to the resource requirement and the resource status of the YARN cluster, calls the YARN cluster to execute the target task according to the scheduling strategy, and optimizes the resource scheduling strategy by combining YARN cluster resource management, data processing, self-supervised learning and the GPT algorithm model, improves the efficiency of target task execution and resource utilization in the cloud computing environment, can formulate a more reasonable resource scheduling strategy, helps to reduce task waiting time and execution time, and improves the efficiency and performance of task execution, thereby effectively solving the problems that the existing technology cannot cope with complex and changeable tasks and resource requirements, uneven resource allocation, and low and insufficient utilization.
[0052] In one application scenario, Figure 2 A flowchart of a resource scheduling method for YARN tasks is shown.
[0053] like Figure 2 As shown in the figure, the core process and module interaction of YARN task resource scheduling are demonstrated.
[0054] First, the specific features of each module include:
[0055] Data collection and processing module: responsible for collecting relevant data in the YARN cluster, including but not limited to historical task data, resource usage data, etc., and performing preprocessing and feature extraction to provide input for subsequent GPT algorithm models.
[0056] GPT algorithm model module: uses the collected data to train the model, and can perform in-depth analysis on the input data and generate task features based on the analysis results. These task features are an abstract representation of task resource requirements, providing an important basis for subsequent task scheduling and resource allocation.
[0057] Task configuration module: Based on the task features generated by the GPT algorithm model, further refine and configure the specific attributes of the task, such as the priority of the task, the amount of resources required, etc. This step is crucial for formulating a reasonable task scheduling strategy.
[0058] YARN resource monitoring module: provides real-time cluster resource information and allocates appropriate resources to each task. This involves effective management and optimization of cluster resources to ensure that tasks can be executed smoothly and maximize resource utilization.
[0059] Task decision module: Receives status updates from the task execution module and monitors and adjusts task execution based on these updates. This step ensures that tasks can be executed according to the predetermined scheduling strategy and resource allocation plan, and can respond and handle problems in a timely manner.
[0060] YARN resource monitoring module: Continuously monitors the resource usage of the cluster throughout the process, including the real-time status and change trends of resources such as CPU and memory. This module provides important reference information for the GPT algorithm model module and task resource allocation module, enabling them to more accurately predict task resource requirements and formulate corresponding scheduling strategies.
[0061] Among them, the task execution information, as the output of the entire process, records the results and status of task execution, including whether the task is successfully completed, resource usage, possible problems, etc. This information is of great significance for evaluating the effectiveness of scheduling solutions, optimizing resource allocation strategies, and improving model training.
[0062] like Figure 2 As shown, implementing YARN task resource scheduling according to the above modules may include the following steps:
[0063] Step 1: Data collection and processing.
[0064] Specifically, task data is collected from the historical records of the YARN cluster, including the task type, execution time, required resources (such as CPU, memory), etc. At the same time, cluster resource usage data is collected, such as the total number of CPUs, total memory, resource allocation of each node, etc. It is also necessary to collect cluster status data, such as the health of the cluster, the active status of the nodes, etc.
[0065] Furthermore, the collected data is cleaned to remove noise and outliers, and the data is formatted to ensure that the data format meets the requirements of the GPT algorithm model. Key features such as task type, data size, and the correlation between tasks are extracted. These features will be used to train the GPT algorithm model.
[0066] Step 2: GPT algorithm model training.
[0067] Specifically, the processed data set is used as input to train the GPT algorithm model. The GPT model learns and understands the relationship between tasks and resources, as well as the patterns and trends of resource usage through self-supervised learning. During the training process, the model will continuously optimize its parameters to improve the accuracy of predicting task resource requirements.
[0068] Furthermore, during or after the training process, a portion of the data that was not involved in the training is used to evaluate the model to evaluate the model's ability to predict task resource requirements and ensure that the model can accurately predict the resources required for new tasks.
[0069] Step 3: Task resource analysis and scheduling strategy formulation.
[0070] Specifically, the feature information of the newly submitted task is extracted, such as task type, data size, task association, etc., and this feature information is input into the trained GPT algorithm model to predict the amount of resources required for the task (such as CPU, memory).
[0071] Furthermore, based on the results of the task resource demand analysis and combined with the current resource status of the YARN cluster (provided by the YARN resource monitoring module), a reasonable resource scheduling strategy is formulated to determine the priority of tasks, the order and quantity of resource allocation, etc., to ensure the rational use of resources and efficient execution of tasks.
[0072] Step 4: Task execution and monitoring.
[0073] Specifically, the resources in the YARN cluster are called according to the scheduling strategy to execute tasks, and the execution process and results of the tasks are recorded for subsequent analysis and optimization.
[0074] Furthermore, we continuously monitor the resource usage of the YARN cluster, including the total amount, allocated amount, and idle amount of various resources, and monitor the execution of tasks, including the execution progress of tasks, resource usage, etc. If unreasonable resource usage or problems with task execution are found, we promptly adjust and optimize the scheduling strategy. At the same time, we use the monitored data to update the GPT algorithm model to improve the prediction accuracy of the model and the effectiveness of the scheduling strategy.
[0075] Through the above process, the embodiment of the present invention combines YARN cluster data collection and processing, GPT algorithm model training, task resource analysis and scheduling strategy formulation, and task execution and monitoring to achieve a significant improvement in resource utilization and optimization of task execution efficiency. The introduction of the GPT algorithm model makes resource scheduling more intelligent and precise, and can predict resource requirements based on task characteristics, thereby formulating reasonable scheduling strategies. The continuous monitoring and data feedback mechanism ensures the dynamic adjustment and optimization of the scheduling strategy, further improves the system's adaptability and stability, and provides strong support for resource management and task execution in a cloud computing environment.
[0076] In another application scenario, Figure 3 A flowchart of a resource scheduling method for YARN tasks is shown.
[0077] like Figure 3 As shown, resource scheduling of YARN tasks can include the following steps:
[0078] Step S1: Configure the basic information of the task.
[0079] Specifically, the basic attributes of each task are clarified, including task type, priority, quantity, and the hierarchical relationship between tasks. This information is the basis for formulating scheduling strategies and resource allocation in subsequent steps.
[0080] Step S2: Determine resource allocation using the GPT algorithm model.
[0081] Specifically, the GPT algorithm model is called to perform resource prediction according to the task characteristics configured in step S1. The GPT model predicts a reasonable resource allocation plan for each task, such as memory size and number of CPUs, by analyzing historical data and task characteristics.
[0082] Step S3: Construct a task graph.
[0083] Specifically, according to the superior-subordinate relationship of tasks, these tasks are organized into a tree structure, namely the task graph. In the task graph, each node represents a task and records the dependent superior data of the task. At the same time, the data of the task graph is saved in an array for subsequent management and query.
[0084] Step S4: Get the YARN cluster resource status.
[0085] Specifically, the YARN resource monitor is used to obtain the current resource status of the cluster, including the total amount of CPU and memory, idle amount, and the number of currently executing tasks. This information is crucial for formulating a reasonable task scheduling strategy.
[0086] Step S5: Count the number of tasks currently being executed.
[0087] Specifically, the redis database is used to count the number of tasks currently being executed by using a specific prefix (STATUS_HASH_PREFIX) and status (IN_PROGRESS). This number will be used to calculate the number of executable tasks later.
[0088] Step S6: Calculate the amount of executable tasks.
[0089] Specifically, the total number of tasks in the YARN resources minus the number of currently executed tasks counted in step S5 is used to calculate the number of tasks that can still be executed in the current cluster. This calculation result provides an important basis for subsequent task scheduling.
[0090] Step S7: Formulate a task scheduling strategy.
[0091] Specifically, a reasonable task scheduling strategy should be formulated based on the task priority, resource requirements, and cluster resource conditions. The strategy should ensure that high-priority tasks are executed first while avoiding resource overload and waste.
[0092] Step S8: Task execution and status update.
[0093] Specifically, the task is executed according to the established scheduling strategy. When the task is executed, the status of the corresponding task in redis is updated to "executing". After the task is completed, the status is updated to "completed" and the relevant information of the task execution is recorded in the log.
[0094] Through the above process, the embodiment of the present invention realizes efficient scheduling and resource optimization of tasks in a cloud computing environment by configuring basic task information, using the GPT algorithm to predict resource allocation, building a task graph, acquiring YARN cluster resources in real time, counting the number of executed tasks, calculating the amount of executable tasks, and formulating a scheduling strategy accordingly. It not only ensures that high-priority tasks are executed first, but also effectively avoids resource overload and waste through accurate resource prediction and dynamic scheduling strategies, improves resource utilization and task execution efficiency, and at the same time, updates task status in real time, enhancing the transparency and manageability of the system.
[0095] In another application scenario, Figure 4 A flowchart of a resource scheduling method for YARN tasks is shown. Figure 5 A schematic diagram showing the construction of a task relationship graph.
[0096] like Figure 4 As shown, resource scheduling of YARN tasks can include the following steps:
[0097] Step 410: Build a task manager.
[0098] Specifically, the task manager is initialized to prepare for subsequent task management and status updates.
[0099] Step 420: Store task information in Redis.
[0100] Specifically, the relevant information of each task (such as parent tasks, dependencies, current status, etc.) is stored in the Redis database. Redis is very suitable for such frequent reading and writing and fast query scenarios with its high performance and flexible data structure.
[0101] Step 430: Query and sort parent node tasks.
[0102] Specifically, the task configuration table is queried through SQL to find all tasks whose parent nodes are empty (i.e., top-level tasks), and then they are sorted in descending order according to conditions such as task priority, required memory resources, and required number of CPU cores to determine the execution order of these top-level tasks.
[0103] Step 440: Execute the task and update the status.
[0104] Specifically, the tasks are executed in the order determined in step 430. Before executing each task, the setTaskStatus() method is called to update the status of the task to "executing" or other corresponding status, and the status information is saved in Redis.
[0105] Step 450: Check task dependencies.
[0106] Specifically, before executing a task, you need to check whether the task depends on the completion of other tasks. This usually involves querying the task dependencies stored in Redis to confirm that all dependent tasks have been completed.
[0107] Step 460: Calculate the number of executable tasks and idle resources.
[0108] Specifically, during the execution of tasks, the idle resources of the cluster (such as the number of CPU cores, memory, etc.) are continuously monitored, and the number of tasks that can be executed is calculated based on the number of tasks currently being executed and the amount of idle resources. This step helps to avoid resource overload and ensure the smooth execution of tasks.
[0109] Step 470: Repeat steps 440 to 460.
[0110] Specifically, for the remaining tasks, steps S4 to S6 are repeatedly executed until all tasks are completed or a certain termination condition (such as resource exhaustion, reaching a time limit, etc.) is reached.
[0111] In the above process, the entire process ends when all tasks are completed or the termination conditions are reached. At this time, the status of all tasks has been updated to Redis, and the final status and related information of the task can be obtained by querying Redis.
[0112] It should be noted that step 450 and step 460 need to be repeated during the task execution process to ensure the correct execution of the task and the rational use of resources. In addition, the specific implementation details (such as how to call the setTaskStatus() method, how to query the task information in Redis, etc.) vary according to the actual application scenario and the technology stack used.
[0113] like Figure 5 As shown, to build a task relationship diagram, you first need to configure the basic information and associations of the tasks. In the configuration interface, users need to set the basic information of the new task, including task type, task priority, task quantity, etc. Users also need to configure the complex hierarchical associations of tasks. These relationships will be used for subsequent task scheduling and execution. Then the system uses the GPT algorithm model to allocate appropriate resources to each task, such as memory size and number of CPU cores, based on the task requirements and the current resource status of the cluster.
[0114] Furthermore, the TaskNode class is defined to represent the task node, including task ID, dependent task list, task status and other information; the TaskGraph class is defined to build and manage the task graph. Through the hierarchical relationship configured in the task, the task association relationship is converted into a tree-like task graph and stored in the task graph data set. The task graph is used as the basis for subsequent task execution to ensure that tasks are executed in the correct order.
[0115] Furthermore, the YARN resource monitor is used to obtain the resource status of the cluster, including the total number of CPUs, the number of idle CPUs, the total memory, the idle memory, the total number of tasks, etc. The key-value pairs with specific prefixes in Redis are used to count the number of tasks that are currently in the "IN_PROGRESS" state. The total number of tasks in the YARN resource minus the number of currently executed tasks is subtracted to calculate the total number of tasks that can be executed by the YARN resource.
[0116] Furthermore, according to factors such as cluster resource conditions and task priority, a task scheduling strategy is formulated to manage the order of task execution to ensure that tasks can be executed efficiently and orderly. The TaskManager class is defined to manage the status and dependencies of tasks, and the task's superior dependencies, status and other information are stored in redis so that it can quickly determine whether the task meets the execution conditions during execution. Functions such as adding tasks, checking whether task dependencies are completed, and setting task status are provided.
[0117] Furthermore, in the task manager, according to the task graph and task status, select the executable root task or the subtask that meets the execution conditions, allocate resources to the selected task, and execute the task. During the task execution, continuously update the task status and check whether there are new tasks that can be executed. During the task execution, continuously monitor the task status and update the task status information to redis as needed. When the task is completed, set its status to "COMPLETED" and trigger the execution of other tasks that depend on the task.
[0118] If an exception or failure occurs during the execution of a task, its status is set to "FAILED" and the relevant error information is recorded. The failed task is retried or otherwise processed as needed.
[0119] Through the above steps, the embodiment of the present invention realizes the refinement and intelligence of task management through detailed configuration of basic task information and setting of association relationships, combined with the precise resource allocation of the GPT algorithm model. The definition and application of TaskNode and TaskGraph ensure the accuracy of the task execution order. Combined with YARN resource monitoring and Redis real-time statistics, efficient scheduling of cluster resources and accurate calculation of task volume are achieved. The introduction of TaskManager not only improves the flexibility and response speed of task management, but also ensures the stability and reliability of task execution through continuous status monitoring and exception handling mechanisms, significantly improving the efficiency and success rate of task scheduling in cloud computing environments.
[0120] The following is an embodiment of the device of the present invention, which can be used to execute the resource scheduling method of the YARN task involved in the present invention. For details not disclosed in the embodiment of the device of the present invention, please refer to the method embodiment of the resource scheduling method of the YARN task involved in the present invention.
[0121] See also Figure 6 , an embodiment of the present invention provides a resource scheduling device 800 for a YARN task.
[0122] The resource scheduling device 800 for the YARN task includes but is not limited to: a data set preparation module 810 , a model training module 830 , a resource demand analysis module 850 and a resource scheduling module 870 .
[0123] The data set preparation module 810 is used to obtain relevant data of the YARN cluster and process the relevant data to obtain a training set and a test set.
[0124] The model training module 830 is used to train the GPT algorithm model using the training set through self-supervised learning, evaluate the GPT algorithm model using the test set, and optimize the GPT algorithm model based on the evaluation results.
[0125] The resource requirement analysis module 850 is used to obtain characteristic information of the target task and obtain the resource requirement of the target task according to the characteristic information through the GPT algorithm model.
[0126] The resource scheduling module 870 is used to formulate a resource scheduling strategy according to resource requirements and the resource status of the YARN cluster, and call the YARN cluster to execute the target task according to the scheduling strategy.
[0127] It should be noted that the resource scheduling of the YARN task provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the resource scheduling device of the YARN task will be divided into different functional modules to complete all or part of the functions described above.
[0128] In addition, the resource scheduling device for YARN tasks and the resource scheduling method for YARN tasks provided in the above embodiments belong to the same concept, and the specific manner in which each module performs operations has been described in detail in the method embodiments and will not be repeated here.
[0129] Figure 7 The present invention schematically illustrates the structure of an electronic device according to an exemplary embodiment.
[0130] It should be noted that the electronic device is only an example adapted to the present invention and cannot be considered to provide any limitation on the scope of use of the present invention. The electronic device cannot be interpreted as requiring dependence on or having Figure 7 One or more components of exemplary electronic device 2000 are shown.
[0131] The hardware structure of the electronic device 2000 may vary greatly due to differences in configuration or performance, such as Figure 7As shown, the electronic device 2000 includes: a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .
[0132] Specifically, the power supply 210 is used to provide operating voltage for each hardware device on the electronic device 2000 .
[0133] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Of course, in other examples adapted by the present invention, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, etc. Figure 7 As shown, this is not intended to be a specific limitation.
[0134] The memory 250 is a carrier for storing resources, which may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 251, an application 253 and data 255, etc. The storage method may be temporary storage or permanent storage.
[0135] Among them, the operating system 251 is used to manage and control various hardware devices and application programs 253 on the electronic device 2000 to enable the central processor 270 to calculate and process the massive data 255 in the memory 250. It can be WindowsServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
[0136] The application program 253 is a computer-readable instruction that performs at least one specific task based on the operating system 251, and may include at least one module ( Figure 7 (not shown), each module may respectively include computer-readable instructions for the electronic device 2000. For example, the resource scheduling device of the YARN task may be regarded as an application 253 deployed on the electronic device 2000.
[0137] The data 255 may be signal information, etc., stored in the memory 250 .
[0138] The central processor 270 may include one or more processors, and is configured to communicate with the memory 250 through at least one communication bus to read the computer-readable instructions stored in the memory 250, thereby realizing the operation and processing of the mass data 255 in the memory 250. For example, the resource scheduling method of the YARN task is completed by the central processor 270 reading a series of computer-readable instructions stored in the memory 250.
[0139] In addition, the present invention can also be implemented by hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present invention is not limited to any specific hardware circuits, software, or a combination of the two.
[0140] See also Figure 8 In an embodiment of the present invention, an electronic device 4000 is provided. The electronic device 4000 may include: a desktop computer, a laptop computer, a server, etc. with sensor recognition capability.
[0141] exist Figure 8 In the embodiment, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.
[0142] The data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0143] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception, etc. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.
[0144] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.
[0145] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 4000, but is not limited to this.
[0146] Computer-readable instructions are stored in the memory 4003 , and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002 .
[0147] The computer-readable instructions are executed by one or more processors 4001 to implement the resource scheduling method for YARN tasks in the above-mentioned embodiments.
[0148] In addition, an embodiment of the present invention provides a storage medium on which computer-readable instructions are stored. The computer-readable instructions are executed by one or more processors to implement the resource scheduling method for YARN tasks as described above.
[0149] A computer program product is provided in an embodiment of the present invention. The computer program product includes computer-readable instructions, which are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the resource scheduling method for YARN tasks as described above.
[0150] Compared with the related art, the present invention has the following beneficial effects:
[0151] 1. The present invention first obtains relevant data of the YARN cluster, processes the relevant data to obtain a training set and a test set, trains a GPT algorithm model using the training set in a self-supervised learning manner, evaluates the GPT algorithm model using the test set and optimizes the GPT algorithm model according to the evaluation result, obtains characteristic information of the target task, obtains the resource requirement of the target task according to the characteristic information through the GPT algorithm model, formulates a resource scheduling strategy according to the resource requirement and the resource status of the YARN cluster, calls the YARN cluster to execute the target task according to the scheduling strategy, optimizes the resource scheduling strategy by combining YARN cluster resource management, data processing, self-supervised learning and the GPT algorithm model, improves the efficiency of target task execution and resource utilization in the cloud computing environment, can formulate a more reasonable resource scheduling strategy, helps to reduce task waiting time and execution time, and improves the efficiency and performance of task execution, thereby effectively solving the problems that the prior art cannot cope with complex and changeable tasks and resource requirements, and has uneven resource allocation and low utilization.
[0152] 2. The present invention uses the GPT algorithm model to perform deep learning and prediction on historical tasks in the YARN cluster, and combines the tree-structured task scheduling rules to achieve reasonable scheduling and efficient utilization of resources.
[0153] 3. The present invention can realize reasonable resource scheduling: the GPT algorithm model can deeply learn and understand the relationship between tasks and resources, realize reasonable resource demand analysis, and improve the rationality and effectiveness of resource scheduling; it provides a tree-structured task scheduling rule: in the task scheduling process, a task graph is constructed according to the upper and lower associations of tasks, and the parallel or serial execution mode of tasks is confirmed between task graphs, and the reasonable allocation and scheduling of upper and lower tasks can be seamlessly connected through the task graph, thereby effectively reducing the waste and idleness of resources, and thus achieving the improvement of the overall utilization rate of resources; it can realize efficient resource utilization: by combining the resource prediction of the GPT algorithm model, in the process of reasonable and seamless scheduling of tasks, it can reduce the waste and idleness of resources, thereby improving the overall utilization rate and performance of resources; it has strong adaptability: the scheduling rules with a tree structure can meet various complex task scheduling scenarios.
[0154] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0155] The above descriptions are only partial embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.
Claims
1. A resource scheduling method for YARN tasks, characterized in that: The method comprises: Obtain relevant data of the YARN cluster, and process the relevant data to obtain a training set and a test set; the relevant data includes task data, resource usage data, and status data; Using the training set to train the GPT algorithm model in a self-supervised learning manner, using the test set to evaluate the GPT algorithm model, and optimizing the GPT algorithm model according to the evaluation results; Acquire characteristic information of the target task, and obtain resource requirements of the target task according to the characteristic information through the GPT algorithm model; the characteristic information includes task type, data size and task dependency; A resource scheduling strategy is formulated according to the resource demand and the resource status of the YARN cluster, and the YARN cluster is called to execute the target task according to the scheduling strategy; the resource scheduling strategy includes the priority of the task, the allocation order and quantity of resources.
2. The resource scheduling method for YARN tasks according to claim 1, characterized in that: The step of processing the relevant data to obtain a training set and a test set includes: Clean the relevant data, remove noise and outliers, and format the relevant data so that the format of the relevant data meets the requirements of the GPT algorithm model; Extract key features of the relevant data, and use the processed relevant data and key features as training sets and test sets; the key features include task type, data size, and correlation between tasks.
3. The resource scheduling method for YARN tasks according to claim 1, characterized in that: The obtaining the resource requirement of the target task according to the feature information by using the GPT algorithm model includes: Input the feature information into the GPT algorithm model to predict the required resource amount of the target task; the required resource amount includes CPU and memory; The priority, resource allocation order and quantity of the target task are set according to the priority, task association relationship and required resource quantity of the target task and serve as the resource requirement of the target task.
4. The resource scheduling method for YARN tasks according to claim 1, characterized in that: The formulating a resource scheduling strategy according to the resource demand and the resource status of the YARN cluster includes: Obtain resource usage and resource availability of the YARN cluster; the resource usage includes the total amount, allocated amount, and free amount of each resource; A resource scheduling strategy is formulated for the target task according to the resource demand of the target task and the resource usage and resource availability of the YARN cluster so that resource allocation can meet the demand of the target task without any resource waste.
5. The resource scheduling method for YARN tasks according to claim 1, characterized in that: The calling the YARN cluster to execute the target task according to the scheduling strategy includes: Calling the YARN cluster to execute the target task according to the scheduling strategy, recording the execution process and execution result of the target task and detecting whether it is executed normally; If an error occurs during task execution, the scheduling strategy is adjusted and optimized in a timely manner to execute the target task.
6. The resource scheduling method for YARN tasks according to claim 5, characterized in that: After executing the target task, the method includes: Continuously observe the resource usage of the YARN cluster and the execution of the target task, wherein the execution status includes the execution progress and resource usage of the target task; The GPT algorithm model is updated using the resource usage of the YARN cluster and the execution status of the target task.
7. The resource scheduling method for YARN tasks according to claim 1, characterized in that: The method further comprises: A tree-like task graph is constructed and stored according to the characteristic relationship of the target task; the root node of the tree-like task represents a task without a preceding dependency; Before executing the target task, it is determined that the dependency of each task is satisfied. When executing the target task, it is performed in the order of the tree-like task diagram. After the target task is executed, the node status is updated.
8. A resource scheduling device for YARN tasks, characterized in that: The device comprises: The data set preparation module is used to obtain relevant data of the YARN cluster and process the relevant data to obtain a training set and a test set; A model training module, used to train the GPT algorithm model using the training set in a self-supervised learning manner, evaluate the GPT algorithm model using the test set, and optimize the GPT algorithm model according to the evaluation result; A resource requirement analysis module, used to obtain characteristic information of a target task, and obtain resource requirements of the target task according to the characteristic information through the GPT algorithm model; The resource scheduling module is used to formulate a resource scheduling strategy according to the resource demand and the resource status of the YARN cluster, and call the YARN cluster to execute the target task according to the scheduling strategy.
9. An electronic device, characterized in that: include: at least one processor and at least one memory, wherein: The memory has computer-readable instructions stored thereon; The computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the resource scheduling method for YARN tasks as described in any one of claims 1 to 7.
10. A storage medium having computer-readable instructions stored thereon, characterized in that: The computer-readable instructions are executed by one or more processors to implement the resource scheduling method for YARN tasks as described in any one of claims 1 to 7.