Resource scheduling method and device and computer equipment
Through the preset model, the implementation information of the full-link task is analyzed and the resource scheduling strategy is dynamically adjusted, which solves the lack of flexibility and efficiency of the resource scheduling system, and achieves more efficient resource utilization and task execution.
Patent Information
- Application Number
- CN202411245218.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-05
- Publication Date
- 2024-12-24
AI Technical Summary
The existing resource scheduling system lacks flexibility in resource allocation and is difficult to dynamically adjust according to real-time load changes and task requirements, resulting in low resource utilization, complex task dependencies and difficult to effectively manage, and serious resource competition, resulting in delays or failures in task execution. .
The execution information of the full-link task is analyzed using a preset model, including a first sub-model for predicting resource requirements, a second sub-model for determining the alarm mechanism, and a third sub-model for determining the resource scheduling strategy. Through these sub-models, dynamically adjust resource scheduling strategies, optimize task execution order and resource allocation, and avoid resource competition.
It improves the flexibility and efficiency of resource scheduling, ensures that tasks are executed in the correct order, optimizes resource utilization, reduces resource competition and waste, and improves the overall performance of the system.
Smart Images
Figure CN119179573A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a resource scheduling method, device and computer equipment. Background Art
[0002] At present, digitalization plays an increasingly important role. It involves data integration, processing, and full-link data processing for service provision. The task nodes in the entire data processing task link will be relatively long and the number of nodes will be relatively large, and some links will even reach thousands of nodes. To ensure the efficient and stable operation of all nodes on the link, the most important thing is to ensure the effective scheduling of resources. To achieve effective resource scheduling, we are currently facing the following problems: Resources cannot be adjusted dynamically: Many existing scheduling systems lack flexibility in resource allocation, and it is difficult to dynamically adjust according to real-time load changes and task requirements. This leads to low resource utilization and the inability to give full play to the performance advantages of the system. Complex task dependencies: The dependencies between data processing tasks are complex, and the scheduling system needs to accurately manage these dependencies to ensure that tasks are executed in the correct order. If resources are not properly scheduled, task execution may be delayed or failed. Resource competition: When resources are limited, multiple tasks may compete for the same resources, resulting in resource contention and conflict. Existing scheduling systems often lack effective resource allocation strategies, making it difficult to balance resource requirements, resulting in some tasks not being processed in a timely manner. Summary of the invention
[0003] The embodiments of the present application provide a resource scheduling method, apparatus, and computer device to at least solve the technical problem of low resource scheduling flexibility in the related art.
[0004] According to one aspect of an embodiment of the present application, a resource scheduling method is provided, comprising: obtaining execution information of a full-link task to be scheduled, wherein the execution information includes at least: the upstream and downstream task relationship of the full-link task, the resource information consumed during the execution of the full-link task, and the execution duration of the full-link task; using a preset model to analyze the execution information of the full-link task to obtain a resource scheduling strategy for executing the full-link task, wherein the preset model includes at least: a first sub-model for predicting the resources required to be consumed during the execution of the full-link task, a second sub-model for determining an alarm mechanism during the execution of the task, and a third sub-model for determining a resource scheduling strategy; scheduling the resources required during the execution of the full-link task according to the resource scheduling strategy of the full-link task.
[0005] Optionally, a preset model is used to analyze the execution information of the full-link task, including: obtaining a training data set, wherein the training data set includes: static information and dynamic information, the static information at least includes: the upstream and downstream task relationships of historical tasks, and the dynamic information at least includes: the resource information consumed during the execution of historical tasks and the task execution time; using the training data set to train a preset initial model to obtain the preset model; using multiple sub-models in the preset model to analyze the execution information of the full-link task.
[0006] Optionally, multiple sub-models in the preset model are used to analyze the execution information of the full-link task to obtain a resource scheduling strategy for executing the full-link task, including: using the first sub-model to analyze the execution information of the full-link task to be scheduled to obtain the amount of resources required to be consumed by the full-link task in each time period during the execution process; using the second sub-model to analyze the upstream and downstream task relationships of the full-link task and the resource information consumed by the full-link task during the execution process, determine the target sub-task in the full-link task that will cause resource competition, and generate alarm information, the resource competition situation includes: multiple sub-tasks consume the same type of resources in the same time period, and the alarm information is used to indicate the target sub-task and the execution time period of the target sub-task; using the third sub-model to analyze the amount of resources required to be consumed in each time period during the execution of the full-link task and the alarm information, to determine the resource scheduling strategy of the full-link task.
[0007] Optionally, after determining the resource scheduling strategy of the full-link task, the method also includes: using the fourth sub-model in the preset model to analyze the upstream and downstream task relationships of the full-link task, the alarm information and the resource scheduling strategy of the full-link task to obtain the execution order of each sub-task in the full-link task.
[0008] Optionally, the fourth sub-model is determined in the following manner, including: constructing a directed graph based on the training data set, in which nodes represent tasks and edges represent dependencies between tasks; performing feature extraction on each node in the directed graph to obtain a feature vector for each node, wherein the feature vector of each node is used to at least characterize the duration, resource requirement and execution priority of the task; based on the feature vector of each node, adjusting the directed graph with the goal of minimizing resource consumption in each time period of task execution until the fourth sub-model converges.
[0009] Optionally, the method also includes: using a pre-trained fifth sub-model to analyze the amount of resources required to be consumed in each time period during the execution of the full-link task, selecting a computing node with the lowest resource utilization from multiple computing nodes as the target computing node, and deploying the full-link task chain to the target computing node for execution.
[0010] Optionally, the method further includes: obtaining resource utilization of the multiple computing nodes during the execution of the full-link task; and expanding the capacity of the multiple computing nodes when the resource utilization of the multiple computing nodes is higher than a preset threshold.
[0011] According to another aspect of an embodiment of the present application, a resource scheduling device is also provided, including: an acquisition module, used to acquire execution information of a full-link task to be scheduled, wherein the execution information includes at least: the upstream and downstream task relationship of the full-link task, the resource information consumed during the execution of the full-link task, and the execution duration of the full-link task; an analysis module, used to analyze the execution information of the full-link task using a preset model to obtain a resource scheduling strategy for executing the full-link task, wherein the preset model includes at least: a first sub-model for predicting the resources required to be consumed during the execution of the full-link task, a second sub-model for determining an alarm mechanism during the execution of the task, and a third sub-model for determining a resource scheduling strategy; a scheduling module, used to schedule the resources required during the execution of the full-link task according to the resource scheduling strategy of the full-link task.
[0012] According to another aspect of the embodiment of the present application, a computer device is provided, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor is connected to the memory and is used to execute the above-mentioned resource scheduling method.
[0013] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned resource scheduling method by running the computer program.
[0014] According to another aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, which implement the above-mentioned resource scheduling method when executed by a processor.
[0015] In an embodiment of the present application, by obtaining the execution information of the full-link task to be scheduled, wherein the execution information includes at least: the upstream and downstream task relationship of the full-link task, the resource information consumed by the full-link task during execution, and the execution duration of the full-link task; using a preset model to analyze the execution information of the full-link task, and obtaining a resource scheduling strategy for executing the full-link task, wherein the preset model includes at least: a first sub-model for predicting the resources required to be consumed during the execution of the full-link task, a second sub-model for determining the alarm mechanism during the task execution, and a third sub-model for determining the resource scheduling strategy; according to the resource scheduling strategy of the full-link task, the resources required during the execution of the full-link task are scheduled, thereby achieving the purpose of dynamically adjusting the resource scheduling strategy according to the execution information of the full-link task using a preset model containing multiple sub-models, thereby achieving the technical effect of improving the flexibility of resource scheduling, and then solving the technical problem of low resource scheduling flexibility in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0017] Figure 1 It is a hardware structure block diagram of a computer terminal for implementing a resource scheduling method according to an embodiment of the present application;
[0018] Figure 2 is a flow chart of a resource scheduling method according to an embodiment of the present application;
[0019] Figure 3 is a flow chart of another resource scheduling method according to an embodiment of the present application;
[0020] Figure 4 It is a structural diagram of a resource scheduling device according to an embodiment of the present application. DETAILED DESCRIPTION
[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0023] The information collected in the embodiments of the present application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or reject automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.
[0024] In order to solve the problems existing in the related art, the embodiment of the present application provides a resource scheduling method, which can be run on Figure 1 In the computer terminal shown, the computer terminal is explained below.
[0025] The resource scheduling method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a resource scheduling method. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1Different configurations are shown.
[0026] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).
[0027] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the resource scheduling method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned resource scheduling method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0028] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0029] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .
[0030] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.
[0031] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a resource scheduling method. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0032] Figure 2 is a flow chart of a resource scheduling method according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:
[0033] Step S202, acquiring execution information of the full-link task to be scheduled, wherein the execution information at least includes: the upstream and downstream task relationship of the full-link task, the resource information consumed during the execution of the full-link task, and the execution duration of the full-link task;
[0034] In the above step S202, the full link task is used to describe the entire process or task chain from the starting point to the end point in a certain field or system, wherein the full link task contains multiple subtask nodes, and the upstream and downstream task relationship contains the dependencies between multiple subtasks. The dependency between tasks means that in the process of completing a project or a series of tasks, some tasks need to be started after other tasks are completed. This relationship is usually manifested as Task A must be performed after Task B is completed, or Task C needs to wait for the result of Task D before it can be started. Dependencies can be mandatory or optional.
[0035] Step S204, using a preset model to analyze the execution information of the full-link task to obtain a resource scheduling strategy for executing the full-link task, wherein the preset model at least includes: a first sub-model for predicting the resources required to be consumed during the execution of the full-link task, a second sub-model for determining an alarm mechanism during the task execution, and a third sub-model for determining a resource scheduling strategy;
[0036] In the above step S204, the first sub-model can be an LSTM (Long Short-Term Memory) model, which is used to predict the resource requirements for executing tasks in a certain period of time in the future, and can monitor the resource usage and task abnormalities in real time; the second sub-model can be a Bayesian network model, and the established early warning mechanism is used to identify potential resource competition situations in advance, for example: multiple tasks use the same type of resources (CPU, memory, etc.) at the same time; the third sub-model can be a random forest model, which is used to determine the adjustment of resource configuration and scheduling strategies, and can also adjust the scheduling strategies in response to task execution.
[0037] Step S206: Scheduling the resources required during the execution of the full-link task according to the resource scheduling strategy of the full-link task.
[0038] Through the above steps S202 to S206, the execution information of the full-link task to be scheduled is obtained, wherein the execution information at least includes: the upstream and downstream task relationship of the full-link task, the resource information consumed by the full-link task during execution, and the execution duration of the full-link task; the execution information of the full-link task is analyzed by using a preset model to obtain a resource scheduling strategy for executing the full-link task, wherein the preset model at least includes: a first sub-model for predicting the resources required to be consumed during the execution of the full-link task, a second sub-model for determining the alarm mechanism during the task execution, and a third sub-model for determining the resource scheduling strategy; the resources required during the execution of the full-link task are scheduled according to the resource scheduling strategy of the full-link task, thereby achieving the purpose of dynamically adjusting the resource scheduling strategy according to the execution information of the full-link task using a preset model containing multiple sub-models, thereby achieving the technical effect of improving the flexibility of resource scheduling, and further solving the technical problem of low resource scheduling flexibility in related technologies. The following is a detailed description.
[0039] In step S204, the specific steps of using the preset model to analyze the execution information of the full-link task include: obtaining a training data set, wherein the training data set includes: static information and dynamic information, the static information at least includes: the upstream and downstream task relationships of historical tasks, and the dynamic information at least includes: the resource information consumed during the execution of historical tasks and the task execution time; using the training data set to train the preset initial model to obtain the preset model; using multiple sub-models in the preset model to analyze the execution information of the full-link task.
[0040] In actual application scenarios, static information includes task execution cycle, execution frequency, task type, task name, upstream and downstream task relationships, etc.; dynamic information includes CPU, memory, network bandwidth, load changes, execution time, execution time, and full-link execution time consumed by task running.
[0041] It should be noted that feature extraction tools can be used to extract features from the training data set to obtain time series features, task features, resource usage features, and task execution status features, among which the time series features are at least used to characterize the execution timing of the full-link task, the task features are at least used to characterize the dependency relationship between each subtask in the full-link task, the resource usage features are at least used to characterize the resource usage of the full-link task during the execution process, and the task execution status features are at least used to characterize the execution status of each task in the full-link task, for example: whether the full-link task is executed and completed in the correct order.
[0042] In some embodiments of the present application, multiple sub-models in the preset model are used to analyze the execution information of the full-link task, and the specific process of obtaining the resource scheduling strategy for executing the full-link task is as follows: the first sub-model is used to analyze the execution information of the full-link task to be scheduled, and the amount of resources required to be consumed by the full-link task in each time period during the execution process is obtained; the second sub-model is used to analyze the upstream and downstream task relationships of the full-link task and the resource information consumed by the full-link task during the execution process, determine the target sub-task in the full-link task that will cause resource competition, and generate alarm information, the resource competition situation includes: multiple sub-tasks consume the same type of resources in the same time period, and the alarm information is used to indicate the target sub-task and the execution time period of the target sub-task; the third sub-model is used to determine the resource scheduling strategy of the full-link task based on the amount of resources required to be consumed in each time period during the execution of the full-link task and the alarm information.
[0043] Taking the full-link task including subtasks A, B and C as an example, when the execution time period of A and B is the same and A and B both need to occupy memory during the execution process, A and B are considered to be target subtasks, and resource competition will occur. Then, an alarm message can be generated before A and B are executed. The alarm message indicates that there will be resource competition when A and B are executed; the third sub-model allocates corresponding resources according to the amount of resources required in each time period predicted by the first sub-model. Taking the above subtasks A and B as an example, the resource scheduling strategy is determined to schedule the predicted resource amount, and increase a preset amount of memory during the execution period of A and B.
[0044] After determining the resource scheduling strategy of the full-link task, the fourth sub-model in the preset model is used to analyze the upstream and downstream task relationships of the full-link task, the alarm information and the resource scheduling strategy of the full-link task to obtain the execution order of each sub-task in the full-link task.
[0045] It should be noted that the fourth sub-model can be GNN (Graph Neural Networks), which can be used to manage and optimize task dependencies to ensure that tasks are executed in the correct order.
[0046] Taking the full-link task containing subtasks A, B and C as an example, when there is no dependency between the execution of tasks A and B (for example, tasks A and B do not need to be executed at the same time), the execution order of tasks A and B can be adjusted. For example, the execution order of tasks A and B can be changed from simultaneous execution to sequential execution to avoid resource competition.
[0047] It should be noted that the fourth sub-model is determined in the following manner, including: constructing a directed graph based on the training data set, in which nodes represent tasks and edges represent dependencies between tasks; performing feature extraction on each node in the directed graph to obtain a feature vector for each node, wherein the feature vector of each node is used to at least characterize the duration, resource requirements and execution priority of the task; based on the feature vector of each node, adjusting the directed graph with the goal of minimizing resource consumption in each time period of task execution until the fourth sub-model converges.
[0048] In some embodiments of the present application, a pre-trained fifth sub-model can also be used to analyze the amount of resources consumed in each time period during the execution of the full-link task, select the computing node with the lowest resource utilization from multiple computing nodes as the target computing node, and deploy the full-link task chain to the target computing node for execution.
[0049] It should be noted that the fifth sub-model can be a Deep Q-Network model (a deep learning network model), which is used to determine the load balancing strategy, dynamically allocate tasks to different computing nodes, and combine with k8s to complete automatic elastic expansion and contraction to ensure efficient operation during peak and trough periods.
[0050] In an optional manner, the resource utilization of the multiple computing nodes during the execution of the full-link task can also be obtained; when the resource utilization of the multiple computing nodes is higher than a preset threshold, the multiple computing nodes are expanded.
[0051] After completing the model training and accuracy verification, the model can be enabled to the scheduling system, such as Figure 3 As shown, step 1 is to perform model training to obtain a preset model; step 2 is to input the model analysis results into the scheduling system, and use the scheduling system to dynamically adjust the resources required during task execution; step 4 is to collect the execution status of the task in real time; step 5 is to adjust the preset model based on the collected task execution status.
[0052] It can be understood that the real-time dynamic data includes: the task execution status during the execution of the task based on the task resource scheduling strategy, including: whether the task is executed normally, the amount of resources occupied during the task execution, etc.
[0053] In actual application scenarios, the execution of the task chain can be analyzed in real time. The model can be updated regularly by re-training the preset model.
[0054] The resource scheduling method provided by the present application collects static information of tasks, including task execution cycle, execution frequency, task type, task name, upstream and downstream task relationship, etc.; collects dynamic information of tasks, including CPU, memory, network bandwidth, load change, execution time, execution time, full-link execution time, etc. consumed by task operation; collects cluster resource information, including CPU, memory, and disk of the cluster. Better monitoring of resources. In the process of scheduling task trigger execution, from the very beginning, only task dependency or time dependency determines task triggering. This method improves the task execution link, and can trigger tasks according to factors such as the concurrency of tasks, whether they can be executed synchronously, and resource conditions, greatly improving the execution efficiency and resource utilization of tasks. The previous scheduling strategy would cause resource competition during the peak period of tasks, and some tasks with relatively large resource consumption could not be effectively monitored, thus occupying resources for a long time, resulting in some important and relatively small tasks being unable to be executed. Based on this method, the execution path of the task will be optimized according to the resources required by the task itself and the available resources of the cluster, and the competition for resources will be minimized as much as possible; at the same time, the problem of the inability to execute tasks due to insufficient resources after task submission can be solved. Due to insufficient resources, tasks cannot be executed normally and can only be queued. There is no load management during peak hours, and dynamic expansion and contraction cannot be performed, resulting in resource waste. Based on this method, the resources required during peak hours can be predicted in advance, and then based on k8s (Kubernetes, an open source container orchestration platform), resources can be expanded to automatically deploy, expand and manage containerized applications, ensure the normal execution of tasks, completely solve the problem of resource competition, and reduce resources during off-peak hours. This improves resource utilization and reduces resource waste.
[0055] Figure 4 is a structural diagram of a resource scheduling device according to an embodiment of the present application, such as Figure 4 As shown, the device comprises:
[0056] An acquisition module 40 is used to acquire execution information of the full-link task to be scheduled, wherein the execution information at least includes: the upstream and downstream task relationship of the full-link task, the resource information consumed during the execution of the full-link task, and the execution duration of the full-link task;
[0057] An analysis module 42 is used to analyze the execution information of the full-link task using a preset model to obtain a resource scheduling strategy for executing the full-link task, wherein the preset model at least includes: a first sub-model for predicting the resources required to be consumed during the execution of the full-link task, a second sub-model for determining an alarm mechanism during the task execution, and a third sub-model for determining a resource scheduling strategy;
[0058] The scheduling module 44 is used to schedule the resources required during the execution of the full-link task according to the resource scheduling strategy of the full-link task.
[0059] Through the above-mentioned resource scheduling device, by obtaining the execution information of the full-link task to be scheduled, wherein the execution information at least includes: the upstream and downstream task relationship of the full-link task, the resource information consumed by the full-link task during the execution process and the execution time of the full-link task; using a preset model to analyze the execution information of the full-link task, and obtaining a resource scheduling strategy for executing the full-link task, wherein the preset model at least includes: a first sub-model for predicting the resources required to be consumed during the execution of the full-link task, a second sub-model for determining the alarm mechanism during the task execution process, and a third sub-model for determining the resource scheduling strategy; according to the resource scheduling strategy of the full-link task, the resources required during the execution of the full-link task are scheduled, thereby achieving the purpose of dynamically adjusting the resource scheduling strategy according to the execution information of the full-link task by using a preset model containing multiple sub-models, thereby achieving the technical effect of improving the flexibility of resource scheduling, and then solving the technical problem of low resource scheduling flexibility in related technologies.
[0060] The analysis module 42 in the above-mentioned resource scheduling device includes: an analysis submodule, which is used to analyze the execution information of the full-link task using a preset model, including: obtaining a training data set, wherein the training data set includes: static information and dynamic information, the static information at least includes: the upstream and downstream task relationships of historical tasks, and the dynamic information at least includes: the resource information consumed during the execution of historical tasks and the task execution time; using the training data set to train the preset initial model to obtain the preset model; using multiple sub-models in the preset model to analyze the execution information of the full-link task.
[0061] The analysis submodule includes: an analysis unit, which is used to use multiple sub-models in the preset model to analyze the execution information of the full-link task, and obtain the resource scheduling strategy for executing the full-link task, including: using the first sub-model to analyze the execution information of the full-link task to be scheduled, and obtain the amount of resources required to be consumed by the full-link task in each time period during the execution process; using the second sub-model for the upstream and downstream task relationship of the full-link task and the resource information consumed by the full-link task during the execution process, determine the target sub-task in the full-link task that will cause resource competition, and generate alarm information, the resource competition situation includes: multiple sub-tasks consume the same type of resources in the same time period, and the alarm information is used to indicate the target sub-task and the execution time period of the target sub-task; using the third sub-model for the amount of resources required to be consumed in each time period during the execution of the full-link task and the alarm information, to determine the resource scheduling strategy of the full-link task.
[0062] The scheduling module 44 includes: an execution sub-module, which is used to, after determining the resource scheduling strategy of the full-link task, use the fourth sub-model in the preset model to analyze the upstream and downstream task relationships of the full-link task, the alarm information and the resource scheduling strategy of the full-link task, and obtain the execution order of each subtask in the full-link task.
[0063] The execution submodule includes: a training unit, which is used to train the fourth sub-model, including: constructing a directed graph based on the training data set, in which nodes represent tasks and edges represent dependencies between tasks; performing feature extraction on each node in the directed graph to obtain a feature vector for each node, wherein the feature vector of each node is used to characterize at least the duration, resource requirement and execution priority of the task; based on the feature vector of each node, adjusting the directed graph with the goal of minimizing resource consumption in each time period of task execution until the fourth sub-model converges.
[0064] The execution submodule also includes: a deployment unit, which is used to use a pre-trained fifth sub-model to analyze the amount of resources required to be consumed in each time period during the execution of the full-link task, select a computing node with the lowest resource utilization from multiple computing nodes as the target computing node, and deploy the full-link task chain to the target computing node for execution.
[0065] The training unit includes an expansion subunit, which is used to: obtain the resource utilization of the multiple computing nodes during the execution of the full-link task; and expand the target computing node when the resource utilization of the multiple computing nodes is higher than a preset threshold.
[0066] It should be noted that Figure 4 The resource scheduling device shown is used to execute Figure 2 The resource scheduling method shown, therefore the relevant explanations in the above resource scheduling method are also applicable to the resource scheduling device, and will not be repeated here.
[0067] An embodiment of the present application also provides a computer device, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor is connected to the memory and is used to execute the above-mentioned resource scheduling method.
[0068] An embodiment of the present application further provides a non-volatile storage medium, which includes a stored computer program, wherein a device where the non-volatile storage medium is located executes the above-mentioned resource scheduling method by running the computer program.
[0069] An embodiment of the present application also provides a computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the resource scheduling method in the present application are implemented.
[0070] An embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the resource scheduling method in the present application.
[0071] An embodiment of the present application also provides a computer program, which, when executed by a processor, implements the steps of the resource scheduling method in the present application.
[0072] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0073] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0074] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0075] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0076] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0077] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0078] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A resource scheduling method, characterized in that: include: Acquire execution information of the full-link task to be scheduled, wherein the execution information at least includes: the upstream and downstream task relationship of the full-link task, the resource information consumed during the execution of the full-link task, and the execution duration of the full-link task; The execution information of the full-link task is analyzed by using a preset model to obtain a resource scheduling strategy for executing the full-link task, wherein the preset model at least includes: a first sub-model for predicting the resources required to be consumed during the execution of the full-link task, a second sub-model for determining an alarm mechanism during the task execution, and a third sub-model for determining a resource scheduling strategy; The resources required during the execution of the full-link task are scheduled according to the resource scheduling strategy of the full-link task.
2. The method according to claim 1, characterized in that The execution information of the full-link task is analyzed using a preset model, including: Acquire a training data set, wherein the training data set includes: static information and dynamic information, the static information at least includes: upstream and downstream task relationships of historical tasks, and the dynamic information at least includes: resource information consumed during the execution of historical tasks and task execution time; Using the training data set to train a preset initial model to obtain the preset model; The execution information of the full-link task is analyzed using multiple sub-models in the preset model.
3. The method according to claim 2, characterized in that The execution information of the full-link task is analyzed by using multiple sub-models in the preset model to obtain a resource scheduling strategy for executing the full-link task, including: The first sub-model is used to analyze the execution information of the full-link task to be scheduled, so as to obtain the amount of resources required to be consumed by the full-link task in each time period during the execution process; The second sub-model is used to determine the target sub-task that will cause resource competition in the full-link task based on the upstream and downstream task relationship of the full-link task and the resource information consumed by the full-link task during execution, and generate alarm information, wherein the resource competition includes: multiple sub-tasks consume the same type of resources in the same period, and the alarm information is used to indicate the target sub-task and the execution period of the target sub-task; The third sub-model is used to determine the resource scheduling strategy of the full-link task based on the amount of resources consumed in each time period during the execution of the full-link task and the alarm information.
4. The method according to claim 3, characterized in that After determining the resource scheduling strategy of the full-link task, the method further includes: The fourth sub-model in the preset model is used to analyze the upstream and downstream task relationships of the full-link task, the alarm information and the resource scheduling strategy of the full-link task to obtain the execution order of each sub-task in the full-link task.
5. The method according to claim 4, characterized in that The fourth sub-model is determined by the following method, including: Constructing a directed graph according to the training data set, wherein the nodes represent tasks and the edges represent dependencies between the tasks; Extracting features from each node in the directed graph to obtain a feature vector for each node, wherein the feature vector for each node is used to characterize at least the duration, resource requirement, and execution priority of the task; Based on the feature vector of each node, the directed graph is adjusted with the goal of minimizing resource consumption in each time period of task execution until the fourth sub-model converges.
6. The method according to claim 3, characterized in that The method further comprises: The pre-trained fifth sub-model is used to analyze the amount of resources consumed in each time period during the execution of the full-link task, and the computing node with the lowest resource utilization is selected from multiple computing nodes to be determined as the target computing node, and the full-link task chain is deployed to the target computing node for execution.
7. The method according to claim 5, characterized in that The method further comprises: Obtain resource utilization rates of multiple computing nodes during the execution of the full-link task; When the resource utilization of the plurality of computing nodes is higher than a preset threshold, the plurality of computing nodes are expanded.
8. A resource scheduling device, characterized in that: include: An acquisition module, used to acquire execution information of the full-link task to be scheduled, wherein the execution information at least includes: the upstream and downstream task relationship of the full-link task, the resource information consumed during the execution of the full-link task, and the execution duration of the full-link task; An analysis module, used to analyze the execution information of the full-link task using a preset model to obtain a resource scheduling strategy for executing the full-link task, wherein the preset model at least includes: a first sub-model for predicting the resources required to be consumed during the execution of the full-link task, a second sub-model for determining an alarm mechanism during the task execution, and a third sub-model for determining a resource scheduling strategy; The scheduling module is used to schedule the resources required during the execution of the full-link task according to the resource scheduling strategy of the full-link task.
9. A computer device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; The processor is connected to the memory, and is used to execute the resource scheduling method described in any one of claims 1 to 7.
10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the resource scheduling method described in any one of claims 1 to 7 is implemented.